2024-03-28T15:35:28.953 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-28T15:35:28.954 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625322 --name mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/iozone}} --owner mchangir --verbose -- /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625322/orig.config.yaml 2024-03-28T15:35:29.007 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-28T15:35:29.097 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625322 branch: wip-mchangir-testing-main-20240318.032620 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/iozone}} email: mchangir@redhat.com first_in_suite: false job_id: '7625322' kernel: flavor: default kdb: true sha1: 1df0c7e8a2354c42d738e5454ba83ea7332deac2 last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-mchangir-testing-main-20240318.032620 ceph: cephfs: max_mds: 1 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - slow metadata IO - SLOW_OPS - slow request sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 subvols: create: 2 subvol_options: --size 25000000000 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: dry_run: true install: ceph: flavor: default sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 kclient: client.0: mount_subvol_num: 1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-mchangir-testing-main-20240318.032620 sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 owner: mchangir priority: 100 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 3645 sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 sleep_before_teardown: 0 subset: 132/256 suite: fs suite_branch: wip-mchangir-testing-main-20240318.032620 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi067.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJhDUpZ2IauSqSlufwftGvEaQ5Vkiy4L5JFlRhYpBj2aTCwofUuXYlSN6ib1OrRxcYTlXmvuJV4XyHSzUqmpg/w= smithi150.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKoK70VPdmC0/eaUeMW21Z+c7KnYoeDaBs5A+2KtrsJOjJKEZ7ly0mD2SHeQpR/0WAIJQPxBMGQVnQWzZw0qjng= smithi183.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEeOeHmr9R9+ygjpgtqLUQT+UiCsT5fjGaFjqEiHSMmwGHnuZq/Ym1IW+aDBvoirIO+eDo7o3EdxLx0EEdGZCP8= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - ceph-fuse: null - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - workunit: clients: all: - suites/iozone.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-27_03:45:23 tube: smithi user: mchangir verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-28T15:35:29.097 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa; will attempt to use it 2024-03-28T15:35:29.098 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks 2024-03-28T15:35:29.098 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-28T15:35:29.099 INFO:teuthology.task.internal:Checking packages... 2024-03-28T15:35:29.124 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '3be9a5c9bc793e11e2800a8c0c696e8b46742033' 2024-03-28T15:35:29.124 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-28T15:35:29.124 INFO:teuthology.packaging:ref: None 2024-03-28T15:35:29.124 INFO:teuthology.packaging:tag: None 2024-03-28T15:35:29.124 INFO:teuthology.packaging:branch: wip-mchangir-testing-main-20240318.032620 2024-03-28T15:35:29.124 INFO:teuthology.packaging:sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T15:35:29.124 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-mchangir-testing-main-20240318.032620 2024-03-28T15:35:29.356 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2145.g3be9a5c9 2024-03-28T15:35:29.357 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-28T15:35:29.367 INFO:teuthology.task.internal:no buildpackages task found 2024-03-28T15:35:29.367 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-28T15:35:29.380 INFO:teuthology.task.internal:Saving configuration 2024-03-28T15:35:29.399 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-28T15:35:29.407 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-28T15:35:29.432 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi067.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625322', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 15:30:11.382265', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJhDUpZ2IauSqSlufwftGvEaQ5Vkiy4L5JFlRhYpBj2aTCwofUuXYlSN6ib1OrRxcYTlXmvuJV4XyHSzUqmpg/w='} 2024-03-28T15:35:29.453 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi150.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625322', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 15:30:11.383491', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKoK70VPdmC0/eaUeMW21Z+c7KnYoeDaBs5A+2KtrsJOjJKEZ7ly0mD2SHeQpR/0WAIJQPxBMGQVnQWzZw0qjng='} 2024-03-28T15:35:29.478 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi183.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625322', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 15:30:11.380785', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEeOeHmr9R9+ygjpgtqLUQT+UiCsT5fjGaFjqEiHSMmwGHnuZq/Ym1IW+aDBvoirIO+eDo7o3EdxLx0EEdGZCP8='} 2024-03-28T15:35:29.478 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-28T15:35:29.487 INFO:teuthology.task.internal:roles: ubuntu@smithi067.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-03-28T15:35:29.487 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-03-28T15:35:29.487 INFO:teuthology.task.internal:roles: ubuntu@smithi183.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-03-28T15:35:29.487 INFO:teuthology.run_tasks:Running task console_log... 2024-03-28T15:35:29.622 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f3d61edd0d0>, signals=[15]) 2024-03-28T15:35:29.623 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-28T15:35:29.633 INFO:teuthology.task.internal:Opening connections... 2024-03-28T15:35:29.634 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi067.front.sepia.ceph.com 2024-03-28T15:35:29.636 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T15:35:29.741 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi150.front.sepia.ceph.com 2024-03-28T15:35:29.742 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T15:35:29.830 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi183.front.sepia.ceph.com 2024-03-28T15:35:29.831 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T15:35:29.918 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-28T15:35:29.927 DEBUG:teuthology.orchestra.run.smithi067:> uname -m 2024-03-28T15:35:29.946 INFO:teuthology.orchestra.run.smithi067.stdout:x86_64 2024-03-28T15:35:29.947 DEBUG:teuthology.orchestra.run.smithi067:> cat /etc/os-release 2024-03-28T15:35:30.004 INFO:teuthology.orchestra.run.smithi067.stdout:NAME="CentOS Stream" 2024-03-28T15:35:30.005 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION="9" 2024-03-28T15:35:30.005 INFO:teuthology.orchestra.run.smithi067.stdout:ID="centos" 2024-03-28T15:35:30.005 INFO:teuthology.orchestra.run.smithi067.stdout:ID_LIKE="rhel fedora" 2024-03-28T15:35:30.005 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION_ID="9" 2024-03-28T15:35:30.005 INFO:teuthology.orchestra.run.smithi067.stdout:PLATFORM_ID="platform:el9" 2024-03-28T15:35:30.005 INFO:teuthology.orchestra.run.smithi067.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-28T15:35:30.005 INFO:teuthology.orchestra.run.smithi067.stdout:ANSI_COLOR="0;31" 2024-03-28T15:35:30.005 INFO:teuthology.orchestra.run.smithi067.stdout:LOGO="fedora-logo-icon" 2024-03-28T15:35:30.005 INFO:teuthology.orchestra.run.smithi067.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-28T15:35:30.005 INFO:teuthology.orchestra.run.smithi067.stdout:HOME_URL="https://centos.org/" 2024-03-28T15:35:30.005 INFO:teuthology.orchestra.run.smithi067.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-28T15:35:30.005 INFO:teuthology.orchestra.run.smithi067.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-28T15:35:30.006 INFO:teuthology.orchestra.run.smithi067.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-28T15:35:30.006 INFO:teuthology.lock.ops:Updating smithi067.front.sepia.ceph.com on lock server 2024-03-28T15:35:30.031 DEBUG:teuthology.orchestra.run.smithi150:> uname -m 2024-03-28T15:35:30.048 INFO:teuthology.orchestra.run.smithi150.stdout:x86_64 2024-03-28T15:35:30.049 DEBUG:teuthology.orchestra.run.smithi150:> cat /etc/os-release 2024-03-28T15:35:30.106 INFO:teuthology.orchestra.run.smithi150.stdout:NAME="CentOS Stream" 2024-03-28T15:35:30.106 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION="9" 2024-03-28T15:35:30.106 INFO:teuthology.orchestra.run.smithi150.stdout:ID="centos" 2024-03-28T15:35:30.106 INFO:teuthology.orchestra.run.smithi150.stdout:ID_LIKE="rhel fedora" 2024-03-28T15:35:30.106 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION_ID="9" 2024-03-28T15:35:30.106 INFO:teuthology.orchestra.run.smithi150.stdout:PLATFORM_ID="platform:el9" 2024-03-28T15:35:30.106 INFO:teuthology.orchestra.run.smithi150.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-28T15:35:30.106 INFO:teuthology.orchestra.run.smithi150.stdout:ANSI_COLOR="0;31" 2024-03-28T15:35:30.106 INFO:teuthology.orchestra.run.smithi150.stdout:LOGO="fedora-logo-icon" 2024-03-28T15:35:30.106 INFO:teuthology.orchestra.run.smithi150.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-28T15:35:30.106 INFO:teuthology.orchestra.run.smithi150.stdout:HOME_URL="https://centos.org/" 2024-03-28T15:35:30.106 INFO:teuthology.orchestra.run.smithi150.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-28T15:35:30.107 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-28T15:35:30.107 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-28T15:35:30.107 INFO:teuthology.lock.ops:Updating smithi150.front.sepia.ceph.com on lock server 2024-03-28T15:35:30.130 DEBUG:teuthology.orchestra.run.smithi183:> uname -m 2024-03-28T15:35:30.148 INFO:teuthology.orchestra.run.smithi183.stdout:x86_64 2024-03-28T15:35:30.148 DEBUG:teuthology.orchestra.run.smithi183:> cat /etc/os-release 2024-03-28T15:35:30.204 INFO:teuthology.orchestra.run.smithi183.stdout:NAME="CentOS Stream" 2024-03-28T15:35:30.204 INFO:teuthology.orchestra.run.smithi183.stdout:VERSION="9" 2024-03-28T15:35:30.204 INFO:teuthology.orchestra.run.smithi183.stdout:ID="centos" 2024-03-28T15:35:30.204 INFO:teuthology.orchestra.run.smithi183.stdout:ID_LIKE="rhel fedora" 2024-03-28T15:35:30.204 INFO:teuthology.orchestra.run.smithi183.stdout:VERSION_ID="9" 2024-03-28T15:35:30.205 INFO:teuthology.orchestra.run.smithi183.stdout:PLATFORM_ID="platform:el9" 2024-03-28T15:35:30.205 INFO:teuthology.orchestra.run.smithi183.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-28T15:35:30.205 INFO:teuthology.orchestra.run.smithi183.stdout:ANSI_COLOR="0;31" 2024-03-28T15:35:30.205 INFO:teuthology.orchestra.run.smithi183.stdout:LOGO="fedora-logo-icon" 2024-03-28T15:35:30.205 INFO:teuthology.orchestra.run.smithi183.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-28T15:35:30.205 INFO:teuthology.orchestra.run.smithi183.stdout:HOME_URL="https://centos.org/" 2024-03-28T15:35:30.205 INFO:teuthology.orchestra.run.smithi183.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-28T15:35:30.205 INFO:teuthology.orchestra.run.smithi183.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-28T15:35:30.205 INFO:teuthology.orchestra.run.smithi183.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-28T15:35:30.205 INFO:teuthology.lock.ops:Updating smithi183.front.sepia.ceph.com on lock server 2024-03-28T15:35:30.230 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-28T15:35:30.240 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-28T15:35:30.250 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-28T15:35:30.250 DEBUG:teuthology.orchestra.run.smithi067:> test '!' -e /home/ubuntu/cephtest 2024-03-28T15:35:30.253 DEBUG:teuthology.orchestra.run.smithi150:> test '!' -e /home/ubuntu/cephtest 2024-03-28T15:35:30.255 DEBUG:teuthology.orchestra.run.smithi183:> test '!' -e /home/ubuntu/cephtest 2024-03-28T15:35:30.270 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-28T15:35:30.281 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-28T15:35:30.281 DEBUG:teuthology.orchestra.run.smithi067:> test -z $(ls -A /var/lib/ceph) 2024-03-28T15:35:30.310 DEBUG:teuthology.orchestra.run.smithi150:> test -z $(ls -A /var/lib/ceph) 2024-03-28T15:35:30.312 DEBUG:teuthology.orchestra.run.smithi183:> test -z $(ls -A /var/lib/ceph) 2024-03-28T15:35:30.422 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-28T15:35:30.503 INFO:teuthology.run_tasks:Running task kernel... 2024-03-28T15:35:30.522 INFO:teuthology.task.kernel:normalize config orig: {'flavor': 'default', 'kdb': True, 'sha1': '1df0c7e8a2354c42d738e5454ba83ea7332deac2'} 2024-03-28T15:35:30.522 INFO:teuthology.task.kernel:config {'mon.a': {'flavor': 'default', 'kdb': True, 'sha1': '1df0c7e8a2354c42d738e5454ba83ea7332deac2'}, 'mon.b': {'flavor': 'default', 'kdb': True, 'sha1': '1df0c7e8a2354c42d738e5454ba83ea7332deac2'}, 'mon.c': {'flavor': 'default', 'kdb': True, 'sha1': '1df0c7e8a2354c42d738e5454ba83ea7332deac2'}}, timeout 300 2024-03-28T15:35:30.522 DEBUG:teuthology.orchestra.run.smithi067:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-28T15:35:30.523 DEBUG:teuthology.orchestra.run.smithi150:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-28T15:35:30.523 DEBUG:teuthology.orchestra.run.smithi183:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-28T15:35:30.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T15:35:30.539 DEBUG:teuthology.task.kernel:sha1 for mon.b is 1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-28T15:35:30.540 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "1df0c7e8a2354c42d738e5454ba83ea7332deac2"... 2024-03-28T15:35:30.540 DEBUG:teuthology.orchestra.run.smithi150:> uname -r 2024-03-28T15:35:30.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T15:35:30.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T15:35:30.541 DEBUG:teuthology.task.kernel:sha1 for mon.a is 1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-28T15:35:30.542 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "1df0c7e8a2354c42d738e5454ba83ea7332deac2"... 2024-03-28T15:35:30.542 DEBUG:teuthology.orchestra.run.smithi067:> uname -r 2024-03-28T15:35:30.542 DEBUG:teuthology.task.kernel:sha1 for mon.c is 1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-28T15:35:30.542 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "1df0c7e8a2354c42d738e5454ba83ea7332deac2"... 2024-03-28T15:35:30.542 DEBUG:teuthology.orchestra.run.smithi183:> uname -r 2024-03-28T15:35:30.596 INFO:teuthology.orchestra.run.smithi150.stdout:5.14.0-361.el9.x86_64 2024-03-28T15:35:30.596 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-28T15:35:30.596 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-03-28T15:35:30.597 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-28T15:35:30.601 INFO:teuthology.orchestra.run.smithi067.stdout:5.14.0-361.el9.x86_64 2024-03-28T15:35:30.601 INFO:teuthology.orchestra.run.smithi183.stdout:5.14.0-361.el9.x86_64 2024-03-28T15:35:30.601 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-28T15:35:30.601 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-03-28T15:35:30.601 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-28T15:35:30.602 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-28T15:35:30.602 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-03-28T15:35:30.603 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-28T15:35:30.741 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum upgrade -y linux-firmware 2024-03-28T15:35:30.745 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum upgrade -y linux-firmware 2024-03-28T15:35:30.750 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum upgrade -y linux-firmware 2024-03-28T15:35:32.725 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-28T15:35:32.858 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-28T15:35:32.923 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-28T15:35:34.126 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS-9 - BaseOS 6.6 MB/s | 2.0 MB 00:00 2024-03-28T15:35:34.325 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS-9 - BaseOS 6.6 MB/s | 2.0 MB 00:00 2024-03-28T15:35:34.515 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS-9 - BaseOS 4.9 MB/s | 2.0 MB 00:00 2024-03-28T15:35:35.041 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - BaseOS 22 MB/s | 8.0 MB 00:00 2024-03-28T15:35:35.347 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.0 MB 00:00 2024-03-28T15:35:36.817 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - AppStream 29 MB/s | 19 MB 00:00 2024-03-28T15:35:37.132 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - AppStream 25 MB/s | 19 MB 00:00 2024-03-28T15:35:41.191 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.1 MB 00:00 2024-03-28T15:35:41.276 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - CRB 7.3 MB/s | 6.1 MB 00:00 2024-03-28T15:35:42.583 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - Extras packages 66 kB/s | 16 kB 00:00 2024-03-28T15:35:42.993 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - Extras packages 30 kB/s | 16 kB 00:00 2024-03-28T15:35:43.340 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - BaseOS 927 kB/s | 8.0 MB 00:08 2024-03-28T15:35:43.678 INFO:teuthology.orchestra.run.smithi067.stdout:Extra Packages for Enterprise Linux 36 MB/s | 21 MB 00:00 2024-03-28T15:35:44.054 INFO:teuthology.orchestra.run.smithi150.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-03-28T15:35:48.845 INFO:teuthology.orchestra.run.smithi067.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-03-28T15:35:49.001 INFO:teuthology.orchestra.run.smithi150.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-03-28T15:35:51.223 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-03-28T15:35:51.232 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-28T15:35:51.232 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-03-28T15:35:51.232 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-28T15:35:51.232 INFO:teuthology.orchestra.run.smithi067.stdout:Upgrading: 2024-03-28T15:35:51.232 INFO:teuthology.orchestra.run.smithi067.stdout: linux-firmware noarch 20240219-143.el9 CentOS-BaseOS 375 M 2024-03-28T15:35:51.233 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T15:35:51.233 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-03-28T15:35:51.233 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-28T15:35:51.233 INFO:teuthology.orchestra.run.smithi067.stdout:Upgrade 1 Package 2024-03-28T15:35:51.233 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T15:35:51.233 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 375 M 2024-03-28T15:35:51.233 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-03-28T15:35:51.263 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-03-28T15:35:51.278 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-03-28T15:35:51.278 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2024-03-28T15:35:51.278 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-03-28T15:35:51.278 INFO:teuthology.orchestra.run.smithi150.stdout:Upgrading: 2024-03-28T15:35:51.278 INFO:teuthology.orchestra.run.smithi150.stdout: linux-firmware noarch 20240219-143.el9 CentOS-BaseOS 375 M 2024-03-28T15:35:51.278 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-28T15:35:51.278 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-03-28T15:35:51.278 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-03-28T15:35:51.278 INFO:teuthology.orchestra.run.smithi150.stdout:Upgrade 1 Package 2024-03-28T15:35:51.279 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-28T15:35:51.279 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 375 M 2024-03-28T15:35:51.279 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-03-28T15:36:01.679 INFO:teuthology.orchestra.run.smithi067.stdout:linux-firmware-20240219-143.el9.noarch.rpm 36 MB/s | 375 MB 00:10 2024-03-28T15:36:01.680 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-03-28T15:36:01.680 INFO:teuthology.orchestra.run.smithi067.stdout:Total 36 MB/s | 375 MB 00:10 2024-03-28T15:36:01.681 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-03-28T15:36:01.782 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-03-28T15:36:01.782 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-03-28T15:36:01.921 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - AppStream 1.1 MB/s | 19 MB 00:17 2024-03-28T15:36:03.452 INFO:teuthology.orchestra.run.smithi150.stdout:linux-firmware-20240219-143.el9.noarch.rpm 31 MB/s | 375 MB 00:12 2024-03-28T15:36:03.452 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-03-28T15:36:03.453 INFO:teuthology.orchestra.run.smithi150.stdout:Total 31 MB/s | 375 MB 00:12 2024-03-28T15:36:03.453 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-03-28T15:36:03.551 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-03-28T15:36:03.551 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-03-28T15:36:04.263 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-03-28T15:36:04.263 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-03-28T15:36:06.025 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-03-28T15:36:06.025 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-03-28T15:36:08.113 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - CRB 2.3 MB/s | 6.1 MB 00:02 2024-03-28T15:36:09.631 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - Extras packages 41 kB/s | 16 kB 00:00 2024-03-28T15:36:10.699 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-03-28T15:36:10.918 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-28T15:36:10.934 INFO:teuthology.orchestra.run.smithi183.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-03-28T15:36:11.006 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-28T15:36:12.422 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-03-28T15:36:12.422 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-28T15:36:12.436 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-03-28T15:36:12.647 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-28T15:36:12.735 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-28T15:36:12.815 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-28T15:36:12.815 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T15:36:12.815 INFO:teuthology.orchestra.run.smithi067.stdout:Upgraded: 2024-03-28T15:36:12.815 INFO:teuthology.orchestra.run.smithi067.stdout: linux-firmware-20240219-143.el9.noarch 2024-03-28T15:36:12.815 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T15:36:12.816 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-03-28T15:36:13.098 INFO:teuthology.task.kernel:Downloading kernel 1df0c7e8a2354c42d738e5454ba83ea7332deac2 on mon.a... 2024-03-28T15:36:13.098 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-28T15:36:13.319 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/centos/9/flavors/default/x86_64/ 2024-03-28T15:36:13.319 DEBUG:teuthology.orchestra.run.smithi067:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.8.0_g1df0c7e8a235-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/centos/9/flavors/default/x86_64/ --input-file=- 2024-03-28T15:36:13.322 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi067.front.sepia.ceph.com to complete... 2024-03-28T15:36:13.939 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-03-28T15:36:13.939 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-28T15:36:14.351 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-28T15:36:14.351 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-28T15:36:14.351 INFO:teuthology.orchestra.run.smithi150.stdout:Upgraded: 2024-03-28T15:36:14.352 INFO:teuthology.orchestra.run.smithi150.stdout: linux-firmware-20240219-143.el9.noarch 2024-03-28T15:36:14.352 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-28T15:36:14.352 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-03-28T15:36:14.633 INFO:teuthology.task.kernel:Downloading kernel 1df0c7e8a2354c42d738e5454ba83ea7332deac2 on mon.b... 2024-03-28T15:36:14.633 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-28T15:36:14.784 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/centos/9/flavors/default/x86_64/ 2024-03-28T15:36:14.784 DEBUG:teuthology.orchestra.run.smithi150:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.8.0_g1df0c7e8a235-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/centos/9/flavors/default/x86_64/ --input-file=- 2024-03-28T15:36:14.786 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi150.front.sepia.ceph.com to complete... 2024-03-28T15:36:15.798 INFO:teuthology.orchestra.run.smithi183.stdout:lab-extras 38 kB/s | 1.7 kB 00:00 2024-03-28T15:36:17.972 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-28T15:36:17.982 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-28T15:36:17.983 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-03-28T15:36:17.983 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-28T15:36:17.983 INFO:teuthology.orchestra.run.smithi183.stdout:Upgrading: 2024-03-28T15:36:17.983 INFO:teuthology.orchestra.run.smithi183.stdout: linux-firmware noarch 20240219-143.el9 CentOS-BaseOS 375 M 2024-03-28T15:36:17.983 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T15:36:17.983 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-28T15:36:17.983 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-28T15:36:17.983 INFO:teuthology.orchestra.run.smithi183.stdout:Upgrade 1 Package 2024-03-28T15:36:17.983 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T15:36:17.983 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 375 M 2024-03-28T15:36:17.983 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-28T15:36:30.848 INFO:teuthology.orchestra.run.smithi183.stdout:linux-firmware-20240219-143.el9.noarch.rpm 29 MB/s | 375 MB 00:12 2024-03-28T15:36:30.849 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-28T15:36:30.849 INFO:teuthology.orchestra.run.smithi183.stdout:Total 29 MB/s | 375 MB 00:12 2024-03-28T15:36:30.849 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-28T15:36:30.937 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-28T15:36:30.937 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-28T15:36:32.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28 15:36:32 URL:https://3.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/centos/9/flavors/default/x86_64/kernel-6.8.0_g1df0c7e8a235-1.x86_64.rpm [336317861/336317861] -> "/tmp/kernel.x86_64.rpm" [1] 2024-03-28T15:36:32.364 INFO:teuthology.orchestra.run.smithi067.stderr:FINISHED --2024-03-28 15:36:32-- 2024-03-28T15:36:32.364 INFO:teuthology.orchestra.run.smithi067.stderr:Total wall clock time: 19s 2024-03-28T15:36:32.364 INFO:teuthology.orchestra.run.smithi067.stderr:Downloaded: 1 files, 321M in 19s (17.2 MB/s) 2024-03-28T15:36:32.367 INFO:teuthology.task.kernel:Installing kernel 1df0c7e8a2354c42d738e5454ba83ea7332deac2 on mon.a... 2024-03-28T15:36:32.367 DEBUG:teuthology.orchestra.run.smithi067:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-03-28T15:36:33.304 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-28T15:36:33.305 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-28T15:36:34.393 INFO:teuthology.orchestra.run.smithi067.stdout:Verifying... ######################################## 2024-03-28T15:36:34.450 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing... ######################################## 2024-03-28T15:36:34.534 INFO:teuthology.orchestra.run.smithi067.stdout:Updating / installing... 2024-03-28T15:36:36.195 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28 15:36:36 URL:https://3.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/centos/9/flavors/default/x86_64/kernel-6.8.0_g1df0c7e8a235-1.x86_64.rpm [336317861/336317861] -> "/tmp/kernel.x86_64.rpm" [1] 2024-03-28T15:36:36.195 INFO:teuthology.orchestra.run.smithi150.stderr:FINISHED --2024-03-28 15:36:36-- 2024-03-28T15:36:36.195 INFO:teuthology.orchestra.run.smithi150.stderr:Total wall clock time: 21s 2024-03-28T15:36:36.195 INFO:teuthology.orchestra.run.smithi150.stderr:Downloaded: 1 files, 321M in 21s (15.4 MB/s) 2024-03-28T15:36:36.198 INFO:teuthology.task.kernel:Installing kernel 1df0c7e8a2354c42d738e5454ba83ea7332deac2 on mon.b... 2024-03-28T15:36:36.199 DEBUG:teuthology.orchestra.run.smithi150:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-03-28T15:36:38.132 INFO:teuthology.orchestra.run.smithi150.stdout:Verifying... ######################################## 2024-03-28T15:36:38.181 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing... ######################################## 2024-03-28T15:36:38.260 INFO:teuthology.orchestra.run.smithi150.stdout:Updating / installing... 2024-03-28T15:36:39.803 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-28T15:36:40.090 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-28T15:36:40.172 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-28T15:36:41.391 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-03-28T15:36:41.391 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-28T15:36:41.800 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-28T15:36:41.800 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T15:36:41.800 INFO:teuthology.orchestra.run.smithi183.stdout:Upgraded: 2024-03-28T15:36:41.800 INFO:teuthology.orchestra.run.smithi183.stdout: linux-firmware-20240219-143.el9.noarch 2024-03-28T15:36:41.800 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T15:36:41.800 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-28T15:36:42.079 INFO:teuthology.task.kernel:Downloading kernel 1df0c7e8a2354c42d738e5454ba83ea7332deac2 on mon.c... 2024-03-28T15:36:42.079 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-28T15:36:42.228 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/centos/9/flavors/default/x86_64/ 2024-03-28T15:36:42.229 DEBUG:teuthology.orchestra.run.smithi183:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.8.0_g1df0c7e8a235-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/centos/9/flavors/default/x86_64/ --input-file=- 2024-03-28T15:36:42.232 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi183.front.sepia.ceph.com to complete... 2024-03-28T15:36:42.352 INFO:teuthology.orchestra.run.smithi067.stdout:kernel-6.8.0_g1df0c7e8a235-1 ######################################## 2024-03-28T15:36:45.780 INFO:teuthology.orchestra.run.smithi150.stdout:kernel-6.8.0_g1df0c7e8a235-1 ######################################## 2024-03-28T15:36:54.067 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28 15:36:54 URL:https://3.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/centos/9/flavors/default/x86_64/kernel-6.8.0_g1df0c7e8a235-1.x86_64.rpm [336317861/336317861] -> "/tmp/kernel.x86_64.rpm" [1] 2024-03-28T15:36:54.067 INFO:teuthology.orchestra.run.smithi183.stderr:FINISHED --2024-03-28 15:36:54-- 2024-03-28T15:36:54.067 INFO:teuthology.orchestra.run.smithi183.stderr:Total wall clock time: 12s 2024-03-28T15:36:54.067 INFO:teuthology.orchestra.run.smithi183.stderr:Downloaded: 1 files, 321M in 11s (28.0 MB/s) 2024-03-28T15:36:54.071 INFO:teuthology.task.kernel:Installing kernel 1df0c7e8a2354c42d738e5454ba83ea7332deac2 on mon.c... 2024-03-28T15:36:54.071 DEBUG:teuthology.orchestra.run.smithi183:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-03-28T15:36:55.998 INFO:teuthology.orchestra.run.smithi183.stdout:Verifying... ######################################## 2024-03-28T15:36:56.048 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing... ######################################## 2024-03-28T15:36:56.128 INFO:teuthology.orchestra.run.smithi183.stdout:Updating / installing... 2024-03-28T15:36:58.141 INFO:teuthology.orchestra.run.smithi067.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-28T15:37:01.635 INFO:teuthology.orchestra.run.smithi150.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-28T15:37:03.609 INFO:teuthology.orchestra.run.smithi183.stdout:kernel-6.8.0_g1df0c7e8a235-1 ######################################## 2024-03-28T15:37:19.115 INFO:teuthology.orchestra.run.smithi183.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-28T15:37:21.392 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi067.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-03-28T15:37:21.392 DEBUG:teuthology.orchestra.run.smithi067:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-03-28T15:37:21.464 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235 2024-03-28T15:37:21.464 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/System.map 2024-03-28T15:37:21.464 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/config 2024-03-28T15:37:21.465 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel 2024-03-28T15:37:21.465 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch 2024-03-28T15:37:21.465 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86 2024-03-28T15:37:21.465 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto 2024-03-28T15:37:21.465 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-03-28T15:37:21.465 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-03-28T15:37:21.465 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-03-28T15:37:21.465 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-03-28T15:37:21.465 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-03-28T15:37:21.465 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-03-28T15:37:21.465 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-03-28T15:37:21.465 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/crc32c-intel.ko 2024-03-28T15:37:21.465 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-03-28T15:37:21.465 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-03-28T15:37:21.466 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-03-28T15:37:21.466 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-03-28T15:37:21.466 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-avx2.ko 2024-03-28T15:37:21.466 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-03-28T15:37:21.466 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-03-28T15:37:21.466 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-03-28T15:37:21.466 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-03-28T15:37:21.466 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-03-28T15:37:21.466 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-03-28T15:37:21.466 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-03-28T15:37:21.466 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel 2024-03-28T15:37:21.466 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu 2024-03-28T15:37:21.466 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu/mce 2024-03-28T15:37:21.466 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-03-28T15:37:21.467 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm 2024-03-28T15:37:21.467 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm-amd.ko 2024-03-28T15:37:21.467 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm-intel.ko 2024-03-28T15:37:21.467 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm.ko 2024-03-28T15:37:21.467 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto 2024-03-28T15:37:21.467 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/ansi_cprng.ko 2024-03-28T15:37:21.467 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/anubis.ko 2024-03-28T15:37:21.467 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/arc4.ko 2024-03-28T15:37:21.467 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx 2024-03-28T15:37:21.467 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_memcpy.ko 2024-03-28T15:37:21.467 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_pq.ko 2024-03-28T15:37:21.467 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_raid6_recov.ko 2024-03-28T15:37:21.467 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_tx.ko 2024-03-28T15:37:21.467 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_xor.ko 2024-03-28T15:37:21.468 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/raid6test.ko 2024-03-28T15:37:21.468 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/authenc.ko 2024-03-28T15:37:21.468 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/authencesn.ko 2024-03-28T15:37:21.468 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blake2b_generic.ko 2024-03-28T15:37:21.468 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blowfish_common.ko 2024-03-28T15:37:21.468 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blowfish_generic.ko 2024-03-28T15:37:21.468 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/camellia_generic.ko 2024-03-28T15:37:21.468 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast5_generic.ko 2024-03-28T15:37:21.468 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast6_generic.ko 2024-03-28T15:37:21.468 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast_common.ko 2024-03-28T15:37:21.468 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/ccm.ko 2024-03-28T15:37:21.468 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cmac.ko 2024-03-28T15:37:21.468 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crc32_generic.ko 2024-03-28T15:37:21.469 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crypto_engine.ko 2024-03-28T15:37:21.469 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crypto_user.ko 2024-03-28T15:37:21.469 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cts.ko 2024-03-28T15:37:21.469 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/des_generic.ko 2024-03-28T15:37:21.469 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/dh_generic.ko 2024-03-28T15:37:21.469 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/echainiv.ko 2024-03-28T15:37:21.469 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/essiv.ko 2024-03-28T15:37:21.469 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/fcrypt.ko 2024-03-28T15:37:21.469 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/khazad.ko 2024-03-28T15:37:21.469 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/lz4.ko 2024-03-28T15:37:21.469 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/lz4hc.ko 2024-03-28T15:37:21.469 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/md4.ko 2024-03-28T15:37:21.469 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/michael_mic.ko 2024-03-28T15:37:21.469 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/pcbc.ko 2024-03-28T15:37:21.470 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/pcrypt.ko 2024-03-28T15:37:21.470 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/rmd160.ko 2024-03-28T15:37:21.470 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/seed.ko 2024-03-28T15:37:21.470 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/serpent_generic.ko 2024-03-28T15:37:21.470 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/tcrypt.ko 2024-03-28T15:37:21.470 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/tea.ko 2024-03-28T15:37:21.470 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/twofish_common.ko 2024-03-28T15:37:21.470 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/twofish_generic.ko 2024-03-28T15:37:21.470 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/vmac.ko 2024-03-28T15:37:21.470 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/wp512.ko 2024-03-28T15:37:21.470 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xcbc.ko 2024-03-28T15:37:21.470 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xor.ko 2024-03-28T15:37:21.470 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xxhash_generic.ko 2024-03-28T15:37:21.470 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers 2024-03-28T15:37:21.471 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi 2024-03-28T15:37:21.471 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/acpi_ipmi.ko 2024-03-28T15:37:21.471 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/acpi_pad.ko 2024-03-28T15:37:21.471 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/custom_method.ko 2024-03-28T15:37:21.471 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/ec_sys.ko 2024-03-28T15:37:21.471 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/platform_profile.ko 2024-03-28T15:37:21.471 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/sbs.ko 2024-03-28T15:37:21.471 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/sbshc.ko 2024-03-28T15:37:21.471 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/video.ko 2024-03-28T15:37:21.471 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata 2024-03-28T15:37:21.471 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/acard-ahci.ko 2024-03-28T15:37:21.471 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/ahci_platform.ko 2024-03-28T15:37:21.471 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/ata_generic.ko 2024-03-28T15:37:21.471 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/libahci_platform.ko 2024-03-28T15:37:21.471 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_acpi.ko 2024-03-28T15:37:21.472 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ali.ko 2024-03-28T15:37:21.472 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_amd.ko 2024-03-28T15:37:21.472 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_artop.ko 2024-03-28T15:37:21.472 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_atiixp.ko 2024-03-28T15:37:21.472 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_atp867x.ko 2024-03-28T15:37:21.472 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cmd640.ko 2024-03-28T15:37:21.472 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cmd64x.ko 2024-03-28T15:37:21.472 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cypress.ko 2024-03-28T15:37:21.472 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_efar.ko 2024-03-28T15:37:21.472 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt366.ko 2024-03-28T15:37:21.472 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt37x.ko 2024-03-28T15:37:21.472 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt3x2n.ko 2024-03-28T15:37:21.472 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt3x3.ko 2024-03-28T15:37:21.472 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_it8213.ko 2024-03-28T15:37:21.472 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_it821x.ko 2024-03-28T15:37:21.472 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_jmicron.ko 2024-03-28T15:37:21.473 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_marvell.ko 2024-03-28T15:37:21.473 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_mpiix.ko 2024-03-28T15:37:21.473 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_netcell.ko 2024-03-28T15:37:21.473 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ninja32.ko 2024-03-28T15:37:21.473 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ns87410.ko 2024-03-28T15:37:21.473 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ns87415.ko 2024-03-28T15:37:21.473 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_oldpiix.ko 2024-03-28T15:37:21.473 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_opti.ko 2024-03-28T15:37:21.473 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_optidma.ko 2024-03-28T15:37:21.473 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pcmcia.ko 2024-03-28T15:37:21.473 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pdc2027x.ko 2024-03-28T15:37:21.473 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-03-28T15:37:21.473 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_piccolo.ko 2024-03-28T15:37:21.473 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_rdc.ko 2024-03-28T15:37:21.473 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sch.ko 2024-03-28T15:37:21.474 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_serverworks.ko 2024-03-28T15:37:21.474 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sil680.ko 2024-03-28T15:37:21.474 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sis.ko 2024-03-28T15:37:21.474 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sl82c105.ko 2024-03-28T15:37:21.474 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_triflex.ko 2024-03-28T15:37:21.474 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_via.ko 2024-03-28T15:37:21.474 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pdc_adma.ko 2024-03-28T15:37:21.474 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_inic162x.ko 2024-03-28T15:37:21.474 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_mv.ko 2024-03-28T15:37:21.474 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_nv.ko 2024-03-28T15:37:21.474 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_promise.ko 2024-03-28T15:37:21.474 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_qstor.ko 2024-03-28T15:37:21.474 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sil.ko 2024-03-28T15:37:21.474 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sil24.ko 2024-03-28T15:37:21.475 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sis.ko 2024-03-28T15:37:21.475 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_svw.ko 2024-03-28T15:37:21.475 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sx4.ko 2024-03-28T15:37:21.475 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_uli.ko 2024-03-28T15:37:21.475 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_via.ko 2024-03-28T15:37:21.475 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_vsc.ko 2024-03-28T15:37:21.475 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/auxdisplay 2024-03-28T15:37:21.475 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/auxdisplay/ks0108.ko 2024-03-28T15:37:21.475 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base 2024-03-28T15:37:21.475 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base/regmap 2024-03-28T15:37:21.475 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base/regmap/regmap-i2c.ko 2024-03-28T15:37:21.475 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base/regmap/regmap-mmio.ko 2024-03-28T15:37:21.475 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/bcma 2024-03-28T15:37:21.475 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/bcma/bcma.ko 2024-03-28T15:37:21.476 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block 2024-03-28T15:37:21.476 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/aoe 2024-03-28T15:37:21.476 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/aoe/aoe.ko 2024-03-28T15:37:21.476 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/brd.ko 2024-03-28T15:37:21.476 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/drbd 2024-03-28T15:37:21.476 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/drbd/drbd.ko 2024-03-28T15:37:21.476 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/floppy.ko 2024-03-28T15:37:21.476 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/loop.ko 2024-03-28T15:37:21.476 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/mtip32xx 2024-03-28T15:37:21.476 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-03-28T15:37:21.476 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/nbd.ko 2024-03-28T15:37:21.476 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/null_blk 2024-03-28T15:37:21.476 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/null_blk/null_blk.ko 2024-03-28T15:37:21.476 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/pktcdvd.ko 2024-03-28T15:37:21.476 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/rbd.ko 2024-03-28T15:37:21.477 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/virtio_blk.ko 2024-03-28T15:37:21.477 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/xen-blkback 2024-03-28T15:37:21.477 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-03-28T15:37:21.477 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/xen-blkfront.ko 2024-03-28T15:37:21.477 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/zram 2024-03-28T15:37:21.477 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/zram/zram.ko 2024-03-28T15:37:21.477 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char 2024-03-28T15:37:21.477 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hangcheck-timer.ko 2024-03-28T15:37:21.477 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random 2024-03-28T15:37:21.477 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/amd-rng.ko 2024-03-28T15:37:21.477 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/intel-rng.ko 2024-03-28T15:37:21.477 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-03-28T15:37:21.477 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/via-rng.ko 2024-03-28T15:37:21.478 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/virtio-rng.ko 2024-03-28T15:37:21.478 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi 2024-03-28T15:37:21.478 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-03-28T15:37:21.478 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-03-28T15:37:21.478 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-03-28T15:37:21.478 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_si.ko 2024-03-28T15:37:21.478 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-03-28T15:37:21.478 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-03-28T15:37:21.478 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/lp.ko 2024-03-28T15:37:21.478 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/mwave 2024-03-28T15:37:21.478 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/mwave/mwave.ko 2024-03-28T15:37:21.478 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ppdev.ko 2024-03-28T15:37:21.478 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tlclk.ko 2024-03-28T15:37:21.478 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm 2024-03-28T15:37:21.479 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm.ko 2024-03-28T15:37:21.479 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_atmel.ko 2024-03-28T15:37:21.479 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_infineon.ko 2024-03-28T15:37:21.479 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_nsc.ko 2024-03-28T15:37:21.479 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_tis.ko 2024-03-28T15:37:21.479 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-03-28T15:37:21.479 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/uv_mmtimer.ko 2024-03-28T15:37:21.479 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/virtio_console.ko 2024-03-28T15:37:21.479 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq 2024-03-28T15:37:21.479 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-03-28T15:37:21.479 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-03-28T15:37:21.479 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/p4-clockmod.ko 2024-03-28T15:37:21.479 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-03-28T15:37:21.480 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/powernow-k8.ko 2024-03-28T15:37:21.480 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/speedstep-lib.ko 2024-03-28T15:37:21.480 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto 2024-03-28T15:37:21.480 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp 2024-03-28T15:37:21.480 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-03-28T15:37:21.480 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp/ccp.ko 2024-03-28T15:37:21.480 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel 2024-03-28T15:37:21.480 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat 2024-03-28T15:37:21.480 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_common 2024-03-28T15:37:21.480 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-03-28T15:37:21.480 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-03-28T15:37:21.480 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-03-28T15:37:21.480 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/padlock-aes.ko 2024-03-28T15:37:21.480 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/padlock-sha.ko 2024-03-28T15:37:21.480 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/virtio 2024-03-28T15:37:21.481 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-03-28T15:37:21.481 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dca 2024-03-28T15:37:21.481 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dca/dca.ko 2024-03-28T15:37:21.481 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/devfreq 2024-03-28T15:37:21.481 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-03-28T15:37:21.481 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma 2024-03-28T15:37:21.481 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/dw 2024-03-28T15:37:21.481 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/dw/dw_dmac.ko 2024-03-28T15:37:21.481 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/ioat 2024-03-28T15:37:21.481 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/ioat/ioatdma.ko 2024-03-28T15:37:21.481 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac 2024-03-28T15:37:21.481 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/amd64_edac.ko 2024-03-28T15:37:21.481 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/e752x_edac.ko 2024-03-28T15:37:21.481 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/edac_mce_amd.ko 2024-03-28T15:37:21.481 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i3000_edac.ko 2024-03-28T15:37:21.482 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i3200_edac.ko 2024-03-28T15:37:21.482 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i5100_edac.ko 2024-03-28T15:37:21.482 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i5400_edac.ko 2024-03-28T15:37:21.482 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i7300_edac.ko 2024-03-28T15:37:21.482 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i7core_edac.ko 2024-03-28T15:37:21.482 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i82975x_edac.ko 2024-03-28T15:37:21.482 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/ie31200_edac.ko 2024-03-28T15:37:21.482 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/sb_edac.ko 2024-03-28T15:37:21.482 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/x38_edac.ko 2024-03-28T15:37:21.482 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware 2024-03-28T15:37:21.482 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware/edd.ko 2024-03-28T15:37:21.482 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware/iscsi_ibft.ko 2024-03-28T15:37:21.482 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio 2024-03-28T15:37:21.482 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-ich.ko 2024-03-28T15:37:21.483 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-viperboard.ko 2024-03-28T15:37:21.483 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid 2024-03-28T15:37:21.483 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-appleir.ko 2024-03-28T15:37:21.483 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-aureal.ko 2024-03-28T15:37:21.483 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-axff.ko 2024-03-28T15:37:21.483 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-dr.ko 2024-03-28T15:37:21.483 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-elecom.ko 2024-03-28T15:37:21.483 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-elo.ko 2024-03-28T15:37:21.483 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-emsff.ko 2024-03-28T15:37:21.483 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gaff.ko 2024-03-28T15:37:21.483 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gt683r.ko 2024-03-28T15:37:21.483 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gyration.ko 2024-03-28T15:37:21.483 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtek-kbd.ko 2024-03-28T15:37:21.483 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtek-mouse.ko 2024-03-28T15:37:21.483 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtekff.ko 2024-03-28T15:37:21.484 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-hyperv.ko 2024-03-28T15:37:21.484 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-icade.ko 2024-03-28T15:37:21.484 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-keytouch.ko 2024-03-28T15:37:21.484 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-kye.ko 2024-03-28T15:37:21.484 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-lcpower.ko 2024-03-28T15:37:21.484 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-led.ko 2024-03-28T15:37:21.484 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-lenovo.ko 2024-03-28T15:37:21.484 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-logitech-dj.ko 2024-03-28T15:37:21.484 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-03-28T15:37:21.484 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-multitouch.ko 2024-03-28T15:37:21.484 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-ortek.ko 2024-03-28T15:37:21.484 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-penmount.ko 2024-03-28T15:37:21.484 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-petalynx.ko 2024-03-28T15:37:21.485 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-picolcd.ko 2024-03-28T15:37:21.485 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-pl.ko 2024-03-28T15:37:21.485 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-plantronics.ko 2024-03-28T15:37:21.485 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-primax.ko 2024-03-28T15:37:21.485 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-rmi.ko 2024-03-28T15:37:21.485 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-arvo.ko 2024-03-28T15:37:21.485 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-common.ko 2024-03-28T15:37:21.485 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-isku.ko 2024-03-28T15:37:21.485 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-kone.ko 2024-03-28T15:37:21.485 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-03-28T15:37:21.485 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-konepure.ko 2024-03-28T15:37:21.485 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-03-28T15:37:21.485 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-lua.ko 2024-03-28T15:37:21.485 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-pyra.ko 2024-03-28T15:37:21.486 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-ryos.ko 2024-03-28T15:37:21.486 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-savu.ko 2024-03-28T15:37:21.486 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat.ko 2024-03-28T15:37:21.486 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-saitek.ko 2024-03-28T15:37:21.486 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-samsung.ko 2024-03-28T15:37:21.486 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sensor-hub.ko 2024-03-28T15:37:21.486 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sjoy.ko 2024-03-28T15:37:21.486 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sony.ko 2024-03-28T15:37:21.486 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-speedlink.ko 2024-03-28T15:37:21.486 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-steelseries.ko 2024-03-28T15:37:21.486 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sunplus.ko 2024-03-28T15:37:21.486 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-thrustmaster.ko 2024-03-28T15:37:21.486 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-tivo.ko 2024-03-28T15:37:21.487 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-tmff.ko 2024-03-28T15:37:21.487 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-topseed.ko 2024-03-28T15:37:21.487 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-twinhan.ko 2024-03-28T15:37:21.487 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-uclogic.ko 2024-03-28T15:37:21.487 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-waltop.ko 2024-03-28T15:37:21.487 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-wiimote.ko 2024-03-28T15:37:21.487 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-xinmo.ko 2024-03-28T15:37:21.487 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-zpff.ko 2024-03-28T15:37:21.487 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-zydacron.ko 2024-03-28T15:37:21.487 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/uhid.ko 2024-03-28T15:37:21.487 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/wacom.ko 2024-03-28T15:37:21.487 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv 2024-03-28T15:37:21.487 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_balloon.ko 2024-03-28T15:37:21.488 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_utils.ko 2024-03-28T15:37:21.488 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_vmbus.ko 2024-03-28T15:37:21.488 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon 2024-03-28T15:37:21.488 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/abituguru.ko 2024-03-28T15:37:21.488 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/abituguru3.ko 2024-03-28T15:37:21.488 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/acpi_power_meter.ko 2024-03-28T15:37:21.488 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ad7414.ko 2024-03-28T15:37:21.488 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ad7418.ko 2024-03-28T15:37:21.488 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adc128d818.ko 2024-03-28T15:37:21.488 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1025.ko 2024-03-28T15:37:21.488 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1026.ko 2024-03-28T15:37:21.488 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1029.ko 2024-03-28T15:37:21.489 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1031.ko 2024-03-28T15:37:21.489 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm9240.ko 2024-03-28T15:37:21.489 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ads7828.ko 2024-03-28T15:37:21.489 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7410.ko 2024-03-28T15:37:21.489 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7411.ko 2024-03-28T15:37:21.489 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7462.ko 2024-03-28T15:37:21.489 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7470.ko 2024-03-28T15:37:21.489 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7475.ko 2024-03-28T15:37:21.489 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7x10.ko 2024-03-28T15:37:21.489 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/amc6821.ko 2024-03-28T15:37:21.489 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/applesmc.ko 2024-03-28T15:37:21.489 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asb100.ko 2024-03-28T15:37:21.489 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asc7621.ko 2024-03-28T15:37:21.490 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asus_atk0110.ko 2024-03-28T15:37:21.490 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/atxp1.ko 2024-03-28T15:37:21.490 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/coretemp.ko 2024-03-28T15:37:21.490 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-03-28T15:37:21.490 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/dme1737.ko 2024-03-28T15:37:21.490 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ds1621.ko 2024-03-28T15:37:21.490 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ds620.ko 2024-03-28T15:37:21.490 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/emc1403.ko 2024-03-28T15:37:21.490 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/emc6w201.ko 2024-03-28T15:37:21.490 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f71805f.ko 2024-03-28T15:37:21.490 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f71882fg.ko 2024-03-28T15:37:21.490 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f75375s.ko 2024-03-28T15:37:21.491 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/fam15h_power.ko 2024-03-28T15:37:21.491 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/fschmd.ko 2024-03-28T15:37:21.491 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/g760a.ko 2024-03-28T15:37:21.491 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/g762.ko 2024-03-28T15:37:21.491 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/gl518sm.ko 2024-03-28T15:37:21.491 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/gl520sm.ko 2024-03-28T15:37:21.491 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/hwmon-vid.ko 2024-03-28T15:37:21.491 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/i5500_temp.ko 2024-03-28T15:37:21.491 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/i5k_amb.ko 2024-03-28T15:37:21.491 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ibmaem.ko 2024-03-28T15:37:21.491 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ibmpex.ko 2024-03-28T15:37:21.491 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ina209.ko 2024-03-28T15:37:21.492 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ina2xx.ko 2024-03-28T15:37:21.492 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/it87.ko 2024-03-28T15:37:21.492 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/k10temp.ko 2024-03-28T15:37:21.492 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/k8temp.ko 2024-03-28T15:37:21.492 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lineage-pem.ko 2024-03-28T15:37:21.492 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm63.ko 2024-03-28T15:37:21.492 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm73.ko 2024-03-28T15:37:21.492 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm75.ko 2024-03-28T15:37:21.492 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm77.ko 2024-03-28T15:37:21.492 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm78.ko 2024-03-28T15:37:21.492 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm80.ko 2024-03-28T15:37:21.492 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm83.ko 2024-03-28T15:37:21.493 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm85.ko 2024-03-28T15:37:21.493 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm87.ko 2024-03-28T15:37:21.493 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm90.ko 2024-03-28T15:37:21.493 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm92.ko 2024-03-28T15:37:21.493 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm93.ko 2024-03-28T15:37:21.493 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95234.ko 2024-03-28T15:37:21.493 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95241.ko 2024-03-28T15:37:21.493 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95245.ko 2024-03-28T15:37:21.493 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc2945.ko 2024-03-28T15:37:21.493 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4151.ko 2024-03-28T15:37:21.493 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4215.ko 2024-03-28T15:37:21.494 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4222.ko 2024-03-28T15:37:21.494 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4245.ko 2024-03-28T15:37:21.494 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4260.ko 2024-03-28T15:37:21.494 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4261.ko 2024-03-28T15:37:21.494 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max16065.ko 2024-03-28T15:37:21.494 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max1619.ko 2024-03-28T15:37:21.494 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max1668.ko 2024-03-28T15:37:21.494 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max197.ko 2024-03-28T15:37:21.494 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6639.ko 2024-03-28T15:37:21.494 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6650.ko 2024-03-28T15:37:21.494 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6697.ko 2024-03-28T15:37:21.495 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/mcp3021.ko 2024-03-28T15:37:21.495 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6683.ko 2024-03-28T15:37:21.495 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6775-core.ko 2024-03-28T15:37:21.495 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6775.ko 2024-03-28T15:37:21.495 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct7802.ko 2024-03-28T15:37:21.495 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pc87360.ko 2024-03-28T15:37:21.495 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pc87427.ko 2024-03-28T15:37:21.495 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pcf8591.ko 2024-03-28T15:37:21.495 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus 2024-03-28T15:37:21.495 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-03-28T15:37:21.495 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-03-28T15:37:21.495 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-03-28T15:37:21.496 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max16064.ko 2024-03-28T15:37:21.496 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max34440.ko 2024-03-28T15:37:21.496 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max8688.ko 2024-03-28T15:37:21.496 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-03-28T15:37:21.496 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-03-28T15:37:21.496 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-03-28T15:37:21.496 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-03-28T15:37:21.496 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-03-28T15:37:21.496 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-03-28T15:37:21.496 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/powr1220.ko 2024-03-28T15:37:21.496 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch5627.ko 2024-03-28T15:37:21.496 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch5636.ko 2024-03-28T15:37:21.496 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch56xx-common.ko 2024-03-28T15:37:21.497 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sht15.ko 2024-03-28T15:37:21.497 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sht21.ko 2024-03-28T15:37:21.497 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/shtc1.ko 2024-03-28T15:37:21.497 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sis5595.ko 2024-03-28T15:37:21.497 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47b397.ko 2024-03-28T15:37:21.497 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47m1.ko 2024-03-28T15:37:21.497 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47m192.ko 2024-03-28T15:37:21.497 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/thmc50.ko 2024-03-28T15:37:21.497 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp102.ko 2024-03-28T15:37:21.497 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp103.ko 2024-03-28T15:37:21.497 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp401.ko 2024-03-28T15:37:21.497 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp421.ko 2024-03-28T15:37:21.498 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/via-cputemp.ko 2024-03-28T15:37:21.498 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/via686a.ko 2024-03-28T15:37:21.498 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/vt1211.ko 2024-03-28T15:37:21.498 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/vt8231.ko 2024-03-28T15:37:21.498 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83627ehf.ko 2024-03-28T15:37:21.498 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83627hf.ko 2024-03-28T15:37:21.498 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83781d.ko 2024-03-28T15:37:21.498 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83791d.ko 2024-03-28T15:37:21.498 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83792d.ko 2024-03-28T15:37:21.498 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83793.ko 2024-03-28T15:37:21.498 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83795.ko 2024-03-28T15:37:21.501 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83l785ts.ko 2024-03-28T15:37:21.502 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83l786ng.ko 2024-03-28T15:37:21.502 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c 2024-03-28T15:37:21.502 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos 2024-03-28T15:37:21.502 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-03-28T15:37:21.502 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-03-28T15:37:21.502 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses 2024-03-28T15:37:21.502 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-03-28T15:37:21.502 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-03-28T15:37:21.502 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-03-28T15:37:21.502 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-03-28T15:37:21.502 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-03-28T15:37:21.502 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-03-28T15:37:21.502 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-03-28T15:37:21.502 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-03-28T15:37:21.503 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-i801.ko 2024-03-28T15:37:21.503 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-isch.ko 2024-03-28T15:37:21.503 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-03-28T15:37:21.503 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-03-28T15:37:21.503 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-03-28T15:37:21.503 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-parport.ko 2024-03-28T15:37:21.503 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-03-28T15:37:21.503 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-03-28T15:37:21.503 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-03-28T15:37:21.503 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-03-28T15:37:21.503 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-03-28T15:37:21.503 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-03-28T15:37:21.503 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-via.ko 2024-03-28T15:37:21.503 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-03-28T15:37:21.504 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-03-28T15:37:21.504 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-dev.ko 2024-03-28T15:37:21.504 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-mux.ko 2024-03-28T15:37:21.504 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-03-28T15:37:21.504 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-smbus.ko 2024-03-28T15:37:21.504 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-stub.ko 2024-03-28T15:37:21.504 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband 2024-03-28T15:37:21.504 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core 2024-03-28T15:37:21.504 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_cm.ko 2024-03-28T15:37:21.504 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_core.ko 2024-03-28T15:37:21.504 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_umad.ko 2024-03-28T15:37:21.504 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-03-28T15:37:21.504 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/iw_cm.ko 2024-03-28T15:37:21.505 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/rdma_cm.ko 2024-03-28T15:37:21.505 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-03-28T15:37:21.505 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw 2024-03-28T15:37:21.505 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/cxgb4 2024-03-28T15:37:21.505 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-03-28T15:37:21.505 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx4 2024-03-28T15:37:21.505 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-03-28T15:37:21.505 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx5 2024-03-28T15:37:21.505 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-03-28T15:37:21.505 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mthca 2024-03-28T15:37:21.505 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-03-28T15:37:21.505 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp 2024-03-28T15:37:21.505 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/ipoib 2024-03-28T15:37:21.505 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-03-28T15:37:21.506 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/iser 2024-03-28T15:37:21.506 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-03-28T15:37:21.506 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/isert 2024-03-28T15:37:21.506 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-03-28T15:37:21.506 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srp 2024-03-28T15:37:21.506 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-03-28T15:37:21.506 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srpt 2024-03-28T15:37:21.506 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-03-28T15:37:21.506 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input 2024-03-28T15:37:21.506 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport 2024-03-28T15:37:21.506 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-03-28T15:37:21.506 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/fm801-gp.ko 2024-03-28T15:37:21.506 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/gameport.ko 2024-03-28T15:37:21.506 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/joydev.ko 2024-03-28T15:37:21.507 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard 2024-03-28T15:37:21.507 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/gpio_keys.ko 2024-03-28T15:37:21.507 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse 2024-03-28T15:37:21.507 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/appletouch.ko 2024-03-28T15:37:21.507 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/bcm5974.ko 2024-03-28T15:37:21.507 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/cyapatp.ko 2024-03-28T15:37:21.507 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/elan_i2c.ko 2024-03-28T15:37:21.507 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/sermouse.ko 2024-03-28T15:37:21.507 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-03-28T15:37:21.507 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/synaptics_usb.ko 2024-03-28T15:37:21.507 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/vsxxxaa.ko 2024-03-28T15:37:21.507 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/rmi4 2024-03-28T15:37:21.507 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/rmi4/rmi_core.ko 2024-03-28T15:37:21.507 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio 2024-03-28T15:37:21.507 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/altera_ps2.ko 2024-03-28T15:37:21.508 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/arc_ps2.ko 2024-03-28T15:37:21.508 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-03-28T15:37:21.508 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/serio_raw.ko 2024-03-28T15:37:21.508 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/sparse-keymap.ko 2024-03-28T15:37:21.508 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds 2024-03-28T15:37:21.508 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-blinkm.ko 2024-03-28T15:37:21.508 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-lm3530.ko 2024-03-28T15:37:21.508 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-lp3944.ko 2024-03-28T15:37:21.508 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-ss4200.ko 2024-03-28T15:37:21.508 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger 2024-03-28T15:37:21.508 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-03-28T15:37:21.508 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-03-28T15:37:21.508 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-03-28T15:37:21.508 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-03-28T15:37:21.509 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-03-28T15:37:21.509 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-03-28T15:37:21.509 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-03-28T15:37:21.509 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-03-28T15:37:21.509 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-03-28T15:37:21.509 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md 2024-03-28T15:37:21.509 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/bcache 2024-03-28T15:37:21.509 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/bcache/bcache.ko 2024-03-28T15:37:21.509 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-bio-prison.ko 2024-03-28T15:37:21.509 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-cache-smq.ko 2024-03-28T15:37:21.509 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-cache.ko 2024-03-28T15:37:21.509 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-crypt.ko 2024-03-28T15:37:21.509 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-delay.ko 2024-03-28T15:37:21.509 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-flakey.ko 2024-03-28T15:37:21.509 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-log-userspace.ko 2024-03-28T15:37:21.510 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-multipath.ko 2024-03-28T15:37:21.510 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-queue-length.ko 2024-03-28T15:37:21.510 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-raid.ko 2024-03-28T15:37:21.510 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-round-robin.ko 2024-03-28T15:37:21.510 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-service-time.ko 2024-03-28T15:37:21.510 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-switch.ko 2024-03-28T15:37:21.510 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-thin-pool.ko 2024-03-28T15:37:21.510 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-verity.ko 2024-03-28T15:37:21.510 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/persistent-data 2024-03-28T15:37:21.510 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-03-28T15:37:21.510 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid0.ko 2024-03-28T15:37:21.510 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid1.ko 2024-03-28T15:37:21.510 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid10.ko 2024-03-28T15:37:21.510 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid456.ko 2024-03-28T15:37:21.511 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message 2024-03-28T15:37:21.511 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion 2024-03-28T15:37:21.511 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptbase.ko 2024-03-28T15:37:21.511 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptctl.ko 2024-03-28T15:37:21.511 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptfc.ko 2024-03-28T15:37:21.511 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptlan.ko 2024-03-28T15:37:21.511 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptsas.ko 2024-03-28T15:37:21.511 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptscsih.ko 2024-03-28T15:37:21.511 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptspi.ko 2024-03-28T15:37:21.511 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd 2024-03-28T15:37:21.511 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/lpc_ich.ko 2024-03-28T15:37:21.511 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/lpc_sch.ko 2024-03-28T15:37:21.511 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/mfd-core.ko 2024-03-28T15:37:21.511 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/sm501.ko 2024-03-28T15:37:21.511 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/viperboard.ko 2024-03-28T15:37:21.511 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/vx855.ko 2024-03-28T15:37:21.512 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/wl1273-core.ko 2024-03-28T15:37:21.512 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc 2024-03-28T15:37:21.512 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/altera-stapl 2024-03-28T15:37:21.512 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-03-28T15:37:21.512 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/apds9802als.ko 2024-03-28T15:37:21.512 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/apds990x.ko 2024-03-28T15:37:21.512 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/bh1770glc.ko 2024-03-28T15:37:21.512 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/cb710 2024-03-28T15:37:21.512 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/cb710/cb710.ko 2024-03-28T15:37:21.512 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom 2024-03-28T15:37:21.512 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/at24.ko 2024-03-28T15:37:21.512 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-03-28T15:37:21.512 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/max6875.ko 2024-03-28T15:37:21.512 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/enclosure.ko 2024-03-28T15:37:21.512 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/hpilo.ko 2024-03-28T15:37:21.513 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/isl29003.ko 2024-03-28T15:37:21.513 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/isl29020.ko 2024-03-28T15:37:21.513 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d 2024-03-28T15:37:21.513 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-03-28T15:37:21.513 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-03-28T15:37:21.513 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei 2024-03-28T15:37:21.513 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei-me.ko 2024-03-28T15:37:21.513 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei-txe.ko 2024-03-28T15:37:21.513 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei.ko 2024-03-28T15:37:21.513 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-gru 2024-03-28T15:37:21.513 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-gru/gru.ko 2024-03-28T15:37:21.513 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp 2024-03-28T15:37:21.513 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp/xp.ko 2024-03-28T15:37:21.513 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp/xpc.ko 2024-03-28T15:37:21.514 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-03-28T15:37:21.514 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tifm_7xx1.ko 2024-03-28T15:37:21.514 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tifm_core.ko 2024-03-28T15:37:21.514 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tsl2550.ko 2024-03-28T15:37:21.514 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_balloon.ko 2024-03-28T15:37:21.514 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_vmci 2024-03-28T15:37:21.514 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-03-28T15:37:21.514 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net 2024-03-28T15:37:21.514 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/bonding 2024-03-28T15:37:21.514 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/bonding/bonding.ko 2024-03-28T15:37:21.514 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa 2024-03-28T15:37:21.514 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/b53 2024-03-28T15:37:21.514 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/b53/b53_common.ko 2024-03-28T15:37:21.514 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/bcm-sf2.ko 2024-03-28T15:37:21.515 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6060.ko 2024-03-28T15:37:21.515 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6xxx 2024-03-28T15:37:21.515 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-03-28T15:37:21.515 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dummy.ko 2024-03-28T15:37:21.515 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/eql.ko 2024-03-28T15:37:21.515 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet 2024-03-28T15:37:21.515 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com 2024-03-28T15:37:21.515 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-03-28T15:37:21.515 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-03-28T15:37:21.515 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-03-28T15:37:21.515 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-03-28T15:37:21.515 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390 2024-03-28T15:37:21.515 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/8390.ko 2024-03-28T15:37:21.516 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-03-28T15:37:21.516 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-03-28T15:37:21.516 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-03-28T15:37:21.516 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/adaptec 2024-03-28T15:37:21.516 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-03-28T15:37:21.516 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/agere 2024-03-28T15:37:21.516 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/agere/et131x.ko 2024-03-28T15:37:21.516 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/alteon 2024-03-28T15:37:21.516 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-03-28T15:37:21.516 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/altera 2024-03-28T15:37:21.516 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-03-28T15:37:21.516 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd 2024-03-28T15:37:21.516 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-03-28T15:37:21.516 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-03-28T15:37:21.517 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-03-28T15:37:21.517 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros 2024-03-28T15:37:21.517 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/alx 2024-03-28T15:37:21.517 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-03-28T15:37:21.517 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1c 2024-03-28T15:37:21.517 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-03-28T15:37:21.517 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1e 2024-03-28T15:37:21.517 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-03-28T15:37:21.517 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx 2024-03-28T15:37:21.517 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-03-28T15:37:21.517 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-03-28T15:37:21.517 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom 2024-03-28T15:37:21.517 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-03-28T15:37:21.518 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-03-28T15:37:21.518 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-03-28T15:37:21.518 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-03-28T15:37:21.518 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-03-28T15:37:21.518 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/genet 2024-03-28T15:37:21.518 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-03-28T15:37:21.518 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-03-28T15:37:21.518 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade 2024-03-28T15:37:21.518 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade/bna 2024-03-28T15:37:21.518 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-03-28T15:37:21.518 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio 2024-03-28T15:37:21.518 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb 2024-03-28T15:37:21.518 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-03-28T15:37:21.518 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-03-28T15:37:21.518 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-03-28T15:37:21.519 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-03-28T15:37:21.519 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-03-28T15:37:21.519 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-03-28T15:37:21.519 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-03-28T15:37:21.519 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-03-28T15:37:21.519 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-03-28T15:37:21.519 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco 2024-03-28T15:37:21.519 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco/enic 2024-03-28T15:37:21.519 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-03-28T15:37:21.519 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec 2024-03-28T15:37:21.519 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip 2024-03-28T15:37:21.519 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-03-28T15:37:21.519 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-03-28T15:37:21.520 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-03-28T15:37:21.520 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-03-28T15:37:21.520 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-03-28T15:37:21.520 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-03-28T15:37:21.520 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink 2024-03-28T15:37:21.520 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-03-28T15:37:21.520 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-03-28T15:37:21.520 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dnet.ko 2024-03-28T15:37:21.520 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex 2024-03-28T15:37:21.520 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex/benet 2024-03-28T15:37:21.520 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-03-28T15:37:21.520 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ethoc.ko 2024-03-28T15:37:21.520 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/fealnx.ko 2024-03-28T15:37:21.520 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel 2024-03-28T15:37:21.521 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e100.ko 2024-03-28T15:37:21.521 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000 2024-03-28T15:37:21.521 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-03-28T15:37:21.521 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000e 2024-03-28T15:37:21.521 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-03-28T15:37:21.521 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/fm10k 2024-03-28T15:37:21.521 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-03-28T15:37:21.521 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/i40e 2024-03-28T15:37:21.521 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-03-28T15:37:21.521 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/iavf 2024-03-28T15:37:21.521 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-03-28T15:37:21.521 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igb 2024-03-28T15:37:21.521 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-03-28T15:37:21.522 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igbvf 2024-03-28T15:37:21.522 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-03-28T15:37:21.522 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbe 2024-03-28T15:37:21.522 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-03-28T15:37:21.522 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbevf 2024-03-28T15:37:21.522 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-03-28T15:37:21.522 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/jme.ko 2024-03-28T15:37:21.522 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell 2024-03-28T15:37:21.522 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-03-28T15:37:21.522 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/skge.ko 2024-03-28T15:37:21.522 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-03-28T15:37:21.522 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox 2024-03-28T15:37:21.522 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4 2024-03-28T15:37:21.523 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-03-28T15:37:21.523 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-03-28T15:37:21.523 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5 2024-03-28T15:37:21.523 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-03-28T15:37:21.523 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-03-28T15:37:21.523 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/micrel 2024-03-28T15:37:21.523 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-03-28T15:37:21.523 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom 2024-03-28T15:37:21.523 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom/myri10ge 2024-03-28T15:37:21.523 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-03-28T15:37:21.523 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi 2024-03-28T15:37:21.523 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-03-28T15:37:21.523 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-03-28T15:37:21.523 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/neterion 2024-03-28T15:37:21.523 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-03-28T15:37:21.523 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/nvidia 2024-03-28T15:37:21.524 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-03-28T15:37:21.524 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines 2024-03-28T15:37:21.524 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-03-28T15:37:21.524 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-03-28T15:37:21.524 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic 2024-03-28T15:37:21.524 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/netxen 2024-03-28T15:37:21.524 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-03-28T15:37:21.524 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-03-28T15:37:21.524 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-03-28T15:37:21.524 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-03-28T15:37:21.524 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rdc 2024-03-28T15:37:21.524 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-03-28T15:37:21.524 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek 2024-03-28T15:37:21.524 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-03-28T15:37:21.524 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-03-28T15:37:21.525 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/atp.ko 2024-03-28T15:37:21.525 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-03-28T15:37:21.525 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rocker 2024-03-28T15:37:21.525 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-03-28T15:37:21.525 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sfc 2024-03-28T15:37:21.525 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-03-28T15:37:21.525 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/silan 2024-03-28T15:37:21.525 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-03-28T15:37:21.525 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis 2024-03-28T15:37:21.525 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis/sis190.ko 2024-03-28T15:37:21.525 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis/sis900.ko 2024-03-28T15:37:21.525 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc 2024-03-28T15:37:21.525 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-03-28T15:37:21.525 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-03-28T15:37:21.525 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-03-28T15:37:21.526 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-03-28T15:37:21.526 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro 2024-03-28T15:37:21.526 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac 2024-03-28T15:37:21.526 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-03-28T15:37:21.526 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-03-28T15:37:21.526 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun 2024-03-28T15:37:21.526 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/cassini.ko 2024-03-28T15:37:21.526 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/niu.ko 2024-03-28T15:37:21.526 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/sungem.ko 2024-03-28T15:37:21.526 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-03-28T15:37:21.526 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/tehuti 2024-03-28T15:37:21.526 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-03-28T15:37:21.526 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ti 2024-03-28T15:37:21.526 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ti/tlan.ko 2024-03-28T15:37:21.526 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via 2024-03-28T15:37:21.527 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-03-28T15:37:21.527 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-03-28T15:37:21.527 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet 2024-03-28T15:37:21.527 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-03-28T15:37:21.527 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-03-28T15:37:21.527 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/xircom 2024-03-28T15:37:21.527 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-03-28T15:37:21.527 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/geneve.ko 2024-03-28T15:37:21.527 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/hyperv 2024-03-28T15:37:21.527 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-03-28T15:37:21.527 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ieee802154 2024-03-28T15:37:21.527 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ieee802154/fakelb.ko 2024-03-28T15:37:21.527 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ifb.ko 2024-03-28T15:37:21.527 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ipvlan 2024-03-28T15:37:21.527 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ipvlan/ipvlan.ko 2024-03-28T15:37:21.528 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/macvlan.ko 2024-03-28T15:37:21.528 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/macvtap.ko 2024-03-28T15:37:21.528 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio 2024-03-28T15:37:21.528 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio.ko 2024-03-28T15:37:21.528 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-03-28T15:37:21.528 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-03-28T15:37:21.528 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-regmap.ko 2024-03-28T15:37:21.528 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mii.ko 2024-03-28T15:37:21.528 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/net_failover.ko 2024-03-28T15:37:21.528 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/netconsole.ko 2024-03-28T15:37:21.528 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/nlmon.ko 2024-03-28T15:37:21.528 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs 2024-03-28T15:37:21.528 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs/pcs-lynx.ko 2024-03-28T15:37:21.528 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-03-28T15:37:21.528 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy 2024-03-28T15:37:21.529 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/amd.ko 2024-03-28T15:37:21.529 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-03-28T15:37:21.529 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-03-28T15:37:21.529 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm7xxx.ko 2024-03-28T15:37:21.529 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm87xx.ko 2024-03-28T15:37:21.529 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/broadcom.ko 2024-03-28T15:37:21.529 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/cicada.ko 2024-03-28T15:37:21.529 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/davicom.ko 2024-03-28T15:37:21.529 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/dp83640.ko 2024-03-28T15:37:21.529 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/et1011c.ko 2024-03-28T15:37:21.529 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/icplus.ko 2024-03-28T15:37:21.529 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/lxt.ko 2024-03-28T15:37:21.529 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/marvell.ko 2024-03-28T15:37:21.529 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/micrel.ko 2024-03-28T15:37:21.529 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/national.ko 2024-03-28T15:37:21.530 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/phylink.ko 2024-03-28T15:37:21.530 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/qsemi.ko 2024-03-28T15:37:21.530 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/realtek.ko 2024-03-28T15:37:21.530 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/smsc.ko 2024-03-28T15:37:21.530 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/ste10Xp.ko 2024-03-28T15:37:21.530 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/vitesse.ko 2024-03-28T15:37:21.530 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp 2024-03-28T15:37:21.530 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/bsd_comp.ko 2024-03-28T15:37:21.530 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_async.ko 2024-03-28T15:37:21.530 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_deflate.ko 2024-03-28T15:37:21.530 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_generic.ko 2024-03-28T15:37:21.530 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_mppe.ko 2024-03-28T15:37:21.530 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_synctty.ko 2024-03-28T15:37:21.530 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pppoe.ko 2024-03-28T15:37:21.531 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pppox.ko 2024-03-28T15:37:21.531 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pptp.ko 2024-03-28T15:37:21.531 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/slip 2024-03-28T15:37:21.531 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/slip/slhc.ko 2024-03-28T15:37:21.531 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/slip/slip.ko 2024-03-28T15:37:21.531 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/sungem_phy.ko 2024-03-28T15:37:21.531 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/tap.ko 2024-03-28T15:37:21.531 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team 2024-03-28T15:37:21.531 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team.ko 2024-03-28T15:37:21.531 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_activebackup.ko 2024-03-28T15:37:21.531 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_broadcast.ko 2024-03-28T15:37:21.531 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-03-28T15:37:21.531 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_random.ko 2024-03-28T15:37:21.531 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-03-28T15:37:21.532 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/tun.ko 2024-03-28T15:37:21.532 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/veth.ko 2024-03-28T15:37:21.532 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/virtio_net.ko 2024-03-28T15:37:21.532 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vmxnet3 2024-03-28T15:37:21.532 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-03-28T15:37:21.532 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vxlan 2024-03-28T15:37:21.532 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vxlan/vxlan.ko 2024-03-28T15:37:21.532 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/xen-netback 2024-03-28T15:37:21.532 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/xen-netback/xen-netback.ko 2024-03-28T15:37:21.532 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/xen-netfront.ko 2024-03-28T15:37:21.532 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ntb 2024-03-28T15:37:21.532 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ntb/ntb.ko 2024-03-28T15:37:21.532 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme 2024-03-28T15:37:21.532 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host 2024-03-28T15:37:21.532 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host/nvme-core.ko 2024-03-28T15:37:21.533 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host/nvme.ko 2024-03-28T15:37:21.533 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport 2024-03-28T15:37:21.533 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport.ko 2024-03-28T15:37:21.533 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_cs.ko 2024-03-28T15:37:21.533 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_pc.ko 2024-03-28T15:37:21.533 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_serial.ko 2024-03-28T15:37:21.533 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci 2024-03-28T15:37:21.533 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/hotplug 2024-03-28T15:37:21.533 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-03-28T15:37:21.533 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/pcie 2024-03-28T15:37:21.533 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/pcie/aer_inject.ko 2024-03-28T15:37:21.533 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/xen-pcifront.ko 2024-03-28T15:37:21.533 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia 2024-03-28T15:37:21.533 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/i82092.ko 2024-03-28T15:37:21.534 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/pd6729.ko 2024-03-28T15:37:21.534 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/yenta_socket.ko 2024-03-28T15:37:21.534 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl 2024-03-28T15:37:21.534 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl/intel 2024-03-28T15:37:21.534 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-03-28T15:37:21.534 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform 2024-03-28T15:37:21.534 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86 2024-03-28T15:37:21.534 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/acer-wmi.ko 2024-03-28T15:37:21.534 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/acerhdf.ko 2024-03-28T15:37:21.534 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/apple-gmux.ko 2024-03-28T15:37:21.534 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-laptop.ko 2024-03-28T15:37:21.534 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-03-28T15:37:21.534 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-wmi.ko 2024-03-28T15:37:21.534 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/classmate-laptop.ko 2024-03-28T15:37:21.535 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-03-28T15:37:21.535 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-03-28T15:37:21.535 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-03-28T15:37:21.535 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-03-28T15:37:21.535 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/hdaps.ko 2024-03-28T15:37:21.535 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel 2024-03-28T15:37:21.535 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-03-28T15:37:21.535 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel_ips.ko 2024-03-28T15:37:21.535 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/msi-wmi.ko 2024-03-28T15:37:21.535 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/mxm-wmi.ko 2024-03-28T15:37:21.535 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-03-28T15:37:21.535 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/samsung-laptop.ko 2024-03-28T15:37:21.535 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/samsung-q10.ko 2024-03-28T15:37:21.535 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/topstar-laptop.ko 2024-03-28T15:37:21.535 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-03-28T15:37:21.536 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/toshiba_haps.ko 2024-03-28T15:37:21.536 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/wmi-bmof.ko 2024-03-28T15:37:21.536 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/wmi.ko 2024-03-28T15:37:21.536 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap 2024-03-28T15:37:21.536 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap/intel_rapl_common.ko 2024-03-28T15:37:21.536 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap/intel_rapl_msr.ko 2024-03-28T15:37:21.536 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps 2024-03-28T15:37:21.536 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients 2024-03-28T15:37:21.536 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps-gpio.ko 2024-03-28T15:37:21.536 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps-ldisc.ko 2024-03-28T15:37:21.536 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps_parport.ko 2024-03-28T15:37:21.536 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/pps_core.ko 2024-03-28T15:37:21.536 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp 2024-03-28T15:37:21.536 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp/ptp.ko 2024-03-28T15:37:21.537 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp/ptp_kvm.ko 2024-03-28T15:37:21.537 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc 2024-03-28T15:37:21.537 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-bq32k.ko 2024-03-28T15:37:21.537 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1286.ko 2024-03-28T15:37:21.537 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1307.ko 2024-03-28T15:37:21.537 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1374.ko 2024-03-28T15:37:21.537 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1511.ko 2024-03-28T15:37:21.537 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1553.ko 2024-03-28T15:37:21.537 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1672.ko 2024-03-28T15:37:21.537 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1742.ko 2024-03-28T15:37:21.537 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds2404.ko 2024-03-28T15:37:21.537 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds3232.ko 2024-03-28T15:37:21.537 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-em3027.ko 2024-03-28T15:37:21.537 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-fm3130.ko 2024-03-28T15:37:21.537 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-isl12022.ko 2024-03-28T15:37:21.538 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-isl1208.ko 2024-03-28T15:37:21.538 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m41t80.ko 2024-03-28T15:37:21.538 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m48t35.ko 2024-03-28T15:37:21.538 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m48t59.ko 2024-03-28T15:37:21.538 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-max6900.ko 2024-03-28T15:37:21.538 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-msm6242.ko 2024-03-28T15:37:21.538 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf2127.ko 2024-03-28T15:37:21.538 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf85063.ko 2024-03-28T15:37:21.538 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8523.ko 2024-03-28T15:37:21.538 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8563.ko 2024-03-28T15:37:21.538 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8583.ko 2024-03-28T15:37:21.538 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rp5c01.ko 2024-03-28T15:37:21.538 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rs5c372.ko 2024-03-28T15:37:21.538 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-03-28T15:37:21.538 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rx8025.ko 2024-03-28T15:37:21.539 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rx8581.ko 2024-03-28T15:37:21.539 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-03-28T15:37:21.539 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-x1205.ko 2024-03-28T15:37:21.539 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi 2024-03-28T15:37:21.539 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-9xxx.ko 2024-03-28T15:37:21.539 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-sas.ko 2024-03-28T15:37:21.539 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-xxxx.ko 2024-03-28T15:37:21.539 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/BusLogic.ko 2024-03-28T15:37:21.539 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/a100u2w.ko 2024-03-28T15:37:21.539 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aacraid 2024-03-28T15:37:21.539 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aacraid/aacraid.ko 2024-03-28T15:37:21.539 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/advansys.ko 2024-03-28T15:37:21.539 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx 2024-03-28T15:37:21.539 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-03-28T15:37:21.539 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-03-28T15:37:21.540 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/am53c974.ko 2024-03-28T15:37:21.540 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/arcmsr 2024-03-28T15:37:21.540 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-03-28T15:37:21.540 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/atp870u.ko 2024-03-28T15:37:21.540 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/be2iscsi 2024-03-28T15:37:21.540 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-03-28T15:37:21.540 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bfa 2024-03-28T15:37:21.540 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bfa/bfa.ko 2024-03-28T15:37:21.540 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2fc 2024-03-28T15:37:21.540 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-03-28T15:37:21.540 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2i 2024-03-28T15:37:21.540 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-03-28T15:37:21.540 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ch.ko 2024-03-28T15:37:21.540 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/csiostor 2024-03-28T15:37:21.540 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/csiostor/csiostor.ko 2024-03-28T15:37:21.541 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi 2024-03-28T15:37:21.541 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb3i 2024-03-28T15:37:21.541 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-03-28T15:37:21.541 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb4i 2024-03-28T15:37:21.541 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-03-28T15:37:21.541 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-03-28T15:37:21.541 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/dc395x.ko 2024-03-28T15:37:21.541 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler 2024-03-28T15:37:21.541 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-03-28T15:37:21.541 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-03-28T15:37:21.541 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-03-28T15:37:21.541 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-03-28T15:37:21.541 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/dmx3191d.ko 2024-03-28T15:37:21.541 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esas2r 2024-03-28T15:37:21.541 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esas2r/esas2r.ko 2024-03-28T15:37:21.541 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esp_scsi.ko 2024-03-28T15:37:21.542 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe 2024-03-28T15:37:21.542 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe/fcoe.ko 2024-03-28T15:37:21.542 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-03-28T15:37:21.542 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fnic 2024-03-28T15:37:21.542 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fnic/fnic.ko 2024-03-28T15:37:21.542 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hpsa.ko 2024-03-28T15:37:21.542 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hptiop.ko 2024-03-28T15:37:21.542 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hv_storvsc.ko 2024-03-28T15:37:21.542 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/initio.ko 2024-03-28T15:37:21.542 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ipr.ko 2024-03-28T15:37:21.542 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ips.ko 2024-03-28T15:37:21.542 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/isci 2024-03-28T15:37:21.542 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/isci/isci.ko 2024-03-28T15:37:21.542 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-03-28T15:37:21.543 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/iscsi_tcp.ko 2024-03-28T15:37:21.543 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libfc 2024-03-28T15:37:21.543 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libfc/libfc.ko 2024-03-28T15:37:21.543 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libiscsi.ko 2024-03-28T15:37:21.543 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libiscsi_tcp.ko 2024-03-28T15:37:21.543 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libsas 2024-03-28T15:37:21.543 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libsas/libsas.ko 2024-03-28T15:37:21.543 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/lpfc 2024-03-28T15:37:21.543 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/lpfc/lpfc.ko 2024-03-28T15:37:21.543 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid 2024-03-28T15:37:21.543 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid.ko 2024-03-28T15:37:21.543 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-03-28T15:37:21.543 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-03-28T15:37:21.543 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-03-28T15:37:21.543 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mpt3sas 2024-03-28T15:37:21.544 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-03-28T15:37:21.544 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvsas 2024-03-28T15:37:21.544 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvsas/mvsas.ko 2024-03-28T15:37:21.544 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvumi.ko 2024-03-28T15:37:21.544 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pm8001 2024-03-28T15:37:21.544 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-03-28T15:37:21.544 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pmcraid.ko 2024-03-28T15:37:21.544 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla1280.ko 2024-03-28T15:37:21.544 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx 2024-03-28T15:37:21.544 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-03-28T15:37:21.544 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-03-28T15:37:21.544 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla4xxx 2024-03-28T15:37:21.544 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-03-28T15:37:21.544 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/raid_class.ko 2024-03-28T15:37:21.544 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_debug.ko 2024-03-28T15:37:21.544 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_fc.ko 2024-03-28T15:37:21.545 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-03-28T15:37:21.545 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_sas.ko 2024-03-28T15:37:21.545 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_spi.ko 2024-03-28T15:37:21.545 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_srp.ko 2024-03-28T15:37:21.545 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ses.ko 2024-03-28T15:37:21.545 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/st.ko 2024-03-28T15:37:21.545 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/stex.ko 2024-03-28T15:37:21.545 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/sym53c8xx_2 2024-03-28T15:37:21.545 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-03-28T15:37:21.545 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/virtio_scsi.ko 2024-03-28T15:37:21.545 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/vmw_pvscsi.ko 2024-03-28T15:37:21.545 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/wd719x.ko 2024-03-28T15:37:21.545 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ssb 2024-03-28T15:37:21.545 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ssb/ssb.ko 2024-03-28T15:37:21.546 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target 2024-03-28T15:37:21.546 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/iscsi 2024-03-28T15:37:21.546 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-03-28T15:37:21.546 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/loopback 2024-03-28T15:37:21.546 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/loopback/tcm_loop.ko 2024-03-28T15:37:21.546 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_file.ko 2024-03-28T15:37:21.546 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_iblock.ko 2024-03-28T15:37:21.546 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_mod.ko 2024-03-28T15:37:21.546 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_pscsi.ko 2024-03-28T15:37:21.546 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_user.ko 2024-03-28T15:37:21.546 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/tcm_fc 2024-03-28T15:37:21.546 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-03-28T15:37:21.546 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal 2024-03-28T15:37:21.546 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel 2024-03-28T15:37:21.546 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal 2024-03-28T15:37:21.547 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-03-28T15:37:21.547 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-03-28T15:37:21.547 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-03-28T15:37:21.547 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-03-28T15:37:21.547 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-03-28T15:37:21.547 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-03-28T15:37:21.547 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-03-28T15:37:21.547 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-03-28T15:37:21.547 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-03-28T15:37:21.547 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-03-28T15:37:21.547 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-03-28T15:37:21.547 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-03-28T15:37:21.547 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-03-28T15:37:21.547 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-03-28T15:37:21.547 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-03-28T15:37:21.548 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-03-28T15:37:21.548 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-03-28T15:37:21.548 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-03-28T15:37:21.548 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty 2024-03-28T15:37:21.548 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/ipwireless 2024-03-28T15:37:21.548 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-03-28T15:37:21.548 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/n_gsm.ko 2024-03-28T15:37:21.548 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/n_hdlc.ko 2024-03-28T15:37:21.548 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/nozomi.ko 2024-03-28T15:37:21.548 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial 2024-03-28T15:37:21.548 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/8250 2024-03-28T15:37:21.548 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-03-28T15:37:21.548 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/arc_uart.ko 2024-03-28T15:37:21.548 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/jsm 2024-03-28T15:37:21.548 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/jsm/jsm.ko 2024-03-28T15:37:21.549 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs 2024-03-28T15:37:21.549 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/core 2024-03-28T15:37:21.549 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/core/ufshcd-core.ko 2024-03-28T15:37:21.549 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/host 2024-03-28T15:37:21.549 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-03-28T15:37:21.549 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio 2024-03-28T15:37:21.549 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio.ko 2024-03-28T15:37:21.549 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_aec.ko 2024-03-28T15:37:21.549 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_cif.ko 2024-03-28T15:37:21.549 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_pci_generic.ko 2024-03-28T15:37:21.549 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_sercos3.ko 2024-03-28T15:37:21.549 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb 2024-03-28T15:37:21.549 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class 2024-03-28T15:37:21.549 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/cdc-acm.ko 2024-03-28T15:37:21.549 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/cdc-wdm.ko 2024-03-28T15:37:21.550 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/usblp.ko 2024-03-28T15:37:21.550 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/usbtmc.ko 2024-03-28T15:37:21.550 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host 2024-03-28T15:37:21.550 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host/sl811-hcd.ko 2024-03-28T15:37:21.550 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image 2024-03-28T15:37:21.550 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image/mdc800.ko 2024-03-28T15:37:21.550 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image/microtek.ko 2024-03-28T15:37:21.550 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc 2024-03-28T15:37:21.550 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/adutux.ko 2024-03-28T15:37:21.550 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/appledisplay.ko 2024-03-28T15:37:21.550 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/emi26.ko 2024-03-28T15:37:21.550 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/emi62.ko 2024-03-28T15:37:21.550 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/ezusb.ko 2024-03-28T15:37:21.551 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/idmouse.ko 2024-03-28T15:37:21.551 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/iowarrior.ko 2024-03-28T15:37:21.551 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/isight_firmware.ko 2024-03-28T15:37:21.551 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/ldusb.ko 2024-03-28T15:37:21.551 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/legousbtower.ko 2024-03-28T15:37:21.551 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/sisusbvga 2024-03-28T15:37:21.551 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-03-28T15:37:21.551 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/trancevibrator.ko 2024-03-28T15:37:21.551 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usb3503.ko 2024-03-28T15:37:21.551 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usblcd.ko 2024-03-28T15:37:21.551 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usbsevseg.ko 2024-03-28T15:37:21.551 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/uss720.ko 2024-03-28T15:37:21.551 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/yurex.ko 2024-03-28T15:37:21.552 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/phy 2024-03-28T15:37:21.552 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/phy/phy-generic.ko 2024-03-28T15:37:21.552 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial 2024-03-28T15:37:21.552 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/aircable.ko 2024-03-28T15:37:21.552 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ark3116.ko 2024-03-28T15:37:21.552 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/belkin_sa.ko 2024-03-28T15:37:21.552 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ch341.ko 2024-03-28T15:37:21.552 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cp210x.ko 2024-03-28T15:37:21.552 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cyberjack.ko 2024-03-28T15:37:21.552 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cypress_m8.ko 2024-03-28T15:37:21.552 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/digi_acceleport.ko 2024-03-28T15:37:21.552 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/empeg.ko 2024-03-28T15:37:21.552 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ftdi_sio.ko 2024-03-28T15:37:21.552 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/garmin_gps.ko 2024-03-28T15:37:21.553 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/io_edgeport.ko 2024-03-28T15:37:21.553 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/io_ti.ko 2024-03-28T15:37:21.553 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ipaq.ko 2024-03-28T15:37:21.553 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ipw.ko 2024-03-28T15:37:21.553 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ir-usb.ko 2024-03-28T15:37:21.553 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-03-28T15:37:21.553 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/keyspan.ko 2024-03-28T15:37:21.553 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/keyspan_pda.ko 2024-03-28T15:37:21.553 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/kl5kusb105.ko 2024-03-28T15:37:21.553 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/kobil_sct.ko 2024-03-28T15:37:21.553 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mct_u232.ko 2024-03-28T15:37:21.553 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mos7720.ko 2024-03-28T15:37:21.553 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mos7840.ko 2024-03-28T15:37:21.553 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/navman.ko 2024-03-28T15:37:21.554 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/omninet.ko 2024-03-28T15:37:21.554 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/opticon.ko 2024-03-28T15:37:21.554 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/option.ko 2024-03-28T15:37:21.554 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/oti6858.ko 2024-03-28T15:37:21.554 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/pl2303.ko 2024-03-28T15:37:21.554 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/qcaux.ko 2024-03-28T15:37:21.554 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/qcserial.ko 2024-03-28T15:37:21.554 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/quatech2.ko 2024-03-28T15:37:21.554 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/safe_serial.ko 2024-03-28T15:37:21.554 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/sierra.ko 2024-03-28T15:37:21.554 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/spcp8x5.ko 2024-03-28T15:37:21.554 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ssu100.ko 2024-03-28T15:37:21.554 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/symbolserial.ko 2024-03-28T15:37:21.554 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-03-28T15:37:21.555 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-03-28T15:37:21.555 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb_debug.ko 2024-03-28T15:37:21.555 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb_wwan.ko 2024-03-28T15:37:21.555 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/visor.ko 2024-03-28T15:37:21.555 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/whiteheat.ko 2024-03-28T15:37:21.555 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/xsens_mt.ko 2024-03-28T15:37:21.555 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage 2024-03-28T15:37:21.555 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/uas.ko 2024-03-28T15:37:21.555 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-alauda.ko 2024-03-28T15:37:21.555 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-cypress.ko 2024-03-28T15:37:21.555 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-datafab.ko 2024-03-28T15:37:21.555 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-03-28T15:37:21.555 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-freecom.ko 2024-03-28T15:37:21.555 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-isd200.ko 2024-03-28T15:37:21.556 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-03-28T15:37:21.556 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-karma.ko 2024-03-28T15:37:21.556 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-onetouch.ko 2024-03-28T15:37:21.556 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-realtek.ko 2024-03-28T15:37:21.556 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-sddr09.ko 2024-03-28T15:37:21.556 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-sddr55.ko 2024-03-28T15:37:21.556 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-usbat.ko 2024-03-28T15:37:21.556 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/usb-storage.ko 2024-03-28T15:37:21.556 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip 2024-03-28T15:37:21.556 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/usbip-core.ko 2024-03-28T15:37:21.556 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/usbip-host.ko 2024-03-28T15:37:21.556 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-03-28T15:37:21.556 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio 2024-03-28T15:37:21.557 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci 2024-03-28T15:37:21.557 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-03-28T15:37:21.557 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci/vfio-pci.ko 2024-03-28T15:37:21.557 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/vfio.ko 2024-03-28T15:37:21.557 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-03-28T15:37:21.557 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost 2024-03-28T15:37:21.557 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost.ko 2024-03-28T15:37:21.557 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_iotlb.ko 2024-03-28T15:37:21.557 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_net.ko 2024-03-28T15:37:21.557 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_scsi.ko 2024-03-28T15:37:21.557 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video 2024-03-28T15:37:21.557 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight 2024-03-28T15:37:21.557 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/apple_bl.ko 2024-03-28T15:37:21.557 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/lcd.ko 2024-03-28T15:37:21.558 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/platform_lcd.ko 2024-03-28T15:37:21.558 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio 2024-03-28T15:37:21.558 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio.ko 2024-03-28T15:37:21.558 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_balloon.ko 2024-03-28T15:37:21.558 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_mmio.ko 2024-03-28T15:37:21.558 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_pci.ko 2024-03-28T15:37:21.558 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-03-28T15:37:21.558 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-03-28T15:37:21.558 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_ring.ko 2024-03-28T15:37:21.558 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1 2024-03-28T15:37:21.558 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters 2024-03-28T15:37:21.558 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters/ds2482.ko 2024-03-28T15:37:21.558 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters/ds2490.ko 2024-03-28T15:37:21.558 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves 2024-03-28T15:37:21.558 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-03-28T15:37:21.559 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-03-28T15:37:21.559 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-03-28T15:37:21.559 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-03-28T15:37:21.559 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-03-28T15:37:21.559 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-03-28T15:37:21.559 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-03-28T15:37:21.559 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-03-28T15:37:21.559 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-03-28T15:37:21.559 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_smem.ko 2024-03-28T15:37:21.559 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_therm.ko 2024-03-28T15:37:21.559 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/wire.ko 2024-03-28T15:37:21.559 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog 2024-03-28T15:37:21.559 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/alim1535_wdt.ko 2024-03-28T15:37:21.559 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/alim7101_wdt.ko 2024-03-28T15:37:21.559 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/f71808e_wdt.ko 2024-03-28T15:37:21.559 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/hpwdt.ko 2024-03-28T15:37:21.560 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/i6300esb.ko 2024-03-28T15:37:21.560 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-03-28T15:37:21.560 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/iTCO_wdt.ko 2024-03-28T15:37:21.560 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ib700wdt.ko 2024-03-28T15:37:21.560 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ibmasr.ko 2024-03-28T15:37:21.560 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-03-28T15:37:21.560 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/it8712f_wdt.ko 2024-03-28T15:37:21.560 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/it87_wdt.ko 2024-03-28T15:37:21.560 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/machzwd.ko 2024-03-28T15:37:21.560 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/nv_tco.ko 2024-03-28T15:37:21.560 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/pcwd_pci.ko 2024-03-28T15:37:21.560 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/pcwd_usb.ko 2024-03-28T15:37:21.560 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-03-28T15:37:21.560 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sch311x_wdt.ko 2024-03-28T15:37:21.560 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/softdog.ko 2024-03-28T15:37:21.560 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sp5100_tco.ko 2024-03-28T15:37:21.561 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/via_wdt.ko 2024-03-28T15:37:21.561 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-03-28T15:37:21.561 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83877f_wdt.ko 2024-03-28T15:37:21.561 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83977f_wdt.ko 2024-03-28T15:37:21.561 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/wdt_pci.ko 2024-03-28T15:37:21.561 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/xen_wdt.ko 2024-03-28T15:37:21.561 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen 2024-03-28T15:37:21.561 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-acpi-processor.ko 2024-03-28T15:37:21.561 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-evtchn.ko 2024-03-28T15:37:21.561 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-gntalloc.ko 2024-03-28T15:37:21.561 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-gntdev.ko 2024-03-28T15:37:21.561 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-pciback 2024-03-28T15:37:21.561 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-03-28T15:37:21.561 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-privcmd.ko 2024-03-28T15:37:21.561 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xenfs 2024-03-28T15:37:21.562 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xenfs/xenfs.ko 2024-03-28T15:37:21.562 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs 2024-03-28T15:37:21.562 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/9p 2024-03-28T15:37:21.562 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/9p/9p.ko 2024-03-28T15:37:21.562 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/affs 2024-03-28T15:37:21.562 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/affs/affs.ko 2024-03-28T15:37:21.562 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/befs 2024-03-28T15:37:21.562 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/befs/befs.ko 2024-03-28T15:37:21.562 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/binfmt_misc.ko 2024-03-28T15:37:21.562 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/btrfs 2024-03-28T15:37:21.562 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/btrfs/btrfs.ko 2024-03-28T15:37:21.562 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cachefiles 2024-03-28T15:37:21.562 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cachefiles/cachefiles.ko 2024-03-28T15:37:21.562 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ceph 2024-03-28T15:37:21.562 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ceph/ceph.ko 2024-03-28T15:37:21.563 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/coda 2024-03-28T15:37:21.563 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/coda/coda.ko 2024-03-28T15:37:21.563 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cramfs 2024-03-28T15:37:21.563 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cramfs/cramfs.ko 2024-03-28T15:37:21.563 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/dlm 2024-03-28T15:37:21.563 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/dlm/dlm.ko 2024-03-28T15:37:21.563 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ecryptfs 2024-03-28T15:37:21.563 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ecryptfs/ecryptfs.ko 2024-03-28T15:37:21.563 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/f2fs 2024-03-28T15:37:21.563 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/f2fs/f2fs.ko 2024-03-28T15:37:21.563 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat 2024-03-28T15:37:21.563 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat/fat.ko 2024-03-28T15:37:21.563 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat/msdos.ko 2024-03-28T15:37:21.563 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat/vfat.ko 2024-03-28T15:37:21.563 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fuse 2024-03-28T15:37:21.564 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fuse/cuse.ko 2024-03-28T15:37:21.564 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fuse/fuse.ko 2024-03-28T15:37:21.564 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/gfs2 2024-03-28T15:37:21.564 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/gfs2/gfs2.ko 2024-03-28T15:37:21.564 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfs 2024-03-28T15:37:21.564 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfs/hfs.ko 2024-03-28T15:37:21.564 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfsplus 2024-03-28T15:37:21.564 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfsplus/hfsplus.ko 2024-03-28T15:37:21.564 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/isofs 2024-03-28T15:37:21.564 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/isofs/isofs.ko 2024-03-28T15:37:21.564 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/jfs 2024-03-28T15:37:21.564 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/jfs/jfs.ko 2024-03-28T15:37:21.564 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/lockd 2024-03-28T15:37:21.564 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/lockd/lockd.ko 2024-03-28T15:37:21.565 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/minix 2024-03-28T15:37:21.565 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/minix/minix.ko 2024-03-28T15:37:21.565 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/netfs 2024-03-28T15:37:21.565 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/netfs/netfs.ko 2024-03-28T15:37:21.565 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs 2024-03-28T15:37:21.565 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/blocklayout 2024-03-28T15:37:21.565 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-03-28T15:37:21.565 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/filelayout 2024-03-28T15:37:21.565 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-03-28T15:37:21.565 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/flexfilelayout 2024-03-28T15:37:21.565 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-03-28T15:37:21.565 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfs.ko 2024-03-28T15:37:21.565 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfsv3.ko 2024-03-28T15:37:21.565 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfsv4.ko 2024-03-28T15:37:21.565 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common 2024-03-28T15:37:21.566 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common/grace.ko 2024-03-28T15:37:21.566 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common/nfs_acl.ko 2024-03-28T15:37:21.566 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfsd 2024-03-28T15:37:21.566 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfsd/nfsd.ko 2024-03-28T15:37:21.566 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nilfs2 2024-03-28T15:37:21.566 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nilfs2/nilfs2.ko 2024-03-28T15:37:21.566 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls 2024-03-28T15:37:21.566 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-celtic.ko 2024-03-28T15:37:21.566 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-centeuro.ko 2024-03-28T15:37:21.566 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-croatian.ko 2024-03-28T15:37:21.566 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-cyrillic.ko 2024-03-28T15:37:21.566 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-gaelic.ko 2024-03-28T15:37:21.566 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-greek.ko 2024-03-28T15:37:21.566 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-iceland.ko 2024-03-28T15:37:21.566 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-inuit.ko 2024-03-28T15:37:21.567 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-roman.ko 2024-03-28T15:37:21.567 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-romanian.ko 2024-03-28T15:37:21.567 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-turkish.ko 2024-03-28T15:37:21.567 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1250.ko 2024-03-28T15:37:21.567 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1251.ko 2024-03-28T15:37:21.567 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1255.ko 2024-03-28T15:37:21.567 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp737.ko 2024-03-28T15:37:21.567 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp775.ko 2024-03-28T15:37:21.567 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp850.ko 2024-03-28T15:37:21.567 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp852.ko 2024-03-28T15:37:21.567 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp855.ko 2024-03-28T15:37:21.567 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp857.ko 2024-03-28T15:37:21.567 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp860.ko 2024-03-28T15:37:21.567 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp861.ko 2024-03-28T15:37:21.567 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp862.ko 2024-03-28T15:37:21.567 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp863.ko 2024-03-28T15:37:21.568 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp864.ko 2024-03-28T15:37:21.568 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp865.ko 2024-03-28T15:37:21.568 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp866.ko 2024-03-28T15:37:21.568 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp869.ko 2024-03-28T15:37:21.568 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp874.ko 2024-03-28T15:37:21.568 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp932.ko 2024-03-28T15:37:21.568 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp936.ko 2024-03-28T15:37:21.568 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp949.ko 2024-03-28T15:37:21.568 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp950.ko 2024-03-28T15:37:21.568 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_euc-jp.ko 2024-03-28T15:37:21.568 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-1.ko 2024-03-28T15:37:21.568 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-13.ko 2024-03-28T15:37:21.568 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-14.ko 2024-03-28T15:37:21.568 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-15.ko 2024-03-28T15:37:21.568 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-2.ko 2024-03-28T15:37:21.568 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-3.ko 2024-03-28T15:37:21.569 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-4.ko 2024-03-28T15:37:21.569 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-5.ko 2024-03-28T15:37:21.569 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-6.ko 2024-03-28T15:37:21.569 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-7.ko 2024-03-28T15:37:21.569 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-9.ko 2024-03-28T15:37:21.569 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-r.ko 2024-03-28T15:37:21.569 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-ru.ko 2024-03-28T15:37:21.569 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-u.ko 2024-03-28T15:37:21.569 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_ucs2_utils.ko 2024-03-28T15:37:21.569 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_utf8.ko 2024-03-28T15:37:21.569 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2 2024-03-28T15:37:21.569 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/cluster 2024-03-28T15:37:21.569 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-03-28T15:37:21.569 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlm 2024-03-28T15:37:21.570 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-03-28T15:37:21.570 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlmfs 2024-03-28T15:37:21.570 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-03-28T15:37:21.570 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2.ko 2024-03-28T15:37:21.570 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-03-28T15:37:21.570 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-03-28T15:37:21.570 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-03-28T15:37:21.570 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/overlayfs 2024-03-28T15:37:21.570 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/overlayfs/overlay.ko 2024-03-28T15:37:21.570 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/pstore 2024-03-28T15:37:21.570 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/pstore/ramoops.ko 2024-03-28T15:37:21.570 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/reiserfs 2024-03-28T15:37:21.570 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/reiserfs/reiserfs.ko 2024-03-28T15:37:21.570 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/romfs 2024-03-28T15:37:21.571 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/romfs/romfs.ko 2024-03-28T15:37:21.571 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb 2024-03-28T15:37:21.571 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/client 2024-03-28T15:37:21.571 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/client/cifs.ko 2024-03-28T15:37:21.571 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common 2024-03-28T15:37:21.571 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common/cifs_arc4.ko 2024-03-28T15:37:21.571 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common/cifs_md4.ko 2024-03-28T15:37:21.571 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/squashfs 2024-03-28T15:37:21.571 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/squashfs/squashfs.ko 2024-03-28T15:37:21.571 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/sysv 2024-03-28T15:37:21.571 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/sysv/sysv.ko 2024-03-28T15:37:21.571 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/udf 2024-03-28T15:37:21.571 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/udf/udf.ko 2024-03-28T15:37:21.571 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ufs 2024-03-28T15:37:21.571 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ufs/ufs.ko 2024-03-28T15:37:21.572 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/xfs 2024-03-28T15:37:21.572 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/xfs/xfs.ko 2024-03-28T15:37:21.572 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel 2024-03-28T15:37:21.572 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/rcu 2024-03-28T15:37:21.572 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/rcu/rcutorture.ko 2024-03-28T15:37:21.572 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/torture.ko 2024-03-28T15:37:21.572 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/trace 2024-03-28T15:37:21.572 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-03-28T15:37:21.572 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib 2024-03-28T15:37:21.572 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/asn1_encoder.ko 2024-03-28T15:37:21.572 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crc-itu-t.ko 2024-03-28T15:37:21.572 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crc8.ko 2024-03-28T15:37:21.572 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto 2024-03-28T15:37:21.572 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto/libarc4.ko 2024-03-28T15:37:21.573 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto/libdes.ko 2024-03-28T15:37:21.573 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/libcrc32c.ko 2024-03-28T15:37:21.573 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lru_cache.ko 2024-03-28T15:37:21.573 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4 2024-03-28T15:37:21.573 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4/lz4_compress.ko 2024-03-28T15:37:21.573 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4/lz4hc_compress.ko 2024-03-28T15:37:21.573 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/math 2024-03-28T15:37:21.573 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/math/cordic.ko 2024-03-28T15:37:21.573 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/raid6 2024-03-28T15:37:21.573 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/raid6/raid6_pq.ko 2024-03-28T15:37:21.573 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/reed_solomon 2024-03-28T15:37:21.573 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/reed_solomon/reed_solomon.ko 2024-03-28T15:37:21.573 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_bm.ko 2024-03-28T15:37:21.573 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_fsm.ko 2024-03-28T15:37:21.573 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_kmp.ko 2024-03-28T15:37:21.573 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/zstd 2024-03-28T15:37:21.573 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/zstd/zstd_compress.ko 2024-03-28T15:37:21.574 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/mm 2024-03-28T15:37:21.574 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/mm/hwpoison-inject.ko 2024-03-28T15:37:21.574 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net 2024-03-28T15:37:21.574 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan 2024-03-28T15:37:21.574 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/6lowpan.ko 2024-03-28T15:37:21.574 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_dest.ko 2024-03-28T15:37:21.574 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_fragment.ko 2024-03-28T15:37:21.574 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_hop.ko 2024-03-28T15:37:21.574 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_ipv6.ko 2024-03-28T15:37:21.574 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_mobility.ko 2024-03-28T15:37:21.574 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_routing.ko 2024-03-28T15:37:21.574 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_udp.ko 2024-03-28T15:37:21.574 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802 2024-03-28T15:37:21.574 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/garp.ko 2024-03-28T15:37:21.574 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/mrp.ko 2024-03-28T15:37:21.575 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/p8022.ko 2024-03-28T15:37:21.575 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/psnap.ko 2024-03-28T15:37:21.575 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/stp.ko 2024-03-28T15:37:21.575 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/8021q 2024-03-28T15:37:21.575 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/8021q/8021q.ko 2024-03-28T15:37:21.575 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p 2024-03-28T15:37:21.575 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet.ko 2024-03-28T15:37:21.575 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_fd.ko 2024-03-28T15:37:21.575 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_rdma.ko 2024-03-28T15:37:21.575 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_virtio.ko 2024-03-28T15:37:21.575 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/batman-adv 2024-03-28T15:37:21.575 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/batman-adv/batman-adv.ko 2024-03-28T15:37:21.575 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge 2024-03-28T15:37:21.575 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/br_netfilter.ko 2024-03-28T15:37:21.575 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/bridge.ko 2024-03-28T15:37:21.575 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter 2024-03-28T15:37:21.575 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-03-28T15:37:21.576 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_among.ko 2024-03-28T15:37:21.576 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_arp.ko 2024-03-28T15:37:21.576 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-03-28T15:37:21.576 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-03-28T15:37:21.576 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_ip.ko 2024-03-28T15:37:21.576 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-03-28T15:37:21.576 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_limit.ko 2024-03-28T15:37:21.576 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_log.ko 2024-03-28T15:37:21.576 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_mark.ko 2024-03-28T15:37:21.576 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-03-28T15:37:21.576 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-03-28T15:37:21.576 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-03-28T15:37:21.576 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-03-28T15:37:21.576 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_snat.ko 2024-03-28T15:37:21.576 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_stp.ko 2024-03-28T15:37:21.577 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-03-28T15:37:21.577 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-03-28T15:37:21.577 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-03-28T15:37:21.577 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-03-28T15:37:21.577 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtables.ko 2024-03-28T15:37:21.577 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-03-28T15:37:21.577 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-03-28T15:37:21.577 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-03-28T15:37:21.577 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ceph 2024-03-28T15:37:21.577 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ceph/libceph.ko 2024-03-28T15:37:21.577 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/core 2024-03-28T15:37:21.577 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/core/failover.ko 2024-03-28T15:37:21.577 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/core/pktgen.ko 2024-03-28T15:37:21.577 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp 2024-03-28T15:37:21.577 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp.ko 2024-03-28T15:37:21.577 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_diag.ko 2024-03-28T15:37:21.578 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_ipv4.ko 2024-03-28T15:37:21.578 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_ipv6.ko 2024-03-28T15:37:21.578 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dns_resolver 2024-03-28T15:37:21.578 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dns_resolver/dns_resolver.ko 2024-03-28T15:37:21.578 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa 2024-03-28T15:37:21.578 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/dsa_core.ko 2024-03-28T15:37:21.578 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_brcm.ko 2024-03-28T15:37:21.578 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_dsa.ko 2024-03-28T15:37:21.578 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_none.ko 2024-03-28T15:37:21.578 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_trailer.ko 2024-03-28T15:37:21.578 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154 2024-03-28T15:37:21.578 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/6lowpan 2024-03-28T15:37:21.578 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-03-28T15:37:21.578 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/ieee802154.ko 2024-03-28T15:37:21.578 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/ieee802154_socket.ko 2024-03-28T15:37:21.579 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4 2024-03-28T15:37:21.579 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ah4.ko 2024-03-28T15:37:21.579 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/esp4.ko 2024-03-28T15:37:21.579 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/fou.ko 2024-03-28T15:37:21.579 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/gre.ko 2024-03-28T15:37:21.579 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/inet_diag.ko 2024-03-28T15:37:21.579 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_gre.ko 2024-03-28T15:37:21.579 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_tunnel.ko 2024-03-28T15:37:21.579 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_vti.ko 2024-03-28T15:37:21.579 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ipcomp.ko 2024-03-28T15:37:21.579 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ipip.ko 2024-03-28T15:37:21.579 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter 2024-03-28T15:37:21.579 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arp_tables.ko 2024-03-28T15:37:21.579 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-03-28T15:37:21.579 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-03-28T15:37:21.580 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-03-28T15:37:21.580 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-03-28T15:37:21.580 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-03-28T15:37:21.580 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-03-28T15:37:21.580 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-03-28T15:37:21.580 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-03-28T15:37:21.580 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-03-28T15:37:21.580 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_security.ko 2024-03-28T15:37:21.580 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-03-28T15:37:21.580 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-03-28T15:37:21.580 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-03-28T15:37:21.580 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-03-28T15:37:21.580 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-03-28T15:37:21.580 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-03-28T15:37:21.581 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-03-28T15:37:21.581 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-03-28T15:37:21.581 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-03-28T15:37:21.581 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-03-28T15:37:21.581 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_bic.ko 2024-03-28T15:37:21.581 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_dctcp.ko 2024-03-28T15:37:21.581 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_diag.ko 2024-03-28T15:37:21.581 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_highspeed.ko 2024-03-28T15:37:21.581 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_htcp.ko 2024-03-28T15:37:21.581 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_hybla.ko 2024-03-28T15:37:21.581 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_illinois.ko 2024-03-28T15:37:21.581 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_lp.ko 2024-03-28T15:37:21.581 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_scalable.ko 2024-03-28T15:37:21.581 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_vegas.ko 2024-03-28T15:37:21.581 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_veno.ko 2024-03-28T15:37:21.581 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_westwood.ko 2024-03-28T15:37:21.581 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_yeah.ko 2024-03-28T15:37:21.582 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tunnel4.ko 2024-03-28T15:37:21.582 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/udp_diag.ko 2024-03-28T15:37:21.582 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/udp_tunnel.ko 2024-03-28T15:37:21.582 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/xfrm4_tunnel.ko 2024-03-28T15:37:21.582 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6 2024-03-28T15:37:21.582 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ah6.ko 2024-03-28T15:37:21.582 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/esp6.ko 2024-03-28T15:37:21.582 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/fou6.ko 2024-03-28T15:37:21.582 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_tunnel.ko 2024-03-28T15:37:21.582 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-03-28T15:37:21.582 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_vti.ko 2024-03-28T15:37:21.582 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ipcomp6.ko 2024-03-28T15:37:21.583 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter 2024-03-28T15:37:21.583 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-03-28T15:37:21.583 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-03-28T15:37:21.583 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-03-28T15:37:21.583 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-03-28T15:37:21.583 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-03-28T15:37:21.583 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-03-28T15:37:21.583 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-03-28T15:37:21.583 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-03-28T15:37:21.583 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-03-28T15:37:21.583 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-03-28T15:37:21.583 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-03-28T15:37:21.583 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-03-28T15:37:21.583 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-03-28T15:37:21.583 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-03-28T15:37:21.583 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-03-28T15:37:21.584 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-03-28T15:37:21.584 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-03-28T15:37:21.584 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-03-28T15:37:21.584 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-03-28T15:37:21.584 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-03-28T15:37:21.584 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-03-28T15:37:21.584 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-03-28T15:37:21.584 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-03-28T15:37:21.584 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-03-28T15:37:21.584 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/sit.ko 2024-03-28T15:37:21.584 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/tunnel6.ko 2024-03-28T15:37:21.584 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/xfrm6_tunnel.ko 2024-03-28T15:37:21.584 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/key 2024-03-28T15:37:21.584 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/key/af_key.ko 2024-03-28T15:37:21.584 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp 2024-03-28T15:37:21.585 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_core.ko 2024-03-28T15:37:21.585 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_debugfs.ko 2024-03-28T15:37:21.585 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_eth.ko 2024-03-28T15:37:21.585 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ip.ko 2024-03-28T15:37:21.585 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ip6.ko 2024-03-28T15:37:21.585 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_netlink.ko 2024-03-28T15:37:21.585 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ppp.ko 2024-03-28T15:37:21.585 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/llc 2024-03-28T15:37:21.585 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/llc/llc.ko 2024-03-28T15:37:21.585 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mac802154 2024-03-28T15:37:21.585 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mac802154/mac802154.ko 2024-03-28T15:37:21.585 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mpls 2024-03-28T15:37:21.585 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mpls/mpls_gso.ko 2024-03-28T15:37:21.585 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter 2024-03-28T15:37:21.585 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset 2024-03-28T15:37:21.585 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set.ko 2024-03-28T15:37:21.586 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-03-28T15:37:21.586 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-03-28T15:37:21.586 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-03-28T15:37:21.586 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-03-28T15:37:21.586 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-03-28T15:37:21.586 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-03-28T15:37:21.586 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-03-28T15:37:21.586 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-03-28T15:37:21.586 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-03-28T15:37:21.586 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-03-28T15:37:21.586 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-03-28T15:37:21.586 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-03-28T15:37:21.586 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-03-28T15:37:21.586 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-03-28T15:37:21.586 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-03-28T15:37:21.587 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs 2024-03-28T15:37:21.587 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs.ko 2024-03-28T15:37:21.587 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-03-28T15:37:21.587 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-03-28T15:37:21.587 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-03-28T15:37:21.587 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-03-28T15:37:21.587 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-03-28T15:37:21.587 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-03-28T15:37:21.587 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-03-28T15:37:21.587 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-03-28T15:37:21.587 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-03-28T15:37:21.587 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-03-28T15:37:21.587 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-03-28T15:37:21.587 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-03-28T15:37:21.587 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-03-28T15:37:21.587 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conncount.ko 2024-03-28T15:37:21.588 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack.ko 2024-03-28T15:37:21.588 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-03-28T15:37:21.588 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-03-28T15:37:21.588 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-03-28T15:37:21.588 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_h323.ko 2024-03-28T15:37:21.588 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_irc.ko 2024-03-28T15:37:21.588 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-03-28T15:37:21.588 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-03-28T15:37:21.588 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-03-28T15:37:21.588 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_sane.ko 2024-03-28T15:37:21.588 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_sip.ko 2024-03-28T15:37:21.588 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-03-28T15:37:21.588 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-03-28T15:37:21.588 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_dup_netdev.ko 2024-03-28T15:37:21.589 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_flow_table.ko 2024-03-28T15:37:21.589 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_flow_table_inet.ko 2024-03-28T15:37:21.589 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_log_syslog.ko 2024-03-28T15:37:21.589 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat.ko 2024-03-28T15:37:21.589 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_amanda.ko 2024-03-28T15:37:21.589 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_ftp.ko 2024-03-28T15:37:21.589 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_irc.ko 2024-03-28T15:37:21.589 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_sip.ko 2024-03-28T15:37:21.589 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_tftp.ko 2024-03-28T15:37:21.589 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_synproxy_core.ko 2024-03-28T15:37:21.589 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_tables.ko 2024-03-28T15:37:21.589 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink.ko 2024-03-28T15:37:21.589 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_acct.ko 2024-03-28T15:37:21.589 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_log.ko 2024-03-28T15:37:21.589 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_osf.ko 2024-03-28T15:37:21.590 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_queue.ko 2024-03-28T15:37:21.590 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_chain_nat.ko 2024-03-28T15:37:21.590 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_compat.ko 2024-03-28T15:37:21.590 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_ct.ko 2024-03-28T15:37:21.590 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_dup_netdev.ko 2024-03-28T15:37:21.590 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_fib.ko 2024-03-28T15:37:21.590 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_fwd_netdev.ko 2024-03-28T15:37:21.590 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_hash.ko 2024-03-28T15:37:21.590 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_limit.ko 2024-03-28T15:37:21.590 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_log.ko 2024-03-28T15:37:21.590 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_masq.ko 2024-03-28T15:37:21.590 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_nat.ko 2024-03-28T15:37:21.590 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_numgen.ko 2024-03-28T15:37:21.590 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_queue.ko 2024-03-28T15:37:21.590 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_quota.ko 2024-03-28T15:37:21.591 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_redir.ko 2024-03-28T15:37:21.591 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_reject.ko 2024-03-28T15:37:21.591 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_reject_inet.ko 2024-03-28T15:37:21.591 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_socket.ko 2024-03-28T15:37:21.591 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_tunnel.ko 2024-03-28T15:37:21.591 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_AUDIT.ko 2024-03-28T15:37:21.591 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CHECKSUM.ko 2024-03-28T15:37:21.591 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CLASSIFY.ko 2024-03-28T15:37:21.591 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-03-28T15:37:21.591 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CT.ko 2024-03-28T15:37:21.591 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_DSCP.ko 2024-03-28T15:37:21.591 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_HL.ko 2024-03-28T15:37:21.591 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_HMARK.ko 2024-03-28T15:37:21.591 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_IDLETIMER.ko 2024-03-28T15:37:21.591 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_LED.ko 2024-03-28T15:37:21.592 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_LOG.ko 2024-03-28T15:37:21.592 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_MASQUERADE.ko 2024-03-28T15:37:21.592 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NETMAP.ko 2024-03-28T15:37:21.592 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NFLOG.ko 2024-03-28T15:37:21.592 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NFQUEUE.ko 2024-03-28T15:37:21.592 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_RATEEST.ko 2024-03-28T15:37:21.592 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_REDIRECT.ko 2024-03-28T15:37:21.592 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_SECMARK.ko 2024-03-28T15:37:21.592 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TCPMSS.ko 2024-03-28T15:37:21.592 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-03-28T15:37:21.592 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TEE.ko 2024-03-28T15:37:21.592 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TPROXY.ko 2024-03-28T15:37:21.592 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TRACE.ko 2024-03-28T15:37:21.592 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_addrtype.ko 2024-03-28T15:37:21.592 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_bpf.ko 2024-03-28T15:37:21.592 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cgroup.ko 2024-03-28T15:37:21.592 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cluster.ko 2024-03-28T15:37:21.593 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_comment.ko 2024-03-28T15:37:21.593 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connbytes.ko 2024-03-28T15:37:21.593 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connlabel.ko 2024-03-28T15:37:21.593 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connlimit.ko 2024-03-28T15:37:21.593 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connmark.ko 2024-03-28T15:37:21.593 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_conntrack.ko 2024-03-28T15:37:21.593 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cpu.ko 2024-03-28T15:37:21.593 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_dccp.ko 2024-03-28T15:37:21.593 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_devgroup.ko 2024-03-28T15:37:21.593 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_dscp.ko 2024-03-28T15:37:21.593 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ecn.ko 2024-03-28T15:37:21.593 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_esp.ko 2024-03-28T15:37:21.593 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_hashlimit.ko 2024-03-28T15:37:21.593 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_helper.ko 2024-03-28T15:37:21.593 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_hl.ko 2024-03-28T15:37:21.593 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ipcomp.ko 2024-03-28T15:37:21.593 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_iprange.ko 2024-03-28T15:37:21.594 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ipvs.ko 2024-03-28T15:37:21.594 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_l2tp.ko 2024-03-28T15:37:21.594 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_length.ko 2024-03-28T15:37:21.594 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_limit.ko 2024-03-28T15:37:21.594 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_mac.ko 2024-03-28T15:37:21.594 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_mark.ko 2024-03-28T15:37:21.594 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_multiport.ko 2024-03-28T15:37:21.594 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_nat.ko 2024-03-28T15:37:21.594 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_nfacct.ko 2024-03-28T15:37:21.594 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_osf.ko 2024-03-28T15:37:21.594 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_owner.ko 2024-03-28T15:37:21.594 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_physdev.ko 2024-03-28T15:37:21.594 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_pkttype.ko 2024-03-28T15:37:21.594 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_policy.ko 2024-03-28T15:37:21.594 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_quota.ko 2024-03-28T15:37:21.595 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_rateest.ko 2024-03-28T15:37:21.595 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_realm.ko 2024-03-28T15:37:21.595 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_recent.ko 2024-03-28T15:37:21.595 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_sctp.ko 2024-03-28T15:37:21.595 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_set.ko 2024-03-28T15:37:21.595 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_socket.ko 2024-03-28T15:37:21.595 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_state.ko 2024-03-28T15:37:21.595 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_statistic.ko 2024-03-28T15:37:21.595 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_string.ko 2024-03-28T15:37:21.595 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_tcpmss.ko 2024-03-28T15:37:21.595 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_time.ko 2024-03-28T15:37:21.595 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_u32.ko 2024-03-28T15:37:21.595 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netlink 2024-03-28T15:37:21.595 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netlink/netlink_diag.ko 2024-03-28T15:37:21.595 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/nsh 2024-03-28T15:37:21.595 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/nsh/nsh.ko 2024-03-28T15:37:21.596 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch 2024-03-28T15:37:21.596 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/openvswitch.ko 2024-03-28T15:37:21.596 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-geneve.ko 2024-03-28T15:37:21.596 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-gre.ko 2024-03-28T15:37:21.596 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-vxlan.ko 2024-03-28T15:37:21.596 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/packet 2024-03-28T15:37:21.596 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/packet/af_packet_diag.ko 2024-03-28T15:37:21.596 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds 2024-03-28T15:37:21.596 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds.ko 2024-03-28T15:37:21.596 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds_rdma.ko 2024-03-28T15:37:21.596 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds_tcp.ko 2024-03-28T15:37:21.596 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched 2024-03-28T15:37:21.596 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_csum.ko 2024-03-28T15:37:21.596 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_gact.ko 2024-03-28T15:37:21.596 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_mirred.ko 2024-03-28T15:37:21.597 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_nat.ko 2024-03-28T15:37:21.597 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_pedit.ko 2024-03-28T15:37:21.597 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_police.ko 2024-03-28T15:37:21.597 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_simple.ko 2024-03-28T15:37:21.597 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_skbedit.ko 2024-03-28T15:37:21.597 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_vlan.ko 2024-03-28T15:37:21.597 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_basic.ko 2024-03-28T15:37:21.597 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_bpf.ko 2024-03-28T15:37:21.597 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_flow.ko 2024-03-28T15:37:21.597 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_fw.ko 2024-03-28T15:37:21.597 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_route.ko 2024-03-28T15:37:21.597 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_u32.ko 2024-03-28T15:37:21.597 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_cmp.ko 2024-03-28T15:37:21.597 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_ipset.ko 2024-03-28T15:37:21.597 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_meta.ko 2024-03-28T15:37:21.597 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_nbyte.ko 2024-03-28T15:37:21.597 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_text.ko 2024-03-28T15:37:21.598 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_u32.ko 2024-03-28T15:37:21.598 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_choke.ko 2024-03-28T15:37:21.598 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_codel.ko 2024-03-28T15:37:21.598 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_drr.ko 2024-03-28T15:37:21.598 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_fq.ko 2024-03-28T15:37:21.598 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_gred.ko 2024-03-28T15:37:21.598 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_hfsc.ko 2024-03-28T15:37:21.598 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_hhf.ko 2024-03-28T15:37:21.598 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_htb.ko 2024-03-28T15:37:21.598 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_ingress.ko 2024-03-28T15:37:21.598 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_mqprio.ko 2024-03-28T15:37:21.598 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_mqprio_lib.ko 2024-03-28T15:37:21.598 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_multiq.ko 2024-03-28T15:37:21.598 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_netem.ko 2024-03-28T15:37:21.598 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_pie.ko 2024-03-28T15:37:21.599 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_plug.ko 2024-03-28T15:37:21.599 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_prio.ko 2024-03-28T15:37:21.599 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_qfq.ko 2024-03-28T15:37:21.599 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_red.ko 2024-03-28T15:37:21.599 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_sfb.ko 2024-03-28T15:37:21.599 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_sfq.ko 2024-03-28T15:37:21.599 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_tbf.ko 2024-03-28T15:37:21.599 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_teql.ko 2024-03-28T15:37:21.599 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp 2024-03-28T15:37:21.599 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp/sctp.ko 2024-03-28T15:37:21.599 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp/sctp_diag.ko 2024-03-28T15:37:21.599 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc 2024-03-28T15:37:21.599 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss 2024-03-28T15:37:21.599 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-03-28T15:37:21.599 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-03-28T15:37:21.599 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/sunrpc.ko 2024-03-28T15:37:21.599 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/xprtrdma 2024-03-28T15:37:21.600 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-03-28T15:37:21.600 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc 2024-03-28T15:37:21.600 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc/diag.ko 2024-03-28T15:37:21.600 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc/tipc.ko 2024-03-28T15:37:21.600 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/unix 2024-03-28T15:37:21.600 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/unix/unix_diag.ko 2024-03-28T15:37:21.600 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock 2024-03-28T15:37:21.600 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-03-28T15:37:21.600 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-03-28T15:37:21.600 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock.ko 2024-03-28T15:37:21.600 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock_diag.ko 2024-03-28T15:37:21.600 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock_loopback.ko 2024-03-28T15:37:21.600 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/xfrm 2024-03-28T15:37:21.600 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/xfrm/xfrm_ipcomp.ko 2024-03-28T15:37:21.600 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security 2024-03-28T15:37:21.601 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys 2024-03-28T15:37:21.601 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/encrypted-keys 2024-03-28T15:37:21.601 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-03-28T15:37:21.601 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/trusted-keys 2024-03-28T15:37:21.601 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/trusted-keys/trusted.ko 2024-03-28T15:37:21.601 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/virt 2024-03-28T15:37:21.601 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/virt/lib 2024-03-28T15:37:21.601 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/virt/lib/irqbypass.ko 2024-03-28T15:37:21.601 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.alias 2024-03-28T15:37:21.601 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.alias.bin 2024-03-28T15:37:21.601 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin 2024-03-28T15:37:21.601 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin.alias.bin 2024-03-28T15:37:21.601 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin.bin 2024-03-28T15:37:21.601 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin.modinfo 2024-03-28T15:37:21.601 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.dep 2024-03-28T15:37:21.602 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.dep.bin 2024-03-28T15:37:21.602 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.devname 2024-03-28T15:37:21.602 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.order 2024-03-28T15:37:21.602 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.softdep 2024-03-28T15:37:21.602 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.symbols 2024-03-28T15:37:21.602 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.symbols.bin 2024-03-28T15:37:21.602 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.8.0-g1df0c7e8a235/vmlinuz 2024-03-28T15:37:21.606 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-g1df0c7e8a235 2024-03-28T15:37:21.606 DEBUG:teuthology.orchestra.run.smithi067:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-03-28T15:37:21.630 INFO:teuthology.orchestra.run.smithi067.stdout:postinstall scriptlet (using /bin/sh): 2024-03-28T15:37:21.630 INFO:teuthology.orchestra.run.smithi067.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-03-28T15:37:21.631 INFO:teuthology.orchestra.run.smithi067.stdout: /usr/bin/kernel-install add 6.8.0-g1df0c7e8a235 /lib/modules/6.8.0-g1df0c7e8a235/vmlinuz 2024-03-28T15:37:21.631 INFO:teuthology.orchestra.run.smithi067.stdout:fi 2024-03-28T15:37:21.631 INFO:teuthology.orchestra.run.smithi067.stdout:for file in vmlinuz System.map config; do 2024-03-28T15:37:21.631 INFO:teuthology.orchestra.run.smithi067.stdout: if ! cmp --silent "/lib/modules/6.8.0-g1df0c7e8a235/${file}" "/boot/${file}-6.8.0-g1df0c7e8a235"; then 2024-03-28T15:37:21.631 INFO:teuthology.orchestra.run.smithi067.stdout: cp "/lib/modules/6.8.0-g1df0c7e8a235/${file}" "/boot/${file}-6.8.0-g1df0c7e8a235" 2024-03-28T15:37:21.631 INFO:teuthology.orchestra.run.smithi067.stdout: fi 2024-03-28T15:37:21.631 INFO:teuthology.orchestra.run.smithi067.stdout:done 2024-03-28T15:37:21.631 INFO:teuthology.orchestra.run.smithi067.stdout:preuninstall scriptlet (using /bin/sh): 2024-03-28T15:37:21.631 INFO:teuthology.orchestra.run.smithi067.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-03-28T15:37:21.631 INFO:teuthology.orchestra.run.smithi067.stdout:new-kernel-pkg --remove 6.8.0-g1df0c7e8a235 --rminitrd --initrdfile=/boot/initramfs-6.8.0-g1df0c7e8a235.img 2024-03-28T15:37:21.631 INFO:teuthology.orchestra.run.smithi067.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-03-28T15:37:21.631 INFO:teuthology.orchestra.run.smithi067.stdout:kernel-install remove 6.8.0-g1df0c7e8a235 2024-03-28T15:37:21.632 INFO:teuthology.orchestra.run.smithi067.stdout:fi 2024-03-28T15:37:21.632 INFO:teuthology.orchestra.run.smithi067.stdout:postuninstall scriptlet (using /bin/sh): 2024-03-28T15:37:21.632 INFO:teuthology.orchestra.run.smithi067.stdout:if [ -x /sbin/update-bootloader ]; then 2024-03-28T15:37:21.632 INFO:teuthology.orchestra.run.smithi067.stdout:/sbin/update-bootloader --remove 6.8.0-g1df0c7e8a235 2024-03-28T15:37:21.632 INFO:teuthology.orchestra.run.smithi067.stdout:fi 2024-03-28T15:37:21.633 DEBUG:teuthology.orchestra.run.smithi067:> sudo rpm -qi grub2-tools 2024-03-28T15:37:21.718 INFO:teuthology.orchestra.run.smithi067.stdout:Name : grub2-tools 2024-03-28T15:37:21.718 INFO:teuthology.orchestra.run.smithi067.stdout:Epoch : 1 2024-03-28T15:37:21.718 INFO:teuthology.orchestra.run.smithi067.stdout:Version : 2.06 2024-03-28T15:37:21.718 INFO:teuthology.orchestra.run.smithi067.stdout:Release : 68.el9 2024-03-28T15:37:21.719 INFO:teuthology.orchestra.run.smithi067.stdout:Architecture: x86_64 2024-03-28T15:37:21.719 INFO:teuthology.orchestra.run.smithi067.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-28T15:37:21.719 INFO:teuthology.orchestra.run.smithi067.stdout:Group : Unspecified 2024-03-28T15:37:21.719 INFO:teuthology.orchestra.run.smithi067.stdout:Size : 8264254 2024-03-28T15:37:21.719 INFO:teuthology.orchestra.run.smithi067.stdout:License : GPLv3+ 2024-03-28T15:37:21.719 INFO:teuthology.orchestra.run.smithi067.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-28T15:37:21.719 INFO:teuthology.orchestra.run.smithi067.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-28T15:37:21.719 INFO:teuthology.orchestra.run.smithi067.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-28T15:37:21.719 INFO:teuthology.orchestra.run.smithi067.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-28T15:37:21.719 INFO:teuthology.orchestra.run.smithi067.stdout:Packager : builder@centos.org 2024-03-28T15:37:21.719 INFO:teuthology.orchestra.run.smithi067.stdout:Vendor : CentOS 2024-03-28T15:37:21.719 INFO:teuthology.orchestra.run.smithi067.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-28T15:37:21.719 INFO:teuthology.orchestra.run.smithi067.stdout:Summary : Support tools for GRUB. 2024-03-28T15:37:21.719 INFO:teuthology.orchestra.run.smithi067.stdout:Description : 2024-03-28T15:37:21.719 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T15:37:21.719 INFO:teuthology.orchestra.run.smithi067.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-28T15:37:21.720 INFO:teuthology.orchestra.run.smithi067.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-28T15:37:21.720 INFO:teuthology.orchestra.run.smithi067.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-28T15:37:21.720 INFO:teuthology.orchestra.run.smithi067.stdout:hardware devices. 2024-03-28T15:37:21.720 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T15:37:21.720 INFO:teuthology.orchestra.run.smithi067.stdout:This subpackage provides tools for support of all platforms. 2024-03-28T15:37:21.721 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-28T15:37:21.721 INFO:teuthology.task.kernel:Updating grub on smithi067 to boot 6.8.0-g1df0c7e8a235 2024-03-28T15:37:21.721 DEBUG:teuthology.orchestra.run.smithi067:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-28T15:37:22.296 INFO:teuthology.orchestra.run.smithi067.stderr:Generating grub configuration file ... 2024-03-28T15:37:23.791 INFO:teuthology.orchestra.run.smithi067.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-28T15:37:23.833 INFO:teuthology.orchestra.run.smithi067.stderr:done 2024-03-28T15:37:23.835 DEBUG:teuthology.orchestra.run.smithi067:> mktemp 2024-03-28T15:37:23.851 INFO:teuthology.orchestra.run.smithi067.stdout:/tmp/tmp.nxohyBQu0C 2024-03-28T15:37:23.851 DEBUG:teuthology.orchestra.run.smithi067:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.nxohyBQu0C 2024-03-28T15:37:23.922 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0666 /tmp/tmp.nxohyBQu0C 2024-03-28T15:37:24.079 DEBUG:teuthology.orchestra.remote:smithi067:/tmp/tmp.nxohyBQu0C is 6KB 2024-03-28T15:37:24.127 DEBUG:teuthology.orchestra.run.smithi067:> rm -fr /tmp/tmp.nxohyBQu0C 2024-03-28T15:37:24.142 DEBUG:teuthology.orchestra.run.smithi067:> sudo /bin/ls /boot/loader/entries || true 2024-03-28T15:37:24.212 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi150.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-03-28T15:37:24.212 DEBUG:teuthology.orchestra.run.smithi150:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-03-28T15:37:24.221 INFO:teuthology.orchestra.run.smithi067.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-28T15:37:24.221 INFO:teuthology.orchestra.run.smithi067.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-28T15:37:24.221 INFO:teuthology.orchestra.run.smithi067.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-28T15:37:24.221 INFO:teuthology.orchestra.run.smithi067.stdout:15949e3c88704300882482eddfd7baae-6.8.0-g1df0c7e8a235.conf 2024-03-28T15:37:24.222 DEBUG:teuthology.orchestra.run.smithi067:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.8.0-g1df0c7e8a235 2024-03-28T15:37:24.282 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235 2024-03-28T15:37:24.282 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/System.map 2024-03-28T15:37:24.282 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/config 2024-03-28T15:37:24.282 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel 2024-03-28T15:37:24.282 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch 2024-03-28T15:37:24.282 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86 2024-03-28T15:37:24.282 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto 2024-03-28T15:37:24.282 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-03-28T15:37:24.282 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-03-28T15:37:24.283 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-03-28T15:37:24.283 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-03-28T15:37:24.283 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-03-28T15:37:24.283 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-03-28T15:37:24.283 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-03-28T15:37:24.283 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/crc32c-intel.ko 2024-03-28T15:37:24.283 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-03-28T15:37:24.283 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-03-28T15:37:24.283 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-03-28T15:37:24.283 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-03-28T15:37:24.283 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-avx2.ko 2024-03-28T15:37:24.283 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-03-28T15:37:24.283 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-03-28T15:37:24.283 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-03-28T15:37:24.284 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-03-28T15:37:24.284 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-03-28T15:37:24.284 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-03-28T15:37:24.284 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-03-28T15:37:24.284 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel 2024-03-28T15:37:24.284 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu 2024-03-28T15:37:24.284 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu/mce 2024-03-28T15:37:24.284 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-03-28T15:37:24.284 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm 2024-03-28T15:37:24.284 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm-amd.ko 2024-03-28T15:37:24.284 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm-intel.ko 2024-03-28T15:37:24.284 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm.ko 2024-03-28T15:37:24.284 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto 2024-03-28T15:37:24.284 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/ansi_cprng.ko 2024-03-28T15:37:24.284 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/anubis.ko 2024-03-28T15:37:24.285 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/arc4.ko 2024-03-28T15:37:24.285 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx 2024-03-28T15:37:24.285 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_memcpy.ko 2024-03-28T15:37:24.285 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_pq.ko 2024-03-28T15:37:24.285 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_raid6_recov.ko 2024-03-28T15:37:24.285 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_tx.ko 2024-03-28T15:37:24.285 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_xor.ko 2024-03-28T15:37:24.285 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/raid6test.ko 2024-03-28T15:37:24.285 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/authenc.ko 2024-03-28T15:37:24.285 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/authencesn.ko 2024-03-28T15:37:24.285 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blake2b_generic.ko 2024-03-28T15:37:24.285 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blowfish_common.ko 2024-03-28T15:37:24.285 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blowfish_generic.ko 2024-03-28T15:37:24.285 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/camellia_generic.ko 2024-03-28T15:37:24.285 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast5_generic.ko 2024-03-28T15:37:24.286 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast6_generic.ko 2024-03-28T15:37:24.286 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast_common.ko 2024-03-28T15:37:24.286 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/ccm.ko 2024-03-28T15:37:24.286 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cmac.ko 2024-03-28T15:37:24.286 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crc32_generic.ko 2024-03-28T15:37:24.286 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crypto_engine.ko 2024-03-28T15:37:24.286 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crypto_user.ko 2024-03-28T15:37:24.286 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cts.ko 2024-03-28T15:37:24.286 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/des_generic.ko 2024-03-28T15:37:24.286 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/dh_generic.ko 2024-03-28T15:37:24.286 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/echainiv.ko 2024-03-28T15:37:24.286 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/essiv.ko 2024-03-28T15:37:24.286 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/fcrypt.ko 2024-03-28T15:37:24.286 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/khazad.ko 2024-03-28T15:37:24.287 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/lz4.ko 2024-03-28T15:37:24.287 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/lz4hc.ko 2024-03-28T15:37:24.287 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/md4.ko 2024-03-28T15:37:24.287 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/michael_mic.ko 2024-03-28T15:37:24.287 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/pcbc.ko 2024-03-28T15:37:24.287 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/pcrypt.ko 2024-03-28T15:37:24.287 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/rmd160.ko 2024-03-28T15:37:24.287 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/seed.ko 2024-03-28T15:37:24.287 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/serpent_generic.ko 2024-03-28T15:37:24.287 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/tcrypt.ko 2024-03-28T15:37:24.287 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/tea.ko 2024-03-28T15:37:24.287 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/twofish_common.ko 2024-03-28T15:37:24.287 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/twofish_generic.ko 2024-03-28T15:37:24.287 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/vmac.ko 2024-03-28T15:37:24.288 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/wp512.ko 2024-03-28T15:37:24.288 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xcbc.ko 2024-03-28T15:37:24.288 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xor.ko 2024-03-28T15:37:24.288 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xxhash_generic.ko 2024-03-28T15:37:24.288 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers 2024-03-28T15:37:24.288 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi 2024-03-28T15:37:24.288 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/acpi_ipmi.ko 2024-03-28T15:37:24.288 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/acpi_pad.ko 2024-03-28T15:37:24.288 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/custom_method.ko 2024-03-28T15:37:24.288 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/ec_sys.ko 2024-03-28T15:37:24.288 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/platform_profile.ko 2024-03-28T15:37:24.288 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/sbs.ko 2024-03-28T15:37:24.288 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/sbshc.ko 2024-03-28T15:37:24.288 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/video.ko 2024-03-28T15:37:24.288 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata 2024-03-28T15:37:24.289 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/acard-ahci.ko 2024-03-28T15:37:24.289 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/ahci_platform.ko 2024-03-28T15:37:24.289 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/ata_generic.ko 2024-03-28T15:37:24.289 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/libahci_platform.ko 2024-03-28T15:37:24.289 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_acpi.ko 2024-03-28T15:37:24.289 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ali.ko 2024-03-28T15:37:24.289 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_amd.ko 2024-03-28T15:37:24.289 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_artop.ko 2024-03-28T15:37:24.289 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_atiixp.ko 2024-03-28T15:37:24.289 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_atp867x.ko 2024-03-28T15:37:24.289 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cmd640.ko 2024-03-28T15:37:24.289 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cmd64x.ko 2024-03-28T15:37:24.289 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cypress.ko 2024-03-28T15:37:24.289 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_efar.ko 2024-03-28T15:37:24.289 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt366.ko 2024-03-28T15:37:24.289 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt37x.ko 2024-03-28T15:37:24.290 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt3x2n.ko 2024-03-28T15:37:24.290 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt3x3.ko 2024-03-28T15:37:24.290 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_it8213.ko 2024-03-28T15:37:24.290 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_it821x.ko 2024-03-28T15:37:24.290 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_jmicron.ko 2024-03-28T15:37:24.290 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_marvell.ko 2024-03-28T15:37:24.290 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_mpiix.ko 2024-03-28T15:37:24.290 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_netcell.ko 2024-03-28T15:37:24.290 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ninja32.ko 2024-03-28T15:37:24.290 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ns87410.ko 2024-03-28T15:37:24.290 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ns87415.ko 2024-03-28T15:37:24.290 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_oldpiix.ko 2024-03-28T15:37:24.290 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_opti.ko 2024-03-28T15:37:24.290 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_optidma.ko 2024-03-28T15:37:24.290 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pcmcia.ko 2024-03-28T15:37:24.291 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pdc2027x.ko 2024-03-28T15:37:24.291 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-03-28T15:37:24.291 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_piccolo.ko 2024-03-28T15:37:24.291 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_rdc.ko 2024-03-28T15:37:24.291 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sch.ko 2024-03-28T15:37:24.291 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_serverworks.ko 2024-03-28T15:37:24.291 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sil680.ko 2024-03-28T15:37:24.291 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sis.ko 2024-03-28T15:37:24.291 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sl82c105.ko 2024-03-28T15:37:24.291 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_triflex.ko 2024-03-28T15:37:24.291 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_via.ko 2024-03-28T15:37:24.291 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pdc_adma.ko 2024-03-28T15:37:24.291 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_inic162x.ko 2024-03-28T15:37:24.291 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_mv.ko 2024-03-28T15:37:24.292 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_nv.ko 2024-03-28T15:37:24.292 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_promise.ko 2024-03-28T15:37:24.292 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_qstor.ko 2024-03-28T15:37:24.292 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sil.ko 2024-03-28T15:37:24.292 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sil24.ko 2024-03-28T15:37:24.292 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sis.ko 2024-03-28T15:37:24.292 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_svw.ko 2024-03-28T15:37:24.292 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sx4.ko 2024-03-28T15:37:24.292 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_uli.ko 2024-03-28T15:37:24.292 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_via.ko 2024-03-28T15:37:24.292 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_vsc.ko 2024-03-28T15:37:24.292 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/auxdisplay 2024-03-28T15:37:24.292 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/auxdisplay/ks0108.ko 2024-03-28T15:37:24.292 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base 2024-03-28T15:37:24.292 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base/regmap 2024-03-28T15:37:24.293 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base/regmap/regmap-i2c.ko 2024-03-28T15:37:24.293 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base/regmap/regmap-mmio.ko 2024-03-28T15:37:24.293 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/bcma 2024-03-28T15:37:24.293 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/bcma/bcma.ko 2024-03-28T15:37:24.293 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block 2024-03-28T15:37:24.293 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/aoe 2024-03-28T15:37:24.293 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/aoe/aoe.ko 2024-03-28T15:37:24.293 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/brd.ko 2024-03-28T15:37:24.293 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/drbd 2024-03-28T15:37:24.293 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/drbd/drbd.ko 2024-03-28T15:37:24.293 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/floppy.ko 2024-03-28T15:37:24.293 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/loop.ko 2024-03-28T15:37:24.293 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/mtip32xx 2024-03-28T15:37:24.293 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-03-28T15:37:24.293 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/nbd.ko 2024-03-28T15:37:24.294 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/null_blk 2024-03-28T15:37:24.294 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/null_blk/null_blk.ko 2024-03-28T15:37:24.294 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/pktcdvd.ko 2024-03-28T15:37:24.294 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/rbd.ko 2024-03-28T15:37:24.294 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/virtio_blk.ko 2024-03-28T15:37:24.294 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/xen-blkback 2024-03-28T15:37:24.294 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-03-28T15:37:24.294 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/xen-blkfront.ko 2024-03-28T15:37:24.294 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/zram 2024-03-28T15:37:24.294 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/zram/zram.ko 2024-03-28T15:37:24.294 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char 2024-03-28T15:37:24.294 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hangcheck-timer.ko 2024-03-28T15:37:24.294 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random 2024-03-28T15:37:24.294 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/amd-rng.ko 2024-03-28T15:37:24.295 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/intel-rng.ko 2024-03-28T15:37:24.295 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-03-28T15:37:24.295 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/via-rng.ko 2024-03-28T15:37:24.295 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/virtio-rng.ko 2024-03-28T15:37:24.295 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi 2024-03-28T15:37:24.295 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-03-28T15:37:24.295 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-03-28T15:37:24.295 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-03-28T15:37:24.295 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_si.ko 2024-03-28T15:37:24.295 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-03-28T15:37:24.295 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-03-28T15:37:24.295 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/lp.ko 2024-03-28T15:37:24.295 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/mwave 2024-03-28T15:37:24.295 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/mwave/mwave.ko 2024-03-28T15:37:24.295 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ppdev.ko 2024-03-28T15:37:24.296 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tlclk.ko 2024-03-28T15:37:24.296 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm 2024-03-28T15:37:24.296 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm.ko 2024-03-28T15:37:24.296 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_atmel.ko 2024-03-28T15:37:24.296 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_infineon.ko 2024-03-28T15:37:24.296 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_nsc.ko 2024-03-28T15:37:24.296 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_tis.ko 2024-03-28T15:37:24.296 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-03-28T15:37:24.296 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/uv_mmtimer.ko 2024-03-28T15:37:24.296 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/virtio_console.ko 2024-03-28T15:37:24.296 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq 2024-03-28T15:37:24.296 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-03-28T15:37:24.296 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-03-28T15:37:24.296 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/p4-clockmod.ko 2024-03-28T15:37:24.297 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-03-28T15:37:24.297 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/powernow-k8.ko 2024-03-28T15:37:24.297 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/speedstep-lib.ko 2024-03-28T15:37:24.297 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto 2024-03-28T15:37:24.297 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp 2024-03-28T15:37:24.297 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-03-28T15:37:24.297 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp/ccp.ko 2024-03-28T15:37:24.297 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel 2024-03-28T15:37:24.297 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat 2024-03-28T15:37:24.297 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_common 2024-03-28T15:37:24.297 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-03-28T15:37:24.297 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-03-28T15:37:24.297 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-03-28T15:37:24.297 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/padlock-aes.ko 2024-03-28T15:37:24.297 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/padlock-sha.ko 2024-03-28T15:37:24.298 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/virtio 2024-03-28T15:37:24.298 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-03-28T15:37:24.298 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dca 2024-03-28T15:37:24.298 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dca/dca.ko 2024-03-28T15:37:24.298 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/devfreq 2024-03-28T15:37:24.298 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-03-28T15:37:24.298 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma 2024-03-28T15:37:24.298 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/dw 2024-03-28T15:37:24.298 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/dw/dw_dmac.ko 2024-03-28T15:37:24.298 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/ioat 2024-03-28T15:37:24.298 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/ioat/ioatdma.ko 2024-03-28T15:37:24.298 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac 2024-03-28T15:37:24.298 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/amd64_edac.ko 2024-03-28T15:37:24.298 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/e752x_edac.ko 2024-03-28T15:37:24.298 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/edac_mce_amd.ko 2024-03-28T15:37:24.299 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i3000_edac.ko 2024-03-28T15:37:24.299 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i3200_edac.ko 2024-03-28T15:37:24.299 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i5100_edac.ko 2024-03-28T15:37:24.299 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i5400_edac.ko 2024-03-28T15:37:24.299 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i7300_edac.ko 2024-03-28T15:37:24.299 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i7core_edac.ko 2024-03-28T15:37:24.299 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i82975x_edac.ko 2024-03-28T15:37:24.299 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/ie31200_edac.ko 2024-03-28T15:37:24.299 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/sb_edac.ko 2024-03-28T15:37:24.299 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/x38_edac.ko 2024-03-28T15:37:24.299 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware 2024-03-28T15:37:24.299 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware/edd.ko 2024-03-28T15:37:24.299 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware/iscsi_ibft.ko 2024-03-28T15:37:24.299 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio 2024-03-28T15:37:24.299 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-ich.ko 2024-03-28T15:37:24.300 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-viperboard.ko 2024-03-28T15:37:24.300 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid 2024-03-28T15:37:24.300 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-appleir.ko 2024-03-28T15:37:24.300 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-aureal.ko 2024-03-28T15:37:24.300 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-axff.ko 2024-03-28T15:37:24.300 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-dr.ko 2024-03-28T15:37:24.300 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-elecom.ko 2024-03-28T15:37:24.300 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-elo.ko 2024-03-28T15:37:24.300 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-emsff.ko 2024-03-28T15:37:24.300 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gaff.ko 2024-03-28T15:37:24.300 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gt683r.ko 2024-03-28T15:37:24.300 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gyration.ko 2024-03-28T15:37:24.300 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtek-kbd.ko 2024-03-28T15:37:24.300 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtek-mouse.ko 2024-03-28T15:37:24.301 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtekff.ko 2024-03-28T15:37:24.301 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-hyperv.ko 2024-03-28T15:37:24.301 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-icade.ko 2024-03-28T15:37:24.301 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-keytouch.ko 2024-03-28T15:37:24.301 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-kye.ko 2024-03-28T15:37:24.301 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-lcpower.ko 2024-03-28T15:37:24.301 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-led.ko 2024-03-28T15:37:24.301 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-lenovo.ko 2024-03-28T15:37:24.301 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-logitech-dj.ko 2024-03-28T15:37:24.301 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-03-28T15:37:24.301 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-multitouch.ko 2024-03-28T15:37:24.301 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-ortek.ko 2024-03-28T15:37:24.301 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-penmount.ko 2024-03-28T15:37:24.301 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-petalynx.ko 2024-03-28T15:37:24.301 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-picolcd.ko 2024-03-28T15:37:24.302 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-pl.ko 2024-03-28T15:37:24.302 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-plantronics.ko 2024-03-28T15:37:24.302 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-primax.ko 2024-03-28T15:37:24.302 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-rmi.ko 2024-03-28T15:37:24.302 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-arvo.ko 2024-03-28T15:37:24.302 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-common.ko 2024-03-28T15:37:24.302 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-isku.ko 2024-03-28T15:37:24.302 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-kone.ko 2024-03-28T15:37:24.302 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-03-28T15:37:24.302 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-konepure.ko 2024-03-28T15:37:24.302 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-03-28T15:37:24.302 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-lua.ko 2024-03-28T15:37:24.302 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-pyra.ko 2024-03-28T15:37:24.302 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-ryos.ko 2024-03-28T15:37:24.303 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-savu.ko 2024-03-28T15:37:24.303 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat.ko 2024-03-28T15:37:24.303 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-saitek.ko 2024-03-28T15:37:24.303 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-samsung.ko 2024-03-28T15:37:24.303 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sensor-hub.ko 2024-03-28T15:37:24.303 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sjoy.ko 2024-03-28T15:37:24.303 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sony.ko 2024-03-28T15:37:24.303 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-speedlink.ko 2024-03-28T15:37:24.303 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-steelseries.ko 2024-03-28T15:37:24.303 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sunplus.ko 2024-03-28T15:37:24.303 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-thrustmaster.ko 2024-03-28T15:37:24.303 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-tivo.ko 2024-03-28T15:37:24.303 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-tmff.ko 2024-03-28T15:37:24.303 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-topseed.ko 2024-03-28T15:37:24.303 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-twinhan.ko 2024-03-28T15:37:24.304 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-uclogic.ko 2024-03-28T15:37:24.304 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-waltop.ko 2024-03-28T15:37:24.304 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-wiimote.ko 2024-03-28T15:37:24.304 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-xinmo.ko 2024-03-28T15:37:24.304 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-zpff.ko 2024-03-28T15:37:24.304 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-zydacron.ko 2024-03-28T15:37:24.304 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/uhid.ko 2024-03-28T15:37:24.304 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/wacom.ko 2024-03-28T15:37:24.304 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv 2024-03-28T15:37:24.304 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_balloon.ko 2024-03-28T15:37:24.304 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_utils.ko 2024-03-28T15:37:24.304 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_vmbus.ko 2024-03-28T15:37:24.304 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon 2024-03-28T15:37:24.304 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/abituguru.ko 2024-03-28T15:37:24.304 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/abituguru3.ko 2024-03-28T15:37:24.304 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/acpi_power_meter.ko 2024-03-28T15:37:24.305 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ad7414.ko 2024-03-28T15:37:24.305 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ad7418.ko 2024-03-28T15:37:24.305 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adc128d818.ko 2024-03-28T15:37:24.305 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1025.ko 2024-03-28T15:37:24.305 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1026.ko 2024-03-28T15:37:24.305 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1029.ko 2024-03-28T15:37:24.305 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1031.ko 2024-03-28T15:37:24.305 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm9240.ko 2024-03-28T15:37:24.305 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ads7828.ko 2024-03-28T15:37:24.305 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7410.ko 2024-03-28T15:37:24.305 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7411.ko 2024-03-28T15:37:24.305 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7462.ko 2024-03-28T15:37:24.305 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7470.ko 2024-03-28T15:37:24.305 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7475.ko 2024-03-28T15:37:24.306 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7x10.ko 2024-03-28T15:37:24.306 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/amc6821.ko 2024-03-28T15:37:24.306 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/applesmc.ko 2024-03-28T15:37:24.306 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asb100.ko 2024-03-28T15:37:24.306 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asc7621.ko 2024-03-28T15:37:24.306 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asus_atk0110.ko 2024-03-28T15:37:24.306 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/atxp1.ko 2024-03-28T15:37:24.306 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/coretemp.ko 2024-03-28T15:37:24.306 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-03-28T15:37:24.306 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/dme1737.ko 2024-03-28T15:37:24.306 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ds1621.ko 2024-03-28T15:37:24.306 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ds620.ko 2024-03-28T15:37:24.306 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/emc1403.ko 2024-03-28T15:37:24.307 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/emc6w201.ko 2024-03-28T15:37:24.307 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f71805f.ko 2024-03-28T15:37:24.307 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f71882fg.ko 2024-03-28T15:37:24.307 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f75375s.ko 2024-03-28T15:37:24.307 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/fam15h_power.ko 2024-03-28T15:37:24.307 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/fschmd.ko 2024-03-28T15:37:24.307 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/g760a.ko 2024-03-28T15:37:24.307 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/g762.ko 2024-03-28T15:37:24.307 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/gl518sm.ko 2024-03-28T15:37:24.307 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/gl520sm.ko 2024-03-28T15:37:24.307 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/hwmon-vid.ko 2024-03-28T15:37:24.307 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/i5500_temp.ko 2024-03-28T15:37:24.307 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/i5k_amb.ko 2024-03-28T15:37:24.307 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ibmaem.ko 2024-03-28T15:37:24.307 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ibmpex.ko 2024-03-28T15:37:24.308 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ina209.ko 2024-03-28T15:37:24.308 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ina2xx.ko 2024-03-28T15:37:24.308 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/it87.ko 2024-03-28T15:37:24.308 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/k10temp.ko 2024-03-28T15:37:24.308 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/k8temp.ko 2024-03-28T15:37:24.308 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lineage-pem.ko 2024-03-28T15:37:24.308 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm63.ko 2024-03-28T15:37:24.308 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm73.ko 2024-03-28T15:37:24.308 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm75.ko 2024-03-28T15:37:24.308 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm77.ko 2024-03-28T15:37:24.308 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm78.ko 2024-03-28T15:37:24.308 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm80.ko 2024-03-28T15:37:24.308 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm83.ko 2024-03-28T15:37:24.308 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm85.ko 2024-03-28T15:37:24.308 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm87.ko 2024-03-28T15:37:24.309 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm90.ko 2024-03-28T15:37:24.309 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm92.ko 2024-03-28T15:37:24.309 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm93.ko 2024-03-28T15:37:24.309 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95234.ko 2024-03-28T15:37:24.309 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95241.ko 2024-03-28T15:37:24.309 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95245.ko 2024-03-28T15:37:24.309 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc2945.ko 2024-03-28T15:37:24.309 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4151.ko 2024-03-28T15:37:24.309 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4215.ko 2024-03-28T15:37:24.309 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4222.ko 2024-03-28T15:37:24.309 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4245.ko 2024-03-28T15:37:24.309 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4260.ko 2024-03-28T15:37:24.309 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4261.ko 2024-03-28T15:37:24.309 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max16065.ko 2024-03-28T15:37:24.309 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max1619.ko 2024-03-28T15:37:24.310 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max1668.ko 2024-03-28T15:37:24.310 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max197.ko 2024-03-28T15:37:24.310 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6639.ko 2024-03-28T15:37:24.310 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6650.ko 2024-03-28T15:37:24.310 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6697.ko 2024-03-28T15:37:24.310 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/mcp3021.ko 2024-03-28T15:37:24.310 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6683.ko 2024-03-28T15:37:24.310 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6775-core.ko 2024-03-28T15:37:24.310 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6775.ko 2024-03-28T15:37:24.310 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct7802.ko 2024-03-28T15:37:24.310 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pc87360.ko 2024-03-28T15:37:24.310 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pc87427.ko 2024-03-28T15:37:24.310 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pcf8591.ko 2024-03-28T15:37:24.310 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus 2024-03-28T15:37:24.311 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-03-28T15:37:24.311 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-03-28T15:37:24.311 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-03-28T15:37:24.311 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max16064.ko 2024-03-28T15:37:24.311 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max34440.ko 2024-03-28T15:37:24.311 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max8688.ko 2024-03-28T15:37:24.311 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-03-28T15:37:24.311 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-03-28T15:37:24.311 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-03-28T15:37:24.311 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-03-28T15:37:24.311 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-03-28T15:37:24.311 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-03-28T15:37:24.311 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/powr1220.ko 2024-03-28T15:37:24.311 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch5627.ko 2024-03-28T15:37:24.311 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch5636.ko 2024-03-28T15:37:24.312 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch56xx-common.ko 2024-03-28T15:37:24.312 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sht15.ko 2024-03-28T15:37:24.312 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sht21.ko 2024-03-28T15:37:24.312 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/shtc1.ko 2024-03-28T15:37:24.312 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sis5595.ko 2024-03-28T15:37:24.312 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47b397.ko 2024-03-28T15:37:24.312 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47m1.ko 2024-03-28T15:37:24.312 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47m192.ko 2024-03-28T15:37:24.312 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/thmc50.ko 2024-03-28T15:37:24.312 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp102.ko 2024-03-28T15:37:24.312 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp103.ko 2024-03-28T15:37:24.312 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp401.ko 2024-03-28T15:37:24.312 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp421.ko 2024-03-28T15:37:24.312 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/via-cputemp.ko 2024-03-28T15:37:24.313 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/via686a.ko 2024-03-28T15:37:24.313 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/vt1211.ko 2024-03-28T15:37:24.313 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/vt8231.ko 2024-03-28T15:37:24.313 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83627ehf.ko 2024-03-28T15:37:24.313 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83627hf.ko 2024-03-28T15:37:24.313 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83781d.ko 2024-03-28T15:37:24.313 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83791d.ko 2024-03-28T15:37:24.313 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83792d.ko 2024-03-28T15:37:24.313 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83793.ko 2024-03-28T15:37:24.313 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83795.ko 2024-03-28T15:37:24.313 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83l785ts.ko 2024-03-28T15:37:24.313 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83l786ng.ko 2024-03-28T15:37:24.313 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c 2024-03-28T15:37:24.313 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos 2024-03-28T15:37:24.313 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-03-28T15:37:24.314 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-03-28T15:37:24.314 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses 2024-03-28T15:37:24.314 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-03-28T15:37:24.314 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-03-28T15:37:24.314 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-03-28T15:37:24.314 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-03-28T15:37:24.314 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-03-28T15:37:24.314 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-03-28T15:37:24.314 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-03-28T15:37:24.314 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-03-28T15:37:24.314 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-i801.ko 2024-03-28T15:37:24.314 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-isch.ko 2024-03-28T15:37:24.314 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-03-28T15:37:24.314 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-03-28T15:37:24.315 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-03-28T15:37:24.315 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-parport.ko 2024-03-28T15:37:24.315 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-03-28T15:37:24.315 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-03-28T15:37:24.315 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-03-28T15:37:24.315 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-03-28T15:37:24.315 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-03-28T15:37:24.315 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-03-28T15:37:24.315 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-via.ko 2024-03-28T15:37:24.315 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-03-28T15:37:24.315 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-03-28T15:37:24.315 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-dev.ko 2024-03-28T15:37:24.315 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-mux.ko 2024-03-28T15:37:24.315 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-03-28T15:37:24.315 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-smbus.ko 2024-03-28T15:37:24.315 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-stub.ko 2024-03-28T15:37:24.316 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband 2024-03-28T15:37:24.316 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core 2024-03-28T15:37:24.316 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_cm.ko 2024-03-28T15:37:24.316 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_core.ko 2024-03-28T15:37:24.316 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_umad.ko 2024-03-28T15:37:24.316 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-03-28T15:37:24.316 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/iw_cm.ko 2024-03-28T15:37:24.316 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/rdma_cm.ko 2024-03-28T15:37:24.316 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-03-28T15:37:24.316 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw 2024-03-28T15:37:24.316 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/cxgb4 2024-03-28T15:37:24.316 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-03-28T15:37:24.316 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx4 2024-03-28T15:37:24.316 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-03-28T15:37:24.317 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx5 2024-03-28T15:37:24.317 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-03-28T15:37:24.317 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mthca 2024-03-28T15:37:24.317 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-03-28T15:37:24.317 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp 2024-03-28T15:37:24.317 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/ipoib 2024-03-28T15:37:24.317 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-03-28T15:37:24.317 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/iser 2024-03-28T15:37:24.317 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-03-28T15:37:24.317 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/isert 2024-03-28T15:37:24.317 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-03-28T15:37:24.317 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srp 2024-03-28T15:37:24.317 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-03-28T15:37:24.317 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srpt 2024-03-28T15:37:24.317 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-03-28T15:37:24.318 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input 2024-03-28T15:37:24.318 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport 2024-03-28T15:37:24.318 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-03-28T15:37:24.318 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/fm801-gp.ko 2024-03-28T15:37:24.318 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/gameport.ko 2024-03-28T15:37:24.318 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/joydev.ko 2024-03-28T15:37:24.318 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard 2024-03-28T15:37:24.318 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/gpio_keys.ko 2024-03-28T15:37:24.318 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse 2024-03-28T15:37:24.318 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/appletouch.ko 2024-03-28T15:37:24.318 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/bcm5974.ko 2024-03-28T15:37:24.318 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/cyapatp.ko 2024-03-28T15:37:24.318 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/elan_i2c.ko 2024-03-28T15:37:24.318 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/sermouse.ko 2024-03-28T15:37:24.319 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-03-28T15:37:24.319 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/synaptics_usb.ko 2024-03-28T15:37:24.319 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/vsxxxaa.ko 2024-03-28T15:37:24.319 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/rmi4 2024-03-28T15:37:24.319 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/rmi4/rmi_core.ko 2024-03-28T15:37:24.319 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio 2024-03-28T15:37:24.319 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/altera_ps2.ko 2024-03-28T15:37:24.319 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/arc_ps2.ko 2024-03-28T15:37:24.319 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-03-28T15:37:24.319 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/serio_raw.ko 2024-03-28T15:37:24.319 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/sparse-keymap.ko 2024-03-28T15:37:24.319 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds 2024-03-28T15:37:24.319 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-blinkm.ko 2024-03-28T15:37:24.319 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-lm3530.ko 2024-03-28T15:37:24.319 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-lp3944.ko 2024-03-28T15:37:24.319 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-ss4200.ko 2024-03-28T15:37:24.320 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger 2024-03-28T15:37:24.320 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-03-28T15:37:24.320 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-03-28T15:37:24.320 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-03-28T15:37:24.320 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-03-28T15:37:24.320 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-03-28T15:37:24.320 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-03-28T15:37:24.320 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-03-28T15:37:24.320 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-03-28T15:37:24.320 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-03-28T15:37:24.320 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md 2024-03-28T15:37:24.320 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/bcache 2024-03-28T15:37:24.320 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/bcache/bcache.ko 2024-03-28T15:37:24.320 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-bio-prison.ko 2024-03-28T15:37:24.320 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-cache-smq.ko 2024-03-28T15:37:24.321 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-cache.ko 2024-03-28T15:37:24.321 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-crypt.ko 2024-03-28T15:37:24.321 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-delay.ko 2024-03-28T15:37:24.321 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-flakey.ko 2024-03-28T15:37:24.321 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-log-userspace.ko 2024-03-28T15:37:24.321 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-multipath.ko 2024-03-28T15:37:24.321 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-queue-length.ko 2024-03-28T15:37:24.321 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-raid.ko 2024-03-28T15:37:24.321 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-round-robin.ko 2024-03-28T15:37:24.321 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-service-time.ko 2024-03-28T15:37:24.321 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-switch.ko 2024-03-28T15:37:24.321 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-thin-pool.ko 2024-03-28T15:37:24.321 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-verity.ko 2024-03-28T15:37:24.321 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/persistent-data 2024-03-28T15:37:24.321 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-03-28T15:37:24.322 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid0.ko 2024-03-28T15:37:24.322 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid1.ko 2024-03-28T15:37:24.322 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid10.ko 2024-03-28T15:37:24.322 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid456.ko 2024-03-28T15:37:24.322 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message 2024-03-28T15:37:24.322 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion 2024-03-28T15:37:24.322 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptbase.ko 2024-03-28T15:37:24.322 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptctl.ko 2024-03-28T15:37:24.322 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptfc.ko 2024-03-28T15:37:24.322 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptlan.ko 2024-03-28T15:37:24.322 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptsas.ko 2024-03-28T15:37:24.322 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptscsih.ko 2024-03-28T15:37:24.322 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptspi.ko 2024-03-28T15:37:24.322 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd 2024-03-28T15:37:24.323 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/lpc_ich.ko 2024-03-28T15:37:24.323 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/lpc_sch.ko 2024-03-28T15:37:24.323 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/mfd-core.ko 2024-03-28T15:37:24.323 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/sm501.ko 2024-03-28T15:37:24.323 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/viperboard.ko 2024-03-28T15:37:24.323 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/vx855.ko 2024-03-28T15:37:24.323 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/wl1273-core.ko 2024-03-28T15:37:24.323 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc 2024-03-28T15:37:24.323 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/altera-stapl 2024-03-28T15:37:24.323 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-03-28T15:37:24.323 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/apds9802als.ko 2024-03-28T15:37:24.323 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/apds990x.ko 2024-03-28T15:37:24.323 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/bh1770glc.ko 2024-03-28T15:37:24.323 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/cb710 2024-03-28T15:37:24.323 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/cb710/cb710.ko 2024-03-28T15:37:24.323 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom 2024-03-28T15:37:24.324 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/at24.ko 2024-03-28T15:37:24.324 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-03-28T15:37:24.324 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/max6875.ko 2024-03-28T15:37:24.324 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/enclosure.ko 2024-03-28T15:37:24.324 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/hpilo.ko 2024-03-28T15:37:24.324 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/isl29003.ko 2024-03-28T15:37:24.324 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/isl29020.ko 2024-03-28T15:37:24.324 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d 2024-03-28T15:37:24.324 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-03-28T15:37:24.324 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-03-28T15:37:24.324 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei 2024-03-28T15:37:24.324 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei-me.ko 2024-03-28T15:37:24.324 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei-txe.ko 2024-03-28T15:37:24.324 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei.ko 2024-03-28T15:37:24.325 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-gru 2024-03-28T15:37:24.325 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-gru/gru.ko 2024-03-28T15:37:24.325 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp 2024-03-28T15:37:24.325 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp/xp.ko 2024-03-28T15:37:24.325 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp/xpc.ko 2024-03-28T15:37:24.325 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-03-28T15:37:24.325 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tifm_7xx1.ko 2024-03-28T15:37:24.325 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tifm_core.ko 2024-03-28T15:37:24.325 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tsl2550.ko 2024-03-28T15:37:24.325 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_balloon.ko 2024-03-28T15:37:24.325 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_vmci 2024-03-28T15:37:24.325 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-03-28T15:37:24.325 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net 2024-03-28T15:37:24.325 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/bonding 2024-03-28T15:37:24.326 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/bonding/bonding.ko 2024-03-28T15:37:24.326 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa 2024-03-28T15:37:24.326 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/b53 2024-03-28T15:37:24.326 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/b53/b53_common.ko 2024-03-28T15:37:24.326 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/bcm-sf2.ko 2024-03-28T15:37:24.326 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6060.ko 2024-03-28T15:37:24.326 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6xxx 2024-03-28T15:37:24.326 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-03-28T15:37:24.326 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dummy.ko 2024-03-28T15:37:24.326 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/eql.ko 2024-03-28T15:37:24.326 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet 2024-03-28T15:37:24.326 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com 2024-03-28T15:37:24.326 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-03-28T15:37:24.326 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-03-28T15:37:24.326 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-03-28T15:37:24.327 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-03-28T15:37:24.327 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390 2024-03-28T15:37:24.327 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/8390.ko 2024-03-28T15:37:24.327 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-03-28T15:37:24.327 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-03-28T15:37:24.327 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-03-28T15:37:24.327 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/adaptec 2024-03-28T15:37:24.327 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-03-28T15:37:24.327 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/agere 2024-03-28T15:37:24.327 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/agere/et131x.ko 2024-03-28T15:37:24.327 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/alteon 2024-03-28T15:37:24.327 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-03-28T15:37:24.327 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/altera 2024-03-28T15:37:24.327 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-03-28T15:37:24.327 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd 2024-03-28T15:37:24.328 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-03-28T15:37:24.328 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-03-28T15:37:24.328 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-03-28T15:37:24.328 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros 2024-03-28T15:37:24.328 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/alx 2024-03-28T15:37:24.328 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-03-28T15:37:24.328 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1c 2024-03-28T15:37:24.328 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-03-28T15:37:24.328 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1e 2024-03-28T15:37:24.328 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-03-28T15:37:24.328 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx 2024-03-28T15:37:24.328 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-03-28T15:37:24.328 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-03-28T15:37:24.328 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom 2024-03-28T15:37:24.329 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-03-28T15:37:24.329 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-03-28T15:37:24.329 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-03-28T15:37:24.329 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-03-28T15:37:24.329 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-03-28T15:37:24.329 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/genet 2024-03-28T15:37:24.329 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-03-28T15:37:24.329 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-03-28T15:37:24.329 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade 2024-03-28T15:37:24.329 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade/bna 2024-03-28T15:37:24.329 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-03-28T15:37:24.329 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio 2024-03-28T15:37:24.329 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb 2024-03-28T15:37:24.329 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-03-28T15:37:24.329 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-03-28T15:37:24.329 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-03-28T15:37:24.330 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-03-28T15:37:24.330 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-03-28T15:37:24.330 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-03-28T15:37:24.330 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-03-28T15:37:24.330 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-03-28T15:37:24.330 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-03-28T15:37:24.330 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco 2024-03-28T15:37:24.330 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco/enic 2024-03-28T15:37:24.330 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-03-28T15:37:24.330 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec 2024-03-28T15:37:24.330 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip 2024-03-28T15:37:24.330 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-03-28T15:37:24.330 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-03-28T15:37:24.330 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-03-28T15:37:24.331 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-03-28T15:37:24.331 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-03-28T15:37:24.331 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-03-28T15:37:24.331 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink 2024-03-28T15:37:24.331 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-03-28T15:37:24.331 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-03-28T15:37:24.331 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dnet.ko 2024-03-28T15:37:24.331 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex 2024-03-28T15:37:24.331 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex/benet 2024-03-28T15:37:24.331 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-03-28T15:37:24.331 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ethoc.ko 2024-03-28T15:37:24.331 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/fealnx.ko 2024-03-28T15:37:24.331 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel 2024-03-28T15:37:24.331 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e100.ko 2024-03-28T15:37:24.331 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000 2024-03-28T15:37:24.332 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-03-28T15:37:24.332 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000e 2024-03-28T15:37:24.332 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-03-28T15:37:24.332 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/fm10k 2024-03-28T15:37:24.332 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-03-28T15:37:24.332 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/i40e 2024-03-28T15:37:24.332 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-03-28T15:37:24.332 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/iavf 2024-03-28T15:37:24.332 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-03-28T15:37:24.332 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igb 2024-03-28T15:37:24.332 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-03-28T15:37:24.332 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igbvf 2024-03-28T15:37:24.332 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-03-28T15:37:24.332 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbe 2024-03-28T15:37:24.332 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-03-28T15:37:24.333 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbevf 2024-03-28T15:37:24.333 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-03-28T15:37:24.333 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/jme.ko 2024-03-28T15:37:24.333 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell 2024-03-28T15:37:24.333 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-03-28T15:37:24.333 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/skge.ko 2024-03-28T15:37:24.333 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-03-28T15:37:24.333 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox 2024-03-28T15:37:24.333 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4 2024-03-28T15:37:24.333 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-03-28T15:37:24.333 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-03-28T15:37:24.333 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5 2024-03-28T15:37:24.333 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-03-28T15:37:24.333 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-03-28T15:37:24.333 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/micrel 2024-03-28T15:37:24.334 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-03-28T15:37:24.334 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom 2024-03-28T15:37:24.334 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom/myri10ge 2024-03-28T15:37:24.334 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-03-28T15:37:24.334 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi 2024-03-28T15:37:24.334 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-03-28T15:37:24.334 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-03-28T15:37:24.334 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/neterion 2024-03-28T15:37:24.334 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-03-28T15:37:24.334 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/nvidia 2024-03-28T15:37:24.334 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-03-28T15:37:24.334 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines 2024-03-28T15:37:24.334 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-03-28T15:37:24.334 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-03-28T15:37:24.335 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic 2024-03-28T15:37:24.335 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/netxen 2024-03-28T15:37:24.335 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-03-28T15:37:24.335 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-03-28T15:37:24.335 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-03-28T15:37:24.335 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-03-28T15:37:24.335 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rdc 2024-03-28T15:37:24.335 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-03-28T15:37:24.335 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek 2024-03-28T15:37:24.335 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-03-28T15:37:24.335 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-03-28T15:37:24.335 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/atp.ko 2024-03-28T15:37:24.335 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-03-28T15:37:24.335 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rocker 2024-03-28T15:37:24.336 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-03-28T15:37:24.336 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sfc 2024-03-28T15:37:24.336 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-03-28T15:37:24.336 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/silan 2024-03-28T15:37:24.336 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-03-28T15:37:24.336 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis 2024-03-28T15:37:24.336 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis/sis190.ko 2024-03-28T15:37:24.336 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis/sis900.ko 2024-03-28T15:37:24.336 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc 2024-03-28T15:37:24.336 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-03-28T15:37:24.336 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-03-28T15:37:24.336 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-03-28T15:37:24.336 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-03-28T15:37:24.336 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro 2024-03-28T15:37:24.336 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac 2024-03-28T15:37:24.336 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-03-28T15:37:24.337 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-03-28T15:37:24.337 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun 2024-03-28T15:37:24.337 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/cassini.ko 2024-03-28T15:37:24.337 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/niu.ko 2024-03-28T15:37:24.337 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/sungem.ko 2024-03-28T15:37:24.337 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-03-28T15:37:24.337 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/tehuti 2024-03-28T15:37:24.337 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-03-28T15:37:24.337 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ti 2024-03-28T15:37:24.337 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ti/tlan.ko 2024-03-28T15:37:24.337 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via 2024-03-28T15:37:24.337 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-03-28T15:37:24.337 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-03-28T15:37:24.337 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet 2024-03-28T15:37:24.338 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-03-28T15:37:24.338 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-03-28T15:37:24.338 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/xircom 2024-03-28T15:37:24.338 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-03-28T15:37:24.338 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/geneve.ko 2024-03-28T15:37:24.338 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/hyperv 2024-03-28T15:37:24.338 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-03-28T15:37:24.338 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ieee802154 2024-03-28T15:37:24.338 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ieee802154/fakelb.ko 2024-03-28T15:37:24.338 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ifb.ko 2024-03-28T15:37:24.338 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ipvlan 2024-03-28T15:37:24.338 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ipvlan/ipvlan.ko 2024-03-28T15:37:24.338 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/macvlan.ko 2024-03-28T15:37:24.338 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/macvtap.ko 2024-03-28T15:37:24.338 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio 2024-03-28T15:37:24.339 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio.ko 2024-03-28T15:37:24.339 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-03-28T15:37:24.339 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-03-28T15:37:24.339 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-regmap.ko 2024-03-28T15:37:24.339 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mii.ko 2024-03-28T15:37:24.339 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/net_failover.ko 2024-03-28T15:37:24.339 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/netconsole.ko 2024-03-28T15:37:24.339 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/nlmon.ko 2024-03-28T15:37:24.339 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs 2024-03-28T15:37:24.339 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs/pcs-lynx.ko 2024-03-28T15:37:24.339 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-03-28T15:37:24.339 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy 2024-03-28T15:37:24.339 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/amd.ko 2024-03-28T15:37:24.340 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-03-28T15:37:24.340 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-03-28T15:37:24.340 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm7xxx.ko 2024-03-28T15:37:24.340 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm87xx.ko 2024-03-28T15:37:24.340 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/broadcom.ko 2024-03-28T15:37:24.340 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/cicada.ko 2024-03-28T15:37:24.340 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/davicom.ko 2024-03-28T15:37:24.340 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/dp83640.ko 2024-03-28T15:37:24.340 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/et1011c.ko 2024-03-28T15:37:24.340 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/icplus.ko 2024-03-28T15:37:24.340 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/lxt.ko 2024-03-28T15:37:24.340 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/marvell.ko 2024-03-28T15:37:24.340 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/micrel.ko 2024-03-28T15:37:24.340 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/national.ko 2024-03-28T15:37:24.340 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/phylink.ko 2024-03-28T15:37:24.341 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/qsemi.ko 2024-03-28T15:37:24.341 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/realtek.ko 2024-03-28T15:37:24.341 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/smsc.ko 2024-03-28T15:37:24.341 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/ste10Xp.ko 2024-03-28T15:37:24.341 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/vitesse.ko 2024-03-28T15:37:24.341 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp 2024-03-28T15:37:24.341 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/bsd_comp.ko 2024-03-28T15:37:24.341 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_async.ko 2024-03-28T15:37:24.341 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_deflate.ko 2024-03-28T15:37:24.341 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_generic.ko 2024-03-28T15:37:24.341 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_mppe.ko 2024-03-28T15:37:24.341 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_synctty.ko 2024-03-28T15:37:24.341 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pppoe.ko 2024-03-28T15:37:24.341 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pppox.ko 2024-03-28T15:37:24.341 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pptp.ko 2024-03-28T15:37:24.342 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/slip 2024-03-28T15:37:24.342 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/slip/slhc.ko 2024-03-28T15:37:24.342 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/slip/slip.ko 2024-03-28T15:37:24.342 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/sungem_phy.ko 2024-03-28T15:37:24.342 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/tap.ko 2024-03-28T15:37:24.342 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team 2024-03-28T15:37:24.342 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team.ko 2024-03-28T15:37:24.342 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_activebackup.ko 2024-03-28T15:37:24.342 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_broadcast.ko 2024-03-28T15:37:24.342 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-03-28T15:37:24.342 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_random.ko 2024-03-28T15:37:24.342 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-03-28T15:37:24.342 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/tun.ko 2024-03-28T15:37:24.342 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/veth.ko 2024-03-28T15:37:24.343 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/virtio_net.ko 2024-03-28T15:37:24.343 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vmxnet3 2024-03-28T15:37:24.343 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-03-28T15:37:24.343 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vxlan 2024-03-28T15:37:24.343 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vxlan/vxlan.ko 2024-03-28T15:37:24.343 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/xen-netback 2024-03-28T15:37:24.343 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/xen-netback/xen-netback.ko 2024-03-28T15:37:24.343 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/xen-netfront.ko 2024-03-28T15:37:24.343 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ntb 2024-03-28T15:37:24.343 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ntb/ntb.ko 2024-03-28T15:37:24.343 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme 2024-03-28T15:37:24.343 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host 2024-03-28T15:37:24.343 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host/nvme-core.ko 2024-03-28T15:37:24.343 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host/nvme.ko 2024-03-28T15:37:24.343 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport 2024-03-28T15:37:24.344 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport.ko 2024-03-28T15:37:24.344 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_cs.ko 2024-03-28T15:37:24.344 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_pc.ko 2024-03-28T15:37:24.344 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_serial.ko 2024-03-28T15:37:24.344 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci 2024-03-28T15:37:24.344 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/hotplug 2024-03-28T15:37:24.344 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-03-28T15:37:24.344 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/pcie 2024-03-28T15:37:24.344 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/pcie/aer_inject.ko 2024-03-28T15:37:24.344 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/xen-pcifront.ko 2024-03-28T15:37:24.344 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia 2024-03-28T15:37:24.344 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/i82092.ko 2024-03-28T15:37:24.344 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/pd6729.ko 2024-03-28T15:37:24.344 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/yenta_socket.ko 2024-03-28T15:37:24.345 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl 2024-03-28T15:37:24.345 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl/intel 2024-03-28T15:37:24.345 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-03-28T15:37:24.345 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform 2024-03-28T15:37:24.345 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86 2024-03-28T15:37:24.345 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/acer-wmi.ko 2024-03-28T15:37:24.345 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/acerhdf.ko 2024-03-28T15:37:24.345 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/apple-gmux.ko 2024-03-28T15:37:24.345 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-laptop.ko 2024-03-28T15:37:24.345 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-03-28T15:37:24.345 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-wmi.ko 2024-03-28T15:37:24.345 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/classmate-laptop.ko 2024-03-28T15:37:24.345 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-03-28T15:37:24.345 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-03-28T15:37:24.345 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-03-28T15:37:24.345 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-03-28T15:37:24.346 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/hdaps.ko 2024-03-28T15:37:24.346 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel 2024-03-28T15:37:24.346 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-03-28T15:37:24.346 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel_ips.ko 2024-03-28T15:37:24.346 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/msi-wmi.ko 2024-03-28T15:37:24.346 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/mxm-wmi.ko 2024-03-28T15:37:24.346 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-03-28T15:37:24.346 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/samsung-laptop.ko 2024-03-28T15:37:24.346 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/samsung-q10.ko 2024-03-28T15:37:24.346 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/topstar-laptop.ko 2024-03-28T15:37:24.346 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-03-28T15:37:24.346 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/toshiba_haps.ko 2024-03-28T15:37:24.346 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/wmi-bmof.ko 2024-03-28T15:37:24.346 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/wmi.ko 2024-03-28T15:37:24.347 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap 2024-03-28T15:37:24.347 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap/intel_rapl_common.ko 2024-03-28T15:37:24.347 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap/intel_rapl_msr.ko 2024-03-28T15:37:24.347 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps 2024-03-28T15:37:24.347 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients 2024-03-28T15:37:24.347 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps-gpio.ko 2024-03-28T15:37:24.347 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps-ldisc.ko 2024-03-28T15:37:24.347 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps_parport.ko 2024-03-28T15:37:24.347 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/pps_core.ko 2024-03-28T15:37:24.347 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp 2024-03-28T15:37:24.347 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp/ptp.ko 2024-03-28T15:37:24.347 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp/ptp_kvm.ko 2024-03-28T15:37:24.347 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc 2024-03-28T15:37:24.347 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-bq32k.ko 2024-03-28T15:37:24.348 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1286.ko 2024-03-28T15:37:24.348 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1307.ko 2024-03-28T15:37:24.348 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1374.ko 2024-03-28T15:37:24.348 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1511.ko 2024-03-28T15:37:24.348 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1553.ko 2024-03-28T15:37:24.348 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1672.ko 2024-03-28T15:37:24.348 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1742.ko 2024-03-28T15:37:24.348 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds2404.ko 2024-03-28T15:37:24.348 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds3232.ko 2024-03-28T15:37:24.348 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-em3027.ko 2024-03-28T15:37:24.348 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-fm3130.ko 2024-03-28T15:37:24.348 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-isl12022.ko 2024-03-28T15:37:24.348 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-isl1208.ko 2024-03-28T15:37:24.348 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m41t80.ko 2024-03-28T15:37:24.348 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m48t35.ko 2024-03-28T15:37:24.349 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m48t59.ko 2024-03-28T15:37:24.349 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-max6900.ko 2024-03-28T15:37:24.349 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-msm6242.ko 2024-03-28T15:37:24.349 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf2127.ko 2024-03-28T15:37:24.349 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf85063.ko 2024-03-28T15:37:24.349 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8523.ko 2024-03-28T15:37:24.349 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8563.ko 2024-03-28T15:37:24.349 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8583.ko 2024-03-28T15:37:24.349 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rp5c01.ko 2024-03-28T15:37:24.349 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rs5c372.ko 2024-03-28T15:37:24.349 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-03-28T15:37:24.349 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rx8025.ko 2024-03-28T15:37:24.349 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rx8581.ko 2024-03-28T15:37:24.349 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-03-28T15:37:24.349 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-x1205.ko 2024-03-28T15:37:24.350 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi 2024-03-28T15:37:24.350 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-9xxx.ko 2024-03-28T15:37:24.350 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-sas.ko 2024-03-28T15:37:24.350 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-xxxx.ko 2024-03-28T15:37:24.350 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/BusLogic.ko 2024-03-28T15:37:24.350 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/a100u2w.ko 2024-03-28T15:37:24.350 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aacraid 2024-03-28T15:37:24.350 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aacraid/aacraid.ko 2024-03-28T15:37:24.350 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/advansys.ko 2024-03-28T15:37:24.350 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx 2024-03-28T15:37:24.350 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-03-28T15:37:24.350 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-03-28T15:37:24.350 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/am53c974.ko 2024-03-28T15:37:24.350 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/arcmsr 2024-03-28T15:37:24.350 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-03-28T15:37:24.351 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/atp870u.ko 2024-03-28T15:37:24.351 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/be2iscsi 2024-03-28T15:37:24.351 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-03-28T15:37:24.351 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bfa 2024-03-28T15:37:24.351 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bfa/bfa.ko 2024-03-28T15:37:24.351 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2fc 2024-03-28T15:37:24.351 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-03-28T15:37:24.351 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2i 2024-03-28T15:37:24.351 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-03-28T15:37:24.351 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ch.ko 2024-03-28T15:37:24.351 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/csiostor 2024-03-28T15:37:24.351 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/csiostor/csiostor.ko 2024-03-28T15:37:24.351 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi 2024-03-28T15:37:24.351 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb3i 2024-03-28T15:37:24.352 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-03-28T15:37:24.352 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb4i 2024-03-28T15:37:24.352 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-03-28T15:37:24.352 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-03-28T15:37:24.352 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/dc395x.ko 2024-03-28T15:37:24.352 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler 2024-03-28T15:37:24.352 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-03-28T15:37:24.352 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-03-28T15:37:24.352 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-03-28T15:37:24.352 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-03-28T15:37:24.352 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/dmx3191d.ko 2024-03-28T15:37:24.352 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esas2r 2024-03-28T15:37:24.352 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esas2r/esas2r.ko 2024-03-28T15:37:24.352 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esp_scsi.ko 2024-03-28T15:37:24.353 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe 2024-03-28T15:37:24.353 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe/fcoe.ko 2024-03-28T15:37:24.353 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-03-28T15:37:24.353 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fnic 2024-03-28T15:37:24.353 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fnic/fnic.ko 2024-03-28T15:37:24.353 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hpsa.ko 2024-03-28T15:37:24.353 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hptiop.ko 2024-03-28T15:37:24.353 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hv_storvsc.ko 2024-03-28T15:37:24.353 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/initio.ko 2024-03-28T15:37:24.353 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ipr.ko 2024-03-28T15:37:24.353 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ips.ko 2024-03-28T15:37:24.353 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/isci 2024-03-28T15:37:24.353 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/isci/isci.ko 2024-03-28T15:37:24.353 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-03-28T15:37:24.353 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/iscsi_tcp.ko 2024-03-28T15:37:24.353 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libfc 2024-03-28T15:37:24.354 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libfc/libfc.ko 2024-03-28T15:37:24.354 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libiscsi.ko 2024-03-28T15:37:24.354 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libiscsi_tcp.ko 2024-03-28T15:37:24.354 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libsas 2024-03-28T15:37:24.354 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libsas/libsas.ko 2024-03-28T15:37:24.354 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/lpfc 2024-03-28T15:37:24.354 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/lpfc/lpfc.ko 2024-03-28T15:37:24.354 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid 2024-03-28T15:37:24.354 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid.ko 2024-03-28T15:37:24.354 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-03-28T15:37:24.354 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-03-28T15:37:24.354 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-03-28T15:37:24.354 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mpt3sas 2024-03-28T15:37:24.354 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-03-28T15:37:24.355 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvsas 2024-03-28T15:37:24.355 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvsas/mvsas.ko 2024-03-28T15:37:24.355 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvumi.ko 2024-03-28T15:37:24.355 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pm8001 2024-03-28T15:37:24.355 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-03-28T15:37:24.355 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pmcraid.ko 2024-03-28T15:37:24.355 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla1280.ko 2024-03-28T15:37:24.355 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx 2024-03-28T15:37:24.355 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-03-28T15:37:24.355 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-03-28T15:37:24.355 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla4xxx 2024-03-28T15:37:24.355 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-03-28T15:37:24.355 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/raid_class.ko 2024-03-28T15:37:24.355 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_debug.ko 2024-03-28T15:37:24.355 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_fc.ko 2024-03-28T15:37:24.356 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-03-28T15:37:24.356 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_sas.ko 2024-03-28T15:37:24.356 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_spi.ko 2024-03-28T15:37:24.356 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_srp.ko 2024-03-28T15:37:24.356 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ses.ko 2024-03-28T15:37:24.356 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/st.ko 2024-03-28T15:37:24.356 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/stex.ko 2024-03-28T15:37:24.356 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/sym53c8xx_2 2024-03-28T15:37:24.356 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-03-28T15:37:24.356 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/virtio_scsi.ko 2024-03-28T15:37:24.356 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/vmw_pvscsi.ko 2024-03-28T15:37:24.356 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/wd719x.ko 2024-03-28T15:37:24.356 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ssb 2024-03-28T15:37:24.356 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ssb/ssb.ko 2024-03-28T15:37:24.357 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target 2024-03-28T15:37:24.357 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/iscsi 2024-03-28T15:37:24.357 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-03-28T15:37:24.357 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/loopback 2024-03-28T15:37:24.357 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/loopback/tcm_loop.ko 2024-03-28T15:37:24.357 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_file.ko 2024-03-28T15:37:24.357 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_iblock.ko 2024-03-28T15:37:24.357 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_mod.ko 2024-03-28T15:37:24.357 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_pscsi.ko 2024-03-28T15:37:24.357 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_user.ko 2024-03-28T15:37:24.357 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/tcm_fc 2024-03-28T15:37:24.357 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-03-28T15:37:24.357 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal 2024-03-28T15:37:24.357 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel 2024-03-28T15:37:24.358 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal 2024-03-28T15:37:24.358 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-03-28T15:37:24.358 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-03-28T15:37:24.358 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-03-28T15:37:24.358 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-03-28T15:37:24.358 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-03-28T15:37:24.358 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-03-28T15:37:24.358 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-03-28T15:37:24.358 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-03-28T15:37:24.358 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-03-28T15:37:24.358 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-03-28T15:37:24.358 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-03-28T15:37:24.358 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-03-28T15:37:24.358 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-03-28T15:37:24.359 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-03-28T15:37:24.359 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-03-28T15:37:24.359 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-03-28T15:37:24.359 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-03-28T15:37:24.359 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-03-28T15:37:24.359 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty 2024-03-28T15:37:24.359 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/ipwireless 2024-03-28T15:37:24.359 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-03-28T15:37:24.359 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/n_gsm.ko 2024-03-28T15:37:24.359 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/n_hdlc.ko 2024-03-28T15:37:24.359 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/nozomi.ko 2024-03-28T15:37:24.359 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial 2024-03-28T15:37:24.359 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/8250 2024-03-28T15:37:24.359 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-03-28T15:37:24.359 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/arc_uart.ko 2024-03-28T15:37:24.360 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/jsm 2024-03-28T15:37:24.360 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/jsm/jsm.ko 2024-03-28T15:37:24.360 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs 2024-03-28T15:37:24.360 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/core 2024-03-28T15:37:24.360 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/core/ufshcd-core.ko 2024-03-28T15:37:24.360 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/host 2024-03-28T15:37:24.360 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-03-28T15:37:24.360 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio 2024-03-28T15:37:24.360 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio.ko 2024-03-28T15:37:24.360 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_aec.ko 2024-03-28T15:37:24.360 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_cif.ko 2024-03-28T15:37:24.360 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_pci_generic.ko 2024-03-28T15:37:24.360 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_sercos3.ko 2024-03-28T15:37:24.360 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb 2024-03-28T15:37:24.361 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class 2024-03-28T15:37:24.361 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/cdc-acm.ko 2024-03-28T15:37:24.361 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/cdc-wdm.ko 2024-03-28T15:37:24.361 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/usblp.ko 2024-03-28T15:37:24.361 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/usbtmc.ko 2024-03-28T15:37:24.361 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host 2024-03-28T15:37:24.361 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host/sl811-hcd.ko 2024-03-28T15:37:24.361 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image 2024-03-28T15:37:24.361 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image/mdc800.ko 2024-03-28T15:37:24.361 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image/microtek.ko 2024-03-28T15:37:24.361 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc 2024-03-28T15:37:24.361 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/adutux.ko 2024-03-28T15:37:24.361 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/appledisplay.ko 2024-03-28T15:37:24.361 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/emi26.ko 2024-03-28T15:37:24.361 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/emi62.ko 2024-03-28T15:37:24.362 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/ezusb.ko 2024-03-28T15:37:24.362 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/idmouse.ko 2024-03-28T15:37:24.362 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/iowarrior.ko 2024-03-28T15:37:24.362 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/isight_firmware.ko 2024-03-28T15:37:24.362 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/ldusb.ko 2024-03-28T15:37:24.362 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/legousbtower.ko 2024-03-28T15:37:24.362 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/sisusbvga 2024-03-28T15:37:24.362 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-03-28T15:37:24.362 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/trancevibrator.ko 2024-03-28T15:37:24.362 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usb3503.ko 2024-03-28T15:37:24.362 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usblcd.ko 2024-03-28T15:37:24.362 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usbsevseg.ko 2024-03-28T15:37:24.362 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/uss720.ko 2024-03-28T15:37:24.362 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/yurex.ko 2024-03-28T15:37:24.363 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/phy 2024-03-28T15:37:24.363 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/phy/phy-generic.ko 2024-03-28T15:37:24.363 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial 2024-03-28T15:37:24.363 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/aircable.ko 2024-03-28T15:37:24.363 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ark3116.ko 2024-03-28T15:37:24.363 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/belkin_sa.ko 2024-03-28T15:37:24.363 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ch341.ko 2024-03-28T15:37:24.363 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cp210x.ko 2024-03-28T15:37:24.363 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cyberjack.ko 2024-03-28T15:37:24.363 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cypress_m8.ko 2024-03-28T15:37:24.363 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/digi_acceleport.ko 2024-03-28T15:37:24.363 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/empeg.ko 2024-03-28T15:37:24.363 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ftdi_sio.ko 2024-03-28T15:37:24.363 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/garmin_gps.ko 2024-03-28T15:37:24.363 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/io_edgeport.ko 2024-03-28T15:37:24.364 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/io_ti.ko 2024-03-28T15:37:24.364 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ipaq.ko 2024-03-28T15:37:24.364 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ipw.ko 2024-03-28T15:37:24.365 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ir-usb.ko 2024-03-28T15:37:24.365 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-03-28T15:37:24.365 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/keyspan.ko 2024-03-28T15:37:24.365 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/keyspan_pda.ko 2024-03-28T15:37:24.365 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/kl5kusb105.ko 2024-03-28T15:37:24.365 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/kobil_sct.ko 2024-03-28T15:37:24.366 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mct_u232.ko 2024-03-28T15:37:24.366 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mos7720.ko 2024-03-28T15:37:24.366 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mos7840.ko 2024-03-28T15:37:24.366 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/navman.ko 2024-03-28T15:37:24.366 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/omninet.ko 2024-03-28T15:37:24.366 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/opticon.ko 2024-03-28T15:37:24.366 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/option.ko 2024-03-28T15:37:24.366 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/oti6858.ko 2024-03-28T15:37:24.366 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/pl2303.ko 2024-03-28T15:37:24.366 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/qcaux.ko 2024-03-28T15:37:24.366 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/qcserial.ko 2024-03-28T15:37:24.366 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/quatech2.ko 2024-03-28T15:37:24.366 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/safe_serial.ko 2024-03-28T15:37:24.366 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/sierra.ko 2024-03-28T15:37:24.367 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/spcp8x5.ko 2024-03-28T15:37:24.367 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ssu100.ko 2024-03-28T15:37:24.367 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/symbolserial.ko 2024-03-28T15:37:24.367 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-03-28T15:37:24.367 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-03-28T15:37:24.367 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb_debug.ko 2024-03-28T15:37:24.367 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb_wwan.ko 2024-03-28T15:37:24.367 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/visor.ko 2024-03-28T15:37:24.367 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/whiteheat.ko 2024-03-28T15:37:24.367 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/xsens_mt.ko 2024-03-28T15:37:24.367 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage 2024-03-28T15:37:24.367 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/uas.ko 2024-03-28T15:37:24.367 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-alauda.ko 2024-03-28T15:37:24.367 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-cypress.ko 2024-03-28T15:37:24.367 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-datafab.ko 2024-03-28T15:37:24.368 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-03-28T15:37:24.368 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-freecom.ko 2024-03-28T15:37:24.368 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-isd200.ko 2024-03-28T15:37:24.368 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-03-28T15:37:24.368 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-karma.ko 2024-03-28T15:37:24.368 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-onetouch.ko 2024-03-28T15:37:24.368 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-realtek.ko 2024-03-28T15:37:24.368 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-sddr09.ko 2024-03-28T15:37:24.368 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-sddr55.ko 2024-03-28T15:37:24.368 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-usbat.ko 2024-03-28T15:37:24.368 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/usb-storage.ko 2024-03-28T15:37:24.368 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip 2024-03-28T15:37:24.368 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/usbip-core.ko 2024-03-28T15:37:24.368 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/usbip-host.ko 2024-03-28T15:37:24.369 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-03-28T15:37:24.369 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio 2024-03-28T15:37:24.369 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci 2024-03-28T15:37:24.369 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-03-28T15:37:24.369 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci/vfio-pci.ko 2024-03-28T15:37:24.369 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/vfio.ko 2024-03-28T15:37:24.369 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-03-28T15:37:24.369 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost 2024-03-28T15:37:24.369 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost.ko 2024-03-28T15:37:24.369 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_iotlb.ko 2024-03-28T15:37:24.369 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_net.ko 2024-03-28T15:37:24.369 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_scsi.ko 2024-03-28T15:37:24.369 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video 2024-03-28T15:37:24.369 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight 2024-03-28T15:37:24.369 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/apple_bl.ko 2024-03-28T15:37:24.370 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/lcd.ko 2024-03-28T15:37:24.370 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/platform_lcd.ko 2024-03-28T15:37:24.370 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio 2024-03-28T15:37:24.370 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio.ko 2024-03-28T15:37:24.370 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_balloon.ko 2024-03-28T15:37:24.370 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_mmio.ko 2024-03-28T15:37:24.370 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_pci.ko 2024-03-28T15:37:24.370 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-03-28T15:37:24.370 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-03-28T15:37:24.370 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_ring.ko 2024-03-28T15:37:24.370 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1 2024-03-28T15:37:24.370 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters 2024-03-28T15:37:24.370 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters/ds2482.ko 2024-03-28T15:37:24.370 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters/ds2490.ko 2024-03-28T15:37:24.371 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves 2024-03-28T15:37:24.371 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-03-28T15:37:24.371 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-03-28T15:37:24.371 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-03-28T15:37:24.371 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-03-28T15:37:24.371 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-03-28T15:37:24.371 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-03-28T15:37:24.371 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-03-28T15:37:24.371 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-03-28T15:37:24.371 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-03-28T15:37:24.371 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_smem.ko 2024-03-28T15:37:24.371 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_therm.ko 2024-03-28T15:37:24.371 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/wire.ko 2024-03-28T15:37:24.371 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog 2024-03-28T15:37:24.371 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/alim1535_wdt.ko 2024-03-28T15:37:24.372 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/alim7101_wdt.ko 2024-03-28T15:37:24.372 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/f71808e_wdt.ko 2024-03-28T15:37:24.372 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/hpwdt.ko 2024-03-28T15:37:24.372 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/i6300esb.ko 2024-03-28T15:37:24.372 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-03-28T15:37:24.372 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/iTCO_wdt.ko 2024-03-28T15:37:24.372 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ib700wdt.ko 2024-03-28T15:37:24.372 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ibmasr.ko 2024-03-28T15:37:24.372 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-03-28T15:37:24.372 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/it8712f_wdt.ko 2024-03-28T15:37:24.372 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/it87_wdt.ko 2024-03-28T15:37:24.372 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/machzwd.ko 2024-03-28T15:37:24.372 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/nv_tco.ko 2024-03-28T15:37:24.372 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/pcwd_pci.ko 2024-03-28T15:37:24.372 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/pcwd_usb.ko 2024-03-28T15:37:24.373 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-03-28T15:37:24.373 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sch311x_wdt.ko 2024-03-28T15:37:24.373 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/softdog.ko 2024-03-28T15:37:24.373 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sp5100_tco.ko 2024-03-28T15:37:24.373 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/via_wdt.ko 2024-03-28T15:37:24.373 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-03-28T15:37:24.373 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83877f_wdt.ko 2024-03-28T15:37:24.373 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83977f_wdt.ko 2024-03-28T15:37:24.373 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/wdt_pci.ko 2024-03-28T15:37:24.373 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/xen_wdt.ko 2024-03-28T15:37:24.373 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen 2024-03-28T15:37:24.373 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-acpi-processor.ko 2024-03-28T15:37:24.373 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-evtchn.ko 2024-03-28T15:37:24.373 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-gntalloc.ko 2024-03-28T15:37:24.374 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-gntdev.ko 2024-03-28T15:37:24.374 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-pciback 2024-03-28T15:37:24.374 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-03-28T15:37:24.374 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-privcmd.ko 2024-03-28T15:37:24.374 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xenfs 2024-03-28T15:37:24.374 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xenfs/xenfs.ko 2024-03-28T15:37:24.374 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs 2024-03-28T15:37:24.374 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/9p 2024-03-28T15:37:24.374 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/9p/9p.ko 2024-03-28T15:37:24.374 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/affs 2024-03-28T15:37:24.374 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/affs/affs.ko 2024-03-28T15:37:24.374 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/befs 2024-03-28T15:37:24.374 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/befs/befs.ko 2024-03-28T15:37:24.374 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/binfmt_misc.ko 2024-03-28T15:37:24.375 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/btrfs 2024-03-28T15:37:24.375 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/btrfs/btrfs.ko 2024-03-28T15:37:24.375 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cachefiles 2024-03-28T15:37:24.375 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cachefiles/cachefiles.ko 2024-03-28T15:37:24.375 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ceph 2024-03-28T15:37:24.375 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ceph/ceph.ko 2024-03-28T15:37:24.375 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/coda 2024-03-28T15:37:24.375 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/coda/coda.ko 2024-03-28T15:37:24.375 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cramfs 2024-03-28T15:37:24.375 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cramfs/cramfs.ko 2024-03-28T15:37:24.375 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/dlm 2024-03-28T15:37:24.375 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/dlm/dlm.ko 2024-03-28T15:37:24.375 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ecryptfs 2024-03-28T15:37:24.375 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ecryptfs/ecryptfs.ko 2024-03-28T15:37:24.376 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/f2fs 2024-03-28T15:37:24.376 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/f2fs/f2fs.ko 2024-03-28T15:37:24.376 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat 2024-03-28T15:37:24.376 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat/fat.ko 2024-03-28T15:37:24.376 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat/msdos.ko 2024-03-28T15:37:24.376 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat/vfat.ko 2024-03-28T15:37:24.376 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fuse 2024-03-28T15:37:24.376 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fuse/cuse.ko 2024-03-28T15:37:24.376 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fuse/fuse.ko 2024-03-28T15:37:24.376 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/gfs2 2024-03-28T15:37:24.376 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/gfs2/gfs2.ko 2024-03-28T15:37:24.376 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfs 2024-03-28T15:37:24.376 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfs/hfs.ko 2024-03-28T15:37:24.376 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfsplus 2024-03-28T15:37:24.376 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfsplus/hfsplus.ko 2024-03-28T15:37:24.377 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/isofs 2024-03-28T15:37:24.377 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/isofs/isofs.ko 2024-03-28T15:37:24.377 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/jfs 2024-03-28T15:37:24.377 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/jfs/jfs.ko 2024-03-28T15:37:24.377 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/lockd 2024-03-28T15:37:24.377 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/lockd/lockd.ko 2024-03-28T15:37:24.377 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/minix 2024-03-28T15:37:24.377 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/minix/minix.ko 2024-03-28T15:37:24.377 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/netfs 2024-03-28T15:37:24.377 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/netfs/netfs.ko 2024-03-28T15:37:24.377 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs 2024-03-28T15:37:24.377 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/blocklayout 2024-03-28T15:37:24.377 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-03-28T15:37:24.377 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/filelayout 2024-03-28T15:37:24.377 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-03-28T15:37:24.378 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/flexfilelayout 2024-03-28T15:37:24.378 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-03-28T15:37:24.378 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfs.ko 2024-03-28T15:37:24.378 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfsv3.ko 2024-03-28T15:37:24.378 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfsv4.ko 2024-03-28T15:37:24.378 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common 2024-03-28T15:37:24.378 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common/grace.ko 2024-03-28T15:37:24.378 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common/nfs_acl.ko 2024-03-28T15:37:24.378 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfsd 2024-03-28T15:37:24.378 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfsd/nfsd.ko 2024-03-28T15:37:24.378 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nilfs2 2024-03-28T15:37:24.378 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nilfs2/nilfs2.ko 2024-03-28T15:37:24.378 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls 2024-03-28T15:37:24.378 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-celtic.ko 2024-03-28T15:37:24.379 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-centeuro.ko 2024-03-28T15:37:24.379 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-croatian.ko 2024-03-28T15:37:24.379 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-cyrillic.ko 2024-03-28T15:37:24.379 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-gaelic.ko 2024-03-28T15:37:24.379 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-greek.ko 2024-03-28T15:37:24.379 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-iceland.ko 2024-03-28T15:37:24.379 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-inuit.ko 2024-03-28T15:37:24.379 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-roman.ko 2024-03-28T15:37:24.379 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-romanian.ko 2024-03-28T15:37:24.379 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-turkish.ko 2024-03-28T15:37:24.379 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1250.ko 2024-03-28T15:37:24.379 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1251.ko 2024-03-28T15:37:24.379 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1255.ko 2024-03-28T15:37:24.379 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp737.ko 2024-03-28T15:37:24.379 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp775.ko 2024-03-28T15:37:24.380 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp850.ko 2024-03-28T15:37:24.380 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp852.ko 2024-03-28T15:37:24.380 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp855.ko 2024-03-28T15:37:24.380 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp857.ko 2024-03-28T15:37:24.380 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp860.ko 2024-03-28T15:37:24.380 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp861.ko 2024-03-28T15:37:24.380 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp862.ko 2024-03-28T15:37:24.380 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp863.ko 2024-03-28T15:37:24.380 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp864.ko 2024-03-28T15:37:24.380 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp865.ko 2024-03-28T15:37:24.380 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp866.ko 2024-03-28T15:37:24.380 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp869.ko 2024-03-28T15:37:24.380 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp874.ko 2024-03-28T15:37:24.380 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp932.ko 2024-03-28T15:37:24.380 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp936.ko 2024-03-28T15:37:24.381 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp949.ko 2024-03-28T15:37:24.381 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp950.ko 2024-03-28T15:37:24.381 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_euc-jp.ko 2024-03-28T15:37:24.381 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-1.ko 2024-03-28T15:37:24.381 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-13.ko 2024-03-28T15:37:24.381 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-14.ko 2024-03-28T15:37:24.381 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-15.ko 2024-03-28T15:37:24.381 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-2.ko 2024-03-28T15:37:24.381 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-3.ko 2024-03-28T15:37:24.381 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-4.ko 2024-03-28T15:37:24.381 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-5.ko 2024-03-28T15:37:24.381 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-6.ko 2024-03-28T15:37:24.381 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-7.ko 2024-03-28T15:37:24.381 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-9.ko 2024-03-28T15:37:24.381 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-r.ko 2024-03-28T15:37:24.382 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-ru.ko 2024-03-28T15:37:24.382 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-u.ko 2024-03-28T15:37:24.382 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_ucs2_utils.ko 2024-03-28T15:37:24.382 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_utf8.ko 2024-03-28T15:37:24.382 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2 2024-03-28T15:37:24.382 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/cluster 2024-03-28T15:37:24.382 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-03-28T15:37:24.382 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlm 2024-03-28T15:37:24.382 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-03-28T15:37:24.382 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlmfs 2024-03-28T15:37:24.382 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-03-28T15:37:24.382 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2.ko 2024-03-28T15:37:24.382 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-03-28T15:37:24.382 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-03-28T15:37:24.383 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-03-28T15:37:24.383 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/overlayfs 2024-03-28T15:37:24.383 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/overlayfs/overlay.ko 2024-03-28T15:37:24.383 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/pstore 2024-03-28T15:37:24.383 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/pstore/ramoops.ko 2024-03-28T15:37:24.383 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/reiserfs 2024-03-28T15:37:24.383 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/reiserfs/reiserfs.ko 2024-03-28T15:37:24.383 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/romfs 2024-03-28T15:37:24.383 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/romfs/romfs.ko 2024-03-28T15:37:24.383 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb 2024-03-28T15:37:24.383 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/client 2024-03-28T15:37:24.383 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/client/cifs.ko 2024-03-28T15:37:24.383 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common 2024-03-28T15:37:24.383 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common/cifs_arc4.ko 2024-03-28T15:37:24.384 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common/cifs_md4.ko 2024-03-28T15:37:24.384 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/squashfs 2024-03-28T15:37:24.384 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/squashfs/squashfs.ko 2024-03-28T15:37:24.384 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/sysv 2024-03-28T15:37:24.384 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/sysv/sysv.ko 2024-03-28T15:37:24.384 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/udf 2024-03-28T15:37:24.384 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/udf/udf.ko 2024-03-28T15:37:24.384 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ufs 2024-03-28T15:37:24.384 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ufs/ufs.ko 2024-03-28T15:37:24.384 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/xfs 2024-03-28T15:37:24.384 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/xfs/xfs.ko 2024-03-28T15:37:24.384 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel 2024-03-28T15:37:24.384 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/rcu 2024-03-28T15:37:24.384 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/rcu/rcutorture.ko 2024-03-28T15:37:24.384 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/torture.ko 2024-03-28T15:37:24.385 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/trace 2024-03-28T15:37:24.385 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-03-28T15:37:24.385 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib 2024-03-28T15:37:24.385 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/asn1_encoder.ko 2024-03-28T15:37:24.385 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crc-itu-t.ko 2024-03-28T15:37:24.385 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crc8.ko 2024-03-28T15:37:24.385 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto 2024-03-28T15:37:24.385 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto/libarc4.ko 2024-03-28T15:37:24.385 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto/libdes.ko 2024-03-28T15:37:24.385 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/libcrc32c.ko 2024-03-28T15:37:24.385 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lru_cache.ko 2024-03-28T15:37:24.385 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4 2024-03-28T15:37:24.385 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4/lz4_compress.ko 2024-03-28T15:37:24.385 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4/lz4hc_compress.ko 2024-03-28T15:37:24.385 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/math 2024-03-28T15:37:24.386 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/math/cordic.ko 2024-03-28T15:37:24.386 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/raid6 2024-03-28T15:37:24.386 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/raid6/raid6_pq.ko 2024-03-28T15:37:24.386 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/reed_solomon 2024-03-28T15:37:24.386 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/reed_solomon/reed_solomon.ko 2024-03-28T15:37:24.386 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_bm.ko 2024-03-28T15:37:24.386 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_fsm.ko 2024-03-28T15:37:24.386 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_kmp.ko 2024-03-28T15:37:24.386 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/zstd 2024-03-28T15:37:24.386 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/zstd/zstd_compress.ko 2024-03-28T15:37:24.386 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/mm 2024-03-28T15:37:24.386 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/mm/hwpoison-inject.ko 2024-03-28T15:37:24.386 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net 2024-03-28T15:37:24.386 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan 2024-03-28T15:37:24.387 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/6lowpan.ko 2024-03-28T15:37:24.387 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_dest.ko 2024-03-28T15:37:24.387 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_fragment.ko 2024-03-28T15:37:24.387 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_hop.ko 2024-03-28T15:37:24.387 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_ipv6.ko 2024-03-28T15:37:24.387 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_mobility.ko 2024-03-28T15:37:24.387 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_routing.ko 2024-03-28T15:37:24.387 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_udp.ko 2024-03-28T15:37:24.387 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802 2024-03-28T15:37:24.387 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/garp.ko 2024-03-28T15:37:24.387 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/mrp.ko 2024-03-28T15:37:24.387 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/p8022.ko 2024-03-28T15:37:24.387 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/psnap.ko 2024-03-28T15:37:24.387 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/stp.ko 2024-03-28T15:37:24.388 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/8021q 2024-03-28T15:37:24.388 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/8021q/8021q.ko 2024-03-28T15:37:24.388 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p 2024-03-28T15:37:24.388 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet.ko 2024-03-28T15:37:24.388 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_fd.ko 2024-03-28T15:37:24.388 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_rdma.ko 2024-03-28T15:37:24.388 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_virtio.ko 2024-03-28T15:37:24.388 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/batman-adv 2024-03-28T15:37:24.388 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/batman-adv/batman-adv.ko 2024-03-28T15:37:24.388 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge 2024-03-28T15:37:24.388 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/br_netfilter.ko 2024-03-28T15:37:24.388 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/bridge.ko 2024-03-28T15:37:24.388 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter 2024-03-28T15:37:24.388 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-03-28T15:37:24.388 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_among.ko 2024-03-28T15:37:24.389 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_arp.ko 2024-03-28T15:37:24.389 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-03-28T15:37:24.389 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-03-28T15:37:24.389 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_ip.ko 2024-03-28T15:37:24.389 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-03-28T15:37:24.389 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_limit.ko 2024-03-28T15:37:24.389 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_log.ko 2024-03-28T15:37:24.389 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_mark.ko 2024-03-28T15:37:24.389 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-03-28T15:37:24.389 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-03-28T15:37:24.389 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-03-28T15:37:24.389 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-03-28T15:37:24.389 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_snat.ko 2024-03-28T15:37:24.389 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_stp.ko 2024-03-28T15:37:24.389 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-03-28T15:37:24.390 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-03-28T15:37:24.390 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-03-28T15:37:24.390 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-03-28T15:37:24.390 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtables.ko 2024-03-28T15:37:24.390 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-03-28T15:37:24.390 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-03-28T15:37:24.390 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-03-28T15:37:24.390 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ceph 2024-03-28T15:37:24.390 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ceph/libceph.ko 2024-03-28T15:37:24.390 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/core 2024-03-28T15:37:24.390 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/core/failover.ko 2024-03-28T15:37:24.390 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/core/pktgen.ko 2024-03-28T15:37:24.390 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp 2024-03-28T15:37:24.390 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp.ko 2024-03-28T15:37:24.391 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_diag.ko 2024-03-28T15:37:24.391 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_ipv4.ko 2024-03-28T15:37:24.391 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_ipv6.ko 2024-03-28T15:37:24.391 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dns_resolver 2024-03-28T15:37:24.391 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dns_resolver/dns_resolver.ko 2024-03-28T15:37:24.391 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa 2024-03-28T15:37:24.391 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/dsa_core.ko 2024-03-28T15:37:24.391 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_brcm.ko 2024-03-28T15:37:24.391 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_dsa.ko 2024-03-28T15:37:24.391 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_none.ko 2024-03-28T15:37:24.391 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_trailer.ko 2024-03-28T15:37:24.391 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154 2024-03-28T15:37:24.391 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/6lowpan 2024-03-28T15:37:24.391 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-03-28T15:37:24.391 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/ieee802154.ko 2024-03-28T15:37:24.392 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/ieee802154_socket.ko 2024-03-28T15:37:24.392 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4 2024-03-28T15:37:24.392 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ah4.ko 2024-03-28T15:37:24.392 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/esp4.ko 2024-03-28T15:37:24.392 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/fou.ko 2024-03-28T15:37:24.392 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/gre.ko 2024-03-28T15:37:24.392 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/inet_diag.ko 2024-03-28T15:37:24.392 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_gre.ko 2024-03-28T15:37:24.392 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_tunnel.ko 2024-03-28T15:37:24.392 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_vti.ko 2024-03-28T15:37:24.392 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ipcomp.ko 2024-03-28T15:37:24.392 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ipip.ko 2024-03-28T15:37:24.392 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter 2024-03-28T15:37:24.392 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arp_tables.ko 2024-03-28T15:37:24.393 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-03-28T15:37:24.393 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-03-28T15:37:24.393 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-03-28T15:37:24.393 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-03-28T15:37:24.393 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-03-28T15:37:24.393 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-03-28T15:37:24.393 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-03-28T15:37:24.393 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-03-28T15:37:24.393 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-03-28T15:37:24.393 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_security.ko 2024-03-28T15:37:24.393 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-03-28T15:37:24.393 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-03-28T15:37:24.393 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-03-28T15:37:24.393 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-03-28T15:37:24.394 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-03-28T15:37:24.394 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-03-28T15:37:24.394 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-03-28T15:37:24.394 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-03-28T15:37:24.394 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-03-28T15:37:24.394 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-03-28T15:37:24.394 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_bic.ko 2024-03-28T15:37:24.394 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_dctcp.ko 2024-03-28T15:37:24.394 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_diag.ko 2024-03-28T15:37:24.394 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_highspeed.ko 2024-03-28T15:37:24.394 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_htcp.ko 2024-03-28T15:37:24.394 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_hybla.ko 2024-03-28T15:37:24.394 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_illinois.ko 2024-03-28T15:37:24.394 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_lp.ko 2024-03-28T15:37:24.395 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_scalable.ko 2024-03-28T15:37:24.395 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_vegas.ko 2024-03-28T15:37:24.395 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_veno.ko 2024-03-28T15:37:24.395 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_westwood.ko 2024-03-28T15:37:24.395 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_yeah.ko 2024-03-28T15:37:24.395 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tunnel4.ko 2024-03-28T15:37:24.395 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/udp_diag.ko 2024-03-28T15:37:24.395 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/udp_tunnel.ko 2024-03-28T15:37:24.395 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/xfrm4_tunnel.ko 2024-03-28T15:37:24.395 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6 2024-03-28T15:37:24.395 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ah6.ko 2024-03-28T15:37:24.395 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/esp6.ko 2024-03-28T15:37:24.395 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/fou6.ko 2024-03-28T15:37:24.395 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_tunnel.ko 2024-03-28T15:37:24.395 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-03-28T15:37:24.396 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_vti.ko 2024-03-28T15:37:24.396 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ipcomp6.ko 2024-03-28T15:37:24.396 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter 2024-03-28T15:37:24.396 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-03-28T15:37:24.396 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-03-28T15:37:24.396 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-03-28T15:37:24.396 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-03-28T15:37:24.396 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-03-28T15:37:24.396 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-03-28T15:37:24.396 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-03-28T15:37:24.396 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-03-28T15:37:24.396 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-03-28T15:37:24.396 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-03-28T15:37:24.396 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-03-28T15:37:24.396 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-03-28T15:37:24.397 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-03-28T15:37:24.397 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-03-28T15:37:24.397 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-03-28T15:37:24.397 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-03-28T15:37:24.397 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-03-28T15:37:24.397 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-03-28T15:37:24.397 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-03-28T15:37:24.397 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-03-28T15:37:24.397 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-03-28T15:37:24.397 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-03-28T15:37:24.397 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-03-28T15:37:24.397 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-03-28T15:37:24.397 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/sit.ko 2024-03-28T15:37:24.397 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/tunnel6.ko 2024-03-28T15:37:24.398 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/xfrm6_tunnel.ko 2024-03-28T15:37:24.398 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/key 2024-03-28T15:37:24.398 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/key/af_key.ko 2024-03-28T15:37:24.398 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp 2024-03-28T15:37:24.398 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_core.ko 2024-03-28T15:37:24.398 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_debugfs.ko 2024-03-28T15:37:24.398 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_eth.ko 2024-03-28T15:37:24.398 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ip.ko 2024-03-28T15:37:24.398 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ip6.ko 2024-03-28T15:37:24.398 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_netlink.ko 2024-03-28T15:37:24.398 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ppp.ko 2024-03-28T15:37:24.398 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/llc 2024-03-28T15:37:24.398 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/llc/llc.ko 2024-03-28T15:37:24.398 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mac802154 2024-03-28T15:37:24.398 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mac802154/mac802154.ko 2024-03-28T15:37:24.399 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mpls 2024-03-28T15:37:24.399 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mpls/mpls_gso.ko 2024-03-28T15:37:24.399 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter 2024-03-28T15:37:24.399 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset 2024-03-28T15:37:24.399 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set.ko 2024-03-28T15:37:24.399 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-03-28T15:37:24.399 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-03-28T15:37:24.399 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-03-28T15:37:24.399 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-03-28T15:37:24.399 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-03-28T15:37:24.399 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-03-28T15:37:24.399 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-03-28T15:37:24.399 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-03-28T15:37:24.399 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-03-28T15:37:24.400 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-03-28T15:37:24.400 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-03-28T15:37:24.400 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-03-28T15:37:24.400 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-03-28T15:37:24.400 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-03-28T15:37:24.400 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-03-28T15:37:24.400 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs 2024-03-28T15:37:24.400 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs.ko 2024-03-28T15:37:24.400 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-03-28T15:37:24.400 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-03-28T15:37:24.400 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-03-28T15:37:24.400 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-03-28T15:37:24.400 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-03-28T15:37:24.400 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-03-28T15:37:24.401 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-03-28T15:37:24.401 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-03-28T15:37:24.401 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-03-28T15:37:24.401 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-03-28T15:37:24.401 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-03-28T15:37:24.401 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-03-28T15:37:24.401 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-03-28T15:37:24.401 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conncount.ko 2024-03-28T15:37:24.401 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack.ko 2024-03-28T15:37:24.401 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-03-28T15:37:24.401 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-03-28T15:37:24.401 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-03-28T15:37:24.401 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_h323.ko 2024-03-28T15:37:24.401 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_irc.ko 2024-03-28T15:37:24.401 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-03-28T15:37:24.402 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-03-28T15:37:24.402 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-03-28T15:37:24.402 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_sane.ko 2024-03-28T15:37:24.402 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_sip.ko 2024-03-28T15:37:24.402 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-03-28T15:37:24.402 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-03-28T15:37:24.402 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_dup_netdev.ko 2024-03-28T15:37:24.402 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_flow_table.ko 2024-03-28T15:37:24.402 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_flow_table_inet.ko 2024-03-28T15:37:24.402 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_log_syslog.ko 2024-03-28T15:37:24.402 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat.ko 2024-03-28T15:37:24.402 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_amanda.ko 2024-03-28T15:37:24.402 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_ftp.ko 2024-03-28T15:37:24.402 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_irc.ko 2024-03-28T15:37:24.403 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_sip.ko 2024-03-28T15:37:24.403 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_tftp.ko 2024-03-28T15:37:24.403 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_synproxy_core.ko 2024-03-28T15:37:24.403 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_tables.ko 2024-03-28T15:37:24.403 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink.ko 2024-03-28T15:37:24.403 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_acct.ko 2024-03-28T15:37:24.403 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_log.ko 2024-03-28T15:37:24.403 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_osf.ko 2024-03-28T15:37:24.403 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_queue.ko 2024-03-28T15:37:24.403 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_chain_nat.ko 2024-03-28T15:37:24.403 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_compat.ko 2024-03-28T15:37:24.403 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_ct.ko 2024-03-28T15:37:24.403 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_dup_netdev.ko 2024-03-28T15:37:24.403 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_fib.ko 2024-03-28T15:37:24.403 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_fwd_netdev.ko 2024-03-28T15:37:24.404 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_hash.ko 2024-03-28T15:37:24.404 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_limit.ko 2024-03-28T15:37:24.404 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_log.ko 2024-03-28T15:37:24.404 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_masq.ko 2024-03-28T15:37:24.404 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_nat.ko 2024-03-28T15:37:24.404 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_numgen.ko 2024-03-28T15:37:24.404 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_queue.ko 2024-03-28T15:37:24.404 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_quota.ko 2024-03-28T15:37:24.404 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_redir.ko 2024-03-28T15:37:24.404 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_reject.ko 2024-03-28T15:37:24.404 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_reject_inet.ko 2024-03-28T15:37:24.404 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_socket.ko 2024-03-28T15:37:24.404 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_tunnel.ko 2024-03-28T15:37:24.404 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_AUDIT.ko 2024-03-28T15:37:24.404 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CHECKSUM.ko 2024-03-28T15:37:24.405 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CLASSIFY.ko 2024-03-28T15:37:24.405 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-03-28T15:37:24.405 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CT.ko 2024-03-28T15:37:24.405 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_DSCP.ko 2024-03-28T15:37:24.405 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_HL.ko 2024-03-28T15:37:24.405 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_HMARK.ko 2024-03-28T15:37:24.405 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_IDLETIMER.ko 2024-03-28T15:37:24.405 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_LED.ko 2024-03-28T15:37:24.405 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_LOG.ko 2024-03-28T15:37:24.405 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_MASQUERADE.ko 2024-03-28T15:37:24.405 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NETMAP.ko 2024-03-28T15:37:24.405 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NFLOG.ko 2024-03-28T15:37:24.405 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NFQUEUE.ko 2024-03-28T15:37:24.405 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_RATEEST.ko 2024-03-28T15:37:24.405 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_REDIRECT.ko 2024-03-28T15:37:24.406 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_SECMARK.ko 2024-03-28T15:37:24.406 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TCPMSS.ko 2024-03-28T15:37:24.406 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-03-28T15:37:24.406 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TEE.ko 2024-03-28T15:37:24.406 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TPROXY.ko 2024-03-28T15:37:24.406 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TRACE.ko 2024-03-28T15:37:24.406 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_addrtype.ko 2024-03-28T15:37:24.406 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_bpf.ko 2024-03-28T15:37:24.406 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cgroup.ko 2024-03-28T15:37:24.406 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cluster.ko 2024-03-28T15:37:24.406 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_comment.ko 2024-03-28T15:37:24.406 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connbytes.ko 2024-03-28T15:37:24.406 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connlabel.ko 2024-03-28T15:37:24.406 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connlimit.ko 2024-03-28T15:37:24.406 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connmark.ko 2024-03-28T15:37:24.407 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_conntrack.ko 2024-03-28T15:37:24.407 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cpu.ko 2024-03-28T15:37:24.407 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_dccp.ko 2024-03-28T15:37:24.407 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_devgroup.ko 2024-03-28T15:37:24.407 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_dscp.ko 2024-03-28T15:37:24.407 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ecn.ko 2024-03-28T15:37:24.407 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_esp.ko 2024-03-28T15:37:24.407 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_hashlimit.ko 2024-03-28T15:37:24.407 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_helper.ko 2024-03-28T15:37:24.407 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_hl.ko 2024-03-28T15:37:24.407 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ipcomp.ko 2024-03-28T15:37:24.407 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_iprange.ko 2024-03-28T15:37:24.407 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ipvs.ko 2024-03-28T15:37:24.407 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_l2tp.ko 2024-03-28T15:37:24.408 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_length.ko 2024-03-28T15:37:24.408 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_limit.ko 2024-03-28T15:37:24.408 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_mac.ko 2024-03-28T15:37:24.408 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_mark.ko 2024-03-28T15:37:24.408 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_multiport.ko 2024-03-28T15:37:24.408 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_nat.ko 2024-03-28T15:37:24.408 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_nfacct.ko 2024-03-28T15:37:24.408 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_osf.ko 2024-03-28T15:37:24.408 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_owner.ko 2024-03-28T15:37:24.408 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_physdev.ko 2024-03-28T15:37:24.408 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_pkttype.ko 2024-03-28T15:37:24.408 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_policy.ko 2024-03-28T15:37:24.408 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_quota.ko 2024-03-28T15:37:24.408 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_rateest.ko 2024-03-28T15:37:24.408 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_realm.ko 2024-03-28T15:37:24.409 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_recent.ko 2024-03-28T15:37:24.409 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_sctp.ko 2024-03-28T15:37:24.409 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_set.ko 2024-03-28T15:37:24.409 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_socket.ko 2024-03-28T15:37:24.409 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_state.ko 2024-03-28T15:37:24.409 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_statistic.ko 2024-03-28T15:37:24.409 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_string.ko 2024-03-28T15:37:24.409 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_tcpmss.ko 2024-03-28T15:37:24.409 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_time.ko 2024-03-28T15:37:24.409 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_u32.ko 2024-03-28T15:37:24.409 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netlink 2024-03-28T15:37:24.409 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netlink/netlink_diag.ko 2024-03-28T15:37:24.409 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/nsh 2024-03-28T15:37:24.409 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/nsh/nsh.ko 2024-03-28T15:37:24.409 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch 2024-03-28T15:37:24.410 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/openvswitch.ko 2024-03-28T15:37:24.410 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-geneve.ko 2024-03-28T15:37:24.410 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-gre.ko 2024-03-28T15:37:24.410 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-vxlan.ko 2024-03-28T15:37:24.410 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/packet 2024-03-28T15:37:24.410 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/packet/af_packet_diag.ko 2024-03-28T15:37:24.410 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds 2024-03-28T15:37:24.410 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds.ko 2024-03-28T15:37:24.410 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds_rdma.ko 2024-03-28T15:37:24.410 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds_tcp.ko 2024-03-28T15:37:24.410 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched 2024-03-28T15:37:24.410 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_csum.ko 2024-03-28T15:37:24.410 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_gact.ko 2024-03-28T15:37:24.410 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_mirred.ko 2024-03-28T15:37:24.410 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_nat.ko 2024-03-28T15:37:24.411 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_pedit.ko 2024-03-28T15:37:24.411 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_police.ko 2024-03-28T15:37:24.411 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_simple.ko 2024-03-28T15:37:24.411 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_skbedit.ko 2024-03-28T15:37:24.411 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_vlan.ko 2024-03-28T15:37:24.411 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_basic.ko 2024-03-28T15:37:24.411 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_bpf.ko 2024-03-28T15:37:24.411 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_flow.ko 2024-03-28T15:37:24.411 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_fw.ko 2024-03-28T15:37:24.411 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_route.ko 2024-03-28T15:37:24.411 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_u32.ko 2024-03-28T15:37:24.411 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_cmp.ko 2024-03-28T15:37:24.411 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_ipset.ko 2024-03-28T15:37:24.411 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_meta.ko 2024-03-28T15:37:24.412 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_nbyte.ko 2024-03-28T15:37:24.412 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_text.ko 2024-03-28T15:37:24.412 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_u32.ko 2024-03-28T15:37:24.412 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_choke.ko 2024-03-28T15:37:24.412 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_codel.ko 2024-03-28T15:37:24.412 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_drr.ko 2024-03-28T15:37:24.412 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_fq.ko 2024-03-28T15:37:24.412 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_gred.ko 2024-03-28T15:37:24.412 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_hfsc.ko 2024-03-28T15:37:24.412 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_hhf.ko 2024-03-28T15:37:24.412 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_htb.ko 2024-03-28T15:37:24.412 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_ingress.ko 2024-03-28T15:37:24.412 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_mqprio.ko 2024-03-28T15:37:24.412 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_mqprio_lib.ko 2024-03-28T15:37:24.412 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_multiq.ko 2024-03-28T15:37:24.413 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_netem.ko 2024-03-28T15:37:24.413 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_pie.ko 2024-03-28T15:37:24.413 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_plug.ko 2024-03-28T15:37:24.413 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_prio.ko 2024-03-28T15:37:24.413 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_qfq.ko 2024-03-28T15:37:24.413 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_red.ko 2024-03-28T15:37:24.413 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_sfb.ko 2024-03-28T15:37:24.413 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_sfq.ko 2024-03-28T15:37:24.413 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_tbf.ko 2024-03-28T15:37:24.413 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_teql.ko 2024-03-28T15:37:24.413 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp 2024-03-28T15:37:24.413 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp/sctp.ko 2024-03-28T15:37:24.413 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp/sctp_diag.ko 2024-03-28T15:37:24.413 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc 2024-03-28T15:37:24.413 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss 2024-03-28T15:37:24.414 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-03-28T15:37:24.414 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-03-28T15:37:24.414 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/sunrpc.ko 2024-03-28T15:37:24.414 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/xprtrdma 2024-03-28T15:37:24.414 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-03-28T15:37:24.414 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc 2024-03-28T15:37:24.414 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc/diag.ko 2024-03-28T15:37:24.414 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc/tipc.ko 2024-03-28T15:37:24.414 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/unix 2024-03-28T15:37:24.414 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/unix/unix_diag.ko 2024-03-28T15:37:24.414 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock 2024-03-28T15:37:24.414 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-03-28T15:37:24.414 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-03-28T15:37:24.414 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock.ko 2024-03-28T15:37:24.414 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock_diag.ko 2024-03-28T15:37:24.415 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock_loopback.ko 2024-03-28T15:37:24.415 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/xfrm 2024-03-28T15:37:24.415 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/xfrm/xfrm_ipcomp.ko 2024-03-28T15:37:24.415 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security 2024-03-28T15:37:24.415 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys 2024-03-28T15:37:24.415 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/encrypted-keys 2024-03-28T15:37:24.415 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-03-28T15:37:24.415 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/trusted-keys 2024-03-28T15:37:24.415 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/trusted-keys/trusted.ko 2024-03-28T15:37:24.415 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/virt 2024-03-28T15:37:24.415 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/virt/lib 2024-03-28T15:37:24.415 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/virt/lib/irqbypass.ko 2024-03-28T15:37:24.415 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.alias 2024-03-28T15:37:24.415 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.alias.bin 2024-03-28T15:37:24.415 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin 2024-03-28T15:37:24.416 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin.alias.bin 2024-03-28T15:37:24.416 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin.bin 2024-03-28T15:37:24.416 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin.modinfo 2024-03-28T15:37:24.416 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.dep 2024-03-28T15:37:24.416 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.dep.bin 2024-03-28T15:37:24.416 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.devname 2024-03-28T15:37:24.416 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.order 2024-03-28T15:37:24.416 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.softdep 2024-03-28T15:37:24.416 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.symbols 2024-03-28T15:37:24.416 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.symbols.bin 2024-03-28T15:37:24.416 INFO:teuthology.orchestra.run.smithi150.stdout:/lib/modules/6.8.0-g1df0c7e8a235/vmlinuz 2024-03-28T15:37:24.420 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-g1df0c7e8a235 2024-03-28T15:37:24.420 DEBUG:teuthology.orchestra.run.smithi150:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-03-28T15:37:24.421 DEBUG:teuthology.orchestra.run.smithi067:> sudo shutdown -r now 2024-03-28T15:37:24.442 INFO:teuthology.orchestra.run.smithi150.stdout:postinstall scriptlet (using /bin/sh): 2024-03-28T15:37:24.442 INFO:teuthology.orchestra.run.smithi150.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-03-28T15:37:24.442 INFO:teuthology.orchestra.run.smithi150.stdout: /usr/bin/kernel-install add 6.8.0-g1df0c7e8a235 /lib/modules/6.8.0-g1df0c7e8a235/vmlinuz 2024-03-28T15:37:24.442 INFO:teuthology.orchestra.run.smithi150.stdout:fi 2024-03-28T15:37:24.442 INFO:teuthology.orchestra.run.smithi150.stdout:for file in vmlinuz System.map config; do 2024-03-28T15:37:24.442 INFO:teuthology.orchestra.run.smithi150.stdout: if ! cmp --silent "/lib/modules/6.8.0-g1df0c7e8a235/${file}" "/boot/${file}-6.8.0-g1df0c7e8a235"; then 2024-03-28T15:37:24.442 INFO:teuthology.orchestra.run.smithi150.stdout: cp "/lib/modules/6.8.0-g1df0c7e8a235/${file}" "/boot/${file}-6.8.0-g1df0c7e8a235" 2024-03-28T15:37:24.442 INFO:teuthology.orchestra.run.smithi150.stdout: fi 2024-03-28T15:37:24.442 INFO:teuthology.orchestra.run.smithi150.stdout:done 2024-03-28T15:37:24.442 INFO:teuthology.orchestra.run.smithi150.stdout:preuninstall scriptlet (using /bin/sh): 2024-03-28T15:37:24.443 INFO:teuthology.orchestra.run.smithi150.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-03-28T15:37:24.443 INFO:teuthology.orchestra.run.smithi150.stdout:new-kernel-pkg --remove 6.8.0-g1df0c7e8a235 --rminitrd --initrdfile=/boot/initramfs-6.8.0-g1df0c7e8a235.img 2024-03-28T15:37:24.443 INFO:teuthology.orchestra.run.smithi150.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-03-28T15:37:24.443 INFO:teuthology.orchestra.run.smithi150.stdout:kernel-install remove 6.8.0-g1df0c7e8a235 2024-03-28T15:37:24.443 INFO:teuthology.orchestra.run.smithi150.stdout:fi 2024-03-28T15:37:24.443 INFO:teuthology.orchestra.run.smithi150.stdout:postuninstall scriptlet (using /bin/sh): 2024-03-28T15:37:24.443 INFO:teuthology.orchestra.run.smithi150.stdout:if [ -x /sbin/update-bootloader ]; then 2024-03-28T15:37:24.443 INFO:teuthology.orchestra.run.smithi150.stdout:/sbin/update-bootloader --remove 6.8.0-g1df0c7e8a235 2024-03-28T15:37:24.443 INFO:teuthology.orchestra.run.smithi150.stdout:fi 2024-03-28T15:37:24.444 DEBUG:teuthology.orchestra.run.smithi150:> sudo rpm -qi grub2-tools 2024-03-28T15:37:24.517 INFO:teuthology.orchestra.run.smithi150.stdout:Name : grub2-tools 2024-03-28T15:37:24.518 INFO:teuthology.orchestra.run.smithi150.stdout:Epoch : 1 2024-03-28T15:37:24.518 INFO:teuthology.orchestra.run.smithi150.stdout:Version : 2.06 2024-03-28T15:37:24.518 INFO:teuthology.orchestra.run.smithi150.stdout:Release : 68.el9 2024-03-28T15:37:24.518 INFO:teuthology.orchestra.run.smithi150.stdout:Architecture: x86_64 2024-03-28T15:37:24.518 INFO:teuthology.orchestra.run.smithi150.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-28T15:37:24.518 INFO:teuthology.orchestra.run.smithi150.stdout:Group : Unspecified 2024-03-28T15:37:24.518 INFO:teuthology.orchestra.run.smithi150.stdout:Size : 8264254 2024-03-28T15:37:24.518 INFO:teuthology.orchestra.run.smithi150.stdout:License : GPLv3+ 2024-03-28T15:37:24.518 INFO:teuthology.orchestra.run.smithi150.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-28T15:37:24.518 INFO:teuthology.orchestra.run.smithi150.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-28T15:37:24.518 INFO:teuthology.orchestra.run.smithi150.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-28T15:37:24.518 INFO:teuthology.orchestra.run.smithi150.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-28T15:37:24.518 INFO:teuthology.orchestra.run.smithi150.stdout:Packager : builder@centos.org 2024-03-28T15:37:24.519 INFO:teuthology.orchestra.run.smithi150.stdout:Vendor : CentOS 2024-03-28T15:37:24.519 INFO:teuthology.orchestra.run.smithi150.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-28T15:37:24.519 INFO:teuthology.orchestra.run.smithi150.stdout:Summary : Support tools for GRUB. 2024-03-28T15:37:24.519 INFO:teuthology.orchestra.run.smithi150.stdout:Description : 2024-03-28T15:37:24.519 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-28T15:37:24.519 INFO:teuthology.orchestra.run.smithi150.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-28T15:37:24.519 INFO:teuthology.orchestra.run.smithi150.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-28T15:37:24.519 INFO:teuthology.orchestra.run.smithi150.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-28T15:37:24.519 INFO:teuthology.orchestra.run.smithi150.stdout:hardware devices. 2024-03-28T15:37:24.519 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-28T15:37:24.519 INFO:teuthology.orchestra.run.smithi150.stdout:This subpackage provides tools for support of all platforms. 2024-03-28T15:37:24.520 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-28T15:37:24.520 INFO:teuthology.task.kernel:Updating grub on smithi150 to boot 6.8.0-g1df0c7e8a235 2024-03-28T15:37:24.520 DEBUG:teuthology.orchestra.run.smithi150:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-28T15:37:25.162 INFO:teuthology.orchestra.run.smithi150.stderr:Generating grub configuration file ... 2024-03-28T15:37:26.545 INFO:teuthology.orchestra.run.smithi150.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-28T15:37:26.574 INFO:teuthology.orchestra.run.smithi150.stderr:done 2024-03-28T15:37:26.578 DEBUG:teuthology.orchestra.run.smithi150:> mktemp 2024-03-28T15:37:26.594 INFO:teuthology.orchestra.run.smithi150.stdout:/tmp/tmp.xbjUQ07JQv 2024-03-28T15:37:26.594 DEBUG:teuthology.orchestra.run.smithi150:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.xbjUQ07JQv 2024-03-28T15:37:26.663 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 0666 /tmp/tmp.xbjUQ07JQv 2024-03-28T15:37:26.801 DEBUG:teuthology.orchestra.remote:smithi150:/tmp/tmp.xbjUQ07JQv is 6KB 2024-03-28T15:37:26.848 DEBUG:teuthology.orchestra.run.smithi150:> rm -fr /tmp/tmp.xbjUQ07JQv 2024-03-28T15:37:26.862 DEBUG:teuthology.orchestra.run.smithi150:> sudo /bin/ls /boot/loader/entries || true 2024-03-28T15:37:26.927 INFO:teuthology.orchestra.run.smithi150.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-28T15:37:26.927 INFO:teuthology.orchestra.run.smithi150.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-28T15:37:26.927 INFO:teuthology.orchestra.run.smithi150.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-28T15:37:26.927 INFO:teuthology.orchestra.run.smithi150.stdout:15949e3c88704300882482eddfd7baae-6.8.0-g1df0c7e8a235.conf 2024-03-28T15:37:26.929 DEBUG:teuthology.orchestra.run.smithi150:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.8.0-g1df0c7e8a235 2024-03-28T15:37:27.094 DEBUG:teuthology.orchestra.run.smithi150:> sudo shutdown -r now 2024-03-28T15:37:41.529 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi183.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-03-28T15:37:41.530 DEBUG:teuthology.orchestra.run.smithi183:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-03-28T15:37:41.600 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235 2024-03-28T15:37:41.601 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/System.map 2024-03-28T15:37:41.601 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/config 2024-03-28T15:37:41.601 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel 2024-03-28T15:37:41.601 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch 2024-03-28T15:37:41.601 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86 2024-03-28T15:37:41.601 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto 2024-03-28T15:37:41.601 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-03-28T15:37:41.601 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-03-28T15:37:41.601 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-03-28T15:37:41.601 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-03-28T15:37:41.601 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-03-28T15:37:41.601 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-03-28T15:37:41.601 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-03-28T15:37:41.602 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/crc32c-intel.ko 2024-03-28T15:37:41.602 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-03-28T15:37:41.602 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-03-28T15:37:41.602 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-03-28T15:37:41.602 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-03-28T15:37:41.602 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-avx2.ko 2024-03-28T15:37:41.602 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-03-28T15:37:41.602 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-03-28T15:37:41.602 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-03-28T15:37:41.602 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-03-28T15:37:41.602 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-03-28T15:37:41.602 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-03-28T15:37:41.602 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-03-28T15:37:41.602 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel 2024-03-28T15:37:41.603 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu 2024-03-28T15:37:41.603 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu/mce 2024-03-28T15:37:41.603 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-03-28T15:37:41.603 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm 2024-03-28T15:37:41.603 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm-amd.ko 2024-03-28T15:37:41.603 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm-intel.ko 2024-03-28T15:37:41.603 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm.ko 2024-03-28T15:37:41.603 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto 2024-03-28T15:37:41.603 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/ansi_cprng.ko 2024-03-28T15:37:41.603 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/anubis.ko 2024-03-28T15:37:41.603 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/arc4.ko 2024-03-28T15:37:41.603 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx 2024-03-28T15:37:41.603 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_memcpy.ko 2024-03-28T15:37:41.603 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_pq.ko 2024-03-28T15:37:41.604 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_raid6_recov.ko 2024-03-28T15:37:41.604 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_tx.ko 2024-03-28T15:37:41.604 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_xor.ko 2024-03-28T15:37:41.604 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/raid6test.ko 2024-03-28T15:37:41.604 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/authenc.ko 2024-03-28T15:37:41.604 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/authencesn.ko 2024-03-28T15:37:41.604 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blake2b_generic.ko 2024-03-28T15:37:41.604 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blowfish_common.ko 2024-03-28T15:37:41.604 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blowfish_generic.ko 2024-03-28T15:37:41.604 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/camellia_generic.ko 2024-03-28T15:37:41.604 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast5_generic.ko 2024-03-28T15:37:41.604 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast6_generic.ko 2024-03-28T15:37:41.604 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast_common.ko 2024-03-28T15:37:41.604 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/ccm.ko 2024-03-28T15:37:41.604 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cmac.ko 2024-03-28T15:37:41.605 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crc32_generic.ko 2024-03-28T15:37:41.605 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crypto_engine.ko 2024-03-28T15:37:41.605 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crypto_user.ko 2024-03-28T15:37:41.605 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cts.ko 2024-03-28T15:37:41.605 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/des_generic.ko 2024-03-28T15:37:41.605 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/dh_generic.ko 2024-03-28T15:37:41.605 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/echainiv.ko 2024-03-28T15:37:41.605 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/essiv.ko 2024-03-28T15:37:41.605 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/fcrypt.ko 2024-03-28T15:37:41.605 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/khazad.ko 2024-03-28T15:37:41.605 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/lz4.ko 2024-03-28T15:37:41.605 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/lz4hc.ko 2024-03-28T15:37:41.605 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/md4.ko 2024-03-28T15:37:41.605 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/michael_mic.ko 2024-03-28T15:37:41.605 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/pcbc.ko 2024-03-28T15:37:41.606 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/pcrypt.ko 2024-03-28T15:37:41.606 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/rmd160.ko 2024-03-28T15:37:41.606 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/seed.ko 2024-03-28T15:37:41.606 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/serpent_generic.ko 2024-03-28T15:37:41.606 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/tcrypt.ko 2024-03-28T15:37:41.606 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/tea.ko 2024-03-28T15:37:41.606 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/twofish_common.ko 2024-03-28T15:37:41.606 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/twofish_generic.ko 2024-03-28T15:37:41.606 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/vmac.ko 2024-03-28T15:37:41.606 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/wp512.ko 2024-03-28T15:37:41.606 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xcbc.ko 2024-03-28T15:37:41.606 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xor.ko 2024-03-28T15:37:41.606 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xxhash_generic.ko 2024-03-28T15:37:41.606 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers 2024-03-28T15:37:41.606 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi 2024-03-28T15:37:41.607 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/acpi_ipmi.ko 2024-03-28T15:37:41.607 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/acpi_pad.ko 2024-03-28T15:37:41.607 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/custom_method.ko 2024-03-28T15:37:41.607 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/ec_sys.ko 2024-03-28T15:37:41.607 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/platform_profile.ko 2024-03-28T15:37:41.607 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/sbs.ko 2024-03-28T15:37:41.607 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/sbshc.ko 2024-03-28T15:37:41.607 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/video.ko 2024-03-28T15:37:41.607 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata 2024-03-28T15:37:41.607 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/acard-ahci.ko 2024-03-28T15:37:41.607 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/ahci_platform.ko 2024-03-28T15:37:41.607 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/ata_generic.ko 2024-03-28T15:37:41.607 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/libahci_platform.ko 2024-03-28T15:37:41.607 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_acpi.ko 2024-03-28T15:37:41.607 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ali.ko 2024-03-28T15:37:41.608 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_amd.ko 2024-03-28T15:37:41.608 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_artop.ko 2024-03-28T15:37:41.608 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_atiixp.ko 2024-03-28T15:37:41.608 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_atp867x.ko 2024-03-28T15:37:41.608 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cmd640.ko 2024-03-28T15:37:41.609 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cmd64x.ko 2024-03-28T15:37:41.609 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cypress.ko 2024-03-28T15:37:41.609 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_efar.ko 2024-03-28T15:37:41.609 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt366.ko 2024-03-28T15:37:41.609 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt37x.ko 2024-03-28T15:37:41.609 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt3x2n.ko 2024-03-28T15:37:41.609 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt3x3.ko 2024-03-28T15:37:41.609 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_it8213.ko 2024-03-28T15:37:41.609 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_it821x.ko 2024-03-28T15:37:41.609 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_jmicron.ko 2024-03-28T15:37:41.609 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_marvell.ko 2024-03-28T15:37:41.609 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_mpiix.ko 2024-03-28T15:37:41.610 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_netcell.ko 2024-03-28T15:37:41.610 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ninja32.ko 2024-03-28T15:37:41.610 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ns87410.ko 2024-03-28T15:37:41.610 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ns87415.ko 2024-03-28T15:37:41.610 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_oldpiix.ko 2024-03-28T15:37:41.610 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_opti.ko 2024-03-28T15:37:41.610 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_optidma.ko 2024-03-28T15:37:41.610 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pcmcia.ko 2024-03-28T15:37:41.610 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pdc2027x.ko 2024-03-28T15:37:41.610 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-03-28T15:37:41.610 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_piccolo.ko 2024-03-28T15:37:41.610 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_rdc.ko 2024-03-28T15:37:41.610 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sch.ko 2024-03-28T15:37:41.610 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_serverworks.ko 2024-03-28T15:37:41.610 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sil680.ko 2024-03-28T15:37:41.611 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sis.ko 2024-03-28T15:37:41.611 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sl82c105.ko 2024-03-28T15:37:41.611 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_triflex.ko 2024-03-28T15:37:41.611 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_via.ko 2024-03-28T15:37:41.611 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pdc_adma.ko 2024-03-28T15:37:41.611 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_inic162x.ko 2024-03-28T15:37:41.611 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_mv.ko 2024-03-28T15:37:41.611 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_nv.ko 2024-03-28T15:37:41.611 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_promise.ko 2024-03-28T15:37:41.611 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_qstor.ko 2024-03-28T15:37:41.611 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sil.ko 2024-03-28T15:37:41.611 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sil24.ko 2024-03-28T15:37:41.611 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sis.ko 2024-03-28T15:37:41.611 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_svw.ko 2024-03-28T15:37:41.611 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sx4.ko 2024-03-28T15:37:41.612 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_uli.ko 2024-03-28T15:37:41.612 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_via.ko 2024-03-28T15:37:41.612 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_vsc.ko 2024-03-28T15:37:41.612 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/auxdisplay 2024-03-28T15:37:41.612 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/auxdisplay/ks0108.ko 2024-03-28T15:37:41.612 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base 2024-03-28T15:37:41.612 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base/regmap 2024-03-28T15:37:41.612 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base/regmap/regmap-i2c.ko 2024-03-28T15:37:41.612 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base/regmap/regmap-mmio.ko 2024-03-28T15:37:41.612 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/bcma 2024-03-28T15:37:41.612 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/bcma/bcma.ko 2024-03-28T15:37:41.612 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block 2024-03-28T15:37:41.612 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/aoe 2024-03-28T15:37:41.612 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/aoe/aoe.ko 2024-03-28T15:37:41.612 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/brd.ko 2024-03-28T15:37:41.613 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/drbd 2024-03-28T15:37:41.613 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/drbd/drbd.ko 2024-03-28T15:37:41.613 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/floppy.ko 2024-03-28T15:37:41.613 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/loop.ko 2024-03-28T15:37:41.613 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/mtip32xx 2024-03-28T15:37:41.613 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-03-28T15:37:41.613 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/nbd.ko 2024-03-28T15:37:41.613 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/null_blk 2024-03-28T15:37:41.613 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/null_blk/null_blk.ko 2024-03-28T15:37:41.613 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/pktcdvd.ko 2024-03-28T15:37:41.613 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/rbd.ko 2024-03-28T15:37:41.613 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/virtio_blk.ko 2024-03-28T15:37:41.613 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/xen-blkback 2024-03-28T15:37:41.613 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-03-28T15:37:41.613 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/xen-blkfront.ko 2024-03-28T15:37:41.614 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/zram 2024-03-28T15:37:41.614 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/zram/zram.ko 2024-03-28T15:37:41.614 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char 2024-03-28T15:37:41.614 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hangcheck-timer.ko 2024-03-28T15:37:41.614 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random 2024-03-28T15:37:41.614 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/amd-rng.ko 2024-03-28T15:37:41.614 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/intel-rng.ko 2024-03-28T15:37:41.614 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-03-28T15:37:41.614 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/via-rng.ko 2024-03-28T15:37:41.614 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/virtio-rng.ko 2024-03-28T15:37:41.614 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi 2024-03-28T15:37:41.614 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-03-28T15:37:41.614 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-03-28T15:37:41.614 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-03-28T15:37:41.614 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_si.ko 2024-03-28T15:37:41.615 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-03-28T15:37:41.615 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-03-28T15:37:41.615 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/lp.ko 2024-03-28T15:37:41.615 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/mwave 2024-03-28T15:37:41.615 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/mwave/mwave.ko 2024-03-28T15:37:41.615 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ppdev.ko 2024-03-28T15:37:41.615 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tlclk.ko 2024-03-28T15:37:41.615 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm 2024-03-28T15:37:41.615 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm.ko 2024-03-28T15:37:41.615 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_atmel.ko 2024-03-28T15:37:41.615 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_infineon.ko 2024-03-28T15:37:41.615 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_nsc.ko 2024-03-28T15:37:41.615 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_tis.ko 2024-03-28T15:37:41.615 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-03-28T15:37:41.615 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/uv_mmtimer.ko 2024-03-28T15:37:41.616 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/virtio_console.ko 2024-03-28T15:37:41.616 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq 2024-03-28T15:37:41.616 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-03-28T15:37:41.616 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-03-28T15:37:41.616 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/p4-clockmod.ko 2024-03-28T15:37:41.616 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-03-28T15:37:41.616 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/powernow-k8.ko 2024-03-28T15:37:41.616 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/speedstep-lib.ko 2024-03-28T15:37:41.616 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto 2024-03-28T15:37:41.616 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp 2024-03-28T15:37:41.616 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-03-28T15:37:41.616 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp/ccp.ko 2024-03-28T15:37:41.616 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel 2024-03-28T15:37:41.616 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat 2024-03-28T15:37:41.616 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_common 2024-03-28T15:37:41.617 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-03-28T15:37:41.617 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-03-28T15:37:41.617 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-03-28T15:37:41.617 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/padlock-aes.ko 2024-03-28T15:37:41.617 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/padlock-sha.ko 2024-03-28T15:37:41.617 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/virtio 2024-03-28T15:37:41.617 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-03-28T15:37:41.617 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dca 2024-03-28T15:37:41.617 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dca/dca.ko 2024-03-28T15:37:41.617 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/devfreq 2024-03-28T15:37:41.617 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-03-28T15:37:41.617 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma 2024-03-28T15:37:41.617 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/dw 2024-03-28T15:37:41.617 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/dw/dw_dmac.ko 2024-03-28T15:37:41.617 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/ioat 2024-03-28T15:37:41.617 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/ioat/ioatdma.ko 2024-03-28T15:37:41.618 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac 2024-03-28T15:37:41.618 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/amd64_edac.ko 2024-03-28T15:37:41.618 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/e752x_edac.ko 2024-03-28T15:37:41.618 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/edac_mce_amd.ko 2024-03-28T15:37:41.618 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i3000_edac.ko 2024-03-28T15:37:41.618 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i3200_edac.ko 2024-03-28T15:37:41.618 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i5100_edac.ko 2024-03-28T15:37:41.618 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i5400_edac.ko 2024-03-28T15:37:41.618 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i7300_edac.ko 2024-03-28T15:37:41.618 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i7core_edac.ko 2024-03-28T15:37:41.618 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i82975x_edac.ko 2024-03-28T15:37:41.618 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/ie31200_edac.ko 2024-03-28T15:37:41.618 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/sb_edac.ko 2024-03-28T15:37:41.618 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/x38_edac.ko 2024-03-28T15:37:41.619 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware 2024-03-28T15:37:41.619 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware/edd.ko 2024-03-28T15:37:41.619 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware/iscsi_ibft.ko 2024-03-28T15:37:41.619 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio 2024-03-28T15:37:41.619 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-ich.ko 2024-03-28T15:37:41.619 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-viperboard.ko 2024-03-28T15:37:41.619 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid 2024-03-28T15:37:41.619 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-appleir.ko 2024-03-28T15:37:41.619 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-aureal.ko 2024-03-28T15:37:41.619 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-axff.ko 2024-03-28T15:37:41.619 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-dr.ko 2024-03-28T15:37:41.619 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-elecom.ko 2024-03-28T15:37:41.619 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-elo.ko 2024-03-28T15:37:41.619 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-emsff.ko 2024-03-28T15:37:41.620 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gaff.ko 2024-03-28T15:37:41.620 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gt683r.ko 2024-03-28T15:37:41.620 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gyration.ko 2024-03-28T15:37:41.620 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtek-kbd.ko 2024-03-28T15:37:41.620 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtek-mouse.ko 2024-03-28T15:37:41.620 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtekff.ko 2024-03-28T15:37:41.620 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-hyperv.ko 2024-03-28T15:37:41.620 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-icade.ko 2024-03-28T15:37:41.620 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-keytouch.ko 2024-03-28T15:37:41.620 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-kye.ko 2024-03-28T15:37:41.620 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-lcpower.ko 2024-03-28T15:37:41.620 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-led.ko 2024-03-28T15:37:41.620 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-lenovo.ko 2024-03-28T15:37:41.620 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-logitech-dj.ko 2024-03-28T15:37:41.620 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-03-28T15:37:41.620 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-multitouch.ko 2024-03-28T15:37:41.621 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-ortek.ko 2024-03-28T15:37:41.621 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-penmount.ko 2024-03-28T15:37:41.621 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-petalynx.ko 2024-03-28T15:37:41.621 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-picolcd.ko 2024-03-28T15:37:41.621 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-pl.ko 2024-03-28T15:37:41.621 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-plantronics.ko 2024-03-28T15:37:41.621 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-primax.ko 2024-03-28T15:37:41.621 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-rmi.ko 2024-03-28T15:37:41.621 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-arvo.ko 2024-03-28T15:37:41.621 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-common.ko 2024-03-28T15:37:41.621 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-isku.ko 2024-03-28T15:37:41.621 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-kone.ko 2024-03-28T15:37:41.621 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-03-28T15:37:41.621 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-konepure.ko 2024-03-28T15:37:41.621 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-03-28T15:37:41.622 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-lua.ko 2024-03-28T15:37:41.622 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-pyra.ko 2024-03-28T15:37:41.622 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-ryos.ko 2024-03-28T15:37:41.622 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-savu.ko 2024-03-28T15:37:41.622 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat.ko 2024-03-28T15:37:41.622 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-saitek.ko 2024-03-28T15:37:41.622 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-samsung.ko 2024-03-28T15:37:41.622 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sensor-hub.ko 2024-03-28T15:37:41.622 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sjoy.ko 2024-03-28T15:37:41.622 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sony.ko 2024-03-28T15:37:41.622 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-speedlink.ko 2024-03-28T15:37:41.622 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-steelseries.ko 2024-03-28T15:37:41.622 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sunplus.ko 2024-03-28T15:37:41.622 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-thrustmaster.ko 2024-03-28T15:37:41.622 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-tivo.ko 2024-03-28T15:37:41.623 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-tmff.ko 2024-03-28T15:37:41.623 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-topseed.ko 2024-03-28T15:37:41.623 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-twinhan.ko 2024-03-28T15:37:41.623 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-uclogic.ko 2024-03-28T15:37:41.623 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-waltop.ko 2024-03-28T15:37:41.623 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-wiimote.ko 2024-03-28T15:37:41.623 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-xinmo.ko 2024-03-28T15:37:41.623 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-zpff.ko 2024-03-28T15:37:41.623 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-zydacron.ko 2024-03-28T15:37:41.623 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/uhid.ko 2024-03-28T15:37:41.623 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/wacom.ko 2024-03-28T15:37:41.623 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv 2024-03-28T15:37:41.623 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_balloon.ko 2024-03-28T15:37:41.623 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_utils.ko 2024-03-28T15:37:41.623 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_vmbus.ko 2024-03-28T15:37:41.624 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon 2024-03-28T15:37:41.624 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/abituguru.ko 2024-03-28T15:37:41.624 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/abituguru3.ko 2024-03-28T15:37:41.624 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/acpi_power_meter.ko 2024-03-28T15:37:41.624 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ad7414.ko 2024-03-28T15:37:41.624 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ad7418.ko 2024-03-28T15:37:41.624 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adc128d818.ko 2024-03-28T15:37:41.624 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1025.ko 2024-03-28T15:37:41.624 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1026.ko 2024-03-28T15:37:41.624 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1029.ko 2024-03-28T15:37:41.624 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1031.ko 2024-03-28T15:37:41.624 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm9240.ko 2024-03-28T15:37:41.624 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ads7828.ko 2024-03-28T15:37:41.624 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7410.ko 2024-03-28T15:37:41.624 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7411.ko 2024-03-28T15:37:41.624 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7462.ko 2024-03-28T15:37:41.625 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7470.ko 2024-03-28T15:37:41.625 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7475.ko 2024-03-28T15:37:41.625 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7x10.ko 2024-03-28T15:37:41.625 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/amc6821.ko 2024-03-28T15:37:41.625 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/applesmc.ko 2024-03-28T15:37:41.625 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asb100.ko 2024-03-28T15:37:41.625 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asc7621.ko 2024-03-28T15:37:41.625 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asus_atk0110.ko 2024-03-28T15:37:41.625 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/atxp1.ko 2024-03-28T15:37:41.625 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/coretemp.ko 2024-03-28T15:37:41.625 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-03-28T15:37:41.625 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/dme1737.ko 2024-03-28T15:37:41.625 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ds1621.ko 2024-03-28T15:37:41.625 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ds620.ko 2024-03-28T15:37:41.625 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/emc1403.ko 2024-03-28T15:37:41.626 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/emc6w201.ko 2024-03-28T15:37:41.626 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f71805f.ko 2024-03-28T15:37:41.626 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f71882fg.ko 2024-03-28T15:37:41.626 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f75375s.ko 2024-03-28T15:37:41.626 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/fam15h_power.ko 2024-03-28T15:37:41.626 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/fschmd.ko 2024-03-28T15:37:41.626 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/g760a.ko 2024-03-28T15:37:41.626 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/g762.ko 2024-03-28T15:37:41.626 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/gl518sm.ko 2024-03-28T15:37:41.626 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/gl520sm.ko 2024-03-28T15:37:41.626 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/hwmon-vid.ko 2024-03-28T15:37:41.626 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/i5500_temp.ko 2024-03-28T15:37:41.626 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/i5k_amb.ko 2024-03-28T15:37:41.626 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ibmaem.ko 2024-03-28T15:37:41.626 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ibmpex.ko 2024-03-28T15:37:41.627 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ina209.ko 2024-03-28T15:37:41.627 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ina2xx.ko 2024-03-28T15:37:41.627 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/it87.ko 2024-03-28T15:37:41.627 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/k10temp.ko 2024-03-28T15:37:41.627 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/k8temp.ko 2024-03-28T15:37:41.627 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lineage-pem.ko 2024-03-28T15:37:41.627 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm63.ko 2024-03-28T15:37:41.627 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm73.ko 2024-03-28T15:37:41.627 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm75.ko 2024-03-28T15:37:41.627 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm77.ko 2024-03-28T15:37:41.627 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm78.ko 2024-03-28T15:37:41.627 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm80.ko 2024-03-28T15:37:41.627 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm83.ko 2024-03-28T15:37:41.627 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm85.ko 2024-03-28T15:37:41.627 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm87.ko 2024-03-28T15:37:41.628 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm90.ko 2024-03-28T15:37:41.628 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm92.ko 2024-03-28T15:37:41.628 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm93.ko 2024-03-28T15:37:41.628 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95234.ko 2024-03-28T15:37:41.628 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95241.ko 2024-03-28T15:37:41.628 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95245.ko 2024-03-28T15:37:41.628 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc2945.ko 2024-03-28T15:37:41.628 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4151.ko 2024-03-28T15:37:41.628 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4215.ko 2024-03-28T15:37:41.628 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4222.ko 2024-03-28T15:37:41.628 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4245.ko 2024-03-28T15:37:41.628 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4260.ko 2024-03-28T15:37:41.628 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4261.ko 2024-03-28T15:37:41.628 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max16065.ko 2024-03-28T15:37:41.628 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max1619.ko 2024-03-28T15:37:41.629 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max1668.ko 2024-03-28T15:37:41.629 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max197.ko 2024-03-28T15:37:41.629 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6639.ko 2024-03-28T15:37:41.629 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6650.ko 2024-03-28T15:37:41.629 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6697.ko 2024-03-28T15:37:41.629 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/mcp3021.ko 2024-03-28T15:37:41.629 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6683.ko 2024-03-28T15:37:41.629 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6775-core.ko 2024-03-28T15:37:41.629 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6775.ko 2024-03-28T15:37:41.629 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct7802.ko 2024-03-28T15:37:41.629 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pc87360.ko 2024-03-28T15:37:41.629 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pc87427.ko 2024-03-28T15:37:41.629 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pcf8591.ko 2024-03-28T15:37:41.629 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus 2024-03-28T15:37:41.629 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-03-28T15:37:41.630 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-03-28T15:37:41.630 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-03-28T15:37:41.630 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max16064.ko 2024-03-28T15:37:41.630 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max34440.ko 2024-03-28T15:37:41.630 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max8688.ko 2024-03-28T15:37:41.630 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-03-28T15:37:41.630 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-03-28T15:37:41.630 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-03-28T15:37:41.630 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-03-28T15:37:41.630 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-03-28T15:37:41.630 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-03-28T15:37:41.630 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/powr1220.ko 2024-03-28T15:37:41.630 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch5627.ko 2024-03-28T15:37:41.630 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch5636.ko 2024-03-28T15:37:41.630 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch56xx-common.ko 2024-03-28T15:37:41.631 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sht15.ko 2024-03-28T15:37:41.631 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sht21.ko 2024-03-28T15:37:41.631 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/shtc1.ko 2024-03-28T15:37:41.631 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sis5595.ko 2024-03-28T15:37:41.631 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47b397.ko 2024-03-28T15:37:41.631 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47m1.ko 2024-03-28T15:37:41.631 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47m192.ko 2024-03-28T15:37:41.631 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/thmc50.ko 2024-03-28T15:37:41.631 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp102.ko 2024-03-28T15:37:41.631 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp103.ko 2024-03-28T15:37:41.631 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp401.ko 2024-03-28T15:37:41.631 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp421.ko 2024-03-28T15:37:41.631 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/via-cputemp.ko 2024-03-28T15:37:41.631 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/via686a.ko 2024-03-28T15:37:41.631 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/vt1211.ko 2024-03-28T15:37:41.632 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/vt8231.ko 2024-03-28T15:37:41.632 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83627ehf.ko 2024-03-28T15:37:41.632 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83627hf.ko 2024-03-28T15:37:41.632 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83781d.ko 2024-03-28T15:37:41.632 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83791d.ko 2024-03-28T15:37:41.632 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83792d.ko 2024-03-28T15:37:41.632 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83793.ko 2024-03-28T15:37:41.632 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83795.ko 2024-03-28T15:37:41.632 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83l785ts.ko 2024-03-28T15:37:41.632 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83l786ng.ko 2024-03-28T15:37:41.632 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c 2024-03-28T15:37:41.632 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos 2024-03-28T15:37:41.632 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-03-28T15:37:41.632 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-03-28T15:37:41.632 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses 2024-03-28T15:37:41.633 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-03-28T15:37:41.633 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-03-28T15:37:41.633 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-03-28T15:37:41.633 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-03-28T15:37:41.633 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-03-28T15:37:41.633 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-03-28T15:37:41.633 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-03-28T15:37:41.633 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-03-28T15:37:41.633 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-i801.ko 2024-03-28T15:37:41.633 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-isch.ko 2024-03-28T15:37:41.633 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-03-28T15:37:41.633 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-03-28T15:37:41.633 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-03-28T15:37:41.633 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-parport.ko 2024-03-28T15:37:41.633 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-03-28T15:37:41.633 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-03-28T15:37:41.634 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-03-28T15:37:41.634 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-03-28T15:37:41.634 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-03-28T15:37:41.634 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-03-28T15:37:41.634 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-via.ko 2024-03-28T15:37:41.634 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-03-28T15:37:41.634 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-03-28T15:37:41.634 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-dev.ko 2024-03-28T15:37:41.634 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-mux.ko 2024-03-28T15:37:41.634 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-03-28T15:37:41.634 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-smbus.ko 2024-03-28T15:37:41.634 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-stub.ko 2024-03-28T15:37:41.634 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband 2024-03-28T15:37:41.635 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core 2024-03-28T15:37:41.635 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_cm.ko 2024-03-28T15:37:41.635 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_core.ko 2024-03-28T15:37:41.635 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_umad.ko 2024-03-28T15:37:41.635 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-03-28T15:37:41.635 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/iw_cm.ko 2024-03-28T15:37:41.635 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/rdma_cm.ko 2024-03-28T15:37:41.635 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-03-28T15:37:41.635 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw 2024-03-28T15:37:41.635 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/cxgb4 2024-03-28T15:37:41.635 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-03-28T15:37:41.635 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx4 2024-03-28T15:37:41.635 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-03-28T15:37:41.635 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx5 2024-03-28T15:37:41.636 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-03-28T15:37:41.636 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mthca 2024-03-28T15:37:41.636 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-03-28T15:37:41.636 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp 2024-03-28T15:37:41.636 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/ipoib 2024-03-28T15:37:41.636 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-03-28T15:37:41.636 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/iser 2024-03-28T15:37:41.636 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-03-28T15:37:41.636 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/isert 2024-03-28T15:37:41.636 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-03-28T15:37:41.636 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srp 2024-03-28T15:37:41.636 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-03-28T15:37:41.636 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srpt 2024-03-28T15:37:41.637 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-03-28T15:37:41.637 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input 2024-03-28T15:37:41.637 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport 2024-03-28T15:37:41.637 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-03-28T15:37:41.637 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/fm801-gp.ko 2024-03-28T15:37:41.637 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/gameport.ko 2024-03-28T15:37:41.637 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/joydev.ko 2024-03-28T15:37:41.637 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard 2024-03-28T15:37:41.637 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/gpio_keys.ko 2024-03-28T15:37:41.637 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse 2024-03-28T15:37:41.637 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/appletouch.ko 2024-03-28T15:37:41.637 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/bcm5974.ko 2024-03-28T15:37:41.637 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/cyapatp.ko 2024-03-28T15:37:41.637 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/elan_i2c.ko 2024-03-28T15:37:41.637 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/sermouse.ko 2024-03-28T15:37:41.638 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-03-28T15:37:41.638 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/synaptics_usb.ko 2024-03-28T15:37:41.638 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/vsxxxaa.ko 2024-03-28T15:37:41.638 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/rmi4 2024-03-28T15:37:41.638 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/rmi4/rmi_core.ko 2024-03-28T15:37:41.638 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio 2024-03-28T15:37:41.638 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/altera_ps2.ko 2024-03-28T15:37:41.638 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/arc_ps2.ko 2024-03-28T15:37:41.638 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-03-28T15:37:41.638 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/serio_raw.ko 2024-03-28T15:37:41.638 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/sparse-keymap.ko 2024-03-28T15:37:41.638 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds 2024-03-28T15:37:41.638 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-blinkm.ko 2024-03-28T15:37:41.638 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-lm3530.ko 2024-03-28T15:37:41.639 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-lp3944.ko 2024-03-28T15:37:41.639 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-ss4200.ko 2024-03-28T15:37:41.639 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger 2024-03-28T15:37:41.639 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-03-28T15:37:41.639 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-03-28T15:37:41.639 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-03-28T15:37:41.639 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-03-28T15:37:41.639 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-03-28T15:37:41.639 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-03-28T15:37:41.639 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-03-28T15:37:41.639 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-03-28T15:37:41.639 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-03-28T15:37:41.639 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md 2024-03-28T15:37:41.639 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/bcache 2024-03-28T15:37:41.640 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/bcache/bcache.ko 2024-03-28T15:37:41.640 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-bio-prison.ko 2024-03-28T15:37:41.640 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-cache-smq.ko 2024-03-28T15:37:41.640 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-cache.ko 2024-03-28T15:37:41.640 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-crypt.ko 2024-03-28T15:37:41.640 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-delay.ko 2024-03-28T15:37:41.640 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-flakey.ko 2024-03-28T15:37:41.640 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-log-userspace.ko 2024-03-28T15:37:41.640 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-multipath.ko 2024-03-28T15:37:41.640 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-queue-length.ko 2024-03-28T15:37:41.640 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-raid.ko 2024-03-28T15:37:41.640 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-round-robin.ko 2024-03-28T15:37:41.640 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-service-time.ko 2024-03-28T15:37:41.640 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-switch.ko 2024-03-28T15:37:41.640 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-thin-pool.ko 2024-03-28T15:37:41.641 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-verity.ko 2024-03-28T15:37:41.641 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/persistent-data 2024-03-28T15:37:41.641 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-03-28T15:37:41.641 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid0.ko 2024-03-28T15:37:41.641 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid1.ko 2024-03-28T15:37:41.641 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid10.ko 2024-03-28T15:37:41.641 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid456.ko 2024-03-28T15:37:41.641 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message 2024-03-28T15:37:41.641 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion 2024-03-28T15:37:41.641 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptbase.ko 2024-03-28T15:37:41.641 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptctl.ko 2024-03-28T15:37:41.641 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptfc.ko 2024-03-28T15:37:41.641 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptlan.ko 2024-03-28T15:37:41.641 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptsas.ko 2024-03-28T15:37:41.641 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptscsih.ko 2024-03-28T15:37:41.642 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptspi.ko 2024-03-28T15:37:41.642 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd 2024-03-28T15:37:41.642 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/lpc_ich.ko 2024-03-28T15:37:41.642 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/lpc_sch.ko 2024-03-28T15:37:41.642 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/mfd-core.ko 2024-03-28T15:37:41.642 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/sm501.ko 2024-03-28T15:37:41.642 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/viperboard.ko 2024-03-28T15:37:41.642 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/vx855.ko 2024-03-28T15:37:41.642 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/wl1273-core.ko 2024-03-28T15:37:41.642 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc 2024-03-28T15:37:41.642 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/altera-stapl 2024-03-28T15:37:41.642 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-03-28T15:37:41.642 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/apds9802als.ko 2024-03-28T15:37:41.642 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/apds990x.ko 2024-03-28T15:37:41.643 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/bh1770glc.ko 2024-03-28T15:37:41.643 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/cb710 2024-03-28T15:37:41.643 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/cb710/cb710.ko 2024-03-28T15:37:41.643 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom 2024-03-28T15:37:41.643 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/at24.ko 2024-03-28T15:37:41.643 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-03-28T15:37:41.643 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/max6875.ko 2024-03-28T15:37:41.643 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/enclosure.ko 2024-03-28T15:37:41.643 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/hpilo.ko 2024-03-28T15:37:41.643 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/isl29003.ko 2024-03-28T15:37:41.643 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/isl29020.ko 2024-03-28T15:37:41.643 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d 2024-03-28T15:37:41.643 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-03-28T15:37:41.643 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-03-28T15:37:41.644 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei 2024-03-28T15:37:41.644 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei-me.ko 2024-03-28T15:37:41.644 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei-txe.ko 2024-03-28T15:37:41.644 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei.ko 2024-03-28T15:37:41.644 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-gru 2024-03-28T15:37:41.644 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-gru/gru.ko 2024-03-28T15:37:41.644 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp 2024-03-28T15:37:41.644 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp/xp.ko 2024-03-28T15:37:41.644 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp/xpc.ko 2024-03-28T15:37:41.644 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-03-28T15:37:41.644 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tifm_7xx1.ko 2024-03-28T15:37:41.644 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tifm_core.ko 2024-03-28T15:37:41.644 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tsl2550.ko 2024-03-28T15:37:41.644 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_balloon.ko 2024-03-28T15:37:41.645 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_vmci 2024-03-28T15:37:41.645 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-03-28T15:37:41.645 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net 2024-03-28T15:37:41.645 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/bonding 2024-03-28T15:37:41.645 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/bonding/bonding.ko 2024-03-28T15:37:41.645 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa 2024-03-28T15:37:41.645 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/b53 2024-03-28T15:37:41.645 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/b53/b53_common.ko 2024-03-28T15:37:41.645 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/bcm-sf2.ko 2024-03-28T15:37:41.645 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6060.ko 2024-03-28T15:37:41.645 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6xxx 2024-03-28T15:37:41.645 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-03-28T15:37:41.645 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dummy.ko 2024-03-28T15:37:41.645 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/eql.ko 2024-03-28T15:37:41.646 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet 2024-03-28T15:37:41.646 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com 2024-03-28T15:37:41.646 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-03-28T15:37:41.646 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-03-28T15:37:41.646 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-03-28T15:37:41.646 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-03-28T15:37:41.646 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390 2024-03-28T15:37:41.646 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/8390.ko 2024-03-28T15:37:41.646 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-03-28T15:37:41.646 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-03-28T15:37:41.646 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-03-28T15:37:41.646 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/adaptec 2024-03-28T15:37:41.646 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-03-28T15:37:41.646 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/agere 2024-03-28T15:37:41.646 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/agere/et131x.ko 2024-03-28T15:37:41.647 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/alteon 2024-03-28T15:37:41.647 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-03-28T15:37:41.647 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/altera 2024-03-28T15:37:41.647 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-03-28T15:37:41.647 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd 2024-03-28T15:37:41.647 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-03-28T15:37:41.647 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-03-28T15:37:41.647 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-03-28T15:37:41.647 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros 2024-03-28T15:37:41.647 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/alx 2024-03-28T15:37:41.647 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-03-28T15:37:41.647 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1c 2024-03-28T15:37:41.647 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-03-28T15:37:41.647 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1e 2024-03-28T15:37:41.647 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-03-28T15:37:41.648 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx 2024-03-28T15:37:41.648 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-03-28T15:37:41.648 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-03-28T15:37:41.648 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom 2024-03-28T15:37:41.648 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-03-28T15:37:41.648 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-03-28T15:37:41.648 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-03-28T15:37:41.648 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-03-28T15:37:41.648 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-03-28T15:37:41.648 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/genet 2024-03-28T15:37:41.648 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-03-28T15:37:41.648 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-03-28T15:37:41.648 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade 2024-03-28T15:37:41.648 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade/bna 2024-03-28T15:37:41.648 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-03-28T15:37:41.649 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio 2024-03-28T15:37:41.649 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb 2024-03-28T15:37:41.649 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-03-28T15:37:41.649 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-03-28T15:37:41.649 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-03-28T15:37:41.649 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-03-28T15:37:41.649 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-03-28T15:37:41.649 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-03-28T15:37:41.649 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-03-28T15:37:41.649 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-03-28T15:37:41.649 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-03-28T15:37:41.649 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco 2024-03-28T15:37:41.649 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco/enic 2024-03-28T15:37:41.649 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-03-28T15:37:41.649 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec 2024-03-28T15:37:41.650 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip 2024-03-28T15:37:41.650 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-03-28T15:37:41.650 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-03-28T15:37:41.650 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-03-28T15:37:41.650 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-03-28T15:37:41.650 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-03-28T15:37:41.650 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-03-28T15:37:41.650 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink 2024-03-28T15:37:41.650 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-03-28T15:37:41.650 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-03-28T15:37:41.650 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dnet.ko 2024-03-28T15:37:41.650 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex 2024-03-28T15:37:41.650 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex/benet 2024-03-28T15:37:41.650 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-03-28T15:37:41.651 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ethoc.ko 2024-03-28T15:37:41.651 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/fealnx.ko 2024-03-28T15:37:41.651 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel 2024-03-28T15:37:41.651 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e100.ko 2024-03-28T15:37:41.651 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000 2024-03-28T15:37:41.651 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-03-28T15:37:41.651 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000e 2024-03-28T15:37:41.651 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-03-28T15:37:41.651 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/fm10k 2024-03-28T15:37:41.651 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-03-28T15:37:41.651 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/i40e 2024-03-28T15:37:41.651 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-03-28T15:37:41.651 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/iavf 2024-03-28T15:37:41.651 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-03-28T15:37:41.652 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igb 2024-03-28T15:37:41.652 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-03-28T15:37:41.652 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igbvf 2024-03-28T15:37:41.652 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-03-28T15:37:41.652 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbe 2024-03-28T15:37:41.652 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-03-28T15:37:41.652 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbevf 2024-03-28T15:37:41.652 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-03-28T15:37:41.652 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/jme.ko 2024-03-28T15:37:41.652 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell 2024-03-28T15:37:41.652 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-03-28T15:37:41.652 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/skge.ko 2024-03-28T15:37:41.652 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-03-28T15:37:41.652 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox 2024-03-28T15:37:41.652 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4 2024-03-28T15:37:41.652 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-03-28T15:37:41.653 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-03-28T15:37:41.653 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5 2024-03-28T15:37:41.653 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-03-28T15:37:41.653 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-03-28T15:37:41.653 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/micrel 2024-03-28T15:37:41.653 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-03-28T15:37:41.653 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom 2024-03-28T15:37:41.653 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom/myri10ge 2024-03-28T15:37:41.653 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-03-28T15:37:41.653 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi 2024-03-28T15:37:41.653 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-03-28T15:37:41.653 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-03-28T15:37:41.653 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/neterion 2024-03-28T15:37:41.653 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-03-28T15:37:41.654 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/nvidia 2024-03-28T15:37:41.654 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-03-28T15:37:41.654 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines 2024-03-28T15:37:41.654 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-03-28T15:37:41.654 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-03-28T15:37:41.654 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic 2024-03-28T15:37:41.654 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/netxen 2024-03-28T15:37:41.654 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-03-28T15:37:41.654 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-03-28T15:37:41.654 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-03-28T15:37:41.654 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-03-28T15:37:41.654 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rdc 2024-03-28T15:37:41.654 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-03-28T15:37:41.655 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek 2024-03-28T15:37:41.655 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-03-28T15:37:41.655 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-03-28T15:37:41.655 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/atp.ko 2024-03-28T15:37:41.655 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-03-28T15:37:41.655 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rocker 2024-03-28T15:37:41.655 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-03-28T15:37:41.655 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sfc 2024-03-28T15:37:41.655 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-03-28T15:37:41.655 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/silan 2024-03-28T15:37:41.655 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-03-28T15:37:41.655 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis 2024-03-28T15:37:41.655 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis/sis190.ko 2024-03-28T15:37:41.655 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis/sis900.ko 2024-03-28T15:37:41.655 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc 2024-03-28T15:37:41.656 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-03-28T15:37:41.656 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-03-28T15:37:41.656 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-03-28T15:37:41.656 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-03-28T15:37:41.656 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro 2024-03-28T15:37:41.656 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac 2024-03-28T15:37:41.656 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-03-28T15:37:41.656 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-03-28T15:37:41.656 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun 2024-03-28T15:37:41.656 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/cassini.ko 2024-03-28T15:37:41.656 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/niu.ko 2024-03-28T15:37:41.656 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/sungem.ko 2024-03-28T15:37:41.656 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-03-28T15:37:41.656 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/tehuti 2024-03-28T15:37:41.657 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-03-28T15:37:41.657 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ti 2024-03-28T15:37:41.657 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ti/tlan.ko 2024-03-28T15:37:41.657 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via 2024-03-28T15:37:41.657 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-03-28T15:37:41.657 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-03-28T15:37:41.657 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet 2024-03-28T15:37:41.657 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-03-28T15:37:41.657 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-03-28T15:37:41.657 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/xircom 2024-03-28T15:37:41.657 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-03-28T15:37:41.657 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/geneve.ko 2024-03-28T15:37:41.657 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/hyperv 2024-03-28T15:37:41.657 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-03-28T15:37:41.658 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ieee802154 2024-03-28T15:37:41.658 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ieee802154/fakelb.ko 2024-03-28T15:37:41.658 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ifb.ko 2024-03-28T15:37:41.658 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ipvlan 2024-03-28T15:37:41.658 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ipvlan/ipvlan.ko 2024-03-28T15:37:41.658 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/macvlan.ko 2024-03-28T15:37:41.658 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/macvtap.ko 2024-03-28T15:37:41.658 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio 2024-03-28T15:37:41.658 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio.ko 2024-03-28T15:37:41.658 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-03-28T15:37:41.658 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-03-28T15:37:41.658 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-regmap.ko 2024-03-28T15:37:41.658 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mii.ko 2024-03-28T15:37:41.658 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/net_failover.ko 2024-03-28T15:37:41.659 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/netconsole.ko 2024-03-28T15:37:41.659 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/nlmon.ko 2024-03-28T15:37:41.659 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs 2024-03-28T15:37:41.659 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs/pcs-lynx.ko 2024-03-28T15:37:41.659 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-03-28T15:37:41.659 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy 2024-03-28T15:37:41.659 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/amd.ko 2024-03-28T15:37:41.659 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-03-28T15:37:41.659 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-03-28T15:37:41.659 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm7xxx.ko 2024-03-28T15:37:41.659 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm87xx.ko 2024-03-28T15:37:41.659 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/broadcom.ko 2024-03-28T15:37:41.659 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/cicada.ko 2024-03-28T15:37:41.659 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/davicom.ko 2024-03-28T15:37:41.660 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/dp83640.ko 2024-03-28T15:37:41.660 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/et1011c.ko 2024-03-28T15:37:41.660 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/icplus.ko 2024-03-28T15:37:41.660 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/lxt.ko 2024-03-28T15:37:41.660 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/marvell.ko 2024-03-28T15:37:41.660 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/micrel.ko 2024-03-28T15:37:41.660 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/national.ko 2024-03-28T15:37:41.660 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/phylink.ko 2024-03-28T15:37:41.660 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/qsemi.ko 2024-03-28T15:37:41.660 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/realtek.ko 2024-03-28T15:37:41.660 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/smsc.ko 2024-03-28T15:37:41.660 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/ste10Xp.ko 2024-03-28T15:37:41.660 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/vitesse.ko 2024-03-28T15:37:41.660 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp 2024-03-28T15:37:41.660 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/bsd_comp.ko 2024-03-28T15:37:41.661 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_async.ko 2024-03-28T15:37:41.661 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_deflate.ko 2024-03-28T15:37:41.661 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_generic.ko 2024-03-28T15:37:41.661 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_mppe.ko 2024-03-28T15:37:41.661 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_synctty.ko 2024-03-28T15:37:41.661 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pppoe.ko 2024-03-28T15:37:41.661 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pppox.ko 2024-03-28T15:37:41.661 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pptp.ko 2024-03-28T15:37:41.661 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/slip 2024-03-28T15:37:41.661 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/slip/slhc.ko 2024-03-28T15:37:41.661 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/slip/slip.ko 2024-03-28T15:37:41.661 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/sungem_phy.ko 2024-03-28T15:37:41.661 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/tap.ko 2024-03-28T15:37:41.661 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team 2024-03-28T15:37:41.661 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team.ko 2024-03-28T15:37:41.662 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_activebackup.ko 2024-03-28T15:37:41.662 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_broadcast.ko 2024-03-28T15:37:41.662 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-03-28T15:37:41.662 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_random.ko 2024-03-28T15:37:41.662 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-03-28T15:37:41.662 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/tun.ko 2024-03-28T15:37:41.662 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/veth.ko 2024-03-28T15:37:41.662 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/virtio_net.ko 2024-03-28T15:37:41.662 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vmxnet3 2024-03-28T15:37:41.662 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-03-28T15:37:41.662 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vxlan 2024-03-28T15:37:41.662 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vxlan/vxlan.ko 2024-03-28T15:37:41.662 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/xen-netback 2024-03-28T15:37:41.662 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/xen-netback/xen-netback.ko 2024-03-28T15:37:41.663 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/xen-netfront.ko 2024-03-28T15:37:41.663 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ntb 2024-03-28T15:37:41.663 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ntb/ntb.ko 2024-03-28T15:37:41.663 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme 2024-03-28T15:37:41.663 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host 2024-03-28T15:37:41.663 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host/nvme-core.ko 2024-03-28T15:37:41.663 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host/nvme.ko 2024-03-28T15:37:41.663 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport 2024-03-28T15:37:41.663 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport.ko 2024-03-28T15:37:41.663 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_cs.ko 2024-03-28T15:37:41.663 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_pc.ko 2024-03-28T15:37:41.663 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_serial.ko 2024-03-28T15:37:41.663 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci 2024-03-28T15:37:41.663 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/hotplug 2024-03-28T15:37:41.664 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-03-28T15:37:41.664 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/pcie 2024-03-28T15:37:41.664 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/pcie/aer_inject.ko 2024-03-28T15:37:41.664 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/xen-pcifront.ko 2024-03-28T15:37:41.664 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia 2024-03-28T15:37:41.664 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/i82092.ko 2024-03-28T15:37:41.664 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/pd6729.ko 2024-03-28T15:37:41.664 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/yenta_socket.ko 2024-03-28T15:37:41.664 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl 2024-03-28T15:37:41.664 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl/intel 2024-03-28T15:37:41.664 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-03-28T15:37:41.664 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform 2024-03-28T15:37:41.664 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86 2024-03-28T15:37:41.664 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/acer-wmi.ko 2024-03-28T15:37:41.664 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/acerhdf.ko 2024-03-28T15:37:41.665 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/apple-gmux.ko 2024-03-28T15:37:41.665 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-laptop.ko 2024-03-28T15:37:41.665 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-03-28T15:37:41.665 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-wmi.ko 2024-03-28T15:37:41.665 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/classmate-laptop.ko 2024-03-28T15:37:41.665 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-03-28T15:37:41.665 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-03-28T15:37:41.665 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-03-28T15:37:41.665 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-03-28T15:37:41.665 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/hdaps.ko 2024-03-28T15:37:41.665 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel 2024-03-28T15:37:41.665 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-03-28T15:37:41.665 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel_ips.ko 2024-03-28T15:37:41.665 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/msi-wmi.ko 2024-03-28T15:37:41.665 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/mxm-wmi.ko 2024-03-28T15:37:41.666 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-03-28T15:37:41.666 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/samsung-laptop.ko 2024-03-28T15:37:41.666 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/samsung-q10.ko 2024-03-28T15:37:41.666 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/topstar-laptop.ko 2024-03-28T15:37:41.666 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-03-28T15:37:41.666 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/toshiba_haps.ko 2024-03-28T15:37:41.666 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/wmi-bmof.ko 2024-03-28T15:37:41.666 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/wmi.ko 2024-03-28T15:37:41.666 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap 2024-03-28T15:37:41.666 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap/intel_rapl_common.ko 2024-03-28T15:37:41.666 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap/intel_rapl_msr.ko 2024-03-28T15:37:41.666 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps 2024-03-28T15:37:41.666 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients 2024-03-28T15:37:41.666 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps-gpio.ko 2024-03-28T15:37:41.667 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps-ldisc.ko 2024-03-28T15:37:41.667 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps_parport.ko 2024-03-28T15:37:41.667 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/pps_core.ko 2024-03-28T15:37:41.667 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp 2024-03-28T15:37:41.667 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp/ptp.ko 2024-03-28T15:37:41.667 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp/ptp_kvm.ko 2024-03-28T15:37:41.667 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc 2024-03-28T15:37:41.667 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-bq32k.ko 2024-03-28T15:37:41.667 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1286.ko 2024-03-28T15:37:41.667 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1307.ko 2024-03-28T15:37:41.667 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1374.ko 2024-03-28T15:37:41.667 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1511.ko 2024-03-28T15:37:41.667 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1553.ko 2024-03-28T15:37:41.667 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1672.ko 2024-03-28T15:37:41.667 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1742.ko 2024-03-28T15:37:41.668 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds2404.ko 2024-03-28T15:37:41.668 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds3232.ko 2024-03-28T15:37:41.668 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-em3027.ko 2024-03-28T15:37:41.668 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-fm3130.ko 2024-03-28T15:37:41.668 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-isl12022.ko 2024-03-28T15:37:41.668 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-isl1208.ko 2024-03-28T15:37:41.668 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m41t80.ko 2024-03-28T15:37:41.668 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m48t35.ko 2024-03-28T15:37:41.668 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m48t59.ko 2024-03-28T15:37:41.668 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-max6900.ko 2024-03-28T15:37:41.668 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-msm6242.ko 2024-03-28T15:37:41.668 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf2127.ko 2024-03-28T15:37:41.668 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf85063.ko 2024-03-28T15:37:41.668 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8523.ko 2024-03-28T15:37:41.668 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8563.ko 2024-03-28T15:37:41.668 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8583.ko 2024-03-28T15:37:41.669 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rp5c01.ko 2024-03-28T15:37:41.669 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rs5c372.ko 2024-03-28T15:37:41.669 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-03-28T15:37:41.669 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rx8025.ko 2024-03-28T15:37:41.669 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rx8581.ko 2024-03-28T15:37:41.669 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-03-28T15:37:41.669 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-x1205.ko 2024-03-28T15:37:41.669 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi 2024-03-28T15:37:41.669 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-9xxx.ko 2024-03-28T15:37:41.669 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-sas.ko 2024-03-28T15:37:41.669 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-xxxx.ko 2024-03-28T15:37:41.669 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/BusLogic.ko 2024-03-28T15:37:41.669 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/a100u2w.ko 2024-03-28T15:37:41.669 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aacraid 2024-03-28T15:37:41.670 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aacraid/aacraid.ko 2024-03-28T15:37:41.670 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/advansys.ko 2024-03-28T15:37:41.670 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx 2024-03-28T15:37:41.670 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-03-28T15:37:41.670 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-03-28T15:37:41.670 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/am53c974.ko 2024-03-28T15:37:41.670 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/arcmsr 2024-03-28T15:37:41.670 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-03-28T15:37:41.670 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/atp870u.ko 2024-03-28T15:37:41.670 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/be2iscsi 2024-03-28T15:37:41.670 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-03-28T15:37:41.670 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bfa 2024-03-28T15:37:41.670 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bfa/bfa.ko 2024-03-28T15:37:41.670 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2fc 2024-03-28T15:37:41.671 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-03-28T15:37:41.671 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2i 2024-03-28T15:37:41.671 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-03-28T15:37:41.671 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ch.ko 2024-03-28T15:37:41.671 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/csiostor 2024-03-28T15:37:41.671 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/csiostor/csiostor.ko 2024-03-28T15:37:41.671 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi 2024-03-28T15:37:41.671 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb3i 2024-03-28T15:37:41.671 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-03-28T15:37:41.671 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb4i 2024-03-28T15:37:41.671 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-03-28T15:37:41.671 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-03-28T15:37:41.671 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/dc395x.ko 2024-03-28T15:37:41.671 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler 2024-03-28T15:37:41.671 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-03-28T15:37:41.672 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-03-28T15:37:41.672 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-03-28T15:37:41.672 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-03-28T15:37:41.672 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/dmx3191d.ko 2024-03-28T15:37:41.672 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esas2r 2024-03-28T15:37:41.672 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esas2r/esas2r.ko 2024-03-28T15:37:41.672 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esp_scsi.ko 2024-03-28T15:37:41.672 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe 2024-03-28T15:37:41.672 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe/fcoe.ko 2024-03-28T15:37:41.672 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-03-28T15:37:41.672 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fnic 2024-03-28T15:37:41.672 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fnic/fnic.ko 2024-03-28T15:37:41.672 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hpsa.ko 2024-03-28T15:37:41.672 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hptiop.ko 2024-03-28T15:37:41.672 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hv_storvsc.ko 2024-03-28T15:37:41.672 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/initio.ko 2024-03-28T15:37:41.673 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ipr.ko 2024-03-28T15:37:41.673 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ips.ko 2024-03-28T15:37:41.673 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/isci 2024-03-28T15:37:41.673 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/isci/isci.ko 2024-03-28T15:37:41.673 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-03-28T15:37:41.673 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/iscsi_tcp.ko 2024-03-28T15:37:41.673 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libfc 2024-03-28T15:37:41.673 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libfc/libfc.ko 2024-03-28T15:37:41.673 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libiscsi.ko 2024-03-28T15:37:41.673 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libiscsi_tcp.ko 2024-03-28T15:37:41.673 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libsas 2024-03-28T15:37:41.673 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libsas/libsas.ko 2024-03-28T15:37:41.673 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/lpfc 2024-03-28T15:37:41.674 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/lpfc/lpfc.ko 2024-03-28T15:37:41.674 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid 2024-03-28T15:37:41.674 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid.ko 2024-03-28T15:37:41.674 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-03-28T15:37:41.674 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-03-28T15:37:41.674 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-03-28T15:37:41.674 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mpt3sas 2024-03-28T15:37:41.674 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-03-28T15:37:41.674 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvsas 2024-03-28T15:37:41.674 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvsas/mvsas.ko 2024-03-28T15:37:41.674 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvumi.ko 2024-03-28T15:37:41.674 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pm8001 2024-03-28T15:37:41.674 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-03-28T15:37:41.674 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pmcraid.ko 2024-03-28T15:37:41.675 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla1280.ko 2024-03-28T15:37:41.675 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx 2024-03-28T15:37:41.675 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-03-28T15:37:41.675 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-03-28T15:37:41.675 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla4xxx 2024-03-28T15:37:41.675 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-03-28T15:37:41.675 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/raid_class.ko 2024-03-28T15:37:41.675 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_debug.ko 2024-03-28T15:37:41.675 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_fc.ko 2024-03-28T15:37:41.675 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-03-28T15:37:41.675 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_sas.ko 2024-03-28T15:37:41.675 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_spi.ko 2024-03-28T15:37:41.675 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_srp.ko 2024-03-28T15:37:41.675 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ses.ko 2024-03-28T15:37:41.676 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/st.ko 2024-03-28T15:37:41.676 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/stex.ko 2024-03-28T15:37:41.676 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/sym53c8xx_2 2024-03-28T15:37:41.676 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-03-28T15:37:41.676 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/virtio_scsi.ko 2024-03-28T15:37:41.676 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/vmw_pvscsi.ko 2024-03-28T15:37:41.676 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/wd719x.ko 2024-03-28T15:37:41.676 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ssb 2024-03-28T15:37:41.676 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ssb/ssb.ko 2024-03-28T15:37:41.676 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target 2024-03-28T15:37:41.676 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/iscsi 2024-03-28T15:37:41.676 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-03-28T15:37:41.676 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/loopback 2024-03-28T15:37:41.676 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/loopback/tcm_loop.ko 2024-03-28T15:37:41.676 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_file.ko 2024-03-28T15:37:41.677 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_iblock.ko 2024-03-28T15:37:41.677 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_mod.ko 2024-03-28T15:37:41.677 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_pscsi.ko 2024-03-28T15:37:41.677 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_user.ko 2024-03-28T15:37:41.677 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/tcm_fc 2024-03-28T15:37:41.677 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-03-28T15:37:41.677 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal 2024-03-28T15:37:41.677 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel 2024-03-28T15:37:41.677 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal 2024-03-28T15:37:41.677 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-03-28T15:37:41.677 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-03-28T15:37:41.677 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-03-28T15:37:41.677 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-03-28T15:37:41.677 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-03-28T15:37:41.678 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-03-28T15:37:41.678 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-03-28T15:37:41.678 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-03-28T15:37:41.678 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-03-28T15:37:41.678 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-03-28T15:37:41.678 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-03-28T15:37:41.678 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-03-28T15:37:41.678 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-03-28T15:37:41.678 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-03-28T15:37:41.678 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-03-28T15:37:41.678 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-03-28T15:37:41.678 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-03-28T15:37:41.678 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-03-28T15:37:41.678 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty 2024-03-28T15:37:41.679 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/ipwireless 2024-03-28T15:37:41.679 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-03-28T15:37:41.679 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/n_gsm.ko 2024-03-28T15:37:41.679 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/n_hdlc.ko 2024-03-28T15:37:41.679 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/nozomi.ko 2024-03-28T15:37:41.679 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial 2024-03-28T15:37:41.679 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/8250 2024-03-28T15:37:41.679 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-03-28T15:37:41.679 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/arc_uart.ko 2024-03-28T15:37:41.679 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/jsm 2024-03-28T15:37:41.679 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/jsm/jsm.ko 2024-03-28T15:37:41.679 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs 2024-03-28T15:37:41.679 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/core 2024-03-28T15:37:41.679 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/core/ufshcd-core.ko 2024-03-28T15:37:41.679 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/host 2024-03-28T15:37:41.680 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-03-28T15:37:41.680 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio 2024-03-28T15:37:41.680 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio.ko 2024-03-28T15:37:41.680 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_aec.ko 2024-03-28T15:37:41.680 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_cif.ko 2024-03-28T15:37:41.680 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_pci_generic.ko 2024-03-28T15:37:41.680 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_sercos3.ko 2024-03-28T15:37:41.680 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb 2024-03-28T15:37:41.680 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class 2024-03-28T15:37:41.680 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/cdc-acm.ko 2024-03-28T15:37:41.680 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/cdc-wdm.ko 2024-03-28T15:37:41.680 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/usblp.ko 2024-03-28T15:37:41.680 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/usbtmc.ko 2024-03-28T15:37:41.680 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host 2024-03-28T15:37:41.680 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host/sl811-hcd.ko 2024-03-28T15:37:41.681 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image 2024-03-28T15:37:41.681 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image/mdc800.ko 2024-03-28T15:37:41.681 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image/microtek.ko 2024-03-28T15:37:41.681 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc 2024-03-28T15:37:41.681 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/adutux.ko 2024-03-28T15:37:41.681 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/appledisplay.ko 2024-03-28T15:37:41.681 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/emi26.ko 2024-03-28T15:37:41.681 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/emi62.ko 2024-03-28T15:37:41.681 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/ezusb.ko 2024-03-28T15:37:41.681 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/idmouse.ko 2024-03-28T15:37:41.681 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/iowarrior.ko 2024-03-28T15:37:41.681 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/isight_firmware.ko 2024-03-28T15:37:41.681 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/ldusb.ko 2024-03-28T15:37:41.681 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/legousbtower.ko 2024-03-28T15:37:41.681 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/sisusbvga 2024-03-28T15:37:41.682 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-03-28T15:37:41.682 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/trancevibrator.ko 2024-03-28T15:37:41.682 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usb3503.ko 2024-03-28T15:37:41.682 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usblcd.ko 2024-03-28T15:37:41.682 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usbsevseg.ko 2024-03-28T15:37:41.682 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/uss720.ko 2024-03-28T15:37:41.682 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/yurex.ko 2024-03-28T15:37:41.682 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/phy 2024-03-28T15:37:41.682 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/phy/phy-generic.ko 2024-03-28T15:37:41.682 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial 2024-03-28T15:37:41.682 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/aircable.ko 2024-03-28T15:37:41.682 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ark3116.ko 2024-03-28T15:37:41.682 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/belkin_sa.ko 2024-03-28T15:37:41.682 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ch341.ko 2024-03-28T15:37:41.683 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cp210x.ko 2024-03-28T15:37:41.683 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cyberjack.ko 2024-03-28T15:37:41.683 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cypress_m8.ko 2024-03-28T15:37:41.683 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/digi_acceleport.ko 2024-03-28T15:37:41.683 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/empeg.ko 2024-03-28T15:37:41.683 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ftdi_sio.ko 2024-03-28T15:37:41.683 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/garmin_gps.ko 2024-03-28T15:37:41.683 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/io_edgeport.ko 2024-03-28T15:37:41.683 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/io_ti.ko 2024-03-28T15:37:41.683 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ipaq.ko 2024-03-28T15:37:41.683 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ipw.ko 2024-03-28T15:37:41.683 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ir-usb.ko 2024-03-28T15:37:41.683 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-03-28T15:37:41.683 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/keyspan.ko 2024-03-28T15:37:41.684 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/keyspan_pda.ko 2024-03-28T15:37:41.684 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/kl5kusb105.ko 2024-03-28T15:37:41.684 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/kobil_sct.ko 2024-03-28T15:37:41.684 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mct_u232.ko 2024-03-28T15:37:41.684 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mos7720.ko 2024-03-28T15:37:41.684 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mos7840.ko 2024-03-28T15:37:41.684 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/navman.ko 2024-03-28T15:37:41.684 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/omninet.ko 2024-03-28T15:37:41.684 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/opticon.ko 2024-03-28T15:37:41.684 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/option.ko 2024-03-28T15:37:41.684 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/oti6858.ko 2024-03-28T15:37:41.684 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/pl2303.ko 2024-03-28T15:37:41.684 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/qcaux.ko 2024-03-28T15:37:41.684 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/qcserial.ko 2024-03-28T15:37:41.684 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/quatech2.ko 2024-03-28T15:37:41.684 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/safe_serial.ko 2024-03-28T15:37:41.685 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/sierra.ko 2024-03-28T15:37:41.685 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/spcp8x5.ko 2024-03-28T15:37:41.685 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ssu100.ko 2024-03-28T15:37:41.685 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/symbolserial.ko 2024-03-28T15:37:41.685 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-03-28T15:37:41.685 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-03-28T15:37:41.685 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb_debug.ko 2024-03-28T15:37:41.685 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb_wwan.ko 2024-03-28T15:37:41.685 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/visor.ko 2024-03-28T15:37:41.685 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/whiteheat.ko 2024-03-28T15:37:41.685 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/xsens_mt.ko 2024-03-28T15:37:41.685 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage 2024-03-28T15:37:41.685 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/uas.ko 2024-03-28T15:37:41.685 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-alauda.ko 2024-03-28T15:37:41.685 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-cypress.ko 2024-03-28T15:37:41.685 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-datafab.ko 2024-03-28T15:37:41.686 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-03-28T15:37:41.686 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-freecom.ko 2024-03-28T15:37:41.686 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-isd200.ko 2024-03-28T15:37:41.686 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-03-28T15:37:41.686 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-karma.ko 2024-03-28T15:37:41.686 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-onetouch.ko 2024-03-28T15:37:41.686 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-realtek.ko 2024-03-28T15:37:41.686 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-sddr09.ko 2024-03-28T15:37:41.686 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-sddr55.ko 2024-03-28T15:37:41.686 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-usbat.ko 2024-03-28T15:37:41.686 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/usb-storage.ko 2024-03-28T15:37:41.686 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip 2024-03-28T15:37:41.686 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/usbip-core.ko 2024-03-28T15:37:41.686 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/usbip-host.ko 2024-03-28T15:37:41.687 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-03-28T15:37:41.687 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio 2024-03-28T15:37:41.687 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci 2024-03-28T15:37:41.687 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-03-28T15:37:41.687 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci/vfio-pci.ko 2024-03-28T15:37:41.687 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/vfio.ko 2024-03-28T15:37:41.687 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-03-28T15:37:41.687 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost 2024-03-28T15:37:41.687 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost.ko 2024-03-28T15:37:41.687 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_iotlb.ko 2024-03-28T15:37:41.687 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_net.ko 2024-03-28T15:37:41.687 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_scsi.ko 2024-03-28T15:37:41.687 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video 2024-03-28T15:37:41.687 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight 2024-03-28T15:37:41.687 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/apple_bl.ko 2024-03-28T15:37:41.688 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/lcd.ko 2024-03-28T15:37:41.688 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/platform_lcd.ko 2024-03-28T15:37:41.688 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio 2024-03-28T15:37:41.688 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio.ko 2024-03-28T15:37:41.688 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_balloon.ko 2024-03-28T15:37:41.688 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_mmio.ko 2024-03-28T15:37:41.688 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_pci.ko 2024-03-28T15:37:41.688 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-03-28T15:37:41.688 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-03-28T15:37:41.688 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_ring.ko 2024-03-28T15:37:41.688 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1 2024-03-28T15:37:41.688 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters 2024-03-28T15:37:41.688 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters/ds2482.ko 2024-03-28T15:37:41.688 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters/ds2490.ko 2024-03-28T15:37:41.688 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves 2024-03-28T15:37:41.688 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-03-28T15:37:41.689 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-03-28T15:37:41.689 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-03-28T15:37:41.689 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-03-28T15:37:41.689 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-03-28T15:37:41.689 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-03-28T15:37:41.689 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-03-28T15:37:41.689 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-03-28T15:37:41.689 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-03-28T15:37:41.689 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_smem.ko 2024-03-28T15:37:41.689 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_therm.ko 2024-03-28T15:37:41.689 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/wire.ko 2024-03-28T15:37:41.689 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog 2024-03-28T15:37:41.689 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/alim1535_wdt.ko 2024-03-28T15:37:41.689 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/alim7101_wdt.ko 2024-03-28T15:37:41.689 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/f71808e_wdt.ko 2024-03-28T15:37:41.690 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/hpwdt.ko 2024-03-28T15:37:41.690 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/i6300esb.ko 2024-03-28T15:37:41.690 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-03-28T15:37:41.690 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/iTCO_wdt.ko 2024-03-28T15:37:41.690 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ib700wdt.ko 2024-03-28T15:37:41.690 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ibmasr.ko 2024-03-28T15:37:41.690 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-03-28T15:37:41.690 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/it8712f_wdt.ko 2024-03-28T15:37:41.690 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/it87_wdt.ko 2024-03-28T15:37:41.690 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/machzwd.ko 2024-03-28T15:37:41.690 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/nv_tco.ko 2024-03-28T15:37:41.690 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/pcwd_pci.ko 2024-03-28T15:37:41.690 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/pcwd_usb.ko 2024-03-28T15:37:41.690 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-03-28T15:37:41.691 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sch311x_wdt.ko 2024-03-28T15:37:41.691 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/softdog.ko 2024-03-28T15:37:41.691 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sp5100_tco.ko 2024-03-28T15:37:41.691 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/via_wdt.ko 2024-03-28T15:37:41.691 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-03-28T15:37:41.691 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83877f_wdt.ko 2024-03-28T15:37:41.691 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83977f_wdt.ko 2024-03-28T15:37:41.691 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/wdt_pci.ko 2024-03-28T15:37:41.691 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/xen_wdt.ko 2024-03-28T15:37:41.691 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen 2024-03-28T15:37:41.691 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-acpi-processor.ko 2024-03-28T15:37:41.691 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-evtchn.ko 2024-03-28T15:37:41.691 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-gntalloc.ko 2024-03-28T15:37:41.691 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-gntdev.ko 2024-03-28T15:37:41.691 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-pciback 2024-03-28T15:37:41.692 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-03-28T15:37:41.692 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-privcmd.ko 2024-03-28T15:37:41.692 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xenfs 2024-03-28T15:37:41.692 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xenfs/xenfs.ko 2024-03-28T15:37:41.692 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs 2024-03-28T15:37:41.692 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/9p 2024-03-28T15:37:41.692 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/9p/9p.ko 2024-03-28T15:37:41.692 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/affs 2024-03-28T15:37:41.692 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/affs/affs.ko 2024-03-28T15:37:41.692 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/befs 2024-03-28T15:37:41.692 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/befs/befs.ko 2024-03-28T15:37:41.692 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/binfmt_misc.ko 2024-03-28T15:37:41.692 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/btrfs 2024-03-28T15:37:41.692 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/btrfs/btrfs.ko 2024-03-28T15:37:41.693 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cachefiles 2024-03-28T15:37:41.693 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cachefiles/cachefiles.ko 2024-03-28T15:37:41.693 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ceph 2024-03-28T15:37:41.693 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ceph/ceph.ko 2024-03-28T15:37:41.693 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/coda 2024-03-28T15:37:41.693 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/coda/coda.ko 2024-03-28T15:37:41.693 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cramfs 2024-03-28T15:37:41.693 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cramfs/cramfs.ko 2024-03-28T15:37:41.693 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/dlm 2024-03-28T15:37:41.693 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/dlm/dlm.ko 2024-03-28T15:37:41.693 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ecryptfs 2024-03-28T15:37:41.693 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ecryptfs/ecryptfs.ko 2024-03-28T15:37:41.693 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/f2fs 2024-03-28T15:37:41.693 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/f2fs/f2fs.ko 2024-03-28T15:37:41.693 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat 2024-03-28T15:37:41.694 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat/fat.ko 2024-03-28T15:37:41.694 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat/msdos.ko 2024-03-28T15:37:41.694 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat/vfat.ko 2024-03-28T15:37:41.694 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fuse 2024-03-28T15:37:41.694 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fuse/cuse.ko 2024-03-28T15:37:41.694 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fuse/fuse.ko 2024-03-28T15:37:41.694 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/gfs2 2024-03-28T15:37:41.694 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/gfs2/gfs2.ko 2024-03-28T15:37:41.694 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfs 2024-03-28T15:37:41.694 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfs/hfs.ko 2024-03-28T15:37:41.694 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfsplus 2024-03-28T15:37:41.694 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfsplus/hfsplus.ko 2024-03-28T15:37:41.694 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/isofs 2024-03-28T15:37:41.694 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/isofs/isofs.ko 2024-03-28T15:37:41.694 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/jfs 2024-03-28T15:37:41.695 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/jfs/jfs.ko 2024-03-28T15:37:41.695 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/lockd 2024-03-28T15:37:41.695 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/lockd/lockd.ko 2024-03-28T15:37:41.695 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/minix 2024-03-28T15:37:41.695 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/minix/minix.ko 2024-03-28T15:37:41.695 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/netfs 2024-03-28T15:37:41.695 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/netfs/netfs.ko 2024-03-28T15:37:41.695 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs 2024-03-28T15:37:41.695 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/blocklayout 2024-03-28T15:37:41.695 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-03-28T15:37:41.695 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/filelayout 2024-03-28T15:37:41.695 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-03-28T15:37:41.695 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/flexfilelayout 2024-03-28T15:37:41.695 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-03-28T15:37:41.695 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfs.ko 2024-03-28T15:37:41.696 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfsv3.ko 2024-03-28T15:37:41.696 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfsv4.ko 2024-03-28T15:37:41.696 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common 2024-03-28T15:37:41.696 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common/grace.ko 2024-03-28T15:37:41.696 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common/nfs_acl.ko 2024-03-28T15:37:41.696 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfsd 2024-03-28T15:37:41.696 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfsd/nfsd.ko 2024-03-28T15:37:41.696 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nilfs2 2024-03-28T15:37:41.696 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nilfs2/nilfs2.ko 2024-03-28T15:37:41.696 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls 2024-03-28T15:37:41.696 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-celtic.ko 2024-03-28T15:37:41.696 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-centeuro.ko 2024-03-28T15:37:41.696 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-croatian.ko 2024-03-28T15:37:41.696 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-cyrillic.ko 2024-03-28T15:37:41.696 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-gaelic.ko 2024-03-28T15:37:41.697 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-greek.ko 2024-03-28T15:37:41.697 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-iceland.ko 2024-03-28T15:37:41.697 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-inuit.ko 2024-03-28T15:37:41.697 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-roman.ko 2024-03-28T15:37:41.697 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-romanian.ko 2024-03-28T15:37:41.697 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-turkish.ko 2024-03-28T15:37:41.697 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1250.ko 2024-03-28T15:37:41.697 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1251.ko 2024-03-28T15:37:41.697 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1255.ko 2024-03-28T15:37:41.697 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp737.ko 2024-03-28T15:37:41.697 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp775.ko 2024-03-28T15:37:41.697 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp850.ko 2024-03-28T15:37:41.697 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp852.ko 2024-03-28T15:37:41.697 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp855.ko 2024-03-28T15:37:41.697 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp857.ko 2024-03-28T15:37:41.697 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp860.ko 2024-03-28T15:37:41.698 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp861.ko 2024-03-28T15:37:41.698 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp862.ko 2024-03-28T15:37:41.698 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp863.ko 2024-03-28T15:37:41.698 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp864.ko 2024-03-28T15:37:41.698 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp865.ko 2024-03-28T15:37:41.698 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp866.ko 2024-03-28T15:37:41.698 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp869.ko 2024-03-28T15:37:41.698 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp874.ko 2024-03-28T15:37:41.698 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp932.ko 2024-03-28T15:37:41.698 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp936.ko 2024-03-28T15:37:41.698 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp949.ko 2024-03-28T15:37:41.698 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp950.ko 2024-03-28T15:37:41.698 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_euc-jp.ko 2024-03-28T15:37:41.698 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-1.ko 2024-03-28T15:37:41.698 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-13.ko 2024-03-28T15:37:41.699 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-14.ko 2024-03-28T15:37:41.699 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-15.ko 2024-03-28T15:37:41.699 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-2.ko 2024-03-28T15:37:41.699 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-3.ko 2024-03-28T15:37:41.699 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-4.ko 2024-03-28T15:37:41.699 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-5.ko 2024-03-28T15:37:41.699 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-6.ko 2024-03-28T15:37:41.699 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-7.ko 2024-03-28T15:37:41.699 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-9.ko 2024-03-28T15:37:41.699 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-r.ko 2024-03-28T15:37:41.699 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-ru.ko 2024-03-28T15:37:41.699 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-u.ko 2024-03-28T15:37:41.699 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_ucs2_utils.ko 2024-03-28T15:37:41.699 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_utf8.ko 2024-03-28T15:37:41.699 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2 2024-03-28T15:37:41.700 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/cluster 2024-03-28T15:37:41.700 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-03-28T15:37:41.700 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlm 2024-03-28T15:37:41.700 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-03-28T15:37:41.700 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlmfs 2024-03-28T15:37:41.700 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-03-28T15:37:41.700 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2.ko 2024-03-28T15:37:41.700 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-03-28T15:37:41.700 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-03-28T15:37:41.700 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-03-28T15:37:41.700 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/overlayfs 2024-03-28T15:37:41.700 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/overlayfs/overlay.ko 2024-03-28T15:37:41.700 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/pstore 2024-03-28T15:37:41.700 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/pstore/ramoops.ko 2024-03-28T15:37:41.701 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/reiserfs 2024-03-28T15:37:41.701 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/reiserfs/reiserfs.ko 2024-03-28T15:37:41.701 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/romfs 2024-03-28T15:37:41.701 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/romfs/romfs.ko 2024-03-28T15:37:41.701 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb 2024-03-28T15:37:41.701 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/client 2024-03-28T15:37:41.701 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/client/cifs.ko 2024-03-28T15:37:41.701 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common 2024-03-28T15:37:41.701 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common/cifs_arc4.ko 2024-03-28T15:37:41.701 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common/cifs_md4.ko 2024-03-28T15:37:41.701 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/squashfs 2024-03-28T15:37:41.701 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/squashfs/squashfs.ko 2024-03-28T15:37:41.701 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/sysv 2024-03-28T15:37:41.701 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/sysv/sysv.ko 2024-03-28T15:37:41.702 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/udf 2024-03-28T15:37:41.702 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/udf/udf.ko 2024-03-28T15:37:41.702 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ufs 2024-03-28T15:37:41.702 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ufs/ufs.ko 2024-03-28T15:37:41.702 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/xfs 2024-03-28T15:37:41.702 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/xfs/xfs.ko 2024-03-28T15:37:41.702 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel 2024-03-28T15:37:41.702 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/rcu 2024-03-28T15:37:41.702 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/rcu/rcutorture.ko 2024-03-28T15:37:41.702 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/torture.ko 2024-03-28T15:37:41.702 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/trace 2024-03-28T15:37:41.702 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-03-28T15:37:41.702 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib 2024-03-28T15:37:41.702 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/asn1_encoder.ko 2024-03-28T15:37:41.703 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crc-itu-t.ko 2024-03-28T15:37:41.703 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crc8.ko 2024-03-28T15:37:41.703 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto 2024-03-28T15:37:41.703 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto/libarc4.ko 2024-03-28T15:37:41.703 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto/libdes.ko 2024-03-28T15:37:41.703 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/libcrc32c.ko 2024-03-28T15:37:41.703 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lru_cache.ko 2024-03-28T15:37:41.703 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4 2024-03-28T15:37:41.703 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4/lz4_compress.ko 2024-03-28T15:37:41.703 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4/lz4hc_compress.ko 2024-03-28T15:37:41.703 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/math 2024-03-28T15:37:41.703 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/math/cordic.ko 2024-03-28T15:37:41.703 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/raid6 2024-03-28T15:37:41.703 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/raid6/raid6_pq.ko 2024-03-28T15:37:41.703 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/reed_solomon 2024-03-28T15:37:41.703 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/reed_solomon/reed_solomon.ko 2024-03-28T15:37:41.704 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_bm.ko 2024-03-28T15:37:41.704 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_fsm.ko 2024-03-28T15:37:41.704 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_kmp.ko 2024-03-28T15:37:41.704 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/zstd 2024-03-28T15:37:41.704 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/zstd/zstd_compress.ko 2024-03-28T15:37:41.704 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/mm 2024-03-28T15:37:41.704 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/mm/hwpoison-inject.ko 2024-03-28T15:37:41.704 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net 2024-03-28T15:37:41.704 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan 2024-03-28T15:37:41.704 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/6lowpan.ko 2024-03-28T15:37:41.704 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_dest.ko 2024-03-28T15:37:41.704 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_fragment.ko 2024-03-28T15:37:41.704 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_hop.ko 2024-03-28T15:37:41.704 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_ipv6.ko 2024-03-28T15:37:41.704 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_mobility.ko 2024-03-28T15:37:41.704 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_routing.ko 2024-03-28T15:37:41.705 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_udp.ko 2024-03-28T15:37:41.705 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802 2024-03-28T15:37:41.705 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/garp.ko 2024-03-28T15:37:41.705 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/mrp.ko 2024-03-28T15:37:41.705 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/p8022.ko 2024-03-28T15:37:41.705 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/psnap.ko 2024-03-28T15:37:41.705 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/stp.ko 2024-03-28T15:37:41.705 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/8021q 2024-03-28T15:37:41.705 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/8021q/8021q.ko 2024-03-28T15:37:41.705 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p 2024-03-28T15:37:41.705 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet.ko 2024-03-28T15:37:41.705 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_fd.ko 2024-03-28T15:37:41.705 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_rdma.ko 2024-03-28T15:37:41.705 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_virtio.ko 2024-03-28T15:37:41.705 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/batman-adv 2024-03-28T15:37:41.705 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/batman-adv/batman-adv.ko 2024-03-28T15:37:41.706 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge 2024-03-28T15:37:41.706 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/br_netfilter.ko 2024-03-28T15:37:41.706 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/bridge.ko 2024-03-28T15:37:41.706 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter 2024-03-28T15:37:41.706 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-03-28T15:37:41.706 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_among.ko 2024-03-28T15:37:41.706 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_arp.ko 2024-03-28T15:37:41.706 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-03-28T15:37:41.706 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-03-28T15:37:41.706 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_ip.ko 2024-03-28T15:37:41.706 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-03-28T15:37:41.706 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_limit.ko 2024-03-28T15:37:41.706 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_log.ko 2024-03-28T15:37:41.706 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_mark.ko 2024-03-28T15:37:41.706 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-03-28T15:37:41.707 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-03-28T15:37:41.707 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-03-28T15:37:41.707 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-03-28T15:37:41.707 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_snat.ko 2024-03-28T15:37:41.707 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_stp.ko 2024-03-28T15:37:41.707 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-03-28T15:37:41.707 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-03-28T15:37:41.707 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-03-28T15:37:41.707 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-03-28T15:37:41.707 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtables.ko 2024-03-28T15:37:41.707 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-03-28T15:37:41.707 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-03-28T15:37:41.707 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-03-28T15:37:41.707 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ceph 2024-03-28T15:37:41.707 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ceph/libceph.ko 2024-03-28T15:37:41.707 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/core 2024-03-28T15:37:41.708 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/core/failover.ko 2024-03-28T15:37:41.708 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/core/pktgen.ko 2024-03-28T15:37:41.708 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp 2024-03-28T15:37:41.708 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp.ko 2024-03-28T15:37:41.708 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_diag.ko 2024-03-28T15:37:41.708 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_ipv4.ko 2024-03-28T15:37:41.708 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_ipv6.ko 2024-03-28T15:37:41.708 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dns_resolver 2024-03-28T15:37:41.708 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dns_resolver/dns_resolver.ko 2024-03-28T15:37:41.708 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa 2024-03-28T15:37:41.708 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/dsa_core.ko 2024-03-28T15:37:41.708 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_brcm.ko 2024-03-28T15:37:41.708 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_dsa.ko 2024-03-28T15:37:41.708 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_none.ko 2024-03-28T15:37:41.708 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_trailer.ko 2024-03-28T15:37:41.709 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154 2024-03-28T15:37:41.709 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/6lowpan 2024-03-28T15:37:41.709 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-03-28T15:37:41.709 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/ieee802154.ko 2024-03-28T15:37:41.709 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/ieee802154_socket.ko 2024-03-28T15:37:41.709 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4 2024-03-28T15:37:41.709 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ah4.ko 2024-03-28T15:37:41.709 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/esp4.ko 2024-03-28T15:37:41.709 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/fou.ko 2024-03-28T15:37:41.709 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/gre.ko 2024-03-28T15:37:41.709 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/inet_diag.ko 2024-03-28T15:37:41.709 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_gre.ko 2024-03-28T15:37:41.709 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_tunnel.ko 2024-03-28T15:37:41.709 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_vti.ko 2024-03-28T15:37:41.709 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ipcomp.ko 2024-03-28T15:37:41.710 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ipip.ko 2024-03-28T15:37:41.710 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter 2024-03-28T15:37:41.710 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arp_tables.ko 2024-03-28T15:37:41.710 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-03-28T15:37:41.710 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-03-28T15:37:41.710 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-03-28T15:37:41.710 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-03-28T15:37:41.710 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-03-28T15:37:41.710 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-03-28T15:37:41.710 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-03-28T15:37:41.710 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-03-28T15:37:41.710 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-03-28T15:37:41.710 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_security.ko 2024-03-28T15:37:41.710 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-03-28T15:37:41.710 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-03-28T15:37:41.711 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-03-28T15:37:41.711 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-03-28T15:37:41.711 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-03-28T15:37:41.711 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-03-28T15:37:41.711 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-03-28T15:37:41.711 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-03-28T15:37:41.711 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-03-28T15:37:41.711 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-03-28T15:37:41.711 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_bic.ko 2024-03-28T15:37:41.711 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_dctcp.ko 2024-03-28T15:37:41.711 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_diag.ko 2024-03-28T15:37:41.711 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_highspeed.ko 2024-03-28T15:37:41.711 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_htcp.ko 2024-03-28T15:37:41.711 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_hybla.ko 2024-03-28T15:37:41.711 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_illinois.ko 2024-03-28T15:37:41.711 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_lp.ko 2024-03-28T15:37:41.712 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_scalable.ko 2024-03-28T15:37:41.712 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_vegas.ko 2024-03-28T15:37:41.712 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_veno.ko 2024-03-28T15:37:41.712 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_westwood.ko 2024-03-28T15:37:41.712 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_yeah.ko 2024-03-28T15:37:41.712 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tunnel4.ko 2024-03-28T15:37:41.712 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/udp_diag.ko 2024-03-28T15:37:41.712 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/udp_tunnel.ko 2024-03-28T15:37:41.712 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/xfrm4_tunnel.ko 2024-03-28T15:37:41.712 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6 2024-03-28T15:37:41.712 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ah6.ko 2024-03-28T15:37:41.712 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/esp6.ko 2024-03-28T15:37:41.712 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/fou6.ko 2024-03-28T15:37:41.712 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_tunnel.ko 2024-03-28T15:37:41.712 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-03-28T15:37:41.712 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_vti.ko 2024-03-28T15:37:41.713 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ipcomp6.ko 2024-03-28T15:37:41.713 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter 2024-03-28T15:37:41.713 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-03-28T15:37:41.713 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-03-28T15:37:41.713 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-03-28T15:37:41.713 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-03-28T15:37:41.713 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-03-28T15:37:41.713 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-03-28T15:37:41.713 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-03-28T15:37:41.713 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-03-28T15:37:41.713 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-03-28T15:37:41.713 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-03-28T15:37:41.713 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-03-28T15:37:41.713 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-03-28T15:37:41.713 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-03-28T15:37:41.713 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-03-28T15:37:41.714 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-03-28T15:37:41.714 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-03-28T15:37:41.714 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-03-28T15:37:41.714 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-03-28T15:37:41.714 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-03-28T15:37:41.714 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-03-28T15:37:41.714 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-03-28T15:37:41.714 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-03-28T15:37:41.714 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-03-28T15:37:41.714 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-03-28T15:37:41.714 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/sit.ko 2024-03-28T15:37:41.714 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/tunnel6.ko 2024-03-28T15:37:41.714 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/xfrm6_tunnel.ko 2024-03-28T15:37:41.714 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/key 2024-03-28T15:37:41.714 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/key/af_key.ko 2024-03-28T15:37:41.715 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp 2024-03-28T15:37:41.715 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_core.ko 2024-03-28T15:37:41.715 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_debugfs.ko 2024-03-28T15:37:41.715 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_eth.ko 2024-03-28T15:37:41.715 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ip.ko 2024-03-28T15:37:41.715 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ip6.ko 2024-03-28T15:37:41.715 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_netlink.ko 2024-03-28T15:37:41.715 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ppp.ko 2024-03-28T15:37:41.715 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/llc 2024-03-28T15:37:41.715 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/llc/llc.ko 2024-03-28T15:37:41.715 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mac802154 2024-03-28T15:37:41.715 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mac802154/mac802154.ko 2024-03-28T15:37:41.715 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mpls 2024-03-28T15:37:41.715 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mpls/mpls_gso.ko 2024-03-28T15:37:41.715 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter 2024-03-28T15:37:41.716 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset 2024-03-28T15:37:41.716 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set.ko 2024-03-28T15:37:41.716 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-03-28T15:37:41.716 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-03-28T15:37:41.716 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-03-28T15:37:41.716 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-03-28T15:37:41.716 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-03-28T15:37:41.716 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-03-28T15:37:41.716 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-03-28T15:37:41.716 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-03-28T15:37:41.716 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-03-28T15:37:41.716 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-03-28T15:37:41.716 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-03-28T15:37:41.716 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-03-28T15:37:41.716 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-03-28T15:37:41.717 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-03-28T15:37:41.717 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-03-28T15:37:41.717 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs 2024-03-28T15:37:41.717 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs.ko 2024-03-28T15:37:41.717 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-03-28T15:37:41.717 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-03-28T15:37:41.717 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-03-28T15:37:41.717 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-03-28T15:37:41.717 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-03-28T15:37:41.717 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-03-28T15:37:41.717 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-03-28T15:37:41.717 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-03-28T15:37:41.717 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-03-28T15:37:41.717 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-03-28T15:37:41.717 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-03-28T15:37:41.718 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-03-28T15:37:41.718 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-03-28T15:37:41.718 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conncount.ko 2024-03-28T15:37:41.718 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack.ko 2024-03-28T15:37:41.718 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-03-28T15:37:41.718 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-03-28T15:37:41.718 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-03-28T15:37:41.718 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_h323.ko 2024-03-28T15:37:41.718 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_irc.ko 2024-03-28T15:37:41.718 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-03-28T15:37:41.718 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-03-28T15:37:41.718 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-03-28T15:37:41.718 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_sane.ko 2024-03-28T15:37:41.718 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_sip.ko 2024-03-28T15:37:41.718 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-03-28T15:37:41.719 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-03-28T15:37:41.719 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_dup_netdev.ko 2024-03-28T15:37:41.719 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_flow_table.ko 2024-03-28T15:37:41.719 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_flow_table_inet.ko 2024-03-28T15:37:41.719 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_log_syslog.ko 2024-03-28T15:37:41.719 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat.ko 2024-03-28T15:37:41.719 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_amanda.ko 2024-03-28T15:37:41.719 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_ftp.ko 2024-03-28T15:37:41.719 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_irc.ko 2024-03-28T15:37:41.719 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_sip.ko 2024-03-28T15:37:41.719 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_tftp.ko 2024-03-28T15:37:41.719 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_synproxy_core.ko 2024-03-28T15:37:41.719 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_tables.ko 2024-03-28T15:37:41.719 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink.ko 2024-03-28T15:37:41.720 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_acct.ko 2024-03-28T15:37:41.720 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_log.ko 2024-03-28T15:37:41.720 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_osf.ko 2024-03-28T15:37:41.720 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_queue.ko 2024-03-28T15:37:41.720 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_chain_nat.ko 2024-03-28T15:37:41.720 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_compat.ko 2024-03-28T15:37:41.720 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_ct.ko 2024-03-28T15:37:41.720 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_dup_netdev.ko 2024-03-28T15:37:41.720 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_fib.ko 2024-03-28T15:37:41.720 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_fwd_netdev.ko 2024-03-28T15:37:41.720 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_hash.ko 2024-03-28T15:37:41.720 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_limit.ko 2024-03-28T15:37:41.720 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_log.ko 2024-03-28T15:37:41.720 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_masq.ko 2024-03-28T15:37:41.720 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_nat.ko 2024-03-28T15:37:41.721 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_numgen.ko 2024-03-28T15:37:41.721 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_queue.ko 2024-03-28T15:37:41.721 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_quota.ko 2024-03-28T15:37:41.721 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_redir.ko 2024-03-28T15:37:41.721 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_reject.ko 2024-03-28T15:37:41.721 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_reject_inet.ko 2024-03-28T15:37:41.721 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_socket.ko 2024-03-28T15:37:41.721 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_tunnel.ko 2024-03-28T15:37:41.721 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_AUDIT.ko 2024-03-28T15:37:41.721 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CHECKSUM.ko 2024-03-28T15:37:41.721 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CLASSIFY.ko 2024-03-28T15:37:41.721 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-03-28T15:37:41.721 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CT.ko 2024-03-28T15:37:41.721 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_DSCP.ko 2024-03-28T15:37:41.721 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_HL.ko 2024-03-28T15:37:41.722 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_HMARK.ko 2024-03-28T15:37:41.722 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_IDLETIMER.ko 2024-03-28T15:37:41.722 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_LED.ko 2024-03-28T15:37:41.722 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_LOG.ko 2024-03-28T15:37:41.722 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_MASQUERADE.ko 2024-03-28T15:37:41.722 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NETMAP.ko 2024-03-28T15:37:41.722 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NFLOG.ko 2024-03-28T15:37:41.722 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NFQUEUE.ko 2024-03-28T15:37:41.722 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_RATEEST.ko 2024-03-28T15:37:41.722 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_REDIRECT.ko 2024-03-28T15:37:41.722 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_SECMARK.ko 2024-03-28T15:37:41.722 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TCPMSS.ko 2024-03-28T15:37:41.722 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-03-28T15:37:41.722 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TEE.ko 2024-03-28T15:37:41.723 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TPROXY.ko 2024-03-28T15:37:41.723 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TRACE.ko 2024-03-28T15:37:41.723 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_addrtype.ko 2024-03-28T15:37:41.723 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_bpf.ko 2024-03-28T15:37:41.723 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cgroup.ko 2024-03-28T15:37:41.723 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cluster.ko 2024-03-28T15:37:41.723 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_comment.ko 2024-03-28T15:37:41.723 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connbytes.ko 2024-03-28T15:37:41.723 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connlabel.ko 2024-03-28T15:37:41.723 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connlimit.ko 2024-03-28T15:37:41.723 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connmark.ko 2024-03-28T15:37:41.723 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_conntrack.ko 2024-03-28T15:37:41.723 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cpu.ko 2024-03-28T15:37:41.723 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_dccp.ko 2024-03-28T15:37:41.723 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_devgroup.ko 2024-03-28T15:37:41.723 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_dscp.ko 2024-03-28T15:37:41.724 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ecn.ko 2024-03-28T15:37:41.724 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_esp.ko 2024-03-28T15:37:41.724 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_hashlimit.ko 2024-03-28T15:37:41.724 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_helper.ko 2024-03-28T15:37:41.724 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_hl.ko 2024-03-28T15:37:41.724 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ipcomp.ko 2024-03-28T15:37:41.724 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_iprange.ko 2024-03-28T15:37:41.724 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ipvs.ko 2024-03-28T15:37:41.724 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_l2tp.ko 2024-03-28T15:37:41.724 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_length.ko 2024-03-28T15:37:41.724 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_limit.ko 2024-03-28T15:37:41.724 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_mac.ko 2024-03-28T15:37:41.724 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_mark.ko 2024-03-28T15:37:41.724 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_multiport.ko 2024-03-28T15:37:41.724 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_nat.ko 2024-03-28T15:37:41.725 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_nfacct.ko 2024-03-28T15:37:41.725 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_osf.ko 2024-03-28T15:37:41.725 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_owner.ko 2024-03-28T15:37:41.725 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_physdev.ko 2024-03-28T15:37:41.725 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_pkttype.ko 2024-03-28T15:37:41.725 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_policy.ko 2024-03-28T15:37:41.725 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_quota.ko 2024-03-28T15:37:41.725 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_rateest.ko 2024-03-28T15:37:41.725 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_realm.ko 2024-03-28T15:37:41.725 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_recent.ko 2024-03-28T15:37:41.725 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_sctp.ko 2024-03-28T15:37:41.725 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_set.ko 2024-03-28T15:37:41.725 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_socket.ko 2024-03-28T15:37:41.725 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_state.ko 2024-03-28T15:37:41.725 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_statistic.ko 2024-03-28T15:37:41.726 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_string.ko 2024-03-28T15:37:41.726 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_tcpmss.ko 2024-03-28T15:37:41.726 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_time.ko 2024-03-28T15:37:41.726 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_u32.ko 2024-03-28T15:37:41.726 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netlink 2024-03-28T15:37:41.726 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netlink/netlink_diag.ko 2024-03-28T15:37:41.726 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/nsh 2024-03-28T15:37:41.726 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/nsh/nsh.ko 2024-03-28T15:37:41.726 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch 2024-03-28T15:37:41.726 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/openvswitch.ko 2024-03-28T15:37:41.726 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-geneve.ko 2024-03-28T15:37:41.726 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-gre.ko 2024-03-28T15:37:41.726 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-vxlan.ko 2024-03-28T15:37:41.726 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/packet 2024-03-28T15:37:41.727 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/packet/af_packet_diag.ko 2024-03-28T15:37:41.727 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds 2024-03-28T15:37:41.727 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds.ko 2024-03-28T15:37:41.727 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds_rdma.ko 2024-03-28T15:37:41.727 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds_tcp.ko 2024-03-28T15:37:41.727 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched 2024-03-28T15:37:41.727 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_csum.ko 2024-03-28T15:37:41.727 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_gact.ko 2024-03-28T15:37:41.727 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_mirred.ko 2024-03-28T15:37:41.727 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_nat.ko 2024-03-28T15:37:41.727 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_pedit.ko 2024-03-28T15:37:41.727 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_police.ko 2024-03-28T15:37:41.727 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_simple.ko 2024-03-28T15:37:41.727 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_skbedit.ko 2024-03-28T15:37:41.728 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_vlan.ko 2024-03-28T15:37:41.728 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_basic.ko 2024-03-28T15:37:41.728 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_bpf.ko 2024-03-28T15:37:41.728 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_flow.ko 2024-03-28T15:37:41.728 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_fw.ko 2024-03-28T15:37:41.728 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_route.ko 2024-03-28T15:37:41.728 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_u32.ko 2024-03-28T15:37:41.728 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_cmp.ko 2024-03-28T15:37:41.728 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_ipset.ko 2024-03-28T15:37:41.728 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_meta.ko 2024-03-28T15:37:41.728 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_nbyte.ko 2024-03-28T15:37:41.728 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_text.ko 2024-03-28T15:37:41.728 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_u32.ko 2024-03-28T15:37:41.728 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_choke.ko 2024-03-28T15:37:41.728 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_codel.ko 2024-03-28T15:37:41.729 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_drr.ko 2024-03-28T15:37:41.729 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_fq.ko 2024-03-28T15:37:41.729 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_gred.ko 2024-03-28T15:37:41.729 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_hfsc.ko 2024-03-28T15:37:41.729 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_hhf.ko 2024-03-28T15:37:41.729 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_htb.ko 2024-03-28T15:37:41.729 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_ingress.ko 2024-03-28T15:37:41.729 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_mqprio.ko 2024-03-28T15:37:41.729 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_mqprio_lib.ko 2024-03-28T15:37:41.729 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_multiq.ko 2024-03-28T15:37:41.729 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_netem.ko 2024-03-28T15:37:41.729 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_pie.ko 2024-03-28T15:37:41.729 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_plug.ko 2024-03-28T15:37:41.729 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_prio.ko 2024-03-28T15:37:41.729 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_qfq.ko 2024-03-28T15:37:41.729 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_red.ko 2024-03-28T15:37:41.730 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_sfb.ko 2024-03-28T15:37:41.730 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_sfq.ko 2024-03-28T15:37:41.730 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_tbf.ko 2024-03-28T15:37:41.730 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_teql.ko 2024-03-28T15:37:41.730 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp 2024-03-28T15:37:41.730 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp/sctp.ko 2024-03-28T15:37:41.730 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp/sctp_diag.ko 2024-03-28T15:37:41.730 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc 2024-03-28T15:37:41.730 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss 2024-03-28T15:37:41.730 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-03-28T15:37:41.730 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-03-28T15:37:41.730 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/sunrpc.ko 2024-03-28T15:37:41.730 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/xprtrdma 2024-03-28T15:37:41.730 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-03-28T15:37:41.730 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc 2024-03-28T15:37:41.731 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc/diag.ko 2024-03-28T15:37:41.731 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc/tipc.ko 2024-03-28T15:37:41.731 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/unix 2024-03-28T15:37:41.731 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/unix/unix_diag.ko 2024-03-28T15:37:41.731 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock 2024-03-28T15:37:41.731 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-03-28T15:37:41.731 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-03-28T15:37:41.731 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock.ko 2024-03-28T15:37:41.731 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock_diag.ko 2024-03-28T15:37:41.731 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock_loopback.ko 2024-03-28T15:37:41.731 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/xfrm 2024-03-28T15:37:41.731 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/xfrm/xfrm_ipcomp.ko 2024-03-28T15:37:41.731 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security 2024-03-28T15:37:41.731 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys 2024-03-28T15:37:41.731 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/encrypted-keys 2024-03-28T15:37:41.732 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-03-28T15:37:41.732 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/trusted-keys 2024-03-28T15:37:41.732 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/trusted-keys/trusted.ko 2024-03-28T15:37:41.732 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/virt 2024-03-28T15:37:41.732 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/virt/lib 2024-03-28T15:37:41.732 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/virt/lib/irqbypass.ko 2024-03-28T15:37:41.732 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.alias 2024-03-28T15:37:41.732 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.alias.bin 2024-03-28T15:37:41.732 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin 2024-03-28T15:37:41.732 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin.alias.bin 2024-03-28T15:37:41.732 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin.bin 2024-03-28T15:37:41.732 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin.modinfo 2024-03-28T15:37:41.732 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.dep 2024-03-28T15:37:41.732 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.dep.bin 2024-03-28T15:37:41.732 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.devname 2024-03-28T15:37:41.732 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.order 2024-03-28T15:37:41.733 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.softdep 2024-03-28T15:37:41.733 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.symbols 2024-03-28T15:37:41.733 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.symbols.bin 2024-03-28T15:37:41.733 INFO:teuthology.orchestra.run.smithi183.stdout:/lib/modules/6.8.0-g1df0c7e8a235/vmlinuz 2024-03-28T15:37:41.736 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-g1df0c7e8a235 2024-03-28T15:37:41.737 DEBUG:teuthology.orchestra.run.smithi183:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-03-28T15:37:41.759 INFO:teuthology.orchestra.run.smithi183.stdout:postinstall scriptlet (using /bin/sh): 2024-03-28T15:37:41.759 INFO:teuthology.orchestra.run.smithi183.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-03-28T15:37:41.759 INFO:teuthology.orchestra.run.smithi183.stdout: /usr/bin/kernel-install add 6.8.0-g1df0c7e8a235 /lib/modules/6.8.0-g1df0c7e8a235/vmlinuz 2024-03-28T15:37:41.760 INFO:teuthology.orchestra.run.smithi183.stdout:fi 2024-03-28T15:37:41.760 INFO:teuthology.orchestra.run.smithi183.stdout:for file in vmlinuz System.map config; do 2024-03-28T15:37:41.760 INFO:teuthology.orchestra.run.smithi183.stdout: if ! cmp --silent "/lib/modules/6.8.0-g1df0c7e8a235/${file}" "/boot/${file}-6.8.0-g1df0c7e8a235"; then 2024-03-28T15:37:41.760 INFO:teuthology.orchestra.run.smithi183.stdout: cp "/lib/modules/6.8.0-g1df0c7e8a235/${file}" "/boot/${file}-6.8.0-g1df0c7e8a235" 2024-03-28T15:37:41.760 INFO:teuthology.orchestra.run.smithi183.stdout: fi 2024-03-28T15:37:41.760 INFO:teuthology.orchestra.run.smithi183.stdout:done 2024-03-28T15:37:41.760 INFO:teuthology.orchestra.run.smithi183.stdout:preuninstall scriptlet (using /bin/sh): 2024-03-28T15:37:41.760 INFO:teuthology.orchestra.run.smithi183.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-03-28T15:37:41.760 INFO:teuthology.orchestra.run.smithi183.stdout:new-kernel-pkg --remove 6.8.0-g1df0c7e8a235 --rminitrd --initrdfile=/boot/initramfs-6.8.0-g1df0c7e8a235.img 2024-03-28T15:37:41.760 INFO:teuthology.orchestra.run.smithi183.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-03-28T15:37:41.760 INFO:teuthology.orchestra.run.smithi183.stdout:kernel-install remove 6.8.0-g1df0c7e8a235 2024-03-28T15:37:41.760 INFO:teuthology.orchestra.run.smithi183.stdout:fi 2024-03-28T15:37:41.760 INFO:teuthology.orchestra.run.smithi183.stdout:postuninstall scriptlet (using /bin/sh): 2024-03-28T15:37:41.761 INFO:teuthology.orchestra.run.smithi183.stdout:if [ -x /sbin/update-bootloader ]; then 2024-03-28T15:37:41.761 INFO:teuthology.orchestra.run.smithi183.stdout:/sbin/update-bootloader --remove 6.8.0-g1df0c7e8a235 2024-03-28T15:37:41.761 INFO:teuthology.orchestra.run.smithi183.stdout:fi 2024-03-28T15:37:41.762 DEBUG:teuthology.orchestra.run.smithi183:> sudo rpm -qi grub2-tools 2024-03-28T15:37:41.837 INFO:teuthology.orchestra.run.smithi183.stdout:Name : grub2-tools 2024-03-28T15:37:41.837 INFO:teuthology.orchestra.run.smithi183.stdout:Epoch : 1 2024-03-28T15:37:41.838 INFO:teuthology.orchestra.run.smithi183.stdout:Version : 2.06 2024-03-28T15:37:41.838 INFO:teuthology.orchestra.run.smithi183.stdout:Release : 68.el9 2024-03-28T15:37:41.838 INFO:teuthology.orchestra.run.smithi183.stdout:Architecture: x86_64 2024-03-28T15:37:41.838 INFO:teuthology.orchestra.run.smithi183.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-28T15:37:41.838 INFO:teuthology.orchestra.run.smithi183.stdout:Group : Unspecified 2024-03-28T15:37:41.838 INFO:teuthology.orchestra.run.smithi183.stdout:Size : 8264254 2024-03-28T15:37:41.838 INFO:teuthology.orchestra.run.smithi183.stdout:License : GPLv3+ 2024-03-28T15:37:41.838 INFO:teuthology.orchestra.run.smithi183.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-28T15:37:41.838 INFO:teuthology.orchestra.run.smithi183.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-28T15:37:41.838 INFO:teuthology.orchestra.run.smithi183.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-28T15:37:41.838 INFO:teuthology.orchestra.run.smithi183.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-28T15:37:41.838 INFO:teuthology.orchestra.run.smithi183.stdout:Packager : builder@centos.org 2024-03-28T15:37:41.838 INFO:teuthology.orchestra.run.smithi183.stdout:Vendor : CentOS 2024-03-28T15:37:41.838 INFO:teuthology.orchestra.run.smithi183.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-28T15:37:41.839 INFO:teuthology.orchestra.run.smithi183.stdout:Summary : Support tools for GRUB. 2024-03-28T15:37:41.839 INFO:teuthology.orchestra.run.smithi183.stdout:Description : 2024-03-28T15:37:41.839 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T15:37:41.839 INFO:teuthology.orchestra.run.smithi183.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-28T15:37:41.839 INFO:teuthology.orchestra.run.smithi183.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-28T15:37:41.839 INFO:teuthology.orchestra.run.smithi183.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-28T15:37:41.839 INFO:teuthology.orchestra.run.smithi183.stdout:hardware devices. 2024-03-28T15:37:41.839 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T15:37:41.839 INFO:teuthology.orchestra.run.smithi183.stdout:This subpackage provides tools for support of all platforms. 2024-03-28T15:37:41.840 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-28T15:37:41.840 INFO:teuthology.task.kernel:Updating grub on smithi183 to boot 6.8.0-g1df0c7e8a235 2024-03-28T15:37:41.840 DEBUG:teuthology.orchestra.run.smithi183:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-28T15:37:42.441 INFO:teuthology.orchestra.run.smithi183.stderr:Generating grub configuration file ... 2024-03-28T15:37:43.759 INFO:teuthology.orchestra.run.smithi183.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-28T15:37:43.797 INFO:teuthology.orchestra.run.smithi183.stderr:done 2024-03-28T15:37:43.800 DEBUG:teuthology.orchestra.run.smithi183:> mktemp 2024-03-28T15:37:43.816 INFO:teuthology.orchestra.run.smithi183.stdout:/tmp/tmp.NafYeVXG9p 2024-03-28T15:37:43.816 DEBUG:teuthology.orchestra.run.smithi183:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.NafYeVXG9p 2024-03-28T15:37:43.884 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 0666 /tmp/tmp.NafYeVXG9p 2024-03-28T15:37:44.035 DEBUG:teuthology.orchestra.remote:smithi183:/tmp/tmp.NafYeVXG9p is 6KB 2024-03-28T15:37:44.084 DEBUG:teuthology.orchestra.run.smithi183:> rm -fr /tmp/tmp.NafYeVXG9p 2024-03-28T15:37:44.098 DEBUG:teuthology.orchestra.run.smithi183:> sudo /bin/ls /boot/loader/entries || true 2024-03-28T15:37:44.166 INFO:teuthology.orchestra.run.smithi183.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-28T15:37:44.166 INFO:teuthology.orchestra.run.smithi183.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-28T15:37:44.166 INFO:teuthology.orchestra.run.smithi183.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-28T15:37:44.166 INFO:teuthology.orchestra.run.smithi183.stdout:15949e3c88704300882482eddfd7baae-6.8.0-g1df0c7e8a235.conf 2024-03-28T15:37:44.168 DEBUG:teuthology.orchestra.run.smithi183:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.8.0-g1df0c7e8a235 2024-03-28T15:37:44.337 DEBUG:teuthology.orchestra.run.smithi183:> sudo shutdown -r now 2024-03-28T15:37:54.434 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-28T15:37:54.434 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-03-28T15:37:54.435 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T15:37:57.098 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-28T15:37:57.098 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-03-28T15:37:57.099 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T15:38:14.354 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-03-28T15:38:14.354 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-03-28T15:38:14.355 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T15:38:29.023 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.67 2024-03-28T15:38:31.582 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-03-28T15:38:38.030 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-03-28T15:38:38.031 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T15:38:40.586 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-03-28T15:38:40.586 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T15:38:56.414 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.67 2024-03-28T15:39:08.422 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-03-28T15:39:08.423 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T15:39:10.289 DEBUG:teuthology.orchestra.run.smithi067:> true 2024-03-28T15:39:10.882 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-03-28T15:39:10.882 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "6.8.0-g1df0c7e8a235"... 2024-03-28T15:39:10.882 DEBUG:teuthology.orchestra.run.smithi067:> uname -r 2024-03-28T15:39:10.921 INFO:teuthology.orchestra.run.smithi067.stdout:6.8.0-g1df0c7e8a235 2024-03-28T15:39:10.922 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-g1df0c7e8a235 vs 6.8.0-g1df0c7e8a235 2024-03-28T15:39:10.922 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-28T15:39:10.922 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-28T15:39:11.923 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-28T15:39:11.923 DEBUG:teuthology.orchestra.run.smithi067:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-28T15:39:12.094 INFO:teuthology.orchestra.run.smithi067.stdout:ttyS1 2024-03-28T15:39:12.145 DEBUG:teuthology.parallel:result is None 2024-03-28T15:39:14.357 DEBUG:teuthology.orchestra.remote:timed out 2024-03-28T15:39:23.358 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-03-28T15:39:23.360 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T15:39:30.897 DEBUG:teuthology.orchestra.run.smithi183:> true 2024-03-28T15:39:31.547 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-03-28T15:39:31.548 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "6.8.0-g1df0c7e8a235"... 2024-03-28T15:39:31.548 DEBUG:teuthology.orchestra.run.smithi183:> uname -r 2024-03-28T15:39:31.590 INFO:teuthology.orchestra.run.smithi183.stdout:6.8.0-g1df0c7e8a235 2024-03-28T15:39:31.590 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-g1df0c7e8a235 vs 6.8.0-g1df0c7e8a235 2024-03-28T15:39:31.590 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-28T15:39:31.590 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-28T15:39:32.591 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-03-28T15:39:32.592 DEBUG:teuthology.orchestra.run.smithi183:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-28T15:39:32.803 INFO:teuthology.orchestra.run.smithi183.stdout:ttyS1 2024-03-28T15:39:32.860 DEBUG:teuthology.parallel:result is None 2024-03-28T15:39:40.589 DEBUG:teuthology.orchestra.remote:timed out 2024-03-28T15:39:52.590 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-03-28T15:39:52.591 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T15:39:53.018 DEBUG:teuthology.orchestra.run.smithi150:> true 2024-03-28T15:39:53.720 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-03-28T15:39:53.720 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "6.8.0-g1df0c7e8a235"... 2024-03-28T15:39:53.720 DEBUG:teuthology.orchestra.run.smithi150:> uname -r 2024-03-28T15:39:53.775 INFO:teuthology.orchestra.run.smithi150.stdout:6.8.0-g1df0c7e8a235 2024-03-28T15:39:53.775 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-g1df0c7e8a235 vs 6.8.0-g1df0c7e8a235 2024-03-28T15:39:53.775 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-28T15:39:53.775 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-28T15:39:54.776 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-28T15:39:54.776 DEBUG:teuthology.orchestra.run.smithi150:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-28T15:39:54.918 INFO:teuthology.orchestra.run.smithi150.stdout:ttyS1 2024-03-28T15:39:54.991 DEBUG:teuthology.parallel:result is None 2024-03-28T15:39:54.992 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-28T15:39:54.999 INFO:teuthology.task.internal:Creating test directory... 2024-03-28T15:39:55.000 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-28T15:39:55.009 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-28T15:39:55.016 DEBUG:teuthology.orchestra.run.smithi183:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-28T15:39:55.076 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-28T15:39:55.083 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-28T15:39:55.096 INFO:teuthology.task.internal:Creating archive directory... 2024-03-28T15:39:55.096 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-28T15:39:55.103 DEBUG:teuthology.orchestra.run.smithi150:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-28T15:39:55.109 DEBUG:teuthology.orchestra.run.smithi183:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-28T15:39:55.178 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-28T15:39:55.189 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-28T15:39:55.189 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-28T15:39:55.196 DEBUG:teuthology.orchestra.run.smithi150:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-28T15:39:55.203 DEBUG:teuthology.orchestra.run.smithi183:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-28T15:39:55.304 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T15:39:55.305 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T15:39:55.306 INFO:teuthology.orchestra.run.smithi183.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T15:39:55.348 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T15:39:55.348 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T15:39:55.349 INFO:teuthology.orchestra.run.smithi183.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T15:39:55.354 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-28T15:39:55.363 INFO:teuthology.task.internal:Configuring sudo... 2024-03-28T15:39:55.364 DEBUG:teuthology.orchestra.run.smithi067:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-28T15:39:55.370 DEBUG:teuthology.orchestra.run.smithi150:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-28T15:39:55.376 DEBUG:teuthology.orchestra.run.smithi183:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-28T15:39:55.475 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-28T15:39:55.484 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-28T15:39:55.484 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-28T15:39:55.491 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-28T15:39:55.498 DEBUG:teuthology.orchestra.run.smithi183:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-28T15:39:55.554 DEBUG:teuthology.orchestra.run.smithi067:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T15:39:55.635 DEBUG:teuthology.orchestra.run.smithi067:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T15:39:55.763 DEBUG:teuthology.orchestra.run.smithi067:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T15:39:55.850 DEBUG:teuthology.orchestra.run.smithi067:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T15:39:55.955 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-28T15:39:55.955 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-28T15:39:56.059 DEBUG:teuthology.orchestra.run.smithi150:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T15:39:56.144 DEBUG:teuthology.orchestra.run.smithi150:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T15:39:56.276 DEBUG:teuthology.orchestra.run.smithi150:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T15:39:56.362 DEBUG:teuthology.orchestra.run.smithi150:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T15:39:56.459 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-28T15:39:56.459 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-28T15:39:56.555 DEBUG:teuthology.orchestra.run.smithi183:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T15:39:56.641 DEBUG:teuthology.orchestra.run.smithi183:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T15:39:56.765 DEBUG:teuthology.orchestra.run.smithi183:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T15:39:56.850 DEBUG:teuthology.orchestra.run.smithi183:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T15:39:56.951 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-28T15:39:56.951 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-28T15:39:57.044 DEBUG:teuthology.orchestra.run.smithi067:> sudo service rsyslog restart 2024-03-28T15:39:57.052 DEBUG:teuthology.orchestra.run.smithi150:> sudo service rsyslog restart 2024-03-28T15:39:57.059 DEBUG:teuthology.orchestra.run.smithi183:> sudo service rsyslog restart 2024-03-28T15:39:57.185 INFO:teuthology.orchestra.run.smithi067.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T15:39:57.186 INFO:teuthology.orchestra.run.smithi150.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T15:39:57.210 INFO:teuthology.orchestra.run.smithi183.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T15:39:57.647 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-28T15:39:57.657 INFO:teuthology.task.internal:Starting timer... 2024-03-28T15:39:57.657 INFO:teuthology.run_tasks:Running task pcp... 2024-03-28T15:39:57.672 INFO:teuthology.run_tasks:Running task selinux... 2024-03-28T15:39:57.686 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-03-28T15:39:57.687 DEBUG:teuthology.orchestra.run.smithi067:> sudo service auditd rotate 2024-03-28T15:39:57.815 INFO:teuthology.orchestra.run.smithi067.stdout:Rotating logs: 2024-03-28T15:39:57.819 DEBUG:teuthology.orchestra.run.smithi150:> sudo service auditd rotate 2024-03-28T15:39:57.943 INFO:teuthology.orchestra.run.smithi150.stdout:Rotating logs: 2024-03-28T15:39:57.948 DEBUG:teuthology.orchestra.run.smithi183:> sudo service auditd rotate 2024-03-28T15:39:58.099 INFO:teuthology.orchestra.run.smithi183.stdout:Rotating logs: 2024-03-28T15:39:58.103 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-28T15:39:58.103 DEBUG:teuthology.orchestra.run.smithi067:> /usr/sbin/getenforce 2024-03-28T15:39:58.157 INFO:teuthology.orchestra.run.smithi067.stdout:Permissive 2024-03-28T15:39:58.158 DEBUG:teuthology.orchestra.run.smithi150:> /usr/sbin/getenforce 2024-03-28T15:39:58.222 INFO:teuthology.orchestra.run.smithi150.stdout:Permissive 2024-03-28T15:39:58.222 DEBUG:teuthology.orchestra.run.smithi183:> /usr/sbin/getenforce 2024-03-28T15:39:58.280 INFO:teuthology.orchestra.run.smithi183.stdout:Permissive 2024-03-28T15:39:58.280 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi067.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi150.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi183.front.sepia.ceph.com': 'permissive'} 2024-03-28T15:39:58.280 DEBUG:teuthology.orchestra.run.smithi067:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T15:39:58.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T15:39:58.364 DEBUG:teuthology.orchestra.run.smithi150:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T15:39:58.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T15:39:58.446 DEBUG:teuthology.orchestra.run.smithi183:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T15:39:58.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T15:39:58.533 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-28T15:39:58.533 DEBUG:teuthology.orchestra.run.smithi067:> sudo /usr/sbin/setenforce permissive 2024-03-28T15:39:58.639 DEBUG:teuthology.orchestra.run.smithi150:> sudo /usr/sbin/setenforce permissive 2024-03-28T15:39:58.736 DEBUG:teuthology.orchestra.run.smithi183:> sudo /usr/sbin/setenforce permissive 2024-03-28T15:39:58.844 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-28T15:39:58.854 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-28T15:39:58.870 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-28T15:39:58.872 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi067.front.sepia.ceph.com,smithi150.front.sepia.ceph.com,smithi183.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-28T15:46:40.041 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi067.front.sepia.ceph.com'), Remote(name='ubuntu@smithi150.front.sepia.ceph.com'), Remote(name='ubuntu@smithi183.front.sepia.ceph.com')] 2024-03-28T15:46:40.042 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-03-28T15:46:40.043 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T15:46:40.194 DEBUG:teuthology.orchestra.run.smithi067:> true 2024-03-28T15:46:40.349 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-03-28T15:46:40.350 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-03-28T15:46:40.351 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T15:46:40.497 DEBUG:teuthology.orchestra.run.smithi150:> true 2024-03-28T15:46:40.651 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-03-28T15:46:40.651 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-03-28T15:46:40.652 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T15:46:40.802 DEBUG:teuthology.orchestra.run.smithi183:> true 2024-03-28T15:46:40.949 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-03-28T15:46:40.950 INFO:teuthology.run_tasks:Running task clock... 2024-03-28T15:46:40.964 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-28T15:46:40.964 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-28T15:46:40.964 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T15:46:40.972 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-28T15:46:40.972 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T15:46:40.979 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-28T15:46:40.979 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T15:46:41.073 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-28T15:46:41.080 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-28T15:46:41.082 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-28T15:46:41.130 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-28T15:46:41.132 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-28T15:46:41.134 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-28T15:46:41.224 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: ntpd: command not found 2024-03-28T15:46:41.231 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: ntpd: command not found 2024-03-28T15:46:41.232 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: ntpd: command not found 2024-03-28T15:46:41.266 INFO:teuthology.orchestra.run.smithi183.stdout:506 Cannot talk to daemon 2024-03-28T15:46:41.273 INFO:teuthology.orchestra.run.smithi150.stdout:506 Cannot talk to daemon 2024-03-28T15:46:41.278 INFO:teuthology.orchestra.run.smithi067.stdout:506 Cannot talk to daemon 2024-03-28T15:46:41.317 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-28T15:46:41.324 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-28T15:46:41.334 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-28T15:46:41.369 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-28T15:46:41.377 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-28T15:46:41.388 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-28T15:46:41.506 INFO:teuthology.orchestra.run.smithi183.stderr:bash: line 1: ntpq: command not found 2024-03-28T15:46:41.508 INFO:teuthology.orchestra.run.smithi150.stderr:bash: line 1: ntpq: command not found 2024-03-28T15:46:41.514 INFO:teuthology.orchestra.run.smithi183.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T15:46:41.514 INFO:teuthology.orchestra.run.smithi183.stdout:=============================================================================== 2024-03-28T15:46:41.514 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T15:46:41.514 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T15:46:41.515 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T15:46:41.515 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T15:46:41.516 INFO:teuthology.orchestra.run.smithi150.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T15:46:41.516 INFO:teuthology.orchestra.run.smithi150.stdout:=============================================================================== 2024-03-28T15:46:41.516 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T15:46:41.516 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T15:46:41.516 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T15:46:41.516 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T15:46:41.528 INFO:teuthology.orchestra.run.smithi067.stderr:bash: line 1: ntpq: command not found 2024-03-28T15:46:41.536 INFO:teuthology.orchestra.run.smithi067.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T15:46:41.536 INFO:teuthology.orchestra.run.smithi067.stdout:=============================================================================== 2024-03-28T15:46:41.537 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T15:46:41.537 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T15:46:41.537 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T15:46:41.537 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T15:46:41.538 INFO:teuthology.run_tasks:Running task install... 2024-03-28T15:46:41.547 DEBUG:teuthology.task.install:project ceph 2024-03-28T15:46:41.547 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033'}} 2024-03-28T15:46:41.547 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033'} 2024-03-28T15:46:41.547 INFO:teuthology.task.install:Using flavor: default 2024-03-28T15:46:41.553 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-28T15:46:41.553 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-28T15:46:41.553 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'tag': None, 'wait_for_package': False} 2024-03-28T15:46:41.554 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T15:46:41.556 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'tag': None, 'wait_for_package': False} 2024-03-28T15:46:41.556 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T15:46:41.557 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'tag': None, 'wait_for_package': False} 2024-03-28T15:46:41.557 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T15:46:41.703 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/ 2024-03-28T15:46:41.703 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2145.g3be9a5c9 2024-03-28T15:46:41.718 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/ 2024-03-28T15:46:41.719 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2145.g3be9a5c9 2024-03-28T15:46:41.722 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/ 2024-03-28T15:46:41.722 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2145.g3be9a5c9 2024-03-28T15:46:41.931 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-28T15:46:41.932 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-28T15:46:41.932 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-28T15:46:41.935 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-28T15:46:41.936 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-28T15:46:41.936 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-28T15:46:41.938 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-28T15:46:41.938 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-28T15:46:41.938 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-28T15:46:42.037 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-28T15:46:42.037 DEBUG:teuthology.orchestra.run.smithi183:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3be9a5c9bc793e11e2800a8c0c696e8b46742033/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-28T15:46:42.038 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-28T15:46:42.038 DEBUG:teuthology.orchestra.run.smithi067:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3be9a5c9bc793e11e2800a8c0c696e8b46742033/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-28T15:46:42.040 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-28T15:46:42.040 DEBUG:teuthology.orchestra.run.smithi150:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3be9a5c9bc793e11e2800a8c0c696e8b46742033/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-28T15:46:42.176 DEBUG:teuthology.orchestra.run.smithi183:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-28T15:46:42.182 DEBUG:teuthology.orchestra.run.smithi150:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-28T15:46:42.183 DEBUG:teuthology.orchestra.run.smithi067:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-28T15:46:42.303 DEBUG:teuthology.orchestra.run.smithi183:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-28T15:46:42.316 DEBUG:teuthology.orchestra.run.smithi150:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-28T15:46:42.327 DEBUG:teuthology.orchestra.run.smithi067:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-28T15:46:42.390 INFO:teuthology.orchestra.run.smithi183.stdout:check_obsoletes = 1 2024-03-28T15:46:42.394 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum clean all 2024-03-28T15:46:42.400 INFO:teuthology.orchestra.run.smithi150.stdout:check_obsoletes = 1 2024-03-28T15:46:42.404 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum clean all 2024-03-28T15:46:42.421 INFO:teuthology.orchestra.run.smithi067.stdout:check_obsoletes = 1 2024-03-28T15:46:42.425 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum clean all 2024-03-28T15:46:42.742 INFO:teuthology.orchestra.run.smithi183.stdout:74 files removed 2024-03-28T15:46:42.761 INFO:teuthology.orchestra.run.smithi150.stdout:74 files removed 2024-03-28T15:46:42.781 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-28T15:46:42.788 INFO:teuthology.orchestra.run.smithi067.stdout:74 files removed 2024-03-28T15:46:42.799 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-28T15:46:42.831 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-28T15:46:43.708 INFO:teuthology.orchestra.run.smithi150.stdout:ceph packages for x86_64 210 kB/s | 83 kB 00:00 2024-03-28T15:46:43.751 INFO:teuthology.orchestra.run.smithi183.stdout:ceph packages for x86_64 181 kB/s | 83 kB 00:00 2024-03-28T15:46:44.043 INFO:teuthology.orchestra.run.smithi067.stdout:ceph packages for x86_64 122 kB/s | 83 kB 00:00 2024-03-28T15:46:44.185 INFO:teuthology.orchestra.run.smithi183.stdout:ceph noarch packages 29 kB/s | 10 kB 00:00 2024-03-28T15:46:44.209 INFO:teuthology.orchestra.run.smithi150.stdout:ceph noarch packages 25 kB/s | 10 kB 00:00 2024-03-28T15:46:44.501 INFO:teuthology.orchestra.run.smithi067.stdout:ceph noarch packages 28 kB/s | 10 kB 00:00 2024-03-28T15:46:44.661 INFO:teuthology.orchestra.run.smithi183.stdout:ceph source packages 4.7 kB/s | 1.9 kB 00:00 2024-03-28T15:46:44.768 INFO:teuthology.orchestra.run.smithi150.stdout:ceph source packages 3.9 kB/s | 1.9 kB 00:00 2024-03-28T15:46:45.094 INFO:teuthology.orchestra.run.smithi067.stdout:ceph source packages 3.6 kB/s | 1.9 kB 00:00 2024-03-28T15:46:45.521 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS-9 - AppStream 9.8 MB/s | 6.7 MB 00:00 2024-03-28T15:46:45.830 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS-9 - AppStream 10 MB/s | 6.7 MB 00:00 2024-03-28T15:46:46.098 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS-9 - AppStream 4.9 MB/s | 6.7 MB 00:01 2024-03-28T15:46:47.839 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS-9 - BaseOS 1.9 MB/s | 2.0 MB 00:01 2024-03-28T15:46:47.873 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS-9 - BaseOS 2.7 MB/s | 2.0 MB 00:00 2024-03-28T15:46:48.106 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS-9 - BaseOS 2.7 MB/s | 2.0 MB 00:00 2024-03-28T15:46:49.544 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - BaseOS 7.9 MB/s | 8.0 MB 00:01 2024-03-28T15:46:49.552 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - BaseOS 6.4 MB/s | 8.0 MB 00:01 2024-03-28T15:46:50.410 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - BaseOS 3.7 MB/s | 8.0 MB 00:02 2024-03-28T15:46:52.733 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - AppStream 9.5 MB/s | 19 MB 00:01 2024-03-28T15:46:53.686 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - AppStream 6.5 MB/s | 19 MB 00:02 2024-03-28T15:46:54.283 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - AppStream 7.1 MB/s | 19 MB 00:02 2024-03-28T15:46:57.578 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - CRB 6.1 MB/s | 6.1 MB 00:01 2024-03-28T15:46:59.071 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - Extras packages 85 kB/s | 16 kB 00:00 2024-03-28T15:46:59.162 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - CRB 4.1 MB/s | 6.1 MB 00:01 2024-03-28T15:47:00.053 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - CRB 3.7 MB/s | 6.1 MB 00:01 2024-03-28T15:47:01.146 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - Extras packages 25 kB/s | 16 kB 00:00 2024-03-28T15:47:01.247 INFO:teuthology.orchestra.run.smithi183.stdout:Extra Packages for Enterprise Linux 10 MB/s | 21 MB 00:02 2024-03-28T15:47:01.629 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - Extras packages 76 kB/s | 16 kB 00:00 2024-03-28T15:47:04.631 INFO:teuthology.orchestra.run.smithi150.stdout:Extra Packages for Enterprise Linux 7.3 MB/s | 21 MB 00:02 2024-03-28T15:47:04.931 INFO:teuthology.orchestra.run.smithi067.stdout:Extra Packages for Enterprise Linux 5.7 MB/s | 21 MB 00:03 2024-03-28T15:47:07.013 INFO:teuthology.orchestra.run.smithi183.stdout:lab-extras 26 kB/s | 1.7 kB 00:00 2024-03-28T15:47:09.757 INFO:teuthology.orchestra.run.smithi183.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T15:47:09.759 INFO:teuthology.orchestra.run.smithi183.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T15:47:09.780 INFO:teuthology.orchestra.run.smithi183.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-28T15:47:09.781 INFO:teuthology.orchestra.run.smithi183.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-28T15:47:09.783 INFO:teuthology.orchestra.run.smithi183.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-28T15:47:09.784 INFO:teuthology.orchestra.run.smithi183.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-28T15:47:09.787 INFO:teuthology.orchestra.run.smithi183.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-28T15:47:09.790 INFO:teuthology.orchestra.run.smithi183.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-28T15:47:09.791 INFO:teuthology.orchestra.run.smithi183.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-28T15:47:09.881 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-28T15:47:09.903 INFO:teuthology.orchestra.run.smithi183.stdout:============================================================================================ 2024-03-28T15:47:09.903 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-03-28T15:47:09.903 INFO:teuthology.orchestra.run.smithi183.stdout:============================================================================================ 2024-03-28T15:47:09.904 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-28T15:47:09.904 INFO:teuthology.orchestra.run.smithi183.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-28T15:47:09.904 INFO:teuthology.orchestra.run.smithi183.stdout: ceph x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 6.6 k 2024-03-28T15:47:09.904 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-base x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.4 M 2024-03-28T15:47:09.904 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 890 k 2024-03-28T15:47:09.904 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 145 k 2024-03-28T15:47:09.904 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 1.5 M 2024-03-28T15:47:09.904 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 139 k 2024-03-28T15:47:09.904 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 1.8 M 2024-03-28T15:47:09.904 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 7.4 M 2024-03-28T15:47:09.904 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-rook noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 49 k 2024-03-28T15:47:09.904 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-radosgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 9.9 M 2024-03-28T15:47:09.904 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-test x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 48 M 2024-03-28T15:47:09.905 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-volume noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 264 k 2024-03-28T15:47:09.905 INFO:teuthology.orchestra.run.smithi183.stdout: cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 748 k 2024-03-28T15:47:09.905 INFO:teuthology.orchestra.run.smithi183.stdout: cephfs-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 220 k 2024-03-28T15:47:09.905 INFO:teuthology.orchestra.run.smithi183.stdout: cephfs-shell x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 27 k 2024-03-28T15:47:09.905 INFO:teuthology.orchestra.run.smithi183.stdout: cephfs-top noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 18 k 2024-03-28T15:47:09.905 INFO:teuthology.orchestra.run.smithi183.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-28T15:47:09.905 INFO:teuthology.orchestra.run.smithi183.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-28T15:47:09.905 INFO:teuthology.orchestra.run.smithi183.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-28T15:47:09.905 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 31 k 2024-03-28T15:47:09.905 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 746 k 2024-03-28T15:47:09.905 INFO:teuthology.orchestra.run.smithi183.stdout: librados-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 127 k 2024-03-28T15:47:09.905 INFO:teuthology.orchestra.run.smithi183.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-28T15:47:09.905 INFO:teuthology.orchestra.run.smithi183.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-28T15:47:09.906 INFO:teuthology.orchestra.run.smithi183.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-28T15:47:09.906 INFO:teuthology.orchestra.run.smithi183.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-03-28T15:47:09.906 INFO:teuthology.orchestra.run.smithi183.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-03-28T15:47:09.906 INFO:teuthology.orchestra.run.smithi183.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-03-28T15:47:09.906 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cephfs x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 162 k 2024-03-28T15:47:09.906 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rados x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 321 k 2024-03-28T15:47:09.906 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 298 k 2024-03-28T15:47:09.906 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 99 k 2024-03-28T15:47:09.906 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 85 k 2024-03-28T15:47:09.906 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-28T15:47:09.906 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-nbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 170 k 2024-03-28T15:47:09.906 INFO:teuthology.orchestra.run.smithi183.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-28T15:47:09.906 INFO:teuthology.orchestra.run.smithi183.stdout:Upgrading: 2024-03-28T15:47:09.907 INFO:teuthology.orchestra.run.smithi183.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-28T15:47:09.907 INFO:teuthology.orchestra.run.smithi183.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-28T15:47:09.907 INFO:teuthology.orchestra.run.smithi183.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-28T15:47:09.907 INFO:teuthology.orchestra.run.smithi183.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-28T15:47:09.907 INFO:teuthology.orchestra.run.smithi183.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-28T15:47:09.907 INFO:teuthology.orchestra.run.smithi183.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-28T15:47:09.907 INFO:teuthology.orchestra.run.smithi183.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-28T15:47:09.907 INFO:teuthology.orchestra.run.smithi183.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-28T15:47:09.907 INFO:teuthology.orchestra.run.smithi183.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-28T15:47:09.907 INFO:teuthology.orchestra.run.smithi183.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-28T15:47:09.907 INFO:teuthology.orchestra.run.smithi183.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-28T15:47:09.907 INFO:teuthology.orchestra.run.smithi183.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-28T15:47:09.907 INFO:teuthology.orchestra.run.smithi183.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-28T15:47:09.907 INFO:teuthology.orchestra.run.smithi183.stdout: librados2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.4 M 2024-03-28T15:47:09.907 INFO:teuthology.orchestra.run.smithi183.stdout: librbd1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-28T15:47:09.908 INFO:teuthology.orchestra.run.smithi183.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-28T15:47:09.908 INFO:teuthology.orchestra.run.smithi183.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-28T15:47:09.908 INFO:teuthology.orchestra.run.smithi183.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-28T15:47:09.908 INFO:teuthology.orchestra.run.smithi183.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-28T15:47:09.908 INFO:teuthology.orchestra.run.smithi183.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-28T15:47:09.908 INFO:teuthology.orchestra.run.smithi183.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-28T15:47:09.908 INFO:teuthology.orchestra.run.smithi183.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-28T15:47:09.908 INFO:teuthology.orchestra.run.smithi183.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-28T15:47:09.908 INFO:teuthology.orchestra.run.smithi183.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-28T15:47:09.908 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-03-28T15:47:09.908 INFO:teuthology.orchestra.run.smithi183.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-28T15:47:09.908 INFO:teuthology.orchestra.run.smithi183.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-28T15:47:09.908 INFO:teuthology.orchestra.run.smithi183.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-28T15:47:09.908 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 20 M 2024-03-28T15:47:09.908 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 34 k 2024-03-28T15:47:09.908 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mds x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 2.3 M 2024-03-28T15:47:09.909 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 248 k 2024-03-28T15:47:09.909 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mon x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 4.7 M 2024-03-28T15:47:09.909 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-osd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 17 M 2024-03-28T15:47:09.909 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 16 k 2024-03-28T15:47:09.909 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-selinux x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 25 k 2024-03-28T15:47:09.909 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-28T15:47:09.909 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-28T15:47:09.909 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-28T15:47:09.909 INFO:teuthology.orchestra.run.smithi183.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-28T15:47:09.909 INFO:teuthology.orchestra.run.smithi183.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-28T15:47:09.909 INFO:teuthology.orchestra.run.smithi183.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-28T15:47:09.909 INFO:teuthology.orchestra.run.smithi183.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-28T15:47:09.909 INFO:teuthology.orchestra.run.smithi183.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-28T15:47:09.909 INFO:teuthology.orchestra.run.smithi183.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-28T15:47:09.910 INFO:teuthology.orchestra.run.smithi183.stdout: libcephsqlite x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 163 k 2024-03-28T15:47:09.910 INFO:teuthology.orchestra.run.smithi183.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-28T15:47:09.910 INFO:teuthology.orchestra.run.smithi183.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-28T15:47:09.910 INFO:teuthology.orchestra.run.smithi183.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-28T15:47:09.910 INFO:teuthology.orchestra.run.smithi183.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-28T15:47:09.910 INFO:teuthology.orchestra.run.smithi183.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-28T15:47:09.910 INFO:teuthology.orchestra.run.smithi183.stdout: libradosstriper1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 501 k 2024-03-28T15:47:09.910 INFO:teuthology.orchestra.run.smithi183.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-28T15:47:09.910 INFO:teuthology.orchestra.run.smithi183.stdout: librgw2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.1 M 2024-03-28T15:47:09.910 INFO:teuthology.orchestra.run.smithi183.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-28T15:47:09.910 INFO:teuthology.orchestra.run.smithi183.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-28T15:47:09.910 INFO:teuthology.orchestra.run.smithi183.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-28T15:47:09.910 INFO:teuthology.orchestra.run.smithi183.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-28T15:47:09.910 INFO:teuthology.orchestra.run.smithi183.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-28T15:47:09.911 INFO:teuthology.orchestra.run.smithi183.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-28T15:47:09.911 INFO:teuthology.orchestra.run.smithi183.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-28T15:47:09.911 INFO:teuthology.orchestra.run.smithi183.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-28T15:47:09.911 INFO:teuthology.orchestra.run.smithi183.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-28T15:47:09.911 INFO:teuthology.orchestra.run.smithi183.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-28T15:47:09.911 INFO:teuthology.orchestra.run.smithi183.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-03-28T15:47:09.911 INFO:teuthology.orchestra.run.smithi183.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-28T15:47:09.911 INFO:teuthology.orchestra.run.smithi183.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-28T15:47:09.911 INFO:teuthology.orchestra.run.smithi183.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-28T15:47:09.911 INFO:teuthology.orchestra.run.smithi183.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-28T15:47:09.911 INFO:teuthology.orchestra.run.smithi183.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-28T15:47:09.911 INFO:teuthology.orchestra.run.smithi183.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-28T15:47:09.911 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-28T15:47:09.911 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ceph-argparse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 45 k 2024-03-28T15:47:09.911 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 131 k 2024-03-28T15:47:09.912 INFO:teuthology.orchestra.run.smithi183.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-28T15:47:09.912 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-28T15:47:09.912 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-28T15:47:09.912 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-28T15:47:09.912 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-28T15:47:09.912 INFO:teuthology.orchestra.run.smithi183.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-28T15:47:09.912 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-28T15:47:09.912 INFO:teuthology.orchestra.run.smithi183.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-28T15:47:09.912 INFO:teuthology.orchestra.run.smithi183.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-28T15:47:09.912 INFO:teuthology.orchestra.run.smithi183.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-28T15:47:09.912 INFO:teuthology.orchestra.run.smithi183.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-28T15:47:09.912 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-28T15:47:09.912 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-28T15:47:09.912 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-28T15:47:09.913 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-28T15:47:09.913 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-28T15:47:09.913 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-28T15:47:09.913 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-28T15:47:09.913 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-28T15:47:09.913 INFO:teuthology.orchestra.run.smithi183.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-28T15:47:09.913 INFO:teuthology.orchestra.run.smithi183.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-28T15:47:09.913 INFO:teuthology.orchestra.run.smithi183.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-28T15:47:09.913 INFO:teuthology.orchestra.run.smithi183.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-28T15:47:09.913 INFO:teuthology.orchestra.run.smithi183.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-28T15:47:09.913 INFO:teuthology.orchestra.run.smithi183.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-28T15:47:09.913 INFO:teuthology.orchestra.run.smithi183.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-28T15:47:09.913 INFO:teuthology.orchestra.run.smithi183.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-28T15:47:09.913 INFO:teuthology.orchestra.run.smithi183.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-28T15:47:09.914 INFO:teuthology.orchestra.run.smithi183.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-28T15:47:09.914 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-28T15:47:09.914 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-28T15:47:09.914 INFO:teuthology.orchestra.run.smithi183.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-28T15:47:09.914 INFO:teuthology.orchestra.run.smithi183.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-28T15:47:09.914 INFO:teuthology.orchestra.run.smithi183.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-28T15:47:09.914 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-28T15:47:09.914 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-28T15:47:09.914 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-28T15:47:09.914 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-28T15:47:09.914 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-28T15:47:09.914 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-28T15:47:09.914 INFO:teuthology.orchestra.run.smithi183.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-28T15:47:09.915 INFO:teuthology.orchestra.run.smithi183.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-28T15:47:09.915 INFO:teuthology.orchestra.run.smithi183.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-28T15:47:09.915 INFO:teuthology.orchestra.run.smithi183.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-28T15:47:09.915 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-28T15:47:09.915 INFO:teuthology.orchestra.run.smithi183.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-28T15:47:09.915 INFO:teuthology.orchestra.run.smithi183.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-28T15:47:09.915 INFO:teuthology.orchestra.run.smithi183.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-28T15:47:09.915 INFO:teuthology.orchestra.run.smithi183.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-28T15:47:09.915 INFO:teuthology.orchestra.run.smithi183.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-28T15:47:09.915 INFO:teuthology.orchestra.run.smithi183.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-28T15:47:09.916 INFO:teuthology.orchestra.run.smithi183.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-28T15:47:09.916 INFO:teuthology.orchestra.run.smithi183.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-28T15:47:09.916 INFO:teuthology.orchestra.run.smithi183.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-28T15:47:09.916 INFO:teuthology.orchestra.run.smithi183.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-28T15:47:09.916 INFO:teuthology.orchestra.run.smithi183.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-28T15:47:09.916 INFO:teuthology.orchestra.run.smithi183.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-28T15:47:09.916 INFO:teuthology.orchestra.run.smithi183.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-28T15:47:09.916 INFO:teuthology.orchestra.run.smithi183.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-28T15:47:09.916 INFO:teuthology.orchestra.run.smithi183.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-28T15:47:09.916 INFO:teuthology.orchestra.run.smithi183.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-28T15:47:09.916 INFO:teuthology.orchestra.run.smithi183.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-28T15:47:09.916 INFO:teuthology.orchestra.run.smithi183.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-28T15:47:09.916 INFO:teuthology.orchestra.run.smithi183.stdout:Installing weak dependencies: 2024-03-28T15:47:09.916 INFO:teuthology.orchestra.run.smithi183.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-28T15:47:09.916 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T15:47:09.917 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-28T15:47:09.917 INFO:teuthology.orchestra.run.smithi183.stdout:============================================================================================ 2024-03-28T15:47:09.917 INFO:teuthology.orchestra.run.smithi183.stdout:Install 150 Packages 2024-03-28T15:47:09.917 INFO:teuthology.orchestra.run.smithi183.stdout:Upgrade 24 Packages 2024-03-28T15:47:09.917 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T15:47:09.917 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 225 M 2024-03-28T15:47:09.917 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-28T15:47:10.438 INFO:teuthology.orchestra.run.smithi150.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-03-28T15:47:10.906 INFO:teuthology.orchestra.run.smithi067.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-03-28T15:47:11.082 INFO:teuthology.orchestra.run.smithi183.stdout:(1/174): ceph-19.0.0-2145.g3be9a5c9.el9.x86_64. 22 kB/s | 6.6 kB 00:00 2024-03-28T15:47:11.548 INFO:teuthology.orchestra.run.smithi183.stdout:(2/174): ceph-fuse-19.0.0-2145.g3be9a5c9.el9.x8 1.9 MB/s | 890 kB 00:00 2024-03-28T15:47:11.648 INFO:teuthology.orchestra.run.smithi183.stdout:(3/174): ceph-base-19.0.0-2145.g3be9a5c9.el9.x8 6.2 MB/s | 5.4 MB 00:00 2024-03-28T15:47:11.682 INFO:teuthology.orchestra.run.smithi183.stdout:(4/174): ceph-immutable-object-cache-19.0.0-214 1.1 MB/s | 145 kB 00:00 2024-03-28T15:47:12.191 INFO:teuthology.orchestra.run.smithi183.stdout:(5/174): ceph-mgr-19.0.0-2145.g3be9a5c9.el9.x86 2.9 MB/s | 1.5 MB 00:00 2024-03-28T15:47:12.249 INFO:teuthology.orchestra.run.smithi183.stdout:(6/174): ceph-mds-19.0.0-2145.g3be9a5c9.el9.x86 3.9 MB/s | 2.3 MB 00:00 2024-03-28T15:47:12.733 INFO:teuthology.orchestra.run.smithi183.stdout:(7/174): ceph-common-19.0.0-2145.g3be9a5c9.el9. 10 MB/s | 20 MB 00:01 2024-03-28T15:47:13.019 INFO:teuthology.orchestra.run.smithi150.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T15:47:13.020 INFO:teuthology.orchestra.run.smithi150.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T15:47:13.040 INFO:teuthology.orchestra.run.smithi150.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-28T15:47:13.041 INFO:teuthology.orchestra.run.smithi150.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-28T15:47:13.042 INFO:teuthology.orchestra.run.smithi150.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-28T15:47:13.043 INFO:teuthology.orchestra.run.smithi150.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-28T15:47:13.046 INFO:teuthology.orchestra.run.smithi150.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-28T15:47:13.049 INFO:teuthology.orchestra.run.smithi150.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-28T15:47:13.051 INFO:teuthology.orchestra.run.smithi150.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-28T15:47:13.138 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-03-28T15:47:13.159 INFO:teuthology.orchestra.run.smithi150.stdout:============================================================================================ 2024-03-28T15:47:13.160 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2024-03-28T15:47:13.160 INFO:teuthology.orchestra.run.smithi150.stdout:============================================================================================ 2024-03-28T15:47:13.160 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-03-28T15:47:13.160 INFO:teuthology.orchestra.run.smithi150.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-28T15:47:13.160 INFO:teuthology.orchestra.run.smithi150.stdout: ceph x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 6.6 k 2024-03-28T15:47:13.160 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-base x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.4 M 2024-03-28T15:47:13.160 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 890 k 2024-03-28T15:47:13.160 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 145 k 2024-03-28T15:47:13.160 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 1.5 M 2024-03-28T15:47:13.160 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 139 k 2024-03-28T15:47:13.160 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 1.8 M 2024-03-28T15:47:13.160 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 7.4 M 2024-03-28T15:47:13.160 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-rook noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 49 k 2024-03-28T15:47:13.160 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-radosgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 9.9 M 2024-03-28T15:47:13.161 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-test x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 48 M 2024-03-28T15:47:13.161 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-volume noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 264 k 2024-03-28T15:47:13.161 INFO:teuthology.orchestra.run.smithi150.stdout: cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 748 k 2024-03-28T15:47:13.161 INFO:teuthology.orchestra.run.smithi150.stdout: cephfs-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 220 k 2024-03-28T15:47:13.161 INFO:teuthology.orchestra.run.smithi150.stdout: cephfs-shell x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 27 k 2024-03-28T15:47:13.161 INFO:teuthology.orchestra.run.smithi150.stdout: cephfs-top noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 18 k 2024-03-28T15:47:13.161 INFO:teuthology.orchestra.run.smithi150.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-28T15:47:13.161 INFO:teuthology.orchestra.run.smithi150.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-28T15:47:13.161 INFO:teuthology.orchestra.run.smithi150.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-28T15:47:13.161 INFO:teuthology.orchestra.run.smithi150.stdout: libcephfs-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 31 k 2024-03-28T15:47:13.161 INFO:teuthology.orchestra.run.smithi150.stdout: libcephfs2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 746 k 2024-03-28T15:47:13.161 INFO:teuthology.orchestra.run.smithi150.stdout: librados-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 127 k 2024-03-28T15:47:13.161 INFO:teuthology.orchestra.run.smithi150.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-28T15:47:13.161 INFO:teuthology.orchestra.run.smithi150.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-28T15:47:13.161 INFO:teuthology.orchestra.run.smithi150.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-28T15:47:13.162 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-03-28T15:47:13.162 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-03-28T15:47:13.162 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-03-28T15:47:13.162 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cephfs x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 162 k 2024-03-28T15:47:13.162 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rados x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 321 k 2024-03-28T15:47:13.162 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 298 k 2024-03-28T15:47:13.162 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 99 k 2024-03-28T15:47:13.162 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 85 k 2024-03-28T15:47:13.162 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-28T15:47:13.162 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-nbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 170 k 2024-03-28T15:47:13.162 INFO:teuthology.orchestra.run.smithi150.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-28T15:47:13.162 INFO:teuthology.orchestra.run.smithi150.stdout:Upgrading: 2024-03-28T15:47:13.162 INFO:teuthology.orchestra.run.smithi150.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-28T15:47:13.162 INFO:teuthology.orchestra.run.smithi150.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-28T15:47:13.163 INFO:teuthology.orchestra.run.smithi150.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-28T15:47:13.163 INFO:teuthology.orchestra.run.smithi150.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-28T15:47:13.163 INFO:teuthology.orchestra.run.smithi150.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-28T15:47:13.163 INFO:teuthology.orchestra.run.smithi150.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-28T15:47:13.163 INFO:teuthology.orchestra.run.smithi150.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-28T15:47:13.163 INFO:teuthology.orchestra.run.smithi150.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-28T15:47:13.163 INFO:teuthology.orchestra.run.smithi150.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-28T15:47:13.163 INFO:teuthology.orchestra.run.smithi150.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-28T15:47:13.163 INFO:teuthology.orchestra.run.smithi150.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-28T15:47:13.163 INFO:teuthology.orchestra.run.smithi150.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-28T15:47:13.163 INFO:teuthology.orchestra.run.smithi150.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-28T15:47:13.163 INFO:teuthology.orchestra.run.smithi150.stdout: librados2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.4 M 2024-03-28T15:47:13.163 INFO:teuthology.orchestra.run.smithi150.stdout: librbd1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-28T15:47:13.163 INFO:teuthology.orchestra.run.smithi150.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-28T15:47:13.163 INFO:teuthology.orchestra.run.smithi150.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-28T15:47:13.164 INFO:teuthology.orchestra.run.smithi150.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-28T15:47:13.164 INFO:teuthology.orchestra.run.smithi150.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-28T15:47:13.164 INFO:teuthology.orchestra.run.smithi150.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-28T15:47:13.164 INFO:teuthology.orchestra.run.smithi150.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-28T15:47:13.164 INFO:teuthology.orchestra.run.smithi150.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-28T15:47:13.164 INFO:teuthology.orchestra.run.smithi150.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-28T15:47:13.164 INFO:teuthology.orchestra.run.smithi150.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-28T15:47:13.164 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2024-03-28T15:47:13.164 INFO:teuthology.orchestra.run.smithi150.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-28T15:47:13.164 INFO:teuthology.orchestra.run.smithi150.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-28T15:47:13.164 INFO:teuthology.orchestra.run.smithi150.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-28T15:47:13.164 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 20 M 2024-03-28T15:47:13.164 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 34 k 2024-03-28T15:47:13.164 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mds x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 2.3 M 2024-03-28T15:47:13.165 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 248 k 2024-03-28T15:47:13.165 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mon x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 4.7 M 2024-03-28T15:47:13.165 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-osd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 17 M 2024-03-28T15:47:13.165 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 16 k 2024-03-28T15:47:13.165 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-selinux x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 25 k 2024-03-28T15:47:13.165 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-28T15:47:13.165 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-28T15:47:13.165 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-28T15:47:13.165 INFO:teuthology.orchestra.run.smithi150.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-28T15:47:13.165 INFO:teuthology.orchestra.run.smithi150.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-28T15:47:13.165 INFO:teuthology.orchestra.run.smithi150.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-28T15:47:13.165 INFO:teuthology.orchestra.run.smithi150.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-28T15:47:13.165 INFO:teuthology.orchestra.run.smithi150.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-28T15:47:13.165 INFO:teuthology.orchestra.run.smithi150.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-28T15:47:13.165 INFO:teuthology.orchestra.run.smithi150.stdout: libcephsqlite x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 163 k 2024-03-28T15:47:13.165 INFO:teuthology.orchestra.run.smithi150.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-28T15:47:13.166 INFO:teuthology.orchestra.run.smithi150.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-28T15:47:13.166 INFO:teuthology.orchestra.run.smithi150.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-28T15:47:13.166 INFO:teuthology.orchestra.run.smithi150.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-28T15:47:13.166 INFO:teuthology.orchestra.run.smithi150.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-28T15:47:13.166 INFO:teuthology.orchestra.run.smithi150.stdout: libradosstriper1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 501 k 2024-03-28T15:47:13.166 INFO:teuthology.orchestra.run.smithi150.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-28T15:47:13.166 INFO:teuthology.orchestra.run.smithi150.stdout: librgw2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.1 M 2024-03-28T15:47:13.166 INFO:teuthology.orchestra.run.smithi150.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-28T15:47:13.166 INFO:teuthology.orchestra.run.smithi150.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-28T15:47:13.166 INFO:teuthology.orchestra.run.smithi150.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-28T15:47:13.166 INFO:teuthology.orchestra.run.smithi150.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-28T15:47:13.166 INFO:teuthology.orchestra.run.smithi150.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-28T15:47:13.166 INFO:teuthology.orchestra.run.smithi150.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-28T15:47:13.166 INFO:teuthology.orchestra.run.smithi150.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-28T15:47:13.166 INFO:teuthology.orchestra.run.smithi150.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-28T15:47:13.167 INFO:teuthology.orchestra.run.smithi150.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-28T15:47:13.167 INFO:teuthology.orchestra.run.smithi150.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-28T15:47:13.167 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-03-28T15:47:13.167 INFO:teuthology.orchestra.run.smithi150.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-28T15:47:13.167 INFO:teuthology.orchestra.run.smithi150.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-28T15:47:13.167 INFO:teuthology.orchestra.run.smithi150.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-28T15:47:13.167 INFO:teuthology.orchestra.run.smithi150.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-28T15:47:13.167 INFO:teuthology.orchestra.run.smithi150.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-28T15:47:13.167 INFO:teuthology.orchestra.run.smithi150.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-28T15:47:13.167 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-28T15:47:13.167 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ceph-argparse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 45 k 2024-03-28T15:47:13.167 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 131 k 2024-03-28T15:47:13.167 INFO:teuthology.orchestra.run.smithi150.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-28T15:47:13.167 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-28T15:47:13.167 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-28T15:47:13.168 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-28T15:47:13.168 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-28T15:47:13.168 INFO:teuthology.orchestra.run.smithi150.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-28T15:47:13.168 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-28T15:47:13.168 INFO:teuthology.orchestra.run.smithi150.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-28T15:47:13.168 INFO:teuthology.orchestra.run.smithi150.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-28T15:47:13.168 INFO:teuthology.orchestra.run.smithi150.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-28T15:47:13.168 INFO:teuthology.orchestra.run.smithi150.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-28T15:47:13.168 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-28T15:47:13.168 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-28T15:47:13.168 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-28T15:47:13.168 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-28T15:47:13.168 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-28T15:47:13.168 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-28T15:47:13.168 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-28T15:47:13.170 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-28T15:47:13.170 INFO:teuthology.orchestra.run.smithi150.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-28T15:47:13.170 INFO:teuthology.orchestra.run.smithi150.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-28T15:47:13.170 INFO:teuthology.orchestra.run.smithi150.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-28T15:47:13.170 INFO:teuthology.orchestra.run.smithi150.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-28T15:47:13.170 INFO:teuthology.orchestra.run.smithi150.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-28T15:47:13.170 INFO:teuthology.orchestra.run.smithi150.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-28T15:47:13.170 INFO:teuthology.orchestra.run.smithi150.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-28T15:47:13.170 INFO:teuthology.orchestra.run.smithi150.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-28T15:47:13.170 INFO:teuthology.orchestra.run.smithi150.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-28T15:47:13.170 INFO:teuthology.orchestra.run.smithi150.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-28T15:47:13.171 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-28T15:47:13.171 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-28T15:47:13.171 INFO:teuthology.orchestra.run.smithi150.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-28T15:47:13.171 INFO:teuthology.orchestra.run.smithi150.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-28T15:47:13.171 INFO:teuthology.orchestra.run.smithi150.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-28T15:47:13.171 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-28T15:47:13.171 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-28T15:47:13.171 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-28T15:47:13.171 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-28T15:47:13.171 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-28T15:47:13.171 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-28T15:47:13.171 INFO:teuthology.orchestra.run.smithi150.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-28T15:47:13.171 INFO:teuthology.orchestra.run.smithi150.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-28T15:47:13.171 INFO:teuthology.orchestra.run.smithi150.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-28T15:47:13.171 INFO:teuthology.orchestra.run.smithi150.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-28T15:47:13.172 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-28T15:47:13.172 INFO:teuthology.orchestra.run.smithi150.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-28T15:47:13.172 INFO:teuthology.orchestra.run.smithi150.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-28T15:47:13.172 INFO:teuthology.orchestra.run.smithi150.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-28T15:47:13.172 INFO:teuthology.orchestra.run.smithi150.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-28T15:47:13.172 INFO:teuthology.orchestra.run.smithi150.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-28T15:47:13.172 INFO:teuthology.orchestra.run.smithi150.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-28T15:47:13.172 INFO:teuthology.orchestra.run.smithi150.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-28T15:47:13.172 INFO:teuthology.orchestra.run.smithi150.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-28T15:47:13.172 INFO:teuthology.orchestra.run.smithi150.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-28T15:47:13.172 INFO:teuthology.orchestra.run.smithi150.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-28T15:47:13.172 INFO:teuthology.orchestra.run.smithi150.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-28T15:47:13.172 INFO:teuthology.orchestra.run.smithi150.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-28T15:47:13.172 INFO:teuthology.orchestra.run.smithi150.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-28T15:47:13.172 INFO:teuthology.orchestra.run.smithi150.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-28T15:47:13.173 INFO:teuthology.orchestra.run.smithi150.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-28T15:47:13.173 INFO:teuthology.orchestra.run.smithi150.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-28T15:47:13.173 INFO:teuthology.orchestra.run.smithi150.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-28T15:47:13.174 INFO:teuthology.orchestra.run.smithi150.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-28T15:47:13.174 INFO:teuthology.orchestra.run.smithi150.stdout:Installing weak dependencies: 2024-03-28T15:47:13.174 INFO:teuthology.orchestra.run.smithi150.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-28T15:47:13.174 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-28T15:47:13.174 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-03-28T15:47:13.174 INFO:teuthology.orchestra.run.smithi150.stdout:============================================================================================ 2024-03-28T15:47:13.175 INFO:teuthology.orchestra.run.smithi150.stdout:Install 150 Packages 2024-03-28T15:47:13.175 INFO:teuthology.orchestra.run.smithi150.stdout:Upgrade 24 Packages 2024-03-28T15:47:13.175 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-28T15:47:13.176 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 225 M 2024-03-28T15:47:13.176 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-03-28T15:47:13.575 INFO:teuthology.orchestra.run.smithi183.stdout:(8/174): ceph-radosgw-19.0.0-2145.g3be9a5c9.el9 12 MB/s | 9.9 MB 00:00 2024-03-28T15:47:13.595 INFO:teuthology.orchestra.run.smithi067.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T15:47:13.596 INFO:teuthology.orchestra.run.smithi067.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T15:47:13.615 INFO:teuthology.orchestra.run.smithi067.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-28T15:47:13.616 INFO:teuthology.orchestra.run.smithi067.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-28T15:47:13.617 INFO:teuthology.orchestra.run.smithi067.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-28T15:47:13.618 INFO:teuthology.orchestra.run.smithi067.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-28T15:47:13.621 INFO:teuthology.orchestra.run.smithi067.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-28T15:47:13.624 INFO:teuthology.orchestra.run.smithi067.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-28T15:47:13.625 INFO:teuthology.orchestra.run.smithi067.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-28T15:47:13.674 INFO:teuthology.orchestra.run.smithi183.stdout:(9/174): ceph-selinux-19.0.0-2145.g3be9a5c9.el9 256 kB/s | 25 kB 00:00 2024-03-28T15:47:13.719 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-03-28T15:47:13.746 INFO:teuthology.orchestra.run.smithi067.stdout:============================================================================================ 2024-03-28T15:47:13.746 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-03-28T15:47:13.746 INFO:teuthology.orchestra.run.smithi067.stdout:============================================================================================ 2024-03-28T15:47:13.746 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-03-28T15:47:13.746 INFO:teuthology.orchestra.run.smithi067.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-28T15:47:13.746 INFO:teuthology.orchestra.run.smithi067.stdout: ceph x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 6.6 k 2024-03-28T15:47:13.746 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-base x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.4 M 2024-03-28T15:47:13.747 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 890 k 2024-03-28T15:47:13.747 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 145 k 2024-03-28T15:47:13.747 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 1.5 M 2024-03-28T15:47:13.747 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 139 k 2024-03-28T15:47:13.747 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 1.8 M 2024-03-28T15:47:13.747 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 7.4 M 2024-03-28T15:47:13.747 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-rook noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 49 k 2024-03-28T15:47:13.747 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-radosgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 9.9 M 2024-03-28T15:47:13.747 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-test x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 48 M 2024-03-28T15:47:13.747 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-volume noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 264 k 2024-03-28T15:47:13.747 INFO:teuthology.orchestra.run.smithi067.stdout: cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 748 k 2024-03-28T15:47:13.747 INFO:teuthology.orchestra.run.smithi067.stdout: cephfs-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 220 k 2024-03-28T15:47:13.747 INFO:teuthology.orchestra.run.smithi067.stdout: cephfs-shell x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 27 k 2024-03-28T15:47:13.747 INFO:teuthology.orchestra.run.smithi067.stdout: cephfs-top noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 18 k 2024-03-28T15:47:13.748 INFO:teuthology.orchestra.run.smithi067.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-28T15:47:13.748 INFO:teuthology.orchestra.run.smithi067.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-28T15:47:13.748 INFO:teuthology.orchestra.run.smithi067.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-28T15:47:13.748 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 31 k 2024-03-28T15:47:13.748 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 746 k 2024-03-28T15:47:13.748 INFO:teuthology.orchestra.run.smithi067.stdout: librados-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 127 k 2024-03-28T15:47:13.748 INFO:teuthology.orchestra.run.smithi067.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-28T15:47:13.748 INFO:teuthology.orchestra.run.smithi067.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-28T15:47:13.748 INFO:teuthology.orchestra.run.smithi067.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-28T15:47:13.748 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-03-28T15:47:13.748 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-03-28T15:47:13.748 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-03-28T15:47:13.748 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cephfs x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 162 k 2024-03-28T15:47:13.749 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rados x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 321 k 2024-03-28T15:47:13.749 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 298 k 2024-03-28T15:47:13.749 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 99 k 2024-03-28T15:47:13.749 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 85 k 2024-03-28T15:47:13.749 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-28T15:47:13.749 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-nbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 170 k 2024-03-28T15:47:13.749 INFO:teuthology.orchestra.run.smithi067.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-28T15:47:13.749 INFO:teuthology.orchestra.run.smithi067.stdout:Upgrading: 2024-03-28T15:47:13.749 INFO:teuthology.orchestra.run.smithi067.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-28T15:47:13.749 INFO:teuthology.orchestra.run.smithi067.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-28T15:47:13.749 INFO:teuthology.orchestra.run.smithi067.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-28T15:47:13.749 INFO:teuthology.orchestra.run.smithi067.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-28T15:47:13.749 INFO:teuthology.orchestra.run.smithi067.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-28T15:47:13.749 INFO:teuthology.orchestra.run.smithi067.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-28T15:47:13.750 INFO:teuthology.orchestra.run.smithi067.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-28T15:47:13.750 INFO:teuthology.orchestra.run.smithi067.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-28T15:47:13.750 INFO:teuthology.orchestra.run.smithi067.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-28T15:47:13.750 INFO:teuthology.orchestra.run.smithi067.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-28T15:47:13.750 INFO:teuthology.orchestra.run.smithi067.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-28T15:47:13.750 INFO:teuthology.orchestra.run.smithi067.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-28T15:47:13.750 INFO:teuthology.orchestra.run.smithi067.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-28T15:47:13.750 INFO:teuthology.orchestra.run.smithi067.stdout: librados2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.4 M 2024-03-28T15:47:13.750 INFO:teuthology.orchestra.run.smithi067.stdout: librbd1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-28T15:47:13.750 INFO:teuthology.orchestra.run.smithi067.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-28T15:47:13.750 INFO:teuthology.orchestra.run.smithi067.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-28T15:47:13.750 INFO:teuthology.orchestra.run.smithi067.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-28T15:47:13.750 INFO:teuthology.orchestra.run.smithi067.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-28T15:47:13.750 INFO:teuthology.orchestra.run.smithi067.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-28T15:47:13.751 INFO:teuthology.orchestra.run.smithi067.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-28T15:47:13.751 INFO:teuthology.orchestra.run.smithi067.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-28T15:47:13.751 INFO:teuthology.orchestra.run.smithi067.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-28T15:47:13.751 INFO:teuthology.orchestra.run.smithi067.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-28T15:47:13.751 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-03-28T15:47:13.751 INFO:teuthology.orchestra.run.smithi067.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-28T15:47:13.751 INFO:teuthology.orchestra.run.smithi067.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-28T15:47:13.751 INFO:teuthology.orchestra.run.smithi067.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-28T15:47:13.751 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 20 M 2024-03-28T15:47:13.752 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 34 k 2024-03-28T15:47:13.752 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mds x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 2.3 M 2024-03-28T15:47:13.752 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 248 k 2024-03-28T15:47:13.752 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mon x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 4.7 M 2024-03-28T15:47:13.752 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-osd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 17 M 2024-03-28T15:47:13.752 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 16 k 2024-03-28T15:47:13.752 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-selinux x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 25 k 2024-03-28T15:47:13.752 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-28T15:47:13.752 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-28T15:47:13.752 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-28T15:47:13.752 INFO:teuthology.orchestra.run.smithi067.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-28T15:47:13.752 INFO:teuthology.orchestra.run.smithi067.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-28T15:47:13.752 INFO:teuthology.orchestra.run.smithi067.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-28T15:47:13.753 INFO:teuthology.orchestra.run.smithi067.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-28T15:47:13.753 INFO:teuthology.orchestra.run.smithi067.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-28T15:47:13.753 INFO:teuthology.orchestra.run.smithi067.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-28T15:47:13.753 INFO:teuthology.orchestra.run.smithi067.stdout: libcephsqlite x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 163 k 2024-03-28T15:47:13.753 INFO:teuthology.orchestra.run.smithi067.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-28T15:47:13.753 INFO:teuthology.orchestra.run.smithi067.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-28T15:47:13.753 INFO:teuthology.orchestra.run.smithi067.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-28T15:47:13.753 INFO:teuthology.orchestra.run.smithi067.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-28T15:47:13.753 INFO:teuthology.orchestra.run.smithi067.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-28T15:47:13.753 INFO:teuthology.orchestra.run.smithi067.stdout: libradosstriper1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 501 k 2024-03-28T15:47:13.753 INFO:teuthology.orchestra.run.smithi067.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-28T15:47:13.753 INFO:teuthology.orchestra.run.smithi067.stdout: librgw2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.1 M 2024-03-28T15:47:13.753 INFO:teuthology.orchestra.run.smithi067.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-28T15:47:13.754 INFO:teuthology.orchestra.run.smithi067.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-28T15:47:13.754 INFO:teuthology.orchestra.run.smithi067.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-28T15:47:13.754 INFO:teuthology.orchestra.run.smithi067.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-28T15:47:13.754 INFO:teuthology.orchestra.run.smithi067.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-28T15:47:13.754 INFO:teuthology.orchestra.run.smithi067.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-28T15:47:13.754 INFO:teuthology.orchestra.run.smithi067.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-28T15:47:13.754 INFO:teuthology.orchestra.run.smithi067.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-28T15:47:13.754 INFO:teuthology.orchestra.run.smithi067.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-28T15:47:13.754 INFO:teuthology.orchestra.run.smithi067.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-28T15:47:13.754 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-03-28T15:47:13.754 INFO:teuthology.orchestra.run.smithi067.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-28T15:47:13.754 INFO:teuthology.orchestra.run.smithi067.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-28T15:47:13.754 INFO:teuthology.orchestra.run.smithi067.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-28T15:47:13.754 INFO:teuthology.orchestra.run.smithi067.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-28T15:47:13.754 INFO:teuthology.orchestra.run.smithi067.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-28T15:47:13.755 INFO:teuthology.orchestra.run.smithi067.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-28T15:47:13.755 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-28T15:47:13.755 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-argparse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 45 k 2024-03-28T15:47:13.755 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 131 k 2024-03-28T15:47:13.755 INFO:teuthology.orchestra.run.smithi067.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-28T15:47:13.755 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-28T15:47:13.755 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-28T15:47:13.755 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-28T15:47:13.755 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-28T15:47:13.755 INFO:teuthology.orchestra.run.smithi067.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-28T15:47:13.755 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-28T15:47:13.755 INFO:teuthology.orchestra.run.smithi067.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-28T15:47:13.755 INFO:teuthology.orchestra.run.smithi067.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-28T15:47:13.755 INFO:teuthology.orchestra.run.smithi067.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-28T15:47:13.755 INFO:teuthology.orchestra.run.smithi067.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-28T15:47:13.756 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-28T15:47:13.756 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-28T15:47:13.756 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-28T15:47:13.756 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-28T15:47:13.756 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-28T15:47:13.756 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-28T15:47:13.756 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-28T15:47:13.756 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-28T15:47:13.756 INFO:teuthology.orchestra.run.smithi067.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-28T15:47:13.756 INFO:teuthology.orchestra.run.smithi067.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-28T15:47:13.756 INFO:teuthology.orchestra.run.smithi067.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-28T15:47:13.756 INFO:teuthology.orchestra.run.smithi067.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-28T15:47:13.756 INFO:teuthology.orchestra.run.smithi067.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-28T15:47:13.756 INFO:teuthology.orchestra.run.smithi067.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-28T15:47:13.756 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-28T15:47:13.757 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-28T15:47:13.757 INFO:teuthology.orchestra.run.smithi067.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-28T15:47:13.757 INFO:teuthology.orchestra.run.smithi067.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-28T15:47:13.757 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-28T15:47:13.757 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-28T15:47:13.757 INFO:teuthology.orchestra.run.smithi067.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-28T15:47:13.757 INFO:teuthology.orchestra.run.smithi067.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-28T15:47:13.757 INFO:teuthology.orchestra.run.smithi067.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-28T15:47:13.757 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-28T15:47:13.757 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-28T15:47:13.757 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-28T15:47:13.757 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-28T15:47:13.757 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-28T15:47:13.757 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-28T15:47:13.757 INFO:teuthology.orchestra.run.smithi067.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-28T15:47:13.758 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-28T15:47:13.758 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-28T15:47:13.758 INFO:teuthology.orchestra.run.smithi067.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-28T15:47:13.758 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-28T15:47:13.758 INFO:teuthology.orchestra.run.smithi067.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-28T15:47:13.758 INFO:teuthology.orchestra.run.smithi067.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-28T15:47:13.758 INFO:teuthology.orchestra.run.smithi067.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-28T15:47:13.758 INFO:teuthology.orchestra.run.smithi067.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-28T15:47:13.758 INFO:teuthology.orchestra.run.smithi067.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-28T15:47:13.758 INFO:teuthology.orchestra.run.smithi067.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-28T15:47:13.758 INFO:teuthology.orchestra.run.smithi067.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-28T15:47:13.758 INFO:teuthology.orchestra.run.smithi067.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-28T15:47:13.758 INFO:teuthology.orchestra.run.smithi067.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-28T15:47:13.758 INFO:teuthology.orchestra.run.smithi067.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-28T15:47:13.759 INFO:teuthology.orchestra.run.smithi067.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-28T15:47:13.759 INFO:teuthology.orchestra.run.smithi067.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-28T15:47:13.759 INFO:teuthology.orchestra.run.smithi067.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-28T15:47:13.759 INFO:teuthology.orchestra.run.smithi067.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-28T15:47:13.759 INFO:teuthology.orchestra.run.smithi067.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-28T15:47:13.760 INFO:teuthology.orchestra.run.smithi067.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-28T15:47:13.760 INFO:teuthology.orchestra.run.smithi067.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-28T15:47:13.760 INFO:teuthology.orchestra.run.smithi067.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-28T15:47:13.761 INFO:teuthology.orchestra.run.smithi067.stdout:Installing weak dependencies: 2024-03-28T15:47:13.761 INFO:teuthology.orchestra.run.smithi067.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-28T15:47:13.761 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T15:47:13.761 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-03-28T15:47:13.761 INFO:teuthology.orchestra.run.smithi067.stdout:============================================================================================ 2024-03-28T15:47:13.761 INFO:teuthology.orchestra.run.smithi067.stdout:Install 150 Packages 2024-03-28T15:47:13.761 INFO:teuthology.orchestra.run.smithi067.stdout:Upgrade 24 Packages 2024-03-28T15:47:13.761 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T15:47:13.763 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 225 M 2024-03-28T15:47:13.763 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-03-28T15:47:13.900 INFO:teuthology.orchestra.run.smithi183.stdout:(10/174): ceph-osd-19.0.0-2145.g3be9a5c9.el9.x8 10 MB/s | 17 MB 00:01 2024-03-28T15:47:13.973 INFO:teuthology.orchestra.run.smithi150.stdout:(1/174): ceph-19.0.0-2145.g3be9a5c9.el9.x86_64. 21 kB/s | 6.6 kB 00:00 2024-03-28T15:47:13.975 INFO:teuthology.orchestra.run.smithi183.stdout:(11/174): cephfs-mirror-19.0.0-2145.g3be9a5c9.e 2.9 MB/s | 220 kB 00:00 2024-03-28T15:47:14.075 INFO:teuthology.orchestra.run.smithi183.stdout:(12/174): cephfs-shell-19.0.0-2145.g3be9a5c9.el 272 kB/s | 27 kB 00:00 2024-03-28T15:47:14.176 INFO:teuthology.orchestra.run.smithi183.stdout:(13/174): libcephfs-devel-19.0.0-2145.g3be9a5c9 316 kB/s | 31 kB 00:00 2024-03-28T15:47:14.368 INFO:teuthology.orchestra.run.smithi183.stdout:(14/174): libcephfs2-19.0.0-2145.g3be9a5c9.el9. 3.8 MB/s | 746 kB 00:00 2024-03-28T15:47:14.476 INFO:teuthology.orchestra.run.smithi183.stdout:(15/174): libcephsqlite-19.0.0-2145.g3be9a5c9.e 1.5 MB/s | 163 kB 00:00 2024-03-28T15:47:14.507 INFO:teuthology.orchestra.run.smithi067.stdout:(1/174): ceph-19.0.0-2145.g3be9a5c9.el9.x86_64. 16 kB/s | 6.6 kB 00:00 2024-03-28T15:47:14.601 INFO:teuthology.orchestra.run.smithi183.stdout:(16/174): ceph-mon-19.0.0-2145.g3be9a5c9.el9.x8 1.9 MB/s | 4.7 MB 00:02 2024-03-28T15:47:14.635 INFO:teuthology.orchestra.run.smithi183.stdout:(17/174): librados-devel-19.0.0-2145.g3be9a5c9. 804 kB/s | 127 kB 00:00 2024-03-28T15:47:14.899 INFO:teuthology.orchestra.run.smithi150.stdout:(2/174): ceph-fuse-19.0.0-2145.g3be9a5c9.el9.x8 964 kB/s | 890 kB 00:00 2024-03-28T15:47:15.043 INFO:teuthology.orchestra.run.smithi183.stdout:(18/174): libradosstriper1-19.0.0-2145.g3be9a5c 1.1 MB/s | 501 kB 00:00 2024-03-28T15:47:15.124 INFO:teuthology.orchestra.run.smithi150.stdout:(3/174): ceph-immutable-object-cache-19.0.0-214 648 kB/s | 145 kB 00:00 2024-03-28T15:47:15.152 INFO:teuthology.orchestra.run.smithi183.stdout:(19/174): python3-ceph-argparse-19.0.0-2145.g3b 420 kB/s | 45 kB 00:00 2024-03-28T15:47:15.419 INFO:teuthology.orchestra.run.smithi183.stdout:(20/174): librgw2-19.0.0-2145.g3be9a5c9.el9.x86 6.6 MB/s | 5.1 MB 00:00 2024-03-28T15:47:15.444 INFO:teuthology.orchestra.run.smithi183.stdout:(21/174): python3-ceph-common-19.0.0-2145.g3be9 451 kB/s | 131 kB 00:00 2024-03-28T15:47:15.503 INFO:teuthology.orchestra.run.smithi183.stdout:(22/174): python3-cephfs-19.0.0-2145.g3be9a5c9. 1.9 MB/s | 162 kB 00:00 2024-03-28T15:47:15.611 INFO:teuthology.orchestra.run.smithi183.stdout:(23/174): python3-rbd-19.0.0-2145.g3be9a5c9.el9 2.7 MB/s | 298 kB 00:00 2024-03-28T15:47:15.632 INFO:teuthology.orchestra.run.smithi150.stdout:(4/174): ceph-base-19.0.0-2145.g3be9a5c9.el9.x8 2.7 MB/s | 5.4 MB 00:01 2024-03-28T15:47:15.687 INFO:teuthology.orchestra.run.smithi183.stdout:(24/174): python3-rados-19.0.0-2145.g3be9a5c9.e 1.3 MB/s | 321 kB 00:00 2024-03-28T15:47:15.691 INFO:teuthology.orchestra.run.smithi067.stdout:(2/174): ceph-fuse-19.0.0-2145.g3be9a5c9.el9.x8 752 kB/s | 890 kB 00:01 2024-03-28T15:47:15.712 INFO:teuthology.orchestra.run.smithi183.stdout:(25/174): python3-rgw-19.0.0-2145.g3be9a5c9.el9 1.0 MB/s | 99 kB 00:00 2024-03-28T15:47:15.787 INFO:teuthology.orchestra.run.smithi183.stdout:(26/174): rbd-fuse-19.0.0-2145.g3be9a5c9.el9.x8 860 kB/s | 85 kB 00:00 2024-03-28T15:47:15.891 INFO:teuthology.orchestra.run.smithi067.stdout:(3/174): ceph-immutable-object-cache-19.0.0-214 733 kB/s | 145 kB 00:00 2024-03-28T15:47:16.012 INFO:teuthology.orchestra.run.smithi183.stdout:(27/174): rbd-nbd-19.0.0-2145.g3be9a5c9.el9.x86 761 kB/s | 170 kB 00:00 2024-03-28T15:47:16.121 INFO:teuthology.orchestra.run.smithi183.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2145.g 315 kB/s | 34 kB 00:00 2024-03-28T15:47:16.141 INFO:teuthology.orchestra.run.smithi150.stdout:(5/174): ceph-mgr-19.0.0-2145.g3be9a5c9.el9.x86 2.9 MB/s | 1.5 MB 00:00 2024-03-28T15:47:16.246 INFO:teuthology.orchestra.run.smithi183.stdout:(29/174): rbd-mirror-19.0.0-2145.g3be9a5c9.el9. 5.8 MB/s | 3.1 MB 00:00 2024-03-28T15:47:16.296 INFO:teuthology.orchestra.run.smithi183.stdout:(30/174): ceph-mgr-cephadm-19.0.0-2145.g3be9a5c 795 kB/s | 139 kB 00:00 2024-03-28T15:47:16.555 INFO:teuthology.orchestra.run.smithi183.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2145.g3be9a 5.8 MB/s | 1.8 MB 00:00 2024-03-28T15:47:16.664 INFO:teuthology.orchestra.run.smithi183.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2145.g3b 2.3 MB/s | 248 kB 00:00 2024-03-28T15:47:16.772 INFO:teuthology.orchestra.run.smithi183.stdout:(33/174): ceph-mgr-rook-19.0.0-2145.g3be9a5c9.e 458 kB/s | 49 kB 00:00 2024-03-28T15:47:16.873 INFO:teuthology.orchestra.run.smithi183.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2145.g3 166 kB/s | 16 kB 00:00 2024-03-28T15:47:16.973 INFO:teuthology.orchestra.run.smithi183.stdout:(35/174): ceph-volume-19.0.0-2145.g3be9a5c9.el9 2.6 MB/s | 264 kB 00:00 2024-03-28T15:47:17.116 INFO:teuthology.orchestra.run.smithi150.stdout:(6/174): ceph-mds-19.0.0-2145.g3be9a5c9.el9.x86 1.2 MB/s | 2.3 MB 00:01 2024-03-28T15:47:17.165 INFO:teuthology.orchestra.run.smithi183.stdout:(36/174): cephadm-19.0.0-2145.g3be9a5c9.el9.noa 3.8 MB/s | 748 kB 00:00 2024-03-28T15:47:17.182 INFO:teuthology.orchestra.run.smithi067.stdout:(4/174): ceph-mds-19.0.0-2145.g3be9a5c9.el9.x86 1.8 MB/s | 2.3 MB 00:01 2024-03-28T15:47:17.249 INFO:teuthology.orchestra.run.smithi183.stdout:(37/174): cephfs-top-19.0.0-2145.g3be9a5c9.el9. 221 kB/s | 18 kB 00:00 2024-03-28T15:47:17.456 INFO:teuthology.orchestra.run.smithi150.stdout:(7/174): ceph-mon-19.0.0-2145.g3be9a5c9.el9.x86 3.6 MB/s | 4.7 MB 00:01 2024-03-28T15:47:17.533 INFO:teuthology.orchestra.run.smithi067.stdout:(5/174): ceph-mgr-19.0.0-2145.g3be9a5c9.el9.x86 4.3 MB/s | 1.5 MB 00:00 2024-03-28T15:47:17.791 INFO:teuthology.orchestra.run.smithi183.stdout:(38/174): ceph-test-19.0.0-2145.g3be9a5c9.el9.x 12 MB/s | 48 MB 00:04 2024-03-28T15:47:17.924 INFO:teuthology.orchestra.run.smithi183.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 982 kB/s | 662 kB 00:00 2024-03-28T15:47:17.966 INFO:teuthology.orchestra.run.smithi183.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 5.3 MB/s | 920 kB 00:00 2024-03-28T15:47:17.991 INFO:teuthology.orchestra.run.smithi183.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.8 MB/s | 106 kB 00:00 2024-03-28T15:47:18.025 INFO:teuthology.orchestra.run.smithi183.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 5.3 MB/s | 307 kB 00:00 2024-03-28T15:47:18.050 INFO:teuthology.orchestra.run.smithi183.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 573 kB/s | 32 kB 00:00 2024-03-28T15:47:18.225 INFO:teuthology.orchestra.run.smithi067.stdout:(6/174): ceph-base-19.0.0-2145.g3be9a5c9.el9.x8 1.3 MB/s | 5.4 MB 00:04 2024-03-28T15:47:18.243 INFO:teuthology.orchestra.run.smithi183.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 14 MB/s | 3.0 MB 00:00 2024-03-28T15:47:18.301 INFO:teuthology.orchestra.run.smithi183.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 73 kB/s | 17 kB 00:00 2024-03-28T15:47:18.325 INFO:teuthology.orchestra.run.smithi067.stdout:(7/174): ceph-mon-19.0.0-2145.g3be9a5c9.el9.x86 5.9 MB/s | 4.7 MB 00:00 2024-03-28T15:47:18.359 INFO:teuthology.orchestra.run.smithi183.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 177 kB/s | 10 kB 00:00 2024-03-28T15:47:18.384 INFO:teuthology.orchestra.run.smithi183.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.9 MB/s | 159 kB 00:00 2024-03-28T15:47:18.426 INFO:teuthology.orchestra.run.smithi183.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 680 kB/s | 44 kB 00:00 2024-03-28T15:47:18.468 INFO:teuthology.orchestra.run.smithi183.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 7.8 MB/s | 661 kB 00:00 2024-03-28T15:47:18.510 INFO:teuthology.orchestra.run.smithi183.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 7.3 MB/s | 578 kB 00:00 2024-03-28T15:47:18.535 INFO:teuthology.orchestra.run.smithi183.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 756 kB/s | 46 kB 00:00 2024-03-28T15:47:18.577 INFO:teuthology.orchestra.run.smithi183.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.2 MB/s | 282 kB 00:00 2024-03-28T15:47:18.611 INFO:teuthology.orchestra.run.smithi183.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 2.4 MB/s | 187 kB 00:00 2024-03-28T15:47:18.644 INFO:teuthology.orchestra.run.smithi183.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.3 MB/s | 217 kB 00:00 2024-03-28T15:47:18.703 INFO:teuthology.orchestra.run.smithi183.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 374 kB/s | 34 kB 00:00 2024-03-28T15:47:18.817 INFO:teuthology.orchestra.run.smithi150.stdout:(8/174): ceph-radosgw-19.0.0-2145.g3be9a5c9.el9 7.2 MB/s | 9.9 MB 00:01 2024-03-28T15:47:18.865 INFO:teuthology.orchestra.run.smithi183.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 22 MB/s | 4.7 MB 00:00 2024-03-28T15:47:18.904 INFO:teuthology.orchestra.run.smithi183.stdout:(57/174): protobuf-3.14.0-13.el9.x86_64.rpm 5.4 MB/s | 1.0 MB 00:00 2024-03-28T15:47:18.929 INFO:teuthology.orchestra.run.smithi183.stdout:(58/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 78 kB 00:00 2024-03-28T15:47:19.084 INFO:teuthology.orchestra.run.smithi150.stdout:(9/174): ceph-common-19.0.0-2145.g3be9a5c9.el9. 3.8 MB/s | 20 MB 00:05 2024-03-28T15:47:19.109 INFO:teuthology.orchestra.run.smithi150.stdout:(10/174): ceph-selinux-19.0.0-2145.g3be9a5c9.el 87 kB/s | 25 kB 00:00 2024-03-28T15:47:19.154 INFO:teuthology.orchestra.run.smithi183.stdout:(59/174): python3-babel-2.9.1-2.el9.noarch.rpm 23 MB/s | 5.8 MB 00:00 2024-03-28T15:47:19.179 INFO:teuthology.orchestra.run.smithi183.stdout:(60/174): python3-jsonpatch-1.21-16.el9.noarch. 103 kB/s | 24 kB 00:00 2024-03-28T15:47:19.184 INFO:teuthology.orchestra.run.smithi150.stdout:(11/174): cephfs-mirror-19.0.0-2145.g3be9a5c9.e 2.9 MB/s | 220 kB 00:00 2024-03-28T15:47:19.213 INFO:teuthology.orchestra.run.smithi183.stdout:(61/174): python3-jsonpointer-2.0-4.el9.noarch. 563 kB/s | 18 kB 00:00 2024-03-28T15:47:19.234 INFO:teuthology.orchestra.run.smithi150.stdout:(12/174): cephfs-shell-19.0.0-2145.g3be9a5c9.el 551 kB/s | 27 kB 00:00 2024-03-28T15:47:19.238 INFO:teuthology.orchestra.run.smithi183.stdout:(62/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.5 MB/s | 144 kB 00:00 2024-03-28T15:47:19.242 INFO:teuthology.orchestra.run.smithi067.stdout:(8/174): ceph-radosgw-19.0.0-2145.g3be9a5c9.el9 11 MB/s | 9.9 MB 00:00 2024-03-28T15:47:19.264 INFO:teuthology.orchestra.run.smithi183.stdout:(63/174): python3-markupsafe-1.1.1-12.el9.x86_6 652 kB/s | 32 kB 00:00 2024-03-28T15:47:19.285 INFO:teuthology.orchestra.run.smithi150.stdout:(13/174): libcephfs-devel-19.0.0-2145.g3be9a5c9 637 kB/s | 31 kB 00:00 2024-03-28T15:47:19.300 INFO:teuthology.orchestra.run.smithi067.stdout:(9/174): ceph-selinux-19.0.0-2145.g3be9a5c9.el9 439 kB/s | 25 kB 00:00 2024-03-28T15:47:19.442 INFO:teuthology.orchestra.run.smithi183.stdout:(64/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 24 MB/s | 4.8 MB 00:00 2024-03-28T15:47:19.473 INFO:teuthology.orchestra.run.smithi183.stdout:(65/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.1 MB/s | 217 kB 00:00 2024-03-28T15:47:19.506 INFO:teuthology.orchestra.run.smithi183.stdout:(66/174): python3-oauthlib-3.1.1-5.el9.noarch.r 2.7 MB/s | 174 kB 00:00 2024-03-28T15:47:19.527 INFO:teuthology.orchestra.run.smithi150.stdout:(14/174): libcephfs2-19.0.0-2145.g3be9a5c9.el9. 3.0 MB/s | 746 kB 00:00 2024-03-28T15:47:19.540 INFO:teuthology.orchestra.run.smithi183.stdout:(67/174): python3-packaging-20.9-5.el9.noarch.r 1.0 MB/s | 69 kB 00:00 2024-03-28T15:47:19.565 INFO:teuthology.orchestra.run.smithi183.stdout:(68/174): python3-prettytable-0.7.2-27.el9.noar 722 kB/s | 41 kB 00:00 2024-03-28T15:47:19.599 INFO:teuthology.orchestra.run.smithi183.stdout:(69/174): python3-protobuf-3.14.0-13.el9.noarch 4.0 MB/s | 238 kB 00:00 2024-03-28T15:47:19.618 INFO:teuthology.orchestra.run.smithi150.stdout:(15/174): libcephsqlite-19.0.0-2145.g3be9a5c9.e 1.8 MB/s | 163 kB 00:00 2024-03-28T15:47:19.624 INFO:teuthology.orchestra.run.smithi183.stdout:(70/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.3 MB/s | 132 kB 00:00 2024-03-28T15:47:19.649 INFO:teuthology.orchestra.run.smithi183.stdout:(71/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.3 MB/s | 211 kB 00:00 2024-03-28T15:47:19.674 INFO:teuthology.orchestra.run.smithi183.stdout:(72/174): python3-pytz-2021.1-5.el9.noarch.rpm 997 kB/s | 47 kB 00:00 2024-03-28T15:47:19.700 INFO:teuthology.orchestra.run.smithi183.stdout:(73/174): python3-requests-oauthlib-1.3.0-12.el 875 kB/s | 43 kB 00:00 2024-03-28T15:47:19.702 INFO:teuthology.orchestra.run.smithi150.stdout:(16/174): librados-devel-19.0.0-2145.g3be9a5c9. 1.5 MB/s | 127 kB 00:00 2024-03-28T15:47:19.877 INFO:teuthology.orchestra.run.smithi150.stdout:(17/174): libradosstriper1-19.0.0-2145.g3be9a5c 2.8 MB/s | 501 kB 00:00 2024-03-28T15:47:19.900 INFO:teuthology.orchestra.run.smithi183.stdout:(74/174): python3-toml-0.10.2-6.el9.noarch.rpm 208 kB/s | 39 kB 00:00 2024-03-28T15:47:20.100 INFO:teuthology.orchestra.run.smithi183.stdout:(75/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 37 MB/s | 16 MB 00:00 2024-03-28T15:47:20.125 INFO:teuthology.orchestra.run.smithi183.stdout:(76/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 191 kB/s | 41 kB 00:00 2024-03-28T15:47:20.159 INFO:teuthology.orchestra.run.smithi183.stdout:(77/174): socat-1.7.4.1-5.el9.x86_64.rpm 5.1 MB/s | 300 kB 00:00 2024-03-28T15:47:20.176 INFO:teuthology.orchestra.run.smithi183.stdout:(78/174): uuid-1.6.2-55.el9.x86_64.rpm 1.2 MB/s | 56 kB 00:00 2024-03-28T15:47:20.193 INFO:teuthology.orchestra.run.smithi183.stdout:(79/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.9 MB/s | 63 kB 00:00 2024-03-28T15:47:20.243 INFO:teuthology.orchestra.run.smithi183.stdout:(80/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 672 kB/s | 44 kB 00:00 2024-03-28T15:47:20.268 INFO:teuthology.orchestra.run.smithi183.stdout:(81/174): fuse-2.9.9-15.el9.x86_64.rpm 1.0 MB/s | 79 kB 00:00 2024-03-28T15:47:20.302 INFO:teuthology.orchestra.run.smithi183.stdout:(82/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.1 MB/s | 241 kB 00:00 2024-03-28T15:47:20.352 INFO:teuthology.orchestra.run.smithi183.stdout:(83/174): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2024-03-28T15:47:20.378 INFO:teuthology.orchestra.run.smithi183.stdout:(84/174): python3-ply-3.11-14.el9.noarch.rpm 1.6 MB/s | 103 kB 00:00 2024-03-28T15:47:20.403 INFO:teuthology.orchestra.run.smithi183.stdout:(85/174): python3-pycparser-2.20-6.el9.noarch.r 2.5 MB/s | 124 kB 00:00 2024-03-28T15:47:20.437 INFO:teuthology.orchestra.run.smithi183.stdout:(86/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.2 MB/s | 187 kB 00:00 2024-03-28T15:47:20.570 INFO:teuthology.orchestra.run.smithi183.stdout:(87/174): jq-1.6-16.el9.x86_64.rpm 1.1 MB/s | 187 kB 00:00 2024-03-28T15:47:20.646 INFO:teuthology.orchestra.run.smithi183.stdout:(88/174): libquadmath-11.4.1-3.el9.x86_64.rpm 2.6 MB/s | 195 kB 00:00 2024-03-28T15:47:20.687 INFO:teuthology.orchestra.run.smithi183.stdout:(89/174): libgfortran-11.4.1-3.el9.x86_64.rpm 3.1 MB/s | 803 kB 00:00 2024-03-28T15:47:20.829 INFO:teuthology.orchestra.run.smithi183.stdout:(90/174): ceph-mgr-diskprediction-local-19.0.0- 1.6 MB/s | 7.4 MB 00:04 2024-03-28T15:47:20.854 INFO:teuthology.orchestra.run.smithi183.stdout:(91/174): python3-requests-2.25.1-8.el9.noarch. 600 kB/s | 125 kB 00:00 2024-03-28T15:47:21.446 INFO:teuthology.orchestra.run.smithi183.stdout:(92/174): elfutils-libelf-devel-0.190-2.el9.x86 28 kB/s | 21 kB 00:00 2024-03-28T15:47:21.472 INFO:teuthology.orchestra.run.smithi183.stdout:(93/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 40 kB/s | 25 kB 00:00 2024-03-28T15:47:21.619 INFO:teuthology.orchestra.run.smithi150.stdout:(18/174): librgw2-19.0.0-2145.g3be9a5c9.el9.x86 3.0 MB/s | 5.1 MB 00:01 2024-03-28T15:47:21.703 INFO:teuthology.orchestra.run.smithi150.stdout:(19/174): python3-ceph-argparse-19.0.0-2145.g3b 548 kB/s | 45 kB 00:00 2024-03-28T15:47:21.786 INFO:teuthology.orchestra.run.smithi150.stdout:(20/174): python3-ceph-common-19.0.0-2145.g3be9 1.6 MB/s | 131 kB 00:00 2024-03-28T15:47:21.870 INFO:teuthology.orchestra.run.smithi150.stdout:(21/174): python3-cephfs-19.0.0-2145.g3be9a5c9. 1.9 MB/s | 162 kB 00:00 2024-03-28T15:47:22.028 INFO:teuthology.orchestra.run.smithi150.stdout:(22/174): python3-rados-19.0.0-2145.g3be9a5c9.e 2.0 MB/s | 321 kB 00:00 2024-03-28T15:47:22.179 INFO:teuthology.orchestra.run.smithi150.stdout:(23/174): python3-rbd-19.0.0-2145.g3be9a5c9.el9 2.0 MB/s | 298 kB 00:00 2024-03-28T15:47:22.256 INFO:teuthology.orchestra.run.smithi150.stdout:(24/174): python3-rgw-19.0.0-2145.g3be9a5c9.el9 1.3 MB/s | 99 kB 00:00 2024-03-28T15:47:22.321 INFO:teuthology.orchestra.run.smithi150.stdout:(25/174): rbd-fuse-19.0.0-2145.g3be9a5c9.el9.x8 1.3 MB/s | 85 kB 00:00 2024-03-28T15:47:22.617 INFO:teuthology.orchestra.run.smithi067.stdout:(10/174): ceph-test-19.0.0-2145.g3be9a5c9.el9.x 14 MB/s | 48 MB 00:03 2024-03-28T15:47:22.747 INFO:teuthology.orchestra.run.smithi150.stdout:(26/174): rbd-mirror-19.0.0-2145.g3be9a5c9.el9. 7.3 MB/s | 3.1 MB 00:00 2024-03-28T15:47:22.805 INFO:teuthology.orchestra.run.smithi150.stdout:(27/174): rbd-nbd-19.0.0-2145.g3be9a5c9.el9.x86 2.9 MB/s | 170 kB 00:00 2024-03-28T15:47:22.834 INFO:teuthology.orchestra.run.smithi067.stdout:(11/174): cephfs-mirror-19.0.0-2145.g3be9a5c9.e 1.0 MB/s | 220 kB 00:00 2024-03-28T15:47:22.855 INFO:teuthology.orchestra.run.smithi150.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2145.g 691 kB/s | 34 kB 00:00 2024-03-28T15:47:22.901 INFO:teuthology.orchestra.run.smithi067.stdout:(12/174): cephfs-shell-19.0.0-2145.g3be9a5c9.el 412 kB/s | 27 kB 00:00 2024-03-28T15:47:22.914 INFO:teuthology.orchestra.run.smithi150.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2145.g3be9a5c 2.4 MB/s | 139 kB 00:00 2024-03-28T15:47:22.968 INFO:teuthology.orchestra.run.smithi067.stdout:(13/174): libcephfs-devel-19.0.0-2145.g3be9a5c9 475 kB/s | 31 kB 00:00 2024-03-28T15:47:23.118 INFO:teuthology.orchestra.run.smithi067.stdout:(14/174): libcephfs2-19.0.0-2145.g3be9a5c9.el9. 4.9 MB/s | 746 kB 00:00 2024-03-28T15:47:23.231 INFO:teuthology.orchestra.run.smithi150.stdout:(30/174): ceph-mgr-dashboard-19.0.0-2145.g3be9a 5.7 MB/s | 1.8 MB 00:00 2024-03-28T15:47:23.252 INFO:teuthology.orchestra.run.smithi067.stdout:(15/174): libcephsqlite-19.0.0-2145.g3be9a5c9.e 1.2 MB/s | 163 kB 00:00 2024-03-28T15:47:23.319 INFO:teuthology.orchestra.run.smithi067.stdout:(16/174): librados-devel-19.0.0-2145.g3be9a5c9. 1.9 MB/s | 127 kB 00:00 2024-03-28T15:47:23.428 INFO:teuthology.orchestra.run.smithi067.stdout:(17/174): libradosstriper1-19.0.0-2145.g3be9a5c 4.5 MB/s | 501 kB 00:00 2024-03-28T15:47:23.447 INFO:teuthology.orchestra.run.smithi183.stdout:(94/174): postgresql-contrib-13.14-1.el9.x86_64 461 kB/s | 911 kB 00:01 2024-03-28T15:47:23.870 INFO:teuthology.orchestra.run.smithi067.stdout:(18/174): librgw2-19.0.0-2145.g3be9a5c9.el9.x86 12 MB/s | 5.1 MB 00:00 2024-03-28T15:47:23.945 INFO:teuthology.orchestra.run.smithi067.stdout:(19/174): python3-ceph-argparse-19.0.0-2145.g3b 610 kB/s | 45 kB 00:00 2024-03-28T15:47:24.028 INFO:teuthology.orchestra.run.smithi067.stdout:(20/174): python3-ceph-common-19.0.0-2145.g3be9 1.6 MB/s | 131 kB 00:00 2024-03-28T15:47:24.112 INFO:teuthology.orchestra.run.smithi067.stdout:(21/174): python3-cephfs-19.0.0-2145.g3be9a5c9. 1.9 MB/s | 162 kB 00:00 2024-03-28T15:47:24.196 INFO:teuthology.orchestra.run.smithi067.stdout:(22/174): python3-rados-19.0.0-2145.g3be9a5c9.e 3.8 MB/s | 321 kB 00:00 2024-03-28T15:47:24.279 INFO:teuthology.orchestra.run.smithi067.stdout:(23/174): python3-rbd-19.0.0-2145.g3be9a5c9.el9 3.5 MB/s | 298 kB 00:00 2024-03-28T15:47:24.280 INFO:teuthology.orchestra.run.smithi183.stdout:(95/174): postgresql-private-libs-13.14-1.el9.x 160 kB/s | 133 kB 00:00 2024-03-28T15:47:24.339 INFO:teuthology.orchestra.run.smithi183.stdout:(96/174): postgresql-13.14-1.el9.x86_64.rpm 563 kB/s | 1.6 MB 00:02 2024-03-28T15:47:24.346 INFO:teuthology.orchestra.run.smithi067.stdout:(24/174): python3-rgw-19.0.0-2145.g3be9a5c9.el9 1.5 MB/s | 99 kB 00:00 2024-03-28T15:47:24.422 INFO:teuthology.orchestra.run.smithi067.stdout:(25/174): rbd-fuse-19.0.0-2145.g3be9a5c9.el9.x8 1.1 MB/s | 85 kB 00:00 2024-03-28T15:47:24.558 INFO:teuthology.orchestra.run.smithi150.stdout:(31/174): ceph-mgr-diskprediction-local-19.0.0- 5.6 MB/s | 7.4 MB 00:01 2024-03-28T15:47:24.665 INFO:teuthology.orchestra.run.smithi150.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2145.g3b 2.3 MB/s | 248 kB 00:00 2024-03-28T15:47:24.672 INFO:teuthology.orchestra.run.smithi067.stdout:(26/174): rbd-mirror-19.0.0-2145.g3be9a5c9.el9. 12 MB/s | 3.1 MB 00:00 2024-03-28T15:47:24.732 INFO:teuthology.orchestra.run.smithi150.stdout:(33/174): ceph-mgr-rook-19.0.0-2145.g3be9a5c9.e 749 kB/s | 49 kB 00:00 2024-03-28T15:47:24.755 INFO:teuthology.orchestra.run.smithi067.stdout:(27/174): rbd-nbd-19.0.0-2145.g3be9a5c9.el9.x86 2.0 MB/s | 170 kB 00:00 2024-03-28T15:47:24.782 INFO:teuthology.orchestra.run.smithi150.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2145.g3 335 kB/s | 16 kB 00:00 2024-03-28T15:47:24.823 INFO:teuthology.orchestra.run.smithi067.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2145.g 516 kB/s | 34 kB 00:00 2024-03-28T15:47:24.889 INFO:teuthology.orchestra.run.smithi067.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2145.g3be9a5c 2.1 MB/s | 139 kB 00:00 2024-03-28T15:47:24.999 INFO:teuthology.orchestra.run.smithi150.stdout:(35/174): ceph-osd-19.0.0-2145.g3be9a5c9.el9.x8 2.1 MB/s | 17 MB 00:07 2024-03-28T15:47:25.033 INFO:teuthology.orchestra.run.smithi150.stdout:(36/174): ceph-volume-19.0.0-2145.g3be9a5c9.el9 1.0 MB/s | 264 kB 00:00 2024-03-28T15:47:25.065 INFO:teuthology.orchestra.run.smithi067.stdout:(30/174): ceph-mgr-dashboard-19.0.0-2145.g3be9a 10 MB/s | 1.8 MB 00:00 2024-03-28T15:47:25.083 INFO:teuthology.orchestra.run.smithi150.stdout:(37/174): cephfs-top-19.0.0-2145.g3be9a5c9.el9. 370 kB/s | 18 kB 00:00 2024-03-28T15:47:25.275 INFO:teuthology.orchestra.run.smithi150.stdout:(38/174): automake-1.16.2-8.el9.noarch.rpm 3.4 MB/s | 662 kB 00:00 2024-03-28T15:47:25.298 INFO:teuthology.orchestra.run.smithi183.stdout:(97/174): python3-devel-3.9.17-2.el9.x86_64.rpm 256 kB/s | 245 kB 00:00 2024-03-28T15:47:25.350 INFO:teuthology.orchestra.run.smithi150.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 12 MB/s | 920 kB 00:00 2024-03-28T15:47:25.392 INFO:teuthology.orchestra.run.smithi150.stdout:(40/174): cephadm-19.0.0-2145.g3be9a5c9.el9.noa 1.9 MB/s | 748 kB 00:00 2024-03-28T15:47:25.417 INFO:teuthology.orchestra.run.smithi150.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.6 MB/s | 106 kB 00:00 2024-03-28T15:47:25.459 INFO:teuthology.orchestra.run.smithi150.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 4.6 MB/s | 307 kB 00:00 2024-03-28T15:47:25.476 INFO:teuthology.orchestra.run.smithi150.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 571 kB/s | 32 kB 00:00 2024-03-28T15:47:25.577 INFO:teuthology.orchestra.run.smithi150.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 26 MB/s | 3.0 MB 00:00 2024-03-28T15:47:25.593 INFO:teuthology.orchestra.run.smithi150.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 185 kB/s | 17 kB 00:00 2024-03-28T15:47:25.610 INFO:teuthology.orchestra.run.smithi150.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 310 kB/s | 10 kB 00:00 2024-03-28T15:47:25.644 INFO:teuthology.orchestra.run.smithi150.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.2 MB/s | 159 kB 00:00 2024-03-28T15:47:25.662 INFO:teuthology.orchestra.run.smithi150.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 904 kB/s | 44 kB 00:00 2024-03-28T15:47:25.730 INFO:teuthology.orchestra.run.smithi150.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 7.6 MB/s | 661 kB 00:00 2024-03-28T15:47:25.732 INFO:teuthology.orchestra.run.smithi067.stdout:(31/174): ceph-osd-19.0.0-2145.g3be9a5c9.el9.x8 2.2 MB/s | 17 MB 00:07 2024-03-28T15:47:25.770 INFO:teuthology.orchestra.run.smithi150.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 5.7 MB/s | 578 kB 00:00 2024-03-28T15:47:25.795 INFO:teuthology.orchestra.run.smithi150.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 781 kB/s | 46 kB 00:00 2024-03-28T15:47:25.829 INFO:teuthology.orchestra.run.smithi150.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.8 MB/s | 282 kB 00:00 2024-03-28T15:47:25.863 INFO:teuthology.orchestra.run.smithi150.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 2.8 MB/s | 187 kB 00:00 2024-03-28T15:47:25.865 INFO:teuthology.orchestra.run.smithi067.stdout:(32/174): ceph-mgr-diskprediction-local-19.0.0- 9.2 MB/s | 7.4 MB 00:00 2024-03-28T15:47:25.896 INFO:teuthology.orchestra.run.smithi150.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.3 MB/s | 217 kB 00:00 2024-03-28T15:47:25.913 INFO:teuthology.orchestra.run.smithi150.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 716 kB/s | 34 kB 00:00 2024-03-28T15:47:26.056 INFO:teuthology.orchestra.run.smithi150.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 30 MB/s | 4.7 MB 00:00 2024-03-28T15:47:26.097 INFO:teuthology.orchestra.run.smithi150.stdout:(57/174): protobuf-3.14.0-13.el9.x86_64.rpm 6.4 MB/s | 1.0 MB 00:00 2024-03-28T15:47:26.124 INFO:teuthology.orchestra.run.smithi067.stdout:(33/174): ceph-common-19.0.0-2145.g3be9a5c9.el9 1.7 MB/s | 20 MB 00:12 2024-03-28T15:47:26.149 INFO:teuthology.orchestra.run.smithi067.stdout:(34/174): ceph-mgr-rook-19.0.0-2145.g3be9a5c9.e 174 kB/s | 49 kB 00:00 2024-03-28T15:47:26.164 INFO:teuthology.orchestra.run.smithi150.stdout:(58/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.2 MB/s | 78 kB 00:00 2024-03-28T15:47:26.183 INFO:teuthology.orchestra.run.smithi067.stdout:(35/174): ceph-mgr-modules-core-19.0.0-2145.g3b 553 kB/s | 248 kB 00:00 2024-03-28T15:47:26.208 INFO:teuthology.orchestra.run.smithi067.stdout:(36/174): ceph-prometheus-alerts-19.0.0-2145.g3 199 kB/s | 16 kB 00:00 2024-03-28T15:47:26.233 INFO:teuthology.orchestra.run.smithi067.stdout:(37/174): ceph-volume-19.0.0-2145.g3be9a5c9.el9 3.1 MB/s | 264 kB 00:00 2024-03-28T15:47:26.323 INFO:teuthology.orchestra.run.smithi150.stdout:(59/174): python3-babel-2.9.1-2.el9.noarch.rpm 26 MB/s | 5.8 MB 00:00 2024-03-28T15:47:26.339 INFO:teuthology.orchestra.run.smithi183.stdout:(98/174): python3-jinja2-2.11.3-5.el9.noarch.rp 238 kB/s | 248 kB 00:01 2024-03-28T15:47:26.348 INFO:teuthology.orchestra.run.smithi150.stdout:(60/174): python3-jsonpatch-1.21-16.el9.noarch. 147 kB/s | 24 kB 00:00 2024-03-28T15:47:26.350 INFO:teuthology.orchestra.run.smithi067.stdout:(38/174): cephfs-top-19.0.0-2145.g3be9a5c9.el9. 129 kB/s | 18 kB 00:00 2024-03-28T15:47:26.373 INFO:teuthology.orchestra.run.smithi150.stdout:(61/174): python3-jsonpointer-2.0-4.el9.noarch. 369 kB/s | 18 kB 00:00 2024-03-28T15:47:26.399 INFO:teuthology.orchestra.run.smithi150.stdout:(62/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.9 MB/s | 144 kB 00:00 2024-03-28T15:47:26.424 INFO:teuthology.orchestra.run.smithi150.stdout:(63/174): python3-markupsafe-1.1.1-12.el9.x86_6 653 kB/s | 32 kB 00:00 2024-03-28T15:47:26.425 INFO:teuthology.orchestra.run.smithi067.stdout:(39/174): cephadm-19.0.0-2145.g3be9a5c9.el9.noa 3.0 MB/s | 748 kB 00:00 2024-03-28T15:47:26.501 INFO:teuthology.orchestra.run.smithi067.stdout:(40/174): automake-1.16.2-8.el9.noarch.rpm 2.4 MB/s | 662 kB 00:00 2024-03-28T15:47:26.574 INFO:teuthology.orchestra.run.smithi150.stdout:(64/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 27 MB/s | 4.8 MB 00:00 2024-03-28T15:47:26.593 INFO:teuthology.orchestra.run.smithi067.stdout:(41/174): bison-3.7.4-5.el9.x86_64.rpm 3.8 MB/s | 920 kB 00:00 2024-03-28T15:47:26.608 INFO:teuthology.orchestra.run.smithi150.stdout:(65/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.3 MB/s | 217 kB 00:00 2024-03-28T15:47:26.642 INFO:teuthology.orchestra.run.smithi150.stdout:(66/174): python3-oauthlib-3.1.1-5.el9.noarch.r 2.6 MB/s | 174 kB 00:00 2024-03-28T15:47:26.651 INFO:teuthology.orchestra.run.smithi067.stdout:(42/174): boost-program-options-1.75.0-8.el9.x8 475 kB/s | 106 kB 00:00 2024-03-28T15:47:26.667 INFO:teuthology.orchestra.run.smithi150.stdout:(67/174): python3-packaging-20.9-5.el9.noarch.r 1.2 MB/s | 69 kB 00:00 2024-03-28T15:47:26.676 INFO:teuthology.orchestra.run.smithi067.stdout:(43/174): flex-2.6.4-9.el9.x86_64.rpm 1.8 MB/s | 307 kB 00:00 2024-03-28T15:47:26.692 INFO:teuthology.orchestra.run.smithi150.stdout:(68/174): python3-prettytable-0.7.2-27.el9.noar 844 kB/s | 41 kB 00:00 2024-03-28T15:47:26.702 INFO:teuthology.orchestra.run.smithi067.stdout:(44/174): flexiblas-3.0.4-8.el9.x86_64.rpm 303 kB/s | 32 kB 00:00 2024-03-28T15:47:26.726 INFO:teuthology.orchestra.run.smithi150.stdout:(69/174): python3-protobuf-3.14.0-13.el9.noarch 4.1 MB/s | 238 kB 00:00 2024-03-28T15:47:26.751 INFO:teuthology.orchestra.run.smithi150.stdout:(70/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.3 MB/s | 132 kB 00:00 2024-03-28T15:47:26.785 INFO:teuthology.orchestra.run.smithi150.stdout:(71/174): python3-pyasn1-modules-0.4.8-6.el9.no 3.6 MB/s | 211 kB 00:00 2024-03-28T15:47:26.810 INFO:teuthology.orchestra.run.smithi150.stdout:(72/174): python3-pytz-2021.1-5.el9.noarch.rpm 846 kB/s | 47 kB 00:00 2024-03-28T15:47:26.835 INFO:teuthology.orchestra.run.smithi150.stdout:(73/174): python3-requests-oauthlib-1.3.0-12.el 871 kB/s | 43 kB 00:00 2024-03-28T15:47:26.860 INFO:teuthology.orchestra.run.smithi067.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 94 kB/s | 17 kB 00:00 2024-03-28T15:47:26.928 INFO:teuthology.orchestra.run.smithi067.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 45 kB/s | 10 kB 00:00 2024-03-28T15:47:27.011 INFO:teuthology.orchestra.run.smithi067.stdout:(47/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 9.0 MB/s | 3.0 MB 00:00 2024-03-28T15:47:27.045 INFO:teuthology.orchestra.run.smithi067.stdout:(48/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 882 kB/s | 159 kB 00:00 2024-03-28T15:47:27.070 INFO:teuthology.orchestra.run.smithi067.stdout:(49/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 321 kB/s | 44 kB 00:00 2024-03-28T15:47:27.073 INFO:teuthology.orchestra.run.smithi183.stdout:(99/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 681 kB/s | 4.1 MB 00:06 2024-03-28T15:47:27.077 INFO:teuthology.orchestra.run.smithi150.stdout:(74/174): python3-toml-0.10.2-6.el9.noarch.rpm 163 kB/s | 39 kB 00:00 2024-03-28T15:47:27.137 INFO:teuthology.orchestra.run.smithi067.stdout:(50/174): librdkafka-1.6.1-102.el9.x86_64.rpm 5.3 MB/s | 661 kB 00:00 2024-03-28T15:47:27.204 INFO:teuthology.orchestra.run.smithi067.stdout:(51/174): libtool-2.4.6-45.el9.x86_64.rpm 4.4 MB/s | 578 kB 00:00 2024-03-28T15:47:27.229 INFO:teuthology.orchestra.run.smithi067.stdout:(52/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 300 kB/s | 46 kB 00:00 2024-03-28T15:47:27.262 INFO:teuthology.orchestra.run.smithi067.stdout:(53/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.3 MB/s | 282 kB 00:00 2024-03-28T15:47:27.304 INFO:teuthology.orchestra.run.smithi067.stdout:(54/174): lua-5.4.4-4.el9.x86_64.rpm 2.5 MB/s | 187 kB 00:00 2024-03-28T15:47:27.338 INFO:teuthology.orchestra.run.smithi067.stdout:(55/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.0 MB/s | 217 kB 00:00 2024-03-28T15:47:27.363 INFO:teuthology.orchestra.run.smithi067.stdout:(56/174): openblas-0.3.21-2.el9.x86_64.rpm 341 kB/s | 34 kB 00:00 2024-03-28T15:47:27.465 INFO:teuthology.orchestra.run.smithi183.stdout:(100/174): qatlib-24.02.0-1.el9.x86_64.rpm 196 kB/s | 221 kB 00:01 2024-03-28T15:47:27.663 INFO:teuthology.orchestra.run.smithi067.stdout:(57/174): openblas-openmp-0.3.21-2.el9.x86_64.r 13 MB/s | 4.7 MB 00:00 2024-03-28T15:47:27.669 INFO:teuthology.orchestra.run.smithi150.stdout:(75/174): ceph-test-19.0.0-2145.g3be9a5c9.el9.x 5.5 MB/s | 48 MB 00:08 2024-03-28T15:47:27.688 INFO:teuthology.orchestra.run.smithi067.stdout:(58/174): python3-attrs-20.3.0-7.el9.noarch.rpm 245 kB/s | 78 kB 00:00 2024-03-28T15:47:27.756 INFO:teuthology.orchestra.run.smithi067.stdout:(59/174): protobuf-3.14.0-13.el9.x86_64.rpm 2.6 MB/s | 1.0 MB 00:00 2024-03-28T15:47:27.789 INFO:teuthology.orchestra.run.smithi067.stdout:(60/174): python3-jsonpatch-1.21-16.el9.noarch. 245 kB/s | 24 kB 00:00 2024-03-28T15:47:27.823 INFO:teuthology.orchestra.run.smithi183.stdout:(101/174): qatlib-service-24.02.0-1.el9.x86_64. 48 kB/s | 36 kB 00:00 2024-03-28T15:47:27.894 INFO:teuthology.orchestra.run.smithi150.stdout:(76/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 14 MB/s | 16 MB 00:01 2024-03-28T15:47:27.919 INFO:teuthology.orchestra.run.smithi150.stdout:(77/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 49 kB/s | 41 kB 00:00 2024-03-28T15:47:27.944 INFO:teuthology.orchestra.run.smithi150.stdout:(78/174): socat-1.7.4.1-5.el9.x86_64.rpm 1.1 MB/s | 300 kB 00:00 2024-03-28T15:47:27.986 INFO:teuthology.orchestra.run.smithi150.stdout:(79/174): uuid-1.6.2-55.el9.x86_64.rpm 1.4 MB/s | 56 kB 00:00 2024-03-28T15:47:28.023 INFO:teuthology.orchestra.run.smithi067.stdout:(61/174): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 5.8 MB 00:00 2024-03-28T15:47:28.037 INFO:teuthology.orchestra.run.smithi150.stdout:(80/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 691 kB/s | 63 kB 00:00 2024-03-28T15:47:28.048 INFO:teuthology.orchestra.run.smithi067.stdout:(62/174): python3-jsonpointer-2.0-4.el9.noarch. 62 kB/s | 18 kB 00:00 2024-03-28T15:47:28.062 INFO:teuthology.orchestra.run.smithi150.stdout:(81/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 381 kB/s | 44 kB 00:00 2024-03-28T15:47:28.073 INFO:teuthology.orchestra.run.smithi067.stdout:(63/174): python3-mako-1.1.4-6.el9.noarch.rpm 536 kB/s | 144 kB 00:00 2024-03-28T15:47:28.087 INFO:teuthology.orchestra.run.smithi150.stdout:(82/174): fuse-2.9.9-15.el9.x86_64.rpm 795 kB/s | 79 kB 00:00 2024-03-28T15:47:28.098 INFO:teuthology.orchestra.run.smithi067.stdout:(64/174): python3-markupsafe-1.1.1-12.el9.x86_6 433 kB/s | 32 kB 00:00 2024-03-28T15:47:28.198 INFO:teuthology.orchestra.run.smithi183.stdout:(102/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 65 kB/s | 47 kB 00:00 2024-03-28T15:47:28.207 INFO:teuthology.orchestra.run.smithi067.stdout:(65/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.6 MB/s | 217 kB 00:00 2024-03-28T15:47:28.296 INFO:teuthology.orchestra.run.smithi150.stdout:(83/174): python3-cryptography-36.0.1-4.el9.x86 4.9 MB/s | 1.1 MB 00:00 2024-03-28T15:47:28.298 INFO:teuthology.orchestra.run.smithi183.stdout:(103/174): lua-devel-5.4.4-4.el9.x86_64.rpm 232 kB/s | 22 kB 00:00 2024-03-28T15:47:28.330 INFO:teuthology.orchestra.run.smithi150.stdout:(84/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 906 kB/s | 241 kB 00:00 2024-03-28T15:47:28.355 INFO:teuthology.orchestra.run.smithi150.stdout:(85/174): python3-ply-3.11-14.el9.noarch.rpm 402 kB/s | 103 kB 00:00 2024-03-28T15:47:28.372 INFO:teuthology.orchestra.run.smithi150.stdout:(86/174): python3-pycparser-2.20-6.el9.noarch.r 1.7 MB/s | 124 kB 00:00 2024-03-28T15:47:28.422 INFO:teuthology.orchestra.run.smithi150.stdout:(87/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.8 MB/s | 187 kB 00:00 2024-03-28T15:47:28.457 INFO:teuthology.orchestra.run.smithi067.stdout:(66/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 12 MB/s | 4.8 MB 00:00 2024-03-28T15:47:28.491 INFO:teuthology.orchestra.run.smithi067.stdout:(67/174): python3-oauthlib-3.1.1-5.el9.noarch.r 451 kB/s | 174 kB 00:00 2024-03-28T15:47:28.516 INFO:teuthology.orchestra.run.smithi067.stdout:(68/174): python3-packaging-20.9-5.el9.noarch.r 234 kB/s | 69 kB 00:00 2024-03-28T15:47:28.541 INFO:teuthology.orchestra.run.smithi067.stdout:(69/174): python3-prettytable-0.7.2-27.el9.noar 511 kB/s | 41 kB 00:00 2024-03-28T15:47:28.600 INFO:teuthology.orchestra.run.smithi067.stdout:(70/174): python3-protobuf-3.14.0-13.el9.noarch 2.8 MB/s | 238 kB 00:00 2024-03-28T15:47:28.634 INFO:teuthology.orchestra.run.smithi067.stdout:(71/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.1 MB/s | 132 kB 00:00 2024-03-28T15:47:28.659 INFO:teuthology.orchestra.run.smithi067.stdout:(72/174): python3-pyasn1-modules-0.4.8-6.el9.no 1.8 MB/s | 211 kB 00:00 2024-03-28T15:47:28.684 INFO:teuthology.orchestra.run.smithi067.stdout:(73/174): python3-pytz-2021.1-5.el9.noarch.rpm 588 kB/s | 47 kB 00:00 2024-03-28T15:47:28.707 INFO:teuthology.orchestra.run.smithi183.stdout:(104/174): protobuf-compiler-3.14.0-13.el9.x86_ 2.1 MB/s | 863 kB 00:00 2024-03-28T15:47:28.709 INFO:teuthology.orchestra.run.smithi067.stdout:(74/174): python3-requests-oauthlib-1.3.0-12.el 878 kB/s | 43 kB 00:00 2024-03-28T15:47:28.766 INFO:teuthology.orchestra.run.smithi183.stdout:(105/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 100 kB/s | 94 kB 00:00 2024-03-28T15:47:28.772 INFO:teuthology.orchestra.run.smithi150.stdout:(88/174): libquadmath-11.4.1-3.el9.x86_64.rpm 558 kB/s | 195 kB 00:00 2024-03-28T15:47:28.798 INFO:teuthology.orchestra.run.smithi150.stdout:(89/174): jq-1.6-16.el9.x86_64.rpm 425 kB/s | 187 kB 00:00 2024-03-28T15:47:28.866 INFO:teuthology.orchestra.run.smithi183.stdout:(106/174): gperftools-libs-2.9.1-2.el9.x86_64.r 3.1 MB/s | 309 kB 00:00 2024-03-28T15:47:28.881 INFO:teuthology.orchestra.run.smithi150.stdout:(90/174): python3-requests-2.25.1-8.el9.noarch. 1.1 MB/s | 125 kB 00:00 2024-03-28T15:47:28.899 INFO:teuthology.orchestra.run.smithi183.stdout:(107/174): abseil-cpp-20211102.0-3.el9.x86_64.r 2.8 MB/s | 548 kB 00:00 2024-03-28T15:47:28.925 INFO:teuthology.orchestra.run.smithi183.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 336 kB/s | 19 kB 00:00 2024-03-28T15:47:28.943 INFO:teuthology.orchestra.run.smithi067.stdout:(75/174): python3-toml-0.10.2-6.el9.noarch.rpm 150 kB/s | 39 kB 00:00 2024-03-28T15:47:28.967 INFO:teuthology.orchestra.run.smithi183.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 636 kB/s | 26 kB 00:00 2024-03-28T15:47:28.968 INFO:teuthology.orchestra.run.smithi067.stdout:(76/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 162 kB/s | 41 kB 00:00 2024-03-28T15:47:28.981 INFO:teuthology.orchestra.run.smithi150.stdout:(91/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 256 kB/s | 25 kB 00:00 2024-03-28T15:47:29.007 INFO:teuthology.orchestra.run.smithi150.stdout:(92/174): elfutils-libelf-devel-0.190-2.el9.x86 103 kB/s | 21 kB 00:00 2024-03-28T15:47:29.008 INFO:teuthology.orchestra.run.smithi183.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 1.2 MB/s | 49 kB 00:00 2024-03-28T15:47:29.040 INFO:teuthology.orchestra.run.smithi150.stdout:(93/174): libgfortran-11.4.1-3.el9.x86_64.rpm 1.2 MB/s | 803 kB 00:00 2024-03-28T15:47:29.051 INFO:teuthology.orchestra.run.smithi183.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-03-28T15:47:29.084 INFO:teuthology.orchestra.run.smithi183.stdout:(112/174): luarocks-3.9.2-1.el9.noarch.rpm 4.6 MB/s | 151 kB 00:00 2024-03-28T15:47:29.193 INFO:teuthology.orchestra.run.smithi183.stdout:(113/174): libarrow-9.0.0-10.el9.x86_64.rpm 15 MB/s | 4.4 MB 00:00 2024-03-28T15:47:29.199 INFO:teuthology.orchestra.run.smithi150.stdout:(94/174): postgresql-13.14-1.el9.x86_64.rpm 8.3 MB/s | 1.6 MB 00:00 2024-03-28T15:47:29.243 INFO:teuthology.orchestra.run.smithi183.stdout:(114/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 5.2 MB/s | 838 kB 00:00 2024-03-28T15:47:29.259 INFO:teuthology.orchestra.run.smithi150.stdout:(95/174): postgresql-private-libs-13.14-1.el9.x 2.2 MB/s | 133 kB 00:00 2024-03-28T15:47:29.285 INFO:teuthology.orchestra.run.smithi183.stdout:(115/174): python3-asyncssh-2.13.2-5.el9.noarch 5.9 MB/s | 548 kB 00:00 2024-03-28T15:47:29.310 INFO:teuthology.orchestra.run.smithi183.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 655 kB/s | 43 kB 00:00 2024-03-28T15:47:29.316 INFO:teuthology.orchestra.run.smithi150.stdout:(96/174): postgresql-contrib-13.14-1.el9.x86_64 3.2 MB/s | 911 kB 00:00 2024-03-28T15:47:29.326 INFO:teuthology.orchestra.run.smithi067.stdout:(77/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 24 MB/s | 16 MB 00:00 2024-03-28T15:47:29.335 INFO:teuthology.orchestra.run.smithi183.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 649 kB/s | 32 kB 00:00 2024-03-28T15:47:29.360 INFO:teuthology.orchestra.run.smithi067.stdout:(78/174): socat-1.7.4.1-5.el9.x86_64.rpm 797 kB/s | 300 kB 00:00 2024-03-28T15:47:29.361 INFO:teuthology.orchestra.run.smithi183.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 302 kB/s | 15 kB 00:00 2024-03-28T15:47:29.385 INFO:teuthology.orchestra.run.smithi067.stdout:(79/174): uuid-1.6.2-55.el9.x86_64.rpm 140 kB/s | 56 kB 00:00 2024-03-28T15:47:29.394 INFO:teuthology.orchestra.run.smithi183.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 2.9 MB/s | 172 kB 00:00 2024-03-28T15:47:29.419 INFO:teuthology.orchestra.run.smithi067.stdout:(80/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 705 kB/s | 63 kB 00:00 2024-03-28T15:47:29.428 INFO:teuthology.orchestra.run.smithi183.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 5.3 MB/s | 358 kB 00:00 2024-03-28T15:47:29.433 INFO:teuthology.orchestra.run.smithi150.stdout:(97/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 9.2 MB/s | 4.1 MB 00:00 2024-03-28T15:47:29.444 INFO:teuthology.orchestra.run.smithi067.stdout:(81/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 770 kB/s | 44 kB 00:00 2024-03-28T15:47:29.461 INFO:teuthology.orchestra.run.smithi067.stdout:(82/174): fuse-2.9.9-15.el9.x86_64.rpm 1.0 MB/s | 79 kB 00:00 2024-03-28T15:47:29.461 INFO:teuthology.orchestra.run.smithi183.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 4.5 MB/s | 303 kB 00:00 2024-03-28T15:47:29.467 INFO:teuthology.orchestra.run.smithi150.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 1.6 MB/s | 245 kB 00:00 2024-03-28T15:47:29.495 INFO:teuthology.orchestra.run.smithi067.stdout:(83/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.2 MB/s | 241 kB 00:00 2024-03-28T15:47:29.500 INFO:teuthology.orchestra.run.smithi150.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 3.7 MB/s | 248 kB 00:00 2024-03-28T15:47:29.506 INFO:teuthology.orchestra.run.smithi183.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 455 kB/s | 35 kB 00:00 2024-03-28T15:47:29.545 INFO:teuthology.orchestra.run.smithi183.stdout:(123/174): python3-google-auth-2.28.2-1.el9.noa 2.4 MB/s | 203 kB 00:00 2024-03-28T15:47:29.553 INFO:teuthology.orchestra.run.smithi067.stdout:(84/174): python3-cryptography-36.0.1-4.el9.x86 13 MB/s | 1.1 MB 00:00 2024-03-28T15:47:29.567 INFO:teuthology.orchestra.run.smithi150.stdout:(100/174): qatlib-service-24.02.0-1.el9.x86_64. 544 kB/s | 36 kB 00:00 2024-03-28T15:47:29.578 INFO:teuthology.orchestra.run.smithi067.stdout:(85/174): python3-ply-3.11-14.el9.noarch.rpm 906 kB/s | 103 kB 00:00 2024-03-28T15:47:29.587 INFO:teuthology.orchestra.run.smithi183.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 3.4 MB/s | 144 kB 00:00 2024-03-28T15:47:29.595 INFO:teuthology.orchestra.run.smithi067.stdout:(86/174): python3-pycparser-2.20-6.el9.noarch.r 1.4 MB/s | 124 kB 00:00 2024-03-28T15:47:29.621 INFO:teuthology.orchestra.run.smithi183.stdout:(125/174): python3-jaraco-8.2.1-3.el9.noarch.rp 327 kB/s | 11 kB 00:00 2024-03-28T15:47:29.626 INFO:teuthology.orchestra.run.smithi150.stdout:(101/174): qatlib-24.02.0-1.el9.x86_64.rpm 1.4 MB/s | 221 kB 00:00 2024-03-28T15:47:29.629 INFO:teuthology.orchestra.run.smithi067.stdout:(87/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.8 MB/s | 187 kB 00:00 2024-03-28T15:47:29.651 INFO:teuthology.orchestra.run.smithi150.stdout:(102/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 572 kB/s | 47 kB 00:00 2024-03-28T15:47:29.688 INFO:teuthology.orchestra.run.smithi183.stdout:(126/174): python3-grpcio-1.46.7-10.el9.x86_64. 11 MB/s | 2.0 MB 00:00 2024-03-28T15:47:29.693 INFO:teuthology.orchestra.run.smithi150.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.4 MB/s | 94 kB 00:00 2024-03-28T15:47:29.730 INFO:teuthology.orchestra.run.smithi183.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 164 kB/s | 18 kB 00:00 2024-03-28T15:47:29.755 INFO:teuthology.orchestra.run.smithi183.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 350 kB/s | 23 kB 00:00 2024-03-28T15:47:29.780 INFO:teuthology.orchestra.run.smithi183.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 393 kB/s | 19 kB 00:00 2024-03-28T15:47:29.806 INFO:teuthology.orchestra.run.smithi183.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 398 kB/s | 20 kB 00:00 2024-03-28T15:47:29.818 INFO:teuthology.orchestra.run.smithi150.stdout:(104/174): postgresql-server-13.14-1.el9.x86_64 10 MB/s | 5.8 MB 00:00 2024-03-28T15:47:29.848 INFO:teuthology.orchestra.run.smithi183.stdout:(131/174): python3-logutils-0.3.5-21.el9.noarch 1.1 MB/s | 46 kB 00:00 2024-03-28T15:47:29.915 INFO:teuthology.orchestra.run.smithi183.stdout:(132/174): python3-kubernetes-26.1.0-2.el9.noar 7.7 MB/s | 1.0 MB 00:00 2024-03-28T15:47:29.948 INFO:teuthology.orchestra.run.smithi183.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 791 kB/s | 79 kB 00:00 2024-03-28T15:47:29.974 INFO:teuthology.orchestra.run.smithi183.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 1.0 MB/s | 58 kB 00:00 2024-03-28T15:47:29.999 INFO:teuthology.orchestra.run.smithi183.stdout:(135/174): python3-pecan-1.4.2-3.el9.noarch.rpm 5.4 MB/s | 272 kB 00:00 2024-03-28T15:47:30.024 INFO:teuthology.orchestra.run.smithi183.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 332 kB/s | 16 kB 00:00 2024-03-28T15:47:30.049 INFO:teuthology.orchestra.run.smithi183.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.8 MB/s | 90 kB 00:00 2024-03-28T15:47:30.074 INFO:teuthology.orchestra.run.smithi183.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 571 kB/s | 28 kB 00:00 2024-03-28T15:47:30.100 INFO:teuthology.orchestra.run.smithi183.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 621 kB/s | 31 kB 00:00 2024-03-28T15:47:30.133 INFO:teuthology.orchestra.run.smithi183.stdout:(140/174): python3-routes-2.5.1-5.el9.noarch.rp 3.2 MB/s | 188 kB 00:00 2024-03-28T15:47:30.157 INFO:teuthology.orchestra.run.smithi183.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-03-28T15:47:30.160 INFO:teuthology.orchestra.run.smithi150.stdout:(105/174): lua-devel-5.4.4-4.el9.x86_64.rpm 44 kB/s | 22 kB 00:00 2024-03-28T15:47:30.182 INFO:teuthology.orchestra.run.smithi183.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 746 kB/s | 36 kB 00:00 2024-03-28T15:47:30.208 INFO:teuthology.orchestra.run.smithi183.stdout:(143/174): python3-typing-extensions-4.4.0-2.el 1.0 MB/s | 51 kB 00:00 2024-03-28T15:47:30.241 INFO:teuthology.orchestra.run.smithi183.stdout:(144/174): python3-webob-1.8.7-6.el9.noarch.rpm 3.9 MB/s | 230 kB 00:00 2024-03-28T15:47:30.290 INFO:teuthology.orchestra.run.smithi183.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 1.1 MB/s | 90 kB 00:00 2024-03-28T15:47:30.325 INFO:teuthology.orchestra.run.smithi183.stdout:(146/174): python3-werkzeug-2.0.3-3.el9.1.noarc 5.0 MB/s | 427 kB 00:00 2024-03-28T15:47:30.351 INFO:teuthology.orchestra.run.smithi183.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 333 kB/s | 20 kB 00:00 2024-03-28T15:47:30.384 INFO:teuthology.orchestra.run.smithi183.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 3.2 MB/s | 192 kB 00:00 2024-03-28T15:47:30.385 INFO:teuthology.orchestra.run.smithi150.stdout:(106/174): protobuf-compiler-3.14.0-13.el9.x86_ 1.2 MB/s | 863 kB 00:00 2024-03-28T15:47:30.451 INFO:teuthology.orchestra.run.smithi183.stdout:(149/174): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-03-28T15:47:30.605 INFO:teuthology.orchestra.run.smithi067.stdout:(88/174): libquadmath-11.4.1-3.el9.x86_64.rpm 200 kB/s | 195 kB 00:00 2024-03-28T15:47:30.630 INFO:teuthology.orchestra.run.smithi067.stdout:(89/174): jq-1.6-16.el9.x86_64.rpm 179 kB/s | 187 kB 00:01 2024-03-28T15:47:30.661 INFO:teuthology.orchestra.run.smithi150.stdout:(107/174): abseil-cpp-20211102.0-3.el9.x86_64.r 651 kB/s | 548 kB 00:00 2024-03-28T15:47:30.803 INFO:teuthology.orchestra.run.smithi150.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 47 kB/s | 19 kB 00:00 2024-03-28T15:47:30.836 INFO:teuthology.orchestra.run.smithi150.stdout:(109/174): gperftools-libs-2.9.1-2.el9.x86_64.r 458 kB/s | 309 kB 00:00 2024-03-28T15:47:30.847 INFO:teuthology.orchestra.run.smithi067.stdout:(90/174): elfutils-libelf-devel-0.190-2.el9.x86 100 kB/s | 21 kB 00:00 2024-03-28T15:47:30.885 INFO:teuthology.orchestra.run.smithi183.stdout:(150/174): librbd1-19.0.0-2145.g3be9a5c9.el9.x8 7.3 MB/s | 3.1 MB 00:00 2024-03-28T15:47:30.927 INFO:teuthology.orchestra.run.smithi183.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 7.7 MB/s | 321 kB 00:00 2024-03-28T15:47:30.928 INFO:teuthology.orchestra.run.smithi150.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 209 kB/s | 26 kB 00:00 2024-03-28T15:47:30.954 INFO:teuthology.orchestra.run.smithi150.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 418 kB/s | 49 kB 00:00 2024-03-28T15:47:31.002 INFO:teuthology.orchestra.run.smithi067.stdout:(91/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 299 kB/s | 25 kB 00:00 2024-03-28T15:47:31.094 INFO:teuthology.orchestra.run.smithi183.stdout:(152/174): NetworkManager-1.47.2-1.el9.x86_64.r 14 MB/s | 2.3 MB 00:00 2024-03-28T15:47:31.121 INFO:teuthology.orchestra.run.smithi150.stdout:(112/174): libarrow-9.0.0-10.el9.x86_64.rpm 9.6 MB/s | 4.4 MB 00:00 2024-03-28T15:47:31.138 INFO:teuthology.orchestra.run.smithi150.stdout:(113/174): libunwind-1.6.2-1.el9.x86_64.rpm 323 kB/s | 67 kB 00:00 2024-03-28T15:47:31.163 INFO:teuthology.orchestra.run.smithi150.stdout:(114/174): luarocks-3.9.2-1.el9.noarch.rpm 725 kB/s | 151 kB 00:00 2024-03-28T15:47:31.202 INFO:teuthology.orchestra.run.smithi183.stdout:(153/174): NetworkManager-libnm-1.47.2-1.el9.x8 17 MB/s | 1.8 MB 00:00 2024-03-28T15:47:31.239 INFO:teuthology.orchestra.run.smithi067.stdout:(92/174): libgfortran-11.4.1-3.el9.x86_64.rpm 489 kB/s | 803 kB 00:01 2024-03-28T15:47:31.252 INFO:teuthology.orchestra.run.smithi183.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 868 kB/s | 42 kB 00:00 2024-03-28T15:47:31.255 INFO:teuthology.orchestra.run.smithi150.stdout:(115/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 6.2 MB/s | 838 kB 00:00 2024-03-28T15:47:31.272 INFO:teuthology.orchestra.run.smithi150.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 402 kB/s | 43 kB 00:00 2024-03-28T15:47:31.294 INFO:teuthology.orchestra.run.smithi183.stdout:(155/174): NetworkManager-tui-1.47.2-1.el9.x86_ 5.9 MB/s | 248 kB 00:00 2024-03-28T15:47:31.323 INFO:teuthology.orchestra.run.smithi067.stdout:(93/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 13 MB/s | 4.1 MB 00:00 2024-03-28T15:47:31.336 INFO:teuthology.orchestra.run.smithi183.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 1.7 MB/s | 71 kB 00:00 2024-03-28T15:47:31.347 INFO:teuthology.orchestra.run.smithi150.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 354 kB/s | 32 kB 00:00 2024-03-28T15:47:31.364 INFO:teuthology.orchestra.run.smithi150.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 164 kB/s | 15 kB 00:00 2024-03-28T15:47:31.378 INFO:teuthology.orchestra.run.smithi183.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 890 kB/s | 36 kB 00:00 2024-03-28T15:47:31.420 INFO:teuthology.orchestra.run.smithi183.stdout:(158/174): elfutils-libelf-0.190-2.el9.x86_64.r 4.6 MB/s | 192 kB 00:00 2024-03-28T15:47:31.477 INFO:teuthology.orchestra.run.smithi150.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.3 MB/s | 172 kB 00:00 2024-03-28T15:47:31.479 INFO:teuthology.orchestra.run.smithi183.stdout:(159/174): elfutils-libs-0.190-2.el9.x86_64.rpm 4.3 MB/s | 256 kB 00:00 2024-03-28T15:47:31.481 INFO:teuthology.orchestra.run.smithi067.stdout:(94/174): postgresql-13.14-1.el9.x86_64.rpm 6.6 MB/s | 1.6 MB 00:00 2024-03-28T15:47:31.521 INFO:teuthology.orchestra.run.smithi183.stdout:(160/174): libacl-2.3.1-4.el9.x86_64.rpm 564 kB/s | 23 kB 00:00 2024-03-28T15:47:31.523 INFO:teuthology.orchestra.run.smithi150.stdout:(120/174): python3-asyncssh-2.13.2-5.el9.noarch 1.4 MB/s | 548 kB 00:00 2024-03-28T15:47:31.523 INFO:teuthology.orchestra.run.smithi067.stdout:(95/174): python3-requests-2.25.1-8.el9.noarch. 136 kB/s | 125 kB 00:00 2024-03-28T15:47:31.557 INFO:teuthology.orchestra.run.smithi150.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 1.8 MB/s | 358 kB 00:00 2024-03-28T15:47:31.563 INFO:teuthology.orchestra.run.smithi183.stdout:(161/174): libblkid-2.37.4-18.el9.x86_64.rpm 2.6 MB/s | 108 kB 00:00 2024-03-28T15:47:31.575 INFO:teuthology.orchestra.run.smithi067.stdout:(96/174): postgresql-private-libs-13.14-1.el9.x 1.5 MB/s | 133 kB 00:00 2024-03-28T15:47:31.590 INFO:teuthology.orchestra.run.smithi150.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.6 MB/s | 303 kB 00:00 2024-03-28T15:47:31.614 INFO:teuthology.orchestra.run.smithi183.stdout:(162/174): libfdisk-2.37.4-18.el9.x86_64.rpm 3.1 MB/s | 155 kB 00:00 2024-03-28T15:47:31.624 INFO:teuthology.orchestra.run.smithi150.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 354 kB/s | 35 kB 00:00 2024-03-28T15:47:31.655 INFO:teuthology.orchestra.run.smithi183.stdout:(163/174): libmount-2.37.4-18.el9.x86_64.rpm 3.2 MB/s | 136 kB 00:00 2024-03-28T15:47:31.691 INFO:teuthology.orchestra.run.smithi150.stdout:(124/174): python3-google-auth-2.28.2-1.el9.noa 1.5 MB/s | 203 kB 00:00 2024-03-28T15:47:31.715 INFO:teuthology.orchestra.run.smithi067.stdout:(97/174): postgresql-contrib-13.14-1.el9.x86_64 2.3 MB/s | 911 kB 00:00 2024-03-28T15:47:31.730 INFO:teuthology.orchestra.run.smithi183.stdout:(164/174): librados2-19.0.0-2145.g3be9a5c9.el9. 2.5 MB/s | 3.4 MB 00:01 2024-03-28T15:47:31.741 INFO:teuthology.orchestra.run.smithi150.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 1.2 MB/s | 144 kB 00:00 2024-03-28T15:47:31.748 INFO:teuthology.orchestra.run.smithi067.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 2.3 MB/s | 245 kB 00:00 2024-03-28T15:47:31.755 INFO:teuthology.orchestra.run.smithi183.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 640 kB/s | 64 kB 00:00 2024-03-28T15:47:31.781 INFO:teuthology.orchestra.run.smithi183.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 581 kB/s | 29 kB 00:00 2024-03-28T15:47:31.799 INFO:teuthology.orchestra.run.smithi067.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 2.9 MB/s | 248 kB 00:00 2024-03-28T15:47:31.816 INFO:teuthology.orchestra.run.smithi150.stdout:(126/174): python3-grpcio-1.46.7-10.el9.x86_64. 9.0 MB/s | 2.0 MB 00:00 2024-03-28T15:47:31.832 INFO:teuthology.orchestra.run.smithi067.stdout:(100/174): qatlib-24.02.0-1.el9.x86_64.rpm 2.6 MB/s | 221 kB 00:00 2024-03-28T15:47:31.841 INFO:teuthology.orchestra.run.smithi150.stdout:(127/174): python3-jaraco-8.2.1-3.el9.noarch.rp 71 kB/s | 11 kB 00:00 2024-03-28T15:47:31.864 INFO:teuthology.orchestra.run.smithi183.stdout:(167/174): openssl-3.0.7-27.el9.x86_64.rpm 11 MB/s | 1.2 MB 00:00 2024-03-28T15:47:31.867 INFO:teuthology.orchestra.run.smithi150.stdout:(128/174): python3-jaraco-classes-3.2.1-5.el9.n 142 kB/s | 18 kB 00:00 2024-03-28T15:47:31.883 INFO:teuthology.orchestra.run.smithi067.stdout:(101/174): qatlib-service-24.02.0-1.el9.x86_64. 434 kB/s | 36 kB 00:00 2024-03-28T15:47:31.916 INFO:teuthology.orchestra.run.smithi067.stdout:(102/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 572 kB/s | 47 kB 00:00 2024-03-28T15:47:31.917 INFO:teuthology.orchestra.run.smithi150.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 232 kB/s | 23 kB 00:00 2024-03-28T15:47:31.942 INFO:teuthology.orchestra.run.smithi150.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 195 kB/s | 19 kB 00:00 2024-03-28T15:47:31.967 INFO:teuthology.orchestra.run.smithi150.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 197 kB/s | 20 kB 00:00 2024-03-28T15:47:31.998 INFO:teuthology.orchestra.run.smithi183.stdout:(168/174): util-linux-2.37.4-18.el9.x86_64.rpm 17 MB/s | 2.3 MB 00:00 2024-03-28T15:47:32.000 INFO:teuthology.orchestra.run.smithi067.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 807 kB/s | 94 kB 00:00 2024-03-28T15:47:32.065 INFO:teuthology.orchestra.run.smithi183.stdout:(169/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 7.6 MB/s | 2.2 MB 00:00 2024-03-28T15:47:32.076 INFO:teuthology.orchestra.run.smithi150.stdout:(132/174): python3-kubernetes-26.1.0-2.el9.noar 6.5 MB/s | 1.0 MB 00:00 2024-03-28T15:47:32.101 INFO:teuthology.orchestra.run.smithi150.stdout:(133/174): python3-logutils-0.3.5-21.el9.noarch 293 kB/s | 46 kB 00:00 2024-03-28T15:47:32.115 INFO:teuthology.orchestra.run.smithi183.stdout:(170/174): util-linux-core-2.37.4-18.el9.x86_64 3.9 MB/s | 465 kB 00:00 2024-03-28T15:47:32.118 INFO:teuthology.orchestra.run.smithi150.stdout:(134/174): python3-more-itertools-8.12.0-2.el9. 525 kB/s | 79 kB 00:00 2024-03-28T15:47:32.141 INFO:teuthology.orchestra.run.smithi183.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 418 kB/s | 31 kB 00:00 2024-03-28T15:47:32.177 INFO:teuthology.orchestra.run.smithi150.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 579 kB/s | 58 kB 00:00 2024-03-28T15:47:32.191 INFO:teuthology.orchestra.run.smithi183.stdout:(172/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 14 MB/s | 1.1 MB 00:00 2024-03-28T15:47:32.225 INFO:teuthology.orchestra.run.smithi067.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 72 kB/s | 22 kB 00:00 2024-03-28T15:47:32.227 INFO:teuthology.orchestra.run.smithi150.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 152 kB/s | 16 kB 00:00 2024-03-28T15:47:32.261 INFO:teuthology.orchestra.run.smithi150.stdout:(137/174): python3-pecan-1.4.2-3.el9.noarch.rpm 1.7 MB/s | 272 kB 00:00 2024-03-28T15:47:32.286 INFO:teuthology.orchestra.run.smithi150.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 832 kB/s | 90 kB 00:00 2024-03-28T15:47:32.328 INFO:teuthology.orchestra.run.smithi150.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 283 kB/s | 28 kB 00:00 2024-03-28T15:47:32.362 INFO:teuthology.orchestra.run.smithi150.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 308 kB/s | 31 kB 00:00 2024-03-28T15:47:32.387 INFO:teuthology.orchestra.run.smithi150.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 1.8 MB/s | 188 kB 00:00 2024-03-28T15:47:32.444 INFO:teuthology.orchestra.run.smithi150.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 513 kB/s | 59 kB 00:00 2024-03-28T15:47:32.469 INFO:teuthology.orchestra.run.smithi150.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 335 kB/s | 36 kB 00:00 2024-03-28T15:47:32.494 INFO:teuthology.orchestra.run.smithi150.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 477 kB/s | 51 kB 00:00 2024-03-28T15:47:32.570 INFO:teuthology.orchestra.run.smithi150.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 1.8 MB/s | 230 kB 00:00 2024-03-28T15:47:32.595 INFO:teuthology.orchestra.run.smithi150.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 716 kB/s | 90 kB 00:00 2024-03-28T15:47:32.654 INFO:teuthology.orchestra.run.smithi150.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 2.6 MB/s | 427 kB 00:00 2024-03-28T15:47:32.688 INFO:teuthology.orchestra.run.smithi150.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 171 kB/s | 20 kB 00:00 2024-03-28T15:47:32.709 INFO:teuthology.orchestra.run.smithi067.stdout:(105/174): postgresql-server-13.14-1.el9.x86_64 4.9 MB/s | 5.8 MB 00:01 2024-03-28T15:47:32.721 INFO:teuthology.orchestra.run.smithi150.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 1.5 MB/s | 192 kB 00:00 2024-03-28T15:47:32.813 INFO:teuthology.orchestra.run.smithi150.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 10 MB/s | 1.6 MB 00:00 2024-03-28T15:47:32.864 INFO:teuthology.orchestra.run.smithi150.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 6.4 MB/s | 321 kB 00:00 2024-03-28T15:47:32.866 INFO:teuthology.orchestra.run.smithi183.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 103 kB/s | 74 kB 00:00 2024-03-28T15:47:32.917 INFO:teuthology.orchestra.run.smithi067.stdout:(106/174): protobuf-compiler-3.14.0-13.el9.x86_ 942 kB/s | 863 kB 00:00 2024-03-28T15:47:32.951 INFO:teuthology.orchestra.run.smithi067.stdout:(107/174): abseil-cpp-20211102.0-3.el9.x86_64.r 756 kB/s | 548 kB 00:00 2024-03-28T15:47:33.106 INFO:teuthology.orchestra.run.smithi150.stdout:(152/174): librbd1-19.0.0-2145.g3be9a5c9.el9.x8 8.2 MB/s | 3.1 MB 00:00 2024-03-28T15:47:33.187 INFO:teuthology.orchestra.run.smithi067.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 72 kB/s | 19 kB 00:00 2024-03-28T15:47:33.266 INFO:teuthology.orchestra.run.smithi183.stdout:(174/174): postgresql-server-13.14-1.el9.x86_64 659 kB/s | 5.8 MB 00:08 2024-03-28T15:47:33.276 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-28T15:47:33.277 INFO:teuthology.orchestra.run.smithi183.stdout:Total 9.6 MB/s | 225 MB 00:23 2024-03-28T15:47:33.335 INFO:teuthology.orchestra.run.smithi067.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 177 kB/s | 26 kB 00:00 2024-03-28T15:47:33.564 INFO:teuthology.orchestra.run.smithi150.stdout:(153/174): librados2-19.0.0-2145.g3be9a5c9.el9. 3.9 MB/s | 3.4 MB 00:00 2024-03-28T15:47:33.631 INFO:teuthology.orchestra.run.smithi150.stdout:(154/174): NetworkManager-1.47.2-1.el9.x86_64.r 3.0 MB/s | 2.3 MB 00:00 2024-03-28T15:47:33.644 INFO:teuthology.orchestra.run.smithi067.stdout:(110/174): libarrow-9.0.0-10.el9.x86_64.rpm 6.4 MB/s | 4.4 MB 00:00 2024-03-28T15:47:33.669 INFO:teuthology.orchestra.run.smithi067.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 146 kB/s | 49 kB 00:00 2024-03-28T15:47:33.703 INFO:teuthology.orchestra.run.smithi067.stdout:(112/174): gperftools-libs-2.9.1-2.el9.x86_64.r 311 kB/s | 309 kB 00:00 2024-03-28T15:47:33.706 INFO:teuthology.orchestra.run.smithi150.stdout:(155/174): NetworkManager-libnm-1.47.2-1.el9.x8 3.1 MB/s | 1.8 MB 00:00 2024-03-28T15:47:33.732 INFO:teuthology.orchestra.run.smithi150.stdout:(156/174): NetworkManager-team-1.47.2-1.el9.x86 253 kB/s | 42 kB 00:00 2024-03-28T15:47:33.753 INFO:teuthology.orchestra.run.smithi067.stdout:(113/174): libunwind-1.6.2-1.el9.x86_64.rpm 624 kB/s | 67 kB 00:00 2024-03-28T15:47:33.782 INFO:teuthology.orchestra.run.smithi150.stdout:(157/174): acl-2.3.1-4.el9.x86_64.rpm 959 kB/s | 71 kB 00:00 2024-03-28T15:47:33.807 INFO:teuthology.orchestra.run.smithi150.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 488 kB/s | 36 kB 00:00 2024-03-28T15:47:33.841 INFO:teuthology.orchestra.run.smithi150.stdout:(159/174): NetworkManager-tui-1.47.2-1.el9.x86_ 1.2 MB/s | 248 kB 00:00 2024-03-28T15:47:33.875 INFO:teuthology.orchestra.run.smithi150.stdout:(160/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.0 MB/s | 192 kB 00:00 2024-03-28T15:47:33.878 INFO:teuthology.orchestra.run.smithi067.stdout:(114/174): python3-asyncssh-2.13.2-5.el9.noarch 4.3 MB/s | 548 kB 00:00 2024-03-28T15:47:33.900 INFO:teuthology.orchestra.run.smithi150.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 398 kB/s | 23 kB 00:00 2024-03-28T15:47:33.928 INFO:teuthology.orchestra.run.smithi067.stdout:(115/174): luarocks-3.9.2-1.el9.noarch.rpm 585 kB/s | 151 kB 00:00 2024-03-28T15:47:33.933 INFO:teuthology.orchestra.run.smithi150.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.9 MB/s | 108 kB 00:00 2024-03-28T15:47:33.967 INFO:teuthology.orchestra.run.smithi150.stdout:(163/174): elfutils-libs-0.190-2.el9.x86_64.rpm 1.6 MB/s | 256 kB 00:00 2024-03-28T15:47:33.979 INFO:teuthology.orchestra.run.smithi067.stdout:(116/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.0 MB/s | 838 kB 00:00 2024-03-28T15:47:33.992 INFO:teuthology.orchestra.run.smithi150.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.7 MB/s | 155 kB 00:00 2024-03-28T15:47:34.004 INFO:teuthology.orchestra.run.smithi067.stdout:(117/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 348 kB/s | 43 kB 00:00 2024-03-28T15:47:34.017 INFO:teuthology.orchestra.run.smithi150.stdout:(165/174): libmount-2.37.4-18.el9.x86_64.rpm 1.6 MB/s | 136 kB 00:00 2024-03-28T15:47:34.029 INFO:teuthology.orchestra.run.smithi067.stdout:(118/174): python3-cachetools-4.2.4-1.el9.noarc 321 kB/s | 32 kB 00:00 2024-03-28T15:47:34.043 INFO:teuthology.orchestra.run.smithi150.stdout:(166/174): libsmartcols-2.37.4-18.el9.x86_64.rp 856 kB/s | 64 kB 00:00 2024-03-28T15:47:34.068 INFO:teuthology.orchestra.run.smithi150.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 384 kB/s | 29 kB 00:00 2024-03-28T15:47:34.074 INFO:teuthology.orchestra.run.smithi067.stdout:(119/174): python3-certifi-2021.10.8-2.el9.noar 164 kB/s | 15 kB 00:00 2024-03-28T15:47:34.105 INFO:teuthology.orchestra.run.smithi067.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.7 MB/s | 172 kB 00:00 2024-03-28T15:47:34.193 INFO:teuthology.orchestra.run.smithi150.stdout:(168/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 14 MB/s | 2.2 MB 00:00 2024-03-28T15:47:34.260 INFO:teuthology.orchestra.run.smithi150.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 6.9 MB/s | 465 kB 00:00 2024-03-28T15:47:34.310 INFO:teuthology.orchestra.run.smithi150.stdout:(170/174): openssl-3.0.7-27.el9.x86_64.rpm 4.5 MB/s | 1.2 MB 00:00 2024-03-28T15:47:34.323 INFO:teuthology.orchestra.run.smithi067.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 1.2 MB/s | 303 kB 00:00 2024-03-28T15:47:34.355 INFO:teuthology.orchestra.run.smithi067.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 141 kB/s | 35 kB 00:00 2024-03-28T15:47:34.359 INFO:teuthology.orchestra.run.smithi150.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 317 kB/s | 31 kB 00:00 2024-03-28T15:47:34.389 INFO:teuthology.orchestra.run.smithi067.stdout:(123/174): python3-cherrypy-18.6.1-2.el9.noarch 999 kB/s | 358 kB 00:00 2024-03-28T15:47:34.439 INFO:teuthology.orchestra.run.smithi067.stdout:(124/174): python3-google-auth-2.28.2-1.el9.noa 1.7 MB/s | 203 kB 00:00 2024-03-28T15:47:34.478 INFO:teuthology.orchestra.run.smithi150.stdout:(172/174): util-linux-2.37.4-18.el9.x86_64.rpm 5.6 MB/s | 2.3 MB 00:00 2024-03-28T15:47:34.503 INFO:teuthology.orchestra.run.smithi150.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 523 kB/s | 74 kB 00:00 2024-03-28T15:47:34.523 INFO:teuthology.orchestra.run.smithi067.stdout:(125/174): python3-grpcio-1.46.7-10.el9.x86_64. 12 MB/s | 2.0 MB 00:00 2024-03-28T15:47:34.545 INFO:teuthology.orchestra.run.smithi150.stdout:(174/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 4.6 MB/s | 1.1 MB 00:00 2024-03-28T15:47:34.548 INFO:teuthology.orchestra.run.smithi067.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 98 kB/s | 11 kB 00:00 2024-03-28T15:47:34.559 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-03-28T15:47:34.559 INFO:teuthology.orchestra.run.smithi150.stdout:Total 10 MB/s | 225 MB 00:21 2024-03-28T15:47:34.573 INFO:teuthology.orchestra.run.smithi067.stdout:(127/174): python3-grpcio-tools-1.46.7-10.el9.x 787 kB/s | 144 kB 00:00 2024-03-28T15:47:34.632 INFO:teuthology.orchestra.run.smithi067.stdout:(128/174): python3-jaraco-classes-3.2.1-5.el9.n 164 kB/s | 18 kB 00:00 2024-03-28T15:47:34.657 INFO:teuthology.orchestra.run.smithi067.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 214 kB/s | 23 kB 00:00 2024-03-28T15:47:34.682 INFO:teuthology.orchestra.run.smithi067.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 179 kB/s | 19 kB 00:00 2024-03-28T15:47:34.724 INFO:teuthology.orchestra.run.smithi067.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 215 kB/s | 20 kB 00:00 2024-03-28T15:47:34.783 INFO:teuthology.orchestra.run.smithi067.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 464 kB/s | 46 kB 00:00 2024-03-28T15:47:34.825 INFO:teuthology.orchestra.run.smithi067.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 790 kB/s | 79 kB 00:00 2024-03-28T15:47:34.867 INFO:teuthology.orchestra.run.smithi067.stdout:(134/174): python3-kubernetes-26.1.0-2.el9.noar 4.9 MB/s | 1.0 MB 00:00 2024-03-28T15:47:34.895 INFO:teuthology.orchestra.run.smithi067.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 534 kB/s | 58 kB 00:00 2024-03-28T15:47:34.916 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-28T15:47:34.926 INFO:teuthology.orchestra.run.smithi067.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 2.7 MB/s | 272 kB 00:00 2024-03-28T15:47:34.959 INFO:teuthology.orchestra.run.smithi067.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 180 kB/s | 16 kB 00:00 2024-03-28T15:47:34.993 INFO:teuthology.orchestra.run.smithi067.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 899 kB/s | 90 kB 00:00 2024-03-28T15:47:35.010 INFO:teuthology.orchestra.run.smithi067.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 338 kB/s | 28 kB 00:00 2024-03-28T15:47:35.044 INFO:teuthology.orchestra.run.smithi067.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 369 kB/s | 31 kB 00:00 2024-03-28T15:47:35.119 INFO:teuthology.orchestra.run.smithi067.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 1.5 MB/s | 188 kB 00:00 2024-03-28T15:47:35.144 INFO:teuthology.orchestra.run.smithi067.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 357 kB/s | 36 kB 00:00 2024-03-28T15:47:35.169 INFO:teuthology.orchestra.run.smithi067.stdout:(143/174): python3-rsa-4.9-2.el9.noarch.rpm 372 kB/s | 59 kB 00:00 2024-03-28T15:47:35.220 INFO:teuthology.orchestra.run.smithi067.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 511 kB/s | 51 kB 00:00 2024-03-28T15:47:35.253 INFO:teuthology.orchestra.run.smithi067.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-03-28T15:47:35.283 INFO:teuthology.orchestra.run.smithi067.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 794 kB/s | 90 kB 00:00 2024-03-28T15:47:35.294 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-28T15:47:35.294 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-28T15:47:35.350 INFO:teuthology.orchestra.run.smithi067.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 208 kB/s | 20 kB 00:00 2024-03-28T15:47:35.384 INFO:teuthology.orchestra.run.smithi067.stdout:(148/174): python3-werkzeug-2.0.3-3.el9.1.noarc 2.6 MB/s | 427 kB 00:00 2024-03-28T15:47:35.426 INFO:teuthology.orchestra.run.smithi067.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 1.3 MB/s | 192 kB 00:00 2024-03-28T15:47:35.526 INFO:teuthology.orchestra.run.smithi067.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 9.0 MB/s | 1.6 MB 00:00 2024-03-28T15:47:35.585 INFO:teuthology.orchestra.run.smithi067.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 5.5 MB/s | 321 kB 00:00 2024-03-28T15:47:36.183 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-03-28T15:47:36.318 INFO:teuthology.orchestra.run.smithi067.stdout:(152/174): librados2-19.0.0-2145.g3be9a5c9.el9. 3.6 MB/s | 3.4 MB 00:00 2024-03-28T15:47:36.562 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-03-28T15:47:36.562 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-03-28T15:47:36.710 INFO:teuthology.orchestra.run.smithi067.stdout:(153/174): librbd1-19.0.0-2145.g3be9a5c9.el9.x8 2.4 MB/s | 3.1 MB 00:01 2024-03-28T15:47:37.561 INFO:teuthology.orchestra.run.smithi067.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 50 kB/s | 42 kB 00:00 2024-03-28T15:47:37.732 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-28T15:47:37.732 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-28T15:47:38.719 INFO:teuthology.orchestra.run.smithi067.stdout:(155/174): NetworkManager-tui-1.47.2-1.el9.x86_ 215 kB/s | 248 kB 00:01 2024-03-28T15:47:39.098 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-03-28T15:47:39.098 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-03-28T15:47:39.627 INFO:teuthology.orchestra.run.smithi067.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 79 kB/s | 71 kB 00:00 2024-03-28T15:47:39.986 INFO:teuthology.orchestra.run.smithi067.stdout:(157/174): NetworkManager-1.47.2-1.el9.x86_64.r 536 kB/s | 2.3 MB 00:04 2024-03-28T15:47:40.045 INFO:teuthology.orchestra.run.smithi067.stdout:(158/174): NetworkManager-libnm-1.47.2-1.el9.x8 504 kB/s | 1.8 MB 00:03 2024-03-28T15:47:40.420 INFO:teuthology.orchestra.run.smithi067.stdout:(159/174): elfutils-debuginfod-client-0.190-2.e 46 kB/s | 36 kB 00:00 2024-03-28T15:47:41.096 INFO:teuthology.orchestra.run.smithi067.stdout:(160/174): elfutils-libelf-0.190-2.el9.x86_64.r 174 kB/s | 192 kB 00:01 2024-03-28T15:47:41.212 INFO:teuthology.orchestra.run.smithi067.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 29 kB/s | 23 kB 00:00 2024-03-28T15:47:41.338 INFO:teuthology.orchestra.run.smithi067.stdout:(162/174): elfutils-libs-0.190-2.el9.x86_64.rpm 198 kB/s | 256 kB 00:01 2024-03-28T15:47:42.113 INFO:teuthology.orchestra.run.smithi067.stdout:(163/174): libblkid-2.37.4-18.el9.x86_64.rpm 107 kB/s | 108 kB 00:01 2024-03-28T15:47:42.230 INFO:teuthology.orchestra.run.smithi067.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 152 kB/s | 155 kB 00:01 2024-03-28T15:47:42.347 INFO:teuthology.orchestra.run.smithi067.stdout:(165/174): libmount-2.37.4-18.el9.x86_64.rpm 135 kB/s | 136 kB 00:01 2024-03-28T15:47:43.056 INFO:teuthology.orchestra.run.smithi067.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 35 kB/s | 29 kB 00:00 2024-03-28T15:47:43.097 INFO:teuthology.orchestra.run.smithi067.stdout:(167/174): libsmartcols-2.37.4-18.el9.x86_64.rp 65 kB/s | 64 kB 00:00 2024-03-28T15:47:45.081 INFO:teuthology.orchestra.run.smithi067.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 453 kB/s | 1.2 MB 00:02 2024-03-28T15:47:45.384 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-28T15:47:45.532 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-28T15:47:45.572 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-28T15:47:45.609 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T15:47:45.654 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T15:47:45.733 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-28T15:47:45.871 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-28T15:47:45.928 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-28T15:47:45.967 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-28T15:47:46.062 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-28T15:47:46.365 INFO:teuthology.orchestra.run.smithi067.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 363 kB/s | 465 kB 00:01 2024-03-28T15:47:46.436 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-03-28T15:47:46.474 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-28T15:47:46.544 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T15:47:46.545 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-28T15:47:46.593 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-28T15:47:46.639 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T15:47:46.682 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T15:47:46.700 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T15:47:46.725 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T15:47:46.776 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-28T15:47:46.846 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T15:47:46.876 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-28T15:47:46.888 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-28T15:47:46.898 INFO:teuthology.orchestra.run.smithi067.stdout:(170/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 575 kB/s | 2.2 MB 00:03 2024-03-28T15:47:46.939 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-28T15:47:46.973 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-28T15:47:46.990 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-03-28T15:47:47.049 INFO:teuthology.orchestra.run.smithi067.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 46 kB/s | 31 kB 00:00 2024-03-28T15:47:47.098 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-28T15:47:47.115 INFO:teuthology.orchestra.run.smithi067.stdout:(172/174): util-linux-2.37.4-18.el9.x86_64.rpm 582 kB/s | 2.3 MB 00:04 2024-03-28T15:47:47.295 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-28T15:47:47.380 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-03-28T15:47:47.390 INFO:teuthology.orchestra.run.smithi067.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 218 kB/s | 74 kB 00:00 2024-03-28T15:47:47.405 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T15:47:47.511 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-28T15:47:47.550 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T15:47:47.554 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T15:47:47.580 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-28T15:47:47.670 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-28T15:47:47.720 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T15:47:47.746 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-28T15:47:47.764 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T15:47:47.782 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T15:47:47.845 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T15:47:47.887 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T15:47:47.921 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-28T15:47:47.987 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-03-28T15:47:48.151 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T15:47:48.200 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T15:47:48.241 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-28T15:47:48.295 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-28T15:47:48.420 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-03-28T15:47:48.444 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T15:47:48.558 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-28T15:47:48.594 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T15:47:48.620 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-28T15:47:48.711 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-28T15:47:48.765 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T15:47:48.789 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-28T15:47:48.822 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T15:47:48.853 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T15:47:48.883 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T15:47:48.899 INFO:teuthology.orchestra.run.smithi067.stdout:(174/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 550 kB/s | 1.1 MB 00:01 2024-03-28T15:47:48.909 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-03-28T15:47:48.909 INFO:teuthology.orchestra.run.smithi067.stdout:Total 6.4 MB/s | 225 MB 00:35 2024-03-28T15:47:49.049 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T15:47:49.109 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T15:47:49.164 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T15:47:49.212 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T15:47:49.272 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-28T15:47:49.446 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T15:47:49.619 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-28T15:47:49.629 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-28T15:47:49.771 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T15:47:49.775 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-28T15:47:49.807 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-28T15:47:49.843 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-28T15:47:49.860 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T15:47:49.901 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-28T15:47:50.013 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-28T15:47:50.050 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T15:47:50.099 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T15:47:50.113 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-28T15:47:50.157 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-28T15:47:50.247 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-28T15:47:50.287 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-28T15:47:50.346 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-28T15:47:50.391 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-28T15:47:50.426 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T15:47:50.454 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-28T15:47:50.576 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-28T15:47:50.611 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 39/198 2024-03-28T15:47:50.649 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-28T15:47:50.651 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-03-28T15:47:50.672 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 40/198 2024-03-28T15:47:50.702 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 41/198 2024-03-28T15:47:50.782 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-28T15:47:50.810 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-28T15:47:50.822 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-28T15:47:50.850 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-28T15:47:50.878 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-28T15:47:50.897 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-28T15:47:50.976 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-28T15:47:51.006 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-28T15:47:51.053 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-03-28T15:47:51.053 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-03-28T15:47:51.090 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-28T15:47:51.138 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-28T15:47:51.148 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-28T15:47:51.215 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-28T15:47:51.229 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-28T15:47:51.268 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-28T15:47:51.328 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-28T15:47:51.372 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-28T15:47:51.405 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-28T15:47:51.517 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-28T15:47:51.552 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 39/198 2024-03-28T15:47:51.601 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 40/198 2024-03-28T15:47:51.660 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 41/198 2024-03-28T15:47:51.787 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-28T15:47:51.846 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-28T15:47:51.906 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-28T15:47:51.944 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-28T15:47:52.005 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-28T15:47:52.124 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-28T15:47:52.131 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-28T15:47:52.172 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T15:47:52.190 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-28T15:47:52.199 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T15:47:52.244 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-03-28T15:47:52.310 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-28T15:47:52.364 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 53/198 2024-03-28T15:47:52.412 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-28T15:47:52.823 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-28T15:47:52.896 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-28T15:47:52.986 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-28T15:47:53.070 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-28T15:47:53.105 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-28T15:47:53.163 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T15:47:53.190 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T15:47:53.244 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-28T15:47:53.244 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-03-28T15:47:53.303 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-28T15:47:53.318 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-28T15:47:53.402 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 53/198 2024-03-28T15:47:53.449 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-28T15:47:53.511 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-28T15:47:53.543 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-03-28T15:47:53.546 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-03-28T15:47:53.564 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-28T15:47:53.660 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-28T15:47:53.761 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-28T15:47:53.797 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-28T15:47:53.836 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-28T15:47:53.896 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-28T15:47:53.903 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-28T15:47:53.944 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-28T15:47:53.998 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-28T15:47:54.072 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-28T15:47:54.117 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-28T15:47:54.210 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-28T15:47:54.263 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-28T15:47:54.267 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-28T15:47:54.318 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-28T15:47:54.356 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-28T15:47:54.439 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-28T15:47:54.473 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-28T15:47:54.564 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-28T15:47:54.581 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-28T15:47:54.620 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-28T15:47:54.644 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-28T15:47:54.667 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-28T15:47:54.695 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-28T15:47:54.719 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-28T15:47:54.735 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-28T15:47:54.773 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-28T15:47:54.811 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-28T15:47:54.869 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-28T15:47:54.879 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-28T15:47:54.919 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-28T15:47:54.981 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-28T15:47:55.040 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-28T15:47:55.052 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-28T15:47:55.185 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-28T15:47:55.239 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-28T15:47:55.255 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-28T15:47:55.300 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-28T15:47:55.309 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-28T15:47:55.347 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-28T15:47:55.374 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-28T15:47:55.404 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-28T15:47:55.532 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-28T15:47:55.570 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-28T15:47:55.614 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-28T15:47:55.615 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-28T15:47:55.644 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-28T15:47:55.674 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-28T15:47:55.693 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-28T15:47:55.838 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-28T15:47:55.843 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-28T15:47:55.880 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-28T15:47:55.949 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-28T15:47:55.985 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-28T15:47:56.010 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-28T15:47:56.027 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-28T15:47:56.073 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-28T15:47:56.132 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-28T15:47:56.156 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-28T15:47:56.182 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-28T15:47:56.213 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-28T15:47:56.232 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-28T15:47:56.300 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-28T15:47:56.360 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-28T15:47:56.376 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-28T15:47:56.403 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-28T15:47:56.438 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-28T15:47:56.502 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-28T15:47:56.541 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-28T15:47:56.622 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-28T15:47:56.675 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-28T15:47:56.684 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-28T15:47:56.807 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-28T15:47:56.848 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-28T15:47:56.861 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-28T15:47:56.898 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-28T15:47:56.898 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-28T15:47:56.937 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-28T15:47:56.953 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-28T15:47:56.984 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-28T15:47:57.025 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-28T15:47:57.072 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-28T15:47:57.092 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-28T15:47:57.148 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-28T15:47:57.163 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-28T15:47:57.213 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-28T15:47:57.256 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-28T15:47:57.383 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-28T15:47:57.433 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-28T15:47:57.468 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-28T15:47:57.519 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-28T15:47:57.562 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-28T15:47:57.705 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-28T15:47:57.825 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-28T15:47:57.876 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-28T15:47:57.912 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-28T15:47:57.967 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-28T15:47:58.106 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-28T15:47:58.115 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-28T15:47:58.163 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-28T15:47:58.178 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-28T15:47:58.188 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-28T15:47:58.244 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-28T15:47:58.363 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-28T15:47:58.405 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-28T15:47:58.635 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-28T15:47:58.669 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-28T15:47:58.761 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-28T15:47:59.134 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-28T15:47:59.185 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-28T15:47:59.218 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-28T15:47:59.273 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-28T15:47:59.384 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-28T15:47:59.427 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-28T15:47:59.550 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-28T15:47:59.677 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-28T15:47:59.687 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-28T15:47:59.716 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-28T15:47:59.808 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-28T15:48:00.608 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-28T15:48:00.736 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-28T15:48:00.907 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-03-28T15:48:01.057 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-28T15:48:01.121 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-28T15:48:01.176 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T15:48:01.194 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-28T15:48:01.221 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T15:48:01.289 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-28T15:48:01.309 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-28T15:48:01.348 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-28T15:48:01.393 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-28T15:48:01.404 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-28T15:48:01.434 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-28T15:48:01.461 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-28T15:48:01.483 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-28T15:48:01.590 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-28T15:48:01.813 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-28T15:48:01.847 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-28T15:48:01.950 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-28T15:48:01.978 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 125/198 2024-03-28T15:48:02.004 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-28T15:48:02.022 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-28T15:48:02.049 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T15:48:02.202 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T15:48:02.233 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T15:48:02.254 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-28T15:48:02.336 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-28T15:48:02.342 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T15:48:02.373 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-28T15:48:02.374 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-28T15:48:02.404 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-28T15:48:02.438 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-03-28T15:48:02.443 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-28T15:48:02.530 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-28T15:48:02.567 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-28T15:48:02.657 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-28T15:48:02.685 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 129/198 2024-03-28T15:48:02.776 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-28T15:48:02.823 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-28T15:48:02.829 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-03-28T15:48:02.851 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-28T15:48:02.855 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T15:48:02.971 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-28T15:48:02.999 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 125/198 2024-03-28T15:48:03.010 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T15:48:03.039 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-28T15:48:03.053 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-28T15:48:03.130 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-28T15:48:03.200 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-28T15:48:03.233 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T15:48:03.310 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T15:48:03.574 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-28T15:48:03.613 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-28T15:48:03.631 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T15:48:03.675 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T15:48:03.712 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-28T15:48:03.757 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 129/198 2024-03-28T15:48:03.758 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-28T15:48:04.138 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-28T15:48:04.265 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T15:48:04.355 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T15:48:04.576 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T15:48:04.668 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T15:48:05.032 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T15:48:05.226 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-28T15:48:05.297 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T15:48:05.353 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-28T15:48:05.392 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-28T15:48:05.428 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-28T15:48:05.488 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-28T15:48:05.605 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-28T15:48:05.648 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T15:48:05.687 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-28T15:48:05.735 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-28T15:48:05.841 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-28T15:48:05.914 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-28T15:48:05.981 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-28T15:48:06.029 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-28T15:48:06.067 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-28T15:48:06.190 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-28T15:48:06.229 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 39/198 2024-03-28T15:48:06.289 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 40/198 2024-03-28T15:48:06.315 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T15:48:06.328 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 41/198 2024-03-28T15:48:06.343 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T15:48:06.351 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-28T15:48:06.378 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T15:48:06.455 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-28T15:48:06.519 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-28T15:48:06.540 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-28T15:48:06.540 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-28T15:48:06.541 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-28T15:48:06.541 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T15:48:06.586 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-28T15:48:06.633 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-28T15:48:06.819 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-28T15:48:06.877 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-28T15:48:07.011 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T15:48:07.048 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-28T15:48:07.227 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-28T15:48:07.227 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-28T15:48:07.227 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-28T15:48:07.227 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-28T15:48:07.274 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-28T15:48:07.594 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-28T15:48:07.705 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-28T15:48:07.819 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-28T15:48:07.907 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T15:48:07.936 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T15:48:07.984 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-03-28T15:48:08.093 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-28T15:48:08.164 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 53/198 2024-03-28T15:48:08.209 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-28T15:48:08.628 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-28T15:48:08.797 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-28T15:48:08.886 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-28T15:48:08.947 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-28T15:48:09.162 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-28T15:48:09.216 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-28T15:48:09.346 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-28T15:48:09.444 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-28T15:48:09.482 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-28T15:48:09.522 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-28T15:48:09.593 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-28T15:48:09.640 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-28T15:48:09.705 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-28T15:48:09.770 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-28T15:48:09.973 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-28T15:48:10.027 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-28T15:48:10.066 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-28T15:48:10.127 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-28T15:48:10.271 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-28T15:48:10.314 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-28T15:48:10.360 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-28T15:48:10.396 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-28T15:48:10.429 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-28T15:48:10.607 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-28T15:48:10.782 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-28T15:48:10.922 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-28T15:48:10.973 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-28T15:48:11.035 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-28T15:48:11.109 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-28T15:48:11.363 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-28T15:48:11.425 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-28T15:48:11.594 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-28T15:48:11.640 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-28T15:48:11.680 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-28T15:48:11.711 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-28T15:48:11.760 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-28T15:48:11.805 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-28T15:48:11.876 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-28T15:48:11.934 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-28T15:48:11.981 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-28T15:48:12.123 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-28T15:48:12.167 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-28T15:48:12.203 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-28T15:48:12.269 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-28T15:48:12.313 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-28T15:48:12.462 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-28T15:48:12.582 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-28T15:48:12.643 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-28T15:48:12.679 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-28T15:48:12.747 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-28T15:48:12.899 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-28T15:48:12.961 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-28T15:48:13.964 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-28T15:48:14.027 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-28T15:48:14.061 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-28T15:48:14.122 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-28T15:48:14.247 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-28T15:48:14.294 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-28T15:48:14.541 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-28T15:48:14.599 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-28T15:48:14.707 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-28T15:48:15.596 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-28T15:48:15.756 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-28T15:48:17.362 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-28T15:48:17.461 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-28T15:48:17.498 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-28T15:48:17.529 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-28T15:48:17.570 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-28T15:48:17.969 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-28T15:48:18.001 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-28T15:48:18.122 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-28T15:48:18.157 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 125/198 2024-03-28T15:48:18.226 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-28T15:48:18.779 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-28T15:48:18.812 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-28T15:48:18.912 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-28T15:48:18.947 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 129/198 2024-03-28T15:48:19.379 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-28T15:48:19.380 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /sys 2024-03-28T15:48:19.380 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /proc 2024-03-28T15:48:19.380 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /mnt 2024-03-28T15:48:19.380 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /var/tmp 2024-03-28T15:48:19.380 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /home 2024-03-28T15:48:19.380 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /root 2024-03-28T15:48:19.380 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /tmp 2024-03-28T15:48:19.380 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T15:48:19.591 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-28T15:48:19.637 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-28T15:48:19.638 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T15:48:19.638 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-28T15:48:19.638 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T15:48:20.622 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-28T15:48:20.622 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /sys 2024-03-28T15:48:20.622 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /proc 2024-03-28T15:48:20.622 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /mnt 2024-03-28T15:48:20.622 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /var/tmp 2024-03-28T15:48:20.622 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /home 2024-03-28T15:48:20.622 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /root 2024-03-28T15:48:20.622 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /tmp 2024-03-28T15:48:20.623 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-28T15:48:20.861 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-28T15:48:20.908 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-28T15:48:20.908 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T15:48:20.908 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-28T15:48:20.909 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-28T15:48:21.227 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-28T15:48:21.303 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-28T15:48:21.304 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T15:48:21.304 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-28T15:48:21.304 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T15:48:21.304 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T15:48:21.304 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T15:48:21.593 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-28T15:48:21.666 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-28T15:48:21.666 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T15:48:21.667 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-28T15:48:21.667 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T15:48:21.667 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T15:48:21.667 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T15:48:21.756 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T15:48:21.794 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T15:48:22.162 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-28T15:48:22.230 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-28T15:48:22.230 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T15:48:22.230 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-28T15:48:22.230 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T15:48:22.230 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T15:48:22.230 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T15:48:22.366 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-28T15:48:22.463 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T15:48:22.505 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-28T15:48:22.565 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-28T15:48:22.644 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-28T15:48:22.644 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T15:48:22.644 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-28T15:48:22.644 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T15:48:22.644 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T15:48:22.644 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-28T15:48:22.722 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-28T15:48:22.722 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-28T15:48:22.722 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-28T15:48:22.723 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T15:48:22.773 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-28T15:48:22.890 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-28T15:48:22.974 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-28T15:48:22.974 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T15:48:22.974 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-28T15:48:22.974 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T15:48:22.974 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T15:48:22.974 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-28T15:48:23.018 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-28T15:48:23.047 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-28T15:48:23.099 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 139/198 2024-03-28T15:48:23.187 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 140/198 2024-03-28T15:48:23.227 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-28T15:48:23.460 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-28T15:48:23.526 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-28T15:48:23.526 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T15:48:23.526 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-28T15:48:23.527 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T15:48:23.527 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T15:48:23.527 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-28T15:48:23.557 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-28T15:48:23.594 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-28T15:48:23.643 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-28T15:48:24.272 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-28T15:48:24.310 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-28T15:48:24.361 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 139/198 2024-03-28T15:48:24.462 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 140/198 2024-03-28T15:48:24.497 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-28T15:48:24.709 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-28T15:48:24.744 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-28T15:48:24.831 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-28T15:48:24.865 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-28T15:48:24.884 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-28T15:48:25.057 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 144/198 2024-03-28T15:48:25.105 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-28T15:48:25.182 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-28T15:48:25.183 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T15:48:25.183 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-28T15:48:25.183 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T15:48:25.183 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T15:48:25.183 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T15:48:25.239 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-28T15:48:25.278 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-28T15:48:25.362 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 147/198 2024-03-28T15:48:25.991 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-28T15:48:26.015 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-28T15:48:26.158 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-28T15:48:26.307 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-28T15:48:26.328 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 144/198 2024-03-28T15:48:26.341 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-28T15:48:26.359 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-28T15:48:26.418 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-28T15:48:26.418 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T15:48:26.418 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-28T15:48:26.418 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T15:48:26.418 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T15:48:26.418 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T15:48:26.429 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-28T15:48:26.429 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T15:48:26.429 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-28T15:48:26.429 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T15:48:26.429 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T15:48:26.429 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-28T15:48:26.474 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-28T15:48:26.493 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-28T15:48:26.531 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-28T15:48:26.561 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-28T15:48:26.562 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T15:48:26.562 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-28T15:48:26.562 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T15:48:26.621 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 147/198 2024-03-28T15:48:26.669 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T15:48:26.770 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T15:48:26.770 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T15:48:26.770 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-28T15:48:26.770 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T15:48:26.771 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T15:48:26.771 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T15:48:27.170 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-28T15:48:27.249 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-28T15:48:27.250 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T15:48:27.250 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-28T15:48:27.250 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T15:48:27.250 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T15:48:27.250 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T15:48:27.588 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-28T15:48:27.629 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-28T15:48:27.697 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-28T15:48:27.697 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T15:48:27.698 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-28T15:48:27.698 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T15:48:27.698 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T15:48:27.698 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-28T15:48:27.754 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-28T15:48:27.818 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-28T15:48:27.818 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T15:48:27.818 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-28T15:48:27.818 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-28T15:48:27.897 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T15:48:27.980 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T15:48:27.981 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T15:48:27.981 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-28T15:48:27.981 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T15:48:27.981 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T15:48:27.981 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-28T15:48:28.364 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-28T15:48:28.434 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-28T15:48:28.434 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T15:48:28.434 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-28T15:48:28.434 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T15:48:28.435 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T15:48:28.435 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-28T15:48:32.085 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-28T15:48:32.141 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 154/198 2024-03-28T15:48:32.174 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 155/198 2024-03-28T15:48:32.213 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-28T15:48:32.490 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 157/198 2024-03-28T15:48:32.618 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 158/198 2024-03-28T15:48:32.674 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 159/198 2024-03-28T15:48:32.724 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-28T15:48:32.810 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-28T15:48:32.860 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-28T15:48:32.897 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 163/198 2024-03-28T15:48:32.950 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-28T15:48:33.001 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-28T15:48:33.092 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-28T15:48:33.190 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-28T15:48:33.250 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-28T15:48:33.261 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-28T15:48:33.306 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 154/198 2024-03-28T15:48:33.372 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 155/198 2024-03-28T15:48:33.415 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-28T15:48:33.696 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 157/198 2024-03-28T15:48:33.824 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 158/198 2024-03-28T15:48:33.877 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 159/198 2024-03-28T15:48:33.926 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-28T15:48:34.009 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-28T15:48:34.046 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-28T15:48:34.082 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 163/198 2024-03-28T15:48:34.135 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-28T15:48:34.145 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T15:48:34.187 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-28T15:48:34.197 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T15:48:34.263 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T15:48:34.276 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-28T15:48:34.394 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-28T15:48:34.436 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-28T15:48:35.356 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T15:48:35.408 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T15:48:35.463 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T15:48:35.592 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 170/198 2024-03-28T15:48:35.686 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-28T15:48:35.741 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-28T15:48:35.891 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-28T15:48:35.947 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-28T15:48:35.980 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-28T15:48:36.015 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-28T15:48:36.056 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-28T15:48:36.056 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T15:48:36.184 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T15:48:36.235 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-28T15:48:36.235 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T15:48:36.278 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-28T15:48:36.279 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /sys 2024-03-28T15:48:36.279 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /proc 2024-03-28T15:48:36.279 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /mnt 2024-03-28T15:48:36.279 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /var/tmp 2024-03-28T15:48:36.279 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /home 2024-03-28T15:48:36.279 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /root 2024-03-28T15:48:36.279 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /tmp 2024-03-28T15:48:36.279 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T15:48:36.365 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T15:48:36.415 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-28T15:48:36.464 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-28T15:48:36.496 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-28T15:48:36.529 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-28T15:48:36.533 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-28T15:48:36.584 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-28T15:48:36.584 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T15:48:36.585 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-28T15:48:36.585 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T15:48:36.596 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-28T15:48:36.710 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-28T15:48:36.759 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-28T15:48:36.792 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-28T15:48:36.798 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 170/198 2024-03-28T15:48:36.826 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-28T15:48:36.859 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-28T15:48:36.893 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-28T15:48:36.896 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-28T15:48:36.930 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T15:48:36.930 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T15:48:36.939 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-28T15:48:37.054 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-28T15:48:37.091 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-28T15:48:37.124 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-28T15:48:37.151 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-28T15:48:37.191 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-28T15:48:37.191 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T15:48:37.243 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T15:48:37.243 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T15:48:37.338 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T15:48:37.382 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T15:48:37.411 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-28T15:48:37.412 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T15:48:37.440 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-28T15:48:37.500 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-28T15:48:37.532 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-28T15:48:37.551 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T15:48:37.572 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-28T15:48:37.596 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-28T15:48:37.642 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-28T15:48:37.674 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-28T15:48:37.700 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T15:48:37.707 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-28T15:48:37.779 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-28T15:48:37.855 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-28T15:48:37.903 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-28T15:48:37.937 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-28T15:48:37.957 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T15:48:37.958 INFO:teuthology.orchestra.run.smithi183.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-03-28T15:48:37.958 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T15:48:37.987 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-28T15:48:38.020 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-28T15:48:38.054 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-28T15:48:38.091 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T15:48:38.091 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T15:48:38.272 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-28T15:48:38.351 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-28T15:48:38.352 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T15:48:38.352 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-28T15:48:38.352 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T15:48:38.352 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T15:48:38.352 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T15:48:38.428 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T15:48:38.428 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T15:48:38.584 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T15:48:38.615 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-28T15:48:38.621 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-28T15:48:38.670 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-28T15:48:38.697 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-28T15:48:38.697 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T15:48:38.697 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-28T15:48:38.698 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T15:48:38.698 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T15:48:38.698 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T15:48:38.709 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-28T15:48:38.750 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-28T15:48:38.870 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T15:48:39.118 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T15:48:39.119 INFO:teuthology.orchestra.run.smithi150.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-03-28T15:48:39.119 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-28T15:48:39.289 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-28T15:48:39.363 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-28T15:48:39.363 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T15:48:39.364 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-28T15:48:39.364 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T15:48:39.364 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T15:48:39.364 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T15:48:39.490 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-28T15:48:40.116 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-28T15:48:40.151 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-28T15:48:40.220 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 139/198 2024-03-28T15:48:40.323 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 140/198 2024-03-28T15:48:40.354 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-28T15:48:40.699 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-28T15:48:40.729 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-28T15:48:41.903 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-28T15:48:41.929 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-28T15:48:42.078 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-28T15:48:42.260 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 144/198 2024-03-28T15:48:42.290 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-28T15:48:42.364 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-28T15:48:42.364 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T15:48:42.364 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-28T15:48:42.365 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T15:48:42.365 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T15:48:42.365 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T15:48:42.424 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-28T15:48:42.464 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-28T15:48:42.529 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 147/198 2024-03-28T15:48:42.826 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 1/198 2024-03-28T15:48:42.827 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2/198 2024-03-28T15:48:42.827 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 3/198 2024-03-28T15:48:42.827 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 4/198 2024-03-28T15:48:42.827 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 5/198 2024-03-28T15:48:42.827 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 6/198 2024-03-28T15:48:42.827 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 7/198 2024-03-28T15:48:42.827 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 8/198 2024-03-28T15:48:42.827 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 9/198 2024-03-28T15:48:42.827 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T15:48:42.827 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T15:48:42.827 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-28T15:48:42.827 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 13/198 2024-03-28T15:48:42.827 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 14/198 2024-03-28T15:48:42.828 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 15/198 2024-03-28T15:48:42.828 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T15:48:42.828 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 17/198 2024-03-28T15:48:42.828 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 18/198 2024-03-28T15:48:42.828 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 19/198 2024-03-28T15:48:42.828 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 20/198 2024-03-28T15:48:42.828 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 21/198 2024-03-28T15:48:42.828 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 22/198 2024-03-28T15:48:42.828 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 23/198 2024-03-28T15:48:42.828 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 24/198 2024-03-28T15:48:42.828 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 25/198 2024-03-28T15:48:42.845 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 26/198 2024-03-28T15:48:42.845 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 27/198 2024-03-28T15:48:42.845 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 28/198 2024-03-28T15:48:42.845 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 29/198 2024-03-28T15:48:42.845 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 30/198 2024-03-28T15:48:42.845 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 31/198 2024-03-28T15:48:42.845 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 32/198 2024-03-28T15:48:42.846 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 33/198 2024-03-28T15:48:42.846 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 34/198 2024-03-28T15:48:42.846 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 35/198 2024-03-28T15:48:42.846 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 36/198 2024-03-28T15:48:42.846 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 37/198 2024-03-28T15:48:42.846 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 38/198 2024-03-28T15:48:42.846 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 39/198 2024-03-28T15:48:42.846 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-28T15:48:42.846 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-28T15:48:42.846 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-28T15:48:42.846 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-28T15:48:42.846 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-28T15:48:42.846 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-28T15:48:42.846 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-28T15:48:42.847 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-28T15:48:42.847 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-28T15:48:42.847 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-28T15:48:42.847 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-28T15:48:42.847 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-28T15:48:42.847 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-28T15:48:42.847 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-28T15:48:42.847 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-28T15:48:42.847 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-28T15:48:42.847 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-28T15:48:42.847 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-28T15:48:42.847 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 58/198 2024-03-28T15:48:42.847 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 59/198 2024-03-28T15:48:42.847 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 60/198 2024-03-28T15:48:42.847 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 61/198 2024-03-28T15:48:42.848 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 62/198 2024-03-28T15:48:42.848 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 63/198 2024-03-28T15:48:42.848 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 64/198 2024-03-28T15:48:42.848 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 65/198 2024-03-28T15:48:42.848 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 66/198 2024-03-28T15:48:42.848 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 67/198 2024-03-28T15:48:42.848 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 68/198 2024-03-28T15:48:42.848 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 69/198 2024-03-28T15:48:42.848 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 70/198 2024-03-28T15:48:42.848 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 71/198 2024-03-28T15:48:42.848 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 72/198 2024-03-28T15:48:42.848 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 73/198 2024-03-28T15:48:42.848 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 74/198 2024-03-28T15:48:42.848 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 75/198 2024-03-28T15:48:42.848 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 76/198 2024-03-28T15:48:42.849 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 77/198 2024-03-28T15:48:42.849 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 78/198 2024-03-28T15:48:42.849 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 79/198 2024-03-28T15:48:42.849 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 80/198 2024-03-28T15:48:42.849 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 81/198 2024-03-28T15:48:42.849 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 82/198 2024-03-28T15:48:42.849 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 83/198 2024-03-28T15:48:42.849 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 84/198 2024-03-28T15:48:42.849 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-ply-3.11-14.el9.noarch 85/198 2024-03-28T15:48:42.849 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 86/198 2024-03-28T15:48:42.849 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 87/198 2024-03-28T15:48:42.849 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : jq-1.6-16.el9.x86_64 88/198 2024-03-28T15:48:42.849 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 89/198 2024-03-28T15:48:42.849 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 90/198 2024-03-28T15:48:42.850 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 91/198 2024-03-28T15:48:42.850 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 92/198 2024-03-28T15:48:42.850 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 93/198 2024-03-28T15:48:42.850 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 94/198 2024-03-28T15:48:42.850 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : postgresql-13.14-1.el9.x86_64 95/198 2024-03-28T15:48:42.850 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 96/198 2024-03-28T15:48:42.850 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 97/198 2024-03-28T15:48:42.850 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 98/198 2024-03-28T15:48:42.850 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-28T15:48:42.850 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-28T15:48:42.850 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 101/198 2024-03-28T15:48:42.850 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 102/198 2024-03-28T15:48:42.850 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 103/198 2024-03-28T15:48:42.850 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-28T15:48:42.851 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-28T15:48:42.851 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-28T15:48:42.851 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-28T15:48:42.851 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-28T15:48:42.851 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-28T15:48:42.851 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-28T15:48:42.851 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-28T15:48:42.851 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-28T15:48:42.851 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-28T15:48:42.851 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-28T15:48:42.851 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-28T15:48:42.851 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-28T15:48:42.851 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-28T15:48:42.851 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-28T15:48:42.851 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-28T15:48:42.852 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-28T15:48:42.852 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-28T15:48:42.852 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-28T15:48:42.852 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-28T15:48:42.852 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-28T15:48:42.852 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-28T15:48:42.852 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-28T15:48:42.852 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-28T15:48:42.852 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-28T15:48:42.852 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-28T15:48:42.852 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-28T15:48:42.852 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-28T15:48:42.852 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-28T15:48:42.852 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-28T15:48:42.852 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-28T15:48:42.853 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-28T15:48:42.853 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-28T15:48:42.853 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-28T15:48:42.853 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-28T15:48:42.853 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-28T15:48:42.853 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-28T15:48:42.853 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-28T15:48:42.853 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-28T15:48:42.853 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-28T15:48:42.853 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-28T15:48:42.853 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-28T15:48:42.853 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-28T15:48:42.853 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-28T15:48:42.853 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-28T15:48:42.853 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-28T15:48:42.854 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-28T15:48:42.854 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T15:48:42.854 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-28T15:48:42.854 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-28T15:48:42.854 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-28T15:48:42.854 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-28T15:48:42.854 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-28T15:48:42.854 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-28T15:48:42.854 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-28T15:48:42.854 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-28T15:48:42.854 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-28T15:48:42.854 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-28T15:48:42.854 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-28T15:48:42.855 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-28T15:48:42.855 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-28T15:48:42.855 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-28T15:48:42.855 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-28T15:48:42.855 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-28T15:48:42.855 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-28T15:48:42.855 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-28T15:48:42.855 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-28T15:48:42.855 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-28T15:48:42.855 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-28T15:48:42.855 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-28T15:48:42.855 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-28T15:48:42.855 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-28T15:48:42.855 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-28T15:48:42.855 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-28T15:48:42.856 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-28T15:48:42.856 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-28T15:48:42.856 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-28T15:48:42.856 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-28T15:48:42.856 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-28T15:48:42.856 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-28T15:48:42.856 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-28T15:48:42.856 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-28T15:48:42.856 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-28T15:48:42.856 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-28T15:48:42.856 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-28T15:48:42.856 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-28T15:48:42.856 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-28T15:48:42.856 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-28T15:48:42.856 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-28T15:48:42.856 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-28T15:48:42.857 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-28T15:48:42.857 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-28T15:48:42.857 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-28T15:48:42.857 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-28T15:48:43.534 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-28T15:48:43.568 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-28T15:48:43.643 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-28T15:48:43.644 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T15:48:43.644 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-28T15:48:43.644 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T15:48:43.644 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T15:48:43.644 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T15:48:43.693 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-28T15:48:43.780 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-28T15:48:43.780 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T15:48:43.780 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-28T15:48:43.780 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T15:48:43.862 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 1/198 2024-03-28T15:48:43.863 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2/198 2024-03-28T15:48:43.863 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 3/198 2024-03-28T15:48:43.863 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 4/198 2024-03-28T15:48:43.863 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 5/198 2024-03-28T15:48:43.863 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 6/198 2024-03-28T15:48:43.863 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 7/198 2024-03-28T15:48:43.863 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 8/198 2024-03-28T15:48:43.863 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 9/198 2024-03-28T15:48:43.863 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T15:48:43.863 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T15:48:43.863 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-28T15:48:43.863 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 13/198 2024-03-28T15:48:43.864 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 14/198 2024-03-28T15:48:43.864 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 15/198 2024-03-28T15:48:43.864 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T15:48:43.864 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 17/198 2024-03-28T15:48:43.864 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 18/198 2024-03-28T15:48:43.864 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 19/198 2024-03-28T15:48:43.864 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 20/198 2024-03-28T15:48:43.864 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 21/198 2024-03-28T15:48:43.864 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 22/198 2024-03-28T15:48:43.864 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 23/198 2024-03-28T15:48:43.864 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 24/198 2024-03-28T15:48:43.864 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 25/198 2024-03-28T15:48:43.864 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 26/198 2024-03-28T15:48:43.864 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 27/198 2024-03-28T15:48:43.864 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 28/198 2024-03-28T15:48:43.865 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T15:48:43.881 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 29/198 2024-03-28T15:48:43.881 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 30/198 2024-03-28T15:48:43.882 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 31/198 2024-03-28T15:48:43.882 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 32/198 2024-03-28T15:48:43.882 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 33/198 2024-03-28T15:48:43.882 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 34/198 2024-03-28T15:48:43.882 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 35/198 2024-03-28T15:48:43.882 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 36/198 2024-03-28T15:48:43.882 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 37/198 2024-03-28T15:48:43.882 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 38/198 2024-03-28T15:48:43.885 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 39/198 2024-03-28T15:48:43.885 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-28T15:48:43.885 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-28T15:48:43.885 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-28T15:48:43.886 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-28T15:48:43.886 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-28T15:48:43.886 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-28T15:48:43.886 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-28T15:48:43.886 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-28T15:48:43.886 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-28T15:48:43.886 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-28T15:48:43.886 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-28T15:48:43.886 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-28T15:48:43.886 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-28T15:48:43.886 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-28T15:48:43.886 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-28T15:48:43.886 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-28T15:48:43.887 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-28T15:48:43.887 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-28T15:48:43.887 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 58/198 2024-03-28T15:48:43.887 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 59/198 2024-03-28T15:48:43.887 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 60/198 2024-03-28T15:48:43.887 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 61/198 2024-03-28T15:48:43.887 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 62/198 2024-03-28T15:48:43.887 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 63/198 2024-03-28T15:48:43.887 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 64/198 2024-03-28T15:48:43.887 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 65/198 2024-03-28T15:48:43.887 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 66/198 2024-03-28T15:48:43.887 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 67/198 2024-03-28T15:48:43.887 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 68/198 2024-03-28T15:48:43.887 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 69/198 2024-03-28T15:48:43.887 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 70/198 2024-03-28T15:48:43.888 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 71/198 2024-03-28T15:48:43.888 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 72/198 2024-03-28T15:48:43.888 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 73/198 2024-03-28T15:48:43.888 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 74/198 2024-03-28T15:48:43.888 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 75/198 2024-03-28T15:48:43.888 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 76/198 2024-03-28T15:48:43.888 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 77/198 2024-03-28T15:48:43.888 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 78/198 2024-03-28T15:48:43.888 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 79/198 2024-03-28T15:48:43.888 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 80/198 2024-03-28T15:48:43.888 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 81/198 2024-03-28T15:48:43.888 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 82/198 2024-03-28T15:48:43.888 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 83/198 2024-03-28T15:48:43.888 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 84/198 2024-03-28T15:48:43.889 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-ply-3.11-14.el9.noarch 85/198 2024-03-28T15:48:43.889 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 86/198 2024-03-28T15:48:43.889 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 87/198 2024-03-28T15:48:43.889 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : jq-1.6-16.el9.x86_64 88/198 2024-03-28T15:48:43.889 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 89/198 2024-03-28T15:48:43.889 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 90/198 2024-03-28T15:48:43.889 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 91/198 2024-03-28T15:48:43.889 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 92/198 2024-03-28T15:48:43.889 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 93/198 2024-03-28T15:48:43.889 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 94/198 2024-03-28T15:48:43.889 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : postgresql-13.14-1.el9.x86_64 95/198 2024-03-28T15:48:43.889 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 96/198 2024-03-28T15:48:43.889 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 97/198 2024-03-28T15:48:43.889 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 98/198 2024-03-28T15:48:43.890 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-28T15:48:43.890 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-28T15:48:43.890 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 101/198 2024-03-28T15:48:43.890 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 102/198 2024-03-28T15:48:43.890 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 103/198 2024-03-28T15:48:43.890 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-28T15:48:43.890 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-28T15:48:43.890 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-28T15:48:43.890 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-28T15:48:43.890 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-28T15:48:43.890 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-28T15:48:43.890 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-28T15:48:43.890 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-28T15:48:43.890 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-28T15:48:43.890 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-28T15:48:43.891 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-28T15:48:43.891 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-28T15:48:43.891 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-28T15:48:43.891 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-28T15:48:43.891 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-28T15:48:43.891 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-28T15:48:43.891 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-28T15:48:43.891 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-28T15:48:43.891 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-28T15:48:43.891 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-28T15:48:43.891 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-28T15:48:43.891 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-28T15:48:43.891 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-28T15:48:43.891 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-28T15:48:43.891 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-28T15:48:43.891 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-28T15:48:43.891 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-28T15:48:43.892 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-28T15:48:43.892 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-28T15:48:43.892 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-28T15:48:43.892 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-28T15:48:43.892 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-28T15:48:43.892 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-28T15:48:43.892 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-28T15:48:43.892 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-28T15:48:43.892 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-28T15:48:43.892 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-28T15:48:43.892 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-28T15:48:43.892 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-28T15:48:43.892 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-28T15:48:43.892 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-28T15:48:43.893 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-28T15:48:43.893 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-28T15:48:43.893 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-28T15:48:43.893 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-28T15:48:43.893 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-28T15:48:43.893 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-28T15:48:43.893 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T15:48:43.893 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-28T15:48:43.893 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-28T15:48:43.893 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-28T15:48:43.893 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-28T15:48:43.893 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-28T15:48:43.893 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-28T15:48:43.893 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-28T15:48:43.893 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-28T15:48:43.893 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-28T15:48:43.893 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-28T15:48:43.894 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-28T15:48:43.894 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-28T15:48:43.894 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-28T15:48:43.894 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-28T15:48:43.894 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-28T15:48:43.894 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-28T15:48:43.894 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-28T15:48:43.894 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-28T15:48:43.894 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-28T15:48:43.894 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-28T15:48:43.894 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-28T15:48:43.894 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-28T15:48:43.894 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-28T15:48:43.894 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-28T15:48:43.895 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-28T15:48:43.895 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-28T15:48:43.895 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-28T15:48:43.895 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-28T15:48:43.895 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-28T15:48:43.895 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-28T15:48:43.895 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-28T15:48:43.895 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-28T15:48:43.895 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-28T15:48:43.895 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-28T15:48:43.895 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-28T15:48:43.895 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-28T15:48:43.895 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-28T15:48:43.895 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-28T15:48:43.895 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-28T15:48:43.895 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-28T15:48:43.896 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-28T15:48:43.896 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-28T15:48:43.896 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-28T15:48:43.896 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-28T15:48:43.896 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-28T15:48:43.896 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-28T15:48:43.940 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T15:48:43.940 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T15:48:43.940 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-28T15:48:43.940 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T15:48:43.940 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T15:48:43.940 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T15:48:44.337 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-28T15:48:44.509 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-28T15:48:44.509 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T15:48:44.509 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-28T15:48:44.509 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T15:48:44.509 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T15:48:44.509 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T15:48:45.532 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-28T15:48:45.533 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T15:48:45.533 INFO:teuthology.orchestra.run.smithi183.stdout:Upgraded: 2024-03-28T15:48:45.533 INFO:teuthology.orchestra.run.smithi183.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-28T15:48:45.533 INFO:teuthology.orchestra.run.smithi183.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-28T15:48:45.533 INFO:teuthology.orchestra.run.smithi183.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-28T15:48:45.533 INFO:teuthology.orchestra.run.smithi183.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-28T15:48:45.533 INFO:teuthology.orchestra.run.smithi183.stdout: acl-2.3.1-4.el9.x86_64 2024-03-28T15:48:45.533 INFO:teuthology.orchestra.run.smithi183.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-28T15:48:45.533 INFO:teuthology.orchestra.run.smithi183.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-28T15:48:45.533 INFO:teuthology.orchestra.run.smithi183.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-28T15:48:45.533 INFO:teuthology.orchestra.run.smithi183.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-28T15:48:45.533 INFO:teuthology.orchestra.run.smithi183.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-28T15:48:45.533 INFO:teuthology.orchestra.run.smithi183.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-28T15:48:45.533 INFO:teuthology.orchestra.run.smithi183.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-28T15:48:45.534 INFO:teuthology.orchestra.run.smithi183.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-28T15:48:45.534 INFO:teuthology.orchestra.run.smithi183.stdout: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:48:45.534 INFO:teuthology.orchestra.run.smithi183.stdout: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:48:45.534 INFO:teuthology.orchestra.run.smithi183.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-28T15:48:45.534 INFO:teuthology.orchestra.run.smithi183.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-28T15:48:45.534 INFO:teuthology.orchestra.run.smithi183.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-28T15:48:45.534 INFO:teuthology.orchestra.run.smithi183.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-28T15:48:45.534 INFO:teuthology.orchestra.run.smithi183.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-28T15:48:45.534 INFO:teuthology.orchestra.run.smithi183.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-28T15:48:45.534 INFO:teuthology.orchestra.run.smithi183.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-28T15:48:45.534 INFO:teuthology.orchestra.run.smithi183.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-28T15:48:45.534 INFO:teuthology.orchestra.run.smithi183.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-28T15:48:45.534 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-28T15:48:45.534 INFO:teuthology.orchestra.run.smithi183.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-28T15:48:45.535 INFO:teuthology.orchestra.run.smithi183.stdout: automake-1.16.2-8.el9.noarch 2024-03-28T15:48:45.535 INFO:teuthology.orchestra.run.smithi183.stdout: bison-3.7.4-5.el9.x86_64 2024-03-28T15:48:45.535 INFO:teuthology.orchestra.run.smithi183.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-28T15:48:45.535 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:48:45.535 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:48:45.535 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:48:45.535 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:48:45.535 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T15:48:45.535 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:48:45.535 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:48:45.535 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:48:45.535 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T15:48:45.535 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T15:48:45.535 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T15:48:45.535 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T15:48:45.535 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T15:48:45.536 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:48:45.536 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:48:45.536 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T15:48:45.536 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:48:45.536 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:48:45.536 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:48:45.536 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T15:48:45.536 INFO:teuthology.orchestra.run.smithi183.stdout: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T15:48:45.536 INFO:teuthology.orchestra.run.smithi183.stdout: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:48:45.536 INFO:teuthology.orchestra.run.smithi183.stdout: cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:48:45.536 INFO:teuthology.orchestra.run.smithi183.stdout: cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T15:48:45.536 INFO:teuthology.orchestra.run.smithi183.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-28T15:48:45.536 INFO:teuthology.orchestra.run.smithi183.stdout: flex-2.6.4-9.el9.x86_64 2024-03-28T15:48:45.536 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-28T15:48:45.536 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-28T15:48:45.536 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-28T15:48:45.537 INFO:teuthology.orchestra.run.smithi183.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-28T15:48:45.537 INFO:teuthology.orchestra.run.smithi183.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-28T15:48:45.537 INFO:teuthology.orchestra.run.smithi183.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-28T15:48:45.537 INFO:teuthology.orchestra.run.smithi183.stdout: jq-1.6-16.el9.x86_64 2024-03-28T15:48:45.537 INFO:teuthology.orchestra.run.smithi183.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-28T15:48:45.537 INFO:teuthology.orchestra.run.smithi183.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-28T15:48:45.537 INFO:teuthology.orchestra.run.smithi183.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-28T15:48:45.537 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:48:45.537 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:48:45.537 INFO:teuthology.orchestra.run.smithi183.stdout: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:48:45.537 INFO:teuthology.orchestra.run.smithi183.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-28T15:48:45.537 INFO:teuthology.orchestra.run.smithi183.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-28T15:48:45.537 INFO:teuthology.orchestra.run.smithi183.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-28T15:48:45.537 INFO:teuthology.orchestra.run.smithi183.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-28T15:48:45.537 INFO:teuthology.orchestra.run.smithi183.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-28T15:48:45.537 INFO:teuthology.orchestra.run.smithi183.stdout: librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:48:45.538 INFO:teuthology.orchestra.run.smithi183.stdout: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:48:45.538 INFO:teuthology.orchestra.run.smithi183.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-28T15:48:45.538 INFO:teuthology.orchestra.run.smithi183.stdout: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:48:45.538 INFO:teuthology.orchestra.run.smithi183.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-28T15:48:45.538 INFO:teuthology.orchestra.run.smithi183.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-28T15:48:45.538 INFO:teuthology.orchestra.run.smithi183.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-28T15:48:45.538 INFO:teuthology.orchestra.run.smithi183.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-28T15:48:45.538 INFO:teuthology.orchestra.run.smithi183.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-28T15:48:45.538 INFO:teuthology.orchestra.run.smithi183.stdout: lua-5.4.4-4.el9.x86_64 2024-03-28T15:48:45.538 INFO:teuthology.orchestra.run.smithi183.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-28T15:48:45.538 INFO:teuthology.orchestra.run.smithi183.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-28T15:48:45.538 INFO:teuthology.orchestra.run.smithi183.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-28T15:48:45.538 INFO:teuthology.orchestra.run.smithi183.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-28T15:48:45.538 INFO:teuthology.orchestra.run.smithi183.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-28T15:48:45.538 INFO:teuthology.orchestra.run.smithi183.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-28T15:48:45.539 INFO:teuthology.orchestra.run.smithi183.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-28T15:48:45.539 INFO:teuthology.orchestra.run.smithi183.stdout: postgresql-13.14-1.el9.x86_64 2024-03-28T15:48:45.539 INFO:teuthology.orchestra.run.smithi183.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-03-28T15:48:45.539 INFO:teuthology.orchestra.run.smithi183.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-03-28T15:48:45.539 INFO:teuthology.orchestra.run.smithi183.stdout: postgresql-server-13.14-1.el9.x86_64 2024-03-28T15:48:45.539 INFO:teuthology.orchestra.run.smithi183.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-28T15:48:45.539 INFO:teuthology.orchestra.run.smithi183.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-28T15:48:45.539 INFO:teuthology.orchestra.run.smithi183.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-28T15:48:45.539 INFO:teuthology.orchestra.run.smithi183.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-28T15:48:45.539 INFO:teuthology.orchestra.run.smithi183.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-28T15:48:45.539 INFO:teuthology.orchestra.run.smithi183.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-28T15:48:45.539 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-28T15:48:45.539 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:48:45.539 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:48:45.539 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:48:45.540 INFO:teuthology.orchestra.run.smithi183.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-28T15:48:45.540 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-28T15:48:45.540 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-28T15:48:45.540 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-28T15:48:45.540 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-28T15:48:45.540 INFO:teuthology.orchestra.run.smithi183.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-28T15:48:45.540 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-28T15:48:45.540 INFO:teuthology.orchestra.run.smithi183.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-28T15:48:45.540 INFO:teuthology.orchestra.run.smithi183.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-28T15:48:45.540 INFO:teuthology.orchestra.run.smithi183.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-28T15:48:45.540 INFO:teuthology.orchestra.run.smithi183.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-28T15:48:45.540 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-28T15:48:45.540 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-28T15:48:45.540 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-28T15:48:45.540 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-28T15:48:45.540 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-28T15:48:45.541 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-28T15:48:45.541 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-28T15:48:45.541 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-28T15:48:45.541 INFO:teuthology.orchestra.run.smithi183.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-28T15:48:45.541 INFO:teuthology.orchestra.run.smithi183.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-28T15:48:45.541 INFO:teuthology.orchestra.run.smithi183.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-28T15:48:45.541 INFO:teuthology.orchestra.run.smithi183.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-28T15:48:45.541 INFO:teuthology.orchestra.run.smithi183.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-28T15:48:45.541 INFO:teuthology.orchestra.run.smithi183.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-28T15:48:45.541 INFO:teuthology.orchestra.run.smithi183.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-28T15:48:45.541 INFO:teuthology.orchestra.run.smithi183.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-28T15:48:45.541 INFO:teuthology.orchestra.run.smithi183.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-28T15:48:45.541 INFO:teuthology.orchestra.run.smithi183.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-28T15:48:45.541 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-28T15:48:45.541 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ply-3.11-14.el9.noarch 2024-03-28T15:48:45.541 INFO:teuthology.orchestra.run.smithi183.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-28T15:48:45.541 INFO:teuthology.orchestra.run.smithi183.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-28T15:48:45.542 INFO:teuthology.orchestra.run.smithi183.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-28T15:48:45.542 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-28T15:48:45.542 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-28T15:48:45.542 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-28T15:48:45.542 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-28T15:48:45.542 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-28T15:48:45.542 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-28T15:48:45.542 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:48:45.542 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:48:45.542 INFO:teuthology.orchestra.run.smithi183.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-28T15:48:45.542 INFO:teuthology.orchestra.run.smithi183.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-28T15:48:45.542 INFO:teuthology.orchestra.run.smithi183.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-28T15:48:45.542 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:48:45.542 INFO:teuthology.orchestra.run.smithi183.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-28T15:48:45.542 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-28T15:48:45.542 INFO:teuthology.orchestra.run.smithi183.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-28T15:48:45.543 INFO:teuthology.orchestra.run.smithi183.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-28T15:48:45.543 INFO:teuthology.orchestra.run.smithi183.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-28T15:48:45.543 INFO:teuthology.orchestra.run.smithi183.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-28T15:48:45.543 INFO:teuthology.orchestra.run.smithi183.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-28T15:48:45.543 INFO:teuthology.orchestra.run.smithi183.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-28T15:48:45.543 INFO:teuthology.orchestra.run.smithi183.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-28T15:48:45.543 INFO:teuthology.orchestra.run.smithi183.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-28T15:48:45.543 INFO:teuthology.orchestra.run.smithi183.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-28T15:48:45.543 INFO:teuthology.orchestra.run.smithi183.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-28T15:48:45.543 INFO:teuthology.orchestra.run.smithi183.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-28T15:48:45.543 INFO:teuthology.orchestra.run.smithi183.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-28T15:48:45.543 INFO:teuthology.orchestra.run.smithi183.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-28T15:48:45.543 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:48:45.543 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:48:45.543 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:48:45.543 INFO:teuthology.orchestra.run.smithi183.stdout: re2-1:20211101-3.el9.x86_64 2024-03-28T15:48:45.544 INFO:teuthology.orchestra.run.smithi183.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-28T15:48:45.544 INFO:teuthology.orchestra.run.smithi183.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-28T15:48:45.544 INFO:teuthology.orchestra.run.smithi183.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-28T15:48:45.544 INFO:teuthology.orchestra.run.smithi183.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-28T15:48:45.544 INFO:teuthology.orchestra.run.smithi183.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-28T15:48:45.544 INFO:teuthology.orchestra.run.smithi183.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-28T15:48:45.544 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T15:48:45.544 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-28T15:48:46.203 DEBUG:teuthology.parallel:result is None 2024-03-28T15:48:46.410 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-28T15:48:46.410 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-28T15:48:46.410 INFO:teuthology.orchestra.run.smithi150.stdout:Upgraded: 2024-03-28T15:48:46.410 INFO:teuthology.orchestra.run.smithi150.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-28T15:48:46.410 INFO:teuthology.orchestra.run.smithi150.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-28T15:48:46.411 INFO:teuthology.orchestra.run.smithi150.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-28T15:48:46.411 INFO:teuthology.orchestra.run.smithi150.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-28T15:48:46.411 INFO:teuthology.orchestra.run.smithi150.stdout: acl-2.3.1-4.el9.x86_64 2024-03-28T15:48:46.411 INFO:teuthology.orchestra.run.smithi150.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-28T15:48:46.411 INFO:teuthology.orchestra.run.smithi150.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-28T15:48:46.411 INFO:teuthology.orchestra.run.smithi150.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-28T15:48:46.411 INFO:teuthology.orchestra.run.smithi150.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-28T15:48:46.411 INFO:teuthology.orchestra.run.smithi150.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-28T15:48:46.411 INFO:teuthology.orchestra.run.smithi150.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-28T15:48:46.411 INFO:teuthology.orchestra.run.smithi150.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-28T15:48:46.411 INFO:teuthology.orchestra.run.smithi150.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-28T15:48:46.411 INFO:teuthology.orchestra.run.smithi150.stdout: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:48:46.411 INFO:teuthology.orchestra.run.smithi150.stdout: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:48:46.411 INFO:teuthology.orchestra.run.smithi150.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-28T15:48:46.411 INFO:teuthology.orchestra.run.smithi150.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-28T15:48:46.411 INFO:teuthology.orchestra.run.smithi150.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-28T15:48:46.412 INFO:teuthology.orchestra.run.smithi150.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-28T15:48:46.412 INFO:teuthology.orchestra.run.smithi150.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-28T15:48:46.412 INFO:teuthology.orchestra.run.smithi150.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-28T15:48:46.412 INFO:teuthology.orchestra.run.smithi150.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-28T15:48:46.412 INFO:teuthology.orchestra.run.smithi150.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-28T15:48:46.412 INFO:teuthology.orchestra.run.smithi150.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-28T15:48:46.412 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-03-28T15:48:46.412 INFO:teuthology.orchestra.run.smithi150.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-28T15:48:46.412 INFO:teuthology.orchestra.run.smithi150.stdout: automake-1.16.2-8.el9.noarch 2024-03-28T15:48:46.412 INFO:teuthology.orchestra.run.smithi150.stdout: bison-3.7.4-5.el9.x86_64 2024-03-28T15:48:46.412 INFO:teuthology.orchestra.run.smithi150.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-28T15:48:46.412 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:48:46.412 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:48:46.412 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:48:46.412 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:48:46.412 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T15:48:46.413 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:48:46.413 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:48:46.413 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:48:46.413 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T15:48:46.413 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T15:48:46.413 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T15:48:46.413 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T15:48:46.413 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T15:48:46.413 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:48:46.413 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:48:46.413 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T15:48:46.413 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:48:46.413 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:48:46.413 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:48:46.413 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T15:48:46.414 INFO:teuthology.orchestra.run.smithi150.stdout: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T15:48:46.414 INFO:teuthology.orchestra.run.smithi150.stdout: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:48:46.414 INFO:teuthology.orchestra.run.smithi150.stdout: cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:48:46.414 INFO:teuthology.orchestra.run.smithi150.stdout: cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T15:48:46.414 INFO:teuthology.orchestra.run.smithi150.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-28T15:48:46.414 INFO:teuthology.orchestra.run.smithi150.stdout: flex-2.6.4-9.el9.x86_64 2024-03-28T15:48:46.414 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-28T15:48:46.414 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-28T15:48:46.414 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-28T15:48:46.414 INFO:teuthology.orchestra.run.smithi150.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-28T15:48:46.414 INFO:teuthology.orchestra.run.smithi150.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-28T15:48:46.414 INFO:teuthology.orchestra.run.smithi150.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-28T15:48:46.414 INFO:teuthology.orchestra.run.smithi150.stdout: jq-1.6-16.el9.x86_64 2024-03-28T15:48:46.414 INFO:teuthology.orchestra.run.smithi150.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-28T15:48:46.415 INFO:teuthology.orchestra.run.smithi150.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-28T15:48:46.415 INFO:teuthology.orchestra.run.smithi150.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-28T15:48:46.415 INFO:teuthology.orchestra.run.smithi150.stdout: libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:48:46.415 INFO:teuthology.orchestra.run.smithi150.stdout: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:48:46.415 INFO:teuthology.orchestra.run.smithi150.stdout: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:48:46.415 INFO:teuthology.orchestra.run.smithi150.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-28T15:48:46.415 INFO:teuthology.orchestra.run.smithi150.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-28T15:48:46.415 INFO:teuthology.orchestra.run.smithi150.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-28T15:48:46.415 INFO:teuthology.orchestra.run.smithi150.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-28T15:48:46.415 INFO:teuthology.orchestra.run.smithi150.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-28T15:48:46.415 INFO:teuthology.orchestra.run.smithi150.stdout: librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:48:46.415 INFO:teuthology.orchestra.run.smithi150.stdout: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:48:46.415 INFO:teuthology.orchestra.run.smithi150.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-28T15:48:46.415 INFO:teuthology.orchestra.run.smithi150.stdout: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:48:46.415 INFO:teuthology.orchestra.run.smithi150.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-28T15:48:46.415 INFO:teuthology.orchestra.run.smithi150.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-28T15:48:46.416 INFO:teuthology.orchestra.run.smithi150.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-28T15:48:46.416 INFO:teuthology.orchestra.run.smithi150.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-28T15:48:46.416 INFO:teuthology.orchestra.run.smithi150.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-28T15:48:46.416 INFO:teuthology.orchestra.run.smithi150.stdout: lua-5.4.4-4.el9.x86_64 2024-03-28T15:48:46.416 INFO:teuthology.orchestra.run.smithi150.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-28T15:48:46.416 INFO:teuthology.orchestra.run.smithi150.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-28T15:48:46.416 INFO:teuthology.orchestra.run.smithi150.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-28T15:48:46.416 INFO:teuthology.orchestra.run.smithi150.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-28T15:48:46.416 INFO:teuthology.orchestra.run.smithi150.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-28T15:48:46.416 INFO:teuthology.orchestra.run.smithi150.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-28T15:48:46.416 INFO:teuthology.orchestra.run.smithi150.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-28T15:48:46.416 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql-13.14-1.el9.x86_64 2024-03-28T15:48:46.416 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-03-28T15:48:46.416 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-03-28T15:48:46.416 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql-server-13.14-1.el9.x86_64 2024-03-28T15:48:46.416 INFO:teuthology.orchestra.run.smithi150.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-28T15:48:46.417 INFO:teuthology.orchestra.run.smithi150.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-28T15:48:46.417 INFO:teuthology.orchestra.run.smithi150.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-28T15:48:46.417 INFO:teuthology.orchestra.run.smithi150.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-28T15:48:46.417 INFO:teuthology.orchestra.run.smithi150.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-28T15:48:46.417 INFO:teuthology.orchestra.run.smithi150.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-28T15:48:46.417 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-28T15:48:46.417 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:48:46.417 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:48:46.417 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:48:46.417 INFO:teuthology.orchestra.run.smithi150.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-28T15:48:46.417 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-28T15:48:46.417 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-28T15:48:46.417 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-28T15:48:46.417 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-28T15:48:46.417 INFO:teuthology.orchestra.run.smithi150.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-28T15:48:46.417 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-28T15:48:46.418 INFO:teuthology.orchestra.run.smithi150.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-28T15:48:46.418 INFO:teuthology.orchestra.run.smithi150.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-28T15:48:46.418 INFO:teuthology.orchestra.run.smithi150.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-28T15:48:46.418 INFO:teuthology.orchestra.run.smithi150.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-28T15:48:46.418 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-28T15:48:46.418 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-28T15:48:46.418 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-28T15:48:46.418 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-28T15:48:46.418 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-28T15:48:46.418 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-28T15:48:46.418 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-28T15:48:46.418 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-28T15:48:46.418 INFO:teuthology.orchestra.run.smithi150.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-28T15:48:46.418 INFO:teuthology.orchestra.run.smithi150.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-28T15:48:46.419 INFO:teuthology.orchestra.run.smithi150.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-28T15:48:46.419 INFO:teuthology.orchestra.run.smithi150.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-28T15:48:46.419 INFO:teuthology.orchestra.run.smithi150.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-28T15:48:46.419 INFO:teuthology.orchestra.run.smithi150.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-28T15:48:46.419 INFO:teuthology.orchestra.run.smithi150.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-28T15:48:46.419 INFO:teuthology.orchestra.run.smithi150.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-28T15:48:46.419 INFO:teuthology.orchestra.run.smithi150.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-28T15:48:46.419 INFO:teuthology.orchestra.run.smithi150.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-28T15:48:46.419 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-28T15:48:46.419 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ply-3.11-14.el9.noarch 2024-03-28T15:48:46.419 INFO:teuthology.orchestra.run.smithi150.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-28T15:48:46.419 INFO:teuthology.orchestra.run.smithi150.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-28T15:48:46.419 INFO:teuthology.orchestra.run.smithi150.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-28T15:48:46.419 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-28T15:48:46.419 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-28T15:48:46.420 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-28T15:48:46.420 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-28T15:48:46.420 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-28T15:48:46.420 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-28T15:48:46.420 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:48:46.420 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:48:46.420 INFO:teuthology.orchestra.run.smithi150.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-28T15:48:46.420 INFO:teuthology.orchestra.run.smithi150.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-28T15:48:46.420 INFO:teuthology.orchestra.run.smithi150.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-28T15:48:46.420 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:48:46.420 INFO:teuthology.orchestra.run.smithi150.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-28T15:48:46.420 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-28T15:48:46.420 INFO:teuthology.orchestra.run.smithi150.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-28T15:48:46.420 INFO:teuthology.orchestra.run.smithi150.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-28T15:48:46.420 INFO:teuthology.orchestra.run.smithi150.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-28T15:48:46.420 INFO:teuthology.orchestra.run.smithi150.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-28T15:48:46.421 INFO:teuthology.orchestra.run.smithi150.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-28T15:48:46.421 INFO:teuthology.orchestra.run.smithi150.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-28T15:48:46.421 INFO:teuthology.orchestra.run.smithi150.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-28T15:48:46.421 INFO:teuthology.orchestra.run.smithi150.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-28T15:48:46.421 INFO:teuthology.orchestra.run.smithi150.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-28T15:48:46.421 INFO:teuthology.orchestra.run.smithi150.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-28T15:48:46.421 INFO:teuthology.orchestra.run.smithi150.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-28T15:48:46.421 INFO:teuthology.orchestra.run.smithi150.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-28T15:48:46.421 INFO:teuthology.orchestra.run.smithi150.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-28T15:48:46.421 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:48:46.421 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:48:46.421 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:48:46.421 INFO:teuthology.orchestra.run.smithi150.stdout: re2-1:20211101-3.el9.x86_64 2024-03-28T15:48:46.421 INFO:teuthology.orchestra.run.smithi150.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-28T15:48:46.421 INFO:teuthology.orchestra.run.smithi150.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-28T15:48:46.421 INFO:teuthology.orchestra.run.smithi150.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-28T15:48:46.422 INFO:teuthology.orchestra.run.smithi150.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-28T15:48:46.422 INFO:teuthology.orchestra.run.smithi150.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-28T15:48:46.422 INFO:teuthology.orchestra.run.smithi150.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-28T15:48:46.422 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-28T15:48:46.422 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-03-28T15:48:47.096 DEBUG:teuthology.parallel:result is None 2024-03-28T15:48:49.511 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-28T15:48:49.565 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 154/198 2024-03-28T15:48:49.599 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 155/198 2024-03-28T15:48:49.638 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-28T15:48:49.927 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 157/198 2024-03-28T15:48:50.073 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 158/198 2024-03-28T15:48:50.125 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 159/198 2024-03-28T15:48:50.188 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-28T15:48:50.242 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-28T15:48:50.296 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-28T15:48:50.339 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 163/198 2024-03-28T15:48:50.383 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-28T15:48:50.435 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-28T15:48:50.526 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-28T15:48:50.650 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-28T15:48:50.692 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-28T15:48:51.795 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T15:48:51.839 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T15:48:51.896 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T15:48:53.305 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 170/198 2024-03-28T15:48:53.402 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-28T15:48:53.453 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-28T15:48:53.583 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-28T15:48:53.623 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-28T15:48:53.655 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-28T15:48:53.689 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-28T15:48:53.731 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-28T15:48:53.731 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T15:48:53.945 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T15:48:53.986 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-28T15:48:53.986 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T15:48:54.166 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T15:48:54.214 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-28T15:48:54.256 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-28T15:48:54.289 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-28T15:48:54.338 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-28T15:48:54.408 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-28T15:48:54.496 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-28T15:48:54.544 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-28T15:48:54.576 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-28T15:48:54.611 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-28T15:48:54.644 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-28T15:48:54.677 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-28T15:48:54.716 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T15:48:54.717 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T15:48:55.060 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T15:48:55.060 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T15:48:55.208 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T15:48:55.238 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-28T15:48:55.292 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-28T15:48:55.324 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-28T15:48:55.364 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-28T15:48:55.518 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T15:48:55.834 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T15:48:55.834 INFO:teuthology.orchestra.run.smithi067.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-03-28T15:48:55.834 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T15:49:00.865 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 1/198 2024-03-28T15:49:00.865 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2/198 2024-03-28T15:49:00.865 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 3/198 2024-03-28T15:49:00.866 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 4/198 2024-03-28T15:49:00.866 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 5/198 2024-03-28T15:49:00.866 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 6/198 2024-03-28T15:49:00.866 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 7/198 2024-03-28T15:49:00.866 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 8/198 2024-03-28T15:49:00.866 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 9/198 2024-03-28T15:49:00.866 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T15:49:00.866 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T15:49:00.866 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-28T15:49:00.866 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 13/198 2024-03-28T15:49:00.866 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 14/198 2024-03-28T15:49:00.866 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 15/198 2024-03-28T15:49:00.867 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T15:49:00.867 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 17/198 2024-03-28T15:49:00.867 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 18/198 2024-03-28T15:49:00.867 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 19/198 2024-03-28T15:49:00.867 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 20/198 2024-03-28T15:49:00.867 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 21/198 2024-03-28T15:49:00.867 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 22/198 2024-03-28T15:49:00.867 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 23/198 2024-03-28T15:49:00.867 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 24/198 2024-03-28T15:49:00.867 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 25/198 2024-03-28T15:49:00.867 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 26/198 2024-03-28T15:49:00.867 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 27/198 2024-03-28T15:49:00.868 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 28/198 2024-03-28T15:49:00.868 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 29/198 2024-03-28T15:49:00.884 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 30/198 2024-03-28T15:49:00.884 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 31/198 2024-03-28T15:49:00.884 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 32/198 2024-03-28T15:49:00.884 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 33/198 2024-03-28T15:49:00.884 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 34/198 2024-03-28T15:49:00.884 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 35/198 2024-03-28T15:49:00.884 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 36/198 2024-03-28T15:49:00.884 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 37/198 2024-03-28T15:49:00.884 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 38/198 2024-03-28T15:49:00.884 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 39/198 2024-03-28T15:49:00.884 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-28T15:49:00.884 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-28T15:49:00.885 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-28T15:49:00.885 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-28T15:49:00.885 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-28T15:49:00.885 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-28T15:49:00.885 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-28T15:49:00.885 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-28T15:49:00.885 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-28T15:49:00.885 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-28T15:49:00.885 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-28T15:49:00.885 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-28T15:49:00.885 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-28T15:49:00.885 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-28T15:49:00.885 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-28T15:49:00.885 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-28T15:49:00.885 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-28T15:49:00.885 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-28T15:49:00.886 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 58/198 2024-03-28T15:49:00.886 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 59/198 2024-03-28T15:49:00.886 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 60/198 2024-03-28T15:49:00.886 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 61/198 2024-03-28T15:49:00.886 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 62/198 2024-03-28T15:49:00.886 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 63/198 2024-03-28T15:49:00.886 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 64/198 2024-03-28T15:49:00.886 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 65/198 2024-03-28T15:49:00.886 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 66/198 2024-03-28T15:49:00.886 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 67/198 2024-03-28T15:49:00.886 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 68/198 2024-03-28T15:49:00.886 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 69/198 2024-03-28T15:49:00.886 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 70/198 2024-03-28T15:49:00.887 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 71/198 2024-03-28T15:49:00.887 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 72/198 2024-03-28T15:49:00.887 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 73/198 2024-03-28T15:49:00.887 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 74/198 2024-03-28T15:49:00.887 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 75/198 2024-03-28T15:49:00.887 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 76/198 2024-03-28T15:49:00.887 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 77/198 2024-03-28T15:49:00.887 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 78/198 2024-03-28T15:49:00.887 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 79/198 2024-03-28T15:49:00.887 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 80/198 2024-03-28T15:49:00.887 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 81/198 2024-03-28T15:49:00.887 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 82/198 2024-03-28T15:49:00.887 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 83/198 2024-03-28T15:49:00.887 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 84/198 2024-03-28T15:49:00.887 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-ply-3.11-14.el9.noarch 85/198 2024-03-28T15:49:00.888 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 86/198 2024-03-28T15:49:00.888 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 87/198 2024-03-28T15:49:00.888 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : jq-1.6-16.el9.x86_64 88/198 2024-03-28T15:49:00.888 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 89/198 2024-03-28T15:49:00.888 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 90/198 2024-03-28T15:49:00.888 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 91/198 2024-03-28T15:49:00.888 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 92/198 2024-03-28T15:49:00.888 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 93/198 2024-03-28T15:49:00.888 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 94/198 2024-03-28T15:49:00.888 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : postgresql-13.14-1.el9.x86_64 95/198 2024-03-28T15:49:00.888 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 96/198 2024-03-28T15:49:00.888 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 97/198 2024-03-28T15:49:00.888 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 98/198 2024-03-28T15:49:00.888 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-28T15:49:00.888 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-28T15:49:00.888 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 101/198 2024-03-28T15:49:00.889 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 102/198 2024-03-28T15:49:00.889 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 103/198 2024-03-28T15:49:00.889 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-28T15:49:00.889 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-28T15:49:00.889 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-28T15:49:00.889 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-28T15:49:00.889 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-28T15:49:00.889 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-28T15:49:00.889 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-28T15:49:00.889 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-28T15:49:00.889 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-28T15:49:00.889 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-28T15:49:00.889 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-28T15:49:00.889 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-28T15:49:00.889 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-28T15:49:00.889 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-28T15:49:00.890 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-28T15:49:00.890 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-28T15:49:00.890 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-28T15:49:00.890 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-28T15:49:00.890 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-28T15:49:00.890 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-28T15:49:00.890 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-28T15:49:00.890 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-28T15:49:00.890 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-28T15:49:00.890 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-28T15:49:00.890 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-28T15:49:00.890 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-28T15:49:00.890 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-28T15:49:00.890 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-28T15:49:00.890 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-28T15:49:00.891 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-28T15:49:00.891 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-28T15:49:00.891 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-28T15:49:00.891 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-28T15:49:00.891 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-28T15:49:00.891 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-28T15:49:00.891 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-28T15:49:00.891 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-28T15:49:00.891 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-28T15:49:00.891 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-28T15:49:00.891 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-28T15:49:00.891 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-28T15:49:00.891 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-28T15:49:00.891 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-28T15:49:00.891 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-28T15:49:00.891 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-28T15:49:00.892 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-28T15:49:00.892 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-28T15:49:00.892 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T15:49:00.892 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-28T15:49:00.892 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-28T15:49:00.892 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-28T15:49:00.892 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-28T15:49:00.892 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-28T15:49:00.892 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-28T15:49:00.892 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-28T15:49:00.892 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-28T15:49:00.892 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-28T15:49:00.892 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-28T15:49:00.892 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-28T15:49:00.892 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-28T15:49:00.893 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-28T15:49:00.893 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-28T15:49:00.893 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-28T15:49:00.893 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-28T15:49:00.893 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-28T15:49:00.893 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-28T15:49:00.893 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-28T15:49:00.893 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-28T15:49:00.893 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-28T15:49:00.893 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-28T15:49:00.893 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-28T15:49:00.893 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-28T15:49:00.893 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-28T15:49:00.893 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-28T15:49:00.893 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-28T15:49:00.893 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-28T15:49:00.894 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-28T15:49:00.894 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-28T15:49:00.894 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-28T15:49:00.894 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-28T15:49:00.894 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-28T15:49:00.894 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-28T15:49:00.894 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-28T15:49:00.894 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-28T15:49:00.894 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-28T15:49:00.894 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-28T15:49:00.894 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-28T15:49:00.894 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-28T15:49:00.894 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-28T15:49:00.894 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-28T15:49:00.894 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-28T15:49:00.895 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-28T15:49:00.895 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-28T15:49:00.895 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-28T15:49:03.324 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-28T15:49:03.325 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T15:49:03.325 INFO:teuthology.orchestra.run.smithi067.stdout:Upgraded: 2024-03-28T15:49:03.325 INFO:teuthology.orchestra.run.smithi067.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-28T15:49:03.325 INFO:teuthology.orchestra.run.smithi067.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-28T15:49:03.325 INFO:teuthology.orchestra.run.smithi067.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-28T15:49:03.325 INFO:teuthology.orchestra.run.smithi067.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-28T15:49:03.325 INFO:teuthology.orchestra.run.smithi067.stdout: acl-2.3.1-4.el9.x86_64 2024-03-28T15:49:03.325 INFO:teuthology.orchestra.run.smithi067.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-28T15:49:03.325 INFO:teuthology.orchestra.run.smithi067.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-28T15:49:03.325 INFO:teuthology.orchestra.run.smithi067.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-28T15:49:03.325 INFO:teuthology.orchestra.run.smithi067.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-28T15:49:03.326 INFO:teuthology.orchestra.run.smithi067.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-28T15:49:03.326 INFO:teuthology.orchestra.run.smithi067.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-28T15:49:03.326 INFO:teuthology.orchestra.run.smithi067.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-28T15:49:03.326 INFO:teuthology.orchestra.run.smithi067.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-28T15:49:03.326 INFO:teuthology.orchestra.run.smithi067.stdout: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:49:03.326 INFO:teuthology.orchestra.run.smithi067.stdout: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:49:03.326 INFO:teuthology.orchestra.run.smithi067.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-28T15:49:03.326 INFO:teuthology.orchestra.run.smithi067.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-28T15:49:03.326 INFO:teuthology.orchestra.run.smithi067.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-28T15:49:03.326 INFO:teuthology.orchestra.run.smithi067.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-28T15:49:03.326 INFO:teuthology.orchestra.run.smithi067.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-28T15:49:03.326 INFO:teuthology.orchestra.run.smithi067.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-28T15:49:03.326 INFO:teuthology.orchestra.run.smithi067.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-28T15:49:03.327 INFO:teuthology.orchestra.run.smithi067.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-28T15:49:03.327 INFO:teuthology.orchestra.run.smithi067.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-28T15:49:03.327 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-03-28T15:49:03.327 INFO:teuthology.orchestra.run.smithi067.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-28T15:49:03.327 INFO:teuthology.orchestra.run.smithi067.stdout: automake-1.16.2-8.el9.noarch 2024-03-28T15:49:03.327 INFO:teuthology.orchestra.run.smithi067.stdout: bison-3.7.4-5.el9.x86_64 2024-03-28T15:49:03.327 INFO:teuthology.orchestra.run.smithi067.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-28T15:49:03.327 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:49:03.327 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:49:03.327 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:49:03.327 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:49:03.327 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T15:49:03.327 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:49:03.327 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:49:03.327 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:49:03.328 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T15:49:03.328 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T15:49:03.328 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T15:49:03.328 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T15:49:03.328 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T15:49:03.328 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:49:03.328 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:49:03.328 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T15:49:03.328 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:49:03.328 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:49:03.328 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:49:03.328 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T15:49:03.328 INFO:teuthology.orchestra.run.smithi067.stdout: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T15:49:03.328 INFO:teuthology.orchestra.run.smithi067.stdout: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:49:03.328 INFO:teuthology.orchestra.run.smithi067.stdout: cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:49:03.329 INFO:teuthology.orchestra.run.smithi067.stdout: cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T15:49:03.329 INFO:teuthology.orchestra.run.smithi067.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-28T15:49:03.329 INFO:teuthology.orchestra.run.smithi067.stdout: flex-2.6.4-9.el9.x86_64 2024-03-28T15:49:03.329 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-28T15:49:03.329 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-28T15:49:03.329 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-28T15:49:03.329 INFO:teuthology.orchestra.run.smithi067.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-28T15:49:03.329 INFO:teuthology.orchestra.run.smithi067.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-28T15:49:03.329 INFO:teuthology.orchestra.run.smithi067.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-28T15:49:03.329 INFO:teuthology.orchestra.run.smithi067.stdout: jq-1.6-16.el9.x86_64 2024-03-28T15:49:03.329 INFO:teuthology.orchestra.run.smithi067.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-28T15:49:03.329 INFO:teuthology.orchestra.run.smithi067.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-28T15:49:03.329 INFO:teuthology.orchestra.run.smithi067.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-28T15:49:03.329 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:49:03.330 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:49:03.330 INFO:teuthology.orchestra.run.smithi067.stdout: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:49:03.330 INFO:teuthology.orchestra.run.smithi067.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-28T15:49:03.330 INFO:teuthology.orchestra.run.smithi067.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-28T15:49:03.330 INFO:teuthology.orchestra.run.smithi067.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-28T15:49:03.330 INFO:teuthology.orchestra.run.smithi067.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-28T15:49:03.330 INFO:teuthology.orchestra.run.smithi067.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-28T15:49:03.330 INFO:teuthology.orchestra.run.smithi067.stdout: librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:49:03.330 INFO:teuthology.orchestra.run.smithi067.stdout: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:49:03.330 INFO:teuthology.orchestra.run.smithi067.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-28T15:49:03.330 INFO:teuthology.orchestra.run.smithi067.stdout: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:49:03.330 INFO:teuthology.orchestra.run.smithi067.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-28T15:49:03.330 INFO:teuthology.orchestra.run.smithi067.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-28T15:49:03.330 INFO:teuthology.orchestra.run.smithi067.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-28T15:49:03.331 INFO:teuthology.orchestra.run.smithi067.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-28T15:49:03.331 INFO:teuthology.orchestra.run.smithi067.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-28T15:49:03.331 INFO:teuthology.orchestra.run.smithi067.stdout: lua-5.4.4-4.el9.x86_64 2024-03-28T15:49:03.331 INFO:teuthology.orchestra.run.smithi067.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-28T15:49:03.331 INFO:teuthology.orchestra.run.smithi067.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-28T15:49:03.331 INFO:teuthology.orchestra.run.smithi067.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-28T15:49:03.331 INFO:teuthology.orchestra.run.smithi067.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-28T15:49:03.331 INFO:teuthology.orchestra.run.smithi067.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-28T15:49:03.331 INFO:teuthology.orchestra.run.smithi067.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-28T15:49:03.331 INFO:teuthology.orchestra.run.smithi067.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-28T15:49:03.331 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql-13.14-1.el9.x86_64 2024-03-28T15:49:03.331 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-03-28T15:49:03.331 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-03-28T15:49:03.331 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql-server-13.14-1.el9.x86_64 2024-03-28T15:49:03.331 INFO:teuthology.orchestra.run.smithi067.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-28T15:49:03.332 INFO:teuthology.orchestra.run.smithi067.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-28T15:49:03.332 INFO:teuthology.orchestra.run.smithi067.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-28T15:49:03.332 INFO:teuthology.orchestra.run.smithi067.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-28T15:49:03.332 INFO:teuthology.orchestra.run.smithi067.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-28T15:49:03.332 INFO:teuthology.orchestra.run.smithi067.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-28T15:49:03.332 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-28T15:49:03.332 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:49:03.332 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:49:03.332 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:49:03.332 INFO:teuthology.orchestra.run.smithi067.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-28T15:49:03.332 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-28T15:49:03.332 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-28T15:49:03.332 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-28T15:49:03.332 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-28T15:49:03.332 INFO:teuthology.orchestra.run.smithi067.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-28T15:49:03.332 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-28T15:49:03.333 INFO:teuthology.orchestra.run.smithi067.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-28T15:49:03.333 INFO:teuthology.orchestra.run.smithi067.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-28T15:49:03.333 INFO:teuthology.orchestra.run.smithi067.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-28T15:49:03.333 INFO:teuthology.orchestra.run.smithi067.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-28T15:49:03.333 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-28T15:49:03.333 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-28T15:49:03.333 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-28T15:49:03.333 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-28T15:49:03.333 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-28T15:49:03.333 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-28T15:49:03.333 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-28T15:49:03.333 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-28T15:49:03.333 INFO:teuthology.orchestra.run.smithi067.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-28T15:49:03.333 INFO:teuthology.orchestra.run.smithi067.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-28T15:49:03.333 INFO:teuthology.orchestra.run.smithi067.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-28T15:49:03.334 INFO:teuthology.orchestra.run.smithi067.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-28T15:49:03.334 INFO:teuthology.orchestra.run.smithi067.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-28T15:49:03.334 INFO:teuthology.orchestra.run.smithi067.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-28T15:49:03.334 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-28T15:49:03.334 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-28T15:49:03.334 INFO:teuthology.orchestra.run.smithi067.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-28T15:49:03.334 INFO:teuthology.orchestra.run.smithi067.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-28T15:49:03.334 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-28T15:49:03.334 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ply-3.11-14.el9.noarch 2024-03-28T15:49:03.334 INFO:teuthology.orchestra.run.smithi067.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-28T15:49:03.334 INFO:teuthology.orchestra.run.smithi067.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-28T15:49:03.334 INFO:teuthology.orchestra.run.smithi067.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-28T15:49:03.334 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-28T15:49:03.334 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-28T15:49:03.335 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-28T15:49:03.335 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-28T15:49:03.335 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-28T15:49:03.335 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-28T15:49:03.335 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:49:03.335 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:49:03.335 INFO:teuthology.orchestra.run.smithi067.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-28T15:49:03.335 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-28T15:49:03.335 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-28T15:49:03.335 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:49:03.335 INFO:teuthology.orchestra.run.smithi067.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-28T15:49:03.335 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-28T15:49:03.335 INFO:teuthology.orchestra.run.smithi067.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-28T15:49:03.335 INFO:teuthology.orchestra.run.smithi067.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-28T15:49:03.336 INFO:teuthology.orchestra.run.smithi067.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-28T15:49:03.336 INFO:teuthology.orchestra.run.smithi067.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-28T15:49:03.336 INFO:teuthology.orchestra.run.smithi067.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-28T15:49:03.336 INFO:teuthology.orchestra.run.smithi067.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-28T15:49:03.336 INFO:teuthology.orchestra.run.smithi067.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-28T15:49:03.336 INFO:teuthology.orchestra.run.smithi067.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-28T15:49:03.336 INFO:teuthology.orchestra.run.smithi067.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-28T15:49:03.336 INFO:teuthology.orchestra.run.smithi067.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-28T15:49:03.336 INFO:teuthology.orchestra.run.smithi067.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-28T15:49:03.336 INFO:teuthology.orchestra.run.smithi067.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-28T15:49:03.336 INFO:teuthology.orchestra.run.smithi067.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-28T15:49:03.336 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:49:03.336 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:49:03.336 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:49:03.336 INFO:teuthology.orchestra.run.smithi067.stdout: re2-1:20211101-3.el9.x86_64 2024-03-28T15:49:03.336 INFO:teuthology.orchestra.run.smithi067.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-28T15:49:03.337 INFO:teuthology.orchestra.run.smithi067.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-28T15:49:03.337 INFO:teuthology.orchestra.run.smithi067.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-28T15:49:03.337 INFO:teuthology.orchestra.run.smithi067.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-28T15:49:03.337 INFO:teuthology.orchestra.run.smithi067.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-28T15:49:03.337 INFO:teuthology.orchestra.run.smithi067.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-28T15:49:03.337 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T15:49:03.337 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-03-28T15:49:04.179 DEBUG:teuthology.parallel:result is None 2024-03-28T15:49:04.180 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T15:49:04.323 DEBUG:teuthology.orchestra.run.smithi067:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-28T15:49:04.379 INFO:teuthology.orchestra.run.smithi067.stdout:19.0.0-2145.g3be9a5c9.el9 2024-03-28T15:49:04.380 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2145.g3be9a5c9.el9 2024-03-28T15:49:04.380 INFO:teuthology.task.install:The correct ceph version 19.0.0-2145.g3be9a5c9 is installed. 2024-03-28T15:49:04.382 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T15:49:04.520 DEBUG:teuthology.orchestra.run.smithi150:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-28T15:49:04.592 INFO:teuthology.orchestra.run.smithi150.stdout:19.0.0-2145.g3be9a5c9.el9 2024-03-28T15:49:04.593 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2145.g3be9a5c9.el9 2024-03-28T15:49:04.593 INFO:teuthology.task.install:The correct ceph version 19.0.0-2145.g3be9a5c9 is installed. 2024-03-28T15:49:04.595 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T15:49:04.948 DEBUG:teuthology.orchestra.run.smithi183:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-28T15:49:05.021 INFO:teuthology.orchestra.run.smithi183.stdout:19.0.0-2145.g3be9a5c9.el9 2024-03-28T15:49:05.021 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2145.g3be9a5c9.el9 2024-03-28T15:49:05.021 INFO:teuthology.task.install:The correct ceph version 19.0.0-2145.g3be9a5c9 is installed. 2024-03-28T15:49:05.023 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-28T15:49:05.023 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-28T15:49:05.023 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-28T15:49:05.108 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-28T15:49:05.108 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-28T15:49:05.211 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-28T15:49:05.211 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-28T15:49:05.313 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-28T15:49:05.314 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-28T15:49:05.314 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/daemon-helper 2024-03-28T15:49:05.410 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-28T15:49:05.487 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-28T15:49:05.487 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/usr/bin/daemon-helper 2024-03-28T15:49:05.583 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-28T15:49:05.714 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-28T15:49:05.714 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/usr/bin/daemon-helper 2024-03-28T15:49:05.811 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-28T15:49:05.949 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-28T15:49:05.949 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-28T15:49:05.949 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-28T15:49:06.027 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-28T15:49:06.140 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-28T15:49:06.140 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-28T15:49:06.236 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-28T15:49:06.376 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-28T15:49:06.376 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-28T15:49:06.477 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-28T15:49:06.609 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-28T15:49:06.609 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-28T15:49:06.609 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/stdin-killer 2024-03-28T15:49:06.690 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-28T15:49:06.804 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-28T15:49:06.805 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/usr/bin/stdin-killer 2024-03-28T15:49:06.904 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-28T15:49:06.998 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-28T15:49:06.998 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/usr/bin/stdin-killer 2024-03-28T15:49:07.097 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-28T15:49:07.192 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-28T15:49:07.292 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 1, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-03-28T15:49:07.293 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T15:49:07.293 INFO:tasks.cephadm:Cluster fsid is b5f8225a-ed1a-11ee-95d0-87774f69a715 2024-03-28T15:49:07.293 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-28T15:49:07.293 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.67', 'mon.b': '172.21.15.150', 'mon.c': '172.21.15.183'} 2024-03-28T15:49:07.293 INFO:tasks.cephadm:First mon is mon.a on smithi067 2024-03-28T15:49:07.294 INFO:tasks.cephadm:First mgr is x 2024-03-28T15:49:07.294 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-28T15:49:07.294 DEBUG:teuthology.orchestra.run.smithi067:> sudo hostname $(hostname -s) 2024-03-28T15:49:07.371 DEBUG:teuthology.orchestra.run.smithi150:> sudo hostname $(hostname -s) 2024-03-28T15:49:07.473 DEBUG:teuthology.orchestra.run.smithi183:> sudo hostname $(hostname -s) 2024-03-28T15:49:07.570 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-28T15:49:07.570 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T15:49:07.707 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77877/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2145-g3be9a5c9', 'node_name': '172.21.5.36+adami06', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2145.g3be9a5c9'}, 'url': 'https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-22 05:51:42.267384', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-mchangir-testing-main-20240318.032620', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-03-28T15:49:07.843 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref wip-mchangir-testing-main-20240318.032620, sha1 3be9a5c9bc793e11e2800a8c0c696e8b46742033 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T15:49:07.844 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/x86_64/flavors/default/cephadm 2024-03-28T15:49:07.844 DEBUG:teuthology.orchestra.run.smithi067:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-28T15:49:09.440 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r--. 1 ubuntu ubuntu 767371 Mar 28 15:49 /home/ubuntu/cephtest/cephadm 2024-03-28T15:49:09.440 DEBUG:teuthology.orchestra.run.smithi150:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-28T15:49:10.720 INFO:teuthology.orchestra.run.smithi150.stdout:-rw-r--r--. 1 ubuntu ubuntu 767371 Mar 28 15:49 /home/ubuntu/cephtest/cephadm 2024-03-28T15:49:10.720 DEBUG:teuthology.orchestra.run.smithi183:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-28T15:49:11.623 INFO:teuthology.orchestra.run.smithi183.stdout:-rw-r--r--. 1 ubuntu ubuntu 767371 Mar 28 15:49 /home/ubuntu/cephtest/cephadm 2024-03-28T15:49:11.624 DEBUG:teuthology.orchestra.run.smithi067:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-28T15:49:11.686 DEBUG:teuthology.orchestra.run.smithi150:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-28T15:49:11.745 DEBUG:teuthology.orchestra.run.smithi183:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-28T15:49:11.818 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 on all hosts... 2024-03-28T15:49:11.818 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 pull 2024-03-28T15:49:11.825 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 pull 2024-03-28T15:49:11.832 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 pull 2024-03-28T15:49:12.940 INFO:teuthology.orchestra.run.smithi150.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033... 2024-03-28T15:49:13.069 INFO:teuthology.orchestra.run.smithi067.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033... 2024-03-28T15:49:13.080 INFO:teuthology.orchestra.run.smithi183.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033... 2024-03-28T15:49:46.649 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-03-28T15:49:46.650 INFO:teuthology.orchestra.run.smithi150.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-28T15:49:46.650 INFO:teuthology.orchestra.run.smithi150.stdout: "image_id": "c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2", 2024-03-28T15:49:46.650 INFO:teuthology.orchestra.run.smithi150.stdout: "repo_digests": [ 2024-03-28T15:49:46.650 INFO:teuthology.orchestra.run.smithi150.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc" 2024-03-28T15:49:46.650 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-03-28T15:49:46.650 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-03-28T15:50:05.509 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-28T15:50:05.510 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-28T15:50:05.510 INFO:teuthology.orchestra.run.smithi067.stdout: "image_id": "c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2", 2024-03-28T15:50:05.510 INFO:teuthology.orchestra.run.smithi067.stdout: "repo_digests": [ 2024-03-28T15:50:05.510 INFO:teuthology.orchestra.run.smithi067.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc" 2024-03-28T15:50:05.510 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-28T15:50:05.510 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-28T15:50:08.821 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-03-28T15:50:08.821 INFO:teuthology.orchestra.run.smithi183.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-28T15:50:08.821 INFO:teuthology.orchestra.run.smithi183.stdout: "image_id": "c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2", 2024-03-28T15:50:08.822 INFO:teuthology.orchestra.run.smithi183.stdout: "repo_digests": [ 2024-03-28T15:50:08.822 INFO:teuthology.orchestra.run.smithi183.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc" 2024-03-28T15:50:08.822 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-03-28T15:50:08.822 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-03-28T15:50:08.887 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /etc/ceph 2024-03-28T15:50:08.999 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /etc/ceph 2024-03-28T15:50:09.107 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /etc/ceph 2024-03-28T15:50:09.196 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 777 /etc/ceph 2024-03-28T15:50:09.298 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 777 /etc/ceph 2024-03-28T15:50:09.402 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 777 /etc/ceph 2024-03-28T15:50:09.505 INFO:tasks.cephadm:Writing seed config... 2024-03-28T15:50:09.506 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-03-28T15:50:09.506 INFO:tasks.cephadm: override: [client] debug client = 20 2024-03-28T15:50:09.506 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-28T15:50:09.507 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-03-28T15:50:09.507 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-03-28T15:50:09.507 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-03-28T15:50:09.507 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-03-28T15:50:09.507 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-03-28T15:50:09.507 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-03-28T15:50:09.507 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-03-28T15:50:09.507 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-03-28T15:50:09.507 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-03-28T15:50:09.507 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-03-28T15:50:09.507 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-03-28T15:50:09.507 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-03-28T15:50:09.507 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-03-28T15:50:09.507 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-03-28T15:50:09.508 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-03-28T15:50:09.508 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-03-28T15:50:09.508 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-03-28T15:50:09.508 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-28T15:50:09.508 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-28T15:50:09.508 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-28T15:50:09.508 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-28T15:50:09.508 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-28T15:50:09.508 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-03-28T15:50:09.508 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-03-28T15:50:09.508 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-03-28T15:50:09.508 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-03-28T15:50:09.508 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-03-28T15:50:09.508 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-03-28T15:50:09.509 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-03-28T15:50:09.509 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-03-28T15:50:09.509 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-28T15:50:09.509 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-28T15:50:09.509 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-03-28T15:50:09.509 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-03-28T15:50:09.509 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-03-28T15:50:09.509 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-03-28T15:50:09.509 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-03-28T15:50:09.509 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-03-28T15:50:09.509 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-03-28T15:50:09.509 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-28T15:50:09.509 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-03-28T15:50:09.510 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-28T15:50:09.510 DEBUG:teuthology.orchestra.run.smithi067:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-28T15:50:09.570 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = b5f8225a-ed1a-11ee-95d0-87774f69a715 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-03-28T15:50:09.571 DEBUG:teuthology.orchestra.run.smithi067:mon.a> sudo journalctl -f -n 0 -u ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@mon.a.service 2024-03-28T15:50:09.618 DEBUG:teuthology.orchestra.run.smithi067:mgr.x> sudo journalctl -f -n 0 -u ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@mgr.x.service 2024-03-28T15:50:09.664 INFO:tasks.cephadm:Bootstrapping... 2024-03-28T15:50:09.664 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 -v bootstrap --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.67 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-28T15:50:09.948 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-03-28T15:50:09.948 INFO:teuthology.orchestra.run.smithi067.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033', '-v', 'bootstrap', '--fsid', 'b5f8225a-ed1a-11ee-95d0-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.67', '--skip-admin-label'] 2024-03-28T15:50:10.003 INFO:teuthology.orchestra.run.smithi067.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-28T15:50:10.003 INFO:teuthology.orchestra.run.smithi067.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-28T15:50:10.003 INFO:teuthology.orchestra.run.smithi067.stdout:Verifying podman|docker is present... 2024-03-28T15:50:10.055 INFO:teuthology.orchestra.run.smithi067.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-28T15:50:10.055 INFO:teuthology.orchestra.run.smithi067.stdout:Verifying lvm2 is present... 2024-03-28T15:50:10.055 INFO:teuthology.orchestra.run.smithi067.stdout:Verifying time synchronization is in place... 2024-03-28T15:50:10.070 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-28T15:50:10.070 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-28T15:50:10.086 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-28T15:50:10.087 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stdout inactive 2024-03-28T15:50:10.104 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stdout enabled 2024-03-28T15:50:10.124 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stdout active 2024-03-28T15:50:10.124 INFO:teuthology.orchestra.run.smithi067.stdout:Unit chronyd.service is enabled and running 2024-03-28T15:50:10.124 INFO:teuthology.orchestra.run.smithi067.stdout:Repeating the final host check... 2024-03-28T15:50:10.179 INFO:teuthology.orchestra.run.smithi067.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-28T15:50:10.179 INFO:teuthology.orchestra.run.smithi067.stdout:podman (/bin/podman) version 4.9.4 is present 2024-03-28T15:50:10.179 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl is present 2024-03-28T15:50:10.179 INFO:teuthology.orchestra.run.smithi067.stdout:lvcreate is present 2024-03-28T15:50:10.196 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-28T15:50:10.196 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-28T15:50:10.213 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-28T15:50:10.214 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stdout inactive 2024-03-28T15:50:10.237 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stdout enabled 2024-03-28T15:50:10.257 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stdout active 2024-03-28T15:50:10.257 INFO:teuthology.orchestra.run.smithi067.stdout:Unit chronyd.service is enabled and running 2024-03-28T15:50:10.257 INFO:teuthology.orchestra.run.smithi067.stdout:Host looks OK 2024-03-28T15:50:10.257 INFO:teuthology.orchestra.run.smithi067.stdout:Cluster fsid: b5f8225a-ed1a-11ee-95d0-87774f69a715 2024-03-28T15:50:10.258 INFO:teuthology.orchestra.run.smithi067.stdout:Acquiring lock 140174380260800 on /run/cephadm/b5f8225a-ed1a-11ee-95d0-87774f69a715.lock 2024-03-28T15:50:10.259 INFO:teuthology.orchestra.run.smithi067.stdout:Lock 140174380260800 acquired on /run/cephadm/b5f8225a-ed1a-11ee-95d0-87774f69a715.lock 2024-03-28T15:50:10.260 INFO:teuthology.orchestra.run.smithi067.stdout:Verifying IP 172.21.15.67 port 3300 ... 2024-03-28T15:50:10.261 INFO:teuthology.orchestra.run.smithi067.stdout:Verifying IP 172.21.15.67 port 6789 ... 2024-03-28T15:50:10.261 INFO:teuthology.orchestra.run.smithi067.stdout:Base mon IP(s) is [172.21.15.67:3300, 172.21.15.67:6789], mon addrv is [v2:172.21.15.67:3300,v1:172.21.15.67:6789] 2024-03-28T15:50:10.275 INFO:teuthology.orchestra.run.smithi067.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.67 metric 100 2024-03-28T15:50:10.276 INFO:teuthology.orchestra.run.smithi067.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.67 metric 100 2024-03-28T15:50:10.288 INFO:teuthology.orchestra.run.smithi067.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-28T15:50:10.289 INFO:teuthology.orchestra.run.smithi067.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-03-28T15:50:10.289 INFO:teuthology.orchestra.run.smithi067.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1786sec hoplimit 64 pref medium 2024-03-28T15:50:10.299 INFO:teuthology.orchestra.run.smithi067.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-28T15:50:10.300 INFO:teuthology.orchestra.run.smithi067.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-28T15:50:10.300 INFO:teuthology.orchestra.run.smithi067.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-28T15:50:10.300 INFO:teuthology.orchestra.run.smithi067.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-03-28T15:50:10.300 INFO:teuthology.orchestra.run.smithi067.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:feb6:fc0b/64 scope link 2024-03-28T15:50:10.301 INFO:teuthology.orchestra.run.smithi067.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-28T15:50:10.303 INFO:teuthology.orchestra.run.smithi067.stdout:Mon IP `172.21.15.67` is in CIDR network `172.21.0.0/20` 2024-03-28T15:50:10.303 INFO:teuthology.orchestra.run.smithi067.stdout:Mon IP `172.21.15.67` is in CIDR network `172.21.0.0/20` 2024-03-28T15:50:10.304 INFO:teuthology.orchestra.run.smithi067.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-28T15:50:10.307 INFO:teuthology.orchestra.run.smithi067.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-28T15:50:10.309 INFO:teuthology.orchestra.run.smithi067.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033... 2024-03-28T15:50:10.992 INFO:teuthology.orchestra.run.smithi067.stdout:/bin/podman: stdout c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 2024-03-28T15:50:10.993 INFO:teuthology.orchestra.run.smithi067.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033... 2024-03-28T15:50:10.993 INFO:teuthology.orchestra.run.smithi067.stdout:/bin/podman: stderr Getting image source signatures 2024-03-28T15:50:10.993 INFO:teuthology.orchestra.run.smithi067.stdout:/bin/podman: stderr Copying blob sha256:4dd83c6ee8921fe17396d77c31e205985ae94c87d3f752465b5f04aaf9ae340a 2024-03-28T15:50:10.993 INFO:teuthology.orchestra.run.smithi067.stdout:/bin/podman: stderr Copying blob sha256:9516588f908cdb14b5e4062a6063db2d9291b355c3f32e2a99864608a4a7b649 2024-03-28T15:50:10.993 INFO:teuthology.orchestra.run.smithi067.stdout:/bin/podman: stderr Copying config sha256:c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 2024-03-28T15:50:10.994 INFO:teuthology.orchestra.run.smithi067.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-28T15:50:12.208 INFO:teuthology.orchestra.run.smithi067.stdout:ceph: stdout ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T15:50:12.208 INFO:teuthology.orchestra.run.smithi067.stdout:Ceph version: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T15:50:12.208 INFO:teuthology.orchestra.run.smithi067.stdout:Extracting ceph user uid/gid from container image... 2024-03-28T15:50:13.258 INFO:teuthology.orchestra.run.smithi067.stdout:stat: stdout 167 167 2024-03-28T15:50:13.258 INFO:teuthology.orchestra.run.smithi067.stdout:Creating initial keys... 2024-03-28T15:50:14.266 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph-authtool: stdout AQC1kQVmCWOGKhAA1e6x7ym5l5qN3pkbIXCL6g== 2024-03-28T15:50:15.484 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph-authtool: stdout AQC2kQVmXU28KhAAp9UMVHl3/eSbX+939gn36Q== 2024-03-28T15:50:16.598 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph-authtool: stdout AQC3kQVmpCHnOBAAB3PcfvFWOK6BrXziHe+6pQ== 2024-03-28T15:50:16.604 INFO:teuthology.orchestra.run.smithi067.stdout:Creating initial monmap... 2024-03-28T15:50:17.739 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-28T15:50:17.739 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-28T15:50:17.739 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to b5f8225a-ed1a-11ee-95d0-87774f69a715 2024-03-28T15:50:17.739 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-28T15:50:17.739 INFO:teuthology.orchestra.run.smithi067.stdout:monmaptool for a [v2:172.21.15.67:3300,v1:172.21.15.67:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-28T15:50:17.740 INFO:teuthology.orchestra.run.smithi067.stdout:setting min_mon_release = quincy 2024-03-28T15:50:17.740 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/monmaptool: set fsid to b5f8225a-ed1a-11ee-95d0-87774f69a715 2024-03-28T15:50:17.740 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-28T15:50:17.740 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T15:50:17.740 INFO:teuthology.orchestra.run.smithi067.stdout:Creating mon... 2024-03-28T15:50:18.967 INFO:teuthology.orchestra.run.smithi067.stdout:create mon.a on 2024-03-28T15:50:19.330 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-28T15:50:19.642 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-28T15:50:19.943 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715.target → /etc/systemd/system/ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715.target. 2024-03-28T15:50:19.943 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715.target → /etc/systemd/system/ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715.target. 2024-03-28T15:50:20.487 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@mon.a 2024-03-28T15:50:20.487 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stderr Failed to reset failed state of unit ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@mon.a.service: Unit ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@mon.a.service not loaded. 2024-03-28T15:50:20.785 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715.target.wants/ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@mon.a.service → /etc/systemd/system/ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@.service. 2024-03-28T15:50:21.470 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:21 smithi067 podman[31323]: 2024-03-28 15:50:21.393797408 +0000 UTC m=+0.347063657 container start e7b62a90ffe2df4503acf0dc87622504d660fbb8ec9200d0f8693234b1aec06b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mon-a, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-28T15:50:21.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:21 smithi067 bash[31323]: e7b62a90ffe2df4503acf0dc87622504d660fbb8ec9200d0f8693234b1aec06b 2024-03-28T15:50:21.500 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-28T15:50:21.500 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stdout disabled 2024-03-28T15:50:21.516 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-28T15:50:21.516 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stdout inactive 2024-03-28T15:50:21.516 INFO:teuthology.orchestra.run.smithi067.stdout:firewalld.service is not enabled 2024-03-28T15:50:21.516 INFO:teuthology.orchestra.run.smithi067.stdout:Not possible to enable service . firewalld.service is not available 2024-03-28T15:50:21.517 INFO:teuthology.orchestra.run.smithi067.stdout:Waiting for mon to start... 2024-03-28T15:50:21.517 INFO:teuthology.orchestra.run.smithi067.stdout:Waiting for mon... 2024-03-28T15:50:21.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:21 smithi067 ceph-mon[31342]: mkfs b5f8225a-ed1a-11ee-95d0-87774f69a715 2024-03-28T15:50:21.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:21 smithi067 systemd[1]: Started Ceph mon.a for b5f8225a-ed1a-11ee-95d0-87774f69a715. 2024-03-28T15:50:21.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:21 smithi067 ceph-mon[31342]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-28T15:50:22.741 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:22 smithi067 ceph-mon[31342]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-28T15:50:22.742 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:22 smithi067 ceph-mon[31342]: monmap e1: 1 mons at {a=[v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T15:50:22.742 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:22 smithi067 ceph-mon[31342]: fsmap 2024-03-28T15:50:22.742 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:22 smithi067 ceph-mon[31342]: osdmap e1: 0 total, 0 up, 0 in 2024-03-28T15:50:22.742 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:22 smithi067 ceph-mon[31342]: mgrmap e1: no daemons active 2024-03-28T15:50:22.742 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:22 smithi067 ceph-mon[31342]: from='client.? 172.21.15.67:0/1466628235' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-28T15:50:23.059 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout cluster: 2024-03-28T15:50:23.065 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout id: b5f8225a-ed1a-11ee-95d0-87774f69a715 2024-03-28T15:50:23.065 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-28T15:50:23.065 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 2024-03-28T15:50:23.065 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout services: 2024-03-28T15:50:23.065 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.758978s) 2024-03-28T15:50:23.065 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-28T15:50:23.065 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-28T15:50:23.066 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 2024-03-28T15:50:23.066 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout data: 2024-03-28T15:50:23.066 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-28T15:50:23.066 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-28T15:50:23.066 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-28T15:50:23.066 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout pgs: 2024-03-28T15:50:23.066 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 2024-03-28T15:50:23.066 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:22.135+0000 7f6b43000700 1 Processor -- start 2024-03-28T15:50:23.066 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:22.135+0000 7f6b43000700 1 -- start start 2024-03-28T15:50:23.066 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:22.136+0000 7f6b43000700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b3c109b80 0x7f6b3c109fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:23.066 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:22.136+0000 7f6b43000700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b3c10a570 con 0x7f6b3c109b80 2024-03-28T15:50:23.066 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:22.137+0000 7f6b3be00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b3c109b80 0x7f6b3c109fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:23.066 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:22.137+0000 7f6b3be00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b3c109b80 0x7f6b3c109fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56360/0 (socket says 172.21.15.67:56360) 2024-03-28T15:50:23.067 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:22.137+0000 7f6b3be00700 1 -- 172.21.15.67:0/4117955248 learned_addr learned my addr 172.21.15.67:0/4117955248 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:50:23.067 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:22.137+0000 7f6b3be00700 1 -- 172.21.15.67:0/4117955248 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b3c10ad80 con 0x7f6b3c109b80 2024-03-28T15:50:23.067 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:22.137+0000 7f6b3be00700 1 --2- 172.21.15.67:0/4117955248 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b3c109b80 0x7f6b3c109fa0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c0098a0 tx=0x7f6b2c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92115a821a5710f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:23.067 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:22.138+0000 7f6b3b400700 1 -- 172.21.15.67:0/4117955248 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6b2c004160 con 0x7f6b3c109b80 2024-03-28T15:50:23.067 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:22.138+0000 7f6b3b400700 1 -- 172.21.15.67:0/4117955248 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b2c004300 con 0x7f6b3c109b80 2024-03-28T15:50:23.067 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:22.138+0000 7f6b3b400700 1 -- 172.21.15.67:0/4117955248 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6b2c0036a0 con 0x7f6b3c109b80 2024-03-28T15:50:23.067 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:22.138+0000 7f6b43000700 1 -- 172.21.15.67:0/4117955248 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b3c109b80 msgr2=0x7f6b3c109fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:23.067 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:22.138+0000 7f6b43000700 1 --2- 172.21.15.67:0/4117955248 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b3c109b80 0x7f6b3c109fa0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c0098a0 tx=0x7f6b2c009bb0 comp rx=0 tx=0).stop 2024-03-28T15:50:23.067 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:22.138+0000 7f6b43000700 1 -- 172.21.15.67:0/4117955248 shutdown_connections 2024-03-28T15:50:23.067 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:22.138+0000 7f6b43000700 1 --2- 172.21.15.67:0/4117955248 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b3c109b80 0x7f6b3c109fa0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:23.067 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:22.138+0000 7f6b43000700 1 -- 172.21.15.67:0/4117955248 >> 172.21.15.67:0/4117955248 conn(0x7f6b3c07cfa0 msgr2=0x7f6b3c07d3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:23.067 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:22.139+0000 7f6b43000700 1 -- 172.21.15.67:0/4117955248 shutdown_connections 2024-03-28T15:50:23.067 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:22.139+0000 7f6b43000700 1 -- 172.21.15.67:0/4117955248 wait complete. 2024-03-28T15:50:23.068 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:22.140+0000 7f6b43000700 1 Processor -- start 2024-03-28T15:50:23.068 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:22.140+0000 7f6b43000700 1 -- start start 2024-03-28T15:50:23.068 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:22.140+0000 7f6b43000700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b3c109b80 0x7f6b3c19d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:23.068 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:22.140+0000 7f6b43000700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b3c10b2b0 con 0x7f6b3c109b80 2024-03-28T15:50:23.068 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:22.141+0000 7f6b3be00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b3c109b80 0x7f6b3c19d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:23.068 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:22.141+0000 7f6b3be00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b3c109b80 0x7f6b3c19d280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56376/0 (socket says 172.21.15.67:56376) 2024-03-28T15:50:23.068 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:22.141+0000 7f6b3be00700 1 -- 172.21.15.67:0/1466628235 learned_addr learned my addr 172.21.15.67:0/1466628235 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:50:23.068 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:22.141+0000 7f6b3be00700 1 -- 172.21.15.67:0/1466628235 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b3c19d7c0 con 0x7f6b3c109b80 2024-03-28T15:50:23.068 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:22.141+0000 7f6b3be00700 1 --2- 172.21.15.67:0/1466628235 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b3c109b80 0x7f6b3c19d280 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c000c00 tx=0x7f6b2c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:23.068 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:22.142+0000 7f6b39600700 1 -- 172.21.15.67:0/1466628235 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6b2c003db0 con 0x7f6b3c109b80 2024-03-28T15:50:23.068 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:22.142+0000 7f6b39600700 1 -- 172.21.15.67:0/1466628235 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b2c011630 con 0x7f6b3c109b80 2024-03-28T15:50:23.068 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:22.142+0000 7f6b43000700 1 -- 172.21.15.67:0/1466628235 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b3c19da50 con 0x7f6b3c109b80 2024-03-28T15:50:23.068 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:22.142+0000 7f6b39600700 1 -- 172.21.15.67:0/1466628235 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6b2c011950 con 0x7f6b3c109b80 2024-03-28T15:50:23.068 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:22.142+0000 7f6b43000700 1 -- 172.21.15.67:0/1466628235 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b3c1a0740 con 0x7f6b3c109b80 2024-03-28T15:50:23.069 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:22.142+0000 7f6b39600700 1 -- 172.21.15.67:0/1466628235 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f6b2c011af0 con 0x7f6b3c109b80 2024-03-28T15:50:23.069 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:22.142+0000 7f6b39600700 1 -- 172.21.15.67:0/1466628235 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f6b2c01c450 con 0x7f6b3c109b80 2024-03-28T15:50:23.069 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:22.143+0000 7f6b43000700 1 -- 172.21.15.67:0/1466628235 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b3c0637c0 con 0x7f6b3c109b80 2024-03-28T15:50:23.069 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:22.146+0000 7f6b39600700 1 -- 172.21.15.67:0/1466628235 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f6b2c032020 con 0x7f6b3c109b80 2024-03-28T15:50:23.069 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:22.228+0000 7f6b43000700 1 -- 172.21.15.67:0/1466628235 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f6b3c02dbb0 con 0x7f6b3c109b80 2024-03-28T15:50:23.069 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:22.229+0000 7f6b39600700 1 -- 172.21.15.67:0/1466628235 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (secure 0 0 0) 0x7f6b2c004260 con 0x7f6b3c109b80 2024-03-28T15:50:23.069 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:22.230+0000 7f6b43000700 1 -- 172.21.15.67:0/1466628235 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b3c109b80 msgr2=0x7f6b3c19d280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:23.069 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:22.230+0000 7f6b43000700 1 --2- 172.21.15.67:0/1466628235 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b3c109b80 0x7f6b3c19d280 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c000c00 tx=0x7f6b2c004750 comp rx=0 tx=0).stop 2024-03-28T15:50:23.069 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:22.230+0000 7f6b43000700 1 -- 172.21.15.67:0/1466628235 shutdown_connections 2024-03-28T15:50:23.069 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:22.230+0000 7f6b43000700 1 --2- 172.21.15.67:0/1466628235 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b3c109b80 0x7f6b3c19d280 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:23.069 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:22.230+0000 7f6b43000700 1 -- 172.21.15.67:0/1466628235 >> 172.21.15.67:0/1466628235 conn(0x7f6b3c07cfa0 msgr2=0x7f6b3c193ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:23.069 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:22.231+0000 7f6b43000700 1 -- 172.21.15.67:0/1466628235 shutdown_connections 2024-03-28T15:50:23.069 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:22.231+0000 7f6b43000700 1 -- 172.21.15.67:0/1466628235 wait complete. 2024-03-28T15:50:23.069 INFO:teuthology.orchestra.run.smithi067.stdout:mon is available 2024-03-28T15:50:23.070 INFO:teuthology.orchestra.run.smithi067.stdout:Assimilating anything we can from ceph.conf... 2024-03-28T15:50:24.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:23 smithi067 ceph-mon[31342]: from='client.? 172.21.15.67:0/1004229560' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-28T15:50:24.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:23 smithi067 ceph-mon[31342]: from='client.? 172.21.15.67:0/1004229560' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-28T15:50:24.488 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 2024-03-28T15:50:24.488 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout [global] 2024-03-28T15:50:24.488 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout fsid = b5f8225a-ed1a-11ee-95d0-87774f69a715 2024-03-28T15:50:24.489 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.67:3300,v1:172.21.15.67:6789] 2024-03-28T15:50:24.489 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-28T15:50:24.489 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-28T15:50:24.489 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-28T15:50:24.489 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-28T15:50:24.489 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 2024-03-28T15:50:24.489 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-28T15:50:24.489 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-28T15:50:24.489 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 2024-03-28T15:50:24.489 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout [osd] 2024-03-28T15:50:24.489 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout bdev_async_discard = True 2024-03-28T15:50:24.489 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-28T15:50:24.489 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-28T15:50:24.489 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-28T15:50:24.490 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-28T15:50:24.490 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-28T15:50:24.490 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:23.748+0000 7f7c6cc00700 1 Processor -- start 2024-03-28T15:50:24.490 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:23.748+0000 7f7c6cc00700 1 -- start start 2024-03-28T15:50:24.490 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:23.749+0000 7f7c6cc00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c68107970 0x7f7c68107d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:24.490 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:23.749+0000 7f7c6cc00700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c68108360 con 0x7f7c68107970 2024-03-28T15:50:24.490 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:23.749+0000 7f7c65e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c68107970 0x7f7c68107d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:24.490 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:23.749+0000 7f7c65e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c68107970 0x7f7c68107d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56386/0 (socket says 172.21.15.67:56386) 2024-03-28T15:50:24.490 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:23.749+0000 7f7c65e00700 1 -- 172.21.15.67:0/3566000630 learned_addr learned my addr 172.21.15.67:0/3566000630 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:50:24.490 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:23.750+0000 7f7c65e00700 1 -- 172.21.15.67:0/3566000630 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c68108b70 con 0x7f7c68107970 2024-03-28T15:50:24.490 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:23.750+0000 7f7c65e00700 1 --2- 172.21.15.67:0/3566000630 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c68107970 0x7f7c68107d90 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f7c580098a0 tx=0x7f7c58009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56877b23d72e63c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:24.490 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:23.750+0000 7f7c65400700 1 -- 172.21.15.67:0/3566000630 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7c58004160 con 0x7f7c68107970 2024-03-28T15:50:24.490 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:23.750+0000 7f7c65400700 1 -- 172.21.15.67:0/3566000630 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c58004300 con 0x7f7c68107970 2024-03-28T15:50:24.490 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:23.750+0000 7f7c65400700 1 -- 172.21.15.67:0/3566000630 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7c580036a0 con 0x7f7c68107970 2024-03-28T15:50:24.491 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:23.751+0000 7f7c6cc00700 1 -- 172.21.15.67:0/3566000630 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c68107970 msgr2=0x7f7c68107d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:24.491 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:23.751+0000 7f7c6cc00700 1 --2- 172.21.15.67:0/3566000630 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c68107970 0x7f7c68107d90 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f7c580098a0 tx=0x7f7c58009bb0 comp rx=0 tx=0).stop 2024-03-28T15:50:24.491 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:23.751+0000 7f7c6cc00700 1 -- 172.21.15.67:0/3566000630 shutdown_connections 2024-03-28T15:50:24.491 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:23.751+0000 7f7c6cc00700 1 --2- 172.21.15.67:0/3566000630 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c68107970 0x7f7c68107d90 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:24.491 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:23.751+0000 7f7c6cc00700 1 -- 172.21.15.67:0/3566000630 >> 172.21.15.67:0/3566000630 conn(0x7f7c68102f30 msgr2=0x7f7c68105350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:24.491 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:23.751+0000 7f7c6cc00700 1 -- 172.21.15.67:0/3566000630 shutdown_connections 2024-03-28T15:50:24.491 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:23.751+0000 7f7c6cc00700 1 -- 172.21.15.67:0/3566000630 wait complete. 2024-03-28T15:50:24.491 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:23.753+0000 7f7c6cc00700 1 Processor -- start 2024-03-28T15:50:24.491 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:23.753+0000 7f7c6cc00700 1 -- start start 2024-03-28T15:50:24.491 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:23.753+0000 7f7c6cc00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c68107970 0x7f7c6819d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:24.491 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:23.753+0000 7f7c6cc00700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c681090a0 con 0x7f7c68107970 2024-03-28T15:50:24.491 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:23.754+0000 7f7c65e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c68107970 0x7f7c6819d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:24.491 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:23.754+0000 7f7c65e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c68107970 0x7f7c6819d2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56398/0 (socket says 172.21.15.67:56398) 2024-03-28T15:50:24.492 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:23.754+0000 7f7c65e00700 1 -- 172.21.15.67:0/1004229560 learned_addr learned my addr 172.21.15.67:0/1004229560 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:50:24.492 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:23.754+0000 7f7c65e00700 1 -- 172.21.15.67:0/1004229560 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c6819d820 con 0x7f7c68107970 2024-03-28T15:50:24.492 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:23.754+0000 7f7c65e00700 1 --2- 172.21.15.67:0/1004229560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c68107970 0x7f7c6819d2e0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f7c58000c00 tx=0x7f7c58004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:24.492 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:23.755+0000 7f7c5ea00700 1 -- 172.21.15.67:0/1004229560 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7c58003db0 con 0x7f7c68107970 2024-03-28T15:50:24.492 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:23.755+0000 7f7c5ea00700 1 -- 172.21.15.67:0/1004229560 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c58011630 con 0x7f7c68107970 2024-03-28T15:50:24.492 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:23.755+0000 7f7c5ea00700 1 -- 172.21.15.67:0/1004229560 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7c58011950 con 0x7f7c68107970 2024-03-28T15:50:24.492 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:23.755+0000 7f7c6cc00700 1 -- 172.21.15.67:0/1004229560 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c6819dab0 con 0x7f7c68107970 2024-03-28T15:50:24.492 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:23.755+0000 7f7c6cc00700 1 -- 172.21.15.67:0/1004229560 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c681a07a0 con 0x7f7c68107970 2024-03-28T15:50:24.492 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:23.755+0000 7f7c5ea00700 1 -- 172.21.15.67:0/1004229560 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f7c58011af0 con 0x7f7c68107970 2024-03-28T15:50:24.492 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:23.755+0000 7f7c5ea00700 1 -- 172.21.15.67:0/1004229560 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f7c5801c450 con 0x7f7c68107970 2024-03-28T15:50:24.492 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:23.756+0000 7f7c6cc00700 1 -- 172.21.15.67:0/1004229560 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c680637c0 con 0x7f7c68107970 2024-03-28T15:50:24.492 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:23.759+0000 7f7c5ea00700 1 -- 172.21.15.67:0/1004229560 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f7c58032020 con 0x7f7c68107970 2024-03-28T15:50:24.492 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:23.838+0000 7f7c6cc00700 1 -- 172.21.15.67:0/1004229560 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f7c681a0970 con 0x7f7c68107970 2024-03-28T15:50:24.492 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:23.847+0000 7f7c5ea00700 1 -- 172.21.15.67:0/1004229560 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+456 (secure 0 0 0) 0x7f7c58004260 con 0x7f7c68107970 2024-03-28T15:50:24.492 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:23.848+0000 7f7c5ea00700 1 -- 172.21.15.67:0/1004229560 <== mon.0 v2:172.21.15.67:3300/0 8 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7f7c5801c650 con 0x7f7c68107970 2024-03-28T15:50:24.493 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:23.850+0000 7f7c6cc00700 1 -- 172.21.15.67:0/1004229560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c68107970 msgr2=0x7f7c6819d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:24.493 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:23.850+0000 7f7c6cc00700 1 --2- 172.21.15.67:0/1004229560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c68107970 0x7f7c6819d2e0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f7c58000c00 tx=0x7f7c58004750 comp rx=0 tx=0).stop 2024-03-28T15:50:24.493 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:23.850+0000 7f7c6cc00700 1 -- 172.21.15.67:0/1004229560 shutdown_connections 2024-03-28T15:50:24.493 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:23.850+0000 7f7c6cc00700 1 --2- 172.21.15.67:0/1004229560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c68107970 0x7f7c6819d2e0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:24.493 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:23.850+0000 7f7c6cc00700 1 -- 172.21.15.67:0/1004229560 >> 172.21.15.67:0/1004229560 conn(0x7f7c68102f30 msgr2=0x7f7c6807c080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:24.493 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:23.850+0000 7f7c6cc00700 1 -- 172.21.15.67:0/1004229560 shutdown_connections 2024-03-28T15:50:24.493 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:23.850+0000 7f7c6cc00700 1 -- 172.21.15.67:0/1004229560 wait complete. 2024-03-28T15:50:24.493 INFO:teuthology.orchestra.run.smithi067.stdout:Generating new minimal ceph.conf... 2024-03-28T15:50:25.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:25 smithi067 ceph-mon[31342]: from='client.? 172.21.15.67:0/4215528163' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:50:25.971 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:25.204+0000 7febea600700 1 Processor -- start 2024-03-28T15:50:25.971 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:25.204+0000 7febea600700 1 -- start start 2024-03-28T15:50:25.972 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:25.205+0000 7febea600700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febe4081910 0x7febe4081d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:25.972 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:25.205+0000 7febea600700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febe4082300 con 0x7febe4081910 2024-03-28T15:50:25.972 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:25.205+0000 7febe3400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febe4081910 0x7febe4081d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:25.972 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:25.205+0000 7febe3400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febe4081910 0x7febe4081d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56410/0 (socket says 172.21.15.67:56410) 2024-03-28T15:50:25.972 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:25.205+0000 7febe3400700 1 -- 172.21.15.67:0/206512886 learned_addr learned my addr 172.21.15.67:0/206512886 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:50:25.972 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:25.206+0000 7febe3400700 1 -- 172.21.15.67:0/206512886 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febe410f180 con 0x7febe4081910 2024-03-28T15:50:25.972 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:25.206+0000 7febe3400700 1 --2- 172.21.15.67:0/206512886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febe4081910 0x7febe4081d30 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7febd40098a0 tx=0x7febd4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e54316e21a1e00a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:25.972 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:25.206+0000 7febe2a00700 1 -- 172.21.15.67:0/206512886 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7febd4004160 con 0x7febe4081910 2024-03-28T15:50:25.973 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:25.206+0000 7febe2a00700 1 -- 172.21.15.67:0/206512886 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7febd4004300 con 0x7febe4081910 2024-03-28T15:50:25.973 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:25.206+0000 7febe2a00700 1 -- 172.21.15.67:0/206512886 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7febd4011a80 con 0x7febe4081910 2024-03-28T15:50:25.973 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:25.207+0000 7febea600700 1 -- 172.21.15.67:0/206512886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febe4081910 msgr2=0x7febe4081d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:25.973 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:25.207+0000 7febea600700 1 --2- 172.21.15.67:0/206512886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febe4081910 0x7febe4081d30 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7febd40098a0 tx=0x7febd4009bb0 comp rx=0 tx=0).stop 2024-03-28T15:50:25.973 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:25.207+0000 7febea600700 1 -- 172.21.15.67:0/206512886 shutdown_connections 2024-03-28T15:50:25.973 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:25.207+0000 7febea600700 1 --2- 172.21.15.67:0/206512886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febe4081910 0x7febe4081d30 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:25.974 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:25.207+0000 7febea600700 1 -- 172.21.15.67:0/206512886 >> 172.21.15.67:0/206512886 conn(0x7febe407cfa0 msgr2=0x7febe407f3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:25.974 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:25.207+0000 7febea600700 1 -- 172.21.15.67:0/206512886 shutdown_connections 2024-03-28T15:50:25.974 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:25.207+0000 7febea600700 1 -- 172.21.15.67:0/206512886 wait complete. 2024-03-28T15:50:25.974 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:25.208+0000 7febea600700 1 Processor -- start 2024-03-28T15:50:25.974 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:25.208+0000 7febea600700 1 -- start start 2024-03-28T15:50:25.974 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:25.209+0000 7febea600700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febe4081910 0x7febe41a1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:25.974 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:25.209+0000 7febea600700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febe4112530 con 0x7febe4081910 2024-03-28T15:50:25.974 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:25.209+0000 7febe3400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febe4081910 0x7febe41a1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:25.974 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:25.209+0000 7febe3400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febe4081910 0x7febe41a1ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56426/0 (socket says 172.21.15.67:56426) 2024-03-28T15:50:25.974 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:25.209+0000 7febe3400700 1 -- 172.21.15.67:0/4215528163 learned_addr learned my addr 172.21.15.67:0/4215528163 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:50:25.975 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:25.209+0000 7febe3400700 1 -- 172.21.15.67:0/4215528163 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febe41a2420 con 0x7febe4081910 2024-03-28T15:50:25.975 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:25.210+0000 7febe3400700 1 --2- 172.21.15.67:0/4215528163 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febe4081910 0x7febe41a1ee0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7febd4000c00 tx=0x7febd4011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:25.975 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:25.210+0000 7febe0c00700 1 -- 172.21.15.67:0/4215528163 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7febd40039f0 con 0x7febe4081910 2024-03-28T15:50:25.975 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:25.210+0000 7febe0c00700 1 -- 172.21.15.67:0/4215528163 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7febd4003b90 con 0x7febe4081910 2024-03-28T15:50:25.975 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:25.210+0000 7febea600700 1 -- 172.21.15.67:0/4215528163 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febe41a26b0 con 0x7febe4081910 2024-03-28T15:50:25.975 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:25.210+0000 7febe0c00700 1 -- 172.21.15.67:0/4215528163 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7febd401a740 con 0x7febe4081910 2024-03-28T15:50:25.975 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:25.210+0000 7febea600700 1 -- 172.21.15.67:0/4215528163 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febe41a53a0 con 0x7febe4081910 2024-03-28T15:50:25.976 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:25.210+0000 7febe0c00700 1 -- 172.21.15.67:0/4215528163 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7febd402d3f0 con 0x7febe4081910 2024-03-28T15:50:25.976 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:25.210+0000 7febe0c00700 1 -- 172.21.15.67:0/4215528163 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7febd4019700 con 0x7febe4081910 2024-03-28T15:50:25.976 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:25.211+0000 7febea600700 1 -- 172.21.15.67:0/4215528163 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febe40637c0 con 0x7febe4081910 2024-03-28T15:50:25.976 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:25.214+0000 7febe0c00700 1 -- 172.21.15.67:0/4215528163 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7febd4004300 con 0x7febe4081910 2024-03-28T15:50:25.976 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:25.293+0000 7febea600700 1 -- 172.21.15.67:0/4215528163 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7febe41a5570 con 0x7febe4081910 2024-03-28T15:50:25.976 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:25.294+0000 7febe0c00700 1 -- 172.21.15.67:0/4215528163 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7febd4032030 con 0x7febe4081910 2024-03-28T15:50:25.976 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:25.295+0000 7febea600700 1 -- 172.21.15.67:0/4215528163 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febe4081910 msgr2=0x7febe41a1ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:25.977 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:25.295+0000 7febea600700 1 --2- 172.21.15.67:0/4215528163 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febe4081910 0x7febe41a1ee0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7febd4000c00 tx=0x7febd4011b00 comp rx=0 tx=0).stop 2024-03-28T15:50:25.977 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:25.295+0000 7febea600700 1 -- 172.21.15.67:0/4215528163 shutdown_connections 2024-03-28T15:50:25.977 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:25.295+0000 7febea600700 1 --2- 172.21.15.67:0/4215528163 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febe4081910 0x7febe41a1ee0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:25.977 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:25.295+0000 7febea600700 1 -- 172.21.15.67:0/4215528163 >> 172.21.15.67:0/4215528163 conn(0x7febe407cfa0 msgr2=0x7febe407dba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:25.977 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:25.295+0000 7febea600700 1 -- 172.21.15.67:0/4215528163 shutdown_connections 2024-03-28T15:50:25.978 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:25.295+0000 7febea600700 1 -- 172.21.15.67:0/4215528163 wait complete. 2024-03-28T15:50:25.978 INFO:teuthology.orchestra.run.smithi067.stdout:Restarting the monitor... 2024-03-28T15:50:26.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:26 smithi067 systemd[1]: Stopping Ceph mon.a for b5f8225a-ed1a-11ee-95d0-87774f69a715... 2024-03-28T15:50:26.791 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:26 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mon-a[31337]: 2024-03-28T15:50:26.485+0000 7fcb88600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T15:50:26.791 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:26 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mon-a[31337]: 2024-03-28T15:50:26.485+0000 7fcb88600700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-28T15:50:26.791 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:26 smithi067 podman[31585]: 2024-03-28 15:50:26.538145162 +0000 UTC m=+0.194271584 container died e7b62a90ffe2df4503acf0dc87622504d660fbb8ec9200d0f8693234b1aec06b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mon-a, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240319, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-28T15:50:27.062 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:26 smithi067 podman[31585]: 2024-03-28 15:50:26.894281605 +0000 UTC m=+0.550408062 container cleanup e7b62a90ffe2df4503acf0dc87622504d660fbb8ec9200d0f8693234b1aec06b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mon-a, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T15:50:27.062 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:26 smithi067 bash[31585]: ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mon-a 2024-03-28T15:50:27.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:27 smithi067 podman[31599]: 2024-03-28 15:50:27.061684243 +0000 UTC m=+0.501175331 container remove e7b62a90ffe2df4503acf0dc87622504d660fbb8ec9200d0f8693234b1aec06b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mon-a, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-28T15:50:27.843 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:27 smithi067 systemd[1]: ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-28T15:50:27.843 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:27 smithi067 systemd[1]: Stopped Ceph mon.a for b5f8225a-ed1a-11ee-95d0-87774f69a715. 2024-03-28T15:50:27.843 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:27 smithi067 systemd[1]: ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@mon.a.service: Consumed 1.978s CPU time. 2024-03-28T15:50:27.843 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:27 smithi067 systemd[1]: Starting Ceph mon.a for b5f8225a-ed1a-11ee-95d0-87774f69a715... 2024-03-28T15:50:28.321 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:27 smithi067 podman[31689]: 2024-03-28 15:50:27.89677758 +0000 UTC m=+0.037823290 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T15:50:28.321 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:27 smithi067 podman[31689]: 2024-03-28 15:50:27.995769844 +0000 UTC m=+0.136815582 container create ecc4fb80b575f73bbc7f18649c2d8c13d6756758b2184d98b2c5a8a68d64a63f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mon-a, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS) 2024-03-28T15:50:34.072 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 podman[31689]: 2024-03-28 15:50:33.812768128 +0000 UTC m=+5.953813881 container init ecc4fb80b575f73bbc7f18649c2d8c13d6756758b2184d98b2c5a8a68d64a63f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-28T15:50:34.073 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 podman[31689]: 2024-03-28 15:50:33.823542257 +0000 UTC m=+5.964588034 container start ecc4fb80b575f73bbc7f18649c2d8c13d6756758b2184d98b2c5a8a68d64a63f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mon-a, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T15:50:34.073 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: set uid:gid to 167:167 (ceph:ceph) 2024-03-28T15:50:34.073 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev), process ceph-mon, pid 2 2024-03-28T15:50:34.073 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: pidfile_write: ignore empty --pid-file 2024-03-28T15:50:34.073 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: load: jerasure load: lrc 2024-03-28T15:50:34.073 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: RocksDB version: 7.9.2 2024-03-28T15:50:34.073 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Git sha 0 2024-03-28T15:50:34.073 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Compile date 2024-03-22 04:58:55 2024-03-28T15:50:34.073 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: DB SUMMARY 2024-03-28T15:50:34.073 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: DB Session ID: BOQIW02SVAX90Y1M82FF 2024-03-28T15:50:34.074 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: CURRENT file: CURRENT 2024-03-28T15:50:34.074 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: IDENTITY file: IDENTITY 2024-03-28T15:50:34.074 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-28T15:50:34.074 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-28T15:50:34.074 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 99432 ; 2024-03-28T15:50:34.074 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.error_if_exists: 0 2024-03-28T15:50:34.074 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.create_if_missing: 0 2024-03-28T15:50:34.074 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.paranoid_checks: 1 2024-03-28T15:50:34.074 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-28T15:50:34.074 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-28T15:50:34.074 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-28T15:50:34.074 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.env: 0x562517331b20 2024-03-28T15:50:34.075 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.fs: PosixFileSystem 2024-03-28T15:50:34.075 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.info_log: 0x562519085960 2024-03-28T15:50:34.075 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.max_file_opening_threads: 16 2024-03-28T15:50:34.075 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.statistics: (nil) 2024-03-28T15:50:34.075 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.use_fsync: 0 2024-03-28T15:50:34.075 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.max_log_file_size: 0 2024-03-28T15:50:34.076 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-28T15:50:34.076 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-28T15:50:34.076 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.keep_log_file_num: 1000 2024-03-28T15:50:34.076 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.recycle_log_file_num: 0 2024-03-28T15:50:34.076 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.allow_fallocate: 1 2024-03-28T15:50:34.076 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.allow_mmap_reads: 0 2024-03-28T15:50:34.076 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.allow_mmap_writes: 0 2024-03-28T15:50:34.076 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.use_direct_reads: 0 2024-03-28T15:50:34.076 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-28T15:50:34.076 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.create_missing_column_families: 0 2024-03-28T15:50:34.076 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.db_log_dir: 2024-03-28T15:50:34.076 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.wal_dir: 2024-03-28T15:50:34.077 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-28T15:50:34.077 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-28T15:50:34.077 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-28T15:50:34.077 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-28T15:50:34.077 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-28T15:50:34.077 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-28T15:50:34.077 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.advise_random_on_open: 1 2024-03-28T15:50:34.077 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.db_write_buffer_size: 0 2024-03-28T15:50:34.077 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.write_buffer_manager: 0x5625190581e0 2024-03-28T15:50:34.077 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-28T15:50:34.077 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-28T15:50:34.077 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-28T15:50:34.078 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.rate_limiter: (nil) 2024-03-28T15:50:34.078 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-28T15:50:34.078 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.wal_recovery_mode: 2 2024-03-28T15:50:34.078 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.enable_thread_tracking: 0 2024-03-28T15:50:34.078 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.enable_pipelined_write: 0 2024-03-28T15:50:34.078 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.unordered_write: 0 2024-03-28T15:50:34.078 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-28T15:50:34.078 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-28T15:50:34.078 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-28T15:50:34.078 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-28T15:50:34.078 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.row_cache: None 2024-03-28T15:50:34.078 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.wal_filter: None 2024-03-28T15:50:34.080 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-28T15:50:34.080 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.allow_ingest_behind: 0 2024-03-28T15:50:34.080 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.two_write_queues: 0 2024-03-28T15:50:34.080 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.manual_wal_flush: 0 2024-03-28T15:50:34.080 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.wal_compression: 0 2024-03-28T15:50:34.080 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.atomic_flush: 0 2024-03-28T15:50:34.080 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-28T15:50:34.080 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-28T15:50:34.081 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-28T15:50:34.081 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.log_readahead_size: 0 2024-03-28T15:50:34.081 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-28T15:50:34.081 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.best_efforts_recovery: 0 2024-03-28T15:50:34.081 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-28T15:50:34.081 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-28T15:50:34.081 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.allow_data_in_errors: 0 2024-03-28T15:50:34.081 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.db_host_id: __hostname__ 2024-03-28T15:50:34.081 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-28T15:50:34.081 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.max_background_jobs: 2 2024-03-28T15:50:34.081 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.max_background_compactions: -1 2024-03-28T15:50:34.082 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.max_subcompactions: 1 2024-03-28T15:50:34.082 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-28T15:50:34.082 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-28T15:50:34.082 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-28T15:50:34.082 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.max_total_wal_size: 0 2024-03-28T15:50:34.082 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-28T15:50:34.082 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-28T15:50:34.082 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-28T15:50:34.082 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-28T15:50:34.082 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.max_open_files: -1 2024-03-28T15:50:34.082 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.bytes_per_sync: 0 2024-03-28T15:50:34.082 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-28T15:50:34.083 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-28T15:50:34.083 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.compaction_readahead_size: 0 2024-03-28T15:50:34.083 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.max_background_flushes: -1 2024-03-28T15:50:34.083 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Compression algorithms supported: 2024-03-28T15:50:34.083 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-28T15:50:34.083 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: kZSTD supported: 0 2024-03-28T15:50:34.083 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: kXpressCompression supported: 0 2024-03-28T15:50:34.083 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: kLZ4HCCompression supported: 1 2024-03-28T15:50:34.083 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: kZlibCompression supported: 1 2024-03-28T15:50:34.083 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: kSnappyCompression supported: 1 2024-03-28T15:50:34.083 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: kLZ4Compression supported: 1 2024-03-28T15:50:34.083 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: kBZip2Compression supported: 0 2024-03-28T15:50:34.084 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-28T15:50:34.084 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-28T15:50:34.084 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-28T15:50:34.084 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-28T15:50:34.084 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-28T15:50:34.084 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.merge_operator: 2024-03-28T15:50:34.084 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.compaction_filter: None 2024-03-28T15:50:34.084 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.compaction_filter_factory: None 2024-03-28T15:50:34.084 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.sst_partitioner_factory: None 2024-03-28T15:50:34.084 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-28T15:50:34.084 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-28T15:50:34.084 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x562519085a60) 2024-03-28T15:50:34.085 INFO:journalctl@ceph.mon.a.smithi067.stdout: cache_index_and_filter_blocks: 1 2024-03-28T15:50:34.085 INFO:journalctl@ceph.mon.a.smithi067.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-28T15:50:34.085 INFO:journalctl@ceph.mon.a.smithi067.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-28T15:50:34.085 INFO:journalctl@ceph.mon.a.smithi067.stdout: pin_top_level_index_and_filter: 1 2024-03-28T15:50:34.085 INFO:journalctl@ceph.mon.a.smithi067.stdout: index_type: 0 2024-03-28T15:50:34.085 INFO:journalctl@ceph.mon.a.smithi067.stdout: data_block_index_type: 0 2024-03-28T15:50:34.085 INFO:journalctl@ceph.mon.a.smithi067.stdout: index_shortening: 1 2024-03-28T15:50:34.085 INFO:journalctl@ceph.mon.a.smithi067.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-28T15:50:34.085 INFO:journalctl@ceph.mon.a.smithi067.stdout: checksum: 4 2024-03-28T15:50:34.085 INFO:journalctl@ceph.mon.a.smithi067.stdout: no_block_cache: 0 2024-03-28T15:50:34.085 INFO:journalctl@ceph.mon.a.smithi067.stdout: block_cache: 0x56251908b090 2024-03-28T15:50:34.085 INFO:journalctl@ceph.mon.a.smithi067.stdout: block_cache_name: BinnedLRUCache 2024-03-28T15:50:34.086 INFO:journalctl@ceph.mon.a.smithi067.stdout: block_cache_options: 2024-03-28T15:50:34.086 INFO:journalctl@ceph.mon.a.smithi067.stdout: capacity : 536870912 2024-03-28T15:50:34.086 INFO:journalctl@ceph.mon.a.smithi067.stdout: num_shard_bits : 4 2024-03-28T15:50:34.086 INFO:journalctl@ceph.mon.a.smithi067.stdout: strict_capacity_limit : 0 2024-03-28T15:50:34.086 INFO:journalctl@ceph.mon.a.smithi067.stdout: high_pri_pool_ratio: 0.000 2024-03-28T15:50:34.086 INFO:journalctl@ceph.mon.a.smithi067.stdout: block_cache_compressed: (nil) 2024-03-28T15:50:34.086 INFO:journalctl@ceph.mon.a.smithi067.stdout: persistent_cache: (nil) 2024-03-28T15:50:34.086 INFO:journalctl@ceph.mon.a.smithi067.stdout: block_size: 4096 2024-03-28T15:50:34.086 INFO:journalctl@ceph.mon.a.smithi067.stdout: block_size_deviation: 10 2024-03-28T15:50:34.086 INFO:journalctl@ceph.mon.a.smithi067.stdout: block_restart_interval: 16 2024-03-28T15:50:34.086 INFO:journalctl@ceph.mon.a.smithi067.stdout: index_block_restart_interval: 1 2024-03-28T15:50:34.086 INFO:journalctl@ceph.mon.a.smithi067.stdout: metadata_block_size: 4096 2024-03-28T15:50:34.087 INFO:journalctl@ceph.mon.a.smithi067.stdout: partition_filters: 0 2024-03-28T15:50:34.087 INFO:journalctl@ceph.mon.a.smithi067.stdout: use_delta_encoding: 1 2024-03-28T15:50:34.087 INFO:journalctl@ceph.mon.a.smithi067.stdout: filter_policy: bloomfilter 2024-03-28T15:50:34.087 INFO:journalctl@ceph.mon.a.smithi067.stdout: whole_key_filtering: 1 2024-03-28T15:50:34.087 INFO:journalctl@ceph.mon.a.smithi067.stdout: verify_compression: 0 2024-03-28T15:50:34.087 INFO:journalctl@ceph.mon.a.smithi067.stdout: read_amp_bytes_per_bit: 0 2024-03-28T15:50:34.087 INFO:journalctl@ceph.mon.a.smithi067.stdout: format_version: 5 2024-03-28T15:50:34.087 INFO:journalctl@ceph.mon.a.smithi067.stdout: enable_index_compression: 1 2024-03-28T15:50:34.087 INFO:journalctl@ceph.mon.a.smithi067.stdout: block_align: 0 2024-03-28T15:50:34.087 INFO:journalctl@ceph.mon.a.smithi067.stdout: max_auto_readahead_size: 262144 2024-03-28T15:50:34.087 INFO:journalctl@ceph.mon.a.smithi067.stdout: prepopulate_block_cache: 0 2024-03-28T15:50:34.087 INFO:journalctl@ceph.mon.a.smithi067.stdout: initial_auto_readahead_size: 8192 2024-03-28T15:50:34.088 INFO:journalctl@ceph.mon.a.smithi067.stdout: num_file_reads_for_auto_readahead: 2 2024-03-28T15:50:34.088 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.write_buffer_size: 33554432 2024-03-28T15:50:34.088 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.max_write_buffer_number: 2 2024-03-28T15:50:34.088 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.compression: NoCompression 2024-03-28T15:50:34.088 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.bottommost_compression: Disabled 2024-03-28T15:50:34.088 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.prefix_extractor: nullptr 2024-03-28T15:50:34.088 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-28T15:50:34.088 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.num_levels: 7 2024-03-28T15:50:34.088 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-28T15:50:34.088 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-28T15:50:34.088 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-28T15:50:34.089 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-28T15:50:34.089 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-28T15:50:34.089 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-28T15:50:34.089 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-28T15:50:34.089 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-28T15:50:34.089 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-28T15:50:34.089 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-28T15:50:34.089 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-28T15:50:34.089 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-28T15:50:34.089 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-28T15:50:34.089 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.compression_opts.level: 32767 2024-03-28T15:50:34.089 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.compression_opts.strategy: 0 2024-03-28T15:50:34.089 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-28T15:50:34.090 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-28T15:50:34.090 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-28T15:50:34.090 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-28T15:50:34.090 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.compression_opts.enabled: false 2024-03-28T15:50:34.090 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-28T15:50:34.090 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-28T15:50:34.090 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-28T15:50:34.090 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-28T15:50:34.090 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.target_file_size_base: 67108864 2024-03-28T15:50:34.090 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-28T15:50:34.090 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-28T15:50:34.090 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-28T15:50:34.091 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-28T15:50:34.091 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-28T15:50:34.091 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-28T15:50:34.091 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-28T15:50:34.091 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-28T15:50:34.091 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-28T15:50:34.091 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-28T15:50:34.091 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-28T15:50:34.091 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-28T15:50:34.091 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-28T15:50:34.091 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-28T15:50:34.091 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.arena_block_size: 1048576 2024-03-28T15:50:34.092 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-28T15:50:34.092 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-28T15:50:34.092 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.disable_auto_compactions: 0 2024-03-28T15:50:34.092 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-28T15:50:34.092 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-28T15:50:34.092 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-28T15:50:34.092 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-28T15:50:34.092 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-28T15:50:34.092 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-28T15:50:34.092 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-28T15:50:34.092 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-28T15:50:34.092 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-28T15:50:34.092 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-28T15:50:34.093 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.table_properties_collectors: 2024-03-28T15:50:34.093 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.inplace_update_support: 0 2024-03-28T15:50:34.093 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-28T15:50:34.093 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-28T15:50:34.093 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-28T15:50:34.093 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-28T15:50:34.093 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.bloom_locality: 0 2024-03-28T15:50:34.093 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.max_successive_merges: 0 2024-03-28T15:50:34.093 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-28T15:50:34.093 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.paranoid_file_checks: 0 2024-03-28T15:50:34.093 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.force_consistency_checks: 1 2024-03-28T15:50:34.093 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.report_bg_io_stats: 0 2024-03-28T15:50:34.093 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.ttl: 2592000 2024-03-28T15:50:34.094 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-28T15:50:34.094 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-28T15:50:34.094 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-28T15:50:34.094 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.enable_blob_files: false 2024-03-28T15:50:34.094 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.min_blob_size: 0 2024-03-28T15:50:34.094 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.blob_file_size: 268435456 2024-03-28T15:50:34.094 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-28T15:50:34.094 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-28T15:50:34.094 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-28T15:50:34.094 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-28T15:50:34.094 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-28T15:50:34.094 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.blob_file_starting_level: 0 2024-03-28T15:50:34.095 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-28T15:50:34.095 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-28T15:50:34.095 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-28T15:50:34.095 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 8d557d78-aef8-4bf5-aae1-865a4b8ed46f 2024-03-28T15:50:34.095 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711641033886820, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-28T15:50:34.095 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-28T15:50:34.095 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711641033888756, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 94738, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 314, "table_properties": {"data_size": 92721, "index_size": 299, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13628, "raw_average_key_size": 49, "raw_value_size": 85255, "raw_average_value_size": 312, "num_data_blocks": 11, "num_entries": 273, "num_filter_entries": 273, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711641033, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "8d557d78-aef8-4bf5-aae1-865a4b8ed46f", "db_session_id": "BOQIW02SVAX90Y1M82FF", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-28T15:50:34.095 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711641033888922, "job": 1, "event": "recovery_finished"} 2024-03-28T15:50:34.095 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-28T15:50:34.095 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-28T15:50:34.095 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x56251917e000 2024-03-28T15:50:34.096 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: DB pointer 0x562519168000 2024-03-28T15:50:34.096 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-28T15:50:34.096 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-28T15:50:34.096 INFO:journalctl@ceph.mon.a.smithi067.stdout: ** DB Stats ** 2024-03-28T15:50:34.096 INFO:journalctl@ceph.mon.a.smithi067.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-28T15:50:34.096 INFO:journalctl@ceph.mon.a.smithi067.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-28T15:50:34.096 INFO:journalctl@ceph.mon.a.smithi067.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-28T15:50:34.096 INFO:journalctl@ceph.mon.a.smithi067.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-28T15:50:34.096 INFO:journalctl@ceph.mon.a.smithi067.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-28T15:50:34.096 INFO:journalctl@ceph.mon.a.smithi067.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-28T15:50:34.096 INFO:journalctl@ceph.mon.a.smithi067.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-28T15:50:34.097 INFO:journalctl@ceph.mon.a.smithi067.stdout: 2024-03-28T15:50:34.097 INFO:journalctl@ceph.mon.a.smithi067.stdout: ** Compaction Stats [default] ** 2024-03-28T15:50:34.097 INFO:journalctl@ceph.mon.a.smithi067.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-28T15:50:34.097 INFO:journalctl@ceph.mon.a.smithi067.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-28T15:50:34.097 INFO:journalctl@ceph.mon.a.smithi067.stdout: L0 2/0 94.35 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 69.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-28T15:50:34.097 INFO:journalctl@ceph.mon.a.smithi067.stdout: Sum 2/0 94.35 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 69.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-28T15:50:34.097 INFO:journalctl@ceph.mon.a.smithi067.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 69.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-28T15:50:34.097 INFO:journalctl@ceph.mon.a.smithi067.stdout: 2024-03-28T15:50:34.097 INFO:journalctl@ceph.mon.a.smithi067.stdout: ** Compaction Stats [default] ** 2024-03-28T15:50:34.097 INFO:journalctl@ceph.mon.a.smithi067.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-28T15:50:34.097 INFO:journalctl@ceph.mon.a.smithi067.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-28T15:50:34.098 INFO:journalctl@ceph.mon.a.smithi067.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 69.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-28T15:50:34.098 INFO:journalctl@ceph.mon.a.smithi067.stdout: 2024-03-28T15:50:34.098 INFO:journalctl@ceph.mon.a.smithi067.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-28T15:50:34.098 INFO:journalctl@ceph.mon.a.smithi067.stdout: 2024-03-28T15:50:34.098 INFO:journalctl@ceph.mon.a.smithi067.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-28T15:50:34.098 INFO:journalctl@ceph.mon.a.smithi067.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-28T15:50:34.098 INFO:journalctl@ceph.mon.a.smithi067.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-28T15:50:34.098 INFO:journalctl@ceph.mon.a.smithi067.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-28T15:50:34.098 INFO:journalctl@ceph.mon.a.smithi067.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-28T15:50:34.098 INFO:journalctl@ceph.mon.a.smithi067.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-28T15:50:34.098 INFO:journalctl@ceph.mon.a.smithi067.stdout: Cumulative compaction: 0.00 GB write, 12.15 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-28T15:50:34.099 INFO:journalctl@ceph.mon.a.smithi067.stdout: Interval compaction: 0.00 GB write, 12.15 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-28T15:50:34.099 INFO:journalctl@ceph.mon.a.smithi067.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-28T15:50:34.099 INFO:journalctl@ceph.mon.a.smithi067.stdout: Block cache BinnedLRUCache@0x56251908b090#2 capacity: 512.00 MB usage: 1.39 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 2.6e-05 secs_since: 0 2024-03-28T15:50:34.099 INFO:journalctl@ceph.mon.a.smithi067.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-03-28T15:50:34.099 INFO:journalctl@ceph.mon.a.smithi067.stdout: 2024-03-28T15:50:34.099 INFO:journalctl@ceph.mon.a.smithi067.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-28T15:50:34.099 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: starting mon.a rank 0 at public addrs [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] at bind addrs [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 2024-03-28T15:50:34.100 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: mon.a@-1(???) e1 preinit fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 2024-03-28T15:50:34.100 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: mon.a@-1(???).mds e1 new map 2024-03-28T15:50:34.100 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: mon.a@-1(???).mds e1 print_map 2024-03-28T15:50:34.100 INFO:journalctl@ceph.mon.a.smithi067.stdout: e1 2024-03-28T15:50:34.100 INFO:journalctl@ceph.mon.a.smithi067.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-28T15:50:34.100 INFO:journalctl@ceph.mon.a.smithi067.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-28T15:50:34.100 INFO:journalctl@ceph.mon.a.smithi067.stdout: legacy client fscid: -1 2024-03-28T15:50:34.100 INFO:journalctl@ceph.mon.a.smithi067.stdout: 2024-03-28T15:50:34.100 INFO:journalctl@ceph.mon.a.smithi067.stdout: No filesystems configured 2024-03-28T15:50:34.100 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-28T15:50:34.100 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-28T15:50:34.100 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-28T15:50:34.101 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-28T15:50:34.101 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-28T15:50:34.101 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-28T15:50:34.101 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: monmap e1: 1 mons at {a=[v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T15:50:34.101 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: fsmap 2024-03-28T15:50:34.101 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: osdmap e1: 0 total, 0 up, 0 in 2024-03-28T15:50:34.101 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:33 smithi067 ceph-mon[31707]: mgrmap e1: no daemons active 2024-03-28T15:50:36.137 INFO:teuthology.orchestra.run.smithi067.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-28T15:50:36.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:36 smithi067 bash[31689]: ecc4fb80b575f73bbc7f18649c2d8c13d6756758b2184d98b2c5a8a68d64a63f 2024-03-28T15:50:36.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:36 smithi067 systemd[1]: Started Ceph mon.a for b5f8225a-ed1a-11ee-95d0-87774f69a715. 2024-03-28T15:50:39.138 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:38.424+0000 7f7303600700 1 Processor -- start 2024-03-28T15:50:39.138 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:38.424+0000 7f7303600700 1 -- start start 2024-03-28T15:50:39.138 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:38.425+0000 7f7303600700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72fc109b80 0x7f72fc109fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:39.138 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:38.425+0000 7f7303600700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72fc10a570 con 0x7f72fc109b80 2024-03-28T15:50:39.138 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:38.426+0000 7f7300c00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72fc109b80 0x7f72fc109fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:39.138 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:38.426+0000 7f7300c00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72fc109b80 0x7f72fc109fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53428/0 (socket says 172.21.15.67:53428) 2024-03-28T15:50:39.138 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:38.426+0000 7f7300c00700 1 -- 172.21.15.67:0/4230767327 learned_addr learned my addr 172.21.15.67:0/4230767327 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:50:39.138 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:38.426+0000 7f7300c00700 1 -- 172.21.15.67:0/4230767327 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72fc10ad80 con 0x7f72fc109b80 2024-03-28T15:50:39.138 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:38.427+0000 7f7300c00700 1 --2- 172.21.15.67:0/4230767327 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72fc109b80 0x7f72fc109fa0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f72f00098a0 tx=0x7f72f0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bff91859f579ea1f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:39.138 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:38.427+0000 7f72fb400700 1 -- 172.21.15.67:0/4230767327 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f72f0004160 con 0x7f72fc109b80 2024-03-28T15:50:39.138 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:38.427+0000 7f72fb400700 1 -- 172.21.15.67:0/4230767327 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7f72f0004300 con 0x7f72fc109b80 2024-03-28T15:50:39.139 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:38.427+0000 7f72fb400700 1 -- 172.21.15.67:0/4230767327 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f72f0011a80 con 0x7f72fc109b80 2024-03-28T15:50:39.139 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:38.428+0000 7f7303600700 1 -- 172.21.15.67:0/4230767327 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72fc109b80 msgr2=0x7f72fc109fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:39.139 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:38.428+0000 7f7303600700 1 --2- 172.21.15.67:0/4230767327 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72fc109b80 0x7f72fc109fa0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f72f00098a0 tx=0x7f72f0009bb0 comp rx=0 tx=0).stop 2024-03-28T15:50:39.139 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:38.428+0000 7f7303600700 1 -- 172.21.15.67:0/4230767327 shutdown_connections 2024-03-28T15:50:39.139 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:38.428+0000 7f7303600700 1 --2- 172.21.15.67:0/4230767327 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72fc109b80 0x7f72fc109fa0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:39.139 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:38.428+0000 7f7303600700 1 -- 172.21.15.67:0/4230767327 >> 172.21.15.67:0/4230767327 conn(0x7f72fc07cf50 msgr2=0x7f72fc07d380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:39.139 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:38.428+0000 7f7303600700 1 -- 172.21.15.67:0/4230767327 shutdown_connections 2024-03-28T15:50:39.139 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:38.428+0000 7f7303600700 1 -- 172.21.15.67:0/4230767327 wait complete. 2024-03-28T15:50:39.139 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:38.429+0000 7f7303600700 1 Processor -- start 2024-03-28T15:50:39.139 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:38.429+0000 7f7303600700 1 -- start start 2024-03-28T15:50:39.139 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:38.430+0000 7f7303600700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72fc109b80 0x7f72fc19db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:39.139 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:38.430+0000 7f7303600700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72fc10e130 con 0x7f72fc109b80 2024-03-28T15:50:39.139 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:38.430+0000 7f7300c00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72fc109b80 0x7f72fc19db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:39.139 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:38.430+0000 7f7300c00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72fc109b80 0x7f72fc19db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53436/0 (socket says 172.21.15.67:53436) 2024-03-28T15:50:39.139 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:38.430+0000 7f7300c00700 1 -- 172.21.15.67:0/344390558 learned_addr learned my addr 172.21.15.67:0/344390558 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:50:39.140 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:38.431+0000 7f7300c00700 1 -- 172.21.15.67:0/344390558 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72fc19e0d0 con 0x7f72fc109b80 2024-03-28T15:50:39.140 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:38.431+0000 7f7300c00700 1 --2- 172.21.15.67:0/344390558 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72fc109b80 0x7f72fc19db90 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f72f0006af0 tx=0x7f72f0011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:39.140 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:38.431+0000 7f72f9600700 1 -- 172.21.15.67:0/344390558 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f72f0003930 con 0x7f72fc109b80 2024-03-28T15:50:39.140 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:38.431+0000 7f72f9600700 1 -- 172.21.15.67:0/344390558 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7f72f0003ad0 con 0x7f72fc109b80 2024-03-28T15:50:39.140 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:38.431+0000 7f7303600700 1 -- 172.21.15.67:0/344390558 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72fc19e360 con 0x7f72fc109b80 2024-03-28T15:50:39.140 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:38.431+0000 7f72f9600700 1 -- 172.21.15.67:0/344390558 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f72f00195e0 con 0x7f72fc109b80 2024-03-28T15:50:39.140 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:38.431+0000 7f7303600700 1 -- 172.21.15.67:0/344390558 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72fc1a1050 con 0x7f72fc109b80 2024-03-28T15:50:39.140 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:38.432+0000 7f72f9600700 1 -- 172.21.15.67:0/344390558 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f72f0019c90 con 0x7f72fc109b80 2024-03-28T15:50:39.140 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:38.432+0000 7f72f9600700 1 -- 172.21.15.67:0/344390558 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f72f0024b60 con 0x7f72fc109b80 2024-03-28T15:50:39.140 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:38.433+0000 7f7303600700 1 -- 172.21.15.67:0/344390558 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72fc0637c0 con 0x7f72fc109b80 2024-03-28T15:50:39.140 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:38.436+0000 7f72f9600700 1 -- 172.21.15.67:0/344390558 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f72f0043dd0 con 0x7f72fc109b80 2024-03-28T15:50:39.140 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:38.512+0000 7f7303600700 1 -- 172.21.15.67:0/344390558 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f72fc1a15f0 con 0x7f72fc109b80 2024-03-28T15:50:39.140 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:38.517+0000 7f72f9600700 1 -- 172.21.15.67:0/344390558 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f72f0032030 con 0x7f72fc109b80 2024-03-28T15:50:39.141 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:38.518+0000 7f7303600700 1 -- 172.21.15.67:0/344390558 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72fc109b80 msgr2=0x7f72fc19db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:39.141 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:38.518+0000 7f7303600700 1 --2- 172.21.15.67:0/344390558 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72fc109b80 0x7f72fc19db90 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f72f0006af0 tx=0x7f72f0011b70 comp rx=0 tx=0).stop 2024-03-28T15:50:39.141 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:38.518+0000 7f7303600700 1 -- 172.21.15.67:0/344390558 shutdown_connections 2024-03-28T15:50:39.141 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:38.518+0000 7f7303600700 1 --2- 172.21.15.67:0/344390558 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72fc109b80 0x7f72fc19db90 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:39.141 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:38.518+0000 7f7303600700 1 -- 172.21.15.67:0/344390558 >> 172.21.15.67:0/344390558 conn(0x7f72fc07cf50 msgr2=0x7f72fc107580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:39.141 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:38.518+0000 7f7303600700 1 -- 172.21.15.67:0/344390558 shutdown_connections 2024-03-28T15:50:39.141 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:38.519+0000 7f7303600700 1 -- 172.21.15.67:0/344390558 wait complete. 2024-03-28T15:50:39.141 INFO:teuthology.orchestra.run.smithi067.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-28T15:50:39.161 INFO:teuthology.orchestra.run.smithi067.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-28T15:50:39.161 INFO:teuthology.orchestra.run.smithi067.stdout:Creating mgr... 2024-03-28T15:50:39.163 INFO:teuthology.orchestra.run.smithi067.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-28T15:50:39.164 INFO:teuthology.orchestra.run.smithi067.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-28T15:50:39.620 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@mgr.x 2024-03-28T15:50:39.620 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stderr Failed to reset failed state of unit ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@mgr.x.service: Unit ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@mgr.x.service not loaded. 2024-03-28T15:50:39.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:39 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/344390558' entity='client.admin' 2024-03-28T15:50:39.893 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715.target.wants/ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@mgr.x.service → /etc/systemd/system/ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@.service. 2024-03-28T15:50:40.279 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:50:39 smithi067 systemd[1]: Starting Ceph mgr.x for b5f8225a-ed1a-11ee-95d0-87774f69a715... 2024-03-28T15:50:40.570 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:50:40 smithi067 podman[31933]: 2024-03-28 15:50:40.182476839 +0000 UTC m=+0.031056322 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T15:50:40.570 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:50:40 smithi067 podman[31933]: 2024-03-28 15:50:40.280891861 +0000 UTC m=+0.129471362 container create a4c2b1cc1e783368f00d60cedae51385a8867410c9931a84cb6aba0639ebffe1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-28T15:50:40.570 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:50:40 smithi067 podman[31933]: 2024-03-28 15:50:40.509412244 +0000 UTC m=+0.357991793 container init a4c2b1cc1e783368f00d60cedae51385a8867410c9931a84cb6aba0639ebffe1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-28T15:50:40.570 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:50:40 smithi067 podman[31933]: 2024-03-28 15:50:40.52169539 +0000 UTC m=+0.370274868 container start a4c2b1cc1e783368f00d60cedae51385a8867410c9931a84cb6aba0639ebffe1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-28T15:50:40.604 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-28T15:50:40.604 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stdout disabled 2024-03-28T15:50:40.620 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-28T15:50:40.620 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stdout inactive 2024-03-28T15:50:40.620 INFO:teuthology.orchestra.run.smithi067.stdout:firewalld.service is not enabled 2024-03-28T15:50:40.620 INFO:teuthology.orchestra.run.smithi067.stdout:Not possible to enable service . firewalld.service is not available 2024-03-28T15:50:40.638 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-28T15:50:40.639 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stdout disabled 2024-03-28T15:50:40.654 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-28T15:50:40.654 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stdout inactive 2024-03-28T15:50:40.654 INFO:teuthology.orchestra.run.smithi067.stdout:firewalld.service is not enabled 2024-03-28T15:50:40.654 INFO:teuthology.orchestra.run.smithi067.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-28T15:50:40.654 INFO:teuthology.orchestra.run.smithi067.stdout:Waiting for mgr to start... 2024-03-28T15:50:40.654 INFO:teuthology.orchestra.run.smithi067.stdout:Waiting for mgr... 2024-03-28T15:50:40.826 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:50:40 smithi067 bash[31933]: a4c2b1cc1e783368f00d60cedae51385a8867410c9931a84cb6aba0639ebffe1 2024-03-28T15:50:40.826 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:50:40 smithi067 systemd[1]: Started Ceph mgr.x for b5f8225a-ed1a-11ee-95d0-87774f69a715. 2024-03-28T15:50:41.452 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:41 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/3746334691' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-28T15:50:41.452 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:50:41 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:50:41.323+0000 7fb56c6b3200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T15:50:41.452 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:50:41 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:50:41.450+0000 7fb56c6b3200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T15:50:42.012 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 2024-03-28T15:50:42.013 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout { 2024-03-28T15:50:42.013 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "fsid": "b5f8225a-ed1a-11ee-95d0-87774f69a715", 2024-03-28T15:50:42.013 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "health": { 2024-03-28T15:50:42.014 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-28T15:50:42.014 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-28T15:50:42.014 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-28T15:50:42.014 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-03-28T15:50:42.014 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-28T15:50:42.014 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-28T15:50:42.021 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 0 2024-03-28T15:50:42.021 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout ], 2024-03-28T15:50:42.021 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-28T15:50:42.022 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "a" 2024-03-28T15:50:42.022 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout ], 2024-03-28T15:50:42.022 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "quorum_age": 7, 2024-03-28T15:50:42.022 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-28T15:50:42.022 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T15:50:42.022 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-28T15:50:42.022 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-28T15:50:42.022 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-03-28T15:50:42.022 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-28T15:50:42.022 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T15:50:42.022 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-28T15:50:42.022 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-28T15:50:42.023 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-28T15:50:42.023 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-28T15:50:42.023 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-28T15:50:42.023 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-28T15:50:42.023 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-03-28T15:50:42.023 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-28T15:50:42.023 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-28T15:50:42.023 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-28T15:50:42.023 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-28T15:50:42.023 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-28T15:50:42.023 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-28T15:50:42.023 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-28T15:50:42.023 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-28T15:50:42.024 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-28T15:50:42.024 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-03-28T15:50:42.024 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-28T15:50:42.024 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T15:50:42.024 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-28T15:50:42.024 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-28T15:50:42.024 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-03-28T15:50:42.024 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-28T15:50:42.024 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-28T15:50:42.024 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-28T15:50:42.024 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-28T15:50:42.024 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-28T15:50:42.024 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-28T15:50:42.024 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "restful" 2024-03-28T15:50:42.025 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout ], 2024-03-28T15:50:42.025 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T15:50:42.025 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-03-28T15:50:42.025 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-28T15:50:42.025 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T15:50:42.025 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "modified": "2024-03-28T15:50:21.472546+0000", 2024-03-28T15:50:42.025 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T15:50:42.025 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-03-28T15:50:42.025 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-28T15:50:42.025 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout } 2024-03-28T15:50:42.025 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:41.218+0000 7f4bc0400700 1 Processor -- start 2024-03-28T15:50:42.025 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:41.218+0000 7f4bc0400700 1 -- start start 2024-03-28T15:50:42.025 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:41.218+0000 7f4bc0400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bb8108480 0x7f4bb810a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:42.025 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:41.218+0000 7f4bc0400700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb8063430 con 0x7f4bb8108480 2024-03-28T15:50:42.030 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:41.219+0000 7f4bbda00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bb8108480 0x7f4bb810a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:42.030 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:41.219+0000 7f4bbda00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bb8108480 0x7f4bb810a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38078/0 (socket says 172.21.15.67:38078) 2024-03-28T15:50:42.030 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:41.219+0000 7f4bbda00700 1 -- 172.21.15.67:0/1072047698 learned_addr learned my addr 172.21.15.67:0/1072047698 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:50:42.030 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:41.219+0000 7f4bbda00700 1 -- 172.21.15.67:0/1072047698 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bb810adb0 con 0x7f4bb8108480 2024-03-28T15:50:42.030 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:41.220+0000 7f4bbda00700 1 --2- 172.21.15.67:0/1072047698 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bb8108480 0x7f4bb810a870 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f4bac0098a0 tx=0x7f4bac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d799261ab0c84272 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:42.030 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:41.220+0000 7f4bb7e00700 1 -- 172.21.15.67:0/1072047698 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4bac004160 con 0x7f4bb8108480 2024-03-28T15:50:42.030 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:41.220+0000 7f4bb7e00700 1 -- 172.21.15.67:0/1072047698 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7f4bac004300 con 0x7f4bb8108480 2024-03-28T15:50:42.030 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:41.220+0000 7f4bb7e00700 1 -- 172.21.15.67:0/1072047698 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4bac011a80 con 0x7f4bb8108480 2024-03-28T15:50:42.030 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:41.221+0000 7f4bc0400700 1 -- 172.21.15.67:0/1072047698 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bb8108480 msgr2=0x7f4bb810a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:42.030 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:41.221+0000 7f4bc0400700 1 --2- 172.21.15.67:0/1072047698 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bb8108480 0x7f4bb810a870 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f4bac0098a0 tx=0x7f4bac009bb0 comp rx=0 tx=0).stop 2024-03-28T15:50:42.031 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:41.221+0000 7f4bc0400700 1 -- 172.21.15.67:0/1072047698 shutdown_connections 2024-03-28T15:50:42.031 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:41.221+0000 7f4bc0400700 1 --2- 172.21.15.67:0/1072047698 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bb8108480 0x7f4bb810a870 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:42.031 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:41.221+0000 7f4bc0400700 1 -- 172.21.15.67:0/1072047698 >> 172.21.15.67:0/1072047698 conn(0x7f4bb8101e70 msgr2=0x7f4bb8104290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:42.031 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:41.221+0000 7f4bc0400700 1 -- 172.21.15.67:0/1072047698 shutdown_connections 2024-03-28T15:50:42.031 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:41.221+0000 7f4bc0400700 1 -- 172.21.15.67:0/1072047698 wait complete. 2024-03-28T15:50:42.031 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:41.222+0000 7f4bc0400700 1 Processor -- start 2024-03-28T15:50:42.031 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:41.223+0000 7f4bc0400700 1 -- start start 2024-03-28T15:50:42.031 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:41.223+0000 7f4bc0400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bb8108480 0x7f4bb8107d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:42.031 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:41.223+0000 7f4bc0400700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb810bd10 con 0x7f4bb8108480 2024-03-28T15:50:42.031 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:41.223+0000 7f4bbda00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bb8108480 0x7f4bb8107d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:42.031 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:41.223+0000 7f4bbda00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bb8108480 0x7f4bb8107d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38080/0 (socket says 172.21.15.67:38080) 2024-03-28T15:50:42.031 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:41.223+0000 7f4bbda00700 1 -- 172.21.15.67:0/3746334691 learned_addr learned my addr 172.21.15.67:0/3746334691 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:50:42.031 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:41.224+0000 7f4bbda00700 1 -- 172.21.15.67:0/3746334691 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bb8108240 con 0x7f4bb8108480 2024-03-28T15:50:42.032 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:41.224+0000 7f4bbda00700 1 --2- 172.21.15.67:0/3746334691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bb8108480 0x7f4bb8107d00 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4bac011a50 tx=0x7f4bac004290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:42.032 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:41.224+0000 7f4bb6000700 1 -- 172.21.15.67:0/3746334691 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4bac0039f0 con 0x7f4bb8108480 2024-03-28T15:50:42.032 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:41.224+0000 7f4bb6000700 1 -- 172.21.15.67:0/3746334691 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7f4bac003b90 con 0x7f4bb8108480 2024-03-28T15:50:42.032 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:41.224+0000 7f4bc0400700 1 -- 172.21.15.67:0/3746334691 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bb8106430 con 0x7f4bb8108480 2024-03-28T15:50:42.032 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:41.224+0000 7f4bb6000700 1 -- 172.21.15.67:0/3746334691 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4bac01a600 con 0x7f4bb8108480 2024-03-28T15:50:42.032 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:41.225+0000 7f4bc0400700 1 -- 172.21.15.67:0/3746334691 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bb8106970 con 0x7f4bb8108480 2024-03-28T15:50:42.032 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:41.225+0000 7f4bb6000700 1 -- 172.21.15.67:0/3746334691 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f4bac01ac60 con 0x7f4bb8108480 2024-03-28T15:50:42.032 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:41.225+0000 7f4bb6000700 1 -- 172.21.15.67:0/3746334691 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f4bac019700 con 0x7f4bb8108480 2024-03-28T15:50:42.032 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:41.226+0000 7f4bc0400700 1 -- 172.21.15.67:0/3746334691 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bb80637c0 con 0x7f4bb8108480 2024-03-28T15:50:42.032 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:41.229+0000 7f4bb6000700 1 -- 172.21.15.67:0/3746334691 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f4bac024a40 con 0x7f4bb8108480 2024-03-28T15:50:42.032 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:41.309+0000 7f4bc0400700 1 -- 172.21.15.67:0/3746334691 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f4bb8106c20 con 0x7f4bb8108480 2024-03-28T15:50:42.032 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:41.309+0000 7f4bb6000700 1 -- 172.21.15.67:0/3746334691 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f4bac024440 con 0x7f4bb8108480 2024-03-28T15:50:42.032 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:41.310+0000 7f4bc0400700 1 -- 172.21.15.67:0/3746334691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bb8108480 msgr2=0x7f4bb8107d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:42.032 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:41.310+0000 7f4bc0400700 1 --2- 172.21.15.67:0/3746334691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bb8108480 0x7f4bb8107d00 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4bac011a50 tx=0x7f4bac004290 comp rx=0 tx=0).stop 2024-03-28T15:50:42.032 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:41.311+0000 7f4bc0400700 1 -- 172.21.15.67:0/3746334691 shutdown_connections 2024-03-28T15:50:42.033 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:41.311+0000 7f4bc0400700 1 --2- 172.21.15.67:0/3746334691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bb8108480 0x7f4bb8107d00 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:42.033 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:41.311+0000 7f4bc0400700 1 -- 172.21.15.67:0/3746334691 >> 172.21.15.67:0/3746334691 conn(0x7f4bb8101e70 msgr2=0x7f4bb81029f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:42.033 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:41.311+0000 7f4bc0400700 1 -- 172.21.15.67:0/3746334691 shutdown_connections 2024-03-28T15:50:42.033 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:41.311+0000 7f4bc0400700 1 -- 172.21.15.67:0/3746334691 wait complete. 2024-03-28T15:50:42.033 INFO:teuthology.orchestra.run.smithi067.stdout:mgr not available, waiting (1/15)... 2024-03-28T15:50:42.569 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:50:42 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:50:42.278+0000 7fb56c6b3200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T15:50:42.569 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:50:42 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:50:42.402+0000 7fb56c6b3200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T15:50:43.069 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:50:42 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:50:42.581+0000 7fb56c6b3200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T15:50:43.069 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:50:42 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:50:42.708+0000 7fb56c6b3200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T15:50:43.070 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:50:42 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:50:42.821+0000 7fb56c6b3200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T15:50:43.419 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:50:43 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:50:43.158+0000 7fb56c6b3200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T15:50:43.419 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:50:43 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:50:43.302+0000 7fb56c6b3200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T15:50:43.766 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:50:43 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:50:43.417+0000 7fb56c6b3200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T15:50:43.766 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:50:43 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:50:43.544+0000 7fb56c6b3200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T15:50:43.767 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:50:43 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:50:43.764+0000 7fb56c6b3200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T15:50:45.063 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:44 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/151003876' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-28T15:50:45.355 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:50:45 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:50:45.175+0000 7fb56c6b3200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T15:50:45.355 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:50:45 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:50:45.354+0000 7fb56c6b3200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T15:50:45.380 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 2024-03-28T15:50:45.391 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout { 2024-03-28T15:50:45.391 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "fsid": "b5f8225a-ed1a-11ee-95d0-87774f69a715", 2024-03-28T15:50:45.391 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "health": { 2024-03-28T15:50:45.391 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-28T15:50:45.391 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-28T15:50:45.391 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-28T15:50:45.391 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-03-28T15:50:45.392 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-28T15:50:45.392 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-28T15:50:45.392 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 0 2024-03-28T15:50:45.392 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout ], 2024-03-28T15:50:45.392 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-28T15:50:45.392 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "a" 2024-03-28T15:50:45.392 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout ], 2024-03-28T15:50:45.392 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-03-28T15:50:45.392 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-28T15:50:45.392 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T15:50:45.392 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-28T15:50:45.392 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-28T15:50:45.392 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-03-28T15:50:45.392 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-28T15:50:45.393 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T15:50:45.393 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-28T15:50:45.393 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-28T15:50:45.393 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-28T15:50:45.393 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-28T15:50:45.393 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-28T15:50:45.393 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-28T15:50:45.393 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-03-28T15:50:45.393 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-28T15:50:45.393 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-28T15:50:45.393 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-28T15:50:45.393 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-28T15:50:45.393 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-28T15:50:45.393 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-28T15:50:45.393 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-28T15:50:45.394 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-28T15:50:45.394 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-28T15:50:45.394 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-03-28T15:50:45.394 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-28T15:50:45.394 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T15:50:45.394 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-28T15:50:45.394 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-28T15:50:45.394 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-03-28T15:50:45.394 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-28T15:50:45.394 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-28T15:50:45.394 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-28T15:50:45.394 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-28T15:50:45.394 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-28T15:50:45.394 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-28T15:50:45.395 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "restful" 2024-03-28T15:50:45.395 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout ], 2024-03-28T15:50:45.395 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T15:50:45.395 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-03-28T15:50:45.395 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-28T15:50:45.395 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T15:50:45.395 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "modified": "2024-03-28T15:50:21.472546+0000", 2024-03-28T15:50:45.395 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T15:50:45.395 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-03-28T15:50:45.395 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-28T15:50:45.395 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout } 2024-03-28T15:50:45.395 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:44.595+0000 7fdcb2000700 1 Processor -- start 2024-03-28T15:50:45.395 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:44.595+0000 7fdcb2000700 1 -- start start 2024-03-28T15:50:45.395 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:44.595+0000 7fdcb2000700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcac107970 0x7fdcac107d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:45.395 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:44.595+0000 7fdcb2000700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcac108360 con 0x7fdcac107970 2024-03-28T15:50:45.396 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:44.596+0000 7fdcab400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcac107970 0x7fdcac107d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:45.396 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:44.596+0000 7fdcab400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcac107970 0x7fdcac107d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38086/0 (socket says 172.21.15.67:38086) 2024-03-28T15:50:45.396 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:44.596+0000 7fdcab400700 1 -- 172.21.15.67:0/3943750390 learned_addr learned my addr 172.21.15.67:0/3943750390 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:50:45.396 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:44.596+0000 7fdcab400700 1 -- 172.21.15.67:0/3943750390 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcac108b70 con 0x7fdcac107970 2024-03-28T15:50:45.396 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:44.597+0000 7fdcab400700 1 --2- 172.21.15.67:0/3943750390 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcac107970 0x7fdcac107d90 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fdc9c0098a0 tx=0x7fdc9c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=980c8fdce1421464 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:45.396 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:44.597+0000 7fdcaa000700 1 -- 172.21.15.67:0/3943750390 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdc9c004160 con 0x7fdcac107970 2024-03-28T15:50:45.396 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:44.597+0000 7fdcaa000700 1 -- 172.21.15.67:0/3943750390 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7fdc9c004300 con 0x7fdcac107970 2024-03-28T15:50:45.396 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:44.597+0000 7fdcaa000700 1 -- 172.21.15.67:0/3943750390 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdc9c011a80 con 0x7fdcac107970 2024-03-28T15:50:45.396 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:44.598+0000 7fdcb2000700 1 -- 172.21.15.67:0/3943750390 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcac107970 msgr2=0x7fdcac107d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:45.396 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:44.598+0000 7fdcb2000700 1 --2- 172.21.15.67:0/3943750390 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcac107970 0x7fdcac107d90 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fdc9c0098a0 tx=0x7fdc9c009bb0 comp rx=0 tx=0).stop 2024-03-28T15:50:45.396 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:44.598+0000 7fdcb2000700 1 -- 172.21.15.67:0/3943750390 shutdown_connections 2024-03-28T15:50:45.396 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:44.598+0000 7fdcb2000700 1 --2- 172.21.15.67:0/3943750390 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcac107970 0x7fdcac107d90 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:45.396 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:44.598+0000 7fdcb2000700 1 -- 172.21.15.67:0/3943750390 >> 172.21.15.67:0/3943750390 conn(0x7fdcac102f70 msgr2=0x7fdcac105390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:45.396 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:44.598+0000 7fdcb2000700 1 -- 172.21.15.67:0/3943750390 shutdown_connections 2024-03-28T15:50:45.397 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:44.598+0000 7fdcb2000700 1 -- 172.21.15.67:0/3943750390 wait complete. 2024-03-28T15:50:45.397 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:44.599+0000 7fdcb2000700 1 Processor -- start 2024-03-28T15:50:45.397 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:44.600+0000 7fdcb2000700 1 -- start start 2024-03-28T15:50:45.397 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:44.600+0000 7fdcb2000700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcac107970 0x7fdcac07ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:45.397 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:44.600+0000 7fdcb2000700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcac10bf20 con 0x7fdcac107970 2024-03-28T15:50:45.397 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:44.600+0000 7fdcab400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcac107970 0x7fdcac07ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:45.397 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:44.600+0000 7fdcab400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcac107970 0x7fdcac07ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38102/0 (socket says 172.21.15.67:38102) 2024-03-28T15:50:45.397 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:44.600+0000 7fdcab400700 1 -- 172.21.15.67:0/151003876 learned_addr learned my addr 172.21.15.67:0/151003876 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:50:45.397 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:44.601+0000 7fdcab400700 1 -- 172.21.15.67:0/151003876 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcac07cf40 con 0x7fdcac107970 2024-03-28T15:50:45.397 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:44.601+0000 7fdcab400700 1 --2- 172.21.15.67:0/151003876 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcac107970 0x7fdcac07ca00 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fdc9c000c00 tx=0x7fdc9c011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:45.397 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:44.601+0000 7fdca3e00700 1 -- 172.21.15.67:0/151003876 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdc9c0039f0 con 0x7fdcac107970 2024-03-28T15:50:45.397 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:44.601+0000 7fdca3e00700 1 -- 172.21.15.67:0/151003876 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7fdc9c003b90 con 0x7fdcac107970 2024-03-28T15:50:45.397 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:44.601+0000 7fdca3e00700 1 -- 172.21.15.67:0/151003876 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdc9c01a740 con 0x7fdcac107970 2024-03-28T15:50:45.397 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:44.601+0000 7fdcb2000700 1 -- 172.21.15.67:0/151003876 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcac07b130 con 0x7fdcac107970 2024-03-28T15:50:45.397 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:44.602+0000 7fdcb2000700 1 -- 172.21.15.67:0/151003876 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcac07b670 con 0x7fdcac107970 2024-03-28T15:50:45.398 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:44.602+0000 7fdca3e00700 1 -- 172.21.15.67:0/151003876 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fdc9c02d3f0 con 0x7fdcac107970 2024-03-28T15:50:45.398 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:44.602+0000 7fdca3e00700 1 -- 172.21.15.67:0/151003876 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fdc9c019700 con 0x7fdcac107970 2024-03-28T15:50:45.398 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:44.603+0000 7fdcb2000700 1 -- 172.21.15.67:0/151003876 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcac0637c0 con 0x7fdcac107970 2024-03-28T15:50:45.398 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:44.607+0000 7fdca3e00700 1 -- 172.21.15.67:0/151003876 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fdc9c024af0 con 0x7fdcac107970 2024-03-28T15:50:45.398 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:44.687+0000 7fdcb2000700 1 -- 172.21.15.67:0/151003876 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fdcac07b9b0 con 0x7fdcac107970 2024-03-28T15:50:45.398 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:44.687+0000 7fdca3e00700 1 -- 172.21.15.67:0/151003876 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1240 (secure 0 0 0) 0x7fdc9c032030 con 0x7fdcac107970 2024-03-28T15:50:45.398 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:44.689+0000 7fdcb2000700 1 -- 172.21.15.67:0/151003876 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcac107970 msgr2=0x7fdcac07ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:45.398 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:44.689+0000 7fdcb2000700 1 --2- 172.21.15.67:0/151003876 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcac107970 0x7fdcac07ca00 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fdc9c000c00 tx=0x7fdc9c011b00 comp rx=0 tx=0).stop 2024-03-28T15:50:45.398 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:44.689+0000 7fdcb2000700 1 -- 172.21.15.67:0/151003876 shutdown_connections 2024-03-28T15:50:45.398 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:44.689+0000 7fdcb2000700 1 --2- 172.21.15.67:0/151003876 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcac107970 0x7fdcac07ca00 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:45.398 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:44.689+0000 7fdcb2000700 1 -- 172.21.15.67:0/151003876 >> 172.21.15.67:0/151003876 conn(0x7fdcac102f70 msgr2=0x7fdcac103ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:45.398 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:44.689+0000 7fdcb2000700 1 -- 172.21.15.67:0/151003876 shutdown_connections 2024-03-28T15:50:45.398 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:44.689+0000 7fdcb2000700 1 -- 172.21.15.67:0/151003876 wait complete. 2024-03-28T15:50:45.398 INFO:teuthology.orchestra.run.smithi067.stdout:mgr not available, waiting (2/15)... 2024-03-28T15:50:46.069 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:50:45 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:50:45.784+0000 7fb56c6b3200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T15:50:46.069 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:50:45 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:50:45.904+0000 7fb56c6b3200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T15:50:46.569 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:50:46 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:50:46.165+0000 7fb56c6b3200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T15:50:46.569 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:50:46 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:50:46.414+0000 7fb56c6b3200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T15:50:47.069 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:50:46 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:50:46.796+0000 7fb56c6b3200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T15:50:47.442 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:50:47 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:50:47.118+0000 7fb56c6b3200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T15:50:47.734 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:50:47 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:50:47.441+0000 7fb56c6b3200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T15:50:48.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:48 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/3348094475' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-28T15:50:48.727 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:50:48 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:50:48.513+0000 7fb56c6b3200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T15:50:48.739 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 2024-03-28T15:50:48.740 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout { 2024-03-28T15:50:48.740 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "fsid": "b5f8225a-ed1a-11ee-95d0-87774f69a715", 2024-03-28T15:50:48.740 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "health": { 2024-03-28T15:50:48.740 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-28T15:50:48.740 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-28T15:50:48.740 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-28T15:50:48.742 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-03-28T15:50:48.742 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-28T15:50:48.742 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-28T15:50:48.742 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 0 2024-03-28T15:50:48.742 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout ], 2024-03-28T15:50:48.742 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-28T15:50:48.742 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "a" 2024-03-28T15:50:48.742 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout ], 2024-03-28T15:50:48.742 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2024-03-28T15:50:48.742 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-28T15:50:48.743 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T15:50:48.743 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-28T15:50:48.743 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-28T15:50:48.743 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-03-28T15:50:48.744 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-28T15:50:48.744 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T15:50:48.744 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-28T15:50:48.744 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-28T15:50:48.744 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-28T15:50:48.744 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-28T15:50:48.744 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-28T15:50:48.744 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-28T15:50:48.744 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-03-28T15:50:48.745 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-28T15:50:48.749 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-28T15:50:48.749 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-28T15:50:48.749 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-28T15:50:48.749 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-28T15:50:48.750 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-28T15:50:48.750 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-28T15:50:48.750 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-28T15:50:48.750 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-28T15:50:48.750 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-03-28T15:50:48.750 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-28T15:50:48.750 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T15:50:48.750 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-28T15:50:48.750 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-28T15:50:48.750 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-03-28T15:50:48.750 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-28T15:50:48.750 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-28T15:50:48.750 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-28T15:50:48.750 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-28T15:50:48.750 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-28T15:50:48.751 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-28T15:50:48.751 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "restful" 2024-03-28T15:50:48.751 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout ], 2024-03-28T15:50:48.751 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T15:50:48.751 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-03-28T15:50:48.751 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-28T15:50:48.751 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T15:50:48.751 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "modified": "2024-03-28T15:50:21.472546+0000", 2024-03-28T15:50:48.751 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T15:50:48.751 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-03-28T15:50:48.751 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-28T15:50:48.751 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout } 2024-03-28T15:50:48.751 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:48.044+0000 7f673d000700 1 Processor -- start 2024-03-28T15:50:48.751 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:48.044+0000 7f673d000700 1 -- start start 2024-03-28T15:50:48.751 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:48.045+0000 7f673d000700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6738106270 0x7f6738106690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:48.752 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:48.045+0000 7f673d000700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6738063430 con 0x7f6738106270 2024-03-28T15:50:48.752 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:48.045+0000 7f6735e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6738106270 0x7f6738106690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:48.752 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:48.045+0000 7f6735e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6738106270 0x7f6738106690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38116/0 (socket says 172.21.15.67:38116) 2024-03-28T15:50:48.752 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:48.045+0000 7f6735e00700 1 -- 172.21.15.67:0/1655930430 learned_addr learned my addr 172.21.15.67:0/1655930430 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:50:48.752 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:48.046+0000 7f6735e00700 1 -- 172.21.15.67:0/1655930430 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6738106bd0 con 0x7f6738106270 2024-03-28T15:50:48.752 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:48.046+0000 7f6735e00700 1 --2- 172.21.15.67:0/1655930430 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6738106270 0x7f6738106690 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f67280098a0 tx=0x7f6728009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dac772a74d6ea87 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:48.752 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:48.046+0000 7f6734a00700 1 -- 172.21.15.67:0/1655930430 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6728004160 con 0x7f6738106270 2024-03-28T15:50:48.752 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:48.046+0000 7f6734a00700 1 -- 172.21.15.67:0/1655930430 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7f6728004300 con 0x7f6738106270 2024-03-28T15:50:48.752 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:48.046+0000 7f6734a00700 1 -- 172.21.15.67:0/1655930430 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6728011a80 con 0x7f6738106270 2024-03-28T15:50:48.752 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:48.047+0000 7f673d000700 1 -- 172.21.15.67:0/1655930430 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6738106270 msgr2=0x7f6738106690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:48.752 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:48.047+0000 7f673d000700 1 --2- 172.21.15.67:0/1655930430 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6738106270 0x7f6738106690 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f67280098a0 tx=0x7f6728009bb0 comp rx=0 tx=0).stop 2024-03-28T15:50:48.752 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:48.047+0000 7f673d000700 1 -- 172.21.15.67:0/1655930430 shutdown_connections 2024-03-28T15:50:48.756 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:48.047+0000 7f673d000700 1 --2- 172.21.15.67:0/1655930430 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6738106270 0x7f6738106690 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:48.756 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:48.047+0000 7f673d000700 1 -- 172.21.15.67:0/1655930430 >> 172.21.15.67:0/1655930430 conn(0x7f6738101e70 msgr2=0x7f6738104290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:48.756 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:48.048+0000 7f673d000700 1 -- 172.21.15.67:0/1655930430 shutdown_connections 2024-03-28T15:50:48.756 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:48.048+0000 7f673d000700 1 -- 172.21.15.67:0/1655930430 wait complete. 2024-03-28T15:50:48.756 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:48.049+0000 7f673d000700 1 Processor -- start 2024-03-28T15:50:48.756 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:48.049+0000 7f673d000700 1 -- start start 2024-03-28T15:50:48.756 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:48.049+0000 7f673d000700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6738106270 0x7f673819db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:48.756 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:48.049+0000 7f673d000700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6738107c00 con 0x7f6738106270 2024-03-28T15:50:48.756 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:48.050+0000 7f6735e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6738106270 0x7f673819db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:48.756 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:48.050+0000 7f6735e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6738106270 0x7f673819db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38124/0 (socket says 172.21.15.67:38124) 2024-03-28T15:50:48.756 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:48.050+0000 7f6735e00700 1 -- 172.21.15.67:0/3348094475 learned_addr learned my addr 172.21.15.67:0/3348094475 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:50:48.757 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:48.050+0000 7f6735e00700 1 -- 172.21.15.67:0/3348094475 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f673819e050 con 0x7f6738106270 2024-03-28T15:50:48.757 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:48.050+0000 7f6735e00700 1 --2- 172.21.15.67:0/3348094475 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6738106270 0x7f673819db10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6728011a50 tx=0x7f6728004290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:48.757 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:48.051+0000 7f672ea00700 1 -- 172.21.15.67:0/3348094475 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f67280039f0 con 0x7f6738106270 2024-03-28T15:50:48.757 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:48.051+0000 7f672ea00700 1 -- 172.21.15.67:0/3348094475 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7f6728003b90 con 0x7f6738106270 2024-03-28T15:50:48.757 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:48.051+0000 7f673d000700 1 -- 172.21.15.67:0/3348094475 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f673819e2e0 con 0x7f6738106270 2024-03-28T15:50:48.757 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:48.051+0000 7f672ea00700 1 -- 172.21.15.67:0/3348094475 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f672801a600 con 0x7f6738106270 2024-03-28T15:50:48.757 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:48.051+0000 7f673d000700 1 -- 172.21.15.67:0/3348094475 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67381a0fd0 con 0x7f6738106270 2024-03-28T15:50:48.757 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:48.051+0000 7f672ea00700 1 -- 172.21.15.67:0/3348094475 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f672801ac60 con 0x7f6738106270 2024-03-28T15:50:48.757 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:48.051+0000 7f672ea00700 1 -- 172.21.15.67:0/3348094475 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f6728019700 con 0x7f6738106270 2024-03-28T15:50:48.757 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:48.052+0000 7f673d000700 1 -- 172.21.15.67:0/3348094475 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67380637c0 con 0x7f6738106270 2024-03-28T15:50:48.757 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:48.055+0000 7f672ea00700 1 -- 172.21.15.67:0/3348094475 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f6728024a40 con 0x7f6738106270 2024-03-28T15:50:48.757 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:48.134+0000 7f673d000700 1 -- 172.21.15.67:0/3348094475 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f673802dca0 con 0x7f6738106270 2024-03-28T15:50:48.757 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:48.134+0000 7f672ea00700 1 -- 172.21.15.67:0/3348094475 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1240 (secure 0 0 0) 0x7f6728024440 con 0x7f6738106270 2024-03-28T15:50:48.757 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:48.136+0000 7f673d000700 1 -- 172.21.15.67:0/3348094475 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6738106270 msgr2=0x7f673819db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:48.757 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:48.136+0000 7f673d000700 1 --2- 172.21.15.67:0/3348094475 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6738106270 0x7f673819db10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6728011a50 tx=0x7f6728004290 comp rx=0 tx=0).stop 2024-03-28T15:50:48.757 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:48.136+0000 7f673d000700 1 -- 172.21.15.67:0/3348094475 shutdown_connections 2024-03-28T15:50:48.758 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:48.136+0000 7f673d000700 1 --2- 172.21.15.67:0/3348094475 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6738106270 0x7f673819db10 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:48.758 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:48.136+0000 7f673d000700 1 -- 172.21.15.67:0/3348094475 >> 172.21.15.67:0/3348094475 conn(0x7f6738101e70 msgr2=0x7f67381947d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:48.758 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:48.136+0000 7f673d000700 1 -- 172.21.15.67:0/3348094475 shutdown_connections 2024-03-28T15:50:48.758 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:48.136+0000 7f673d000700 1 -- 172.21.15.67:0/3348094475 wait complete. 2024-03-28T15:50:48.758 INFO:teuthology.orchestra.run.smithi067.stdout:mgr not available, waiting (3/15)... 2024-03-28T15:50:49.569 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:50:49 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:50:49.230+0000 7fb56c6b3200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T15:50:49.569 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:50:49 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:50:49.360+0000 7fb56c6b3200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T15:50:50.069 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:50:49 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:50:49.646+0000 7fb56c6b3200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T15:50:50.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:49 smithi067 ceph-mon[31707]: Activating manager daemon x 2024-03-28T15:50:50.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:49 smithi067 ceph-mon[31707]: mgrmap e2: x(active, starting, since 0.0106407s) 2024-03-28T15:50:50.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:49 smithi067 ceph-mon[31707]: from='mgr.14100 172.21.15.67:0/1227260394' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-28T15:50:50.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:49 smithi067 ceph-mon[31707]: from='mgr.14100 172.21.15.67:0/1227260394' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-28T15:50:50.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:49 smithi067 ceph-mon[31707]: from='mgr.14100 172.21.15.67:0/1227260394' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-28T15:50:50.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:49 smithi067 ceph-mon[31707]: from='mgr.14100 172.21.15.67:0/1227260394' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T15:50:50.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:49 smithi067 ceph-mon[31707]: from='mgr.14100 172.21.15.67:0/1227260394' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-28T15:50:50.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:49 smithi067 ceph-mon[31707]: Manager daemon x is now available 2024-03-28T15:50:50.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:49 smithi067 ceph-mon[31707]: from='mgr.14100 172.21.15.67:0/1227260394' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-28T15:50:50.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:49 smithi067 ceph-mon[31707]: from='mgr.14100 172.21.15.67:0/1227260394' entity='mgr.x' 2024-03-28T15:50:50.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:49 smithi067 ceph-mon[31707]: from='mgr.14100 172.21.15.67:0/1227260394' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-28T15:50:50.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:49 smithi067 ceph-mon[31707]: from='mgr.14100 172.21.15.67:0/1227260394' entity='mgr.x' 2024-03-28T15:50:50.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:49 smithi067 ceph-mon[31707]: from='mgr.14100 172.21.15.67:0/1227260394' entity='mgr.x' 2024-03-28T15:50:51.999 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:51 smithi067 ceph-mon[31707]: mgrmap e3: x(active, since 1.0222s) 2024-03-28T15:50:52.281 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 2024-03-28T15:50:52.281 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout { 2024-03-28T15:50:52.281 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "fsid": "b5f8225a-ed1a-11ee-95d0-87774f69a715", 2024-03-28T15:50:52.281 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "health": { 2024-03-28T15:50:52.281 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-28T15:50:52.283 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-28T15:50:52.283 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-28T15:50:52.283 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-03-28T15:50:52.283 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-28T15:50:52.283 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-28T15:50:52.283 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 0 2024-03-28T15:50:52.283 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout ], 2024-03-28T15:50:52.283 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-28T15:50:52.283 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "a" 2024-03-28T15:50:52.283 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout ], 2024-03-28T15:50:52.286 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "quorum_age": 17, 2024-03-28T15:50:52.286 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-28T15:50:52.286 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T15:50:52.286 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-28T15:50:52.286 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-28T15:50:52.286 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-03-28T15:50:52.286 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-28T15:50:52.286 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T15:50:52.286 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-28T15:50:52.286 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-28T15:50:52.286 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-28T15:50:52.286 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-28T15:50:52.287 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-28T15:50:52.287 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-28T15:50:52.287 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-03-28T15:50:52.287 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-28T15:50:52.287 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-28T15:50:52.287 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-28T15:50:52.287 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-28T15:50:52.287 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-28T15:50:52.291 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-28T15:50:52.291 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-28T15:50:52.291 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-28T15:50:52.291 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-28T15:50:52.291 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-03-28T15:50:52.291 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-28T15:50:52.291 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T15:50:52.291 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-28T15:50:52.291 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-28T15:50:52.291 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-03-28T15:50:52.291 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-28T15:50:52.292 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-28T15:50:52.292 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-28T15:50:52.292 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-28T15:50:52.292 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-28T15:50:52.292 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-28T15:50:52.292 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "restful" 2024-03-28T15:50:52.292 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout ], 2024-03-28T15:50:52.292 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T15:50:52.292 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-03-28T15:50:52.292 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-28T15:50:52.292 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T15:50:52.292 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "modified": "2024-03-28T15:50:21.472546+0000", 2024-03-28T15:50:52.292 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T15:50:52.292 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-03-28T15:50:52.292 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-28T15:50:52.293 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout } 2024-03-28T15:50:52.293 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:51.356+0000 7f3927400700 1 Processor -- start 2024-03-28T15:50:52.293 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:51.356+0000 7f3927400700 1 -- start start 2024-03-28T15:50:52.293 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:51.357+0000 7f3927400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3920106250 0x7f3920106670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:52.293 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:51.357+0000 7f3927400700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f392007bdd0 con 0x7f3920106250 2024-03-28T15:50:52.293 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:51.357+0000 7f3924a00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3920106250 0x7f3920106670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:52.293 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:51.357+0000 7f3924a00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3920106250 0x7f3920106670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37846/0 (socket says 172.21.15.67:37846) 2024-03-28T15:50:52.293 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:51.357+0000 7f3924a00700 1 -- 172.21.15.67:0/2612748256 learned_addr learned my addr 172.21.15.67:0/2612748256 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:50:52.298 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:51.358+0000 7f3924a00700 1 -- 172.21.15.67:0/2612748256 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3920106bb0 con 0x7f3920106250 2024-03-28T15:50:52.298 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:51.358+0000 7f3924a00700 1 --2- 172.21.15.67:0/2612748256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3920106250 0x7f3920106670 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f39140098a0 tx=0x7f3914009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d60403a96b0c3783 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:52.298 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:51.358+0000 7f391f400700 1 -- 172.21.15.67:0/2612748256 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3914004160 con 0x7f3920106250 2024-03-28T15:50:52.298 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:51.358+0000 7f391f400700 1 -- 172.21.15.67:0/2612748256 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7f3914004300 con 0x7f3920106250 2024-03-28T15:50:52.298 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:51.358+0000 7f391f400700 1 -- 172.21.15.67:0/2612748256 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3914011a80 con 0x7f3920106250 2024-03-28T15:50:52.298 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:51.359+0000 7f3927400700 1 -- 172.21.15.67:0/2612748256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3920106250 msgr2=0x7f3920106670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:52.298 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:51.359+0000 7f3927400700 1 --2- 172.21.15.67:0/2612748256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3920106250 0x7f3920106670 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f39140098a0 tx=0x7f3914009bb0 comp rx=0 tx=0).stop 2024-03-28T15:50:52.298 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:51.359+0000 7f3927400700 1 -- 172.21.15.67:0/2612748256 shutdown_connections 2024-03-28T15:50:52.298 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:51.359+0000 7f3927400700 1 --2- 172.21.15.67:0/2612748256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3920106250 0x7f3920106670 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:52.298 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:51.359+0000 7f3927400700 1 -- 172.21.15.67:0/2612748256 >> 172.21.15.67:0/2612748256 conn(0x7f3920101e70 msgr2=0x7f3920104290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:52.298 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:51.359+0000 7f3927400700 1 -- 172.21.15.67:0/2612748256 shutdown_connections 2024-03-28T15:50:52.299 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:51.359+0000 7f3927400700 1 -- 172.21.15.67:0/2612748256 wait complete. 2024-03-28T15:50:52.299 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:51.360+0000 7f3927400700 1 Processor -- start 2024-03-28T15:50:52.299 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:51.360+0000 7f3927400700 1 -- start start 2024-03-28T15:50:52.299 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:51.361+0000 7f3927400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3920106250 0x7f392019da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:52.299 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:51.361+0000 7f3927400700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3920107be0 con 0x7f3920106250 2024-03-28T15:50:52.299 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:51.361+0000 7f3924a00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3920106250 0x7f392019da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:52.299 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:51.361+0000 7f3924a00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3920106250 0x7f392019da80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37858/0 (socket says 172.21.15.67:37858) 2024-03-28T15:50:52.299 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:51.361+0000 7f3924a00700 1 -- 172.21.15.67:0/2696722408 learned_addr learned my addr 172.21.15.67:0/2696722408 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:50:52.299 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:51.361+0000 7f3924a00700 1 -- 172.21.15.67:0/2696722408 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f392019dfc0 con 0x7f3920106250 2024-03-28T15:50:52.299 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:51.362+0000 7f3924a00700 1 --2- 172.21.15.67:0/2696722408 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3920106250 0x7f392019da80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3914011a50 tx=0x7f3914011b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:52.299 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:51.362+0000 7f391d600700 1 -- 172.21.15.67:0/2696722408 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3914003a80 con 0x7f3920106250 2024-03-28T15:50:52.299 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:51.362+0000 7f391d600700 1 -- 172.21.15.67:0/2696722408 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7f3914003c20 con 0x7f3920106250 2024-03-28T15:50:52.299 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:51.362+0000 7f391d600700 1 -- 172.21.15.67:0/2696722408 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f391401a650 con 0x7f3920106250 2024-03-28T15:50:52.299 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:51.362+0000 7f3927400700 1 -- 172.21.15.67:0/2696722408 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f392019e250 con 0x7f3920106250 2024-03-28T15:50:52.299 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:51.362+0000 7f3927400700 1 -- 172.21.15.67:0/2696722408 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39201a0f40 con 0x7f3920106250 2024-03-28T15:50:52.300 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:51.363+0000 7f391d600700 1 -- 172.21.15.67:0/2696722408 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 3) v1 ==== 45989+0+0 (secure 0 0 0) 0x7f3914024ad0 con 0x7f3920106250 2024-03-28T15:50:52.300 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:51.363+0000 7f3927400700 1 -- 172.21.15.67:0/2696722408 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39200637c0 con 0x7f3920106250 2024-03-28T15:50:52.300 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:51.364+0000 7f391d600700 1 --2- 172.21.15.67:0/2696722408 >> [v2:172.21.15.67:6800/3580491071,v1:172.21.15.67:6801/3580491071] conn(0x7f3908038e00 0x7f390803b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:52.300 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:51.364+0000 7f391d600700 1 -- 172.21.15.67:0/2696722408 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f391404d090 con 0x7f3920106250 2024-03-28T15:50:52.300 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:51.364+0000 7f391fe00700 1 --2- 172.21.15.67:0/2696722408 >> [v2:172.21.15.67:6800/3580491071,v1:172.21.15.67:6801/3580491071] conn(0x7f3908038e00 0x7f390803b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:52.300 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:51.365+0000 7f391fe00700 1 --2- 172.21.15.67:0/2696722408 >> [v2:172.21.15.67:6800/3580491071,v1:172.21.15.67:6801/3580491071] conn(0x7f3908038e00 0x7f390803b2c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f390c006fd0 tx=0x7f390c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:52.300 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:51.370+0000 7f391d600700 1 -- 172.21.15.67:0/2696722408 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f39140244f0 con 0x7f3920106250 2024-03-28T15:50:52.300 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:51.671+0000 7f391d600700 1 -- 172.21.15.67:0/2696722408 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 4) v1 ==== 46095+0+0 (secure 0 0 0) 0x7f391402d3f0 con 0x7f3920106250 2024-03-28T15:50:52.300 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:51.688+0000 7f3927400700 1 -- 172.21.15.67:0/2696722408 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f39200008d0 con 0x7f3920106250 2024-03-28T15:50:52.300 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:51.689+0000 7f391d600700 1 -- 172.21.15.67:0/2696722408 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f391404f520 con 0x7f3920106250 2024-03-28T15:50:52.300 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:51.692+0000 7f3927400700 1 -- 172.21.15.67:0/2696722408 >> [v2:172.21.15.67:6800/3580491071,v1:172.21.15.67:6801/3580491071] conn(0x7f3908038e00 msgr2=0x7f390803b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:52.300 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:51.692+0000 7f3927400700 1 --2- 172.21.15.67:0/2696722408 >> [v2:172.21.15.67:6800/3580491071,v1:172.21.15.67:6801/3580491071] conn(0x7f3908038e00 0x7f390803b2c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f390c006fd0 tx=0x7f390c006e40 comp rx=0 tx=0).stop 2024-03-28T15:50:52.300 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:51.692+0000 7f3927400700 1 -- 172.21.15.67:0/2696722408 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3920106250 msgr2=0x7f392019da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:52.301 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:51.692+0000 7f3927400700 1 --2- 172.21.15.67:0/2696722408 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3920106250 0x7f392019da80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3914011a50 tx=0x7f3914011b30 comp rx=0 tx=0).stop 2024-03-28T15:50:52.301 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:51.692+0000 7f3927400700 1 -- 172.21.15.67:0/2696722408 shutdown_connections 2024-03-28T15:50:52.301 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:51.692+0000 7f3927400700 1 --2- 172.21.15.67:0/2696722408 >> [v2:172.21.15.67:6800/3580491071,v1:172.21.15.67:6801/3580491071] conn(0x7f3908038e00 0x7f390803b2c0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:52.301 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:51.692+0000 7f3927400700 1 --2- 172.21.15.67:0/2696722408 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3920106250 0x7f392019da80 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:52.301 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:51.692+0000 7f3927400700 1 -- 172.21.15.67:0/2696722408 >> 172.21.15.67:0/2696722408 conn(0x7f3920101e70 msgr2=0x7f3920194760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:52.301 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:51.693+0000 7f3927400700 1 -- 172.21.15.67:0/2696722408 shutdown_connections 2024-03-28T15:50:52.301 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:51.693+0000 7f3927400700 1 -- 172.21.15.67:0/2696722408 wait complete. 2024-03-28T15:50:52.301 INFO:teuthology.orchestra.run.smithi067.stdout:mgr is available 2024-03-28T15:50:52.907 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:52 smithi067 ceph-mon[31707]: mgrmap e4: x(active, since 2s) 2024-03-28T15:50:52.908 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:52 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/2696722408' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-28T15:50:53.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:53 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/276695443' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-28T15:50:53.864 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 2024-03-28T15:50:53.864 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout [global] 2024-03-28T15:50:53.865 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout fsid = b5f8225a-ed1a-11ee-95d0-87774f69a715 2024-03-28T15:50:53.865 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-28T15:50:53.865 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-28T15:50:53.865 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-28T15:50:53.865 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-28T15:50:53.865 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 2024-03-28T15:50:53.865 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-28T15:50:53.865 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-28T15:50:53.865 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 2024-03-28T15:50:53.865 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout [osd] 2024-03-28T15:50:53.865 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout bdev_async_discard = True 2024-03-28T15:50:53.865 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-28T15:50:53.865 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-28T15:50:53.865 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-28T15:50:53.865 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-28T15:50:53.866 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-28T15:50:53.866 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:52.906+0000 7fa57c200700 1 Processor -- start 2024-03-28T15:50:53.866 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:52.907+0000 7fa57c200700 1 -- start start 2024-03-28T15:50:53.866 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:52.907+0000 7fa57c200700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa57407cd10 0x7fa57407b170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:53.866 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:52.907+0000 7fa57c200700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa57407b740 con 0x7fa57407cd10 2024-03-28T15:50:53.866 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:52.908+0000 7fa579800700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa57407cd10 0x7fa57407b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:53.866 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:52.908+0000 7fa579800700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa57407cd10 0x7fa57407b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37862/0 (socket says 172.21.15.67:37862) 2024-03-28T15:50:53.866 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:52.908+0000 7fa579800700 1 -- 172.21.15.67:0/1832687227 learned_addr learned my addr 172.21.15.67:0/1832687227 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:50:53.866 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:52.908+0000 7fa579800700 1 -- 172.21.15.67:0/1832687227 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa57407b8c0 con 0x7fa57407cd10 2024-03-28T15:50:53.866 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:52.909+0000 7fa579800700 1 --2- 172.21.15.67:0/1832687227 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa57407cd10 0x7fa57407b170 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fa5680098a0 tx=0x7fa568009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b5de349f474fc4d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:53.866 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:52.909+0000 7fa573e00700 1 -- 172.21.15.67:0/1832687227 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa568004160 con 0x7fa57407cd10 2024-03-28T15:50:53.866 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:52.909+0000 7fa573e00700 1 -- 172.21.15.67:0/1832687227 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7fa568004300 con 0x7fa57407cd10 2024-03-28T15:50:53.866 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:52.909+0000 7fa573e00700 1 -- 172.21.15.67:0/1832687227 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa568011a80 con 0x7fa57407cd10 2024-03-28T15:50:53.867 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:52.910+0000 7fa57c200700 1 -- 172.21.15.67:0/1832687227 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa57407cd10 msgr2=0x7fa57407b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:53.867 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:52.910+0000 7fa57c200700 1 --2- 172.21.15.67:0/1832687227 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa57407cd10 0x7fa57407b170 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fa5680098a0 tx=0x7fa568009bb0 comp rx=0 tx=0).stop 2024-03-28T15:50:53.867 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:52.910+0000 7fa57c200700 1 -- 172.21.15.67:0/1832687227 shutdown_connections 2024-03-28T15:50:53.867 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:52.910+0000 7fa57c200700 1 --2- 172.21.15.67:0/1832687227 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa57407cd10 0x7fa57407b170 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:53.867 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:52.910+0000 7fa57c200700 1 -- 172.21.15.67:0/1832687227 >> 172.21.15.67:0/1832687227 conn(0x7fa574102f70 msgr2=0x7fa574105390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:53.867 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:52.910+0000 7fa57c200700 1 -- 172.21.15.67:0/1832687227 shutdown_connections 2024-03-28T15:50:53.867 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:52.910+0000 7fa57c200700 1 -- 172.21.15.67:0/1832687227 wait complete. 2024-03-28T15:50:53.867 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:52.911+0000 7fa57c200700 1 Processor -- start 2024-03-28T15:50:53.867 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:52.911+0000 7fa57c200700 1 -- start start 2024-03-28T15:50:53.867 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:52.912+0000 7fa57c200700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa57407cd10 0x7fa57419daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:53.867 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:52.912+0000 7fa57c200700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa57410df40 con 0x7fa57407cd10 2024-03-28T15:50:53.867 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:52.912+0000 7fa579800700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa57407cd10 0x7fa57419daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:53.867 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:52.912+0000 7fa579800700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa57407cd10 0x7fa57419daa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37876/0 (socket says 172.21.15.67:37876) 2024-03-28T15:50:53.867 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:52.912+0000 7fa579800700 1 -- 172.21.15.67:0/276695443 learned_addr learned my addr 172.21.15.67:0/276695443 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:50:53.868 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:52.912+0000 7fa579800700 1 -- 172.21.15.67:0/276695443 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa57419dfe0 con 0x7fa57407cd10 2024-03-28T15:50:53.868 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:52.913+0000 7fa579800700 1 --2- 172.21.15.67:0/276695443 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa57407cd10 0x7fa57419daa0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fa568011a50 tx=0x7fa568011b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:53.868 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:52.913+0000 7fa572000700 1 -- 172.21.15.67:0/276695443 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa568003a80 con 0x7fa57407cd10 2024-03-28T15:50:53.868 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:52.913+0000 7fa572000700 1 -- 172.21.15.67:0/276695443 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7fa568003c20 con 0x7fa57407cd10 2024-03-28T15:50:53.868 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:52.913+0000 7fa572000700 1 -- 172.21.15.67:0/276695443 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa56801a650 con 0x7fa57407cd10 2024-03-28T15:50:53.868 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:52.913+0000 7fa57c200700 1 -- 172.21.15.67:0/276695443 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa57419e270 con 0x7fa57407cd10 2024-03-28T15:50:53.868 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:52.913+0000 7fa57c200700 1 -- 172.21.15.67:0/276695443 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5741a0f60 con 0x7fa57407cd10 2024-03-28T15:50:53.868 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:52.914+0000 7fa572000700 1 -- 172.21.15.67:0/276695443 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 4) v1 ==== 46095+0+0 (secure 0 0 0) 0x7fa568024ad0 con 0x7fa57407cd10 2024-03-28T15:50:53.868 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:52.914+0000 7fa57c200700 1 -- 172.21.15.67:0/276695443 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5740637c0 con 0x7fa57407cd10 2024-03-28T15:50:53.868 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:52.915+0000 7fa572000700 1 --2- 172.21.15.67:0/276695443 >> [v2:172.21.15.67:6800/3580491071,v1:172.21.15.67:6801/3580491071] conn(0x7fa55c038ed0 0x7fa55c03b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:53.868 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:52.915+0000 7fa572000700 1 -- 172.21.15.67:0/276695443 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fa56804d220 con 0x7fa57407cd10 2024-03-28T15:50:53.868 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:52.915+0000 7fa578e00700 1 --2- 172.21.15.67:0/276695443 >> [v2:172.21.15.67:6800/3580491071,v1:172.21.15.67:6801/3580491071] conn(0x7fa55c038ed0 0x7fa55c03b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:53.868 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:52.915+0000 7fa578e00700 1 --2- 172.21.15.67:0/276695443 >> [v2:172.21.15.67:6800/3580491071,v1:172.21.15.67:6801/3580491071] conn(0x7fa55c038ed0 0x7fa55c03b390 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa560006fd0 tx=0x7fa560006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:53.868 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:52.920+0000 7fa572000700 1 -- 172.21.15.67:0/276695443 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa56802d3f0 con 0x7fa57407cd10 2024-03-28T15:50:53.868 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:53.146+0000 7fa57c200700 1 -- 172.21.15.67:0/276695443 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fa5741a1130 con 0x7fa57407cd10 2024-03-28T15:50:53.869 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:53.149+0000 7fa572000700 1 -- 172.21.15.67:0/276695443 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+400 (secure 0 0 0) 0x7fa5741a1130 con 0x7fa57407cd10 2024-03-28T15:50:53.869 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:53.152+0000 7fa57c200700 1 -- 172.21.15.67:0/276695443 >> [v2:172.21.15.67:6800/3580491071,v1:172.21.15.67:6801/3580491071] conn(0x7fa55c038ed0 msgr2=0x7fa55c03b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:53.869 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:53.152+0000 7fa57c200700 1 --2- 172.21.15.67:0/276695443 >> [v2:172.21.15.67:6800/3580491071,v1:172.21.15.67:6801/3580491071] conn(0x7fa55c038ed0 0x7fa55c03b390 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa560006fd0 tx=0x7fa560006e40 comp rx=0 tx=0).stop 2024-03-28T15:50:53.869 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:53.152+0000 7fa57c200700 1 -- 172.21.15.67:0/276695443 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa57407cd10 msgr2=0x7fa57419daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:53.869 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:53.152+0000 7fa57c200700 1 --2- 172.21.15.67:0/276695443 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa57407cd10 0x7fa57419daa0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fa568011a50 tx=0x7fa568011b30 comp rx=0 tx=0).stop 2024-03-28T15:50:53.869 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:53.153+0000 7fa57c200700 1 -- 172.21.15.67:0/276695443 shutdown_connections 2024-03-28T15:50:53.869 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:53.153+0000 7fa57c200700 1 --2- 172.21.15.67:0/276695443 >> [v2:172.21.15.67:6800/3580491071,v1:172.21.15.67:6801/3580491071] conn(0x7fa55c038ed0 0x7fa55c03b390 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:53.869 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:53.153+0000 7fa57c200700 1 --2- 172.21.15.67:0/276695443 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa57407cd10 0x7fa57419daa0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:53.869 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:53.153+0000 7fa57c200700 1 -- 172.21.15.67:0/276695443 >> 172.21.15.67:0/276695443 conn(0x7fa574102f70 msgr2=0x7fa574103aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:53.869 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:53.153+0000 7fa57c200700 1 -- 172.21.15.67:0/276695443 shutdown_connections 2024-03-28T15:50:53.870 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:53.153+0000 7fa57c200700 1 -- 172.21.15.67:0/276695443 wait complete. 2024-03-28T15:50:53.870 INFO:teuthology.orchestra.run.smithi067.stdout:Enabling cephadm module... 2024-03-28T15:50:55.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:54 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/160775448' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-28T15:50:55.069 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:50:54 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: ignoring --setuser ceph since I am not root 2024-03-28T15:50:55.070 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:50:54 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: ignoring --setgroup ceph since I am not root 2024-03-28T15:50:55.524 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:54.442+0000 7fb9e4a00700 1 Processor -- start 2024-03-28T15:50:55.524 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:54.443+0000 7fb9e4a00700 1 -- start start 2024-03-28T15:50:55.525 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:54.443+0000 7fb9e4a00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9e0107780 0x7fb9e0107ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:55.525 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:54.443+0000 7fb9e4a00700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e0108170 con 0x7fb9e0107780 2024-03-28T15:50:55.525 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:54.443+0000 7fb9df400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9e0107780 0x7fb9e0107ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:55.525 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:54.444+0000 7fb9df400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9e0107780 0x7fb9e0107ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37878/0 (socket says 172.21.15.67:37878) 2024-03-28T15:50:55.525 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:54.444+0000 7fb9df400700 1 -- 172.21.15.67:0/1962837149 learned_addr learned my addr 172.21.15.67:0/1962837149 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:50:55.525 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:54.444+0000 7fb9df400700 1 -- 172.21.15.67:0/1962837149 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9e0108980 con 0x7fb9e0107780 2024-03-28T15:50:55.525 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:54.444+0000 7fb9df400700 1 --2- 172.21.15.67:0/1962837149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9e0107780 0x7fb9e0107ba0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fb9d00098a0 tx=0x7fb9d0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1e5cb33036ba001 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:55.525 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:54.445+0000 7fb9dea00700 1 -- 172.21.15.67:0/1962837149 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb9d0004160 con 0x7fb9e0107780 2024-03-28T15:50:55.525 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:54.445+0000 7fb9dea00700 1 -- 172.21.15.67:0/1962837149 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7fb9d0004300 con 0x7fb9e0107780 2024-03-28T15:50:55.525 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:54.445+0000 7fb9dea00700 1 -- 172.21.15.67:0/1962837149 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb9d0011a80 con 0x7fb9e0107780 2024-03-28T15:50:55.525 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:54.445+0000 7fb9e4a00700 1 -- 172.21.15.67:0/1962837149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9e0107780 msgr2=0x7fb9e0107ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:55.525 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:54.445+0000 7fb9e4a00700 1 --2- 172.21.15.67:0/1962837149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9e0107780 0x7fb9e0107ba0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fb9d00098a0 tx=0x7fb9d0009bb0 comp rx=0 tx=0).stop 2024-03-28T15:50:55.525 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:54.446+0000 7fb9e4a00700 1 -- 172.21.15.67:0/1962837149 shutdown_connections 2024-03-28T15:50:55.530 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:54.446+0000 7fb9e4a00700 1 --2- 172.21.15.67:0/1962837149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9e0107780 0x7fb9e0107ba0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:55.530 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:54.446+0000 7fb9e4a00700 1 -- 172.21.15.67:0/1962837149 >> 172.21.15.67:0/1962837149 conn(0x7fb9e0102d20 msgr2=0x7fb9e0105160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:55.530 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:54.446+0000 7fb9e4a00700 1 -- 172.21.15.67:0/1962837149 shutdown_connections 2024-03-28T15:50:55.530 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:54.446+0000 7fb9e4a00700 1 -- 172.21.15.67:0/1962837149 wait complete. 2024-03-28T15:50:55.530 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:54.447+0000 7fb9e4a00700 1 Processor -- start 2024-03-28T15:50:55.530 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:54.447+0000 7fb9e4a00700 1 -- start start 2024-03-28T15:50:55.530 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:54.447+0000 7fb9e4a00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9e0107780 0x7fb9e019b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:55.530 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:54.447+0000 7fb9e4a00700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e010bd30 con 0x7fb9e0107780 2024-03-28T15:50:55.531 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:54.448+0000 7fb9df400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9e0107780 0x7fb9e019b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:55.531 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:54.448+0000 7fb9df400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9e0107780 0x7fb9e019b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37894/0 (socket says 172.21.15.67:37894) 2024-03-28T15:50:55.531 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:54.448+0000 7fb9df400700 1 -- 172.21.15.67:0/160775448 learned_addr learned my addr 172.21.15.67:0/160775448 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:50:55.531 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:54.448+0000 7fb9df400700 1 -- 172.21.15.67:0/160775448 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9e019bc70 con 0x7fb9e0107780 2024-03-28T15:50:55.531 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:54.448+0000 7fb9df400700 1 --2- 172.21.15.67:0/160775448 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9e0107780 0x7fb9e019b730 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb9d0000c00 tx=0x7fb9d0011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:55.531 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:54.449+0000 7fb9dcc00700 1 -- 172.21.15.67:0/160775448 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb9d00039f0 con 0x7fb9e0107780 2024-03-28T15:50:55.531 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:54.449+0000 7fb9dcc00700 1 -- 172.21.15.67:0/160775448 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7fb9d0003b90 con 0x7fb9e0107780 2024-03-28T15:50:55.531 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:54.449+0000 7fb9e4a00700 1 -- 172.21.15.67:0/160775448 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9e019bf00 con 0x7fb9e0107780 2024-03-28T15:50:55.531 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:54.449+0000 7fb9dcc00700 1 -- 172.21.15.67:0/160775448 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb9d001a740 con 0x7fb9e0107780 2024-03-28T15:50:55.531 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:54.449+0000 7fb9e4a00700 1 -- 172.21.15.67:0/160775448 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9e019ebf0 con 0x7fb9e0107780 2024-03-28T15:50:55.531 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:54.450+0000 7fb9dcc00700 1 -- 172.21.15.67:0/160775448 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 4) v1 ==== 46095+0+0 (secure 0 0 0) 0x7fb9d0024440 con 0x7fb9e0107780 2024-03-28T15:50:55.531 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:54.450+0000 7fb9e4a00700 1 -- 172.21.15.67:0/160775448 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9e00637c0 con 0x7fb9e0107780 2024-03-28T15:50:55.531 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:54.450+0000 7fb9dcc00700 1 --2- 172.21.15.67:0/160775448 >> [v2:172.21.15.67:6800/3580491071,v1:172.21.15.67:6801/3580491071] conn(0x7fb9c4038f20 0x7fb9c403b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:55.532 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:54.450+0000 7fb9dcc00700 1 -- 172.21.15.67:0/160775448 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fb9d004d0d0 con 0x7fb9e0107780 2024-03-28T15:50:55.532 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:54.451+0000 7fb9d6a00700 1 --2- 172.21.15.67:0/160775448 >> [v2:172.21.15.67:6800/3580491071,v1:172.21.15.67:6801/3580491071] conn(0x7fb9c4038f20 0x7fb9c403b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:55.532 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:54.451+0000 7fb9d6a00700 1 --2- 172.21.15.67:0/160775448 >> [v2:172.21.15.67:6800/3580491071,v1:172.21.15.67:6801/3580491071] conn(0x7fb9c4038f20 0x7fb9c403b3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb9c8006fd0 tx=0x7fb9c8006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:55.532 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:54.456+0000 7fb9dcc00700 1 -- 172.21.15.67:0/160775448 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb9d0020240 con 0x7fb9e0107780 2024-03-28T15:50:55.532 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:54.745+0000 7fb9e4a00700 1 -- 172.21.15.67:0/160775448 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7fb9e019edc0 con 0x7fb9e0107780 2024-03-28T15:50:55.532 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:54.799+0000 7fb9dcc00700 1 -- 172.21.15.67:0/160775448 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7fb9e019edc0 con 0x7fb9e0107780 2024-03-28T15:50:55.532 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:54.803+0000 7fb9e4a00700 1 -- 172.21.15.67:0/160775448 >> [v2:172.21.15.67:6800/3580491071,v1:172.21.15.67:6801/3580491071] conn(0x7fb9c4038f20 msgr2=0x7fb9c403b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:55.532 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:54.803+0000 7fb9e4a00700 1 --2- 172.21.15.67:0/160775448 >> [v2:172.21.15.67:6800/3580491071,v1:172.21.15.67:6801/3580491071] conn(0x7fb9c4038f20 0x7fb9c403b3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb9c8006fd0 tx=0x7fb9c8006e40 comp rx=0 tx=0).stop 2024-03-28T15:50:55.532 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:54.803+0000 7fb9e4a00700 1 -- 172.21.15.67:0/160775448 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9e0107780 msgr2=0x7fb9e019b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:55.532 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:54.804+0000 7fb9e4a00700 1 --2- 172.21.15.67:0/160775448 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9e0107780 0x7fb9e019b730 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb9d0000c00 tx=0x7fb9d0011b00 comp rx=0 tx=0).stop 2024-03-28T15:50:55.532 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:54.804+0000 7fb9e4a00700 1 -- 172.21.15.67:0/160775448 shutdown_connections 2024-03-28T15:50:55.532 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:54.804+0000 7fb9e4a00700 1 --2- 172.21.15.67:0/160775448 >> [v2:172.21.15.67:6800/3580491071,v1:172.21.15.67:6801/3580491071] conn(0x7fb9c4038f20 0x7fb9c403b3e0 secure :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb9c8006fd0 tx=0x7fb9c8006e40 comp rx=0 tx=0).stop 2024-03-28T15:50:55.532 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:54.804+0000 7fb9e4a00700 1 --2- 172.21.15.67:0/160775448 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9e0107780 0x7fb9e019b730 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:55.533 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:54.804+0000 7fb9e4a00700 1 -- 172.21.15.67:0/160775448 >> 172.21.15.67:0/160775448 conn(0x7fb9e0102d20 msgr2=0x7fb9e01038b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:55.533 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:54.805+0000 7fb9e4a00700 1 -- 172.21.15.67:0/160775448 shutdown_connections 2024-03-28T15:50:55.533 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:54.805+0000 7fb9e4a00700 1 -- 172.21.15.67:0/160775448 wait complete. 2024-03-28T15:50:55.807 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:50:55 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:50:55.614+0000 7f9cf936c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T15:50:55.808 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:50:55 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:50:55.738+0000 7f9cf936c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T15:50:56.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:55 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/160775448' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-28T15:50:56.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:55 smithi067 ceph-mon[31707]: mgrmap e5: x(active, since 5s) 2024-03-28T15:50:56.671 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:50:56 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:50:56.562+0000 7f9cf936c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T15:50:57.002 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:50:56 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/918854937' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-28T15:50:57.003 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:50:56 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:50:56.688+0000 7f9cf936c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T15:50:57.003 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:50:56 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:50:56.872+0000 7f9cf936c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T15:50:57.003 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:50:57 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:50:57.002+0000 7f9cf936c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T15:50:57.054 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout { 2024-03-28T15:50:57.054 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-28T15:50:57.054 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-28T15:50:57.054 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-28T15:50:57.054 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-28T15:50:57.054 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout } 2024-03-28T15:50:57.054 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:56.099+0000 7fadb0e00700 1 Processor -- start 2024-03-28T15:50:57.054 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:56.099+0000 7fadb0e00700 1 -- start start 2024-03-28T15:50:57.054 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:56.099+0000 7fadb0e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fadac107970 0x7fadac107d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:57.055 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:56.100+0000 7fadb0e00700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadac108360 con 0x7fadac107970 2024-03-28T15:50:57.055 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:56.100+0000 7fada9e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fadac107970 0x7fadac107d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:57.055 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:56.100+0000 7fada9e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fadac107970 0x7fadac107d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37914/0 (socket says 172.21.15.67:37914) 2024-03-28T15:50:57.055 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:56.100+0000 7fada9e00700 1 -- 172.21.15.67:0/4263530020 learned_addr learned my addr 172.21.15.67:0/4263530020 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:50:57.055 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:56.100+0000 7fada9e00700 1 -- 172.21.15.67:0/4263530020 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadac108b70 con 0x7fadac107970 2024-03-28T15:50:57.055 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:56.101+0000 7fada9e00700 1 --2- 172.21.15.67:0/4263530020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fadac107970 0x7fadac107d90 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fad9c0098a0 tx=0x7fad9c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5d8025c39a91e05 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:57.055 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:56.101+0000 7fada8a00700 1 -- 172.21.15.67:0/4263530020 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fad9c004160 con 0x7fadac107970 2024-03-28T15:50:57.055 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:56.101+0000 7fada8a00700 1 -- 172.21.15.67:0/4263530020 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7fad9c004300 con 0x7fadac107970 2024-03-28T15:50:57.055 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:56.101+0000 7fada8a00700 1 -- 172.21.15.67:0/4263530020 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fad9c011a80 con 0x7fadac107970 2024-03-28T15:50:57.055 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:56.101+0000 7fadb0e00700 1 -- 172.21.15.67:0/4263530020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fadac107970 msgr2=0x7fadac107d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:57.055 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:56.101+0000 7fadb0e00700 1 --2- 172.21.15.67:0/4263530020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fadac107970 0x7fadac107d90 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fad9c0098a0 tx=0x7fad9c009bb0 comp rx=0 tx=0).stop 2024-03-28T15:50:57.055 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:56.102+0000 7fadb0e00700 1 -- 172.21.15.67:0/4263530020 shutdown_connections 2024-03-28T15:50:57.055 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:56.102+0000 7fadb0e00700 1 --2- 172.21.15.67:0/4263530020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fadac107970 0x7fadac107d90 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:57.055 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:56.102+0000 7fadb0e00700 1 -- 172.21.15.67:0/4263530020 >> 172.21.15.67:0/4263530020 conn(0x7fadac102f70 msgr2=0x7fadac105390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:57.056 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:56.102+0000 7fadb0e00700 1 -- 172.21.15.67:0/4263530020 shutdown_connections 2024-03-28T15:50:57.056 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:56.102+0000 7fadb0e00700 1 -- 172.21.15.67:0/4263530020 wait complete. 2024-03-28T15:50:57.056 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:56.103+0000 7fadb0e00700 1 Processor -- start 2024-03-28T15:50:57.056 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:56.103+0000 7fadb0e00700 1 -- start start 2024-03-28T15:50:57.056 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:56.104+0000 7fadb0e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fadac107970 0x7fadac19b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:57.056 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:56.104+0000 7fadb0e00700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadac10bf20 con 0x7fadac107970 2024-03-28T15:50:57.056 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:56.104+0000 7fada9e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fadac107970 0x7fadac19b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:57.056 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:56.104+0000 7fada9e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fadac107970 0x7fadac19b920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37916/0 (socket says 172.21.15.67:37916) 2024-03-28T15:50:57.056 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:56.104+0000 7fada9e00700 1 -- 172.21.15.67:0/918854937 learned_addr learned my addr 172.21.15.67:0/918854937 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:50:57.056 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:56.104+0000 7fada9e00700 1 -- 172.21.15.67:0/918854937 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadac19be60 con 0x7fadac107970 2024-03-28T15:50:57.056 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:56.105+0000 7fada9e00700 1 --2- 172.21.15.67:0/918854937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fadac107970 0x7fadac19b920 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fad9c000c00 tx=0x7fad9c011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:57.056 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:56.105+0000 7fada2a00700 1 -- 172.21.15.67:0/918854937 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fad9c0039f0 con 0x7fadac107970 2024-03-28T15:50:57.056 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:56.105+0000 7fada2a00700 1 -- 172.21.15.67:0/918854937 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7fad9c003b90 con 0x7fadac107970 2024-03-28T15:50:57.056 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:56.105+0000 7fadb0e00700 1 -- 172.21.15.67:0/918854937 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadac19c0f0 con 0x7fadac107970 2024-03-28T15:50:57.057 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:56.105+0000 7fadb0e00700 1 -- 172.21.15.67:0/918854937 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadac19ede0 con 0x7fadac107970 2024-03-28T15:50:57.057 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:56.105+0000 7fada2a00700 1 -- 172.21.15.67:0/918854937 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fad9c01a740 con 0x7fadac107970 2024-03-28T15:50:57.057 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:56.106+0000 7fadb0e00700 1 -- 172.21.15.67:0/918854937 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadac19c5d0 con 0x7fadac107970 2024-03-28T15:50:57.057 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:56.106+0000 7fada2a00700 1 -- 172.21.15.67:0/918854937 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 46106+0+0 (secure 0 0 0) 0x7fad9c01aa40 con 0x7fadac107970 2024-03-28T15:50:57.057 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:56.107+0000 7fada2a00700 1 --2- 172.21.15.67:0/918854937 >> [v2:172.21.15.67:6800/3580491071,v1:172.21.15.67:6801/3580491071] conn(0x7fad90038f20 0x7fad9003b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:57.057 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:56.107+0000 7fada2a00700 1 -- 172.21.15.67:0/918854937 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fad9c04d220 con 0x7fadac107970 2024-03-28T15:50:57.057 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:56.107+0000 7fada9400700 1 -- 172.21.15.67:0/918854937 >> [v2:172.21.15.67:6800/3580491071,v1:172.21.15.67:6801/3580491071] conn(0x7fad90038f20 msgr2=0x7fad9003b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/3580491071 2024-03-28T15:50:57.057 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:56.107+0000 7fada9400700 1 --2- 172.21.15.67:0/918854937 >> [v2:172.21.15.67:6800/3580491071,v1:172.21.15.67:6801/3580491071] conn(0x7fad90038f20 0x7fad9003b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T15:50:57.057 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:56.113+0000 7fada2a00700 1 -- 172.21.15.67:0/918854937 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fadac19c5d0 con 0x7fadac107970 2024-03-28T15:50:57.057 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:56.308+0000 7fada9400700 1 -- 172.21.15.67:0/918854937 >> [v2:172.21.15.67:6800/3580491071,v1:172.21.15.67:6801/3580491071] conn(0x7fad90038f20 msgr2=0x7fad9003b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/3580491071 2024-03-28T15:50:57.057 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:56.308+0000 7fada9400700 1 --2- 172.21.15.67:0/918854937 >> [v2:172.21.15.67:6800/3580491071,v1:172.21.15.67:6801/3580491071] conn(0x7fad90038f20 0x7fad9003b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T15:50:57.057 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:56.388+0000 7fadb0e00700 1 -- 172.21.15.67:0/918854937 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fadac19efb0 con 0x7fadac107970 2024-03-28T15:50:57.057 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:56.389+0000 7fada2a00700 1 -- 172.21.15.67:0/918854937 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7fadac19efb0 con 0x7fadac107970 2024-03-28T15:50:57.057 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:56.393+0000 7fadb0e00700 1 -- 172.21.15.67:0/918854937 >> [v2:172.21.15.67:6800/3580491071,v1:172.21.15.67:6801/3580491071] conn(0x7fad90038f20 msgr2=0x7fad9003b3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T15:50:57.057 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:56.393+0000 7fadb0e00700 1 --2- 172.21.15.67:0/918854937 >> [v2:172.21.15.67:6800/3580491071,v1:172.21.15.67:6801/3580491071] conn(0x7fad90038f20 0x7fad9003b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:57.057 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:56.393+0000 7fadb0e00700 1 -- 172.21.15.67:0/918854937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fadac107970 msgr2=0x7fadac19b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:57.058 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:56.393+0000 7fadb0e00700 1 --2- 172.21.15.67:0/918854937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fadac107970 0x7fadac19b920 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fad9c000c00 tx=0x7fad9c011b00 comp rx=0 tx=0).stop 2024-03-28T15:50:57.058 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:56.393+0000 7fadb0e00700 1 -- 172.21.15.67:0/918854937 shutdown_connections 2024-03-28T15:50:57.058 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:56.393+0000 7fadb0e00700 1 --2- 172.21.15.67:0/918854937 >> [v2:172.21.15.67:6800/3580491071,v1:172.21.15.67:6801/3580491071] conn(0x7fad90038f20 0x7fad9003b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:57.058 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:56.393+0000 7fadb0e00700 1 --2- 172.21.15.67:0/918854937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fadac107970 0x7fadac19b920 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:57.058 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:56.393+0000 7fadb0e00700 1 -- 172.21.15.67:0/918854937 >> 172.21.15.67:0/918854937 conn(0x7fadac102f70 msgr2=0x7fadac103ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:57.058 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:56.393+0000 7fadb0e00700 1 -- 172.21.15.67:0/918854937 shutdown_connections 2024-03-28T15:50:57.058 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:56.394+0000 7fadb0e00700 1 -- 172.21.15.67:0/918854937 wait complete. 2024-03-28T15:50:57.058 INFO:teuthology.orchestra.run.smithi067.stdout:Waiting for the mgr to restart... 2024-03-28T15:50:57.058 INFO:teuthology.orchestra.run.smithi067.stdout:Waiting for mgr epoch 5... 2024-03-28T15:50:57.319 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:50:57 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:50:57.127+0000 7f9cf936c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T15:50:57.621 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:50:57 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:50:57.473+0000 7f9cf936c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T15:50:57.906 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:50:57 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:50:57.621+0000 7f9cf936c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T15:50:57.906 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:50:57 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:50:57.738+0000 7f9cf936c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T15:50:57.906 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:50:57 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:50:57.864+0000 7f9cf936c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T15:50:58.306 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:50:58 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:50:58.088+0000 7f9cf936c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T15:50:59.819 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:50:59 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:50:59.398+0000 7f9cf936c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T15:50:59.819 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:50:59 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:50:59.574+0000 7f9cf936c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T15:51:00.319 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:51:00 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:51:00.001+0000 7f9cf936c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T15:51:00.319 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:51:00 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:51:00.122+0000 7f9cf936c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T15:51:00.641 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:51:00 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:51:00.388+0000 7f9cf936c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T15:51:01.026 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:51:00 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:51:00.640+0000 7f9cf936c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T15:51:01.319 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:51:01 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:51:01.025+0000 7f9cf936c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T15:51:01.661 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:51:01 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:51:01.346+0000 7f9cf936c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T15:51:02.069 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:51:01 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:51:01.659+0000 7f9cf936c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T15:51:03.070 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:51:02 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:51:02.720+0000 7f9cf936c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T15:51:03.819 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:51:03 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:51:03.436+0000 7f9cf936c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T15:51:03.819 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:51:03 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:51:03.569+0000 7f9cf936c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T15:51:04.319 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:51:03 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:51:03.856+0000 7f9cf936c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T15:51:04.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:03 smithi067 ceph-mon[31707]: Active manager daemon x restarted 2024-03-28T15:51:04.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:03 smithi067 ceph-mon[31707]: Activating manager daemon x 2024-03-28T15:51:04.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:03 smithi067 ceph-mon[31707]: osdmap e2: 0 total, 0 up, 0 in 2024-03-28T15:51:04.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:03 smithi067 ceph-mon[31707]: mgrmap e6: x(active, starting, since 0.00792422s) 2024-03-28T15:51:04.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:03 smithi067 ceph-mon[31707]: from='mgr.14120 172.21.15.67:0/2156043519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T15:51:04.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:03 smithi067 ceph-mon[31707]: from='mgr.14120 172.21.15.67:0/2156043519' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-28T15:51:04.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:03 smithi067 ceph-mon[31707]: from='mgr.14120 172.21.15.67:0/2156043519' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-28T15:51:04.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:03 smithi067 ceph-mon[31707]: from='mgr.14120 172.21.15.67:0/2156043519' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-28T15:51:04.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:03 smithi067 ceph-mon[31707]: from='mgr.14120 172.21.15.67:0/2156043519' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-28T15:51:04.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:03 smithi067 ceph-mon[31707]: Manager daemon x is now available 2024-03-28T15:51:04.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:03 smithi067 ceph-mon[31707]: from='mgr.14120 172.21.15.67:0/2156043519' entity='mgr.x' 2024-03-28T15:51:05.253 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:04 smithi067 ceph-mon[31707]: Found migration_current of "None". Setting to last migration. 2024-03-28T15:51:05.253 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:04 smithi067 ceph-mon[31707]: from='mgr.14120 172.21.15.67:0/2156043519' entity='mgr.x' 2024-03-28T15:51:05.253 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:04 smithi067 ceph-mon[31707]: from='mgr.14120 172.21.15.67:0/2156043519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:51:05.253 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:04 smithi067 ceph-mon[31707]: from='mgr.14120 172.21.15.67:0/2156043519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:51:05.254 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:04 smithi067 ceph-mon[31707]: from='mgr.14120 172.21.15.67:0/2156043519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:51:05.254 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:04 smithi067 ceph-mon[31707]: from='mgr.14120 172.21.15.67:0/2156043519' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-28T15:51:05.254 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:04 smithi067 ceph-mon[31707]: from='mgr.14120 172.21.15.67:0/2156043519' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-28T15:51:05.254 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:04 smithi067 ceph-mon[31707]: from='mgr.14120 172.21.15.67:0/2156043519' entity='mgr.x' 2024-03-28T15:51:05.254 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:04 smithi067 ceph-mon[31707]: from='mgr.14120 172.21.15.67:0/2156043519' entity='mgr.x' 2024-03-28T15:51:05.254 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:04 smithi067 ceph-mon[31707]: mgrmap e7: x(active, since 1.01527s) 2024-03-28T15:51:05.568 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout { 2024-03-28T15:51:05.568 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-28T15:51:05.568 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-28T15:51:05.568 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout } 2024-03-28T15:51:05.569 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:57.696+0000 7fea78a00700 1 Processor -- start 2024-03-28T15:51:05.569 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:57.696+0000 7fea78a00700 1 -- start start 2024-03-28T15:51:05.569 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:57.697+0000 7fea78a00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea741061f0 0x7fea74106610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:05.569 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:57.697+0000 7fea78a00700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea740634b0 con 0x7fea741061f0 2024-03-28T15:51:05.569 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:57.697+0000 7fea71e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea741061f0 0x7fea74106610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:05.569 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:57.698+0000 7fea71e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea741061f0 0x7fea74106610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37920/0 (socket says 172.21.15.67:37920) 2024-03-28T15:51:05.569 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:57.698+0000 7fea71e00700 1 -- 172.21.15.67:0/2500049509 learned_addr learned my addr 172.21.15.67:0/2500049509 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:51:05.569 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:57.698+0000 7fea71e00700 1 -- 172.21.15.67:0/2500049509 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea74106b50 con 0x7fea741061f0 2024-03-28T15:51:05.569 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:57.698+0000 7fea71e00700 1 --2- 172.21.15.67:0/2500049509 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea741061f0 0x7fea74106610 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fea640098a0 tx=0x7fea64009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c756b27b7c682729 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:05.569 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:57.699+0000 7fea70a00700 1 -- 172.21.15.67:0/2500049509 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fea64004160 con 0x7fea741061f0 2024-03-28T15:51:05.569 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:57.699+0000 7fea70a00700 1 -- 172.21.15.67:0/2500049509 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7fea64004300 con 0x7fea741061f0 2024-03-28T15:51:05.569 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:57.699+0000 7fea70a00700 1 -- 172.21.15.67:0/2500049509 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fea64011a80 con 0x7fea741061f0 2024-03-28T15:51:05.569 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:57.699+0000 7fea78a00700 1 -- 172.21.15.67:0/2500049509 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea741061f0 msgr2=0x7fea74106610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:05.569 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:57.699+0000 7fea78a00700 1 --2- 172.21.15.67:0/2500049509 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea741061f0 0x7fea74106610 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fea640098a0 tx=0x7fea64009bb0 comp rx=0 tx=0).stop 2024-03-28T15:51:05.569 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:57.699+0000 7fea78a00700 1 -- 172.21.15.67:0/2500049509 shutdown_connections 2024-03-28T15:51:05.569 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:57.699+0000 7fea78a00700 1 --2- 172.21.15.67:0/2500049509 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea741061f0 0x7fea74106610 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:05.570 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:57.699+0000 7fea78a00700 1 -- 172.21.15.67:0/2500049509 >> 172.21.15.67:0/2500049509 conn(0x7fea74101e70 msgr2=0x7fea74104290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:05.570 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:57.700+0000 7fea78a00700 1 -- 172.21.15.67:0/2500049509 shutdown_connections 2024-03-28T15:51:05.570 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:57.700+0000 7fea78a00700 1 -- 172.21.15.67:0/2500049509 wait complete. 2024-03-28T15:51:05.570 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:57.701+0000 7fea78a00700 1 Processor -- start 2024-03-28T15:51:05.570 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:57.701+0000 7fea78a00700 1 -- start start 2024-03-28T15:51:05.570 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:57.701+0000 7fea78a00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea741061f0 0x7fea7419db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:05.570 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:57.701+0000 7fea78a00700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea74107b80 con 0x7fea741061f0 2024-03-28T15:51:05.570 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:57.701+0000 7fea71e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea741061f0 0x7fea7419db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:05.570 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:57.701+0000 7fea71e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea741061f0 0x7fea7419db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37926/0 (socket says 172.21.15.67:37926) 2024-03-28T15:51:05.570 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:57.701+0000 7fea71e00700 1 -- 172.21.15.67:0/3091992222 learned_addr learned my addr 172.21.15.67:0/3091992222 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:51:05.570 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:57.702+0000 7fea71e00700 1 -- 172.21.15.67:0/3091992222 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea7419e040 con 0x7fea741061f0 2024-03-28T15:51:05.570 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:57.702+0000 7fea71e00700 1 --2- 172.21.15.67:0/3091992222 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea741061f0 0x7fea7419db00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fea64011a50 tx=0x7fea64004290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:05.570 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:57.702+0000 7fea6aa00700 1 -- 172.21.15.67:0/3091992222 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fea640039f0 con 0x7fea741061f0 2024-03-28T15:51:05.571 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:57.702+0000 7fea78a00700 1 -- 172.21.15.67:0/3091992222 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea7419e2d0 con 0x7fea741061f0 2024-03-28T15:51:05.571 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:57.702+0000 7fea6aa00700 1 -- 172.21.15.67:0/3091992222 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7fea64003b90 con 0x7fea741061f0 2024-03-28T15:51:05.571 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:57.702+0000 7fea6aa00700 1 -- 172.21.15.67:0/3091992222 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fea6401a600 con 0x7fea741061f0 2024-03-28T15:51:05.571 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:57.702+0000 7fea78a00700 1 -- 172.21.15.67:0/3091992222 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea741a0fc0 con 0x7fea741061f0 2024-03-28T15:51:05.571 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:57.703+0000 7fea6aa00700 1 -- 172.21.15.67:0/3091992222 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 46106+0+0 (secure 0 0 0) 0x7fea6401a900 con 0x7fea741061f0 2024-03-28T15:51:05.571 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:57.704+0000 7fea6aa00700 1 --2- 172.21.15.67:0/3091992222 >> [v2:172.21.15.67:6800/3580491071,v1:172.21.15.67:6801/3580491071] conn(0x7fea58038f20 0x7fea5803b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:05.571 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:57.704+0000 7fea6aa00700 1 -- 172.21.15.67:0/3091992222 --> [v2:172.21.15.67:6800/3580491071,v1:172.21.15.67:6801/3580491071] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fea5803baf0 con 0x7fea58038f20 2024-03-28T15:51:05.571 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:57.704+0000 7fea6aa00700 1 -- 172.21.15.67:0/3091992222 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fea6404d290 con 0x7fea741061f0 2024-03-28T15:51:05.571 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:57.704+0000 7fea71400700 1 -- 172.21.15.67:0/3091992222 >> [v2:172.21.15.67:6800/3580491071,v1:172.21.15.67:6801/3580491071] conn(0x7fea58038f20 msgr2=0x7fea5803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/3580491071 2024-03-28T15:51:05.571 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:57.704+0000 7fea71400700 1 --2- 172.21.15.67:0/3091992222 >> [v2:172.21.15.67:6800/3580491071,v1:172.21.15.67:6801/3580491071] conn(0x7fea58038f20 0x7fea5803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T15:51:05.571 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:57.905+0000 7fea71400700 1 -- 172.21.15.67:0/3091992222 >> [v2:172.21.15.67:6800/3580491071,v1:172.21.15.67:6801/3580491071] conn(0x7fea58038f20 msgr2=0x7fea5803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/3580491071 2024-03-28T15:51:05.571 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:57.905+0000 7fea71400700 1 --2- 172.21.15.67:0/3091992222 >> [v2:172.21.15.67:6800/3580491071,v1:172.21.15.67:6801/3580491071] conn(0x7fea58038f20 0x7fea5803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T15:51:05.571 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:58.306+0000 7fea71400700 1 -- 172.21.15.67:0/3091992222 >> [v2:172.21.15.67:6800/3580491071,v1:172.21.15.67:6801/3580491071] conn(0x7fea58038f20 msgr2=0x7fea5803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/3580491071 2024-03-28T15:51:05.571 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:58.306+0000 7fea71400700 1 --2- 172.21.15.67:0/3091992222 >> [v2:172.21.15.67:6800/3580491071,v1:172.21.15.67:6801/3580491071] conn(0x7fea58038f20 0x7fea5803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T15:51:05.571 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:59.107+0000 7fea71400700 1 -- 172.21.15.67:0/3091992222 >> [v2:172.21.15.67:6800/3580491071,v1:172.21.15.67:6801/3580491071] conn(0x7fea58038f20 msgr2=0x7fea5803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/3580491071 2024-03-28T15:51:05.572 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:50:59.107+0000 7fea71400700 1 --2- 172.21.15.67:0/3091992222 >> [v2:172.21.15.67:6800/3580491071,v1:172.21.15.67:6801/3580491071] conn(0x7fea58038f20 0x7fea5803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T15:51:05.572 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:00.709+0000 7fea71400700 1 -- 172.21.15.67:0/3091992222 >> [v2:172.21.15.67:6800/3580491071,v1:172.21.15.67:6801/3580491071] conn(0x7fea58038f20 msgr2=0x7fea5803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/3580491071 2024-03-28T15:51:05.572 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:00.709+0000 7fea71400700 1 --2- 172.21.15.67:0/3091992222 >> [v2:172.21.15.67:6800/3580491071,v1:172.21.15.67:6801/3580491071] conn(0x7fea58038f20 0x7fea5803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-28T15:51:05.572 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:03.864+0000 7fea6aa00700 1 -- 172.21.15.67:0/3091992222 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mgrmap(e 6) v1 ==== 45873+0+0 (secure 0 0 0) 0x7fea64024440 con 0x7fea741061f0 2024-03-28T15:51:05.572 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:03.864+0000 7fea6aa00700 1 -- 172.21.15.67:0/3091992222 >> [v2:172.21.15.67:6800/3580491071,v1:172.21.15.67:6801/3580491071] conn(0x7fea58038f20 msgr2=0x7fea5803b3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T15:51:05.572 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:03.865+0000 7fea6aa00700 1 --2- 172.21.15.67:0/3091992222 >> [v2:172.21.15.67:6800/3580491071,v1:172.21.15.67:6801/3580491071] conn(0x7fea58038f20 0x7fea5803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:05.572 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:04.868+0000 7fea6aa00700 1 -- 172.21.15.67:0/3091992222 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 7) v1 ==== 46000+0+0 (secure 0 0 0) 0x7fea6404f2a0 con 0x7fea741061f0 2024-03-28T15:51:05.572 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:04.869+0000 7fea6aa00700 1 --2- 172.21.15.67:0/3091992222 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7fea5803c810 0x7fea5803ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:05.572 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:04.869+0000 7fea6aa00700 1 -- 172.21.15.67:0/3091992222 --> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fea6400af30 con 0x7fea5803c810 2024-03-28T15:51:05.572 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:04.871+0000 7fea71400700 1 --2- 172.21.15.67:0/3091992222 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7fea5803c810 0x7fea5803ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:05.572 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:04.874+0000 7fea71400700 1 --2- 172.21.15.67:0/3091992222 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7fea5803c810 0x7fea5803ec00 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fea60003a10 tx=0x7fea60009290 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:05.572 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:04.875+0000 7fea6aa00700 1 -- 172.21.15.67:0/3091992222 <== mgr.14120 v2:172.21.15.67:6800/1631001968 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7fea6400af30 con 0x7fea5803c810 2024-03-28T15:51:05.572 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:04.884+0000 7fea5fe00700 1 -- 172.21.15.67:0/3091992222 --> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fea4c002600 con 0x7fea5803c810 2024-03-28T15:51:05.572 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:04.884+0000 7fea6aa00700 1 -- 172.21.15.67:0/3091992222 <== mgr.14120 v2:172.21.15.67:6800/1631001968 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fea4c002600 con 0x7fea5803c810 2024-03-28T15:51:05.572 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:04.885+0000 7fea5fe00700 1 -- 172.21.15.67:0/3091992222 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7fea5803c810 msgr2=0x7fea5803ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:05.572 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:04.885+0000 7fea5fe00700 1 --2- 172.21.15.67:0/3091992222 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7fea5803c810 0x7fea5803ec00 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fea60003a10 tx=0x7fea60009290 comp rx=0 tx=0).stop 2024-03-28T15:51:05.573 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:04.885+0000 7fea5fe00700 1 -- 172.21.15.67:0/3091992222 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea741061f0 msgr2=0x7fea7419db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:05.573 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:04.885+0000 7fea5fe00700 1 --2- 172.21.15.67:0/3091992222 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea741061f0 0x7fea7419db00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fea64011a50 tx=0x7fea64004290 comp rx=0 tx=0).stop 2024-03-28T15:51:05.573 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:04.885+0000 7fea5fe00700 1 -- 172.21.15.67:0/3091992222 shutdown_connections 2024-03-28T15:51:05.573 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:04.885+0000 7fea5fe00700 1 --2- 172.21.15.67:0/3091992222 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7fea5803c810 0x7fea5803ec00 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:05.573 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:04.885+0000 7fea5fe00700 1 --2- 172.21.15.67:0/3091992222 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea741061f0 0x7fea7419db00 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:05.573 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:04.885+0000 7fea5fe00700 1 --2- 172.21.15.67:0/3091992222 >> [v2:172.21.15.67:6800/3580491071,v1:172.21.15.67:6801/3580491071] conn(0x7fea58038f20 0x7fea5803b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:05.573 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:04.885+0000 7fea5fe00700 1 -- 172.21.15.67:0/3091992222 >> 172.21.15.67:0/3091992222 conn(0x7fea74101e70 msgr2=0x7fea741947a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:05.573 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:04.885+0000 7fea5fe00700 1 -- 172.21.15.67:0/3091992222 shutdown_connections 2024-03-28T15:51:05.573 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:04.885+0000 7fea5fe00700 1 -- 172.21.15.67:0/3091992222 wait complete. 2024-03-28T15:51:05.573 INFO:teuthology.orchestra.run.smithi067.stdout:mgr epoch 5 is available 2024-03-28T15:51:05.573 INFO:teuthology.orchestra.run.smithi067.stdout:Setting orchestrator backend to cephadm... 2024-03-28T15:51:06.156 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:05 smithi067 ceph-mon[31707]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-28T15:51:06.156 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:05 smithi067 ceph-mon[31707]: [28/Mar/2024:15:51:04] ENGINE Bus STARTING 2024-03-28T15:51:06.156 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:05 smithi067 ceph-mon[31707]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-28T15:51:06.156 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:05 smithi067 ceph-mon[31707]: [28/Mar/2024:15:51:04] ENGINE Serving on http://172.21.15.67:8765 2024-03-28T15:51:06.156 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:05 smithi067 ceph-mon[31707]: [28/Mar/2024:15:51:05] ENGINE Serving on https://172.21.15.67:7150 2024-03-28T15:51:06.156 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:05 smithi067 ceph-mon[31707]: [28/Mar/2024:15:51:05] ENGINE Bus STARTED 2024-03-28T15:51:06.156 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:05 smithi067 ceph-mon[31707]: from='mgr.14120 172.21.15.67:0/2156043519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:51:07.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:06 smithi067 ceph-mon[31707]: mgrmap e8: x(active, since 2s) 2024-03-28T15:51:07.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:06 smithi067 ceph-mon[31707]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:51:07.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:06 smithi067 ceph-mon[31707]: from='mgr.14120 172.21.15.67:0/2156043519' entity='mgr.x' 2024-03-28T15:51:07.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:06 smithi067 ceph-mon[31707]: from='mgr.14120 172.21.15.67:0/2156043519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:51:07.142 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:06.155+0000 7ff663e00700 1 Processor -- start 2024-03-28T15:51:07.148 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:06.155+0000 7ff663e00700 1 -- start start 2024-03-28T15:51:07.148 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:06.156+0000 7ff663e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6641066a0 0x7ff664106ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:07.148 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:06.156+0000 7ff663e00700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff664107090 con 0x7ff6641066a0 2024-03-28T15:51:07.149 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:06.156+0000 7ff662a00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6641066a0 0x7ff664106ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:07.149 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:06.156+0000 7ff662a00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6641066a0 0x7ff664106ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49952/0 (socket says 172.21.15.67:49952) 2024-03-28T15:51:07.149 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:06.156+0000 7ff662a00700 1 -- 172.21.15.67:0/3483165828 learned_addr learned my addr 172.21.15.67:0/3483165828 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:51:07.149 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:06.157+0000 7ff662a00700 1 -- 172.21.15.67:0/3483165828 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6641078a0 con 0x7ff6641066a0 2024-03-28T15:51:07.149 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:06.157+0000 7ff662a00700 1 --2- 172.21.15.67:0/3483165828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6641066a0 0x7ff664106ac0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff6540098a0 tx=0x7ff654009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d33ef8c0c98213f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:07.149 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:06.157+0000 7ff661600700 1 -- 172.21.15.67:0/3483165828 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff654004160 con 0x7ff6641066a0 2024-03-28T15:51:07.149 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:06.157+0000 7ff661600700 1 -- 172.21.15.67:0/3483165828 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7ff654004300 con 0x7ff6641066a0 2024-03-28T15:51:07.149 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:06.157+0000 7ff661600700 1 -- 172.21.15.67:0/3483165828 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff654011a80 con 0x7ff6641066a0 2024-03-28T15:51:07.149 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:06.158+0000 7ff663e00700 1 -- 172.21.15.67:0/3483165828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6641066a0 msgr2=0x7ff664106ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:07.149 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:06.158+0000 7ff663e00700 1 --2- 172.21.15.67:0/3483165828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6641066a0 0x7ff664106ac0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff6540098a0 tx=0x7ff654009bb0 comp rx=0 tx=0).stop 2024-03-28T15:51:07.149 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:06.158+0000 7ff663e00700 1 -- 172.21.15.67:0/3483165828 shutdown_connections 2024-03-28T15:51:07.149 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:06.159+0000 7ff663e00700 1 --2- 172.21.15.67:0/3483165828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6641066a0 0x7ff664106ac0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:07.149 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:06.159+0000 7ff663e00700 1 -- 172.21.15.67:0/3483165828 >> 172.21.15.67:0/3483165828 conn(0x7ff664101cc0 msgr2=0x7ff6641040e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:07.149 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:06.159+0000 7ff663e00700 1 -- 172.21.15.67:0/3483165828 shutdown_connections 2024-03-28T15:51:07.150 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:06.159+0000 7ff663e00700 1 -- 172.21.15.67:0/3483165828 wait complete. 2024-03-28T15:51:07.150 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:06.160+0000 7ff663e00700 1 Processor -- start 2024-03-28T15:51:07.150 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:06.160+0000 7ff663e00700 1 -- start start 2024-03-28T15:51:07.150 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:06.160+0000 7ff663e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6641066a0 0x7ff664191810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:07.150 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:06.160+0000 7ff663e00700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff66410ac50 con 0x7ff6641066a0 2024-03-28T15:51:07.150 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:06.161+0000 7ff662a00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6641066a0 0x7ff664191810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:07.150 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:06.161+0000 7ff662a00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6641066a0 0x7ff664191810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49968/0 (socket says 172.21.15.67:49968) 2024-03-28T15:51:07.150 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:06.161+0000 7ff662a00700 1 -- 172.21.15.67:0/1459917613 learned_addr learned my addr 172.21.15.67:0/1459917613 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:51:07.150 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:06.161+0000 7ff662a00700 1 -- 172.21.15.67:0/1459917613 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff664191d50 con 0x7ff6641066a0 2024-03-28T15:51:07.150 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:06.161+0000 7ff662a00700 1 --2- 172.21.15.67:0/1459917613 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6641066a0 0x7ff664191810 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff654000c00 tx=0x7ff654011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:07.150 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:06.162+0000 7ff65b400700 1 -- 172.21.15.67:0/1459917613 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff6540039f0 con 0x7ff6641066a0 2024-03-28T15:51:07.150 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:06.162+0000 7ff65b400700 1 -- 172.21.15.67:0/1459917613 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7ff654003b90 con 0x7ff6641066a0 2024-03-28T15:51:07.150 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:06.162+0000 7ff65b400700 1 -- 172.21.15.67:0/1459917613 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff65401a740 con 0x7ff6641066a0 2024-03-28T15:51:07.150 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:06.162+0000 7ff663e00700 1 -- 172.21.15.67:0/1459917613 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff66418ff40 con 0x7ff6641066a0 2024-03-28T15:51:07.151 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:06.162+0000 7ff663e00700 1 -- 172.21.15.67:0/1459917613 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff664190480 con 0x7ff6641066a0 2024-03-28T15:51:07.151 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:06.163+0000 7ff663e00700 1 -- 172.21.15.67:0/1459917613 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6640637c0 con 0x7ff6641066a0 2024-03-28T15:51:07.151 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:06.163+0000 7ff65b400700 1 -- 172.21.15.67:0/1459917613 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7ff654024af0 con 0x7ff6641066a0 2024-03-28T15:51:07.151 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:06.163+0000 7ff65b400700 1 --2- 172.21.15.67:0/1459917613 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7ff648038f20 0x7ff64803b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:07.151 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:06.163+0000 7ff65b400700 1 -- 172.21.15.67:0/1459917613 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ff65404d210 con 0x7ff6641066a0 2024-03-28T15:51:07.151 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:06.164+0000 7ff662000700 1 --2- 172.21.15.67:0/1459917613 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7ff648038f20 0x7ff64803b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:07.151 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:06.164+0000 7ff662000700 1 --2- 172.21.15.67:0/1459917613 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7ff648038f20 0x7ff64803b3e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff650006fd0 tx=0x7ff650006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:07.151 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:06.169+0000 7ff65b400700 1 -- 172.21.15.67:0/1459917613 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff65400ba20 con 0x7ff6641066a0 2024-03-28T15:51:07.151 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:06.411+0000 7ff663e00700 1 -- 172.21.15.67:0/1459917613 --> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7ff664195620 con 0x7ff648038f20 2024-03-28T15:51:07.151 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:06.425+0000 7ff65b400700 1 -- 172.21.15.67:0/1459917613 <== mgr.14120 v2:172.21.15.67:6800/1631001968 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ff664195620 con 0x7ff648038f20 2024-03-28T15:51:07.151 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:06.428+0000 7ff663e00700 1 -- 172.21.15.67:0/1459917613 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7ff648038f20 msgr2=0x7ff64803b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:07.151 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:06.428+0000 7ff663e00700 1 --2- 172.21.15.67:0/1459917613 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7ff648038f20 0x7ff64803b3e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff650006fd0 tx=0x7ff650006e40 comp rx=0 tx=0).stop 2024-03-28T15:51:07.151 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:06.428+0000 7ff663e00700 1 -- 172.21.15.67:0/1459917613 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6641066a0 msgr2=0x7ff664191810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:07.151 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:06.428+0000 7ff663e00700 1 --2- 172.21.15.67:0/1459917613 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6641066a0 0x7ff664191810 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff654000c00 tx=0x7ff654011b00 comp rx=0 tx=0).stop 2024-03-28T15:51:07.152 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:06.428+0000 7ff663e00700 1 -- 172.21.15.67:0/1459917613 shutdown_connections 2024-03-28T15:51:07.152 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:06.428+0000 7ff663e00700 1 --2- 172.21.15.67:0/1459917613 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7ff648038f20 0x7ff64803b3e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:07.152 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:06.428+0000 7ff663e00700 1 --2- 172.21.15.67:0/1459917613 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6641066a0 0x7ff664191810 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:07.152 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:06.428+0000 7ff663e00700 1 -- 172.21.15.67:0/1459917613 >> 172.21.15.67:0/1459917613 conn(0x7ff664101cc0 msgr2=0x7ff664102ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:07.152 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:06.429+0000 7ff663e00700 1 -- 172.21.15.67:0/1459917613 shutdown_connections 2024-03-28T15:51:07.152 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:06.429+0000 7ff663e00700 1 -- 172.21.15.67:0/1459917613 wait complete. 2024-03-28T15:51:08.658 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-28T15:51:08.658 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:07.734+0000 7f6d1ce00700 1 Processor -- start 2024-03-28T15:51:08.658 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:07.734+0000 7f6d1ce00700 1 -- start start 2024-03-28T15:51:08.658 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:07.735+0000 7f6d1ce00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d18109b80 0x7f6d18109fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:08.658 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:07.735+0000 7f6d1ce00700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d1810a570 con 0x7f6d18109b80 2024-03-28T15:51:08.658 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:07.735+0000 7f6d15e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d18109b80 0x7f6d18109fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:08.658 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:07.736+0000 7f6d15e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d18109b80 0x7f6d18109fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49982/0 (socket says 172.21.15.67:49982) 2024-03-28T15:51:08.658 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:07.736+0000 7f6d15e00700 1 -- 172.21.15.67:0/1575791737 learned_addr learned my addr 172.21.15.67:0/1575791737 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:51:08.658 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:07.736+0000 7f6d15e00700 1 -- 172.21.15.67:0/1575791737 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d1810ad80 con 0x7f6d18109b80 2024-03-28T15:51:08.659 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:07.737+0000 7f6d15e00700 1 --2- 172.21.15.67:0/1575791737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d18109b80 0x7f6d18109fa0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6d080098a0 tx=0x7f6d08009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e115fa254f5ef6b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:08.659 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:07.737+0000 7f6d14a00700 1 -- 172.21.15.67:0/1575791737 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6d08004160 con 0x7f6d18109b80 2024-03-28T15:51:08.659 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:07.737+0000 7f6d14a00700 1 -- 172.21.15.67:0/1575791737 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7f6d08004300 con 0x7f6d18109b80 2024-03-28T15:51:08.659 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:07.737+0000 7f6d14a00700 1 -- 172.21.15.67:0/1575791737 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6d08011a80 con 0x7f6d18109b80 2024-03-28T15:51:08.659 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:07.738+0000 7f6d1ce00700 1 -- 172.21.15.67:0/1575791737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d18109b80 msgr2=0x7f6d18109fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:08.659 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:07.738+0000 7f6d1ce00700 1 --2- 172.21.15.67:0/1575791737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d18109b80 0x7f6d18109fa0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6d080098a0 tx=0x7f6d08009bb0 comp rx=0 tx=0).stop 2024-03-28T15:51:08.659 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:07.738+0000 7f6d1ce00700 1 -- 172.21.15.67:0/1575791737 shutdown_connections 2024-03-28T15:51:08.662 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:07.738+0000 7f6d1ce00700 1 --2- 172.21.15.67:0/1575791737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d18109b80 0x7f6d18109fa0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:08.662 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:07.738+0000 7f6d1ce00700 1 -- 172.21.15.67:0/1575791737 >> 172.21.15.67:0/1575791737 conn(0x7f6d1807cfa0 msgr2=0x7f6d1807d3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:08.663 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:07.738+0000 7f6d1ce00700 1 -- 172.21.15.67:0/1575791737 shutdown_connections 2024-03-28T15:51:08.663 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:07.738+0000 7f6d1ce00700 1 -- 172.21.15.67:0/1575791737 wait complete. 2024-03-28T15:51:08.663 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:07.739+0000 7f6d1ce00700 1 Processor -- start 2024-03-28T15:51:08.663 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:07.739+0000 7f6d1ce00700 1 -- start start 2024-03-28T15:51:08.663 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:07.740+0000 7f6d1ce00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d18109b80 0x7f6d1819db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:08.663 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:07.740+0000 7f6d1ce00700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d1810e130 con 0x7f6d18109b80 2024-03-28T15:51:08.663 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:07.740+0000 7f6d15e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d18109b80 0x7f6d1819db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:08.663 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:07.740+0000 7f6d15e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d18109b80 0x7f6d1819db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49994/0 (socket says 172.21.15.67:49994) 2024-03-28T15:51:08.663 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:07.740+0000 7f6d15e00700 1 -- 172.21.15.67:0/1223381816 learned_addr learned my addr 172.21.15.67:0/1223381816 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:51:08.663 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:07.740+0000 7f6d15e00700 1 -- 172.21.15.67:0/1223381816 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d1819e050 con 0x7f6d18109b80 2024-03-28T15:51:08.663 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:07.741+0000 7f6d15e00700 1 --2- 172.21.15.67:0/1223381816 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d18109b80 0x7f6d1819db10 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6d08000c00 tx=0x7f6d08011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:08.663 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:07.741+0000 7f6d0ea00700 1 -- 172.21.15.67:0/1223381816 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6d080039f0 con 0x7f6d18109b80 2024-03-28T15:51:08.663 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:07.741+0000 7f6d0ea00700 1 -- 172.21.15.67:0/1223381816 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7f6d08003b90 con 0x7f6d18109b80 2024-03-28T15:51:08.663 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:07.741+0000 7f6d1ce00700 1 -- 172.21.15.67:0/1223381816 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d1819e2e0 con 0x7f6d18109b80 2024-03-28T15:51:08.664 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:07.741+0000 7f6d0ea00700 1 -- 172.21.15.67:0/1223381816 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6d0801a740 con 0x7f6d18109b80 2024-03-28T15:51:08.664 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:07.741+0000 7f6d1ce00700 1 -- 172.21.15.67:0/1223381816 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d181a0fd0 con 0x7f6d18109b80 2024-03-28T15:51:08.664 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:07.742+0000 7f6d1ce00700 1 -- 172.21.15.67:0/1223381816 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d180637c0 con 0x7f6d18109b80 2024-03-28T15:51:08.664 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:07.743+0000 7f6d0ea00700 1 -- 172.21.15.67:0/1223381816 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f6d08024af0 con 0x7f6d18109b80 2024-03-28T15:51:08.664 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:07.744+0000 7f6d0ea00700 1 --2- 172.21.15.67:0/1223381816 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7f6cfc038f20 0x7f6cfc03b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:08.664 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:07.744+0000 7f6d0ea00700 1 -- 172.21.15.67:0/1223381816 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f6d0804d480 con 0x7f6d18109b80 2024-03-28T15:51:08.664 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:07.744+0000 7f6d15400700 1 --2- 172.21.15.67:0/1223381816 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7f6cfc038f20 0x7f6cfc03b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:08.664 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:07.745+0000 7f6d15400700 1 --2- 172.21.15.67:0/1223381816 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7f6cfc038f20 0x7f6cfc03b3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6d04006fd0 tx=0x7f6d04006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:08.664 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:07.749+0000 7f6d0ea00700 1 -- 172.21.15.67:0/1223381816 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6d0800b5a0 con 0x7f6d18109b80 2024-03-28T15:51:08.664 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:07.977+0000 7f6d1ce00700 1 -- 172.21.15.67:0/1223381816 --> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f6d18107c10 con 0x7f6cfc038f20 2024-03-28T15:51:08.664 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:07.978+0000 7f6d0ea00700 1 -- 172.21.15.67:0/1223381816 <== mgr.14120 v2:172.21.15.67:6800/1631001968 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f6d18107c10 con 0x7f6cfc038f20 2024-03-28T15:51:08.664 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:07.981+0000 7f6d1ce00700 1 -- 172.21.15.67:0/1223381816 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7f6cfc038f20 msgr2=0x7f6cfc03b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:08.664 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:07.981+0000 7f6d1ce00700 1 --2- 172.21.15.67:0/1223381816 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7f6cfc038f20 0x7f6cfc03b3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6d04006fd0 tx=0x7f6d04006e40 comp rx=0 tx=0).stop 2024-03-28T15:51:08.664 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:07.981+0000 7f6d1ce00700 1 -- 172.21.15.67:0/1223381816 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d18109b80 msgr2=0x7f6d1819db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:08.664 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:07.981+0000 7f6d1ce00700 1 --2- 172.21.15.67:0/1223381816 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d18109b80 0x7f6d1819db10 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6d08000c00 tx=0x7f6d08011b00 comp rx=0 tx=0).stop 2024-03-28T15:51:08.665 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:07.981+0000 7f6d1ce00700 1 -- 172.21.15.67:0/1223381816 shutdown_connections 2024-03-28T15:51:08.665 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:07.981+0000 7f6d1ce00700 1 --2- 172.21.15.67:0/1223381816 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7f6cfc038f20 0x7f6cfc03b3e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:08.666 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:07.981+0000 7f6d1ce00700 1 --2- 172.21.15.67:0/1223381816 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d18109b80 0x7f6d1819db10 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:08.666 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:07.981+0000 7f6d1ce00700 1 -- 172.21.15.67:0/1223381816 >> 172.21.15.67:0/1223381816 conn(0x7f6d1807cfa0 msgr2=0x7f6d18107500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:08.666 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:07.981+0000 7f6d1ce00700 1 -- 172.21.15.67:0/1223381816 shutdown_connections 2024-03-28T15:51:08.666 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:07.982+0000 7f6d1ce00700 1 -- 172.21.15.67:0/1223381816 wait complete. 2024-03-28T15:51:08.666 INFO:teuthology.orchestra.run.smithi067.stdout:Generating ssh key... 2024-03-28T15:51:09.242 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:08 smithi067 ceph-mon[31707]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:51:10.064 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:51:09 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: Generating public/private rsa key pair. 2024-03-28T15:51:10.064 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:51:09 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: Your identification has been saved in /tmp/tmp2ac5pv17/key. 2024-03-28T15:51:10.064 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:51:09 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: Your public key has been saved in /tmp/tmp2ac5pv17/key.pub. 2024-03-28T15:51:10.065 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:51:09 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: The key fingerprint is: 2024-03-28T15:51:10.065 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:51:09 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: SHA256:/3WGvRg4EGg3K+/BifX0joOD0Jm1vVq4zO4ERnWrHeI ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715 2024-03-28T15:51:10.065 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:51:09 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: The key's randomart image is: 2024-03-28T15:51:10.065 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:51:09 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: +---[RSA 3072]----+ 2024-03-28T15:51:10.065 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:51:09 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: | . . | 2024-03-28T15:51:10.065 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:51:09 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: | o . . | 2024-03-28T15:51:10.065 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:51:09 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: | + = o | 2024-03-28T15:51:10.065 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:51:09 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: | o o.B . | 2024-03-28T15:51:10.065 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:51:09 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: | .+SEoo | 2024-03-28T15:51:10.065 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:51:09 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: | ..+Bo*.o o | 2024-03-28T15:51:10.065 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:51:09 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: | ...Bo=.oo +| 2024-03-28T15:51:10.065 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:51:09 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: | .=o+++.oo.| 2024-03-28T15:51:10.065 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:51:09 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: | oOo.oo . | 2024-03-28T15:51:10.065 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:51:09 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: +----[SHA256]-----+ 2024-03-28T15:51:10.391 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:09.241+0000 7f4cfca00700 1 Processor -- start 2024-03-28T15:51:10.391 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:09.242+0000 7f4cfca00700 1 -- start start 2024-03-28T15:51:10.391 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:09.242+0000 7f4cfca00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4cf807cd10 0x7f4cf807b170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:10.391 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:09.242+0000 7f4cfca00700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf807b740 con 0x7f4cf807cd10 2024-03-28T15:51:10.391 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:09.242+0000 7f4cf5e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4cf807cd10 0x7f4cf807b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:10.392 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:09.242+0000 7f4cf5e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4cf807cd10 0x7f4cf807b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50010/0 (socket says 172.21.15.67:50010) 2024-03-28T15:51:10.392 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:09.243+0000 7f4cf5e00700 1 -- 172.21.15.67:0/1353237120 learned_addr learned my addr 172.21.15.67:0/1353237120 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:51:10.392 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:09.243+0000 7f4cf5e00700 1 -- 172.21.15.67:0/1353237120 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cf807b8c0 con 0x7f4cf807cd10 2024-03-28T15:51:10.392 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:09.243+0000 7f4cf5e00700 1 --2- 172.21.15.67:0/1353237120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4cf807cd10 0x7f4cf807b170 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4ce80098a0 tx=0x7f4ce8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d294895b613cef71 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:10.392 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:09.244+0000 7f4cf4a00700 1 -- 172.21.15.67:0/1353237120 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4ce8004160 con 0x7f4cf807cd10 2024-03-28T15:51:10.392 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:09.244+0000 7f4cf4a00700 1 -- 172.21.15.67:0/1353237120 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7f4ce8004300 con 0x7f4cf807cd10 2024-03-28T15:51:10.392 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:09.244+0000 7f4cf4a00700 1 -- 172.21.15.67:0/1353237120 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4ce8011a80 con 0x7f4cf807cd10 2024-03-28T15:51:10.392 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:09.244+0000 7f4cfca00700 1 -- 172.21.15.67:0/1353237120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4cf807cd10 msgr2=0x7f4cf807b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:10.392 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:09.244+0000 7f4cfca00700 1 --2- 172.21.15.67:0/1353237120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4cf807cd10 0x7f4cf807b170 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4ce80098a0 tx=0x7f4ce8009bb0 comp rx=0 tx=0).stop 2024-03-28T15:51:10.392 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:09.244+0000 7f4cfca00700 1 -- 172.21.15.67:0/1353237120 shutdown_connections 2024-03-28T15:51:10.392 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:09.244+0000 7f4cfca00700 1 --2- 172.21.15.67:0/1353237120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4cf807cd10 0x7f4cf807b170 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:10.392 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:09.245+0000 7f4cfca00700 1 -- 172.21.15.67:0/1353237120 >> 172.21.15.67:0/1353237120 conn(0x7f4cf8102f70 msgr2=0x7f4cf8105390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:10.392 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:09.245+0000 7f4cfca00700 1 -- 172.21.15.67:0/1353237120 shutdown_connections 2024-03-28T15:51:10.392 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:09.245+0000 7f4cfca00700 1 -- 172.21.15.67:0/1353237120 wait complete. 2024-03-28T15:51:10.393 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:09.246+0000 7f4cfca00700 1 Processor -- start 2024-03-28T15:51:10.393 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:09.246+0000 7f4cfca00700 1 -- start start 2024-03-28T15:51:10.393 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:09.246+0000 7f4cfca00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4cf807cd10 0x7f4cf819daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:10.393 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:09.246+0000 7f4cfca00700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf810df40 con 0x7f4cf807cd10 2024-03-28T15:51:10.393 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:09.246+0000 7f4cf5e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4cf807cd10 0x7f4cf819daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:10.393 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:09.246+0000 7f4cf5e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4cf807cd10 0x7f4cf819daa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50012/0 (socket says 172.21.15.67:50012) 2024-03-28T15:51:10.393 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:09.246+0000 7f4cf5e00700 1 -- 172.21.15.67:0/1567231444 learned_addr learned my addr 172.21.15.67:0/1567231444 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:51:10.393 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:09.247+0000 7f4cf5e00700 1 -- 172.21.15.67:0/1567231444 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cf819dfe0 con 0x7f4cf807cd10 2024-03-28T15:51:10.393 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:09.247+0000 7f4cf5e00700 1 --2- 172.21.15.67:0/1567231444 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4cf807cd10 0x7f4cf819daa0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4ce8011a50 tx=0x7f4ce8011b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:10.393 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:09.247+0000 7f4ceea00700 1 -- 172.21.15.67:0/1567231444 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4ce8003a80 con 0x7f4cf807cd10 2024-03-28T15:51:10.393 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:09.247+0000 7f4ceea00700 1 -- 172.21.15.67:0/1567231444 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7f4ce8003c20 con 0x7f4cf807cd10 2024-03-28T15:51:10.393 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:09.247+0000 7f4cfca00700 1 -- 172.21.15.67:0/1567231444 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cf819e270 con 0x7f4cf807cd10 2024-03-28T15:51:10.393 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:09.248+0000 7f4ceea00700 1 -- 172.21.15.67:0/1567231444 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4ce801a650 con 0x7f4cf807cd10 2024-03-28T15:51:10.393 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:09.248+0000 7f4cfca00700 1 -- 172.21.15.67:0/1567231444 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cf81a0f60 con 0x7f4cf807cd10 2024-03-28T15:51:10.393 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:09.249+0000 7f4ceea00700 1 -- 172.21.15.67:0/1567231444 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f4ce8024440 con 0x7f4cf807cd10 2024-03-28T15:51:10.394 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:09.249+0000 7f4cfca00700 1 -- 172.21.15.67:0/1567231444 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cf80637c0 con 0x7f4cf807cd10 2024-03-28T15:51:10.394 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:09.249+0000 7f4ceea00700 1 --2- 172.21.15.67:0/1567231444 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7f4cdc038f20 0x7f4cdc03b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:10.394 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:09.249+0000 7f4ceea00700 1 -- 172.21.15.67:0/1567231444 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f4ce804d400 con 0x7f4cf807cd10 2024-03-28T15:51:10.394 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:09.250+0000 7f4cf5400700 1 --2- 172.21.15.67:0/1567231444 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7f4cdc038f20 0x7f4cdc03b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:10.394 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:09.250+0000 7f4cf5400700 1 --2- 172.21.15.67:0/1567231444 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7f4cdc038f20 0x7f4cdc03b3e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4ce4006fd0 tx=0x7f4ce4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:10.394 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:09.255+0000 7f4ceea00700 1 -- 172.21.15.67:0/1567231444 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4ce804cd50 con 0x7f4cf807cd10 2024-03-28T15:51:10.394 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:09.481+0000 7f4cfca00700 1 -- 172.21.15.67:0/1567231444 --> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f4cf81041b0 con 0x7f4cdc038f20 2024-03-28T15:51:10.394 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:09.665+0000 7f4ceea00700 1 -- 172.21.15.67:0/1567231444 <== mgr.14120 v2:172.21.15.67:6800/1631001968 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f4cf81041b0 con 0x7f4cdc038f20 2024-03-28T15:51:10.394 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:09.669+0000 7f4cfca00700 1 -- 172.21.15.67:0/1567231444 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7f4cdc038f20 msgr2=0x7f4cdc03b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:10.394 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:09.669+0000 7f4cfca00700 1 --2- 172.21.15.67:0/1567231444 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7f4cdc038f20 0x7f4cdc03b3e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4ce4006fd0 tx=0x7f4ce4006e40 comp rx=0 tx=0).stop 2024-03-28T15:51:10.394 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:09.669+0000 7f4cfca00700 1 -- 172.21.15.67:0/1567231444 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4cf807cd10 msgr2=0x7f4cf819daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:10.394 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:09.669+0000 7f4cfca00700 1 --2- 172.21.15.67:0/1567231444 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4cf807cd10 0x7f4cf819daa0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4ce8011a50 tx=0x7f4ce8011b30 comp rx=0 tx=0).stop 2024-03-28T15:51:10.394 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:09.669+0000 7f4cfca00700 1 -- 172.21.15.67:0/1567231444 shutdown_connections 2024-03-28T15:51:10.395 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:09.669+0000 7f4cfca00700 1 --2- 172.21.15.67:0/1567231444 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7f4cdc038f20 0x7f4cdc03b3e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:10.395 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:09.669+0000 7f4cfca00700 1 --2- 172.21.15.67:0/1567231444 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4cf807cd10 0x7f4cf819daa0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:10.395 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:09.669+0000 7f4cfca00700 1 -- 172.21.15.67:0/1567231444 >> 172.21.15.67:0/1567231444 conn(0x7f4cf8102f70 msgr2=0x7f4cf8103aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:10.395 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:09.670+0000 7f4cfca00700 1 -- 172.21.15.67:0/1567231444 shutdown_connections 2024-03-28T15:51:10.395 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:09.670+0000 7f4cfca00700 1 -- 172.21.15.67:0/1567231444 wait complete. 2024-03-28T15:51:10.847 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:10 smithi067 ceph-mon[31707]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:51:10.848 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:10 smithi067 ceph-mon[31707]: Generating ssh key... 2024-03-28T15:51:10.848 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:10 smithi067 ceph-mon[31707]: from='mgr.14120 172.21.15.67:0/2156043519' entity='mgr.x' 2024-03-28T15:51:10.848 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:10 smithi067 ceph-mon[31707]: from='mgr.14120 172.21.15.67:0/2156043519' entity='mgr.x' 2024-03-28T15:51:11.961 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6KW4xZdBV/fUBqKs/kJRL4iaqOTp2NS5gBusRScWfIJfUYjUNmJZ4MHz8hgGUYjAjxgo58AqD9y0cjS9va8z0cxXaPUWtNsUNGek4Rnit1AZhcgfhrpA/0zs7kkNy5MwEtmf8gfxO/4uPilioYEk4TSUrbashODf450mBFn4AW284H99pmcu2JCDQ7H6MwTNLWMU+WxUNls32jyNiKjY7PiQOSNk1tjKNFAb5adfRE9GU+NJfaUiIiflnaGCKHeC4Uqgf8yCbNMPj2Bcz61QNNvlCmMwn2pNCKDbswmaa5Aw8G+3W5pVIUK+sgsPWJKedjubfHkSDEnlDMU4Pir0Debj0gxejCAI7aQGLSlpV2954ynqndMBorThzg+UYxjLT+E5gN4k7zTE/UnXVUYZSlx6U9AnUC1UMiLqG+3Vn1jS2Ib8J7nu0rCRXlTo3G+9MuqvGlm7xdJH64xftzC/fNM1jFxwZZJAA1vCinXpgfVMkrjQP24Wu+W4Iu1N37ak= ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715 2024-03-28T15:51:11.962 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:11.013+0000 7fbf97c00700 1 Processor -- start 2024-03-28T15:51:11.962 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:11.013+0000 7fbf97c00700 1 -- start start 2024-03-28T15:51:11.962 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:11.014+0000 7fbf97c00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf90081910 0x7fbf90081d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:11.962 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:11.014+0000 7fbf97c00700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf90082300 con 0x7fbf90081910 2024-03-28T15:51:11.962 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:11.014+0000 7fbf95200700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf90081910 0x7fbf90081d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:11.962 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:11.014+0000 7fbf95200700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf90081910 0x7fbf90081d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44344/0 (socket says 172.21.15.67:44344) 2024-03-28T15:51:11.963 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:11.014+0000 7fbf95200700 1 -- 172.21.15.67:0/1083258161 learned_addr learned my addr 172.21.15.67:0/1083258161 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:51:11.963 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:11.015+0000 7fbf95200700 1 -- 172.21.15.67:0/1083258161 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf9010f180 con 0x7fbf90081910 2024-03-28T15:51:11.963 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:11.015+0000 7fbf95200700 1 --2- 172.21.15.67:0/1083258161 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf90081910 0x7fbf90081d30 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fbf840098a0 tx=0x7fbf84009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e16f770d344b6f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:11.963 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:11.015+0000 7fbf8f400700 1 -- 172.21.15.67:0/1083258161 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbf84004160 con 0x7fbf90081910 2024-03-28T15:51:11.963 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:11.015+0000 7fbf8f400700 1 -- 172.21.15.67:0/1083258161 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7fbf84004300 con 0x7fbf90081910 2024-03-28T15:51:11.963 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:11.015+0000 7fbf8f400700 1 -- 172.21.15.67:0/1083258161 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbf84011a80 con 0x7fbf90081910 2024-03-28T15:51:11.964 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:11.016+0000 7fbf97c00700 1 -- 172.21.15.67:0/1083258161 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf90081910 msgr2=0x7fbf90081d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:11.964 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:11.016+0000 7fbf97c00700 1 --2- 172.21.15.67:0/1083258161 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf90081910 0x7fbf90081d30 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fbf840098a0 tx=0x7fbf84009bb0 comp rx=0 tx=0).stop 2024-03-28T15:51:11.964 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:11.016+0000 7fbf97c00700 1 -- 172.21.15.67:0/1083258161 shutdown_connections 2024-03-28T15:51:11.964 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:11.016+0000 7fbf97c00700 1 --2- 172.21.15.67:0/1083258161 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf90081910 0x7fbf90081d30 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:11.964 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:11.016+0000 7fbf97c00700 1 -- 172.21.15.67:0/1083258161 >> 172.21.15.67:0/1083258161 conn(0x7fbf9007cfa0 msgr2=0x7fbf9007f3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:11.964 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:11.016+0000 7fbf97c00700 1 -- 172.21.15.67:0/1083258161 shutdown_connections 2024-03-28T15:51:11.964 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:11.016+0000 7fbf97c00700 1 -- 172.21.15.67:0/1083258161 wait complete. 2024-03-28T15:51:11.964 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:11.017+0000 7fbf97c00700 1 Processor -- start 2024-03-28T15:51:11.965 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:11.018+0000 7fbf97c00700 1 -- start start 2024-03-28T15:51:11.965 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:11.018+0000 7fbf97c00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf90081910 0x7fbf901a1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:11.965 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:11.018+0000 7fbf97c00700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf90112530 con 0x7fbf90081910 2024-03-28T15:51:11.965 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:11.018+0000 7fbf95200700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf90081910 0x7fbf901a1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:11.965 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:11.018+0000 7fbf95200700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf90081910 0x7fbf901a1ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44358/0 (socket says 172.21.15.67:44358) 2024-03-28T15:51:11.965 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:11.018+0000 7fbf95200700 1 -- 172.21.15.67:0/2162943085 learned_addr learned my addr 172.21.15.67:0/2162943085 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:51:11.965 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:11.019+0000 7fbf95200700 1 -- 172.21.15.67:0/2162943085 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf901a2420 con 0x7fbf90081910 2024-03-28T15:51:11.965 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:11.019+0000 7fbf95200700 1 --2- 172.21.15.67:0/2162943085 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf90081910 0x7fbf901a1ee0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbf84000c00 tx=0x7fbf84011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:11.965 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:11.019+0000 7fbf8d600700 1 -- 172.21.15.67:0/2162943085 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbf840039f0 con 0x7fbf90081910 2024-03-28T15:51:11.965 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:11.019+0000 7fbf8d600700 1 -- 172.21.15.67:0/2162943085 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7fbf84003b90 con 0x7fbf90081910 2024-03-28T15:51:11.965 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:11.019+0000 7fbf8d600700 1 -- 172.21.15.67:0/2162943085 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbf8401a740 con 0x7fbf90081910 2024-03-28T15:51:11.965 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:11.019+0000 7fbf97c00700 1 -- 172.21.15.67:0/2162943085 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf901a26b0 con 0x7fbf90081910 2024-03-28T15:51:11.965 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:11.020+0000 7fbf97c00700 1 -- 172.21.15.67:0/2162943085 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf901a53a0 con 0x7fbf90081910 2024-03-28T15:51:11.965 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:11.021+0000 7fbf8d600700 1 -- 172.21.15.67:0/2162943085 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7fbf84024af0 con 0x7fbf90081910 2024-03-28T15:51:11.966 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:11.021+0000 7fbf97c00700 1 -- 172.21.15.67:0/2162943085 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf900637c0 con 0x7fbf90081910 2024-03-28T15:51:11.966 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:11.021+0000 7fbf8d600700 1 --2- 172.21.15.67:0/2162943085 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7fbf78038f70 0x7fbf7803b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:11.966 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:11.021+0000 7fbf8d600700 1 -- 172.21.15.67:0/2162943085 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fbf8404d230 con 0x7fbf90081910 2024-03-28T15:51:11.966 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:11.022+0000 7fbf8fe00700 1 --2- 172.21.15.67:0/2162943085 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7fbf78038f70 0x7fbf7803b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:11.966 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:11.022+0000 7fbf8fe00700 1 --2- 172.21.15.67:0/2162943085 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7fbf78038f70 0x7fbf7803b430 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fbf7c006fd0 tx=0x7fbf7c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:11.966 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:11.027+0000 7fbf8d600700 1 -- 172.21.15.67:0/2162943085 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fbf8400e260 con 0x7fbf90081910 2024-03-28T15:51:11.966 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:11.254+0000 7fbf97c00700 1 -- 172.21.15.67:0/2162943085 --> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fbf9007e2b0 con 0x7fbf78038f70 2024-03-28T15:51:11.966 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:11.256+0000 7fbf8d600700 1 -- 172.21.15.67:0/2162943085 <== mgr.14120 v2:172.21.15.67:6800/1631001968 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7fbf9007e2b0 con 0x7fbf78038f70 2024-03-28T15:51:11.966 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:11.258+0000 7fbf97c00700 1 -- 172.21.15.67:0/2162943085 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7fbf78038f70 msgr2=0x7fbf7803b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:11.966 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:11.258+0000 7fbf97c00700 1 --2- 172.21.15.67:0/2162943085 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7fbf78038f70 0x7fbf7803b430 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fbf7c006fd0 tx=0x7fbf7c006e40 comp rx=0 tx=0).stop 2024-03-28T15:51:11.966 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:11.258+0000 7fbf97c00700 1 -- 172.21.15.67:0/2162943085 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf90081910 msgr2=0x7fbf901a1ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:11.966 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:11.258+0000 7fbf97c00700 1 --2- 172.21.15.67:0/2162943085 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf90081910 0x7fbf901a1ee0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbf84000c00 tx=0x7fbf84011b00 comp rx=0 tx=0).stop 2024-03-28T15:51:11.966 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:11.259+0000 7fbf97c00700 1 -- 172.21.15.67:0/2162943085 shutdown_connections 2024-03-28T15:51:11.967 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:11.259+0000 7fbf97c00700 1 --2- 172.21.15.67:0/2162943085 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7fbf78038f70 0x7fbf7803b430 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:11.967 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:11.259+0000 7fbf97c00700 1 --2- 172.21.15.67:0/2162943085 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf90081910 0x7fbf901a1ee0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:11.967 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:11.259+0000 7fbf97c00700 1 -- 172.21.15.67:0/2162943085 >> 172.21.15.67:0/2162943085 conn(0x7fbf9007cfa0 msgr2=0x7fbf9007dba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:11.967 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:11.259+0000 7fbf97c00700 1 -- 172.21.15.67:0/2162943085 shutdown_connections 2024-03-28T15:51:11.967 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:11.259+0000 7fbf97c00700 1 -- 172.21.15.67:0/2162943085 wait complete. 2024-03-28T15:51:11.967 INFO:teuthology.orchestra.run.smithi067.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-28T15:51:11.967 INFO:teuthology.orchestra.run.smithi067.stdout:Adding key to root@localhost authorized_keys... 2024-03-28T15:51:11.967 INFO:teuthology.orchestra.run.smithi067.stdout:Adding host smithi067... 2024-03-28T15:51:12.256 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:11 smithi067 ceph-mon[31707]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:51:13.166 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:12 smithi067 ceph-mon[31707]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi067", "addr": "172.21.15.67", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:51:15.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:14 smithi067 ceph-mon[31707]: Deploying cephadm binary to smithi067 2024-03-28T15:51:20.372 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout Added host 'smithi067' with addr '172.21.15.67' 2024-03-28T15:51:20.372 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:12.567+0000 7f554ee00700 1 Processor -- start 2024-03-28T15:51:20.372 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:12.568+0000 7f554ee00700 1 -- start start 2024-03-28T15:51:20.372 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:12.568+0000 7f554ee00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5548107a00 0x7f5548107e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:20.372 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:12.568+0000 7f554ee00700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55481083f0 con 0x7f5548107a00 2024-03-28T15:51:20.372 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:12.568+0000 7f5547e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5548107a00 0x7f5548107e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:20.372 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:12.569+0000 7f5547e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5548107a00 0x7f5548107e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44372/0 (socket says 172.21.15.67:44372) 2024-03-28T15:51:20.372 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:12.569+0000 7f5547e00700 1 -- 172.21.15.67:0/3637263232 learned_addr learned my addr 172.21.15.67:0/3637263232 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:51:20.373 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:12.569+0000 7f5547e00700 1 -- 172.21.15.67:0/3637263232 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5548108c00 con 0x7f5548107a00 2024-03-28T15:51:20.373 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:12.569+0000 7f5547e00700 1 --2- 172.21.15.67:0/3637263232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5548107a00 0x7f5548107e20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f55380098a0 tx=0x7f5538009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=193147077f97ba29 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:20.373 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:12.570+0000 7f5546a00700 1 -- 172.21.15.67:0/3637263232 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5538004160 con 0x7f5548107a00 2024-03-28T15:51:20.373 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:12.570+0000 7f5546a00700 1 -- 172.21.15.67:0/3637263232 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7f5538004300 con 0x7f5548107a00 2024-03-28T15:51:20.373 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:12.570+0000 7f5546a00700 1 -- 172.21.15.67:0/3637263232 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5538011a80 con 0x7f5548107a00 2024-03-28T15:51:20.373 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:12.571+0000 7f554ee00700 1 -- 172.21.15.67:0/3637263232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5548107a00 msgr2=0x7f5548107e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:20.373 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:12.571+0000 7f554ee00700 1 --2- 172.21.15.67:0/3637263232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5548107a00 0x7f5548107e20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f55380098a0 tx=0x7f5538009bb0 comp rx=0 tx=0).stop 2024-03-28T15:51:20.373 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:12.571+0000 7f554ee00700 1 -- 172.21.15.67:0/3637263232 shutdown_connections 2024-03-28T15:51:20.374 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:12.571+0000 7f554ee00700 1 --2- 172.21.15.67:0/3637263232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5548107a00 0x7f5548107e20 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:20.374 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:12.571+0000 7f554ee00700 1 -- 172.21.15.67:0/3637263232 >> 172.21.15.67:0/3637263232 conn(0x7f5548103000 msgr2=0x7f5548105420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:20.374 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:12.571+0000 7f554ee00700 1 -- 172.21.15.67:0/3637263232 shutdown_connections 2024-03-28T15:51:20.374 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:12.571+0000 7f554ee00700 1 -- 172.21.15.67:0/3637263232 wait complete. 2024-03-28T15:51:20.374 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:12.572+0000 7f554ee00700 1 Processor -- start 2024-03-28T15:51:20.374 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:12.572+0000 7f554ee00700 1 -- start start 2024-03-28T15:51:20.374 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:12.573+0000 7f554ee00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5548107a00 0x7f554819db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:20.374 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:12.573+0000 7f554ee00700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f554810be30 con 0x7f5548107a00 2024-03-28T15:51:20.374 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:12.573+0000 7f5547e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5548107a00 0x7f554819db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:20.374 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:12.573+0000 7f5547e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5548107a00 0x7f554819db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44378/0 (socket says 172.21.15.67:44378) 2024-03-28T15:51:20.374 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:12.573+0000 7f5547e00700 1 -- 172.21.15.67:0/2745912855 learned_addr learned my addr 172.21.15.67:0/2745912855 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:51:20.374 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:12.574+0000 7f5547e00700 1 -- 172.21.15.67:0/2745912855 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f554819e0a0 con 0x7f5548107a00 2024-03-28T15:51:20.374 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:12.574+0000 7f5547e00700 1 --2- 172.21.15.67:0/2745912855 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5548107a00 0x7f554819db60 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f5538000c00 tx=0x7f5538011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:20.374 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:12.574+0000 7f5544c00700 1 -- 172.21.15.67:0/2745912855 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f55380039f0 con 0x7f5548107a00 2024-03-28T15:51:20.374 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:12.574+0000 7f5544c00700 1 -- 172.21.15.67:0/2745912855 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7f5538003b90 con 0x7f5548107a00 2024-03-28T15:51:20.375 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:12.574+0000 7f554ee00700 1 -- 172.21.15.67:0/2745912855 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f554819e330 con 0x7f5548107a00 2024-03-28T15:51:20.375 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:12.574+0000 7f5544c00700 1 -- 172.21.15.67:0/2745912855 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f553801a740 con 0x7f5548107a00 2024-03-28T15:51:20.375 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:12.575+0000 7f554ee00700 1 -- 172.21.15.67:0/2745912855 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55481a1020 con 0x7f5548107a00 2024-03-28T15:51:20.375 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:12.576+0000 7f5544c00700 1 -- 172.21.15.67:0/2745912855 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f5538024af0 con 0x7f5548107a00 2024-03-28T15:51:20.375 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:12.576+0000 7f554ee00700 1 -- 172.21.15.67:0/2745912855 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f554810bfb0 con 0x7f5548107a00 2024-03-28T15:51:20.375 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:12.576+0000 7f5544c00700 1 --2- 172.21.15.67:0/2745912855 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7f552c038f20 0x7f552c03b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:20.375 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:12.576+0000 7f5544c00700 1 -- 172.21.15.67:0/2745912855 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f553804d260 con 0x7f5548107a00 2024-03-28T15:51:20.375 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:12.577+0000 7f5547400700 1 --2- 172.21.15.67:0/2745912855 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7f552c038f20 0x7f552c03b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:20.375 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:12.578+0000 7f5547400700 1 --2- 172.21.15.67:0/2745912855 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7f552c038f20 0x7f552c03b3e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5534006fd0 tx=0x7f5534006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:20.375 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:12.584+0000 7f5544c00700 1 -- 172.21.15.67:0/2745912855 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f553800ba20 con 0x7f5548107a00 2024-03-28T15:51:20.375 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:12.817+0000 7f554ee00700 1 -- 172.21.15.67:0/2745912855 --> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi067", "addr": "172.21.15.67", "target": ["mon-mgr", ""]}) v1 -- 0x7f55481041b0 con 0x7f552c038f20 2024-03-28T15:51:20.375 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:19.656+0000 7f5544c00700 1 -- 172.21.15.67:0/2745912855 <== mgr.14120 v2:172.21.15.67:6800/1631001968 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f55481041b0 con 0x7f552c038f20 2024-03-28T15:51:20.375 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:19.660+0000 7f554ee00700 1 -- 172.21.15.67:0/2745912855 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7f552c038f20 msgr2=0x7f552c03b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:20.375 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:19.660+0000 7f554ee00700 1 --2- 172.21.15.67:0/2745912855 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7f552c038f20 0x7f552c03b3e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5534006fd0 tx=0x7f5534006e40 comp rx=0 tx=0).stop 2024-03-28T15:51:20.375 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:19.660+0000 7f554ee00700 1 -- 172.21.15.67:0/2745912855 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5548107a00 msgr2=0x7f554819db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:20.376 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:19.660+0000 7f554ee00700 1 --2- 172.21.15.67:0/2745912855 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5548107a00 0x7f554819db60 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f5538000c00 tx=0x7f5538011b00 comp rx=0 tx=0).stop 2024-03-28T15:51:20.376 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:19.661+0000 7f554ee00700 1 -- 172.21.15.67:0/2745912855 shutdown_connections 2024-03-28T15:51:20.376 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:19.661+0000 7f554ee00700 1 --2- 172.21.15.67:0/2745912855 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7f552c038f20 0x7f552c03b3e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:20.376 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:19.661+0000 7f554ee00700 1 --2- 172.21.15.67:0/2745912855 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5548107a00 0x7f554819db60 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:20.376 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:19.661+0000 7f554ee00700 1 -- 172.21.15.67:0/2745912855 >> 172.21.15.67:0/2745912855 conn(0x7f5548103000 msgr2=0x7f5548103aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:20.376 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:19.661+0000 7f554ee00700 1 -- 172.21.15.67:0/2745912855 shutdown_connections 2024-03-28T15:51:20.376 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:19.661+0000 7f554ee00700 1 -- 172.21.15.67:0/2745912855 wait complete. 2024-03-28T15:51:20.376 INFO:teuthology.orchestra.run.smithi067.stdout:Deploying unmanaged mon service... 2024-03-28T15:51:21.006 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:20 smithi067 ceph-mon[31707]: from='mgr.14120 172.21.15.67:0/2156043519' entity='mgr.x' 2024-03-28T15:51:21.006 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:20 smithi067 ceph-mon[31707]: Added host smithi067 2024-03-28T15:51:21.006 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:20 smithi067 ceph-mon[31707]: from='mgr.14120 172.21.15.67:0/2156043519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:51:22.453 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-28T15:51:22.454 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:21.029+0000 7f5973e00700 1 Processor -- start 2024-03-28T15:51:22.454 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:21.030+0000 7f5973e00700 1 -- start start 2024-03-28T15:51:22.454 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:21.030+0000 7f5973e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f597407cd70 0x7f597407b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:22.454 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:21.030+0000 7f5973e00700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f597407b7a0 con 0x7f597407cd70 2024-03-28T15:51:22.454 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:21.031+0000 7f5972a00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f597407cd70 0x7f597407b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:22.454 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:21.031+0000 7f5972a00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f597407cd70 0x7f597407b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49798/0 (socket says 172.21.15.67:49798) 2024-03-28T15:51:22.454 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:21.031+0000 7f5972a00700 1 -- 172.21.15.67:0/3049188243 learned_addr learned my addr 172.21.15.67:0/3049188243 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:51:22.454 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:21.031+0000 7f5972a00700 1 -- 172.21.15.67:0/3049188243 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f597407b920 con 0x7f597407cd70 2024-03-28T15:51:22.454 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:21.032+0000 7f5972a00700 1 --2- 172.21.15.67:0/3049188243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f597407cd70 0x7f597407b1d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f59640098a0 tx=0x7f5964009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e828c827fa3a13d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:22.454 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:21.032+0000 7f5971600700 1 -- 172.21.15.67:0/3049188243 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5964004160 con 0x7f597407cd70 2024-03-28T15:51:22.454 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:21.032+0000 7f5971600700 1 -- 172.21.15.67:0/3049188243 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7f5964004300 con 0x7f597407cd70 2024-03-28T15:51:22.455 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:21.032+0000 7f5971600700 1 -- 172.21.15.67:0/3049188243 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5964011a80 con 0x7f597407cd70 2024-03-28T15:51:22.455 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:21.032+0000 7f5973e00700 1 -- 172.21.15.67:0/3049188243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f597407cd70 msgr2=0x7f597407b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:22.455 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:21.032+0000 7f5973e00700 1 --2- 172.21.15.67:0/3049188243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f597407cd70 0x7f597407b1d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f59640098a0 tx=0x7f5964009bb0 comp rx=0 tx=0).stop 2024-03-28T15:51:22.455 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:21.033+0000 7f5973e00700 1 -- 172.21.15.67:0/3049188243 shutdown_connections 2024-03-28T15:51:22.455 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:21.033+0000 7f5973e00700 1 --2- 172.21.15.67:0/3049188243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f597407cd70 0x7f597407b1d0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:22.455 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:21.033+0000 7f5973e00700 1 -- 172.21.15.67:0/3049188243 >> 172.21.15.67:0/3049188243 conn(0x7f5974102d20 msgr2=0x7f5974105160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:22.455 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:21.033+0000 7f5973e00700 1 -- 172.21.15.67:0/3049188243 shutdown_connections 2024-03-28T15:51:22.455 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:21.033+0000 7f5973e00700 1 -- 172.21.15.67:0/3049188243 wait complete. 2024-03-28T15:51:22.455 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:21.034+0000 7f5973e00700 1 Processor -- start 2024-03-28T15:51:22.455 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:21.034+0000 7f5973e00700 1 -- start start 2024-03-28T15:51:22.455 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:21.034+0000 7f5973e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f597407cd70 0x7f59741a1d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:22.455 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:21.034+0000 7f5973e00700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5974109930 con 0x7f597407cd70 2024-03-28T15:51:22.455 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:21.035+0000 7f5972a00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f597407cd70 0x7f59741a1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:22.455 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:21.035+0000 7f5972a00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f597407cd70 0x7f59741a1d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49814/0 (socket says 172.21.15.67:49814) 2024-03-28T15:51:22.455 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:21.035+0000 7f5972a00700 1 -- 172.21.15.67:0/444202181 learned_addr learned my addr 172.21.15.67:0/444202181 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:51:22.456 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:21.035+0000 7f5972a00700 1 -- 172.21.15.67:0/444202181 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59741a2260 con 0x7f597407cd70 2024-03-28T15:51:22.456 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:21.035+0000 7f5972a00700 1 --2- 172.21.15.67:0/444202181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f597407cd70 0x7f59741a1d20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5964000c00 tx=0x7f5964011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:22.456 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:21.036+0000 7f596b400700 1 -- 172.21.15.67:0/444202181 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f59640039f0 con 0x7f597407cd70 2024-03-28T15:51:22.456 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:21.036+0000 7f596b400700 1 -- 172.21.15.67:0/444202181 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7f5964003b90 con 0x7f597407cd70 2024-03-28T15:51:22.456 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:21.036+0000 7f596b400700 1 -- 172.21.15.67:0/444202181 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f596401a740 con 0x7f597407cd70 2024-03-28T15:51:22.456 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:21.036+0000 7f5973e00700 1 -- 172.21.15.67:0/444202181 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59741a24f0 con 0x7f597407cd70 2024-03-28T15:51:22.456 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:21.036+0000 7f5973e00700 1 -- 172.21.15.67:0/444202181 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59741a51d0 con 0x7f597407cd70 2024-03-28T15:51:22.456 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:21.037+0000 7f5973e00700 1 -- 172.21.15.67:0/444202181 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59740637c0 con 0x7f597407cd70 2024-03-28T15:51:22.456 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:21.037+0000 7f596b400700 1 -- 172.21.15.67:0/444202181 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f5964024af0 con 0x7f597407cd70 2024-03-28T15:51:22.456 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:21.037+0000 7f596b400700 1 --2- 172.21.15.67:0/444202181 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7f5958038f70 0x7f595803b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:22.456 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:21.037+0000 7f596b400700 1 -- 172.21.15.67:0/444202181 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f596404d230 con 0x7f597407cd70 2024-03-28T15:51:22.456 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:21.038+0000 7f5972000700 1 --2- 172.21.15.67:0/444202181 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7f5958038f70 0x7f595803b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:22.456 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:21.038+0000 7f5972000700 1 --2- 172.21.15.67:0/444202181 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7f5958038f70 0x7f595803b430 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f5960006fd0 tx=0x7f5960006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:22.456 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:21.043+0000 7f596b400700 1 -- 172.21.15.67:0/444202181 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f596400e260 con 0x7f597407cd70 2024-03-28T15:51:22.457 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:21.318+0000 7f5973e00700 1 -- 172.21.15.67:0/444202181 --> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f5974074a00 con 0x7f5958038f70 2024-03-28T15:51:22.457 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:21.328+0000 7f596b400700 1 -- 172.21.15.67:0/444202181 <== mgr.14120 v2:172.21.15.67:6800/1631001968 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f5974074a00 con 0x7f5958038f70 2024-03-28T15:51:22.457 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:21.333+0000 7f5973e00700 1 -- 172.21.15.67:0/444202181 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7f5958038f70 msgr2=0x7f595803b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:22.457 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:21.333+0000 7f5973e00700 1 --2- 172.21.15.67:0/444202181 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7f5958038f70 0x7f595803b430 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f5960006fd0 tx=0x7f5960006e40 comp rx=0 tx=0).stop 2024-03-28T15:51:22.457 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:21.333+0000 7f5973e00700 1 -- 172.21.15.67:0/444202181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f597407cd70 msgr2=0x7f59741a1d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:22.457 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:21.333+0000 7f5973e00700 1 --2- 172.21.15.67:0/444202181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f597407cd70 0x7f59741a1d20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5964000c00 tx=0x7f5964011b00 comp rx=0 tx=0).stop 2024-03-28T15:51:22.457 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:21.333+0000 7f5973e00700 1 -- 172.21.15.67:0/444202181 shutdown_connections 2024-03-28T15:51:22.457 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:21.333+0000 7f5973e00700 1 --2- 172.21.15.67:0/444202181 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7f5958038f70 0x7f595803b430 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:22.457 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:21.333+0000 7f5973e00700 1 --2- 172.21.15.67:0/444202181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f597407cd70 0x7f59741a1d20 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:22.457 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:21.333+0000 7f5973e00700 1 -- 172.21.15.67:0/444202181 >> 172.21.15.67:0/444202181 conn(0x7f5974102d20 msgr2=0x7f5974105160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:22.457 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:21.333+0000 7f5973e00700 1 -- 172.21.15.67:0/444202181 shutdown_connections 2024-03-28T15:51:22.457 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:21.333+0000 7f5973e00700 1 -- 172.21.15.67:0/444202181 wait complete. 2024-03-28T15:51:22.457 INFO:teuthology.orchestra.run.smithi067.stdout:Deploying unmanaged mgr service... 2024-03-28T15:51:22.609 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:22 smithi067 ceph-mon[31707]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:51:22.610 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:22 smithi067 ceph-mon[31707]: Saving service mon spec with placement count:5 2024-03-28T15:51:22.610 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:22 smithi067 ceph-mon[31707]: from='mgr.14120 172.21.15.67:0/2156043519' entity='mgr.x' 2024-03-28T15:51:24.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:24 smithi067 ceph-mon[31707]: from='mgr.14120 172.21.15.67:0/2156043519' entity='mgr.x' 2024-03-28T15:51:24.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:24 smithi067 ceph-mon[31707]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:51:24.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:24 smithi067 ceph-mon[31707]: Saving service mgr spec with placement count:2 2024-03-28T15:51:24.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:24 smithi067 ceph-mon[31707]: from='mgr.14120 172.21.15.67:0/2156043519' entity='mgr.x' 2024-03-28T15:51:24.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:24 smithi067 ceph-mon[31707]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:51:24.386 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-28T15:51:24.386 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:23.422+0000 7f17fa000700 1 Processor -- start 2024-03-28T15:51:24.386 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:23.422+0000 7f17fa000700 1 -- start start 2024-03-28T15:51:24.386 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:23.422+0000 7f17fa000700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17f4106880 0x7f17f4106ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:24.386 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:23.422+0000 7f17fa000700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f4107270 con 0x7f17f4106880 2024-03-28T15:51:24.386 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:23.423+0000 7f17f3400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17f4106880 0x7f17f4106ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:24.386 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:23.423+0000 7f17f3400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17f4106880 0x7f17f4106ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49820/0 (socket says 172.21.15.67:49820) 2024-03-28T15:51:24.386 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:23.423+0000 7f17f3400700 1 -- 172.21.15.67:0/4167289049 learned_addr learned my addr 172.21.15.67:0/4167289049 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:51:24.387 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:23.424+0000 7f17f3400700 1 -- 172.21.15.67:0/4167289049 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17f4107a80 con 0x7f17f4106880 2024-03-28T15:51:24.387 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:23.424+0000 7f17f3400700 1 --2- 172.21.15.67:0/4167289049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17f4106880 0x7f17f4106ca0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f17e400bb40 tx=0x7f17e400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1f6e3e5e7ffa849 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:24.387 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:23.424+0000 7f17f2000700 1 -- 172.21.15.67:0/4167289049 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f17e400dbe0 con 0x7f17f4106880 2024-03-28T15:51:24.387 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:23.424+0000 7f17f2000700 1 -- 172.21.15.67:0/4167289049 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f17e40044e0 con 0x7f17f4106880 2024-03-28T15:51:24.387 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:23.424+0000 7f17f2000700 1 -- 172.21.15.67:0/4167289049 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f17e4010470 con 0x7f17f4106880 2024-03-28T15:51:24.387 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:23.425+0000 7f17fa000700 1 -- 172.21.15.67:0/4167289049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17f4106880 msgr2=0x7f17f4106ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:24.387 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:23.425+0000 7f17fa000700 1 --2- 172.21.15.67:0/4167289049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17f4106880 0x7f17f4106ca0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f17e400bb40 tx=0x7f17e400be50 comp rx=0 tx=0).stop 2024-03-28T15:51:24.387 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:23.426+0000 7f17fa000700 1 -- 172.21.15.67:0/4167289049 shutdown_connections 2024-03-28T15:51:24.387 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:23.426+0000 7f17fa000700 1 --2- 172.21.15.67:0/4167289049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17f4106880 0x7f17f4106ca0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:24.387 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:23.426+0000 7f17fa000700 1 -- 172.21.15.67:0/4167289049 >> 172.21.15.67:0/4167289049 conn(0x7f17f4101e40 msgr2=0x7f17f4104260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:24.387 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:23.426+0000 7f17fa000700 1 -- 172.21.15.67:0/4167289049 shutdown_connections 2024-03-28T15:51:24.388 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:23.426+0000 7f17fa000700 1 -- 172.21.15.67:0/4167289049 wait complete. 2024-03-28T15:51:24.388 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:23.427+0000 7f17fa000700 1 Processor -- start 2024-03-28T15:51:24.388 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:23.427+0000 7f17fa000700 1 -- start start 2024-03-28T15:51:24.388 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:23.427+0000 7f17fa000700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17f4106880 0x7f17f419daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:24.388 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:23.427+0000 7f17fa000700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f40625d0 con 0x7f17f4106880 2024-03-28T15:51:24.388 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:23.428+0000 7f17f3400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17f4106880 0x7f17f419daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:24.388 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:23.428+0000 7f17f3400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17f4106880 0x7f17f419daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49824/0 (socket says 172.21.15.67:49824) 2024-03-28T15:51:24.388 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:23.428+0000 7f17f3400700 1 -- 172.21.15.67:0/558864064 learned_addr learned my addr 172.21.15.67:0/558864064 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:51:24.388 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:23.428+0000 7f17f3400700 1 -- 172.21.15.67:0/558864064 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17f419e030 con 0x7f17f4106880 2024-03-28T15:51:24.388 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:23.428+0000 7f17f3400700 1 --2- 172.21.15.67:0/558864064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17f4106880 0x7f17f419daf0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f17e400b340 tx=0x7f17e4003fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:24.388 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:23.429+0000 7f17ebe00700 1 -- 172.21.15.67:0/558864064 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f17e4010720 con 0x7f17f4106880 2024-03-28T15:51:24.388 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:23.429+0000 7f17ebe00700 1 -- 172.21.15.67:0/558864064 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f17e401d070 con 0x7f17f4106880 2024-03-28T15:51:24.389 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:23.429+0000 7f17ebe00700 1 -- 172.21.15.67:0/558864064 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f17e4021b90 con 0x7f17f4106880 2024-03-28T15:51:24.389 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:23.429+0000 7f17fa000700 1 -- 172.21.15.67:0/558864064 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17f419e2c0 con 0x7f17f4106880 2024-03-28T15:51:24.389 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:23.429+0000 7f17fa000700 1 -- 172.21.15.67:0/558864064 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17f419e6c0 con 0x7f17f4106880 2024-03-28T15:51:24.389 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:23.430+0000 7f17fa000700 1 -- 172.21.15.67:0/558864064 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17f40637c0 con 0x7f17f4106880 2024-03-28T15:51:24.389 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:23.431+0000 7f17ebe00700 1 -- 172.21.15.67:0/558864064 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f17e4029060 con 0x7f17f4106880 2024-03-28T15:51:24.389 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:23.431+0000 7f17ebe00700 1 --2- 172.21.15.67:0/558864064 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7f17d8038f70 0x7f17d803b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:24.389 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:23.431+0000 7f17ebe00700 1 -- 172.21.15.67:0/558864064 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f17e404df20 con 0x7f17f4106880 2024-03-28T15:51:24.389 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:23.432+0000 7f17f2a00700 1 --2- 172.21.15.67:0/558864064 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7f17d8038f70 0x7f17d803b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:24.389 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:23.432+0000 7f17f2a00700 1 --2- 172.21.15.67:0/558864064 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7f17d8038f70 0x7f17d803b430 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f17e0006fd0 tx=0x7f17e0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:24.389 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:23.438+0000 7f17ebe00700 1 -- 172.21.15.67:0/558864064 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f17e4030070 con 0x7f17f4106880 2024-03-28T15:51:24.389 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:23.680+0000 7f17fa000700 1 -- 172.21.15.67:0/558864064 --> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f17f4074a00 con 0x7f17d8038f70 2024-03-28T15:51:24.389 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:23.689+0000 7f17ebe00700 1 -- 172.21.15.67:0/558864064 <== mgr.14120 v2:172.21.15.67:6800/1631001968 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f17f4074a00 con 0x7f17d8038f70 2024-03-28T15:51:24.390 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:23.692+0000 7f17fa000700 1 -- 172.21.15.67:0/558864064 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7f17d8038f70 msgr2=0x7f17d803b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:24.390 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:23.692+0000 7f17fa000700 1 --2- 172.21.15.67:0/558864064 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7f17d8038f70 0x7f17d803b430 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f17e0006fd0 tx=0x7f17e0006e40 comp rx=0 tx=0).stop 2024-03-28T15:51:24.390 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:23.693+0000 7f17fa000700 1 -- 172.21.15.67:0/558864064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17f4106880 msgr2=0x7f17f419daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:24.390 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:23.693+0000 7f17fa000700 1 --2- 172.21.15.67:0/558864064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17f4106880 0x7f17f419daf0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f17e400b340 tx=0x7f17e4003fb0 comp rx=0 tx=0).stop 2024-03-28T15:51:24.390 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:23.693+0000 7f17fa000700 1 -- 172.21.15.67:0/558864064 shutdown_connections 2024-03-28T15:51:24.390 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:23.693+0000 7f17fa000700 1 --2- 172.21.15.67:0/558864064 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7f17d8038f70 0x7f17d803b430 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:24.390 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:23.693+0000 7f17fa000700 1 --2- 172.21.15.67:0/558864064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17f4106880 0x7f17f419daf0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:24.390 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:23.693+0000 7f17fa000700 1 -- 172.21.15.67:0/558864064 >> 172.21.15.67:0/558864064 conn(0x7f17f4101e40 msgr2=0x7f17f418fd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:24.390 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:23.693+0000 7f17fa000700 1 -- 172.21.15.67:0/558864064 shutdown_connections 2024-03-28T15:51:24.390 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:23.693+0000 7f17fa000700 1 -- 172.21.15.67:0/558864064 wait complete. 2024-03-28T15:51:25.306 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:25 smithi067 ceph-mon[31707]: from='mgr.14120 172.21.15.67:0/2156043519' entity='mgr.x' 2024-03-28T15:51:26.502 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:25.006+0000 7f1e21600700 1 Processor -- start 2024-03-28T15:51:26.502 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:25.007+0000 7f1e21600700 1 -- start start 2024-03-28T15:51:26.502 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:25.007+0000 7f1e21600700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e1c1084e0 0x7f1e1c10a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:26.502 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:25.007+0000 7f1e21600700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e1c0633d0 con 0x7f1e1c1084e0 2024-03-28T15:51:26.502 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:25.007+0000 7f1e1aa00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e1c1084e0 0x7f1e1c10a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:26.505 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:25.007+0000 7f1e1aa00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e1c1084e0 0x7f1e1c10a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49826/0 (socket says 172.21.15.67:49826) 2024-03-28T15:51:26.505 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:25.007+0000 7f1e1aa00700 1 -- 172.21.15.67:0/4132261232 learned_addr learned my addr 172.21.15.67:0/4132261232 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:51:26.505 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:25.008+0000 7f1e1aa00700 1 -- 172.21.15.67:0/4132261232 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e1c10ae10 con 0x7f1e1c1084e0 2024-03-28T15:51:26.505 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:25.008+0000 7f1e1aa00700 1 --2- 172.21.15.67:0/4132261232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e1c1084e0 0x7f1e1c10a8d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c0098a0 tx=0x7f1e0c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34b4ccd44114ac4a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:26.505 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:25.008+0000 7f1e1a000700 1 -- 172.21.15.67:0/4132261232 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1e0c004160 con 0x7f1e1c1084e0 2024-03-28T15:51:26.505 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:25.009+0000 7f1e1a000700 1 -- 172.21.15.67:0/4132261232 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1e0c004300 con 0x7f1e1c1084e0 2024-03-28T15:51:26.505 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:25.009+0000 7f1e1a000700 1 -- 172.21.15.67:0/4132261232 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1e0c011a50 con 0x7f1e1c1084e0 2024-03-28T15:51:26.505 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:25.010+0000 7f1e21600700 1 -- 172.21.15.67:0/4132261232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e1c1084e0 msgr2=0x7f1e1c10a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:26.505 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:25.010+0000 7f1e21600700 1 --2- 172.21.15.67:0/4132261232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e1c1084e0 0x7f1e1c10a8d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c0098a0 tx=0x7f1e0c009bb0 comp rx=0 tx=0).stop 2024-03-28T15:51:26.505 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:25.010+0000 7f1e21600700 1 -- 172.21.15.67:0/4132261232 shutdown_connections 2024-03-28T15:51:26.505 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:25.010+0000 7f1e21600700 1 --2- 172.21.15.67:0/4132261232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e1c1084e0 0x7f1e1c10a8d0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:26.506 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:25.010+0000 7f1e21600700 1 -- 172.21.15.67:0/4132261232 >> 172.21.15.67:0/4132261232 conn(0x7f1e1c101e70 msgr2=0x7f1e1c104290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:26.506 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:25.010+0000 7f1e21600700 1 -- 172.21.15.67:0/4132261232 shutdown_connections 2024-03-28T15:51:26.506 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:25.010+0000 7f1e21600700 1 -- 172.21.15.67:0/4132261232 wait complete. 2024-03-28T15:51:26.506 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:25.011+0000 7f1e21600700 1 Processor -- start 2024-03-28T15:51:26.506 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:25.011+0000 7f1e21600700 1 -- start start 2024-03-28T15:51:26.508 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:25.012+0000 7f1e21600700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e1c1084e0 0x7f1e1c107d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:26.508 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:25.012+0000 7f1e21600700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e1c10bd50 con 0x7f1e1c1084e0 2024-03-28T15:51:26.508 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:25.012+0000 7f1e1aa00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e1c1084e0 0x7f1e1c107d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:26.508 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:25.012+0000 7f1e1aa00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e1c1084e0 0x7f1e1c107d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49836/0 (socket says 172.21.15.67:49836) 2024-03-28T15:51:26.508 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:25.012+0000 7f1e1aa00700 1 -- 172.21.15.67:0/3386546634 learned_addr learned my addr 172.21.15.67:0/3386546634 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:51:26.509 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:25.013+0000 7f1e1aa00700 1 -- 172.21.15.67:0/3386546634 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e1c1082a0 con 0x7f1e1c1084e0 2024-03-28T15:51:26.509 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:25.013+0000 7f1e1aa00700 1 --2- 172.21.15.67:0/3386546634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e1c1084e0 0x7f1e1c107d60 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c011a00 tx=0x7f1e0c011ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:26.509 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:25.013+0000 7f1e13400700 1 -- 172.21.15.67:0/3386546634 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1e0c003af0 con 0x7f1e1c1084e0 2024-03-28T15:51:26.509 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:25.013+0000 7f1e13400700 1 -- 172.21.15.67:0/3386546634 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1e0c01a8e0 con 0x7f1e1c1084e0 2024-03-28T15:51:26.509 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:25.013+0000 7f1e21600700 1 -- 172.21.15.67:0/3386546634 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e1c106490 con 0x7f1e1c1084e0 2024-03-28T15:51:26.509 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:25.013+0000 7f1e13400700 1 -- 172.21.15.67:0/3386546634 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1e0c024840 con 0x7f1e1c1084e0 2024-03-28T15:51:26.509 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:25.013+0000 7f1e21600700 1 -- 172.21.15.67:0/3386546634 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e1c1069d0 con 0x7f1e1c1084e0 2024-03-28T15:51:26.509 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:25.015+0000 7f1e21600700 1 -- 172.21.15.67:0/3386546634 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e1c0637c0 con 0x7f1e1c1084e0 2024-03-28T15:51:26.509 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:25.015+0000 7f1e13400700 1 -- 172.21.15.67:0/3386546634 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f1e0c022020 con 0x7f1e1c1084e0 2024-03-28T15:51:26.509 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:25.015+0000 7f1e13400700 1 --2- 172.21.15.67:0/3386546634 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7f1e00038f70 0x7f1e0003b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:26.509 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:25.015+0000 7f1e13400700 1 -- 172.21.15.67:0/3386546634 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f1e0c04d1b0 con 0x7f1e1c1084e0 2024-03-28T15:51:26.509 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:25.016+0000 7f1e13e00700 1 --2- 172.21.15.67:0/3386546634 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7f1e00038f70 0x7f1e0003b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:26.509 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:25.017+0000 7f1e13e00700 1 --2- 172.21.15.67:0/3386546634 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7f1e00038f70 0x7f1e0003b430 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1e04006fd0 tx=0x7f1e04006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:26.509 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:25.021+0000 7f1e13400700 1 -- 172.21.15.67:0/3386546634 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1e0c02ddb0 con 0x7f1e1c1084e0 2024-03-28T15:51:26.509 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:25.305+0000 7f1e21600700 1 -- 172.21.15.67:0/3386546634 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f1e1c107270 con 0x7f1e1c1084e0 2024-03-28T15:51:26.510 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:25.308+0000 7f1e13400700 1 -- 172.21.15.67:0/3386546634 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f1e0c01a430 con 0x7f1e1c1084e0 2024-03-28T15:51:26.510 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:25.311+0000 7f1e21600700 1 -- 172.21.15.67:0/3386546634 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7f1e00038f70 msgr2=0x7f1e0003b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:26.511 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:25.311+0000 7f1e21600700 1 --2- 172.21.15.67:0/3386546634 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7f1e00038f70 0x7f1e0003b430 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1e04006fd0 tx=0x7f1e04006e40 comp rx=0 tx=0).stop 2024-03-28T15:51:26.511 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:25.311+0000 7f1e21600700 1 -- 172.21.15.67:0/3386546634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e1c1084e0 msgr2=0x7f1e1c107d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:26.511 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:25.311+0000 7f1e21600700 1 --2- 172.21.15.67:0/3386546634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e1c1084e0 0x7f1e1c107d60 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c011a00 tx=0x7f1e0c011ae0 comp rx=0 tx=0).stop 2024-03-28T15:51:26.511 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:25.311+0000 7f1e21600700 1 -- 172.21.15.67:0/3386546634 shutdown_connections 2024-03-28T15:51:26.511 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:25.311+0000 7f1e21600700 1 --2- 172.21.15.67:0/3386546634 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7f1e00038f70 0x7f1e0003b430 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:26.511 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:25.311+0000 7f1e21600700 1 --2- 172.21.15.67:0/3386546634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e1c1084e0 0x7f1e1c107d60 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:26.511 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:25.311+0000 7f1e21600700 1 -- 172.21.15.67:0/3386546634 >> 172.21.15.67:0/3386546634 conn(0x7f1e1c101e70 msgr2=0x7f1e1c1029d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:26.511 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:25.311+0000 7f1e21600700 1 -- 172.21.15.67:0/3386546634 shutdown_connections 2024-03-28T15:51:26.511 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:25.311+0000 7f1e21600700 1 -- 172.21.15.67:0/3386546634 wait complete. 2024-03-28T15:51:26.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:26 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/3386546634' entity='client.admin' 2024-03-28T15:51:26.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:26 smithi067 ceph-mon[31707]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:51:28.020 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:27 smithi067 ceph-mon[31707]: from='mgr.14120 172.21.15.67:0/2156043519' entity='mgr.x' 2024-03-28T15:51:28.020 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:27 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/629551669' entity='client.admin' 2024-03-28T15:51:28.031 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:27.094+0000 7f1b7ce00700 1 Processor -- start 2024-03-28T15:51:28.031 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:27.095+0000 7f1b7ce00700 1 -- start start 2024-03-28T15:51:28.031 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:27.095+0000 7f1b7ce00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b7807cd50 0x7f1b7807b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:28.031 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:27.095+0000 7f1b7ce00700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b7807b780 con 0x7f1b7807cd50 2024-03-28T15:51:28.031 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:27.096+0000 7f1b75e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b7807cd50 0x7f1b7807b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:28.035 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:27.096+0000 7f1b75e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b7807cd50 0x7f1b7807b1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49838/0 (socket says 172.21.15.67:49838) 2024-03-28T15:51:28.035 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:27.096+0000 7f1b75e00700 1 -- 172.21.15.67:0/2419383869 learned_addr learned my addr 172.21.15.67:0/2419383869 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:51:28.035 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:27.096+0000 7f1b75e00700 1 -- 172.21.15.67:0/2419383869 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b7807b900 con 0x7f1b7807cd50 2024-03-28T15:51:28.035 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:27.096+0000 7f1b75e00700 1 --2- 172.21.15.67:0/2419383869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b7807cd50 0x7f1b7807b1b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1b680098a0 tx=0x7f1b68009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a50189877be7868 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:28.035 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:27.097+0000 7f1b74a00700 1 -- 172.21.15.67:0/2419383869 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1b68004160 con 0x7f1b7807cd50 2024-03-28T15:51:28.035 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:27.097+0000 7f1b74a00700 1 -- 172.21.15.67:0/2419383869 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1b68004300 con 0x7f1b7807cd50 2024-03-28T15:51:28.035 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:27.097+0000 7f1b74a00700 1 -- 172.21.15.67:0/2419383869 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1b68011a50 con 0x7f1b7807cd50 2024-03-28T15:51:28.035 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:27.098+0000 7f1b7ce00700 1 -- 172.21.15.67:0/2419383869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b7807cd50 msgr2=0x7f1b7807b1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:28.036 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:27.098+0000 7f1b7ce00700 1 --2- 172.21.15.67:0/2419383869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b7807cd50 0x7f1b7807b1b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1b680098a0 tx=0x7f1b68009bb0 comp rx=0 tx=0).stop 2024-03-28T15:51:28.036 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:27.098+0000 7f1b7ce00700 1 -- 172.21.15.67:0/2419383869 shutdown_connections 2024-03-28T15:51:28.036 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:27.098+0000 7f1b7ce00700 1 --2- 172.21.15.67:0/2419383869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b7807cd50 0x7f1b7807b1b0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:28.036 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:27.098+0000 7f1b7ce00700 1 -- 172.21.15.67:0/2419383869 >> 172.21.15.67:0/2419383869 conn(0x7f1b78103000 msgr2=0x7f1b78105420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:28.036 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:27.098+0000 7f1b7ce00700 1 -- 172.21.15.67:0/2419383869 shutdown_connections 2024-03-28T15:51:28.036 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:27.098+0000 7f1b7ce00700 1 -- 172.21.15.67:0/2419383869 wait complete. 2024-03-28T15:51:28.036 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:27.099+0000 7f1b7ce00700 1 Processor -- start 2024-03-28T15:51:28.036 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:27.099+0000 7f1b7ce00700 1 -- start start 2024-03-28T15:51:28.036 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:27.100+0000 7f1b7ce00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b7807cd50 0x7f1b7819daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:28.036 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:27.100+0000 7f1b7ce00700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b7810dfe0 con 0x7f1b7807cd50 2024-03-28T15:51:28.036 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:27.100+0000 7f1b75e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b7807cd50 0x7f1b7819daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:28.036 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:27.100+0000 7f1b75e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b7807cd50 0x7f1b7819daa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49842/0 (socket says 172.21.15.67:49842) 2024-03-28T15:51:28.036 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:27.100+0000 7f1b75e00700 1 -- 172.21.15.67:0/629551669 learned_addr learned my addr 172.21.15.67:0/629551669 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:51:28.036 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:27.101+0000 7f1b75e00700 1 -- 172.21.15.67:0/629551669 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b7819dfe0 con 0x7f1b7807cd50 2024-03-28T15:51:28.036 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:27.101+0000 7f1b75e00700 1 --2- 172.21.15.67:0/629551669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b7807cd50 0x7f1b7819daa0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1b68006b20 tx=0x7f1b68011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:28.037 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:27.101+0000 7f1b6ea00700 1 -- 172.21.15.67:0/629551669 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1b680039e0 con 0x7f1b7807cd50 2024-03-28T15:51:28.037 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:27.101+0000 7f1b6ea00700 1 -- 172.21.15.67:0/629551669 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1b6801a8e0 con 0x7f1b7807cd50 2024-03-28T15:51:28.037 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:27.101+0000 7f1b7ce00700 1 -- 172.21.15.67:0/629551669 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b7819e270 con 0x7f1b7807cd50 2024-03-28T15:51:28.037 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:27.101+0000 7f1b6ea00700 1 -- 172.21.15.67:0/629551669 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1b680247e0 con 0x7f1b7807cd50 2024-03-28T15:51:28.037 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:27.102+0000 7f1b7ce00700 1 -- 172.21.15.67:0/629551669 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b781a0f60 con 0x7f1b7807cd50 2024-03-28T15:51:28.037 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:27.103+0000 7f1b7ce00700 1 -- 172.21.15.67:0/629551669 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b780637c0 con 0x7f1b7807cd50 2024-03-28T15:51:28.037 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:27.103+0000 7f1b6ea00700 1 -- 172.21.15.67:0/629551669 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f1b68022020 con 0x7f1b7807cd50 2024-03-28T15:51:28.037 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:27.103+0000 7f1b6ea00700 1 --2- 172.21.15.67:0/629551669 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7f1b5c038fc0 0x7f1b5c03b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:28.037 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:27.103+0000 7f1b6ea00700 1 -- 172.21.15.67:0/629551669 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f1b6804d160 con 0x7f1b7807cd50 2024-03-28T15:51:28.037 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:27.104+0000 7f1b75400700 1 --2- 172.21.15.67:0/629551669 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7f1b5c038fc0 0x7f1b5c03b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:28.037 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:27.104+0000 7f1b75400700 1 --2- 172.21.15.67:0/629551669 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7f1b5c038fc0 0x7f1b5c03b480 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f1b64006fd0 tx=0x7f1b64006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:28.037 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:27.109+0000 7f1b6ea00700 1 -- 172.21.15.67:0/629551669 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1b68016260 con 0x7f1b7807cd50 2024-03-28T15:51:28.037 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:27.350+0000 7f1b7ce00700 1 -- 172.21.15.67:0/629551669 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f1b781a1580 con 0x7f1b7807cd50 2024-03-28T15:51:28.038 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:27.354+0000 7f1b6ea00700 1 -- 172.21.15.67:0/629551669 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f1b6802ad60 con 0x7f1b7807cd50 2024-03-28T15:51:28.038 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:27.357+0000 7f1b7ce00700 1 -- 172.21.15.67:0/629551669 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7f1b5c038fc0 msgr2=0x7f1b5c03b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:28.038 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:27.357+0000 7f1b7ce00700 1 --2- 172.21.15.67:0/629551669 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7f1b5c038fc0 0x7f1b5c03b480 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f1b64006fd0 tx=0x7f1b64006e40 comp rx=0 tx=0).stop 2024-03-28T15:51:28.038 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:27.357+0000 7f1b7ce00700 1 -- 172.21.15.67:0/629551669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b7807cd50 msgr2=0x7f1b7819daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:28.038 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:27.357+0000 7f1b7ce00700 1 --2- 172.21.15.67:0/629551669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b7807cd50 0x7f1b7819daa0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1b68006b20 tx=0x7f1b68011ad0 comp rx=0 tx=0).stop 2024-03-28T15:51:28.038 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:27.357+0000 7f1b7ce00700 1 -- 172.21.15.67:0/629551669 shutdown_connections 2024-03-28T15:51:28.038 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:27.357+0000 7f1b7ce00700 1 --2- 172.21.15.67:0/629551669 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7f1b5c038fc0 0x7f1b5c03b480 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:28.039 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:27.357+0000 7f1b7ce00700 1 --2- 172.21.15.67:0/629551669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b7807cd50 0x7f1b7819daa0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:28.039 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:27.357+0000 7f1b7ce00700 1 -- 172.21.15.67:0/629551669 >> 172.21.15.67:0/629551669 conn(0x7f1b78103000 msgr2=0x7f1b78103ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:28.039 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:27.358+0000 7f1b7ce00700 1 -- 172.21.15.67:0/629551669 shutdown_connections 2024-03-28T15:51:28.039 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:27.358+0000 7f1b7ce00700 1 -- 172.21.15.67:0/629551669 wait complete. 2024-03-28T15:51:28.039 INFO:teuthology.orchestra.run.smithi067.stdout:Enabling the dashboard module... 2024-03-28T15:51:28.943 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:28 smithi067 ceph-mon[31707]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:51:29.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:29 smithi067 ceph-mon[31707]: from='mgr.14120 172.21.15.67:0/2156043519' entity='mgr.x' 2024-03-28T15:51:29.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:29 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/1912417399' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-28T15:51:29.955 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:51:29 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: ignoring --setuser ceph since I am not root 2024-03-28T15:51:29.955 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:51:29 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: ignoring --setgroup ceph since I am not root 2024-03-28T15:51:30.661 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:51:30 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:51:30.535+0000 7f4ffb273200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T15:51:30.661 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:51:30 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:51:30.660+0000 7f4ffb273200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T15:51:31.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:30 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/1912417399' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-28T15:51:31.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:30 smithi067 ceph-mon[31707]: mgrmap e9: x(active, since 25s) 2024-03-28T15:51:31.117 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:28.617+0000 7f7fff000700 1 Processor -- start 2024-03-28T15:51:31.117 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:28.617+0000 7f7fff000700 1 -- start start 2024-03-28T15:51:31.117 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:28.618+0000 7f7fff000700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ff8108430 0x7f7ff810a820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:31.117 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:28.618+0000 7f7fff000700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff8063470 con 0x7f7ff8108430 2024-03-28T15:51:31.117 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:28.618+0000 7f7ff7e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ff8108430 0x7f7ff810a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:31.118 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:28.618+0000 7f7ff7e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ff8108430 0x7f7ff810a820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49844/0 (socket says 172.21.15.67:49844) 2024-03-28T15:51:31.118 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:28.619+0000 7f7ff7e00700 1 -- 172.21.15.67:0/1956913817 learned_addr learned my addr 172.21.15.67:0/1956913817 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:51:31.118 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:28.619+0000 7f7ff7e00700 1 -- 172.21.15.67:0/1956913817 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ff810ad60 con 0x7f7ff8108430 2024-03-28T15:51:31.118 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:28.619+0000 7f7ff7e00700 1 --2- 172.21.15.67:0/1956913817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ff8108430 0x7f7ff810a820 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7fe80098a0 tx=0x7f7fe8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2d37f7c6b62541c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:31.118 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:28.621+0000 7f7ff6a00700 1 -- 172.21.15.67:0/1956913817 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7fe8004160 con 0x7f7ff8108430 2024-03-28T15:51:31.118 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:28.621+0000 7f7ff6a00700 1 -- 172.21.15.67:0/1956913817 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7fe8004300 con 0x7f7ff8108430 2024-03-28T15:51:31.118 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:28.621+0000 7f7ff6a00700 1 -- 172.21.15.67:0/1956913817 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7fe8011a50 con 0x7f7ff8108430 2024-03-28T15:51:31.118 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:28.622+0000 7f7fff000700 1 -- 172.21.15.67:0/1956913817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ff8108430 msgr2=0x7f7ff810a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:31.118 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:28.622+0000 7f7fff000700 1 --2- 172.21.15.67:0/1956913817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ff8108430 0x7f7ff810a820 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7fe80098a0 tx=0x7f7fe8009bb0 comp rx=0 tx=0).stop 2024-03-28T15:51:31.118 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:28.622+0000 7f7fff000700 1 -- 172.21.15.67:0/1956913817 shutdown_connections 2024-03-28T15:51:31.118 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:28.622+0000 7f7fff000700 1 --2- 172.21.15.67:0/1956913817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ff8108430 0x7f7ff810a820 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:31.118 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:28.622+0000 7f7fff000700 1 -- 172.21.15.67:0/1956913817 >> 172.21.15.67:0/1956913817 conn(0x7f7ff8101e40 msgr2=0x7f7ff8104260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:31.119 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:28.622+0000 7f7fff000700 1 -- 172.21.15.67:0/1956913817 shutdown_connections 2024-03-28T15:51:31.119 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:28.622+0000 7f7fff000700 1 -- 172.21.15.67:0/1956913817 wait complete. 2024-03-28T15:51:31.119 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:28.623+0000 7f7fff000700 1 Processor -- start 2024-03-28T15:51:31.119 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:28.623+0000 7f7fff000700 1 -- start start 2024-03-28T15:51:31.119 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:28.624+0000 7f7fff000700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ff8108430 0x7f7ff8107cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:31.119 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:28.624+0000 7f7fff000700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff810bd60 con 0x7f7ff8108430 2024-03-28T15:51:31.119 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:28.624+0000 7f7ff7e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ff8108430 0x7f7ff8107cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:31.119 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:28.624+0000 7f7ff7e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ff8108430 0x7f7ff8107cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49858/0 (socket says 172.21.15.67:49858) 2024-03-28T15:51:31.119 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:28.624+0000 7f7ff7e00700 1 -- 172.21.15.67:0/1912417399 learned_addr learned my addr 172.21.15.67:0/1912417399 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:51:31.119 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:28.624+0000 7f7ff7e00700 1 -- 172.21.15.67:0/1912417399 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ff81081f0 con 0x7f7ff8108430 2024-03-28T15:51:31.119 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:28.625+0000 7f7ff7e00700 1 --2- 172.21.15.67:0/1912417399 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ff8108430 0x7f7ff8107cb0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7fe8011a00 tx=0x7f7fe8011ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:31.119 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:28.625+0000 7f7ff4c00700 1 -- 172.21.15.67:0/1912417399 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7fe8003af0 con 0x7f7ff8108430 2024-03-28T15:51:31.119 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:28.625+0000 7f7ff4c00700 1 -- 172.21.15.67:0/1912417399 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7fe801a8e0 con 0x7f7ff8108430 2024-03-28T15:51:31.119 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:28.625+0000 7f7ff4c00700 1 -- 172.21.15.67:0/1912417399 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7fe8024840 con 0x7f7ff8108430 2024-03-28T15:51:31.120 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:28.625+0000 7f7fff000700 1 -- 172.21.15.67:0/1912417399 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ff81063e0 con 0x7f7ff8108430 2024-03-28T15:51:31.120 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:28.625+0000 7f7fff000700 1 -- 172.21.15.67:0/1912417399 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ff8106920 con 0x7f7ff8108430 2024-03-28T15:51:31.120 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:28.626+0000 7f7ff4c00700 1 -- 172.21.15.67:0/1912417399 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f7fe8022020 con 0x7f7ff8108430 2024-03-28T15:51:31.120 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:28.626+0000 7f7fff000700 1 -- 172.21.15.67:0/1912417399 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ff80637c0 con 0x7f7ff8108430 2024-03-28T15:51:31.120 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:28.626+0000 7f7ff4c00700 1 --2- 172.21.15.67:0/1912417399 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7f7fdc038f70 0x7f7fdc03b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:31.120 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:28.626+0000 7f7ff4c00700 1 -- 172.21.15.67:0/1912417399 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f7fe804d1b0 con 0x7f7ff8108430 2024-03-28T15:51:31.120 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:28.627+0000 7f7ff7400700 1 --2- 172.21.15.67:0/1912417399 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7f7fdc038f70 0x7f7fdc03b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:31.120 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:28.627+0000 7f7ff7400700 1 --2- 172.21.15.67:0/1912417399 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7f7fdc038f70 0x7f7fdc03b430 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7fe4006fd0 tx=0x7f7fe4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:31.120 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:28.632+0000 7f7ff4c00700 1 -- 172.21.15.67:0/1912417399 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7fe802ddb0 con 0x7f7ff8108430 2024-03-28T15:51:31.120 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:28.941+0000 7f7fff000700 1 -- 172.21.15.67:0/1912417399 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f7ff8106ba0 con 0x7f7ff8108430 2024-03-28T15:51:31.120 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:29.667+0000 7f7ff4c00700 1 -- 172.21.15.67:0/1912417399 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f7fe801a430 con 0x7f7ff8108430 2024-03-28T15:51:31.120 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:29.672+0000 7f7ff4c00700 1 -- 172.21.15.67:0/1912417399 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mgrmap(e 9) v1 ==== 46119+0+0 (secure 0 0 0) 0x7f7fe80212c0 con 0x7f7ff8108430 2024-03-28T15:51:31.120 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:29.673+0000 7f7fff000700 1 -- 172.21.15.67:0/1912417399 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7f7fdc038f70 msgr2=0x7f7fdc03b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:31.120 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:29.673+0000 7f7fff000700 1 --2- 172.21.15.67:0/1912417399 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7f7fdc038f70 0x7f7fdc03b430 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7fe4006fd0 tx=0x7f7fe4006e40 comp rx=0 tx=0).stop 2024-03-28T15:51:31.121 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:29.673+0000 7f7fff000700 1 -- 172.21.15.67:0/1912417399 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ff8108430 msgr2=0x7f7ff8107cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:31.121 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:29.673+0000 7f7fff000700 1 --2- 172.21.15.67:0/1912417399 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ff8108430 0x7f7ff8107cb0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7fe8011a00 tx=0x7f7fe8011ae0 comp rx=0 tx=0).stop 2024-03-28T15:51:31.121 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:29.673+0000 7f7fff000700 1 -- 172.21.15.67:0/1912417399 shutdown_connections 2024-03-28T15:51:31.121 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:29.673+0000 7f7fff000700 1 --2- 172.21.15.67:0/1912417399 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7f7fdc038f70 0x7f7fdc03b430 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:31.121 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:29.673+0000 7f7fff000700 1 --2- 172.21.15.67:0/1912417399 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ff8108430 0x7f7ff8107cb0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:31.121 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:29.673+0000 7f7fff000700 1 -- 172.21.15.67:0/1912417399 >> 172.21.15.67:0/1912417399 conn(0x7f7ff8101e40 msgr2=0x7f7ff8102970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:31.121 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:29.674+0000 7f7fff000700 1 -- 172.21.15.67:0/1912417399 shutdown_connections 2024-03-28T15:51:31.121 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:29.674+0000 7f7fff000700 1 -- 172.21.15.67:0/1912417399 wait complete. 2024-03-28T15:51:31.668 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:51:31 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:51:31.483+0000 7f4ffb273200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T15:51:31.668 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:51:31 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:51:31.613+0000 7f4ffb273200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T15:51:31.928 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:51:31 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:51:31.798+0000 7f4ffb273200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T15:51:31.929 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:51:31 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:51:31.927+0000 7f4ffb273200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T15:51:32.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:32 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/3034351996' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-28T15:51:32.319 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:51:32 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:51:32.047+0000 7f4ffb273200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T15:51:32.668 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:51:32 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:51:32.406+0000 7f4ffb273200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T15:51:32.668 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:51:32 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:51:32.551+0000 7f4ffb273200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T15:51:32.668 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:51:32 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:51:32.667+0000 7f4ffb273200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T15:51:32.687 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout { 2024-03-28T15:51:32.687 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-28T15:51:32.687 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-28T15:51:32.688 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-28T15:51:32.688 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-28T15:51:32.688 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout } 2024-03-28T15:51:32.688 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:31.667+0000 7fe0fbc00700 1 Processor -- start 2024-03-28T15:51:32.688 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:31.667+0000 7fe0fbc00700 1 -- start start 2024-03-28T15:51:32.688 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:31.668+0000 7fe0fbc00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0f4106270 0x7fe0f4106690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:32.688 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:31.668+0000 7fe0fbc00700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f4063430 con 0x7fe0f4106270 2024-03-28T15:51:32.688 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:31.669+0000 7fe0f9200700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0f4106270 0x7fe0f4106690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:32.688 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:31.669+0000 7fe0f9200700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0f4106270 0x7fe0f4106690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34766/0 (socket says 172.21.15.67:34766) 2024-03-28T15:51:32.688 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:31.669+0000 7fe0f9200700 1 -- 172.21.15.67:0/3672931353 learned_addr learned my addr 172.21.15.67:0/3672931353 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:51:32.688 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:31.669+0000 7fe0f9200700 1 -- 172.21.15.67:0/3672931353 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0f4106bd0 con 0x7fe0f4106270 2024-03-28T15:51:32.688 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:31.670+0000 7fe0f9200700 1 --2- 172.21.15.67:0/3672931353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0f4106270 0x7fe0f4106690 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe0e80098a0 tx=0x7fe0e8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e1d691be14b6f4a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:32.688 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:31.670+0000 7fe0f3400700 1 -- 172.21.15.67:0/3672931353 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe0e8004160 con 0x7fe0f4106270 2024-03-28T15:51:32.688 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:31.670+0000 7fe0f3400700 1 -- 172.21.15.67:0/3672931353 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe0e8004300 con 0x7fe0f4106270 2024-03-28T15:51:32.689 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:31.670+0000 7fe0f3400700 1 -- 172.21.15.67:0/3672931353 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe0e8011a50 con 0x7fe0f4106270 2024-03-28T15:51:32.689 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:31.671+0000 7fe0fbc00700 1 -- 172.21.15.67:0/3672931353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0f4106270 msgr2=0x7fe0f4106690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:32.689 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:31.671+0000 7fe0fbc00700 1 --2- 172.21.15.67:0/3672931353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0f4106270 0x7fe0f4106690 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe0e80098a0 tx=0x7fe0e8009bb0 comp rx=0 tx=0).stop 2024-03-28T15:51:32.689 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:31.671+0000 7fe0fbc00700 1 -- 172.21.15.67:0/3672931353 shutdown_connections 2024-03-28T15:51:32.689 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:31.671+0000 7fe0fbc00700 1 --2- 172.21.15.67:0/3672931353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0f4106270 0x7fe0f4106690 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:32.689 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:31.671+0000 7fe0fbc00700 1 -- 172.21.15.67:0/3672931353 >> 172.21.15.67:0/3672931353 conn(0x7fe0f4101e70 msgr2=0x7fe0f4104290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:32.689 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:31.671+0000 7fe0fbc00700 1 -- 172.21.15.67:0/3672931353 shutdown_connections 2024-03-28T15:51:32.689 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:31.671+0000 7fe0fbc00700 1 -- 172.21.15.67:0/3672931353 wait complete. 2024-03-28T15:51:32.689 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:31.672+0000 7fe0fbc00700 1 Processor -- start 2024-03-28T15:51:32.689 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:31.672+0000 7fe0fbc00700 1 -- start start 2024-03-28T15:51:32.689 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:31.672+0000 7fe0fbc00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0f4106270 0x7fe0f419da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:32.689 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:31.672+0000 7fe0fbc00700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f4107c40 con 0x7fe0f4106270 2024-03-28T15:51:32.689 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:31.673+0000 7fe0f9200700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0f4106270 0x7fe0f419da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:32.689 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:31.673+0000 7fe0f9200700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0f4106270 0x7fe0f419da80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34782/0 (socket says 172.21.15.67:34782) 2024-03-28T15:51:32.690 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:31.673+0000 7fe0f9200700 1 -- 172.21.15.67:0/3034351996 learned_addr learned my addr 172.21.15.67:0/3034351996 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:51:32.690 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:31.673+0000 7fe0f9200700 1 -- 172.21.15.67:0/3034351996 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0f419dfc0 con 0x7fe0f4106270 2024-03-28T15:51:32.690 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:31.673+0000 7fe0f9200700 1 --2- 172.21.15.67:0/3034351996 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0f4106270 0x7fe0f419da80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe0e8011a00 tx=0x7fe0e8011ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:32.690 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:31.674+0000 7fe0f1600700 1 -- 172.21.15.67:0/3034351996 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe0e8003af0 con 0x7fe0f4106270 2024-03-28T15:51:32.690 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:31.674+0000 7fe0f1600700 1 -- 172.21.15.67:0/3034351996 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe0e801a8e0 con 0x7fe0f4106270 2024-03-28T15:51:32.690 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:31.674+0000 7fe0fbc00700 1 -- 172.21.15.67:0/3034351996 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0f419e250 con 0x7fe0f4106270 2024-03-28T15:51:32.690 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:31.674+0000 7fe0f1600700 1 -- 172.21.15.67:0/3034351996 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe0e8024840 con 0x7fe0f4106270 2024-03-28T15:51:32.690 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:31.674+0000 7fe0fbc00700 1 -- 172.21.15.67:0/3034351996 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0f41a0f40 con 0x7fe0f4106270 2024-03-28T15:51:32.690 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:31.675+0000 7fe0f1600700 1 -- 172.21.15.67:0/3034351996 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 9) v1 ==== 46119+0+0 (secure 0 0 0) 0x7fe0e8022020 con 0x7fe0f4106270 2024-03-28T15:51:32.690 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:31.675+0000 7fe0e6a00700 1 -- 172.21.15.67:0/3034351996 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0d0005330 con 0x7fe0f4106270 2024-03-28T15:51:32.690 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:31.675+0000 7fe0f1600700 1 --2- 172.21.15.67:0/3034351996 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7fe0dc038f70 0x7fe0dc03b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:32.690 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:31.675+0000 7fe0f1600700 1 -- 172.21.15.67:0/3034351996 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fe0e804d2f0 con 0x7fe0f4106270 2024-03-28T15:51:32.690 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:31.676+0000 7fe0f3e00700 1 -- 172.21.15.67:0/3034351996 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7fe0dc038f70 msgr2=0x7fe0dc03b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/1631001968 2024-03-28T15:51:32.691 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:31.676+0000 7fe0f3e00700 1 --2- 172.21.15.67:0/3034351996 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7fe0dc038f70 0x7fe0dc03b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T15:51:32.691 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:31.681+0000 7fe0f1600700 1 -- 172.21.15.67:0/3034351996 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe0e8016260 con 0x7fe0f4106270 2024-03-28T15:51:32.691 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:31.877+0000 7fe0f3e00700 1 -- 172.21.15.67:0/3034351996 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7fe0dc038f70 msgr2=0x7fe0dc03b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/1631001968 2024-03-28T15:51:32.691 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:31.877+0000 7fe0f3e00700 1 --2- 172.21.15.67:0/3034351996 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7fe0dc038f70 0x7fe0dc03b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T15:51:32.691 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:31.967+0000 7fe0e6a00700 1 -- 172.21.15.67:0/3034351996 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fe0d0005e30 con 0x7fe0f4106270 2024-03-28T15:51:32.691 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:31.967+0000 7fe0f1600700 1 -- 172.21.15.67:0/3034351996 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7fe0e804cc40 con 0x7fe0f4106270 2024-03-28T15:51:32.691 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:31.970+0000 7fe0e6a00700 1 -- 172.21.15.67:0/3034351996 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7fe0dc038f70 msgr2=0x7fe0dc03b430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T15:51:32.691 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:31.970+0000 7fe0e6a00700 1 --2- 172.21.15.67:0/3034351996 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7fe0dc038f70 0x7fe0dc03b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:32.691 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:31.970+0000 7fe0e6a00700 1 -- 172.21.15.67:0/3034351996 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0f4106270 msgr2=0x7fe0f419da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:32.691 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:31.970+0000 7fe0e6a00700 1 --2- 172.21.15.67:0/3034351996 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0f4106270 0x7fe0f419da80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe0e8011a00 tx=0x7fe0e8011ae0 comp rx=0 tx=0).stop 2024-03-28T15:51:32.691 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:31.970+0000 7fe0e6a00700 1 -- 172.21.15.67:0/3034351996 shutdown_connections 2024-03-28T15:51:32.691 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:31.970+0000 7fe0e6a00700 1 --2- 172.21.15.67:0/3034351996 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7fe0dc038f70 0x7fe0dc03b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:32.691 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:31.970+0000 7fe0e6a00700 1 --2- 172.21.15.67:0/3034351996 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0f4106270 0x7fe0f419da80 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:32.691 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:31.970+0000 7fe0e6a00700 1 -- 172.21.15.67:0/3034351996 >> 172.21.15.67:0/3034351996 conn(0x7fe0f4101e70 msgr2=0x7fe0f41946e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:32.691 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:31.970+0000 7fe0e6a00700 1 -- 172.21.15.67:0/3034351996 shutdown_connections 2024-03-28T15:51:32.692 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:31.971+0000 7fe0e6a00700 1 -- 172.21.15.67:0/3034351996 wait complete. 2024-03-28T15:51:32.692 INFO:teuthology.orchestra.run.smithi067.stdout:Waiting for the mgr to restart... 2024-03-28T15:51:32.692 INFO:teuthology.orchestra.run.smithi067.stdout:Waiting for mgr epoch 9... 2024-03-28T15:51:32.974 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:51:32 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:51:32.813+0000 7f4ffb273200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T15:51:33.286 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:51:33 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:51:33.039+0000 7f4ffb273200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T15:51:34.696 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:51:34 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:51:34.358+0000 7f4ffb273200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T15:51:34.696 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:51:34 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:51:34.534+0000 7f4ffb273200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T15:51:35.319 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:51:34 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:51:34.962+0000 7f4ffb273200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T15:51:35.319 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:51:35 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:51:35.084+0000 7f4ffb273200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T15:51:35.616 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:51:35 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:51:35.351+0000 7f4ffb273200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T15:51:35.974 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:51:35 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:51:35.616+0000 7f4ffb273200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T15:51:36.297 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:51:36 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:51:36.012+0000 7f4ffb273200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T15:51:36.569 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:51:36 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:51:36.338+0000 7f4ffb273200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T15:51:37.069 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:51:36 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:51:36.654+0000 7f4ffb273200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T15:51:38.069 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:51:37 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:51:37.727+0000 7f4ffb273200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T15:51:38.819 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:51:38 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:51:38.444+0000 7f4ffb273200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T15:51:38.819 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:51:38 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:51:38.580+0000 7f4ffb273200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T15:51:39.319 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:51:38 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:51:38.872+0000 7f4ffb273200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T15:51:39.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:38 smithi067 ceph-mon[31707]: Active manager daemon x restarted 2024-03-28T15:51:39.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:38 smithi067 ceph-mon[31707]: Activating manager daemon x 2024-03-28T15:51:39.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:38 smithi067 ceph-mon[31707]: osdmap e3: 0 total, 0 up, 0 in 2024-03-28T15:51:39.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:38 smithi067 ceph-mon[31707]: mgrmap e10: x(active, starting, since 0.00479014s) 2024-03-28T15:51:39.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:38 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T15:51:39.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:38 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-28T15:51:39.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:38 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-28T15:51:39.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:38 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-28T15:51:39.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:38 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-28T15:51:39.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:38 smithi067 ceph-mon[31707]: Manager daemon x is now available 2024-03-28T15:51:40.291 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:39 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:51:40.291 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:39 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:51:40.291 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:39 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-28T15:51:40.292 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:39 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-28T15:51:40.292 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:39 smithi067 ceph-mon[31707]: [28/Mar/2024:15:51:39] ENGINE Bus STARTING 2024-03-28T15:51:40.292 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:39 smithi067 ceph-mon[31707]: [28/Mar/2024:15:51:39] ENGINE Serving on http://172.21.15.67:8765 2024-03-28T15:51:40.292 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:39 smithi067 ceph-mon[31707]: [28/Mar/2024:15:51:39] ENGINE Serving on https://172.21.15.67:7150 2024-03-28T15:51:40.292 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:39 smithi067 ceph-mon[31707]: [28/Mar/2024:15:51:39] ENGINE Bus STARTED 2024-03-28T15:51:40.292 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:39 smithi067 ceph-mon[31707]: mgrmap e11: x(active, since 1.01025s) 2024-03-28T15:51:40.610 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout { 2024-03-28T15:51:40.613 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-28T15:51:40.613 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-28T15:51:40.614 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout } 2024-03-28T15:51:40.614 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:33.283+0000 7f4cb5000700 1 Processor -- start 2024-03-28T15:51:40.614 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:33.284+0000 7f4cb5000700 1 -- start start 2024-03-28T15:51:40.614 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:33.284+0000 7f4cb5000700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4cb007cd50 0x7f4cb007b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:40.614 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:33.284+0000 7f4cb5000700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb007b780 con 0x7f4cb007cd50 2024-03-28T15:51:40.614 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:33.285+0000 7f4cade00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4cb007cd50 0x7f4cb007b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:40.614 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:33.285+0000 7f4cade00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4cb007cd50 0x7f4cb007b1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34798/0 (socket says 172.21.15.67:34798) 2024-03-28T15:51:40.614 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:33.285+0000 7f4cade00700 1 -- 172.21.15.67:0/2906676185 learned_addr learned my addr 172.21.15.67:0/2906676185 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:51:40.614 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:33.285+0000 7f4cade00700 1 -- 172.21.15.67:0/2906676185 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cb007b900 con 0x7f4cb007cd50 2024-03-28T15:51:40.614 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:33.285+0000 7f4cade00700 1 --2- 172.21.15.67:0/2906676185 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4cb007cd50 0x7f4cb007b1b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4ca00098a0 tx=0x7f4ca0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=feb912a2d60ca389 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:40.614 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:33.286+0000 7f4caca00700 1 -- 172.21.15.67:0/2906676185 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4ca0004160 con 0x7f4cb007cd50 2024-03-28T15:51:40.614 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:33.286+0000 7f4caca00700 1 -- 172.21.15.67:0/2906676185 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4ca0004300 con 0x7f4cb007cd50 2024-03-28T15:51:40.615 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:33.286+0000 7f4caca00700 1 -- 172.21.15.67:0/2906676185 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4ca0011a50 con 0x7f4cb007cd50 2024-03-28T15:51:40.615 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:33.286+0000 7f4cb5000700 1 -- 172.21.15.67:0/2906676185 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4cb007cd50 msgr2=0x7f4cb007b1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:40.615 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:33.286+0000 7f4cb5000700 1 --2- 172.21.15.67:0/2906676185 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4cb007cd50 0x7f4cb007b1b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4ca00098a0 tx=0x7f4ca0009bb0 comp rx=0 tx=0).stop 2024-03-28T15:51:40.615 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:33.287+0000 7f4cb5000700 1 -- 172.21.15.67:0/2906676185 shutdown_connections 2024-03-28T15:51:40.615 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:33.287+0000 7f4cb5000700 1 --2- 172.21.15.67:0/2906676185 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4cb007cd50 0x7f4cb007b1b0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:40.615 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:33.287+0000 7f4cb5000700 1 -- 172.21.15.67:0/2906676185 >> 172.21.15.67:0/2906676185 conn(0x7f4cb0102f70 msgr2=0x7f4cb0105390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:40.615 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:33.287+0000 7f4cb5000700 1 -- 172.21.15.67:0/2906676185 shutdown_connections 2024-03-28T15:51:40.615 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:33.287+0000 7f4cb5000700 1 -- 172.21.15.67:0/2906676185 wait complete. 2024-03-28T15:51:40.615 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:33.288+0000 7f4cb5000700 1 Processor -- start 2024-03-28T15:51:40.615 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:33.288+0000 7f4cb5000700 1 -- start start 2024-03-28T15:51:40.618 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:33.288+0000 7f4cb5000700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4cb007cd50 0x7f4cb01a1e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:40.618 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:33.288+0000 7f4cb5000700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb0109b40 con 0x7f4cb007cd50 2024-03-28T15:51:40.618 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:33.289+0000 7f4cade00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4cb007cd50 0x7f4cb01a1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:40.619 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:33.289+0000 7f4cade00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4cb007cd50 0x7f4cb01a1e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34806/0 (socket says 172.21.15.67:34806) 2024-03-28T15:51:40.619 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:33.289+0000 7f4cade00700 1 -- 172.21.15.67:0/2038982424 learned_addr learned my addr 172.21.15.67:0/2038982424 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:51:40.619 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:33.289+0000 7f4cade00700 1 -- 172.21.15.67:0/2038982424 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cb01a23c0 con 0x7f4cb007cd50 2024-03-28T15:51:40.619 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:33.289+0000 7f4cade00700 1 --2- 172.21.15.67:0/2038982424 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4cb007cd50 0x7f4cb01a1e80 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4ca0006ba0 tx=0x7f4ca0011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:40.619 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:33.290+0000 7f4ca6a00700 1 -- 172.21.15.67:0/2038982424 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4ca0003af0 con 0x7f4cb007cd50 2024-03-28T15:51:40.619 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:33.290+0000 7f4ca6a00700 1 -- 172.21.15.67:0/2038982424 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4ca001a8e0 con 0x7f4cb007cd50 2024-03-28T15:51:40.619 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:33.290+0000 7f4cb5000700 1 -- 172.21.15.67:0/2038982424 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cb01a2650 con 0x7f4cb007cd50 2024-03-28T15:51:40.619 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:33.290+0000 7f4ca6a00700 1 -- 172.21.15.67:0/2038982424 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4ca0024900 con 0x7f4cb007cd50 2024-03-28T15:51:40.619 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:33.290+0000 7f4cb5000700 1 -- 172.21.15.67:0/2038982424 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cb01a5340 con 0x7f4cb007cd50 2024-03-28T15:51:40.619 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:33.291+0000 7f4ca6a00700 1 -- 172.21.15.67:0/2038982424 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 9) v1 ==== 46119+0+0 (secure 0 0 0) 0x7f4ca0022020 con 0x7f4cb007cd50 2024-03-28T15:51:40.619 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:33.292+0000 7f4ca6a00700 1 --2- 172.21.15.67:0/2038982424 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7f4c94038f20 0x7f4c9403b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:40.619 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:33.292+0000 7f4ca6a00700 1 -- 172.21.15.67:0/2038982424 --> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4c9403baf0 con 0x7f4c94038f20 2024-03-28T15:51:40.619 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:33.292+0000 7f4ca6a00700 1 -- 172.21.15.67:0/2038982424 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f4ca004d360 con 0x7f4cb007cd50 2024-03-28T15:51:40.619 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:33.292+0000 7f4cad400700 1 -- 172.21.15.67:0/2038982424 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7f4c94038f20 msgr2=0x7f4c9403b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/1631001968 2024-03-28T15:51:40.620 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:33.292+0000 7f4cad400700 1 --2- 172.21.15.67:0/2038982424 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7f4c94038f20 0x7f4c9403b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T15:51:40.620 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:33.493+0000 7f4cad400700 1 -- 172.21.15.67:0/2038982424 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7f4c94038f20 msgr2=0x7f4c9403b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/1631001968 2024-03-28T15:51:40.620 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:33.493+0000 7f4cad400700 1 --2- 172.21.15.67:0/2038982424 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7f4c94038f20 0x7f4c9403b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T15:51:40.620 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:33.894+0000 7f4cad400700 1 -- 172.21.15.67:0/2038982424 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7f4c94038f20 msgr2=0x7f4c9403b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/1631001968 2024-03-28T15:51:40.620 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:33.894+0000 7f4cad400700 1 --2- 172.21.15.67:0/2038982424 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7f4c94038f20 0x7f4c9403b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T15:51:40.620 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:34.695+0000 7f4cad400700 1 -- 172.21.15.67:0/2038982424 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7f4c94038f20 msgr2=0x7f4c9403b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/1631001968 2024-03-28T15:51:40.620 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:34.695+0000 7f4cad400700 1 --2- 172.21.15.67:0/2038982424 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7f4c94038f20 0x7f4c9403b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T15:51:40.622 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:36.297+0000 7f4cad400700 1 -- 172.21.15.67:0/2038982424 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7f4c94038f20 msgr2=0x7f4c9403b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/1631001968 2024-03-28T15:51:40.622 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:36.297+0000 7f4cad400700 1 --2- 172.21.15.67:0/2038982424 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7f4c94038f20 0x7f4c9403b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-28T15:51:40.622 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:38.879+0000 7f4ca6a00700 1 -- 172.21.15.67:0/2038982424 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mgrmap(e 10) v1 ==== 45886+0+0 (secure 0 0 0) 0x7f4ca000b3e0 con 0x7f4cb007cd50 2024-03-28T15:51:40.622 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:38.880+0000 7f4ca6a00700 1 -- 172.21.15.67:0/2038982424 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7f4c94038f20 msgr2=0x7f4c9403b3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T15:51:40.623 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:38.880+0000 7f4ca6a00700 1 --2- 172.21.15.67:0/2038982424 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7f4c94038f20 0x7f4c9403b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:40.623 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:39.882+0000 7f4ca6a00700 1 -- 172.21.15.67:0/2038982424 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 11) v1 ==== 46013+0+0 (secure 0 0 0) 0x7f4ca000b930 con 0x7f4cb007cd50 2024-03-28T15:51:40.623 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:39.883+0000 7f4ca6a00700 1 --2- 172.21.15.67:0/2038982424 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f4c9403c9a0 0x7f4c9403ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:40.623 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:39.883+0000 7f4ca6a00700 1 -- 172.21.15.67:0/2038982424 --> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4ca0008ea0 con 0x7f4c9403c9a0 2024-03-28T15:51:40.623 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:39.884+0000 7f4cad400700 1 --2- 172.21.15.67:0/2038982424 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f4c9403c9a0 0x7f4c9403ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:40.623 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:39.887+0000 7f4cad400700 1 --2- 172.21.15.67:0/2038982424 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f4c9403c9a0 0x7f4c9403ed90 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c003a10 tx=0x7f4c9c0092b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:40.623 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:39.888+0000 7f4ca6a00700 1 -- 172.21.15.67:0/2038982424 <== mgr.14152 v2:172.21.15.67:6800/208726654 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7f4ca0008ea0 con 0x7f4c9403c9a0 2024-03-28T15:51:40.623 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:39.895+0000 7f4cb5000700 1 -- 172.21.15.67:0/2038982424 --> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f4cb00637c0 con 0x7f4c9403c9a0 2024-03-28T15:51:40.623 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:39.895+0000 7f4ca6a00700 1 -- 172.21.15.67:0/2038982424 <== mgr.14152 v2:172.21.15.67:6800/208726654 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f4cb00637c0 con 0x7f4c9403c9a0 2024-03-28T15:51:40.623 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:39.896+0000 7f4cb5000700 1 -- 172.21.15.67:0/2038982424 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f4c9403c9a0 msgr2=0x7f4c9403ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:40.623 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:39.896+0000 7f4cb5000700 1 --2- 172.21.15.67:0/2038982424 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f4c9403c9a0 0x7f4c9403ed90 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c003a10 tx=0x7f4c9c0092b0 comp rx=0 tx=0).stop 2024-03-28T15:51:40.623 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:39.896+0000 7f4cb5000700 1 -- 172.21.15.67:0/2038982424 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4cb007cd50 msgr2=0x7f4cb01a1e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:40.624 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:39.896+0000 7f4cb5000700 1 --2- 172.21.15.67:0/2038982424 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4cb007cd50 0x7f4cb01a1e80 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4ca0006ba0 tx=0x7f4ca0011ad0 comp rx=0 tx=0).stop 2024-03-28T15:51:40.624 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:39.896+0000 7f4cb5000700 1 -- 172.21.15.67:0/2038982424 shutdown_connections 2024-03-28T15:51:40.624 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:39.896+0000 7f4cb5000700 1 --2- 172.21.15.67:0/2038982424 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4cb007cd50 0x7f4cb01a1e80 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:40.625 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:39.896+0000 7f4cb5000700 1 --2- 172.21.15.67:0/2038982424 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f4c9403c9a0 0x7f4c9403ed90 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:40.625 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:39.896+0000 7f4cb5000700 1 --2- 172.21.15.67:0/2038982424 >> [v2:172.21.15.67:6800/1631001968,v1:172.21.15.67:6801/1631001968] conn(0x7f4c94038f20 0x7f4c9403b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:40.625 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:39.896+0000 7f4cb5000700 1 -- 172.21.15.67:0/2038982424 >> 172.21.15.67:0/2038982424 conn(0x7f4cb0102f70 msgr2=0x7f4cb0105360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:40.625 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:39.896+0000 7f4cb5000700 1 -- 172.21.15.67:0/2038982424 shutdown_connections 2024-03-28T15:51:40.625 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:39.896+0000 7f4cb5000700 1 -- 172.21.15.67:0/2038982424 wait complete. 2024-03-28T15:51:40.625 INFO:teuthology.orchestra.run.smithi067.stdout:mgr epoch 9 is available 2024-03-28T15:51:40.625 INFO:teuthology.orchestra.run.smithi067.stdout:Generating a dashboard self-signed certificate... 2024-03-28T15:51:41.202 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:40 smithi067 ceph-mon[31707]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-28T15:51:41.202 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:40 smithi067 ceph-mon[31707]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-28T15:51:42.260 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:41 smithi067 ceph-mon[31707]: mgrmap e12: x(active, since 2s) 2024-03-28T15:51:42.260 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:41 smithi067 ceph-mon[31707]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:51:42.260 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:41 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:51:42.261 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:41 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:51:42.261 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:41 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:51:42.577 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-28T15:51:42.577 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:41.365+0000 7f3fe7200700 1 Processor -- start 2024-03-28T15:51:42.578 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:41.365+0000 7f3fe7200700 1 -- start start 2024-03-28T15:51:42.578 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:41.365+0000 7f3fe7200700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fe0081910 0x7f3fe0081d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:42.578 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:41.365+0000 7f3fe7200700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe0082300 con 0x7f3fe0081910 2024-03-28T15:51:42.578 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:41.365+0000 7f3fdfe00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fe0081910 0x7f3fe0081d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:42.578 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:41.366+0000 7f3fdfe00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fe0081910 0x7f3fe0081d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38084/0 (socket says 172.21.15.67:38084) 2024-03-28T15:51:42.578 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:41.366+0000 7f3fdfe00700 1 -- 172.21.15.67:0/3157675864 learned_addr learned my addr 172.21.15.67:0/3157675864 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:51:42.578 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:41.366+0000 7f3fdfe00700 1 -- 172.21.15.67:0/3157675864 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe010f180 con 0x7f3fe0081910 2024-03-28T15:51:42.578 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:41.366+0000 7f3fdfe00700 1 --2- 172.21.15.67:0/3157675864 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fe0081910 0x7f3fe0081d30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3fd00098a0 tx=0x7f3fd0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59d23ae88aee35a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:42.578 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:41.367+0000 7f3fdf400700 1 -- 172.21.15.67:0/3157675864 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3fd0004160 con 0x7f3fe0081910 2024-03-28T15:51:42.578 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:41.367+0000 7f3fdf400700 1 -- 172.21.15.67:0/3157675864 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3fd0004300 con 0x7f3fe0081910 2024-03-28T15:51:42.578 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:41.367+0000 7f3fdf400700 1 -- 172.21.15.67:0/3157675864 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3fd0011a50 con 0x7f3fe0081910 2024-03-28T15:51:42.578 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:41.367+0000 7f3fe7200700 1 -- 172.21.15.67:0/3157675864 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fe0081910 msgr2=0x7f3fe0081d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:42.578 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:41.367+0000 7f3fe7200700 1 --2- 172.21.15.67:0/3157675864 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fe0081910 0x7f3fe0081d30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3fd00098a0 tx=0x7f3fd0009bb0 comp rx=0 tx=0).stop 2024-03-28T15:51:42.578 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:41.367+0000 7f3fe7200700 1 -- 172.21.15.67:0/3157675864 shutdown_connections 2024-03-28T15:51:42.579 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:41.367+0000 7f3fe7200700 1 --2- 172.21.15.67:0/3157675864 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fe0081910 0x7f3fe0081d30 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:42.579 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:41.367+0000 7f3fe7200700 1 -- 172.21.15.67:0/3157675864 >> 172.21.15.67:0/3157675864 conn(0x7f3fe007cfa0 msgr2=0x7f3fe007f3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:42.579 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:41.368+0000 7f3fe7200700 1 -- 172.21.15.67:0/3157675864 shutdown_connections 2024-03-28T15:51:42.579 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:41.368+0000 7f3fe7200700 1 -- 172.21.15.67:0/3157675864 wait complete. 2024-03-28T15:51:42.579 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:41.369+0000 7f3fe7200700 1 Processor -- start 2024-03-28T15:51:42.579 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:41.369+0000 7f3fe7200700 1 -- start start 2024-03-28T15:51:42.579 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:41.369+0000 7f3fe7200700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fe0081910 0x7f3fe01a1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:42.579 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:41.369+0000 7f3fe7200700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe0112530 con 0x7f3fe0081910 2024-03-28T15:51:42.579 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:41.369+0000 7f3fdfe00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fe0081910 0x7f3fe01a1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:42.579 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:41.369+0000 7f3fdfe00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fe0081910 0x7f3fe01a1ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38096/0 (socket says 172.21.15.67:38096) 2024-03-28T15:51:42.579 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:41.369+0000 7f3fdfe00700 1 -- 172.21.15.67:0/2253025385 learned_addr learned my addr 172.21.15.67:0/2253025385 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:51:42.579 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:41.370+0000 7f3fdfe00700 1 -- 172.21.15.67:0/2253025385 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe01a2430 con 0x7f3fe0081910 2024-03-28T15:51:42.579 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:41.370+0000 7f3fdfe00700 1 --2- 172.21.15.67:0/2253025385 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fe0081910 0x7f3fe01a1ef0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3fd0006b20 tx=0x7f3fd0011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:42.579 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:41.370+0000 7f3fdd600700 1 -- 172.21.15.67:0/2253025385 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3fd00039e0 con 0x7f3fe0081910 2024-03-28T15:51:42.579 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:41.370+0000 7f3fdd600700 1 -- 172.21.15.67:0/2253025385 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3fd001a8e0 con 0x7f3fe0081910 2024-03-28T15:51:42.580 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:41.370+0000 7f3fe7200700 1 -- 172.21.15.67:0/2253025385 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fe01a26c0 con 0x7f3fe0081910 2024-03-28T15:51:42.580 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:41.371+0000 7f3fe7200700 1 -- 172.21.15.67:0/2253025385 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fe01a53b0 con 0x7f3fe0081910 2024-03-28T15:51:42.580 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:41.371+0000 7f3fdd600700 1 -- 172.21.15.67:0/2253025385 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3fd00247e0 con 0x7f3fe0081910 2024-03-28T15:51:42.580 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:41.372+0000 7f3fdd600700 1 -- 172.21.15.67:0/2253025385 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 12) v1 ==== 46119+0+0 (secure 0 0 0) 0x7f3fd0022020 con 0x7f3fe0081910 2024-03-28T15:51:42.580 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:41.372+0000 7f3fe7200700 1 -- 172.21.15.67:0/2253025385 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fbc005360 con 0x7f3fe0081910 2024-03-28T15:51:42.580 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:41.372+0000 7f3fdd600700 1 --2- 172.21.15.67:0/2253025385 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f3fc4038fc0 0x7f3fc403b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:42.580 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:41.372+0000 7f3fdd600700 1 -- 172.21.15.67:0/2253025385 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f3fd004d3d0 con 0x7f3fe0081910 2024-03-28T15:51:42.580 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:41.373+0000 7f3fd7e00700 1 --2- 172.21.15.67:0/2253025385 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f3fc4038fc0 0x7f3fc403b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:42.580 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:41.373+0000 7f3fd7e00700 1 --2- 172.21.15.67:0/2253025385 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f3fc4038fc0 0x7f3fc403b480 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3fc8006fd0 tx=0x7f3fc8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:42.580 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:41.378+0000 7f3fdd600700 1 -- 172.21.15.67:0/2253025385 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3fd00172b0 con 0x7f3fe0081910 2024-03-28T15:51:42.580 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:41.615+0000 7f3fe7200700 1 -- 172.21.15.67:0/2253025385 --> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f3fbc000bf0 con 0x7f3fc4038fc0 2024-03-28T15:51:42.580 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:41.748+0000 7f3fdd600700 1 -- 172.21.15.67:0/2253025385 <== mgr.14152 v2:172.21.15.67:6800/208726654 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f3fbc000bf0 con 0x7f3fc4038fc0 2024-03-28T15:51:42.580 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:41.752+0000 7f3fe7200700 1 -- 172.21.15.67:0/2253025385 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f3fc4038fc0 msgr2=0x7f3fc403b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:42.580 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:41.752+0000 7f3fe7200700 1 --2- 172.21.15.67:0/2253025385 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f3fc4038fc0 0x7f3fc403b480 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3fc8006fd0 tx=0x7f3fc8006e40 comp rx=0 tx=0).stop 2024-03-28T15:51:42.580 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:41.752+0000 7f3fe7200700 1 -- 172.21.15.67:0/2253025385 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fe0081910 msgr2=0x7f3fe01a1ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:42.581 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:41.752+0000 7f3fe7200700 1 --2- 172.21.15.67:0/2253025385 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fe0081910 0x7f3fe01a1ef0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3fd0006b20 tx=0x7f3fd0011ad0 comp rx=0 tx=0).stop 2024-03-28T15:51:42.581 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:41.752+0000 7f3fe7200700 1 -- 172.21.15.67:0/2253025385 shutdown_connections 2024-03-28T15:51:42.581 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:41.752+0000 7f3fe7200700 1 --2- 172.21.15.67:0/2253025385 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f3fc4038fc0 0x7f3fc403b480 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:42.581 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:41.752+0000 7f3fe7200700 1 --2- 172.21.15.67:0/2253025385 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fe0081910 0x7f3fe01a1ef0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:42.581 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:41.752+0000 7f3fe7200700 1 -- 172.21.15.67:0/2253025385 >> 172.21.15.67:0/2253025385 conn(0x7f3fe007cfa0 msgr2=0x7f3fe007da90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:42.581 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:41.753+0000 7f3fe7200700 1 -- 172.21.15.67:0/2253025385 shutdown_connections 2024-03-28T15:51:42.581 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:41.753+0000 7f3fe7200700 1 -- 172.21.15.67:0/2253025385 wait complete. 2024-03-28T15:51:42.581 INFO:teuthology.orchestra.run.smithi067.stdout:Creating initial admin user... 2024-03-28T15:51:44.592 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$L5mLquk2DNRiLOg85RmR1O4IetpOl0S8JTPuAw981Gr6G2ISHWriy", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711641103, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-28T15:51:44.593 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:43.267+0000 7f147dc00700 1 Processor -- start 2024-03-28T15:51:44.593 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:43.267+0000 7f147dc00700 1 -- start start 2024-03-28T15:51:44.593 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:43.267+0000 7f147dc00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1478108440 0x7f147810a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:44.593 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:43.267+0000 7f147dc00700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1478063470 con 0x7f1478108440 2024-03-28T15:51:44.593 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:43.268+0000 7f1476a00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1478108440 0x7f147810a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:44.593 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:43.268+0000 7f1476a00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1478108440 0x7f147810a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38098/0 (socket says 172.21.15.67:38098) 2024-03-28T15:51:44.593 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:43.268+0000 7f1476a00700 1 -- 172.21.15.67:0/2753189037 learned_addr learned my addr 172.21.15.67:0/2753189037 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:51:44.593 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:43.268+0000 7f1476a00700 1 -- 172.21.15.67:0/2753189037 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f147810ad70 con 0x7f1478108440 2024-03-28T15:51:44.593 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:43.269+0000 7f1476a00700 1 --2- 172.21.15.67:0/2753189037 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1478108440 0x7f147810a830 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f14680098a0 tx=0x7f1468009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a239aab1e706df66 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:44.594 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:43.269+0000 7f1476000700 1 -- 172.21.15.67:0/2753189037 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1468004160 con 0x7f1478108440 2024-03-28T15:51:44.594 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:43.269+0000 7f1476000700 1 -- 172.21.15.67:0/2753189037 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1468004300 con 0x7f1478108440 2024-03-28T15:51:44.594 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:43.269+0000 7f1476000700 1 -- 172.21.15.67:0/2753189037 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1468011a50 con 0x7f1478108440 2024-03-28T15:51:44.594 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:43.270+0000 7f147dc00700 1 -- 172.21.15.67:0/2753189037 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1478108440 msgr2=0x7f147810a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:44.594 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:43.270+0000 7f147dc00700 1 --2- 172.21.15.67:0/2753189037 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1478108440 0x7f147810a830 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f14680098a0 tx=0x7f1468009bb0 comp rx=0 tx=0).stop 2024-03-28T15:51:44.594 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:43.270+0000 7f147dc00700 1 -- 172.21.15.67:0/2753189037 shutdown_connections 2024-03-28T15:51:44.594 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:43.270+0000 7f147dc00700 1 --2- 172.21.15.67:0/2753189037 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1478108440 0x7f147810a830 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:44.594 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:43.270+0000 7f147dc00700 1 -- 172.21.15.67:0/2753189037 >> 172.21.15.67:0/2753189037 conn(0x7f1478101e70 msgr2=0x7f1478104290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:44.594 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:43.270+0000 7f147dc00700 1 -- 172.21.15.67:0/2753189037 shutdown_connections 2024-03-28T15:51:44.595 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:43.270+0000 7f147dc00700 1 -- 172.21.15.67:0/2753189037 wait complete. 2024-03-28T15:51:44.595 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:43.271+0000 7f147dc00700 1 Processor -- start 2024-03-28T15:51:44.595 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:43.272+0000 7f147dc00700 1 -- start start 2024-03-28T15:51:44.595 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:43.272+0000 7f147dc00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1478108440 0x7f147819daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:44.595 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:43.272+0000 7f147dc00700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f147810bd70 con 0x7f1478108440 2024-03-28T15:51:44.595 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:43.272+0000 7f1476a00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1478108440 0x7f147819daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:44.595 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:43.272+0000 7f1476a00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1478108440 0x7f147819daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38100/0 (socket says 172.21.15.67:38100) 2024-03-28T15:51:44.595 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:43.272+0000 7f1476a00700 1 -- 172.21.15.67:0/165315443 learned_addr learned my addr 172.21.15.67:0/165315443 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:51:44.595 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:43.273+0000 7f1476a00700 1 -- 172.21.15.67:0/165315443 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f147819e030 con 0x7f1478108440 2024-03-28T15:51:44.595 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:43.273+0000 7f1476a00700 1 --2- 172.21.15.67:0/165315443 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1478108440 0x7f147819daf0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1468011a20 tx=0x7f1468011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:44.595 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:43.273+0000 7f146f400700 1 -- 172.21.15.67:0/165315443 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f14680039c0 con 0x7f1478108440 2024-03-28T15:51:44.595 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:43.273+0000 7f146f400700 1 -- 172.21.15.67:0/165315443 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1468003b60 con 0x7f1478108440 2024-03-28T15:51:44.595 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:43.273+0000 7f147dc00700 1 -- 172.21.15.67:0/165315443 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f147819e2c0 con 0x7f1478108440 2024-03-28T15:51:44.596 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:43.273+0000 7f146f400700 1 -- 172.21.15.67:0/165315443 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f146801a6b0 con 0x7f1478108440 2024-03-28T15:51:44.596 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:43.274+0000 7f147dc00700 1 -- 172.21.15.67:0/165315443 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14781a0fb0 con 0x7f1478108440 2024-03-28T15:51:44.596 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:43.275+0000 7f146f400700 1 -- 172.21.15.67:0/165315443 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 12) v1 ==== 46119+0+0 (secure 0 0 0) 0x7f1468024430 con 0x7f1478108440 2024-03-28T15:51:44.596 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:43.275+0000 7f147dc00700 1 -- 172.21.15.67:0/165315443 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14780637c0 con 0x7f1478108440 2024-03-28T15:51:44.596 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:43.275+0000 7f146f400700 1 --2- 172.21.15.67:0/165315443 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f145c038f20 0x7f145c03b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:44.596 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:43.275+0000 7f146f400700 1 -- 172.21.15.67:0/165315443 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f146804d5d0 con 0x7f1478108440 2024-03-28T15:51:44.596 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:43.276+0000 7f146fe00700 1 --2- 172.21.15.67:0/165315443 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f145c038f20 0x7f145c03b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:44.596 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:43.276+0000 7f146fe00700 1 --2- 172.21.15.67:0/165315443 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f145c038f20 0x7f145c03b3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1460006fd0 tx=0x7f1460006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:44.596 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:43.281+0000 7f146f400700 1 -- 172.21.15.67:0/165315443 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1468016260 con 0x7f1478108440 2024-03-28T15:51:44.596 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:43.527+0000 7f147dc00700 1 -- 172.21.15.67:0/165315443 --> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f147819e7a0 con 0x7f145c038f20 2024-03-28T15:51:44.596 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:43.778+0000 7f146f400700 1 -- 172.21.15.67:0/165315443 <== mgr.14152 v2:172.21.15.67:6800/208726654 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f147819e7a0 con 0x7f145c038f20 2024-03-28T15:51:44.596 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:43.781+0000 7f147dc00700 1 -- 172.21.15.67:0/165315443 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f145c038f20 msgr2=0x7f145c03b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:44.596 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:43.781+0000 7f147dc00700 1 --2- 172.21.15.67:0/165315443 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f145c038f20 0x7f145c03b3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1460006fd0 tx=0x7f1460006e40 comp rx=0 tx=0).stop 2024-03-28T15:51:44.596 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:43.781+0000 7f147dc00700 1 -- 172.21.15.67:0/165315443 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1478108440 msgr2=0x7f147819daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:44.596 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:43.781+0000 7f147dc00700 1 --2- 172.21.15.67:0/165315443 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1478108440 0x7f147819daf0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1468011a20 tx=0x7f1468011b00 comp rx=0 tx=0).stop 2024-03-28T15:51:44.597 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:43.781+0000 7f147dc00700 1 -- 172.21.15.67:0/165315443 shutdown_connections 2024-03-28T15:51:44.597 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:43.781+0000 7f147dc00700 1 --2- 172.21.15.67:0/165315443 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f145c038f20 0x7f145c03b3e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:44.597 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:43.781+0000 7f147dc00700 1 --2- 172.21.15.67:0/165315443 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1478108440 0x7f147819daf0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:44.597 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:43.781+0000 7f147dc00700 1 -- 172.21.15.67:0/165315443 >> 172.21.15.67:0/165315443 conn(0x7f1478101e70 msgr2=0x7f14781028d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:44.597 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:43.781+0000 7f147dc00700 1 -- 172.21.15.67:0/165315443 shutdown_connections 2024-03-28T15:51:44.597 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:43.781+0000 7f147dc00700 1 -- 172.21.15.67:0/165315443 wait complete. 2024-03-28T15:51:44.597 INFO:teuthology.orchestra.run.smithi067.stdout:Fetching dashboard port number... 2024-03-28T15:51:44.881 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:44 smithi067 ceph-mon[31707]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:51:44.881 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:44 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:51:44.881 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:44 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:51:46.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:45 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/2870243387' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-28T15:51:46.890 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:46 smithi067 ceph-mon[31707]: mgrmap e13: x(active, since 6s) 2024-03-28T15:51:47.088 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 8443 2024-03-28T15:51:47.089 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:45.326+0000 7f65c8400700 1 Processor -- start 2024-03-28T15:51:47.089 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:45.326+0000 7f65c8400700 1 -- start start 2024-03-28T15:51:47.089 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:45.327+0000 7f65c8400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65c0109c10 0x7f65c010a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:47.089 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:45.327+0000 7f65c8400700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65c010a600 con 0x7f65c0109c10 2024-03-28T15:51:47.089 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:45.327+0000 7f65c5a00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65c0109c10 0x7f65c010a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:47.089 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:45.327+0000 7f65c5a00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65c0109c10 0x7f65c010a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38106/0 (socket says 172.21.15.67:38106) 2024-03-28T15:51:47.089 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:45.327+0000 7f65c5a00700 1 -- 172.21.15.67:0/289376424 learned_addr learned my addr 172.21.15.67:0/289376424 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:51:47.089 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:45.328+0000 7f65c5a00700 1 -- 172.21.15.67:0/289376424 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65c010ae10 con 0x7f65c0109c10 2024-03-28T15:51:47.089 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:45.328+0000 7f65c5a00700 1 --2- 172.21.15.67:0/289376424 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65c0109c10 0x7f65c010a030 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f65b40098a0 tx=0x7f65b4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ad5a9d217ab321f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:47.089 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:45.328+0000 7f65bfe00700 1 -- 172.21.15.67:0/289376424 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f65b4004160 con 0x7f65c0109c10 2024-03-28T15:51:47.089 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:45.328+0000 7f65bfe00700 1 -- 172.21.15.67:0/289376424 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f65b4004300 con 0x7f65c0109c10 2024-03-28T15:51:47.089 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:45.328+0000 7f65bfe00700 1 -- 172.21.15.67:0/289376424 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f65b4011a50 con 0x7f65c0109c10 2024-03-28T15:51:47.089 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:45.329+0000 7f65c8400700 1 -- 172.21.15.67:0/289376424 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65c0109c10 msgr2=0x7f65c010a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:47.090 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:45.329+0000 7f65c8400700 1 --2- 172.21.15.67:0/289376424 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65c0109c10 0x7f65c010a030 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f65b40098a0 tx=0x7f65b4009bb0 comp rx=0 tx=0).stop 2024-03-28T15:51:47.090 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:45.329+0000 7f65c8400700 1 -- 172.21.15.67:0/289376424 shutdown_connections 2024-03-28T15:51:47.090 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:45.329+0000 7f65c8400700 1 --2- 172.21.15.67:0/289376424 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65c0109c10 0x7f65c010a030 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:47.090 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:45.329+0000 7f65c8400700 1 -- 172.21.15.67:0/289376424 >> 172.21.15.67:0/289376424 conn(0x7f65c0105350 msgr2=0x7f65c0107740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:47.090 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:45.329+0000 7f65c8400700 1 -- 172.21.15.67:0/289376424 shutdown_connections 2024-03-28T15:51:47.090 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:45.329+0000 7f65c8400700 1 -- 172.21.15.67:0/289376424 wait complete. 2024-03-28T15:51:47.090 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:45.330+0000 7f65c8400700 1 Processor -- start 2024-03-28T15:51:47.090 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:45.330+0000 7f65c8400700 1 -- start start 2024-03-28T15:51:47.090 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:45.331+0000 7f65c8400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65c0109c10 0x7f65c019dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:47.090 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:45.331+0000 7f65c8400700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65c010e1c0 con 0x7f65c0109c10 2024-03-28T15:51:47.090 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:45.331+0000 7f65c5a00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65c0109c10 0x7f65c019dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:47.090 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:45.331+0000 7f65c5a00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65c0109c10 0x7f65c019dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38114/0 (socket says 172.21.15.67:38114) 2024-03-28T15:51:47.090 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:45.331+0000 7f65c5a00700 1 -- 172.21.15.67:0/2870243387 learned_addr learned my addr 172.21.15.67:0/2870243387 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:51:47.091 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:45.331+0000 7f65c5a00700 1 -- 172.21.15.67:0/2870243387 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65c019e010 con 0x7f65c0109c10 2024-03-28T15:51:47.091 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:45.332+0000 7f65c5a00700 1 --2- 172.21.15.67:0/2870243387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65c0109c10 0x7f65c019dad0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f65b4006b20 tx=0x7f65b4011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:47.091 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:45.332+0000 7f65be000700 1 -- 172.21.15.67:0/2870243387 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f65b40039e0 con 0x7f65c0109c10 2024-03-28T15:51:47.091 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:45.332+0000 7f65be000700 1 -- 172.21.15.67:0/2870243387 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f65b401a8e0 con 0x7f65c0109c10 2024-03-28T15:51:47.091 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:45.332+0000 7f65c8400700 1 -- 172.21.15.67:0/2870243387 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65c019e2a0 con 0x7f65c0109c10 2024-03-28T15:51:47.091 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:45.332+0000 7f65be000700 1 -- 172.21.15.67:0/2870243387 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f65b40247e0 con 0x7f65c0109c10 2024-03-28T15:51:47.091 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:45.332+0000 7f65c8400700 1 -- 172.21.15.67:0/2870243387 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65c01a0f90 con 0x7f65c0109c10 2024-03-28T15:51:47.091 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:45.333+0000 7f65c8400700 1 -- 172.21.15.67:0/2870243387 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65c010e340 con 0x7f65c0109c10 2024-03-28T15:51:47.091 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:45.333+0000 7f65be000700 1 -- 172.21.15.67:0/2870243387 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 12) v1 ==== 46119+0+0 (secure 0 0 0) 0x7f65b4022020 con 0x7f65c0109c10 2024-03-28T15:51:47.091 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:45.334+0000 7f65be000700 1 --2- 172.21.15.67:0/2870243387 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f65a8038fc0 0x7f65a803b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:47.091 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:45.334+0000 7f65be000700 1 -- 172.21.15.67:0/2870243387 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f65b404d400 con 0x7f65c0109c10 2024-03-28T15:51:47.091 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:45.334+0000 7f65c5000700 1 --2- 172.21.15.67:0/2870243387 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f65a8038fc0 0x7f65a803b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:47.091 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:45.335+0000 7f65c5000700 1 --2- 172.21.15.67:0/2870243387 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f65a8038fc0 0x7f65a803b480 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f65ac006fd0 tx=0x7f65ac006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:47.091 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:45.340+0000 7f65be000700 1 -- 172.21.15.67:0/2870243387 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f65b402ad60 con 0x7f65c0109c10 2024-03-28T15:51:47.091 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:45.571+0000 7f65c8400700 1 -- 172.21.15.67:0/2870243387 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f65c01a1540 con 0x7f65c0109c10 2024-03-28T15:51:47.092 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:45.572+0000 7f65be000700 1 -- 172.21.15.67:0/2870243387 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f65b4021d60 con 0x7f65c0109c10 2024-03-28T15:51:47.092 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:45.575+0000 7f65c8400700 1 -- 172.21.15.67:0/2870243387 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f65a8038fc0 msgr2=0x7f65a803b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:47.092 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:45.575+0000 7f65c8400700 1 --2- 172.21.15.67:0/2870243387 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f65a8038fc0 0x7f65a803b480 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f65ac006fd0 tx=0x7f65ac006e40 comp rx=0 tx=0).stop 2024-03-28T15:51:47.092 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:45.575+0000 7f65c8400700 1 -- 172.21.15.67:0/2870243387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65c0109c10 msgr2=0x7f65c019dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:47.092 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:45.575+0000 7f65c8400700 1 --2- 172.21.15.67:0/2870243387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65c0109c10 0x7f65c019dad0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f65b4006b20 tx=0x7f65b4011ad0 comp rx=0 tx=0).stop 2024-03-28T15:51:47.092 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:45.575+0000 7f65c8400700 1 -- 172.21.15.67:0/2870243387 shutdown_connections 2024-03-28T15:51:47.092 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:45.575+0000 7f65c8400700 1 --2- 172.21.15.67:0/2870243387 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f65a8038fc0 0x7f65a803b480 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:47.092 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:45.575+0000 7f65c8400700 1 --2- 172.21.15.67:0/2870243387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65c0109c10 0x7f65c019dad0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:47.092 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:45.575+0000 7f65c8400700 1 -- 172.21.15.67:0/2870243387 >> 172.21.15.67:0/2870243387 conn(0x7f65c0105350 msgr2=0x7f65c0106e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:47.092 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:45.575+0000 7f65c8400700 1 -- 172.21.15.67:0/2870243387 shutdown_connections 2024-03-28T15:51:47.092 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:45.575+0000 7f65c8400700 1 -- 172.21.15.67:0/2870243387 wait complete. 2024-03-28T15:51:47.118 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-28T15:51:47.118 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stdout disabled 2024-03-28T15:51:47.144 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-28T15:51:47.144 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stdout inactive 2024-03-28T15:51:47.144 INFO:teuthology.orchestra.run.smithi067.stdout:firewalld.service is not enabled 2024-03-28T15:51:47.144 INFO:teuthology.orchestra.run.smithi067.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-28T15:51:47.147 INFO:teuthology.orchestra.run.smithi067.stdout:Ceph Dashboard is now available at: 2024-03-28T15:51:47.147 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T15:51:47.148 INFO:teuthology.orchestra.run.smithi067.stdout: URL: https://smithi067.front.sepia.ceph.com:8443/ 2024-03-28T15:51:47.148 INFO:teuthology.orchestra.run.smithi067.stdout: User: admin 2024-03-28T15:51:47.148 INFO:teuthology.orchestra.run.smithi067.stdout: Password: skwe681hj9 2024-03-28T15:51:47.148 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T15:51:47.148 INFO:teuthology.orchestra.run.smithi067.stdout:Saving cluster configuration to /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/config directory 2024-03-28T15:51:47.150 INFO:teuthology.orchestra.run.smithi067.stdout:Enabling autotune for osd_memory_target 2024-03-28T15:51:48.717 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:47.768+0000 7f914a800700 1 Processor -- start 2024-03-28T15:51:48.717 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:47.768+0000 7f914a800700 1 -- start start 2024-03-28T15:51:48.717 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:47.769+0000 7f914a800700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91441099b0 0x7f9144109dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:48.719 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:47.769+0000 7f914a800700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f914410a3a0 con 0x7f91441099b0 2024-03-28T15:51:48.719 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:47.769+0000 7f9149400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91441099b0 0x7f9144109dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:48.719 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:47.769+0000 7f9149400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91441099b0 0x7f9144109dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38118/0 (socket says 172.21.15.67:38118) 2024-03-28T15:51:48.719 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:47.769+0000 7f9149400700 1 -- 172.21.15.67:0/1992744311 learned_addr learned my addr 172.21.15.67:0/1992744311 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:51:48.719 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:47.770+0000 7f9149400700 1 -- 172.21.15.67:0/1992744311 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f914410abb0 con 0x7f91441099b0 2024-03-28T15:51:48.719 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:47.770+0000 7f9149400700 1 --2- 172.21.15.67:0/1992744311 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91441099b0 0x7f9144109dd0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f91380098a0 tx=0x7f9138009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e950bca7c20e93 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:48.719 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:47.770+0000 7f9143e00700 1 -- 172.21.15.67:0/1992744311 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9138004160 con 0x7f91441099b0 2024-03-28T15:51:48.719 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:47.770+0000 7f9143e00700 1 -- 172.21.15.67:0/1992744311 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9138004300 con 0x7f91441099b0 2024-03-28T15:51:48.719 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:47.770+0000 7f9143e00700 1 -- 172.21.15.67:0/1992744311 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9138011a50 con 0x7f91441099b0 2024-03-28T15:51:48.719 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:47.771+0000 7f914a800700 1 -- 172.21.15.67:0/1992744311 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91441099b0 msgr2=0x7f9144109dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:48.719 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:47.771+0000 7f914a800700 1 --2- 172.21.15.67:0/1992744311 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91441099b0 0x7f9144109dd0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f91380098a0 tx=0x7f9138009bb0 comp rx=0 tx=0).stop 2024-03-28T15:51:48.722 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:47.771+0000 7f914a800700 1 -- 172.21.15.67:0/1992744311 shutdown_connections 2024-03-28T15:51:48.722 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:47.771+0000 7f914a800700 1 --2- 172.21.15.67:0/1992744311 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91441099b0 0x7f9144109dd0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:48.722 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:47.771+0000 7f914a800700 1 -- 172.21.15.67:0/1992744311 >> 172.21.15.67:0/1992744311 conn(0x7f914407ce00 msgr2=0x7f914407d210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:48.722 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:47.771+0000 7f914a800700 1 -- 172.21.15.67:0/1992744311 shutdown_connections 2024-03-28T15:51:48.722 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:47.771+0000 7f914a800700 1 -- 172.21.15.67:0/1992744311 wait complete. 2024-03-28T15:51:48.722 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:47.772+0000 7f914a800700 1 Processor -- start 2024-03-28T15:51:48.722 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:47.773+0000 7f914a800700 1 -- start start 2024-03-28T15:51:48.722 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:47.773+0000 7f914a800700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91441099b0 0x7f914419d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:48.722 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:47.773+0000 7f914a800700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f914410df60 con 0x7f91441099b0 2024-03-28T15:51:48.723 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:47.773+0000 7f9149400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91441099b0 0x7f914419d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:48.723 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:47.773+0000 7f9149400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91441099b0 0x7f914419d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38126/0 (socket says 172.21.15.67:38126) 2024-03-28T15:51:48.723 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:47.773+0000 7f9149400700 1 -- 172.21.15.67:0/3552504581 learned_addr learned my addr 172.21.15.67:0/3552504581 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:51:48.723 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:47.774+0000 7f9149400700 1 -- 172.21.15.67:0/3552504581 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f914419de40 con 0x7f91441099b0 2024-03-28T15:51:48.723 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:47.774+0000 7f9149400700 1 --2- 172.21.15.67:0/3552504581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91441099b0 0x7f914419d900 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9138006b20 tx=0x7f9138011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:48.723 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:47.774+0000 7f9142000700 1 -- 172.21.15.67:0/3552504581 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f91380039e0 con 0x7f91441099b0 2024-03-28T15:51:48.723 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:47.774+0000 7f9142000700 1 -- 172.21.15.67:0/3552504581 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f913801a8e0 con 0x7f91441099b0 2024-03-28T15:51:48.723 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:47.774+0000 7f9142000700 1 -- 172.21.15.67:0/3552504581 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f91380247e0 con 0x7f91441099b0 2024-03-28T15:51:48.723 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:47.774+0000 7f914a800700 1 -- 172.21.15.67:0/3552504581 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f914419e0d0 con 0x7f91441099b0 2024-03-28T15:51:48.723 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:47.775+0000 7f914a800700 1 -- 172.21.15.67:0/3552504581 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91441a0dc0 con 0x7f91441099b0 2024-03-28T15:51:48.723 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:47.776+0000 7f9142000700 1 -- 172.21.15.67:0/3552504581 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f9138022020 con 0x7f91441099b0 2024-03-28T15:51:48.725 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:47.776+0000 7f914a800700 1 -- 172.21.15.67:0/3552504581 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91440637c0 con 0x7f91441099b0 2024-03-28T15:51:48.725 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:47.776+0000 7f9142000700 1 --2- 172.21.15.67:0/3552504581 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f912c039060 0x7f912c03b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:48.726 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:47.776+0000 7f9142000700 1 -- 172.21.15.67:0/3552504581 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f913804d400 con 0x7f91441099b0 2024-03-28T15:51:48.726 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:47.776+0000 7f9148a00700 1 --2- 172.21.15.67:0/3552504581 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f912c039060 0x7f912c03b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:48.726 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:47.777+0000 7f9148a00700 1 --2- 172.21.15.67:0/3552504581 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f912c039060 0x7f912c03b520 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9130006fd0 tx=0x7f9130006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:48.726 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:47.782+0000 7f9142000700 1 -- 172.21.15.67:0/3552504581 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f913802c3a0 con 0x7f91441099b0 2024-03-28T15:51:48.726 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:48.011+0000 7f914a800700 1 -- 172.21.15.67:0/3552504581 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f91441a13e0 con 0x7f91441099b0 2024-03-28T15:51:48.726 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:48.012+0000 7f9142000700 1 -- 172.21.15.67:0/3552504581 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f91380212c0 con 0x7f91441099b0 2024-03-28T15:51:48.726 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:48.015+0000 7f914a800700 1 -- 172.21.15.67:0/3552504581 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f912c039060 msgr2=0x7f912c03b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:48.726 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:48.015+0000 7f914a800700 1 --2- 172.21.15.67:0/3552504581 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f912c039060 0x7f912c03b520 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9130006fd0 tx=0x7f9130006e40 comp rx=0 tx=0).stop 2024-03-28T15:51:48.726 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:48.015+0000 7f914a800700 1 -- 172.21.15.67:0/3552504581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91441099b0 msgr2=0x7f914419d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:48.726 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:48.015+0000 7f914a800700 1 --2- 172.21.15.67:0/3552504581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91441099b0 0x7f914419d900 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9138006b20 tx=0x7f9138011ad0 comp rx=0 tx=0).stop 2024-03-28T15:51:48.726 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:48.016+0000 7f914a800700 1 -- 172.21.15.67:0/3552504581 shutdown_connections 2024-03-28T15:51:48.726 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:48.016+0000 7f914a800700 1 --2- 172.21.15.67:0/3552504581 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f912c039060 0x7f912c03b520 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:48.726 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:48.016+0000 7f914a800700 1 --2- 172.21.15.67:0/3552504581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91441099b0 0x7f914419d900 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:48.726 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:48.016+0000 7f914a800700 1 -- 172.21.15.67:0/3552504581 >> 172.21.15.67:0/3552504581 conn(0x7f914407ce00 msgr2=0x7f91441074b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:48.727 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:48.016+0000 7f914a800700 1 -- 172.21.15.67:0/3552504581 shutdown_connections 2024-03-28T15:51:48.727 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:48.016+0000 7f914a800700 1 -- 172.21.15.67:0/3552504581 wait complete. 2024-03-28T15:51:50.396 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:49.334+0000 7fce23200700 1 Processor -- start 2024-03-28T15:51:50.396 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:49.334+0000 7fce23200700 1 -- start start 2024-03-28T15:51:50.396 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:49.334+0000 7fce23200700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce24109990 0x7fce24109db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:50.396 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:49.334+0000 7fce23200700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce2410a380 con 0x7fce24109990 2024-03-28T15:51:50.396 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:49.335+0000 7fce21e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce24109990 0x7fce24109db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:50.396 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:49.335+0000 7fce21e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce24109990 0x7fce24109db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46042/0 (socket says 172.21.15.67:46042) 2024-03-28T15:51:50.396 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:49.335+0000 7fce21e00700 1 -- 172.21.15.67:0/3215838369 learned_addr learned my addr 172.21.15.67:0/3215838369 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:51:50.396 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:49.335+0000 7fce21e00700 1 -- 172.21.15.67:0/3215838369 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce2410ab90 con 0x7fce24109990 2024-03-28T15:51:50.396 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:49.336+0000 7fce21e00700 1 --2- 172.21.15.67:0/3215838369 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce24109990 0x7fce24109db0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fce140098a0 tx=0x7fce14009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97a7d76439bf17a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:50.396 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:49.336+0000 7fce20a00700 1 -- 172.21.15.67:0/3215838369 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fce14004160 con 0x7fce24109990 2024-03-28T15:51:50.396 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:49.336+0000 7fce20a00700 1 -- 172.21.15.67:0/3215838369 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fce14004300 con 0x7fce24109990 2024-03-28T15:51:50.397 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:49.336+0000 7fce20a00700 1 -- 172.21.15.67:0/3215838369 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fce14011a50 con 0x7fce24109990 2024-03-28T15:51:50.397 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:49.336+0000 7fce23200700 1 -- 172.21.15.67:0/3215838369 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce24109990 msgr2=0x7fce24109db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:50.397 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:49.336+0000 7fce23200700 1 --2- 172.21.15.67:0/3215838369 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce24109990 0x7fce24109db0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fce140098a0 tx=0x7fce14009bb0 comp rx=0 tx=0).stop 2024-03-28T15:51:50.397 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:49.337+0000 7fce23200700 1 -- 172.21.15.67:0/3215838369 shutdown_connections 2024-03-28T15:51:50.397 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:49.337+0000 7fce23200700 1 --2- 172.21.15.67:0/3215838369 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce24109990 0x7fce24109db0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:50.397 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:49.337+0000 7fce23200700 1 -- 172.21.15.67:0/3215838369 >> 172.21.15.67:0/3215838369 conn(0x7fce2407cd60 msgr2=0x7fce2407d190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:50.397 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:49.337+0000 7fce23200700 1 -- 172.21.15.67:0/3215838369 shutdown_connections 2024-03-28T15:51:50.397 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:49.337+0000 7fce23200700 1 -- 172.21.15.67:0/3215838369 wait complete. 2024-03-28T15:51:50.397 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:49.338+0000 7fce23200700 1 Processor -- start 2024-03-28T15:51:50.397 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:49.338+0000 7fce23200700 1 -- start start 2024-03-28T15:51:50.397 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:49.338+0000 7fce23200700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce24109990 0x7fce2419d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:50.397 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:49.338+0000 7fce23200700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce2410df40 con 0x7fce24109990 2024-03-28T15:51:50.397 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:49.339+0000 7fce21e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce24109990 0x7fce2419d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:50.397 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:49.339+0000 7fce21e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce24109990 0x7fce2419d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46046/0 (socket says 172.21.15.67:46046) 2024-03-28T15:51:50.398 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:49.339+0000 7fce21e00700 1 -- 172.21.15.67:0/4108080098 learned_addr learned my addr 172.21.15.67:0/4108080098 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:51:50.398 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:49.339+0000 7fce21e00700 1 -- 172.21.15.67:0/4108080098 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce2419de00 con 0x7fce24109990 2024-03-28T15:51:50.398 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:49.339+0000 7fce21e00700 1 --2- 172.21.15.67:0/4108080098 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce24109990 0x7fce2419d8c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fce14006b20 tx=0x7fce14011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:50.398 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:49.340+0000 7fce1aa00700 1 -- 172.21.15.67:0/4108080098 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fce140039e0 con 0x7fce24109990 2024-03-28T15:51:50.398 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:49.340+0000 7fce1aa00700 1 -- 172.21.15.67:0/4108080098 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fce1401a8e0 con 0x7fce24109990 2024-03-28T15:51:50.398 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:49.340+0000 7fce23200700 1 -- 172.21.15.67:0/4108080098 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce2419e090 con 0x7fce24109990 2024-03-28T15:51:50.398 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:49.340+0000 7fce1aa00700 1 -- 172.21.15.67:0/4108080098 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fce140247e0 con 0x7fce24109990 2024-03-28T15:51:50.398 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:49.340+0000 7fce23200700 1 -- 172.21.15.67:0/4108080098 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce241a0d80 con 0x7fce24109990 2024-03-28T15:51:50.398 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:49.341+0000 7fce1aa00700 1 -- 172.21.15.67:0/4108080098 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7fce14022020 con 0x7fce24109990 2024-03-28T15:51:50.398 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:49.341+0000 7fce23200700 1 -- 172.21.15.67:0/4108080098 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce240637c0 con 0x7fce24109990 2024-03-28T15:51:50.398 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:49.342+0000 7fce1aa00700 1 --2- 172.21.15.67:0/4108080098 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fce08039060 0x7fce0803b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:50.398 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:49.342+0000 7fce1aa00700 1 -- 172.21.15.67:0/4108080098 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fce1404d400 con 0x7fce24109990 2024-03-28T15:51:50.398 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:49.342+0000 7fce21400700 1 --2- 172.21.15.67:0/4108080098 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fce08039060 0x7fce0803b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:50.399 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:49.343+0000 7fce21400700 1 --2- 172.21.15.67:0/4108080098 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fce08039060 0x7fce0803b520 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fce10006fd0 tx=0x7fce10006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:50.399 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:49.347+0000 7fce1aa00700 1 -- 172.21.15.67:0/4108080098 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fce1402c3a0 con 0x7fce24109990 2024-03-28T15:51:50.399 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:49.698+0000 7fce23200700 1 -- 172.21.15.67:0/4108080098 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7fce241a1330 con 0x7fce24109990 2024-03-28T15:51:50.399 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:49.702+0000 7fce1aa00700 1 -- 172.21.15.67:0/4108080098 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) v1 ==== 153+0+0 (secure 0 0 0) 0x7fce140212c0 con 0x7fce24109990 2024-03-28T15:51:50.399 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:49.705+0000 7fce23200700 1 -- 172.21.15.67:0/4108080098 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fce08039060 msgr2=0x7fce0803b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:50.399 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:49.705+0000 7fce23200700 1 --2- 172.21.15.67:0/4108080098 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fce08039060 0x7fce0803b520 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fce10006fd0 tx=0x7fce10006e40 comp rx=0 tx=0).stop 2024-03-28T15:51:50.399 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:49.705+0000 7fce23200700 1 -- 172.21.15.67:0/4108080098 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce24109990 msgr2=0x7fce2419d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:50.399 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:49.705+0000 7fce23200700 1 --2- 172.21.15.67:0/4108080098 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce24109990 0x7fce2419d8c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fce14006b20 tx=0x7fce14011ad0 comp rx=0 tx=0).stop 2024-03-28T15:51:50.399 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:49.706+0000 7fce23200700 1 -- 172.21.15.67:0/4108080098 shutdown_connections 2024-03-28T15:51:50.399 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:49.706+0000 7fce23200700 1 --2- 172.21.15.67:0/4108080098 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fce08039060 0x7fce0803b520 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:50.399 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:49.706+0000 7fce23200700 1 --2- 172.21.15.67:0/4108080098 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce24109990 0x7fce2419d8c0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:50.399 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:49.706+0000 7fce23200700 1 -- 172.21.15.67:0/4108080098 >> 172.21.15.67:0/4108080098 conn(0x7fce2407cd60 msgr2=0x7fce24107440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:50.399 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:49.706+0000 7fce23200700 1 -- 172.21.15.67:0/4108080098 shutdown_connections 2024-03-28T15:51:50.399 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-03-28T15:51:49.706+0000 7fce23200700 1 -- 172.21.15.67:0/4108080098 wait complete. 2024-03-28T15:51:50.399 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-28T15:51:50.400 INFO:teuthology.orchestra.run.smithi067.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-28T15:51:50.400 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T15:51:50.400 INFO:teuthology.orchestra.run.smithi067.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-28T15:51:50.400 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T15:51:50.400 INFO:teuthology.orchestra.run.smithi067.stdout:Or, if you are only running a single cluster on this host: 2024-03-28T15:51:50.401 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T15:51:50.401 INFO:teuthology.orchestra.run.smithi067.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-28T15:51:50.401 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T15:51:50.401 INFO:teuthology.orchestra.run.smithi067.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-28T15:51:50.401 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T15:51:50.401 INFO:teuthology.orchestra.run.smithi067.stdout: ceph telemetry on 2024-03-28T15:51:50.401 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T15:51:50.401 INFO:teuthology.orchestra.run.smithi067.stdout:For more information see: 2024-03-28T15:51:50.401 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T15:51:50.401 INFO:teuthology.orchestra.run.smithi067.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-28T15:51:50.401 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T15:51:50.401 INFO:teuthology.orchestra.run.smithi067.stdout:Bootstrap complete. 2024-03-28T15:51:50.515 INFO:tasks.cephadm:Fetching config... 2024-03-28T15:51:50.515 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-28T15:51:50.516 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-28T15:51:50.570 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-28T15:51:50.571 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-28T15:51:50.571 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-28T15:51:50.612 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-28T15:51:50.612 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-28T15:51:50.612 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-28T15:51:50.748 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-28T15:51:50.748 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-28T15:51:50.749 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-28T15:51:50.802 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-28T15:51:50.803 DEBUG:teuthology.orchestra.run.smithi067:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6KW4xZdBV/fUBqKs/kJRL4iaqOTp2NS5gBusRScWfIJfUYjUNmJZ4MHz8hgGUYjAjxgo58AqD9y0cjS9va8z0cxXaPUWtNsUNGek4Rnit1AZhcgfhrpA/0zs7kkNy5MwEtmf8gfxO/4uPilioYEk4TSUrbashODf450mBFn4AW284H99pmcu2JCDQ7H6MwTNLWMU+WxUNls32jyNiKjY7PiQOSNk1tjKNFAb5adfRE9GU+NJfaUiIiflnaGCKHeC4Uqgf8yCbNMPj2Bcz61QNNvlCmMwn2pNCKDbswmaa5Aw8G+3W5pVIUK+sgsPWJKedjubfHkSDEnlDMU4Pir0Debj0gxejCAI7aQGLSlpV2954ynqndMBorThzg+UYxjLT+E5gN4k7zTE/UnXVUYZSlx6U9AnUC1UMiLqG+3Vn1jS2Ib8J7nu0rCRXlTo3G+9MuqvGlm7xdJH64xftzC/fNM1jFxwZZJAA1vCinXpgfVMkrjQP24Wu+W4Iu1N37ak= ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-28T15:51:50.941 INFO:teuthology.orchestra.run.smithi067.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6KW4xZdBV/fUBqKs/kJRL4iaqOTp2NS5gBusRScWfIJfUYjUNmJZ4MHz8hgGUYjAjxgo58AqD9y0cjS9va8z0cxXaPUWtNsUNGek4Rnit1AZhcgfhrpA/0zs7kkNy5MwEtmf8gfxO/4uPilioYEk4TSUrbashODf450mBFn4AW284H99pmcu2JCDQ7H6MwTNLWMU+WxUNls32jyNiKjY7PiQOSNk1tjKNFAb5adfRE9GU+NJfaUiIiflnaGCKHeC4Uqgf8yCbNMPj2Bcz61QNNvlCmMwn2pNCKDbswmaa5Aw8G+3W5pVIUK+sgsPWJKedjubfHkSDEnlDMU4Pir0Debj0gxejCAI7aQGLSlpV2954ynqndMBorThzg+UYxjLT+E5gN4k7zTE/UnXVUYZSlx6U9AnUC1UMiLqG+3Vn1jS2Ib8J7nu0rCRXlTo3G+9MuqvGlm7xdJH64xftzC/fNM1jFxwZZJAA1vCinXpgfVMkrjQP24Wu+W4Iu1N37ak= ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715 2024-03-28T15:51:50.973 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:50 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/4108080098' entity='client.admin' 2024-03-28T15:51:50.985 DEBUG:teuthology.orchestra.run.smithi150:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6KW4xZdBV/fUBqKs/kJRL4iaqOTp2NS5gBusRScWfIJfUYjUNmJZ4MHz8hgGUYjAjxgo58AqD9y0cjS9va8z0cxXaPUWtNsUNGek4Rnit1AZhcgfhrpA/0zs7kkNy5MwEtmf8gfxO/4uPilioYEk4TSUrbashODf450mBFn4AW284H99pmcu2JCDQ7H6MwTNLWMU+WxUNls32jyNiKjY7PiQOSNk1tjKNFAb5adfRE9GU+NJfaUiIiflnaGCKHeC4Uqgf8yCbNMPj2Bcz61QNNvlCmMwn2pNCKDbswmaa5Aw8G+3W5pVIUK+sgsPWJKedjubfHkSDEnlDMU4Pir0Debj0gxejCAI7aQGLSlpV2954ynqndMBorThzg+UYxjLT+E5gN4k7zTE/UnXVUYZSlx6U9AnUC1UMiLqG+3Vn1jS2Ib8J7nu0rCRXlTo3G+9MuqvGlm7xdJH64xftzC/fNM1jFxwZZJAA1vCinXpgfVMkrjQP24Wu+W4Iu1N37ak= ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-28T15:51:51.119 INFO:teuthology.orchestra.run.smithi150.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6KW4xZdBV/fUBqKs/kJRL4iaqOTp2NS5gBusRScWfIJfUYjUNmJZ4MHz8hgGUYjAjxgo58AqD9y0cjS9va8z0cxXaPUWtNsUNGek4Rnit1AZhcgfhrpA/0zs7kkNy5MwEtmf8gfxO/4uPilioYEk4TSUrbashODf450mBFn4AW284H99pmcu2JCDQ7H6MwTNLWMU+WxUNls32jyNiKjY7PiQOSNk1tjKNFAb5adfRE9GU+NJfaUiIiflnaGCKHeC4Uqgf8yCbNMPj2Bcz61QNNvlCmMwn2pNCKDbswmaa5Aw8G+3W5pVIUK+sgsPWJKedjubfHkSDEnlDMU4Pir0Debj0gxejCAI7aQGLSlpV2954ynqndMBorThzg+UYxjLT+E5gN4k7zTE/UnXVUYZSlx6U9AnUC1UMiLqG+3Vn1jS2Ib8J7nu0rCRXlTo3G+9MuqvGlm7xdJH64xftzC/fNM1jFxwZZJAA1vCinXpgfVMkrjQP24Wu+W4Iu1N37ak= ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715 2024-03-28T15:51:51.178 DEBUG:teuthology.orchestra.run.smithi183:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6KW4xZdBV/fUBqKs/kJRL4iaqOTp2NS5gBusRScWfIJfUYjUNmJZ4MHz8hgGUYjAjxgo58AqD9y0cjS9va8z0cxXaPUWtNsUNGek4Rnit1AZhcgfhrpA/0zs7kkNy5MwEtmf8gfxO/4uPilioYEk4TSUrbashODf450mBFn4AW284H99pmcu2JCDQ7H6MwTNLWMU+WxUNls32jyNiKjY7PiQOSNk1tjKNFAb5adfRE9GU+NJfaUiIiflnaGCKHeC4Uqgf8yCbNMPj2Bcz61QNNvlCmMwn2pNCKDbswmaa5Aw8G+3W5pVIUK+sgsPWJKedjubfHkSDEnlDMU4Pir0Debj0gxejCAI7aQGLSlpV2954ynqndMBorThzg+UYxjLT+E5gN4k7zTE/UnXVUYZSlx6U9AnUC1UMiLqG+3Vn1jS2Ib8J7nu0rCRXlTo3G+9MuqvGlm7xdJH64xftzC/fNM1jFxwZZJAA1vCinXpgfVMkrjQP24Wu+W4Iu1N37ak= ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-28T15:51:51.316 INFO:teuthology.orchestra.run.smithi183.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6KW4xZdBV/fUBqKs/kJRL4iaqOTp2NS5gBusRScWfIJfUYjUNmJZ4MHz8hgGUYjAjxgo58AqD9y0cjS9va8z0cxXaPUWtNsUNGek4Rnit1AZhcgfhrpA/0zs7kkNy5MwEtmf8gfxO/4uPilioYEk4TSUrbashODf450mBFn4AW284H99pmcu2JCDQ7H6MwTNLWMU+WxUNls32jyNiKjY7PiQOSNk1tjKNFAb5adfRE9GU+NJfaUiIiflnaGCKHeC4Uqgf8yCbNMPj2Bcz61QNNvlCmMwn2pNCKDbswmaa5Aw8G+3W5pVIUK+sgsPWJKedjubfHkSDEnlDMU4Pir0Debj0gxejCAI7aQGLSlpV2954ynqndMBorThzg+UYxjLT+E5gN4k7zTE/UnXVUYZSlx6U9AnUC1UMiLqG+3Vn1jS2Ib8J7nu0rCRXlTo3G+9MuqvGlm7xdJH64xftzC/fNM1jFxwZZJAA1vCinXpgfVMkrjQP24Wu+W4Iu1N37ak= ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715 2024-03-28T15:51:51.379 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-28T15:51:51.825 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:51:54.694 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:54 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:51:54.694 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:54 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:51:54.694 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:54 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi067", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:51:54.695 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:54 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:51:54.695 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:54 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:51:54.695 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:54 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:51:54.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:54.682+0000 7ff502800700 1 -- 172.21.15.67:0/1313078550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4fc0761c0 msgr2=0x7ff4fc074620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:54.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:54.682+0000 7ff502800700 1 --2- 172.21.15.67:0/1313078550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4fc0761c0 0x7ff4fc074620 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff4f0009910 tx=0x7ff4f0009c20 comp rx=0 tx=0).stop 2024-03-28T15:51:54.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:54.682+0000 7ff502800700 1 -- 172.21.15.67:0/1313078550 shutdown_connections 2024-03-28T15:51:54.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:54.682+0000 7ff502800700 1 --2- 172.21.15.67:0/1313078550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4fc0761c0 0x7ff4fc074620 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:54.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:54.682+0000 7ff502800700 1 -- 172.21.15.67:0/1313078550 >> 172.21.15.67:0/1313078550 conn(0x7ff4fc0fcfd0 msgr2=0x7ff4fc0ff3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:54.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:54.683+0000 7ff502800700 1 -- 172.21.15.67:0/1313078550 shutdown_connections 2024-03-28T15:51:54.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:54.683+0000 7ff502800700 1 -- 172.21.15.67:0/1313078550 wait complete. 2024-03-28T15:51:54.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:54.684+0000 7ff502800700 1 Processor -- start 2024-03-28T15:51:54.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:54.684+0000 7ff502800700 1 -- start start 2024-03-28T15:51:54.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:54.684+0000 7ff502800700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4fc0761c0 0x7ff4fc199240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:54.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:54.684+0000 7ff502800700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4fc109b50 con 0x7ff4fc0761c0 2024-03-28T15:51:54.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:54.684+0000 7ff501400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4fc0761c0 0x7ff4fc199240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:54.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:54.685+0000 7ff501400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4fc0761c0 0x7ff4fc199240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46060/0 (socket says 172.21.15.67:46060) 2024-03-28T15:51:54.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:54.685+0000 7ff501400700 1 -- 172.21.15.67:0/85231801 learned_addr learned my addr 172.21.15.67:0/85231801 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:51:54.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:54.685+0000 7ff501400700 1 -- 172.21.15.67:0/85231801 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4fc199780 con 0x7ff4fc0761c0 2024-03-28T15:51:54.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:54.685+0000 7ff501400700 1 --2- 172.21.15.67:0/85231801 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4fc0761c0 0x7ff4fc199240 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff4f0005950 tx=0x7ff4f0004fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:54.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:54.686+0000 7ff4fa000700 1 -- 172.21.15.67:0/85231801 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff4f0026070 con 0x7ff4fc0761c0 2024-03-28T15:51:54.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:54.686+0000 7ff4fa000700 1 -- 172.21.15.67:0/85231801 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff4f000bc00 con 0x7ff4fc0761c0 2024-03-28T15:51:54.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:54.686+0000 7ff4fa000700 1 -- 172.21.15.67:0/85231801 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff4f00039a0 con 0x7ff4fc0761c0 2024-03-28T15:51:54.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:54.686+0000 7ff502800700 1 -- 172.21.15.67:0/85231801 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4fc199a10 con 0x7ff4fc0761c0 2024-03-28T15:51:54.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:54.686+0000 7ff502800700 1 -- 172.21.15.67:0/85231801 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4fc19c700 con 0x7ff4fc0761c0 2024-03-28T15:51:54.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:54.687+0000 7ff4fa000700 1 -- 172.21.15.67:0/85231801 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7ff4f0029070 con 0x7ff4fc0761c0 2024-03-28T15:51:54.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:54.687+0000 7ff4ef400700 1 -- 172.21.15.67:0/85231801 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4d8005330 con 0x7ff4fc0761c0 2024-03-28T15:51:54.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:54.688+0000 7ff4fa000700 1 --2- 172.21.15.67:0/85231801 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7ff4e4039020 0x7ff4e403b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:54.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:54.688+0000 7ff4fa000700 1 -- 172.21.15.67:0/85231801 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff4f001e070 con 0x7ff4fc0761c0 2024-03-28T15:51:54.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:54.688+0000 7ff500a00700 1 --2- 172.21.15.67:0/85231801 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7ff4e4039020 0x7ff4e403b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:54.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:54.689+0000 7ff500a00700 1 --2- 172.21.15.67:0/85231801 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7ff4e4039020 0x7ff4e403b4e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff4e8006fd0 tx=0x7ff4e8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:54.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:54.694+0000 7ff4fa000700 1 -- 172.21.15.67:0/85231801 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff4f0013070 con 0x7ff4fc0761c0 2024-03-28T15:51:54.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:54.926+0000 7ff4ef400700 1 -- 172.21.15.67:0/85231801 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7ff4d8005620 con 0x7ff4fc0761c0 2024-03-28T15:51:54.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:54.930+0000 7ff4fa000700 1 -- 172.21.15.67:0/85231801 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7ff4f0030020 con 0x7ff4fc0761c0 2024-03-28T15:51:54.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:54.933+0000 7ff4ef400700 1 -- 172.21.15.67:0/85231801 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7ff4e4039020 msgr2=0x7ff4e403b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:54.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:54.933+0000 7ff4ef400700 1 --2- 172.21.15.67:0/85231801 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7ff4e4039020 0x7ff4e403b4e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff4e8006fd0 tx=0x7ff4e8006e40 comp rx=0 tx=0).stop 2024-03-28T15:51:54.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:54.933+0000 7ff4ef400700 1 -- 172.21.15.67:0/85231801 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4fc0761c0 msgr2=0x7ff4fc199240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:54.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:54.933+0000 7ff4ef400700 1 --2- 172.21.15.67:0/85231801 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4fc0761c0 0x7ff4fc199240 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff4f0005950 tx=0x7ff4f0004fa0 comp rx=0 tx=0).stop 2024-03-28T15:51:54.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:54.933+0000 7ff4ef400700 1 -- 172.21.15.67:0/85231801 shutdown_connections 2024-03-28T15:51:54.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:54.933+0000 7ff4ef400700 1 --2- 172.21.15.67:0/85231801 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7ff4e4039020 0x7ff4e403b4e0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:54.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:54.933+0000 7ff4ef400700 1 --2- 172.21.15.67:0/85231801 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4fc0761c0 0x7ff4fc199240 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:54.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:54.933+0000 7ff4ef400700 1 -- 172.21.15.67:0/85231801 >> 172.21.15.67:0/85231801 conn(0x7ff4fc0fcfd0 msgr2=0x7ff4fc0fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:54.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:54.933+0000 7ff4ef400700 1 -- 172.21.15.67:0/85231801 shutdown_connections 2024-03-28T15:51:54.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:54.933+0000 7ff4ef400700 1 -- 172.21.15.67:0/85231801 wait complete. 2024-03-28T15:51:55.688 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-28T15:51:55.689 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-28T15:51:56.249 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:55 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/85231801' entity='client.admin' 2024-03-28T15:51:56.250 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:55 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:51:56.250 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:55 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:51:56.610 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:51:58.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:58.639+0000 7f7054200700 1 -- 172.21.15.67:0/2744684131 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f704c1083a0 msgr2=0x7f704c1087c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:58.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:58.639+0000 7f7054200700 1 --2- 172.21.15.67:0/2744684131 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f704c1083a0 0x7f704c1087c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7040009910 tx=0x7f7040009c20 comp rx=0 tx=0).stop 2024-03-28T15:51:58.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:58.639+0000 7f7054200700 1 -- 172.21.15.67:0/2744684131 shutdown_connections 2024-03-28T15:51:58.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:58.639+0000 7f7054200700 1 --2- 172.21.15.67:0/2744684131 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f704c1083a0 0x7f704c1087c0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:58.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:58.639+0000 7f7054200700 1 -- 172.21.15.67:0/2744684131 >> 172.21.15.67:0/2744684131 conn(0x7f704c077280 msgr2=0x7f704c079700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:58.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:58.639+0000 7f7054200700 1 -- 172.21.15.67:0/2744684131 shutdown_connections 2024-03-28T15:51:58.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:58.639+0000 7f7054200700 1 -- 172.21.15.67:0/2744684131 wait complete. 2024-03-28T15:51:58.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:58.640+0000 7f7054200700 1 Processor -- start 2024-03-28T15:51:58.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:58.640+0000 7f7054200700 1 -- start start 2024-03-28T15:51:58.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:58.641+0000 7f7054200700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f704c1083a0 0x7f704c19d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:58.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:58.641+0000 7f7054200700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f704c10e0c0 con 0x7f704c1083a0 2024-03-28T15:51:58.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:58.641+0000 7f7051800700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f704c1083a0 0x7f704c19d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:58.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:58.641+0000 7f7051800700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f704c1083a0 0x7f704c19d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46066/0 (socket says 172.21.15.67:46066) 2024-03-28T15:51:58.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:58.641+0000 7f7051800700 1 -- 172.21.15.67:0/3540520895 learned_addr learned my addr 172.21.15.67:0/3540520895 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:51:58.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:58.641+0000 7f7051800700 1 -- 172.21.15.67:0/3540520895 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f704c19dd00 con 0x7f704c1083a0 2024-03-28T15:51:58.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:58.642+0000 7f7051800700 1 --2- 172.21.15.67:0/3540520895 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f704c1083a0 0x7f704c19d7c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7040005190 tx=0x7f7040009ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:58.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:58.642+0000 7f704a000700 1 -- 172.21.15.67:0/3540520895 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7040027070 con 0x7f704c1083a0 2024-03-28T15:51:58.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:58.642+0000 7f704a000700 1 -- 172.21.15.67:0/3540520895 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7040022470 con 0x7f704c1083a0 2024-03-28T15:51:58.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:58.642+0000 7f704a000700 1 -- 172.21.15.67:0/3540520895 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f704000f5f0 con 0x7f704c1083a0 2024-03-28T15:51:58.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:58.642+0000 7f7054200700 1 -- 172.21.15.67:0/3540520895 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f704c19df90 con 0x7f704c1083a0 2024-03-28T15:51:58.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:58.642+0000 7f7054200700 1 -- 172.21.15.67:0/3540520895 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f704c1a0c80 con 0x7f704c1083a0 2024-03-28T15:51:58.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:58.643+0000 7f704a000700 1 -- 172.21.15.67:0/3540520895 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f704002a070 con 0x7f704c1083a0 2024-03-28T15:51:58.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:58.643+0000 7f7054200700 1 -- 172.21.15.67:0/3540520895 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f704c10e840 con 0x7f704c1083a0 2024-03-28T15:51:58.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:58.644+0000 7f704a000700 1 --2- 172.21.15.67:0/3540520895 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7034038fc0 0x7f703403b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:58.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:58.644+0000 7f704a000700 1 -- 172.21.15.67:0/3540520895 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f704001e030 con 0x7f704c1083a0 2024-03-28T15:51:58.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:58.644+0000 7f7050e00700 1 --2- 172.21.15.67:0/3540520895 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7034038fc0 0x7f703403b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:58.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:58.645+0000 7f7050e00700 1 --2- 172.21.15.67:0/3540520895 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7034038fc0 0x7f703403b480 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7038006fd0 tx=0x7f7038006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:58.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:58.650+0000 7f704a000700 1 -- 172.21.15.67:0/3540520895 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7040013070 con 0x7f704c1083a0 2024-03-28T15:51:58.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:58.879+0000 7f7054200700 1 -- 172.21.15.67:0/3540520895 --> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f704c02db40 con 0x7f7034038fc0 2024-03-28T15:51:58.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:58.887+0000 7f704a000700 1 -- 172.21.15.67:0/3540520895 <== mgr.14152 v2:172.21.15.67:6800/208726654 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f704c02db40 con 0x7f7034038fc0 2024-03-28T15:51:58.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:58.890+0000 7f7054200700 1 -- 172.21.15.67:0/3540520895 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7034038fc0 msgr2=0x7f703403b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:58.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:58.890+0000 7f7054200700 1 --2- 172.21.15.67:0/3540520895 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7034038fc0 0x7f703403b480 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7038006fd0 tx=0x7f7038006e40 comp rx=0 tx=0).stop 2024-03-28T15:51:58.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:58.890+0000 7f7054200700 1 -- 172.21.15.67:0/3540520895 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f704c1083a0 msgr2=0x7f704c19d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:58.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:58.890+0000 7f7054200700 1 --2- 172.21.15.67:0/3540520895 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f704c1083a0 0x7f704c19d7c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7040005190 tx=0x7f7040009ea0 comp rx=0 tx=0).stop 2024-03-28T15:51:58.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:58.890+0000 7f7054200700 1 -- 172.21.15.67:0/3540520895 shutdown_connections 2024-03-28T15:51:58.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:58.890+0000 7f7054200700 1 --2- 172.21.15.67:0/3540520895 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7034038fc0 0x7f703403b480 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:58.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:58.890+0000 7f7054200700 1 --2- 172.21.15.67:0/3540520895 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f704c1083a0 0x7f704c19d7c0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:58.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:58.890+0000 7f7054200700 1 -- 172.21.15.67:0/3540520895 >> 172.21.15.67:0/3540520895 conn(0x7f704c077280 msgr2=0x7f704c077d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:58.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:58.890+0000 7f7054200700 1 -- 172.21.15.67:0/3540520895 shutdown_connections 2024-03-28T15:51:58.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:51:58.890+0000 7f7054200700 1 -- 172.21.15.67:0/3540520895 wait complete. 2024-03-28T15:51:59.733 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi150 2024-03-28T15:51:59.733 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-28T15:51:59.733 DEBUG:teuthology.orchestra.run.smithi150:> dd of=/etc/ceph/ceph.conf 2024-03-28T15:51:59.789 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-28T15:51:59.789 DEBUG:teuthology.orchestra.run.smithi150:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-28T15:51:59.881 INFO:tasks.cephadm:Adding host smithi150 to orchestrator... 2024-03-28T15:51:59.881 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph orch host add smithi150 2024-03-28T15:51:59.957 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:59 smithi067 ceph-mon[31707]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:51:59.957 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:59 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:51:59.957 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:59 smithi067 ceph-mon[31707]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:51:59.957 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:59 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:51:59.957 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:59 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:51:59.957 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:59 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:51:59.957 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:59 smithi067 ceph-mon[31707]: Updating smithi067:/etc/ceph/ceph.conf 2024-03-28T15:51:59.958 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:51:59 smithi067 ceph-mon[31707]: Updating smithi067:/var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T15:52:00.305 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:52:01.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:00 smithi067 ceph-mon[31707]: Updating smithi067:/etc/ceph/ceph.client.admin.keyring 2024-03-28T15:52:02.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:02.315+0000 7efcf2200700 1 -- 172.21.15.67:0/2392348633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcec101dd0 msgr2=0x7efcec1021b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:02.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:02.315+0000 7efcf2200700 1 --2- 172.21.15.67:0/2392348633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcec101dd0 0x7efcec1021b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7efcdc009910 tx=0x7efcdc009c20 comp rx=0 tx=0).stop 2024-03-28T15:52:02.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:02.315+0000 7efcf2200700 1 -- 172.21.15.67:0/2392348633 shutdown_connections 2024-03-28T15:52:02.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:02.315+0000 7efcf2200700 1 --2- 172.21.15.67:0/2392348633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcec101dd0 0x7efcec1021b0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:02.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:02.315+0000 7efcf2200700 1 -- 172.21.15.67:0/2392348633 >> 172.21.15.67:0/2392348633 conn(0x7efcec0fd6a0 msgr2=0x7efcec0ffac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:02.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:02.315+0000 7efcf2200700 1 -- 172.21.15.67:0/2392348633 shutdown_connections 2024-03-28T15:52:02.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:02.316+0000 7efcf2200700 1 -- 172.21.15.67:0/2392348633 wait complete. 2024-03-28T15:52:02.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:02.317+0000 7efcf2200700 1 Processor -- start 2024-03-28T15:52:02.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:02.317+0000 7efcf2200700 1 -- start start 2024-03-28T15:52:02.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:02.318+0000 7efcf2200700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcec101dd0 0x7efcec1975e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:02.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:02.318+0000 7efcf2200700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcec10c050 con 0x7efcec101dd0 2024-03-28T15:52:02.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:02.318+0000 7efceb400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcec101dd0 0x7efcec1975e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:02.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:02.318+0000 7efceb400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcec101dd0 0x7efcec1975e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44122/0 (socket says 172.21.15.67:44122) 2024-03-28T15:52:02.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:02.318+0000 7efceb400700 1 -- 172.21.15.67:0/1660530440 learned_addr learned my addr 172.21.15.67:0/1660530440 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:52:02.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:02.318+0000 7efceb400700 1 -- 172.21.15.67:0/1660530440 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcec197b20 con 0x7efcec101dd0 2024-03-28T15:52:02.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:02.319+0000 7efceb400700 1 --2- 172.21.15.67:0/1660530440 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcec101dd0 0x7efcec1975e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7efcdc006010 tx=0x7efcdc005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:02.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:02.319+0000 7efce3e00700 1 -- 172.21.15.67:0/1660530440 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efcdc026070 con 0x7efcec101dd0 2024-03-28T15:52:02.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:02.319+0000 7efce3e00700 1 -- 172.21.15.67:0/1660530440 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7efcdc00bba0 con 0x7efcec101dd0 2024-03-28T15:52:02.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:02.319+0000 7efce3e00700 1 -- 172.21.15.67:0/1660530440 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efcdc0038a0 con 0x7efcec101dd0 2024-03-28T15:52:02.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:02.319+0000 7efcf2200700 1 -- 172.21.15.67:0/1660530440 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcec197e10 con 0x7efcec101dd0 2024-03-28T15:52:02.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:02.319+0000 7efcf2200700 1 -- 172.21.15.67:0/1660530440 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcec19beb0 con 0x7efcec101dd0 2024-03-28T15:52:02.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:02.320+0000 7efce3e00700 1 -- 172.21.15.67:0/1660530440 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7efcdc029070 con 0x7efcec101dd0 2024-03-28T15:52:02.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:02.320+0000 7efcf2200700 1 -- 172.21.15.67:0/1660530440 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcec1032b0 con 0x7efcec101dd0 2024-03-28T15:52:02.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:02.321+0000 7efce3e00700 1 --2- 172.21.15.67:0/1660530440 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7efcd0039020 0x7efcd003b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:02.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:02.321+0000 7efce3e00700 1 -- 172.21.15.67:0/1660530440 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7efcdc01e070 con 0x7efcec101dd0 2024-03-28T15:52:02.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:02.321+0000 7efceaa00700 1 --2- 172.21.15.67:0/1660530440 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7efcd0039020 0x7efcd003b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:02.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:02.322+0000 7efceaa00700 1 --2- 172.21.15.67:0/1660530440 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7efcd0039020 0x7efcd003b4e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7efcd8006fd0 tx=0x7efcd8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:02.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:02.327+0000 7efce3e00700 1 -- 172.21.15.67:0/1660530440 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7efcdc013070 con 0x7efcec101dd0 2024-03-28T15:52:02.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:02.558+0000 7efcf2200700 1 -- 172.21.15.67:0/1660530440 --> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi150", "target": ["mon-mgr", ""]}) v1 -- 0x7efcec198940 con 0x7efcd0039020 2024-03-28T15:52:02.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:02 smithi067 ceph-mon[31707]: Updating smithi067:/var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T15:52:02.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:02 smithi067 ceph-mon[31707]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:52:02.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:02 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:52:02.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:02 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:52:02.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:02 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:52:04.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:03 smithi067 ceph-mon[31707]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi150", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:52:04.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:04 smithi067 ceph-mon[31707]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:52:05.997 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:05 smithi067 ceph-mon[31707]: Deploying cephadm binary to smithi150 2024-03-28T15:52:06.666 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:06 smithi067 ceph-mon[31707]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:52:09.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:08 smithi067 ceph-mon[31707]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:52:09.285 INFO:teuthology.orchestra.run.smithi067.stdout:Added host 'smithi150' with addr '172.21.15.150' 2024-03-28T15:52:09.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:09.282+0000 7efce3e00700 1 -- 172.21.15.67:0/1660530440 <== mgr.14152 v2:172.21.15.67:6800/208726654 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7efcec198940 con 0x7efcd0039020 2024-03-28T15:52:09.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:09.286+0000 7efcf2200700 1 -- 172.21.15.67:0/1660530440 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7efcd0039020 msgr2=0x7efcd003b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:09.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:09.286+0000 7efcf2200700 1 --2- 172.21.15.67:0/1660530440 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7efcd0039020 0x7efcd003b4e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7efcd8006fd0 tx=0x7efcd8006e40 comp rx=0 tx=0).stop 2024-03-28T15:52:09.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:09.286+0000 7efcf2200700 1 -- 172.21.15.67:0/1660530440 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcec101dd0 msgr2=0x7efcec1975e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:09.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:09.286+0000 7efcf2200700 1 --2- 172.21.15.67:0/1660530440 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcec101dd0 0x7efcec1975e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7efcdc006010 tx=0x7efcdc005780 comp rx=0 tx=0).stop 2024-03-28T15:52:09.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:09.286+0000 7efcf2200700 1 -- 172.21.15.67:0/1660530440 shutdown_connections 2024-03-28T15:52:09.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:09.286+0000 7efcf2200700 1 --2- 172.21.15.67:0/1660530440 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7efcd0039020 0x7efcd003b4e0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:09.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:09.286+0000 7efcf2200700 1 --2- 172.21.15.67:0/1660530440 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcec101dd0 0x7efcec1975e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:09.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:09.286+0000 7efcf2200700 1 -- 172.21.15.67:0/1660530440 >> 172.21.15.67:0/1660530440 conn(0x7efcec0fd6a0 msgr2=0x7efcec1037a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:09.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:09.286+0000 7efcf2200700 1 -- 172.21.15.67:0/1660530440 shutdown_connections 2024-03-28T15:52:09.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:09.287+0000 7efcf2200700 1 -- 172.21.15.67:0/1660530440 wait complete. 2024-03-28T15:52:10.085 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph orch host ls --format=json 2024-03-28T15:52:10.538 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:52:10.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:10 smithi067 ceph-mon[31707]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:52:10.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:10 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:52:10.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:10 smithi067 ceph-mon[31707]: Added host smithi150 2024-03-28T15:52:10.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:10 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:52:10.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:10 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:52:12.234 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:11 smithi067 ceph-mon[31707]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:52:12.234 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:11 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:52:12.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:12.513+0000 7f6a2ee00700 1 -- 172.21.15.67:0/2699294384 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a28101dd0 msgr2=0x7f6a281021b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:12.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:12.513+0000 7f6a2ee00700 1 --2- 172.21.15.67:0/2699294384 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a28101dd0 0x7f6a281021b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6a18009910 tx=0x7f6a18009c20 comp rx=0 tx=0).stop 2024-03-28T15:52:12.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:12.513+0000 7f6a2ee00700 1 -- 172.21.15.67:0/2699294384 shutdown_connections 2024-03-28T15:52:12.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:12.513+0000 7f6a2ee00700 1 --2- 172.21.15.67:0/2699294384 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a28101dd0 0x7f6a281021b0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:12.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:12.513+0000 7f6a2ee00700 1 -- 172.21.15.67:0/2699294384 >> 172.21.15.67:0/2699294384 conn(0x7f6a280fd680 msgr2=0x7f6a280ffaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:12.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:12.513+0000 7f6a2ee00700 1 -- 172.21.15.67:0/2699294384 shutdown_connections 2024-03-28T15:52:12.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:12.513+0000 7f6a2ee00700 1 -- 172.21.15.67:0/2699294384 wait complete. 2024-03-28T15:52:12.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:12.514+0000 7f6a2ee00700 1 Processor -- start 2024-03-28T15:52:12.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:12.514+0000 7f6a2ee00700 1 -- start start 2024-03-28T15:52:12.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:12.515+0000 7f6a2ee00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a28101dd0 0x7f6a28074490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:12.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:12.515+0000 7f6a2ee00700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a2810c050 con 0x7f6a28101dd0 2024-03-28T15:52:12.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:12.515+0000 7f6a27e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a28101dd0 0x7f6a28074490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:12.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:12.515+0000 7f6a27e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a28101dd0 0x7f6a28074490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43418/0 (socket says 172.21.15.67:43418) 2024-03-28T15:52:12.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:12.515+0000 7f6a27e00700 1 -- 172.21.15.67:0/2502914791 learned_addr learned my addr 172.21.15.67:0/2502914791 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:52:12.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:12.516+0000 7f6a27e00700 1 -- 172.21.15.67:0/2502914791 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a28076380 con 0x7f6a28101dd0 2024-03-28T15:52:12.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:12.516+0000 7f6a27e00700 1 --2- 172.21.15.67:0/2502914791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a28101dd0 0x7f6a28074490 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6a18006010 tx=0x7f6a18005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:12.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:12.516+0000 7f6a24c00700 1 -- 172.21.15.67:0/2502914791 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6a18026070 con 0x7f6a28101dd0 2024-03-28T15:52:12.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:12.516+0000 7f6a24c00700 1 -- 172.21.15.67:0/2502914791 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6a1800bc00 con 0x7f6a28101dd0 2024-03-28T15:52:12.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:12.516+0000 7f6a2ee00700 1 -- 172.21.15.67:0/2502914791 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a28074a30 con 0x7f6a28101dd0 2024-03-28T15:52:12.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:12.516+0000 7f6a24c00700 1 -- 172.21.15.67:0/2502914791 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6a18003930 con 0x7f6a28101dd0 2024-03-28T15:52:12.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:12.516+0000 7f6a2ee00700 1 -- 172.21.15.67:0/2502914791 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a28074ef0 con 0x7f6a28101dd0 2024-03-28T15:52:12.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:12.518+0000 7f6a24c00700 1 -- 172.21.15.67:0/2502914791 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f6a18029070 con 0x7f6a28101dd0 2024-03-28T15:52:12.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:12.518+0000 7f6a2ee00700 1 -- 172.21.15.67:0/2502914791 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a281032b0 con 0x7f6a28101dd0 2024-03-28T15:52:12.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:12.518+0000 7f6a24c00700 1 --2- 172.21.15.67:0/2502914791 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f6a0c039020 0x7f6a0c03b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:12.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:12.518+0000 7f6a24c00700 1 -- 172.21.15.67:0/2502914791 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f6a1801e070 con 0x7f6a28101dd0 2024-03-28T15:52:12.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:12.519+0000 7f6a27400700 1 --2- 172.21.15.67:0/2502914791 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f6a0c039020 0x7f6a0c03b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:12.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:12.519+0000 7f6a27400700 1 --2- 172.21.15.67:0/2502914791 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f6a0c039020 0x7f6a0c03b4e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f6a14006fd0 tx=0x7f6a14006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:12.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:12.524+0000 7f6a24c00700 1 -- 172.21.15.67:0/2502914791 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6a18013070 con 0x7f6a28101dd0 2024-03-28T15:52:12.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:12.759+0000 7f6a2ee00700 1 -- 172.21.15.67:0/2502914791 --> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6a280759c0 con 0x7f6a0c039020 2024-03-28T15:52:12.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:12.763+0000 7f6a24c00700 1 -- 172.21.15.67:0/2502914791 <== mgr.14152 v2:172.21.15.67:6800/208726654 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+160 (secure 0 0 0) 0x7f6a280759c0 con 0x7f6a0c039020 2024-03-28T15:52:12.764 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T15:52:12.764 INFO:teuthology.orchestra.run.smithi067.stdout:[{"addr": "172.21.15.67", "hostname": "smithi067", "labels": [], "status": ""}, {"addr": "172.21.15.150", "hostname": "smithi150", "labels": [], "status": ""}] 2024-03-28T15:52:12.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:12.766+0000 7f6a2ee00700 1 -- 172.21.15.67:0/2502914791 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f6a0c039020 msgr2=0x7f6a0c03b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:12.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:12.766+0000 7f6a2ee00700 1 --2- 172.21.15.67:0/2502914791 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f6a0c039020 0x7f6a0c03b4e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f6a14006fd0 tx=0x7f6a14006e40 comp rx=0 tx=0).stop 2024-03-28T15:52:12.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:12.766+0000 7f6a2ee00700 1 -- 172.21.15.67:0/2502914791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a28101dd0 msgr2=0x7f6a28074490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:12.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:12.766+0000 7f6a2ee00700 1 --2- 172.21.15.67:0/2502914791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a28101dd0 0x7f6a28074490 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6a18006010 tx=0x7f6a18005780 comp rx=0 tx=0).stop 2024-03-28T15:52:12.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:12.766+0000 7f6a2ee00700 1 -- 172.21.15.67:0/2502914791 shutdown_connections 2024-03-28T15:52:12.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:12.766+0000 7f6a2ee00700 1 --2- 172.21.15.67:0/2502914791 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f6a0c039020 0x7f6a0c03b4e0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:12.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:12.766+0000 7f6a2ee00700 1 --2- 172.21.15.67:0/2502914791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a28101dd0 0x7f6a28074490 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:12.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:12.766+0000 7f6a2ee00700 1 -- 172.21.15.67:0/2502914791 >> 172.21.15.67:0/2502914791 conn(0x7f6a280fd680 msgr2=0x7f6a281037a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:12.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:12.766+0000 7f6a2ee00700 1 -- 172.21.15.67:0/2502914791 shutdown_connections 2024-03-28T15:52:12.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:12.766+0000 7f6a2ee00700 1 -- 172.21.15.67:0/2502914791 wait complete. 2024-03-28T15:52:13.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:12 smithi067 ceph-mon[31707]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T15:52:13.475 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi183 2024-03-28T15:52:13.475 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-28T15:52:13.476 DEBUG:teuthology.orchestra.run.smithi183:> dd of=/etc/ceph/ceph.conf 2024-03-28T15:52:13.534 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-28T15:52:13.534 DEBUG:teuthology.orchestra.run.smithi183:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-28T15:52:13.631 INFO:tasks.cephadm:Adding host smithi183 to orchestrator... 2024-03-28T15:52:13.632 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph orch host add smithi183 2024-03-28T15:52:14.098 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:52:14.428 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:14 smithi067 ceph-mon[31707]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:52:14.429 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:14 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:52:16.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:16.099+0000 7f4428200700 1 -- 172.21.15.67:0/2877675888 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4420101dc0 msgr2=0x7f44201021a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:16.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:16.099+0000 7f4428200700 1 --2- 172.21.15.67:0/2877675888 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4420101dc0 0x7f44201021a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4414009910 tx=0x7f4414009c20 comp rx=0 tx=0).stop 2024-03-28T15:52:16.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:16.099+0000 7f4428200700 1 -- 172.21.15.67:0/2877675888 shutdown_connections 2024-03-28T15:52:16.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:16.099+0000 7f4428200700 1 --2- 172.21.15.67:0/2877675888 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4420101dc0 0x7f44201021a0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:16.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:16.099+0000 7f4428200700 1 -- 172.21.15.67:0/2877675888 >> 172.21.15.67:0/2877675888 conn(0x7f44200fd670 msgr2=0x7f44200ffa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:16.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:16.099+0000 7f4428200700 1 -- 172.21.15.67:0/2877675888 shutdown_connections 2024-03-28T15:52:16.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:16.099+0000 7f4428200700 1 -- 172.21.15.67:0/2877675888 wait complete. 2024-03-28T15:52:16.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:16.100+0000 7f4428200700 1 Processor -- start 2024-03-28T15:52:16.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:16.100+0000 7f4428200700 1 -- start start 2024-03-28T15:52:16.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:16.101+0000 7f4428200700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f442019f6a0 0x7f442019fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:16.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:16.101+0000 7f4428200700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f442010c040 con 0x7f442019f6a0 2024-03-28T15:52:16.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:16.101+0000 7f4425800700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f442019f6a0 0x7f442019fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:16.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:16.101+0000 7f4425800700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f442019f6a0 0x7f442019fa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43432/0 (socket says 172.21.15.67:43432) 2024-03-28T15:52:16.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:16.101+0000 7f4425800700 1 -- 172.21.15.67:0/1031384372 learned_addr learned my addr 172.21.15.67:0/1031384372 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:52:16.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:16.101+0000 7f4425800700 1 -- 172.21.15.67:0/1031384372 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f442019ffc0 con 0x7f442019f6a0 2024-03-28T15:52:16.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:16.102+0000 7f4425800700 1 --2- 172.21.15.67:0/1031384372 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f442019f6a0 0x7f442019fa80 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f441400b5c0 tx=0x7f4414005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:16.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:16.102+0000 7f441e000700 1 -- 172.21.15.67:0/1031384372 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4414026070 con 0x7f442019f6a0 2024-03-28T15:52:16.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:16.102+0000 7f441e000700 1 -- 172.21.15.67:0/1031384372 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f441400b6f0 con 0x7f442019f6a0 2024-03-28T15:52:16.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:16.102+0000 7f4428200700 1 -- 172.21.15.67:0/1031384372 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4420199820 con 0x7f442019f6a0 2024-03-28T15:52:16.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:16.102+0000 7f441e000700 1 -- 172.21.15.67:0/1031384372 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4414003810 con 0x7f442019f6a0 2024-03-28T15:52:16.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:16.102+0000 7f4428200700 1 -- 172.21.15.67:0/1031384372 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4420199ce0 con 0x7f442019f6a0 2024-03-28T15:52:16.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:16.103+0000 7f4428200700 1 -- 172.21.15.67:0/1031384372 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44201032a0 con 0x7f442019f6a0 2024-03-28T15:52:16.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:16.103+0000 7f441e000700 1 -- 172.21.15.67:0/1031384372 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f4414029070 con 0x7f442019f6a0 2024-03-28T15:52:16.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:16.104+0000 7f441e000700 1 --2- 172.21.15.67:0/1031384372 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f4408038fc0 0x7f440803b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:16.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:16.104+0000 7f441e000700 1 -- 172.21.15.67:0/1031384372 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f441401e070 con 0x7f442019f6a0 2024-03-28T15:52:16.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:16.104+0000 7f4424e00700 1 --2- 172.21.15.67:0/1031384372 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f4408038fc0 0x7f440803b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:16.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:16.105+0000 7f4424e00700 1 --2- 172.21.15.67:0/1031384372 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f4408038fc0 0x7f440803b480 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f440c006fd0 tx=0x7f440c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:16.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:16.110+0000 7f441e000700 1 -- 172.21.15.67:0/1031384372 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4414013070 con 0x7f442019f6a0 2024-03-28T15:52:16.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:16 smithi067 ceph-mon[31707]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:52:16.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:16.346+0000 7f4428200700 1 -- 172.21.15.67:0/1031384372 --> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi183", "target": ["mon-mgr", ""]}) v1 -- 0x7f44201a0190 con 0x7f4408038fc0 2024-03-28T15:52:17.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:17 smithi067 ceph-mon[31707]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi183", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:52:18.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:18 smithi067 ceph-mon[31707]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:52:18.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:18 smithi067 ceph-mon[31707]: Deploying cephadm binary to smithi183 2024-03-28T15:52:18.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:18 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:52:18.571 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:18 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:52:18.571 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:18 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:52:18.571 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:18 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:52:18.571 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:18 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi150", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:52:18.571 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:18 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:52:18.571 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:18 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:52:19.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:19 smithi067 ceph-mon[31707]: Updating smithi150:/etc/ceph/ceph.conf 2024-03-28T15:52:20.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:20 smithi067 ceph-mon[31707]: Updating smithi150:/var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T15:52:20.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:20 smithi067 ceph-mon[31707]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:52:22.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:22 smithi067 ceph-mon[31707]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:52:23.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:23.092+0000 7f441e000700 1 -- 172.21.15.67:0/1031384372 <== mgr.14152 v2:172.21.15.67:6800/208726654 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f44201a0190 con 0x7f4408038fc0 2024-03-28T15:52:23.093 INFO:teuthology.orchestra.run.smithi067.stdout:Added host 'smithi183' with addr '172.21.15.183' 2024-03-28T15:52:23.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:23.096+0000 7f4428200700 1 -- 172.21.15.67:0/1031384372 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f4408038fc0 msgr2=0x7f440803b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:23.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:23.096+0000 7f4428200700 1 --2- 172.21.15.67:0/1031384372 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f4408038fc0 0x7f440803b480 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f440c006fd0 tx=0x7f440c006e40 comp rx=0 tx=0).stop 2024-03-28T15:52:23.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:23.096+0000 7f4428200700 1 -- 172.21.15.67:0/1031384372 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f442019f6a0 msgr2=0x7f442019fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:23.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:23.096+0000 7f4428200700 1 --2- 172.21.15.67:0/1031384372 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f442019f6a0 0x7f442019fa80 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f441400b5c0 tx=0x7f4414005780 comp rx=0 tx=0).stop 2024-03-28T15:52:23.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:23.097+0000 7f4428200700 1 -- 172.21.15.67:0/1031384372 shutdown_connections 2024-03-28T15:52:23.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:23.097+0000 7f4428200700 1 --2- 172.21.15.67:0/1031384372 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f4408038fc0 0x7f440803b480 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:23.097+0000 7f4428200700 1 --2- 172.21.15.67:0/1031384372 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f442019f6a0 0x7f442019fa80 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:23.097+0000 7f4428200700 1 -- 172.21.15.67:0/1031384372 >> 172.21.15.67:0/1031384372 conn(0x7f44200fd670 msgr2=0x7f4420074c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:23.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:23.097+0000 7f4428200700 1 -- 172.21.15.67:0/1031384372 shutdown_connections 2024-03-28T15:52:23.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:23.097+0000 7f4428200700 1 -- 172.21.15.67:0/1031384372 wait complete. 2024-03-28T15:52:23.514 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:23 smithi067 ceph-mon[31707]: Updating smithi150:/etc/ceph/ceph.client.admin.keyring 2024-03-28T15:52:23.514 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:23 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:52:23.914 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph orch host ls --format=json 2024-03-28T15:52:24.333 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:52:25.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:24 smithi067 ceph-mon[31707]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:52:25.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:24 smithi067 ceph-mon[31707]: Updating smithi150:/var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T15:52:25.071 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:24 smithi067 ceph-mon[31707]: Added host smithi183 2024-03-28T15:52:25.071 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:24 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:52:25.071 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:24 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:52:25.071 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:24 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:52:25.071 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:24 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:52:26.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:26.253+0000 7f24af000700 1 -- 172.21.15.67:0/2916649266 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24a8074560 msgr2=0x7f24a8074940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:26.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:26.253+0000 7f24af000700 1 --2- 172.21.15.67:0/2916649266 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24a8074560 0x7f24a8074940 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f2498009910 tx=0x7f2498009c20 comp rx=0 tx=0).stop 2024-03-28T15:52:26.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:26.253+0000 7f24af000700 1 -- 172.21.15.67:0/2916649266 shutdown_connections 2024-03-28T15:52:26.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:26.253+0000 7f24af000700 1 --2- 172.21.15.67:0/2916649266 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24a8074560 0x7f24a8074940 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:26.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:26.253+0000 7f24af000700 1 -- 172.21.15.67:0/2916649266 >> 172.21.15.67:0/2916649266 conn(0x7f24a80fd6a0 msgr2=0x7f24a80ffac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:26.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:26.253+0000 7f24af000700 1 -- 172.21.15.67:0/2916649266 shutdown_connections 2024-03-28T15:52:26.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:26.253+0000 7f24af000700 1 -- 172.21.15.67:0/2916649266 wait complete. 2024-03-28T15:52:26.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:26.255+0000 7f24af000700 1 Processor -- start 2024-03-28T15:52:26.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:26.255+0000 7f24af000700 1 -- start start 2024-03-28T15:52:26.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:26.255+0000 7f24af000700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24a8074560 0x7f24a819f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:26.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:26.255+0000 7f24af000700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24a810e2c0 con 0x7f24a8074560 2024-03-28T15:52:26.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:26.256+0000 7f24a7e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24a8074560 0x7f24a819f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:26.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:26.256+0000 7f24a7e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24a8074560 0x7f24a819f590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59036/0 (socket says 172.21.15.67:59036) 2024-03-28T15:52:26.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:26.256+0000 7f24a7e00700 1 -- 172.21.15.67:0/2311980720 learned_addr learned my addr 172.21.15.67:0/2311980720 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:52:26.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:26.256+0000 7f24a7e00700 1 -- 172.21.15.67:0/2311980720 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24a819fad0 con 0x7f24a8074560 2024-03-28T15:52:26.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:26.256+0000 7f24a7e00700 1 --2- 172.21.15.67:0/2311980720 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24a8074560 0x7f24a819f590 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f24980056d0 tx=0x7f24980057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:26.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:26.257+0000 7f24a4c00700 1 -- 172.21.15.67:0/2311980720 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2498026070 con 0x7f24a8074560 2024-03-28T15:52:26.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:26.257+0000 7f24a4c00700 1 -- 172.21.15.67:0/2311980720 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f2498004370 con 0x7f24a8074560 2024-03-28T15:52:26.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:26.257+0000 7f24af000700 1 -- 172.21.15.67:0/2311980720 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24a819fd60 con 0x7f24a8074560 2024-03-28T15:52:26.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:26.257+0000 7f24a4c00700 1 -- 172.21.15.67:0/2311980720 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f24980039b0 con 0x7f24a8074560 2024-03-28T15:52:26.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:26.257+0000 7f24af000700 1 -- 172.21.15.67:0/2311980720 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24a8199660 con 0x7f24a8074560 2024-03-28T15:52:26.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:26.258+0000 7f24af000700 1 -- 172.21.15.67:0/2311980720 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24a8075c10 con 0x7f24a8074560 2024-03-28T15:52:26.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:26.258+0000 7f24a4c00700 1 -- 172.21.15.67:0/2311980720 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f2498029070 con 0x7f24a8074560 2024-03-28T15:52:26.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:26.259+0000 7f24a4c00700 1 --2- 172.21.15.67:0/2311980720 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f248c039020 0x7f248c03b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:26.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:26.259+0000 7f24a4c00700 1 -- 172.21.15.67:0/2311980720 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f249801e070 con 0x7f24a8074560 2024-03-28T15:52:26.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:26.259+0000 7f24a7400700 1 --2- 172.21.15.67:0/2311980720 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f248c039020 0x7f248c03b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:26.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:26.260+0000 7f24a7400700 1 --2- 172.21.15.67:0/2311980720 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f248c039020 0x7f248c03b4e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2490006fd0 tx=0x7f2490006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:26.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:26.264+0000 7f24a4c00700 1 -- 172.21.15.67:0/2311980720 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2498013070 con 0x7f24a8074560 2024-03-28T15:52:26.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:26.497+0000 7f24af000700 1 -- 172.21.15.67:0/2311980720 --> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f24a819ab90 con 0x7f248c039020 2024-03-28T15:52:26.579 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:26 smithi067 ceph-mon[31707]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:52:26.579 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:26 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:52:26.579 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:26 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:52:26.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:26.499+0000 7f24a4c00700 1 -- 172.21.15.67:0/2311980720 <== mgr.14152 v2:172.21.15.67:6800/208726654 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+240 (secure 0 0 0) 0x7f24a819ab90 con 0x7f248c039020 2024-03-28T15:52:26.580 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T15:52:26.580 INFO:teuthology.orchestra.run.smithi067.stdout:[{"addr": "172.21.15.67", "hostname": "smithi067", "labels": [], "status": ""}, {"addr": "172.21.15.150", "hostname": "smithi150", "labels": [], "status": ""}, {"addr": "172.21.15.183", "hostname": "smithi183", "labels": [], "status": ""}] 2024-03-28T15:52:26.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:26.503+0000 7f24af000700 1 -- 172.21.15.67:0/2311980720 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f248c039020 msgr2=0x7f248c03b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:26.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:26.503+0000 7f24af000700 1 --2- 172.21.15.67:0/2311980720 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f248c039020 0x7f248c03b4e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2490006fd0 tx=0x7f2490006e40 comp rx=0 tx=0).stop 2024-03-28T15:52:26.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:26.503+0000 7f24af000700 1 -- 172.21.15.67:0/2311980720 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24a8074560 msgr2=0x7f24a819f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:26.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:26.503+0000 7f24af000700 1 --2- 172.21.15.67:0/2311980720 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24a8074560 0x7f24a819f590 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f24980056d0 tx=0x7f24980057b0 comp rx=0 tx=0).stop 2024-03-28T15:52:26.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:26.504+0000 7f24af000700 1 -- 172.21.15.67:0/2311980720 shutdown_connections 2024-03-28T15:52:26.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:26.504+0000 7f24af000700 1 --2- 172.21.15.67:0/2311980720 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f248c039020 0x7f248c03b4e0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:26.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:26.504+0000 7f24af000700 1 --2- 172.21.15.67:0/2311980720 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24a8074560 0x7f24a819f590 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:26.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:26.504+0000 7f24af000700 1 -- 172.21.15.67:0/2311980720 >> 172.21.15.67:0/2311980720 conn(0x7f24a80fd6a0 msgr2=0x7f24a8105e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:26.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:26.504+0000 7f24af000700 1 -- 172.21.15.67:0/2311980720 shutdown_connections 2024-03-28T15:52:26.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:26.504+0000 7f24af000700 1 -- 172.21.15.67:0/2311980720 wait complete. 2024-03-28T15:52:27.220 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-28T15:52:27.220 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph osd crush tunables default 2024-03-28T15:52:27.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:27 smithi067 ceph-mon[31707]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T15:52:27.684 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:52:28.559 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:28 smithi067 ceph-mon[31707]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:52:28.559 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:28 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:52:29.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:29.623+0000 7f9988e00700 1 -- 172.21.15.67:0/225216336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9984102d20 msgr2=0x7f9984103100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:29.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:29.623+0000 7f9988e00700 1 --2- 172.21.15.67:0/225216336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9984102d20 0x7f9984103100 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f997400b0a0 tx=0x7f997400b3b0 comp rx=0 tx=0).stop 2024-03-28T15:52:29.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:29.623+0000 7f9988e00700 1 -- 172.21.15.67:0/225216336 shutdown_connections 2024-03-28T15:52:29.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:29.623+0000 7f9988e00700 1 --2- 172.21.15.67:0/225216336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9984102d20 0x7f9984103100 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:29.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:29.623+0000 7f9988e00700 1 -- 172.21.15.67:0/225216336 >> 172.21.15.67:0/225216336 conn(0x7f99840776d0 msgr2=0x7f99840fe9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:29.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:29.623+0000 7f9988e00700 1 -- 172.21.15.67:0/225216336 shutdown_connections 2024-03-28T15:52:29.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:29.623+0000 7f9988e00700 1 -- 172.21.15.67:0/225216336 wait complete. 2024-03-28T15:52:29.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:29.625+0000 7f9988e00700 1 Processor -- start 2024-03-28T15:52:29.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:29.625+0000 7f9988e00700 1 -- start start 2024-03-28T15:52:29.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:29.625+0000 7f9988e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9984102d20 0x7f9984197580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:29.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:29.625+0000 7f9988e00700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f998410c0c0 con 0x7f9984102d20 2024-03-28T15:52:29.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:29.626+0000 7f9981e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9984102d20 0x7f9984197580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:29.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:29.626+0000 7f9981e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9984102d20 0x7f9984197580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50658/0 (socket says 172.21.15.67:50658) 2024-03-28T15:52:29.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:29.626+0000 7f9981e00700 1 -- 172.21.15.67:0/3373292310 learned_addr learned my addr 172.21.15.67:0/3373292310 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:52:29.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:29.626+0000 7f9981e00700 1 -- 172.21.15.67:0/3373292310 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9984197ac0 con 0x7f9984102d20 2024-03-28T15:52:29.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:29.627+0000 7f9981e00700 1 --2- 172.21.15.67:0/3373292310 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9984102d20 0x7f9984197580 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f9974000f80 tx=0x7f9974009790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:29.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:29.627+0000 7f997b400700 1 -- 172.21.15.67:0/3373292310 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9974017040 con 0x7f9984102d20 2024-03-28T15:52:29.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:29.627+0000 7f997b400700 1 -- 172.21.15.67:0/3373292310 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9974022070 con 0x7f9984102d20 2024-03-28T15:52:29.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:29.627+0000 7f9988e00700 1 -- 172.21.15.67:0/3373292310 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9984197d50 con 0x7f9984102d20 2024-03-28T15:52:29.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:29.627+0000 7f997b400700 1 -- 172.21.15.67:0/3373292310 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9974004910 con 0x7f9984102d20 2024-03-28T15:52:29.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:29.627+0000 7f9988e00700 1 -- 172.21.15.67:0/3373292310 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99841981b0 con 0x7f9984102d20 2024-03-28T15:52:29.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:29.628+0000 7f9988e00700 1 -- 172.21.15.67:0/3373292310 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9984069e60 con 0x7f9984102d20 2024-03-28T15:52:29.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:29.628+0000 7f997b400700 1 -- 172.21.15.67:0/3373292310 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f9974004040 con 0x7f9984102d20 2024-03-28T15:52:29.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:29.629+0000 7f997b400700 1 --2- 172.21.15.67:0/3373292310 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f9968039060 0x7f996803b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:29.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:29.629+0000 7f997b400700 1 -- 172.21.15.67:0/3373292310 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f997400f050 con 0x7f9984102d20 2024-03-28T15:52:29.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:29.629+0000 7f9979400700 1 --2- 172.21.15.67:0/3373292310 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f9968039060 0x7f996803b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:29.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:29.630+0000 7f9979400700 1 --2- 172.21.15.67:0/3373292310 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f9968039060 0x7f996803b520 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f996c006fd0 tx=0x7f996c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:29.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:29.635+0000 7f997b400700 1 -- 172.21.15.67:0/3373292310 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f997401c5f0 con 0x7f9984102d20 2024-03-28T15:52:29.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:29.868+0000 7f9988e00700 1 -- 172.21.15.67:0/3373292310 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f99841988e0 con 0x7f9984102d20 2024-03-28T15:52:30.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:30.124+0000 7f997b400700 1 -- 172.21.15.67:0/3373292310 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f9974022220 con 0x7f9984102d20 2024-03-28T15:52:30.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:30.129+0000 7f9988e00700 1 -- 172.21.15.67:0/3373292310 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f9968039060 msgr2=0x7f996803b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:30.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:30.129+0000 7f9988e00700 1 --2- 172.21.15.67:0/3373292310 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f9968039060 0x7f996803b520 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f996c006fd0 tx=0x7f996c006e40 comp rx=0 tx=0).stop 2024-03-28T15:52:30.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:30.129+0000 7f9988e00700 1 -- 172.21.15.67:0/3373292310 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9984102d20 msgr2=0x7f9984197580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:30.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:30.129+0000 7f9988e00700 1 --2- 172.21.15.67:0/3373292310 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9984102d20 0x7f9984197580 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f9974000f80 tx=0x7f9974009790 comp rx=0 tx=0).stop 2024-03-28T15:52:30.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:30.129+0000 7f9988e00700 1 -- 172.21.15.67:0/3373292310 shutdown_connections 2024-03-28T15:52:30.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:30.129+0000 7f9988e00700 1 --2- 172.21.15.67:0/3373292310 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f9968039060 0x7f996803b520 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:30.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:30.129+0000 7f9988e00700 1 --2- 172.21.15.67:0/3373292310 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9984102d20 0x7f9984197580 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:30.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:30.129+0000 7f9988e00700 1 -- 172.21.15.67:0/3373292310 >> 172.21.15.67:0/3373292310 conn(0x7f99840776d0 msgr2=0x7f9984103ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:30.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:30.130+0000 7f9988e00700 1 -- 172.21.15.67:0/3373292310 shutdown_connections 2024-03-28T15:52:30.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:52:30.130+0000 7f9988e00700 1 -- 172.21.15.67:0/3373292310 wait complete. 2024-03-28T15:52:30.132 INFO:teuthology.orchestra.run.smithi067.stderr:adjusted tunables profile to default 2024-03-28T15:52:30.480 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:30 smithi067 ceph-mon[31707]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:52:30.480 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:30 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/3373292310' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-28T15:52:30.871 INFO:tasks.cephadm:Adding mon.a on smithi067 2024-03-28T15:52:30.871 INFO:tasks.cephadm:Adding mon.b on smithi150 2024-03-28T15:52:30.871 INFO:tasks.cephadm:Adding mon.c on smithi183 2024-03-28T15:52:30.871 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph orch apply mon '3;smithi067:172.21.15.67=a;smithi150:172.21.15.150=b;smithi183:172.21.15.183=c' 2024-03-28T15:52:31.297 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-28T15:52:31.403 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-28T15:52:31.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:31 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/3373292310' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-28T15:52:31.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:31 smithi067 ceph-mon[31707]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T15:52:32.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:32 smithi067 ceph-mon[31707]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:52:34.101 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:34.100+0000 7f020e400700 1 -- 172.21.15.183:0/3461158540 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0208101bc0 msgr2=0x7f0208101fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:34.101 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:34.100+0000 7f020e400700 1 --2- 172.21.15.183:0/3461158540 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0208101bc0 0x7f0208101fe0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f01f8009910 tx=0x7f01f8009c20 comp rx=0 tx=0).stop 2024-03-28T15:52:34.101 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:34.100+0000 7f020e400700 1 -- 172.21.15.183:0/3461158540 shutdown_connections 2024-03-28T15:52:34.101 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:34.100+0000 7f020e400700 1 --2- 172.21.15.183:0/3461158540 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0208101bc0 0x7f0208101fe0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:34.101 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:34.100+0000 7f020e400700 1 -- 172.21.15.183:0/3461158540 >> 172.21.15.183:0/3461158540 conn(0x7f02080fd160 msgr2=0x7f02080ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:34.101 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:34.100+0000 7f020e400700 1 -- 172.21.15.183:0/3461158540 shutdown_connections 2024-03-28T15:52:34.101 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:34.100+0000 7f020e400700 1 -- 172.21.15.183:0/3461158540 wait complete. 2024-03-28T15:52:34.102 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:34.101+0000 7f020e400700 1 Processor -- start 2024-03-28T15:52:34.102 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:34.101+0000 7f020e400700 1 -- start start 2024-03-28T15:52:34.102 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:34.101+0000 7f020e400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0208101bc0 0x7f0208199420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:34.102 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:34.101+0000 7f020e400700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0208107b00 con 0x7f0208101bc0 2024-03-28T15:52:34.103 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:34.102+0000 7f0207400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0208101bc0 0x7f0208199420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:34.103 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:34.102+0000 7f0207400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0208101bc0 0x7f0208199420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.183:37128/0 (socket says 172.21.15.183:37128) 2024-03-28T15:52:34.103 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:34.102+0000 7f0207400700 1 -- 172.21.15.183:0/1112342491 learned_addr learned my addr 172.21.15.183:0/1112342491 (peer_addr_for_me v2:172.21.15.183:0/0) 2024-03-28T15:52:34.103 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:34.103+0000 7f0207400700 1 -- 172.21.15.183:0/1112342491 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0208199960 con 0x7f0208101bc0 2024-03-28T15:52:34.103 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:34.103+0000 7f0207400700 1 --2- 172.21.15.183:0/1112342491 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0208101bc0 0x7f0208199420 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f01f8005950 tx=0x7f01f8004fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:34.104 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:34.104+0000 7f0204c00700 1 -- 172.21.15.183:0/1112342491 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f01f8026070 con 0x7f0208101bc0 2024-03-28T15:52:34.104 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:34.104+0000 7f020e400700 1 -- 172.21.15.183:0/1112342491 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0208199bf0 con 0x7f0208101bc0 2024-03-28T15:52:34.104 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:34.104+0000 7f0204c00700 1 -- 172.21.15.183:0/1112342491 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f01f800bc00 con 0x7f0208101bc0 2024-03-28T15:52:34.104 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:34.104+0000 7f0204c00700 1 -- 172.21.15.183:0/1112342491 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f01f80039a0 con 0x7f0208101bc0 2024-03-28T15:52:34.104 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:34.104+0000 7f020e400700 1 -- 172.21.15.183:0/1112342491 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f020819c8e0 con 0x7f0208101bc0 2024-03-28T15:52:34.105 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:34.105+0000 7f020e400700 1 -- 172.21.15.183:0/1112342491 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0208108280 con 0x7f0208101bc0 2024-03-28T15:52:34.105 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:34.106+0000 7f0204c00700 1 -- 172.21.15.183:0/1112342491 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f01f8029070 con 0x7f0208101bc0 2024-03-28T15:52:34.106 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:34.106+0000 7f0204c00700 1 --2- 172.21.15.183:0/1112342491 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f01ec039020 0x7f01ec03b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:34.106 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:34.106+0000 7f0204c00700 1 -- 172.21.15.183:0/1112342491 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f01f801e070 con 0x7f0208101bc0 2024-03-28T15:52:34.107 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:34.107+0000 7f01ffe00700 1 --2- 172.21.15.183:0/1112342491 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f01ec039020 0x7f01ec03b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:34.108 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:34.108+0000 7f01ffe00700 1 --2- 172.21.15.183:0/1112342491 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f01ec039020 0x7f01ec03b4e0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f01f0006fd0 tx=0x7f01f0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:34.123 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:34.123+0000 7f0204c00700 1 -- 172.21.15.183:0/1112342491 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f01f8013070 con 0x7f0208101bc0 2024-03-28T15:52:34.363 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:34.363+0000 7f020e400700 1 -- 172.21.15.183:0/1112342491 --> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi067:172.21.15.67=a;smithi150:172.21.15.150=b;smithi183:172.21.15.183=c", "target": ["mon-mgr", ""]}) v1 -- 0x7f020802dcc0 con 0x7f01ec039020 2024-03-28T15:52:34.380 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:34.379+0000 7f0204c00700 1 -- 172.21.15.183:0/1112342491 <== mgr.14152 v2:172.21.15.67:6800/208726654 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f020802dcc0 con 0x7f01ec039020 2024-03-28T15:52:34.380 INFO:teuthology.orchestra.run.smithi183.stdout:Scheduled mon update... 2024-03-28T15:52:34.383 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:34.382+0000 7f020e400700 1 -- 172.21.15.183:0/1112342491 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f01ec039020 msgr2=0x7f01ec03b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:34.383 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:34.382+0000 7f020e400700 1 --2- 172.21.15.183:0/1112342491 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f01ec039020 0x7f01ec03b4e0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f01f0006fd0 tx=0x7f01f0006e40 comp rx=0 tx=0).stop 2024-03-28T15:52:34.383 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:34.382+0000 7f020e400700 1 -- 172.21.15.183:0/1112342491 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0208101bc0 msgr2=0x7f0208199420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:34.383 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:34.382+0000 7f020e400700 1 --2- 172.21.15.183:0/1112342491 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0208101bc0 0x7f0208199420 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f01f8005950 tx=0x7f01f8004fa0 comp rx=0 tx=0).stop 2024-03-28T15:52:34.383 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:34.382+0000 7f020e400700 1 -- 172.21.15.183:0/1112342491 shutdown_connections 2024-03-28T15:52:34.383 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:34.382+0000 7f020e400700 1 --2- 172.21.15.183:0/1112342491 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f01ec039020 0x7f01ec03b4e0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:34.383 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:34.382+0000 7f020e400700 1 --2- 172.21.15.183:0/1112342491 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0208101bc0 0x7f0208199420 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:34.383 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:34.382+0000 7f020e400700 1 -- 172.21.15.183:0/1112342491 >> 172.21.15.183:0/1112342491 conn(0x7f02080fd160 msgr2=0x7f02080ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:34.383 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:34.383+0000 7f020e400700 1 -- 172.21.15.183:0/1112342491 shutdown_connections 2024-03-28T15:52:34.383 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:34.383+0000 7f020e400700 1 -- 172.21.15.183:0/1112342491 wait complete. 2024-03-28T15:52:34.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:34 smithi067 ceph-mon[31707]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:52:34.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:34 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:52:34.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:34 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:52:34.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:34 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:52:34.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:34 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:52:34.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:34 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi183", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:52:34.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:34 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:52:34.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:34 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:52:34.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:34 smithi067 ceph-mon[31707]: Updating smithi183:/etc/ceph/ceph.conf 2024-03-28T15:52:34.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:34 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:52:35.241 DEBUG:teuthology.orchestra.run.smithi150:mon.b> sudo journalctl -f -n 0 -u ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@mon.b.service 2024-03-28T15:52:35.248 DEBUG:teuthology.orchestra.run.smithi183:mon.c> sudo journalctl -f -n 0 -u ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@mon.c.service 2024-03-28T15:52:35.252 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-28T15:52:35.253 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-28T15:52:35.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:35 smithi067 ceph-mon[31707]: Updating smithi183:/var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T15:52:35.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:35 smithi067 ceph-mon[31707]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi067:172.21.15.67=a;smithi150:172.21.15.150=b;smithi183:172.21.15.183=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:52:35.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:35 smithi067 ceph-mon[31707]: Saving service mon spec with placement smithi067:172.21.15.67=a;smithi150:172.21.15.150=b;smithi183:172.21.15.183=c;count:3 2024-03-28T15:52:35.981 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T15:52:36.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:36 smithi067 ceph-mon[31707]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:52:36.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:36 smithi067 ceph-mon[31707]: Updating smithi183:/etc/ceph/ceph.client.admin.keyring 2024-03-28T15:52:36.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:36 smithi067 ceph-mon[31707]: Updating smithi183:/var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T15:52:36.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:36 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:52:36.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:36 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:52:36.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:36 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:52:36.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:36 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T15:52:36.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:36 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:52:37.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:37 smithi067 ceph-mon[31707]: Deploying daemon mon.c on smithi183 2024-03-28T15:52:38.559 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:38.291+0000 7f2c1e000700 1 -- 172.21.15.183:0/2206185178 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2c10005460 con 0x7f2c20101dc0 2024-03-28T15:52:38.559 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:38.292+0000 7f2c26600700 1 -- 172.21.15.183:0/2206185178 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c20101dc0 msgr2=0x7f2c201021a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:38.559 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:38.292+0000 7f2c26600700 1 --2- 172.21.15.183:0/2206185178 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c20101dc0 0x7f2c201021a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2c10009910 tx=0x7f2c10009c20 comp rx=0 tx=0).stop 2024-03-28T15:52:38.559 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:38.292+0000 7f2c26600700 1 -- 172.21.15.183:0/2206185178 shutdown_connections 2024-03-28T15:52:38.560 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:38.292+0000 7f2c26600700 1 --2- 172.21.15.183:0/2206185178 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c20101dc0 0x7f2c201021a0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:38.560 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:38.292+0000 7f2c26600700 1 -- 172.21.15.183:0/2206185178 >> 172.21.15.183:0/2206185178 conn(0x7f2c200fd670 msgr2=0x7f2c200ffa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:38.560 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:38.292+0000 7f2c26600700 1 -- 172.21.15.183:0/2206185178 shutdown_connections 2024-03-28T15:52:38.560 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:38.293+0000 7f2c26600700 1 -- 172.21.15.183:0/2206185178 wait complete. 2024-03-28T15:52:38.560 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:38.294+0000 7f2c26600700 1 Processor -- start 2024-03-28T15:52:38.560 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:38.294+0000 7f2c26600700 1 -- start start 2024-03-28T15:52:38.560 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:38.294+0000 7f2c26600700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c20101dc0 0x7f2c201975c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:38.560 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:38.294+0000 7f2c26600700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c2010c040 con 0x7f2c20101dc0 2024-03-28T15:52:38.560 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:38.295+0000 7f2c1f400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c20101dc0 0x7f2c201975c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:38.560 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:38.295+0000 7f2c1f400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c20101dc0 0x7f2c201975c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.183:37144/0 (socket says 172.21.15.183:37144) 2024-03-28T15:52:38.560 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:38.295+0000 7f2c1f400700 1 -- 172.21.15.183:0/3722418283 learned_addr learned my addr 172.21.15.183:0/3722418283 (peer_addr_for_me v2:172.21.15.183:0/0) 2024-03-28T15:52:38.560 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:38.296+0000 7f2c1f400700 1 -- 172.21.15.183:0/3722418283 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c20197b00 con 0x7f2c20101dc0 2024-03-28T15:52:38.560 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:38.296+0000 7f2c1f400700 1 --2- 172.21.15.183:0/3722418283 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c20101dc0 0x7f2c201975c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f2c100056d0 tx=0x7f2c100057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:38.560 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:38.296+0000 7f2c17e00700 1 -- 172.21.15.183:0/3722418283 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2c10026070 con 0x7f2c20101dc0 2024-03-28T15:52:38.561 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:38.296+0000 7f2c26600700 1 -- 172.21.15.183:0/3722418283 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c20197df0 con 0x7f2c20101dc0 2024-03-28T15:52:38.561 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:38.296+0000 7f2c17e00700 1 -- 172.21.15.183:0/3722418283 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f2c1000bba0 con 0x7f2c20101dc0 2024-03-28T15:52:38.561 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:38.296+0000 7f2c17e00700 1 -- 172.21.15.183:0/3722418283 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2c100037f0 con 0x7f2c20101dc0 2024-03-28T15:52:38.561 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:38.297+0000 7f2c26600700 1 -- 172.21.15.183:0/3722418283 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c2019be90 con 0x7f2c20101dc0 2024-03-28T15:52:38.561 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:38.298+0000 7f2c26600700 1 -- 172.21.15.183:0/3722418283 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c201032a0 con 0x7f2c20101dc0 2024-03-28T15:52:38.561 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:38.298+0000 7f2c17e00700 1 -- 172.21.15.183:0/3722418283 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f2c10029070 con 0x7f2c20101dc0 2024-03-28T15:52:38.561 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:38.299+0000 7f2c17e00700 1 --2- 172.21.15.183:0/3722418283 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f2c04038fc0 0x7f2c0403b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:38.561 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:38.299+0000 7f2c17e00700 1 -- 172.21.15.183:0/3722418283 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f2c1001e070 con 0x7f2c20101dc0 2024-03-28T15:52:38.561 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:38.300+0000 7f2c1ea00700 1 --2- 172.21.15.183:0/3722418283 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f2c04038fc0 0x7f2c0403b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:38.561 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:38.300+0000 7f2c1ea00700 1 --2- 172.21.15.183:0/3722418283 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f2c04038fc0 0x7f2c0403b480 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c006fd0 tx=0x7f2c0c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:38.561 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:38.315+0000 7f2c17e00700 1 -- 172.21.15.183:0/3722418283 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2c10013070 con 0x7f2c20101dc0 2024-03-28T15:52:38.621 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:38.621+0000 7f2c26600700 1 -- 172.21.15.183:0/3722418283 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2c20198920 con 0x7f2c20101dc0 2024-03-28T15:52:38.622 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:38.622+0000 7f2c17e00700 1 -- 172.21.15.183:0/3722418283 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+745 (secure 0 0 0) 0x7f2c10030030 con 0x7f2c20101dc0 2024-03-28T15:52:38.622 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T15:52:38.623 INFO:teuthology.orchestra.run.smithi183.stdout:{"epoch":1,"fsid":"b5f8225a-ed1a-11ee-95d0-87774f69a715","modified":"2024-03-28T15:50:17.125940Z","created":"2024-03-28T15:50:17.125940Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-28T15:52:38.626 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:38.625+0000 7f2c26600700 1 -- 172.21.15.183:0/3722418283 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f2c04038fc0 msgr2=0x7f2c0403b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:38.626 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:38.625+0000 7f2c26600700 1 --2- 172.21.15.183:0/3722418283 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f2c04038fc0 0x7f2c0403b480 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c006fd0 tx=0x7f2c0c006e40 comp rx=0 tx=0).stop 2024-03-28T15:52:38.626 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:38.625+0000 7f2c26600700 1 -- 172.21.15.183:0/3722418283 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c20101dc0 msgr2=0x7f2c201975c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:38.626 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:38.625+0000 7f2c26600700 1 --2- 172.21.15.183:0/3722418283 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c20101dc0 0x7f2c201975c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f2c100056d0 tx=0x7f2c100057b0 comp rx=0 tx=0).stop 2024-03-28T15:52:38.626 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:38.626+0000 7f2c26600700 1 -- 172.21.15.183:0/3722418283 shutdown_connections 2024-03-28T15:52:38.626 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:38.626+0000 7f2c26600700 1 --2- 172.21.15.183:0/3722418283 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f2c04038fc0 0x7f2c0403b480 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:38.626 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:38.626+0000 7f2c26600700 1 --2- 172.21.15.183:0/3722418283 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c20101dc0 0x7f2c201975c0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:38.626 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:38.626+0000 7f2c26600700 1 -- 172.21.15.183:0/3722418283 >> 172.21.15.183:0/3722418283 conn(0x7f2c200fd670 msgr2=0x7f2c20103790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:38.627 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:38.626+0000 7f2c26600700 1 -- 172.21.15.183:0/3722418283 shutdown_connections 2024-03-28T15:52:38.627 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:38.626+0000 7f2c26600700 1 -- 172.21.15.183:0/3722418283 wait complete. 2024-03-28T15:52:38.628 INFO:teuthology.orchestra.run.smithi183.stderr:dumped monmap epoch 1 2024-03-28T15:52:38.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:38 smithi067 ceph-mon[31707]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:52:39.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:39 smithi067 ceph-mon[31707]: from='client.? 172.21.15.183:0/3722418283' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-28T15:52:40.771 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-28T15:52:40.772 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-28T15:52:40.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:40 smithi067 ceph-mon[31707]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:52:41.233 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T15:52:42.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:42 smithi067 ceph-mon[31707]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:52:43.715 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:43.699+0000 7f5a0e600700 1 -- 172.21.15.183:0/992761716 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5a08100a20 msgr2=0x7f5a08100e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:43.716 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:43.699+0000 7f5a0e600700 1 --2- 172.21.15.183:0/992761716 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5a08100a20 0x7f5a08100e00 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f59f8009910 tx=0x7f59f8009c20 comp rx=0 tx=0).stop 2024-03-28T15:52:43.716 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:43.699+0000 7f5a0e600700 1 -- 172.21.15.183:0/992761716 shutdown_connections 2024-03-28T15:52:43.716 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:43.699+0000 7f5a0e600700 1 --2- 172.21.15.183:0/992761716 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5a08100a20 0x7f5a08100e00 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:43.716 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:43.699+0000 7f5a0e600700 1 -- 172.21.15.183:0/992761716 >> 172.21.15.183:0/992761716 conn(0x7f5a080fc640 msgr2=0x7f5a080fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:43.716 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:43.699+0000 7f5a0e600700 1 -- 172.21.15.183:0/992761716 shutdown_connections 2024-03-28T15:52:43.716 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:43.699+0000 7f5a0e600700 1 -- 172.21.15.183:0/992761716 wait complete. 2024-03-28T15:52:43.716 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:43.700+0000 7f5a0e600700 1 Processor -- start 2024-03-28T15:52:43.716 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:43.701+0000 7f5a0e600700 1 -- start start 2024-03-28T15:52:43.716 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:43.701+0000 7f5a0e600700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5a08100a20 0x7f5a0819f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:43.716 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:43.701+0000 7f5a0e600700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a0810e390 con 0x7f5a08100a20 2024-03-28T15:52:43.716 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:43.702+0000 7f5a07400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5a08100a20 0x7f5a0819f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:43.716 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:43.702+0000 7f5a07400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5a08100a20 0x7f5a0819f640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.183:43776/0 (socket says 172.21.15.183:43776) 2024-03-28T15:52:43.716 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:43.702+0000 7f5a07400700 1 -- 172.21.15.183:0/1556970433 learned_addr learned my addr 172.21.15.183:0/1556970433 (peer_addr_for_me v2:172.21.15.183:0/0) 2024-03-28T15:52:43.716 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:43.702+0000 7f5a07400700 1 -- 172.21.15.183:0/1556970433 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a0819fb80 con 0x7f5a08100a20 2024-03-28T15:52:43.717 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:43.703+0000 7f5a07400700 1 --2- 172.21.15.183:0/1556970433 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5a08100a20 0x7f5a0819f640 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f59f80056d0 tx=0x7f59f80057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:43.717 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:43.703+0000 7f59ffe00700 1 -- 172.21.15.183:0/1556970433 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f59f8026070 con 0x7f5a08100a20 2024-03-28T15:52:43.717 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:43.703+0000 7f5a0e600700 1 -- 172.21.15.183:0/1556970433 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a0819fe10 con 0x7f5a08100a20 2024-03-28T15:52:43.717 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:43.703+0000 7f59ffe00700 1 -- 172.21.15.183:0/1556970433 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f59f800bc00 con 0x7f5a08100a20 2024-03-28T15:52:43.717 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:43.703+0000 7f59ffe00700 1 -- 172.21.15.183:0/1556970433 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f59f80039b0 con 0x7f5a08100a20 2024-03-28T15:52:43.717 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:43.704+0000 7f5a0e600700 1 -- 172.21.15.183:0/1556970433 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a081997c0 con 0x7f5a08100a20 2024-03-28T15:52:43.717 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:43.705+0000 7f5a0e600700 1 -- 172.21.15.183:0/1556970433 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a0806a340 con 0x7f5a08100a20 2024-03-28T15:52:43.717 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:43.705+0000 7f59ffe00700 1 -- 172.21.15.183:0/1556970433 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f59f8029070 con 0x7f5a08100a20 2024-03-28T15:52:43.717 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:43.706+0000 7f59ffe00700 1 --2- 172.21.15.183:0/1556970433 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f59ec038fc0 0x7f59ec03b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:43.717 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:43.706+0000 7f59ffe00700 1 -- 172.21.15.183:0/1556970433 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f59f801e070 con 0x7f5a08100a20 2024-03-28T15:52:43.717 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:43.706+0000 7f5a06a00700 1 --2- 172.21.15.183:0/1556970433 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f59ec038fc0 0x7f59ec03b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:43.717 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:43.707+0000 7f5a06a00700 1 --2- 172.21.15.183:0/1556970433 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f59ec038fc0 0x7f59ec03b480 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f59f4006fd0 tx=0x7f59f4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:43.721 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:43.722+0000 7f59ffe00700 1 -- 172.21.15.183:0/1556970433 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f59f8013070 con 0x7f5a08100a20 2024-03-28T15:52:43.759 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:43.759+0000 7f59ffe00700 1 -- 172.21.15.183:0/1556970433 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f59f802ada0 con 0x7f5a08100a20 2024-03-28T15:52:44.043 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:44.042+0000 7f5a0e600700 1 -- 172.21.15.183:0/1556970433 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5a080637c0 con 0x7f5a08100a20 2024-03-28T15:52:48.789 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:48.788+0000 7f59ffe00700 1 -- 172.21.15.183:0/1556970433 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1015 (secure 0 0 0) 0x7f59f8030090 con 0x7f5a08100a20 2024-03-28T15:52:48.790 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T15:52:48.790 INFO:teuthology.orchestra.run.smithi183.stdout:{"epoch":2,"fsid":"b5f8225a-ed1a-11ee-95d0-87774f69a715","modified":"2024-03-28T15:52:43.755433Z","created":"2024-03-28T15:50:17.125940Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:3300","nonce":0},{"type":"v1","addr":"172.21.15.183:6789","nonce":0}]},"addr":"172.21.15.183:6789/0","public_addr":"172.21.15.183:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-28T15:52:48.794 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:48.792+0000 7f5a0e600700 1 -- 172.21.15.183:0/1556970433 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f59ec038fc0 msgr2=0x7f59ec03b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:48.794 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:48.792+0000 7f5a0e600700 1 --2- 172.21.15.183:0/1556970433 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f59ec038fc0 0x7f59ec03b480 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f59f4006fd0 tx=0x7f59f4006e40 comp rx=0 tx=0).stop 2024-03-28T15:52:48.794 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:48.792+0000 7f5a0e600700 1 -- 172.21.15.183:0/1556970433 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5a08100a20 msgr2=0x7f5a0819f640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:48.794 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:48.792+0000 7f5a0e600700 1 --2- 172.21.15.183:0/1556970433 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5a08100a20 0x7f5a0819f640 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f59f80056d0 tx=0x7f59f80057b0 comp rx=0 tx=0).stop 2024-03-28T15:52:48.794 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:48.792+0000 7f5a0e600700 1 -- 172.21.15.183:0/1556970433 shutdown_connections 2024-03-28T15:52:48.794 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:48.792+0000 7f5a0e600700 1 --2- 172.21.15.183:0/1556970433 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f59ec038fc0 0x7f59ec03b480 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:48.794 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:48.792+0000 7f5a0e600700 1 --2- 172.21.15.183:0/1556970433 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5a08100a20 0x7f5a0819f640 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:48.794 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:48.792+0000 7f5a0e600700 1 -- 172.21.15.183:0/1556970433 >> 172.21.15.183:0/1556970433 conn(0x7f5a080fc640 msgr2=0x7f5a080fd6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:48.794 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:48.793+0000 7f5a0e600700 1 -- 172.21.15.183:0/1556970433 shutdown_connections 2024-03-28T15:52:48.795 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:48.793+0000 7f5a0e600700 1 -- 172.21.15.183:0/1556970433 wait complete. 2024-03-28T15:52:48.795 INFO:teuthology.orchestra.run.smithi183.stderr:dumped monmap epoch 2 2024-03-28T15:52:49.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:52:48 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:52:49.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:48 smithi067 ceph-mon[31707]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:52:49.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:48 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T15:52:49.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:48 smithi067 ceph-mon[31707]: mon.a calling monitor election 2024-03-28T15:52:49.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:48 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T15:52:49.321 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:48 smithi067 ceph-mon[31707]: from='client.? 172.21.15.183:0/1556970433' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-28T15:52:49.321 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:48 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T15:52:49.321 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:48 smithi067 ceph-mon[31707]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:52:49.321 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:48 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T15:52:49.321 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:48 smithi067 ceph-mon[31707]: mon.c calling monitor election 2024-03-28T15:52:49.321 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:48 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T15:52:49.321 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:48 smithi067 ceph-mon[31707]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:52:49.321 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:48 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T15:52:49.321 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:48 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T15:52:49.321 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:48 smithi067 ceph-mon[31707]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-28T15:52:49.321 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:48 smithi067 ceph-mon[31707]: monmap e2: 2 mons at {a=[v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0],c=[v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T15:52:49.322 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:48 smithi067 ceph-mon[31707]: fsmap 2024-03-28T15:52:49.322 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:48 smithi067 ceph-mon[31707]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T15:52:49.322 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:48 smithi067 ceph-mon[31707]: mgrmap e13: x(active, since 69s) 2024-03-28T15:52:49.322 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:48 smithi067 ceph-mon[31707]: overall HEALTH_OK 2024-03-28T15:52:49.322 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:48 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:52:49.322 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:48 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:52:50.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:49 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:52:50.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:49 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T15:52:50.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:49 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:52:50.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:49 smithi067 ceph-mon[31707]: Deploying daemon mon.b on smithi150 2024-03-28T15:52:50.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:49 smithi067 ceph-mon[31707]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:52:50.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:49 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T15:52:50.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:52:49 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:52:50.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:52:49 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T15:52:50.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:52:49 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:52:50.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:52:49 smithi183 ceph-mon[35580]: Deploying daemon mon.b on smithi150 2024-03-28T15:52:50.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:52:49 smithi183 ceph-mon[35580]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:52:50.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:52:49 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T15:52:50.528 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-28T15:52:50.529 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-28T15:52:50.974 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T15:52:51.069 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 15:52:50 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T15:52:50.756+0000 7f4fb0400700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-28T15:52:52.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:52:51 smithi183 ceph-mon[35580]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:52:52.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:51 smithi067 ceph-mon[31707]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:52:52.911 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:52.909+0000 7f4bfb200700 1 -- 172.21.15.183:0/1319268444 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bf4101ec0 msgr2=0x7f4bf41022a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:52.911 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:52.909+0000 7f4bfb200700 1 --2- 172.21.15.183:0/1319268444 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bf4101ec0 0x7f4bf41022a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4be4009910 tx=0x7f4be4009c20 comp rx=0 tx=0).stop 2024-03-28T15:52:52.911 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:52.909+0000 7f4bfb200700 1 -- 172.21.15.183:0/1319268444 shutdown_connections 2024-03-28T15:52:52.912 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:52.909+0000 7f4bfb200700 1 --2- 172.21.15.183:0/1319268444 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bf4101ec0 0x7f4bf41022a0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:52.912 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:52.910+0000 7f4bfb200700 1 -- 172.21.15.183:0/1319268444 >> 172.21.15.183:0/1319268444 conn(0x7f4bf40fd730 msgr2=0x7f4bf40ffb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:52.912 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:52.910+0000 7f4bfb200700 1 -- 172.21.15.183:0/1319268444 shutdown_connections 2024-03-28T15:52:52.912 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:52.910+0000 7f4bfb200700 1 -- 172.21.15.183:0/1319268444 wait complete. 2024-03-28T15:52:52.912 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:52.911+0000 7f4bfb200700 1 Processor -- start 2024-03-28T15:52:52.912 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:52.911+0000 7f4bfb200700 1 -- start start 2024-03-28T15:52:52.913 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:52.912+0000 7f4bfb200700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4bf4101ec0 0x7f4bf419f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:52.913 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:52.912+0000 7f4bfb200700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bf419fce0 0x7f4bf4199960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:52.913 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:52.912+0000 7f4bfb200700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bf410c140 con 0x7f4bf419fce0 2024-03-28T15:52:52.913 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:52.912+0000 7f4bfb200700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bf410bfc0 con 0x7f4bf4101ec0 2024-03-28T15:52:52.914 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:52.912+0000 7f4bf3e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4bf4101ec0 0x7f4bf419f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:52.914 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:52.912+0000 7f4bf3400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bf419fce0 0x7f4bf4199960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:52.914 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:52.913+0000 7f4bf3e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4bf4101ec0 0x7f4bf419f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.183:35212/0 (socket says 172.21.15.183:35212) 2024-03-28T15:52:52.914 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:52.913+0000 7f4bf3e00700 1 -- 172.21.15.183:0/1959221366 learned_addr learned my addr 172.21.15.183:0/1959221366 (peer_addr_for_me v2:172.21.15.183:0/0) 2024-03-28T15:52:52.914 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:52.913+0000 7f4bf3e00700 1 -- 172.21.15.183:0/1959221366 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4bf4101ec0 msgr2=0x7f4bf419f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-28T15:52:52.915 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:52.913+0000 7f4bf3e00700 1 -- 172.21.15.183:0/1959221366 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4bf4101ec0 msgr2=0x7f4bf419f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-28T15:52:52.915 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:52.913+0000 7f4bf3e00700 1 --2- 172.21.15.183:0/1959221366 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4bf4101ec0 0x7f4bf419f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-28T15:52:52.915 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:52.913+0000 7f4bf3e00700 1 --2- 172.21.15.183:0/1959221366 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4bf4101ec0 0x7f4bf419f7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T15:52:52.915 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:52.913+0000 7f4bf3400700 1 -- 172.21.15.183:0/1959221366 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4bf4101ec0 msgr2=0x7f4bf419f7a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T15:52:52.915 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:52.913+0000 7f4bf3400700 1 --2- 172.21.15.183:0/1959221366 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4bf4101ec0 0x7f4bf419f7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:52.915 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:52.913+0000 7f4bf3400700 1 -- 172.21.15.183:0/1959221366 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bf41a0320 con 0x7f4bf419fce0 2024-03-28T15:52:52.918 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:52.914+0000 7f4bf3400700 1 --2- 172.21.15.183:0/1959221366 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bf419fce0 0x7f4bf4199960 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4be000b4d0 tx=0x7f4be000b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:52.918 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:52.914+0000 7f4bf0c00700 1 -- 172.21.15.183:0/1959221366 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f4be0011630 con 0x7f4bf419fce0 2024-03-28T15:52:52.918 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:52.914+0000 7f4bfb200700 1 -- 172.21.15.183:0/1959221366 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bf4199ea0 con 0x7f4bf419fce0 2024-03-28T15:52:52.918 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:52.914+0000 7f4bf0c00700 1 -- 172.21.15.183:0/1959221366 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4be0011c80 con 0x7f4bf419fce0 2024-03-28T15:52:52.918 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:52.914+0000 7f4bf0c00700 1 -- 172.21.15.183:0/1959221366 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f4be001adc0 con 0x7f4bf419fce0 2024-03-28T15:52:52.918 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:52.914+0000 7f4bfb200700 1 -- 172.21.15.183:0/1959221366 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bf419a430 con 0x7f4bf419fce0 2024-03-28T15:52:52.919 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:52.915+0000 7f4bfb200700 1 -- 172.21.15.183:0/1959221366 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bf41033a0 con 0x7f4bf419fce0 2024-03-28T15:52:52.919 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:52.916+0000 7f4bf0c00700 1 -- 172.21.15.183:0/1959221366 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f4be0018020 con 0x7f4bf419fce0 2024-03-28T15:52:52.919 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:52.916+0000 7f4bf0c00700 1 --2- 172.21.15.183:0/1959221366 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f4bd8038db0 0x7f4bd803b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:52.919 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:52.916+0000 7f4bf0c00700 1 -- 172.21.15.183:0/1959221366 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f4be004bbf0 con 0x7f4bf419fce0 2024-03-28T15:52:52.920 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:52.917+0000 7f4bf3e00700 1 --2- 172.21.15.183:0/1959221366 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f4bd8038db0 0x7f4bd803b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:52.920 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:52.919+0000 7f4bf3e00700 1 --2- 172.21.15.183:0/1959221366 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f4bd8038db0 0x7f4bd803b270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4be4009750 tx=0x7f4be401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:52.933 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:52.932+0000 7f4bf0c00700 1 -- 172.21.15.183:0/1959221366 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4be001f020 con 0x7f4bf419fce0 2024-03-28T15:52:53.246 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:53.245+0000 7f4bfb200700 1 -- 172.21.15.183:0/1959221366 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4bf40637c0 con 0x7f4bf419fce0 2024-03-28T15:52:53.248 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:53.247+0000 7f4bf0c00700 1 -- 172.21.15.183:0/1959221366 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1015 (secure 0 0 0) 0x7f4be0021bc0 con 0x7f4bf419fce0 2024-03-28T15:52:53.248 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T15:52:53.248 INFO:teuthology.orchestra.run.smithi183.stdout:{"epoch":2,"fsid":"b5f8225a-ed1a-11ee-95d0-87774f69a715","modified":"2024-03-28T15:52:43.755433Z","created":"2024-03-28T15:50:17.125940Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:3300","nonce":0},{"type":"v1","addr":"172.21.15.183:6789","nonce":0}]},"addr":"172.21.15.183:6789/0","public_addr":"172.21.15.183:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-28T15:52:53.250 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:53.249+0000 7f4bfb200700 1 -- 172.21.15.183:0/1959221366 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f4bd8038db0 msgr2=0x7f4bd803b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:53.250 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:53.249+0000 7f4bfb200700 1 --2- 172.21.15.183:0/1959221366 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f4bd8038db0 0x7f4bd803b270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4be4009750 tx=0x7f4be401a040 comp rx=0 tx=0).stop 2024-03-28T15:52:53.250 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:53.249+0000 7f4bfb200700 1 -- 172.21.15.183:0/1959221366 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bf419fce0 msgr2=0x7f4bf4199960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:53.251 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:53.249+0000 7f4bfb200700 1 --2- 172.21.15.183:0/1959221366 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bf419fce0 0x7f4bf4199960 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4be000b4d0 tx=0x7f4be000b890 comp rx=0 tx=0).stop 2024-03-28T15:52:53.251 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:53.249+0000 7f4bfb200700 1 -- 172.21.15.183:0/1959221366 shutdown_connections 2024-03-28T15:52:53.251 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:53.249+0000 7f4bfb200700 1 --2- 172.21.15.183:0/1959221366 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f4bd8038db0 0x7f4bd803b270 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:53.251 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:53.249+0000 7f4bfb200700 1 --2- 172.21.15.183:0/1959221366 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4bf4101ec0 0x7f4bf419f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:53.251 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:53.249+0000 7f4bfb200700 1 --2- 172.21.15.183:0/1959221366 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bf419fce0 0x7f4bf4199960 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:53.251 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:53.249+0000 7f4bfb200700 1 -- 172.21.15.183:0/1959221366 >> 172.21.15.183:0/1959221366 conn(0x7f4bf40fd730 msgr2=0x7f4bf40743c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:53.252 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:53.249+0000 7f4bfb200700 1 -- 172.21.15.183:0/1959221366 shutdown_connections 2024-03-28T15:52:53.252 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:52:53.250+0000 7f4bfb200700 1 -- 172.21.15.183:0/1959221366 wait complete. 2024-03-28T15:52:53.252 INFO:teuthology.orchestra.run.smithi183.stderr:dumped monmap epoch 2 2024-03-28T15:52:54.251 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:53 smithi150 podman[34807]: 2024-03-28 15:52:53.978709361 +0000 UTC m=+0.132898213 container create df1c336724bc41d00a40bde07ba59648cd9fcf3f8d1dea8eef68a17a91296af5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mon-b, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-28T15:52:54.251 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 podman[34807]: 2024-03-28 15:52:54.248968623 +0000 UTC m=+0.403157545 container init df1c336724bc41d00a40bde07ba59648cd9fcf3f8d1dea8eef68a17a91296af5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mon-b, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-28T15:52:54.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:53 smithi067 ceph-mon[31707]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:52:54.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:53 smithi067 ceph-mon[31707]: from='client.? 172.21.15.183:0/1959221366' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-28T15:52:54.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:52:53 smithi183 ceph-mon[35580]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:52:54.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:52:53 smithi183 ceph-mon[35580]: from='client.? 172.21.15.183:0/1959221366' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-28T15:52:54.554 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 podman[34807]: 2024-03-28 15:52:54.259160396 +0000 UTC m=+0.413349222 container start df1c336724bc41d00a40bde07ba59648cd9fcf3f8d1dea8eef68a17a91296af5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mon-b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-28T15:52:54.555 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: set uid:gid to 167:167 (ceph:ceph) 2024-03-28T15:52:54.555 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev), process ceph-mon, pid 2 2024-03-28T15:52:54.555 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: pidfile_write: ignore empty --pid-file 2024-03-28T15:52:54.555 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: load: jerasure load: lrc 2024-03-28T15:52:54.555 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: RocksDB version: 7.9.2 2024-03-28T15:52:54.555 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Git sha 0 2024-03-28T15:52:54.555 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Compile date 2024-03-22 04:58:55 2024-03-28T15:52:54.555 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: DB SUMMARY 2024-03-28T15:52:54.555 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: DB Session ID: 66A6MX4TYPG3GQF7YV3O 2024-03-28T15:52:54.556 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: CURRENT file: CURRENT 2024-03-28T15:52:54.556 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: IDENTITY file: IDENTITY 2024-03-28T15:52:54.556 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-28T15:52:54.556 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-03-28T15:52:54.556 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-03-28T15:52:54.556 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.error_if_exists: 0 2024-03-28T15:52:54.556 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.create_if_missing: 0 2024-03-28T15:52:54.556 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.paranoid_checks: 1 2024-03-28T15:52:54.556 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-28T15:52:54.556 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-28T15:52:54.557 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-28T15:52:54.557 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.env: 0x55f796f74b20 2024-03-28T15:52:54.557 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.fs: PosixFileSystem 2024-03-28T15:52:54.557 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.info_log: 0x55f797a81960 2024-03-28T15:52:54.557 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.max_file_opening_threads: 16 2024-03-28T15:52:54.557 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.statistics: (nil) 2024-03-28T15:52:54.557 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.use_fsync: 0 2024-03-28T15:52:54.558 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.max_log_file_size: 0 2024-03-28T15:52:54.558 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-28T15:52:54.558 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-28T15:52:54.558 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.keep_log_file_num: 1000 2024-03-28T15:52:54.558 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.recycle_log_file_num: 0 2024-03-28T15:52:54.558 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.allow_fallocate: 1 2024-03-28T15:52:54.559 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.allow_mmap_reads: 0 2024-03-28T15:52:54.560 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.allow_mmap_writes: 0 2024-03-28T15:52:54.560 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.use_direct_reads: 0 2024-03-28T15:52:54.560 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-28T15:52:54.560 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.create_missing_column_families: 0 2024-03-28T15:52:54.560 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.db_log_dir: 2024-03-28T15:52:54.560 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.wal_dir: 2024-03-28T15:52:54.560 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-28T15:52:54.560 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-28T15:52:54.560 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-28T15:52:54.561 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-28T15:52:54.561 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-28T15:52:54.561 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-28T15:52:54.561 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.advise_random_on_open: 1 2024-03-28T15:52:54.561 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.db_write_buffer_size: 0 2024-03-28T15:52:54.561 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.write_buffer_manager: 0x55f797a541e0 2024-03-28T15:52:54.561 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-28T15:52:54.561 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-28T15:52:54.561 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-28T15:52:54.561 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.rate_limiter: (nil) 2024-03-28T15:52:54.561 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-28T15:52:54.561 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.wal_recovery_mode: 2 2024-03-28T15:52:54.561 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.enable_thread_tracking: 0 2024-03-28T15:52:54.562 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.enable_pipelined_write: 0 2024-03-28T15:52:54.562 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.unordered_write: 0 2024-03-28T15:52:54.562 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-28T15:52:54.562 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-28T15:52:54.562 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-28T15:52:54.562 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-28T15:52:54.562 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.row_cache: None 2024-03-28T15:52:54.562 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.wal_filter: None 2024-03-28T15:52:54.562 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-28T15:52:54.562 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.allow_ingest_behind: 0 2024-03-28T15:52:54.562 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.two_write_queues: 0 2024-03-28T15:52:54.562 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.manual_wal_flush: 0 2024-03-28T15:52:54.563 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.wal_compression: 0 2024-03-28T15:52:54.563 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.atomic_flush: 0 2024-03-28T15:52:54.563 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-28T15:52:54.563 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-28T15:52:54.563 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-28T15:52:54.563 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.log_readahead_size: 0 2024-03-28T15:52:54.563 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-28T15:52:54.563 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.best_efforts_recovery: 0 2024-03-28T15:52:54.563 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-28T15:52:54.563 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-28T15:52:54.563 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.allow_data_in_errors: 0 2024-03-28T15:52:54.563 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.db_host_id: __hostname__ 2024-03-28T15:52:54.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-28T15:52:54.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.max_background_jobs: 2 2024-03-28T15:52:54.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.max_background_compactions: -1 2024-03-28T15:52:54.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.max_subcompactions: 1 2024-03-28T15:52:54.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-28T15:52:54.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-28T15:52:54.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-28T15:52:54.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.max_total_wal_size: 0 2024-03-28T15:52:54.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-28T15:52:54.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-28T15:52:54.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-28T15:52:54.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-28T15:52:54.564 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.max_open_files: -1 2024-03-28T15:52:54.565 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.bytes_per_sync: 0 2024-03-28T15:52:54.565 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-28T15:52:54.565 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-28T15:52:54.565 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.compaction_readahead_size: 0 2024-03-28T15:52:54.565 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.max_background_flushes: -1 2024-03-28T15:52:54.565 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Compression algorithms supported: 2024-03-28T15:52:54.565 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-28T15:52:54.565 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: kZSTD supported: 0 2024-03-28T15:52:54.566 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: kXpressCompression supported: 0 2024-03-28T15:52:54.566 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: kLZ4HCCompression supported: 1 2024-03-28T15:52:54.566 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: kZlibCompression supported: 1 2024-03-28T15:52:54.566 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: kSnappyCompression supported: 1 2024-03-28T15:52:54.566 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: kLZ4Compression supported: 1 2024-03-28T15:52:54.566 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: kBZip2Compression supported: 0 2024-03-28T15:52:54.566 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-28T15:52:54.566 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-28T15:52:54.566 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-03-28T15:52:54.566 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-28T15:52:54.566 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-28T15:52:54.567 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.merge_operator: 2024-03-28T15:52:54.567 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.compaction_filter: None 2024-03-28T15:52:54.567 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.compaction_filter_factory: None 2024-03-28T15:52:54.567 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.sst_partitioner_factory: None 2024-03-28T15:52:54.567 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-28T15:52:54.567 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-28T15:52:54.567 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55f797a81a60) 2024-03-28T15:52:54.567 INFO:journalctl@ceph.mon.b.smithi150.stdout: cache_index_and_filter_blocks: 1 2024-03-28T15:52:54.567 INFO:journalctl@ceph.mon.b.smithi150.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-28T15:52:54.567 INFO:journalctl@ceph.mon.b.smithi150.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-28T15:52:54.568 INFO:journalctl@ceph.mon.b.smithi150.stdout: pin_top_level_index_and_filter: 1 2024-03-28T15:52:54.568 INFO:journalctl@ceph.mon.b.smithi150.stdout: index_type: 0 2024-03-28T15:52:54.568 INFO:journalctl@ceph.mon.b.smithi150.stdout: data_block_index_type: 0 2024-03-28T15:52:54.568 INFO:journalctl@ceph.mon.b.smithi150.stdout: index_shortening: 1 2024-03-28T15:52:54.568 INFO:journalctl@ceph.mon.b.smithi150.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-28T15:52:54.568 INFO:journalctl@ceph.mon.b.smithi150.stdout: checksum: 4 2024-03-28T15:52:54.568 INFO:journalctl@ceph.mon.b.smithi150.stdout: no_block_cache: 0 2024-03-28T15:52:54.568 INFO:journalctl@ceph.mon.b.smithi150.stdout: block_cache: 0x55f797a87090 2024-03-28T15:52:54.568 INFO:journalctl@ceph.mon.b.smithi150.stdout: block_cache_name: BinnedLRUCache 2024-03-28T15:52:54.568 INFO:journalctl@ceph.mon.b.smithi150.stdout: block_cache_options: 2024-03-28T15:52:54.568 INFO:journalctl@ceph.mon.b.smithi150.stdout: capacity : 536870912 2024-03-28T15:52:54.568 INFO:journalctl@ceph.mon.b.smithi150.stdout: num_shard_bits : 4 2024-03-28T15:52:54.569 INFO:journalctl@ceph.mon.b.smithi150.stdout: strict_capacity_limit : 0 2024-03-28T15:52:54.569 INFO:journalctl@ceph.mon.b.smithi150.stdout: high_pri_pool_ratio: 0.000 2024-03-28T15:52:54.569 INFO:journalctl@ceph.mon.b.smithi150.stdout: block_cache_compressed: (nil) 2024-03-28T15:52:54.569 INFO:journalctl@ceph.mon.b.smithi150.stdout: persistent_cache: (nil) 2024-03-28T15:52:54.569 INFO:journalctl@ceph.mon.b.smithi150.stdout: block_size: 4096 2024-03-28T15:52:54.569 INFO:journalctl@ceph.mon.b.smithi150.stdout: block_size_deviation: 10 2024-03-28T15:52:54.569 INFO:journalctl@ceph.mon.b.smithi150.stdout: block_restart_interval: 16 2024-03-28T15:52:54.569 INFO:journalctl@ceph.mon.b.smithi150.stdout: index_block_restart_interval: 1 2024-03-28T15:52:54.569 INFO:journalctl@ceph.mon.b.smithi150.stdout: metadata_block_size: 4096 2024-03-28T15:52:54.569 INFO:journalctl@ceph.mon.b.smithi150.stdout: partition_filters: 0 2024-03-28T15:52:54.569 INFO:journalctl@ceph.mon.b.smithi150.stdout: use_delta_encoding: 1 2024-03-28T15:52:54.570 INFO:journalctl@ceph.mon.b.smithi150.stdout: filter_policy: bloomfilter 2024-03-28T15:52:54.570 INFO:journalctl@ceph.mon.b.smithi150.stdout: whole_key_filtering: 1 2024-03-28T15:52:54.570 INFO:journalctl@ceph.mon.b.smithi150.stdout: verify_compression: 0 2024-03-28T15:52:54.570 INFO:journalctl@ceph.mon.b.smithi150.stdout: read_amp_bytes_per_bit: 0 2024-03-28T15:52:54.570 INFO:journalctl@ceph.mon.b.smithi150.stdout: format_version: 5 2024-03-28T15:52:54.570 INFO:journalctl@ceph.mon.b.smithi150.stdout: enable_index_compression: 1 2024-03-28T15:52:54.570 INFO:journalctl@ceph.mon.b.smithi150.stdout: block_align: 0 2024-03-28T15:52:54.570 INFO:journalctl@ceph.mon.b.smithi150.stdout: max_auto_readahead_size: 262144 2024-03-28T15:52:54.570 INFO:journalctl@ceph.mon.b.smithi150.stdout: prepopulate_block_cache: 0 2024-03-28T15:52:54.570 INFO:journalctl@ceph.mon.b.smithi150.stdout: initial_auto_readahead_size: 8192 2024-03-28T15:52:54.571 INFO:journalctl@ceph.mon.b.smithi150.stdout: num_file_reads_for_auto_readahead: 2 2024-03-28T15:52:54.571 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.write_buffer_size: 33554432 2024-03-28T15:52:54.571 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.max_write_buffer_number: 2 2024-03-28T15:52:54.571 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.compression: NoCompression 2024-03-28T15:52:54.571 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.bottommost_compression: Disabled 2024-03-28T15:52:54.571 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.prefix_extractor: nullptr 2024-03-28T15:52:54.571 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-28T15:52:54.571 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.num_levels: 7 2024-03-28T15:52:54.571 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-28T15:52:54.571 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-28T15:52:54.571 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-28T15:52:54.571 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-28T15:52:54.571 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-28T15:52:54.571 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-28T15:52:54.571 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-28T15:52:54.572 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-28T15:52:54.572 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-28T15:52:54.572 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-28T15:52:54.572 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-28T15:52:54.572 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-28T15:52:54.572 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-28T15:52:54.572 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.compression_opts.level: 32767 2024-03-28T15:52:54.572 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.compression_opts.strategy: 0 2024-03-28T15:52:54.572 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-28T15:52:54.572 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-28T15:52:54.572 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-28T15:52:54.572 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-28T15:52:54.572 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.compression_opts.enabled: false 2024-03-28T15:52:54.572 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-28T15:52:54.573 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-28T15:52:54.573 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-28T15:52:54.573 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-28T15:52:54.573 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.target_file_size_base: 67108864 2024-03-28T15:52:54.573 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-28T15:52:54.573 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-28T15:52:54.573 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-28T15:52:54.573 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-28T15:52:54.573 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-28T15:52:54.573 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-28T15:52:54.573 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-28T15:52:54.573 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-28T15:52:54.573 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-28T15:52:54.573 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-28T15:52:54.574 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-28T15:52:54.574 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-28T15:52:54.574 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-28T15:52:54.574 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-28T15:52:54.575 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.arena_block_size: 1048576 2024-03-28T15:52:54.575 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-28T15:52:54.575 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-28T15:52:54.575 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.disable_auto_compactions: 0 2024-03-28T15:52:54.575 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-28T15:52:54.575 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-28T15:52:54.575 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-28T15:52:54.575 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-28T15:52:54.576 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-28T15:52:54.576 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-28T15:52:54.576 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-28T15:52:54.576 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-28T15:52:54.576 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-28T15:52:54.576 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-28T15:52:54.576 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.table_properties_collectors: 2024-03-28T15:52:54.576 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.inplace_update_support: 0 2024-03-28T15:52:54.576 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-28T15:52:54.576 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-28T15:52:54.576 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-28T15:52:54.576 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-28T15:52:54.576 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.bloom_locality: 0 2024-03-28T15:52:54.577 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.max_successive_merges: 0 2024-03-28T15:52:54.577 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-28T15:52:54.577 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.paranoid_file_checks: 0 2024-03-28T15:52:54.577 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.force_consistency_checks: 1 2024-03-28T15:52:54.577 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.report_bg_io_stats: 0 2024-03-28T15:52:54.577 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.ttl: 2592000 2024-03-28T15:52:54.577 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-28T15:52:54.577 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-28T15:52:54.577 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-28T15:52:54.577 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.enable_blob_files: false 2024-03-28T15:52:54.577 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.min_blob_size: 0 2024-03-28T15:52:54.577 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.blob_file_size: 268435456 2024-03-28T15:52:54.577 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-28T15:52:54.577 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-28T15:52:54.577 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-28T15:52:54.578 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-28T15:52:54.578 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-28T15:52:54.578 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.blob_file_starting_level: 0 2024-03-28T15:52:54.578 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-28T15:52:54.578 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-28T15:52:54.578 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-28T15:52:54.578 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 7a9407ed-0abf-4eb2-afdb-ab63f2883e04 2024-03-28T15:52:54.578 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711641174317124, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-28T15:52:54.578 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-28T15:52:54.579 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711641174318071, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711641174, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "7a9407ed-0abf-4eb2-afdb-ab63f2883e04", "db_session_id": "66A6MX4TYPG3GQF7YV3O", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-28T15:52:54.579 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711641174318310, "job": 1, "event": "recovery_finished"} 2024-03-28T15:52:54.579 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-28T15:52:54.579 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-28T15:52:54.579 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55f797b7a000 2024-03-28T15:52:54.579 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: DB pointer 0x55f797b64000 2024-03-28T15:52:54.579 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-28T15:52:54.579 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-28T15:52:54.579 INFO:journalctl@ceph.mon.b.smithi150.stdout: ** DB Stats ** 2024-03-28T15:52:54.579 INFO:journalctl@ceph.mon.b.smithi150.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-28T15:52:54.579 INFO:journalctl@ceph.mon.b.smithi150.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-28T15:52:54.579 INFO:journalctl@ceph.mon.b.smithi150.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-28T15:52:54.579 INFO:journalctl@ceph.mon.b.smithi150.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-28T15:52:54.579 INFO:journalctl@ceph.mon.b.smithi150.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-28T15:52:54.580 INFO:journalctl@ceph.mon.b.smithi150.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-28T15:52:54.580 INFO:journalctl@ceph.mon.b.smithi150.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-28T15:52:54.580 INFO:journalctl@ceph.mon.b.smithi150.stdout: 2024-03-28T15:52:54.580 INFO:journalctl@ceph.mon.b.smithi150.stdout: ** Compaction Stats [default] ** 2024-03-28T15:52:54.580 INFO:journalctl@ceph.mon.b.smithi150.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-28T15:52:54.580 INFO:journalctl@ceph.mon.b.smithi150.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-28T15:52:54.580 INFO:journalctl@ceph.mon.b.smithi150.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-28T15:52:54.580 INFO:journalctl@ceph.mon.b.smithi150.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-28T15:52:54.580 INFO:journalctl@ceph.mon.b.smithi150.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-28T15:52:54.580 INFO:journalctl@ceph.mon.b.smithi150.stdout: 2024-03-28T15:52:54.580 INFO:journalctl@ceph.mon.b.smithi150.stdout: ** Compaction Stats [default] ** 2024-03-28T15:52:54.580 INFO:journalctl@ceph.mon.b.smithi150.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-28T15:52:54.580 INFO:journalctl@ceph.mon.b.smithi150.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-28T15:52:54.581 INFO:journalctl@ceph.mon.b.smithi150.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-28T15:52:54.581 INFO:journalctl@ceph.mon.b.smithi150.stdout: 2024-03-28T15:52:54.581 INFO:journalctl@ceph.mon.b.smithi150.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-28T15:52:54.581 INFO:journalctl@ceph.mon.b.smithi150.stdout: 2024-03-28T15:52:54.581 INFO:journalctl@ceph.mon.b.smithi150.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-28T15:52:54.581 INFO:journalctl@ceph.mon.b.smithi150.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-28T15:52:54.581 INFO:journalctl@ceph.mon.b.smithi150.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-28T15:52:54.581 INFO:journalctl@ceph.mon.b.smithi150.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-28T15:52:54.581 INFO:journalctl@ceph.mon.b.smithi150.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-28T15:52:54.581 INFO:journalctl@ceph.mon.b.smithi150.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-28T15:52:54.582 INFO:journalctl@ceph.mon.b.smithi150.stdout: Cumulative compaction: 0.00 GB write, 0.30 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-28T15:52:54.582 INFO:journalctl@ceph.mon.b.smithi150.stdout: Interval compaction: 0.00 GB write, 0.30 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-28T15:52:54.582 INFO:journalctl@ceph.mon.b.smithi150.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-28T15:52:54.582 INFO:journalctl@ceph.mon.b.smithi150.stdout: Block cache BinnedLRUCache@0x55f797a87090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 2.4e-05 secs_since: 0 2024-03-28T15:52:54.582 INFO:journalctl@ceph.mon.b.smithi150.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-28T15:52:54.582 INFO:journalctl@ceph.mon.b.smithi150.stdout: 2024-03-28T15:52:54.582 INFO:journalctl@ceph.mon.b.smithi150.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-28T15:52:54.582 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-03-28T15:52:54.582 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: using public_addr v2:172.21.15.150:0/0 -> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-03-28T15:52:54.582 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: starting mon.b rank -1 at public addrs [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] at bind addrs [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 2024-03-28T15:52:54.582 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: mon.b@-1(???) e0 preinit fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 2024-03-28T15:52:54.582 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 bash[34807]: df1c336724bc41d00a40bde07ba59648cd9fcf3f8d1dea8eef68a17a91296af5 2024-03-28T15:52:54.582 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 systemd[1]: Started Ceph mon.b for b5f8225a-ed1a-11ee-95d0-87774f69a715. 2024-03-28T15:52:54.582 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: mon.b@-1(synchronizing).mds e1 new map 2024-03-28T15:52:54.583 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: mon.b@-1(synchronizing).mds e1 print_map 2024-03-28T15:52:54.583 INFO:journalctl@ceph.mon.b.smithi150.stdout: e1 2024-03-28T15:52:54.583 INFO:journalctl@ceph.mon.b.smithi150.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-28T15:52:54.583 INFO:journalctl@ceph.mon.b.smithi150.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-28T15:52:54.583 INFO:journalctl@ceph.mon.b.smithi150.stdout: legacy client fscid: -1 2024-03-28T15:52:54.583 INFO:journalctl@ceph.mon.b.smithi150.stdout: 2024-03-28T15:52:54.583 INFO:journalctl@ceph.mon.b.smithi150.stdout: No filesystems configured 2024-03-28T15:52:54.583 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-28T15:52:54.583 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-28T15:52:54.583 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-28T15:52:54.584 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-28T15:52:54.584 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-28T15:52:54.584 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-28T15:52:54.584 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-28T15:52:54.584 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-28T15:52:54.584 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-28T15:52:54.584 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-28T15:52:54.584 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: Deploying cephadm binary to smithi150 2024-03-28T15:52:54.584 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:52:54.584 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:52:54.584 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:52:54.584 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:52:54.584 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: Added host smithi150 2024-03-28T15:52:54.584 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:52:54.584 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:52:54.585 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:52:54.585 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:52:54.585 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T15:52:54.585 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:52:54.585 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:52:54.585 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:52:54.585 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi183", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:52:54.585 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:52:54.585 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: Deploying cephadm binary to smithi183 2024-03-28T15:52:54.585 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:52:54.586 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:52:54.586 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:52:54.586 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:52:54.586 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi150", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:52:54.586 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:52:54.586 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:52:54.586 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: Updating smithi150:/etc/ceph/ceph.conf 2024-03-28T15:52:54.586 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: Updating smithi150:/var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T15:52:54.586 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:52:54.586 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:52:54.586 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: Updating smithi150:/etc/ceph/ceph.client.admin.keyring 2024-03-28T15:52:54.586 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:52:54.587 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:52:54.587 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: Updating smithi150:/var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T15:52:54.587 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: Added host smithi183 2024-03-28T15:52:54.587 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:52:54.587 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:52:54.587 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:52:54.587 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:52:54.587 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:52:54.587 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:52:54.587 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:52:54.587 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T15:52:54.588 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:52:54.588 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:52:54.588 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:52:54.588 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/3373292310' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-28T15:52:54.588 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/3373292310' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-28T15:52:54.588 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T15:52:54.588 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:52:54.588 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:52:54.589 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:52:54.589 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:52:54.589 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:52:54.589 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:52:54.589 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi183", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:52:54.589 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:52:54.589 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:52:54.589 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: Updating smithi183:/etc/ceph/ceph.conf 2024-03-28T15:52:54.590 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:52:54.590 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: Updating smithi183:/var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T15:52:54.590 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi067:172.21.15.67=a;smithi150:172.21.15.150=b;smithi183:172.21.15.183=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:52:54.590 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: Saving service mon spec with placement smithi067:172.21.15.67=a;smithi150:172.21.15.150=b;smithi183:172.21.15.183=c;count:3 2024-03-28T15:52:54.590 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:52:54.590 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: Updating smithi183:/etc/ceph/ceph.client.admin.keyring 2024-03-28T15:52:54.590 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: Updating smithi183:/var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T15:52:54.591 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:52:54.591 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:52:54.591 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:52:54.591 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T15:52:54.591 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:52:54.591 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: Deploying daemon mon.c on smithi183 2024-03-28T15:52:54.591 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:52:54.591 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: from='client.? 172.21.15.183:0/3722418283' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-28T15:52:54.591 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:52:54.592 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:52:54.592 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:52:54.592 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T15:52:54.592 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: mon.a calling monitor election 2024-03-28T15:52:54.592 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T15:52:54.592 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: from='client.? 172.21.15.183:0/1556970433' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-28T15:52:54.592 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T15:52:54.592 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:52:54.592 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T15:52:54.592 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: mon.c calling monitor election 2024-03-28T15:52:54.593 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T15:52:54.593 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:52:54.593 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T15:52:54.593 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T15:52:54.593 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-28T15:52:54.593 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: monmap e2: 2 mons at {a=[v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0],c=[v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T15:52:54.593 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: fsmap 2024-03-28T15:52:54.593 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T15:52:54.593 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: mgrmap e13: x(active, since 69s) 2024-03-28T15:52:54.593 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: overall HEALTH_OK 2024-03-28T15:52:54.593 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:52:54.593 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:52:54.593 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:52:54.593 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T15:52:54.594 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:52:54.594 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: Deploying daemon mon.b on smithi150 2024-03-28T15:52:54.594 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:52:54.594 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T15:52:54.594 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:52:54.594 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:52:54.594 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: from='client.? 172.21.15.183:0/1959221366' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-28T15:52:54.594 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:54 smithi150 ceph-mon[34825]: mon.b@-1(synchronizing).paxosservice(auth 1..4) refresh upgraded, format 0 -> 3 2024-03-28T15:52:54.966 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-28T15:52:54.966 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-28T15:52:55.379 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T15:52:59.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:59 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T15:52:59.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:59 smithi150 ceph-mon[34825]: mon.a calling monitor election 2024-03-28T15:52:59.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:59 smithi150 ceph-mon[34825]: mon.c calling monitor election 2024-03-28T15:52:59.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:59 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T15:52:59.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:59 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T15:52:59.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:59 smithi150 ceph-mon[34825]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:52:59.804 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:59 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T15:52:59.804 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:59 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T15:52:59.804 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:59 smithi150 ceph-mon[34825]: mon.b calling monitor election 2024-03-28T15:52:59.804 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:59 smithi150 ceph-mon[34825]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:52:59.804 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:59 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T15:52:59.804 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:59 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T15:52:59.804 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:59 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T15:52:59.804 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:59 smithi150 ceph-mon[34825]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-28T15:52:59.804 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:59 smithi150 ceph-mon[34825]: monmap e3: 3 mons at {a=[v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0],b=[v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0],c=[v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T15:52:59.804 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:59 smithi150 ceph-mon[34825]: fsmap 2024-03-28T15:52:59.804 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:59 smithi150 ceph-mon[34825]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T15:52:59.804 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:59 smithi150 ceph-mon[34825]: mgrmap e13: x(active, since 80s) 2024-03-28T15:52:59.804 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:59 smithi150 ceph-mon[34825]: overall HEALTH_OK 2024-03-28T15:52:59.804 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:59 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:52:59.805 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:52:59 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:52:59.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:59 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T15:52:59.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:59 smithi067 ceph-mon[31707]: mon.a calling monitor election 2024-03-28T15:52:59.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:59 smithi067 ceph-mon[31707]: mon.c calling monitor election 2024-03-28T15:52:59.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:59 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T15:52:59.821 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:59 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T15:52:59.821 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:59 smithi067 ceph-mon[31707]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:52:59.821 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:59 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T15:52:59.821 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:59 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T15:52:59.821 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:59 smithi067 ceph-mon[31707]: mon.b calling monitor election 2024-03-28T15:52:59.821 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:59 smithi067 ceph-mon[31707]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:52:59.821 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:59 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T15:52:59.821 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:59 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T15:52:59.821 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:59 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T15:52:59.821 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:59 smithi067 ceph-mon[31707]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-28T15:52:59.821 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:59 smithi067 ceph-mon[31707]: monmap e3: 3 mons at {a=[v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0],b=[v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0],c=[v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T15:52:59.821 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:59 smithi067 ceph-mon[31707]: fsmap 2024-03-28T15:52:59.821 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:59 smithi067 ceph-mon[31707]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T15:52:59.821 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:59 smithi067 ceph-mon[31707]: mgrmap e13: x(active, since 80s) 2024-03-28T15:52:59.822 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:59 smithi067 ceph-mon[31707]: overall HEALTH_OK 2024-03-28T15:52:59.822 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:59 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:52:59.822 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:52:59 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:52:59.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:52:59 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T15:52:59.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:52:59 smithi183 ceph-mon[35580]: mon.a calling monitor election 2024-03-28T15:52:59.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:52:59 smithi183 ceph-mon[35580]: mon.c calling monitor election 2024-03-28T15:52:59.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:52:59 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T15:52:59.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:52:59 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T15:52:59.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:52:59 smithi183 ceph-mon[35580]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:52:59.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:52:59 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T15:52:59.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:52:59 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T15:52:59.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:52:59 smithi183 ceph-mon[35580]: mon.b calling monitor election 2024-03-28T15:52:59.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:52:59 smithi183 ceph-mon[35580]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:52:59.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:52:59 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T15:52:59.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:52:59 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T15:52:59.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:52:59 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T15:52:59.852 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:52:59 smithi183 ceph-mon[35580]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-28T15:52:59.852 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:52:59 smithi183 ceph-mon[35580]: monmap e3: 3 mons at {a=[v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0],b=[v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0],c=[v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T15:52:59.852 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:52:59 smithi183 ceph-mon[35580]: fsmap 2024-03-28T15:52:59.852 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:52:59 smithi183 ceph-mon[35580]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T15:52:59.852 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:52:59 smithi183 ceph-mon[35580]: mgrmap e13: x(active, since 80s) 2024-03-28T15:52:59.852 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:52:59 smithi183 ceph-mon[35580]: overall HEALTH_OK 2024-03-28T15:52:59.852 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:52:59 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:52:59.852 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:52:59 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:00.290 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:00.287+0000 7f9bfd200700 1 -- 172.21.15.183:0/2728500487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9bf8101eb0 msgr2=0x7f9bdc005610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:53:00.290 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:00.287+0000 7f9bfd200700 1 --2- 172.21.15.183:0/2728500487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9bf8101eb0 0x7f9bdc005610 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9be8005fa0 tx=0x7f9be800ffc0 comp rx=0 tx=0).stop 2024-03-28T15:53:00.290 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:00.288+0000 7f9bfd200700 1 -- 172.21.15.183:0/2728500487 shutdown_connections 2024-03-28T15:53:00.290 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:00.288+0000 7f9bfd200700 1 --2- 172.21.15.183:0/2728500487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9bf8101eb0 0x7f9bdc005610 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:00.290 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:00.288+0000 7f9bfd200700 1 -- 172.21.15.183:0/2728500487 >> 172.21.15.183:0/2728500487 conn(0x7f9bf80fd720 msgr2=0x7f9bf80ffb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:53:00.290 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:00.288+0000 7f9bfd200700 1 -- 172.21.15.183:0/2728500487 shutdown_connections 2024-03-28T15:53:00.290 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:00.288+0000 7f9bfd200700 1 -- 172.21.15.183:0/2728500487 wait complete. 2024-03-28T15:53:00.292 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:00.290+0000 7f9bfd200700 1 Processor -- start 2024-03-28T15:53:00.292 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:00.290+0000 7f9bfd200700 1 -- start start 2024-03-28T15:53:00.292 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:00.291+0000 7f9bfd200700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9bf8101eb0 0x7f9bf8074490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:00.292 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:00.291+0000 7f9bfd200700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9bf80749d0 0x7f9bf8074e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:00.292 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:00.291+0000 7f9bfd200700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9bf81a3930 0x7f9bf81a5d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:00.293 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:00.291+0000 7f9bfd200700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf8109b40 con 0x7f9bf80749d0 2024-03-28T15:53:00.293 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:00.291+0000 7f9bfd200700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf81099c0 con 0x7f9bf8101eb0 2024-03-28T15:53:00.293 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:00.291+0000 7f9bfd200700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf8109cc0 con 0x7f9bf81a3930 2024-03-28T15:53:00.294 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:00.291+0000 7f9bf5400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9bf80749d0 0x7f9bf8074e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:00.294 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:00.291+0000 7f9bf5400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9bf80749d0 0x7f9bf8074e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.183:56900/0 (socket says 172.21.15.183:56900) 2024-03-28T15:53:00.294 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:00.291+0000 7f9bf5400700 1 -- 172.21.15.183:0/1577246667 learned_addr learned my addr 172.21.15.183:0/1577246667 (peer_addr_for_me v2:172.21.15.183:0/0) 2024-03-28T15:53:00.294 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:00.292+0000 7f9bf5e00700 1 --2- 172.21.15.183:0/1577246667 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9bf8101eb0 0x7f9bf8074490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:00.294 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:00.292+0000 7f9bf6800700 1 --2- 172.21.15.183:0/1577246667 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9bf81a3930 0x7f9bf81a5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:00.294 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:00.292+0000 7f9bf5e00700 1 -- 172.21.15.183:0/1577246667 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9bf8101eb0 msgr2=0x7f9bf8074490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-28T15:53:00.294 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:00.292+0000 7f9bf5e00700 1 -- 172.21.15.183:0/1577246667 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9bf8101eb0 msgr2=0x7f9bf8074490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-28T15:53:00.294 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:00.292+0000 7f9bf5e00700 1 --2- 172.21.15.183:0/1577246667 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9bf8101eb0 0x7f9bf8074490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-28T15:53:00.294 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:00.292+0000 7f9bf5e00700 1 --2- 172.21.15.183:0/1577246667 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9bf8101eb0 0x7f9bf8074490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T15:53:00.294 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:00.292+0000 7f9bf5400700 1 -- 172.21.15.183:0/1577246667 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9bf81a3930 msgr2=0x7f9bf81a5d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:53:00.295 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:00.292+0000 7f9bf5400700 1 --2- 172.21.15.183:0/1577246667 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9bf81a3930 0x7f9bf81a5d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:00.295 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:00.292+0000 7f9bf5400700 1 -- 172.21.15.183:0/1577246667 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9bf8101eb0 msgr2=0x7f9bf8074490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T15:53:00.295 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:00.292+0000 7f9bf5400700 1 --2- 172.21.15.183:0/1577246667 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9bf8101eb0 0x7f9bf8074490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:00.295 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:00.292+0000 7f9bf5400700 1 -- 172.21.15.183:0/1577246667 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf81a6450 con 0x7f9bf80749d0 2024-03-28T15:53:00.295 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:00.293+0000 7f9bf5400700 1 --2- 172.21.15.183:0/1577246667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9bf80749d0 0x7f9bf8074e50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9be400b780 tx=0x7f9be400ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:53:00.295 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:00.293+0000 7f9beea00700 1 -- 172.21.15.183:0/1577246667 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be4004230 con 0x7f9bf80749d0 2024-03-28T15:53:00.295 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:00.293+0000 7f9bfd200700 1 -- 172.21.15.183:0/1577246667 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bf81a6740 con 0x7f9bf80749d0 2024-03-28T15:53:00.296 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:00.293+0000 7f9beea00700 1 -- 172.21.15.183:0/1577246667 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9be40043d0 con 0x7f9bf80749d0 2024-03-28T15:53:00.296 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:00.293+0000 7f9beea00700 1 -- 172.21.15.183:0/1577246667 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be4011af0 con 0x7f9bf80749d0 2024-03-28T15:53:00.296 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:00.293+0000 7f9bfd200700 1 -- 172.21.15.183:0/1577246667 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bf81a6cd0 con 0x7f9bf80749d0 2024-03-28T15:53:00.296 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:00.295+0000 7f9beea00700 1 -- 172.21.15.183:0/1577246667 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f9be40036a0 con 0x7f9bf80749d0 2024-03-28T15:53:00.296 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:00.295+0000 7f9bfd200700 1 -- 172.21.15.183:0/1577246667 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bf80637c0 con 0x7f9bf80749d0 2024-03-28T15:53:00.296 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:00.295+0000 7f9beea00700 1 --2- 172.21.15.183:0/1577246667 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f9bd8039450 0x7f9bd803b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:00.297 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:00.295+0000 7f9beea00700 1 -- 172.21.15.183:0/1577246667 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f9be404bb50 con 0x7f9bf80749d0 2024-03-28T15:53:00.297 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:00.296+0000 7f9bf5e00700 1 --2- 172.21.15.183:0/1577246667 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f9bd8039450 0x7f9bd803b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:00.298 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:00.297+0000 7f9bf5e00700 1 --2- 172.21.15.183:0/1577246667 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f9bd8039450 0x7f9bd803b910 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9be800fb20 tx=0x7f9be8005c00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:53:00.313 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:00.312+0000 7f9beea00700 1 -- 172.21.15.183:0/1577246667 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9be404de20 con 0x7f9bf80749d0 2024-03-28T15:53:00.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:00 smithi183 ceph-mon[35580]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:00.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:00 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:00.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:00 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:00.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:00 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:53:00.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:00 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T15:53:00.641 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:00.640+0000 7f9bfd200700 1 -- 172.21.15.183:0/1577246667 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9bf802dbf0 con 0x7f9bf80749d0 2024-03-28T15:53:00.642 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:00.641+0000 7f9beea00700 1 -- 172.21.15.183:0/1577246667 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1285 (secure 0 0 0) 0x7f9be4020910 con 0x7f9bf80749d0 2024-03-28T15:53:00.642 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T15:53:00.643 INFO:teuthology.orchestra.run.smithi183.stdout:{"epoch":3,"fsid":"b5f8225a-ed1a-11ee-95d0-87774f69a715","modified":"2024-03-28T15:52:54.367854Z","created":"2024-03-28T15:50:17.125940Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:3300","nonce":0},{"type":"v1","addr":"172.21.15.183:6789","nonce":0}]},"addr":"172.21.15.183:6789/0","public_addr":"172.21.15.183:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-03-28T15:53:00.645 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:00.643+0000 7f9bfd200700 1 -- 172.21.15.183:0/1577246667 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f9bd8039450 msgr2=0x7f9bd803b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:53:00.645 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:00.643+0000 7f9bfd200700 1 --2- 172.21.15.183:0/1577246667 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f9bd8039450 0x7f9bd803b910 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9be800fb20 tx=0x7f9be8005c00 comp rx=0 tx=0).stop 2024-03-28T15:53:00.645 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:00.643+0000 7f9bfd200700 1 -- 172.21.15.183:0/1577246667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9bf80749d0 msgr2=0x7f9bf8074e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:53:00.645 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:00.643+0000 7f9bfd200700 1 --2- 172.21.15.183:0/1577246667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9bf80749d0 0x7f9bf8074e50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9be400b780 tx=0x7f9be400ba90 comp rx=0 tx=0).stop 2024-03-28T15:53:00.645 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:00.643+0000 7f9bfd200700 1 -- 172.21.15.183:0/1577246667 shutdown_connections 2024-03-28T15:53:00.645 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:00.643+0000 7f9bfd200700 1 --2- 172.21.15.183:0/1577246667 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f9bd8039450 0x7f9bd803b910 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:00.645 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:00.643+0000 7f9bfd200700 1 --2- 172.21.15.183:0/1577246667 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9bf81a3930 0x7f9bf81a5d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:00.645 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:00.643+0000 7f9bfd200700 1 --2- 172.21.15.183:0/1577246667 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9bf8101eb0 0x7f9bf8074490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:00.645 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:00.643+0000 7f9bfd200700 1 --2- 172.21.15.183:0/1577246667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9bf80749d0 0x7f9bf8074e50 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:00.646 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:00.643+0000 7f9bfd200700 1 -- 172.21.15.183:0/1577246667 >> 172.21.15.183:0/1577246667 conn(0x7f9bf80fd720 msgr2=0x7f9bf80fe960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:53:00.646 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:00.644+0000 7f9bfd200700 1 -- 172.21.15.183:0/1577246667 shutdown_connections 2024-03-28T15:53:00.646 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:00.644+0000 7f9bfd200700 1 -- 172.21.15.183:0/1577246667 wait complete. 2024-03-28T15:53:00.646 INFO:teuthology.orchestra.run.smithi183.stderr:dumped monmap epoch 3 2024-03-28T15:53:00.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:00 smithi150 ceph-mon[34825]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:00.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:00 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:00.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:00 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:00.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:00 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:53:00.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:00 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T15:53:00.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:00 smithi067 ceph-mon[31707]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:00.821 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:00 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:00.821 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:00 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:00.821 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:00 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:53:00.822 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:00 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T15:53:01.409 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-28T15:53:01.409 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph config generate-minimal-conf 2024-03-28T15:53:01.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:01 smithi183 ceph-mon[35580]: from='client.? 172.21.15.183:0/1577246667' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-28T15:53:01.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:01 smithi150 ceph-mon[34825]: from='client.? 172.21.15.183:0/1577246667' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-28T15:53:01.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:01 smithi067 ceph-mon[31707]: from='client.? 172.21.15.183:0/1577246667' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-28T15:53:01.875 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:53:02.875 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:02 smithi067 ceph-mon[31707]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:02.875 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:02 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:02.875 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:02 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:02.875 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:02 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:02.875 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:02 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:02.876 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:02 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:53:02.876 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:02 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:53:03.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:02 smithi150 ceph-mon[34825]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:03.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:02 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:03.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:02 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:03.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:02 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:03.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:02 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:03.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:02 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:53:03.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:02 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:53:03.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:02 smithi183 ceph-mon[35580]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:03.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:02 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:03.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:02 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:03.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:02 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:03.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:02 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:03.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:02 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:53:03.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:02 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:53:03.765 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:03 smithi067 ceph-mon[31707]: Updating smithi067:/etc/ceph/ceph.conf 2024-03-28T15:53:03.765 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:03 smithi067 ceph-mon[31707]: Updating smithi150:/etc/ceph/ceph.conf 2024-03-28T15:53:03.765 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:03 smithi067 ceph-mon[31707]: Updating smithi183:/etc/ceph/ceph.conf 2024-03-28T15:53:03.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:03.940+0000 7f2fd3200700 1 -- 172.21.15.67:0/1806259921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fcc100160 msgr2=0x7f2fcc100540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:53:03.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:03.940+0000 7f2fd3200700 1 --2- 172.21.15.67:0/1806259921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fcc100160 0x7f2fcc100540 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2fc0009910 tx=0x7f2fc0009c20 comp rx=0 tx=0).stop 2024-03-28T15:53:03.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:03.940+0000 7f2fd3200700 1 -- 172.21.15.67:0/1806259921 shutdown_connections 2024-03-28T15:53:03.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:03.940+0000 7f2fd3200700 1 --2- 172.21.15.67:0/1806259921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fcc100160 0x7f2fcc100540 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:03.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:03.940+0000 7f2fd3200700 1 -- 172.21.15.67:0/1806259921 >> 172.21.15.67:0/1806259921 conn(0x7f2fcc0fbb20 msgr2=0x7f2fcc0fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:53:03.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:03.940+0000 7f2fd3200700 1 -- 172.21.15.67:0/1806259921 shutdown_connections 2024-03-28T15:53:03.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:03.941+0000 7f2fd3200700 1 -- 172.21.15.67:0/1806259921 wait complete. 2024-03-28T15:53:03.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:03.942+0000 7f2fd3200700 1 Processor -- start 2024-03-28T15:53:03.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:03.942+0000 7f2fd3200700 1 -- start start 2024-03-28T15:53:03.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:03.942+0000 7f2fd3200700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2fcc100160 0x7f2fcc19f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:03.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:03.942+0000 7f2fd3200700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2fcc19fb40 0x7f2fcc1996d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:03.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:03.942+0000 7f2fd3200700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fcc199c10 0x7f2fcc19a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:03.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:03.942+0000 7f2fd3200700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc10bea0 con 0x7f2fcc199c10 2024-03-28T15:53:03.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:03.942+0000 7f2fd3200700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc10bd20 con 0x7f2fcc100160 2024-03-28T15:53:03.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:03.942+0000 7f2fd3200700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc10c020 con 0x7f2fcc19fb40 2024-03-28T15:53:03.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:03.943+0000 7f2fd2800700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fcc199c10 0x7f2fcc19a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:03.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:03.943+0000 7f2fd2800700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fcc199c10 0x7f2fcc19a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45034/0 (socket says 172.21.15.67:45034) 2024-03-28T15:53:03.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:03.943+0000 7f2fd2800700 1 -- 172.21.15.67:0/517759233 learned_addr learned my addr 172.21.15.67:0/517759233 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:53:03.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:03.943+0000 7f2fd1400700 1 --2- 172.21.15.67:0/517759233 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2fcc19fb40 0x7f2fcc1996d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:03.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:03.943+0000 7f2fd2800700 1 -- 172.21.15.67:0/517759233 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2fcc19fb40 msgr2=0x7f2fcc1996d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:53:03.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:03.943+0000 7f2fd2800700 1 --2- 172.21.15.67:0/517759233 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2fcc19fb40 0x7f2fcc1996d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:03.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:03.943+0000 7f2fd2800700 1 -- 172.21.15.67:0/517759233 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2fcc100160 msgr2=0x7f2fcc19f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:53:03.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:03.943+0000 7f2fd2800700 1 --2- 172.21.15.67:0/517759233 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2fcc100160 0x7f2fcc19f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:03.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:03.943+0000 7f2fd2800700 1 -- 172.21.15.67:0/517759233 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fcc1a63b0 con 0x7f2fcc199c10 2024-03-28T15:53:03.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:03.944+0000 7f2fd2800700 1 --2- 172.21.15.67:0/517759233 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fcc199c10 0x7f2fcc19a0c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2fc800ba50 tx=0x7f2fc800be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:53:03.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:03.945+0000 7f2fc6a00700 1 -- 172.21.15.67:0/517759233 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc8019070 con 0x7f2fcc199c10 2024-03-28T15:53:03.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:03.946+0000 7f2fd3200700 1 -- 172.21.15.67:0/517759233 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fcc1a6640 con 0x7f2fcc199c10 2024-03-28T15:53:03.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:03.946+0000 7f2fc6a00700 1 -- 172.21.15.67:0/517759233 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f2fc8014440 con 0x7f2fcc199c10 2024-03-28T15:53:03.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:03.946+0000 7f2fc6a00700 1 -- 172.21.15.67:0/517759233 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc8007590 con 0x7f2fcc199c10 2024-03-28T15:53:03.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:03.946+0000 7f2fd3200700 1 -- 172.21.15.67:0/517759233 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fcc1a6b30 con 0x7f2fcc199c10 2024-03-28T15:53:03.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:03.947+0000 7f2fd3200700 1 -- 172.21.15.67:0/517759233 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fcc103220 con 0x7f2fcc199c10 2024-03-28T15:53:03.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:03.947+0000 7f2fc6a00700 1 -- 172.21.15.67:0/517759233 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f2fc801c070 con 0x7f2fcc199c10 2024-03-28T15:53:03.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:03.948+0000 7f2fc6a00700 1 --2- 172.21.15.67:0/517759233 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f2fb4039450 0x7f2fb403b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:03.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:03.948+0000 7f2fc6a00700 1 -- 172.21.15.67:0/517759233 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f2fc804b770 con 0x7f2fcc199c10 2024-03-28T15:53:03.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:03.948+0000 7f2fd1e00700 1 --2- 172.21.15.67:0/517759233 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f2fb4039450 0x7f2fb403b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:03.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:03.949+0000 7f2fd1e00700 1 --2- 172.21.15.67:0/517759233 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f2fb4039450 0x7f2fb403b910 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2fc000b5c0 tx=0x7f2fc0005da0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:53:03.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:03.954+0000 7f2fc6a00700 1 -- 172.21.15.67:0/517759233 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2fc8014980 con 0x7f2fcc199c10 2024-03-28T15:53:04.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:03 smithi150 ceph-mon[34825]: Updating smithi067:/etc/ceph/ceph.conf 2024-03-28T15:53:04.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:03 smithi150 ceph-mon[34825]: Updating smithi150:/etc/ceph/ceph.conf 2024-03-28T15:53:04.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:03 smithi150 ceph-mon[34825]: Updating smithi183:/etc/ceph/ceph.conf 2024-03-28T15:53:04.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:03 smithi183 ceph-mon[35580]: Updating smithi067:/etc/ceph/ceph.conf 2024-03-28T15:53:04.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:03 smithi183 ceph-mon[35580]: Updating smithi150:/etc/ceph/ceph.conf 2024-03-28T15:53:04.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:03 smithi183 ceph-mon[35580]: Updating smithi183:/etc/ceph/ceph.conf 2024-03-28T15:53:04.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:04.187+0000 7f2fd3200700 1 -- 172.21.15.67:0/517759233 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f2fcc1a6df0 con 0x7f2fcc199c10 2024-03-28T15:53:04.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:04.188+0000 7f2fc6a00700 1 -- 172.21.15.67:0/517759233 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+275 (secure 0 0 0) 0x7f2fc8007730 con 0x7f2fcc199c10 2024-03-28T15:53:04.189 INFO:teuthology.orchestra.run.smithi067.stdout:# minimal ceph.conf for b5f8225a-ed1a-11ee-95d0-87774f69a715 2024-03-28T15:53:04.189 INFO:teuthology.orchestra.run.smithi067.stdout:[global] 2024-03-28T15:53:04.189 INFO:teuthology.orchestra.run.smithi067.stdout: fsid = b5f8225a-ed1a-11ee-95d0-87774f69a715 2024-03-28T15:53:04.189 INFO:teuthology.orchestra.run.smithi067.stdout: mon_host = [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2024-03-28T15:53:04.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:04.191+0000 7f2fd3200700 1 -- 172.21.15.67:0/517759233 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f2fb4039450 msgr2=0x7f2fb403b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:53:04.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:04.191+0000 7f2fd3200700 1 --2- 172.21.15.67:0/517759233 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f2fb4039450 0x7f2fb403b910 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2fc000b5c0 tx=0x7f2fc0005da0 comp rx=0 tx=0).stop 2024-03-28T15:53:04.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:04.191+0000 7f2fd3200700 1 -- 172.21.15.67:0/517759233 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fcc199c10 msgr2=0x7f2fcc19a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:53:04.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:04.191+0000 7f2fd3200700 1 --2- 172.21.15.67:0/517759233 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fcc199c10 0x7f2fcc19a0c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2fc800ba50 tx=0x7f2fc800be10 comp rx=0 tx=0).stop 2024-03-28T15:53:04.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:04.191+0000 7f2fd3200700 1 -- 172.21.15.67:0/517759233 shutdown_connections 2024-03-28T15:53:04.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:04.191+0000 7f2fd3200700 1 --2- 172.21.15.67:0/517759233 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f2fb4039450 0x7f2fb403b910 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:04.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:04.191+0000 7f2fd3200700 1 --2- 172.21.15.67:0/517759233 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fcc199c10 0x7f2fcc19a0c0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:04.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:04.191+0000 7f2fd3200700 1 --2- 172.21.15.67:0/517759233 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2fcc100160 0x7f2fcc19f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:04.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:04.191+0000 7f2fd3200700 1 --2- 172.21.15.67:0/517759233 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2fcc19fb40 0x7f2fcc1996d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:04.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:04.191+0000 7f2fd3200700 1 -- 172.21.15.67:0/517759233 >> 172.21.15.67:0/517759233 conn(0x7f2fcc0fbb20 msgr2=0x7f2fcc109540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:53:04.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:04.191+0000 7f2fd3200700 1 -- 172.21.15.67:0/517759233 shutdown_connections 2024-03-28T15:53:04.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:04.191+0000 7f2fd3200700 1 -- 172.21.15.67:0/517759233 wait complete. 2024-03-28T15:53:04.901 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:04 smithi067 ceph-mon[31707]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:04.901 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:04 smithi067 ceph-mon[31707]: Updating smithi067:/var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T15:53:04.901 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:04 smithi067 ceph-mon[31707]: Updating smithi183:/var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T15:53:04.901 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:04 smithi067 ceph-mon[31707]: Updating smithi150:/var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T15:53:04.901 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:04 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:04.901 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:04 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:04.901 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:04 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:04.902 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:04 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:04.902 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:04 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:04.902 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:04 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:04.902 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:04 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:04.902 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:04 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:04.902 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:04 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/517759233' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:53:04.902 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:04 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:04.902 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:04 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:04.902 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:04 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:04.902 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:04 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:04.902 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:04 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T15:53:04.902 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:04 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T15:53:04.903 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:04 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:53:04.983 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-28T15:53:04.983 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-28T15:53:04.983 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/ceph/ceph.conf 2024-03-28T15:53:05.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:04 smithi150 ceph-mon[34825]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:05.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:04 smithi150 ceph-mon[34825]: Updating smithi067:/var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T15:53:05.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:04 smithi150 ceph-mon[34825]: Updating smithi183:/var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T15:53:05.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:04 smithi150 ceph-mon[34825]: Updating smithi150:/var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T15:53:05.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:04 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:05.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:04 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:05.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:04 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:05.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:04 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:05.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:04 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:05.054 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:04 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:05.054 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:04 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:05.054 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:04 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:05.054 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:04 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/517759233' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:53:05.054 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:04 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:05.054 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:04 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:05.054 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:04 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:05.054 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:04 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:05.054 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:04 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T15:53:05.054 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:04 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T15:53:05.054 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:04 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:53:05.066 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-28T15:53:05.066 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-28T15:53:05.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:04 smithi183 ceph-mon[35580]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:05.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:04 smithi183 ceph-mon[35580]: Updating smithi067:/var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T15:53:05.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:04 smithi183 ceph-mon[35580]: Updating smithi183:/var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T15:53:05.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:04 smithi183 ceph-mon[35580]: Updating smithi150:/var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T15:53:05.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:04 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:05.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:04 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:05.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:04 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:05.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:04 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:05.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:04 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:05.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:04 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:05.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:04 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:05.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:04 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:05.102 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:04 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/517759233' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:53:05.102 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:04 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:05.102 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:04 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:05.102 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:04 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:05.102 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:04 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:05.102 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:04 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T15:53:05.102 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:04 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T15:53:05.102 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:04 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:53:05.181 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-28T15:53:05.181 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/ceph/ceph.conf 2024-03-28T15:53:05.284 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-28T15:53:05.284 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-28T15:53:05.424 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-28T15:53:05.424 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/ceph/ceph.conf 2024-03-28T15:53:05.527 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-28T15:53:05.527 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-28T15:53:05.666 INFO:tasks.cephadm:Adding mgr.x on smithi067 2024-03-28T15:53:05.666 INFO:tasks.cephadm:Adding mgr.y on smithi150 2024-03-28T15:53:05.666 INFO:tasks.cephadm:Adding mgr.z on smithi183 2024-03-28T15:53:05.666 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph orch apply mgr '3;smithi067=x;smithi150=y;smithi183=z' 2024-03-28T15:53:05.790 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:05 smithi183 ceph-mon[35580]: Reconfiguring mon.a (unknown last config time)... 2024-03-28T15:53:05.791 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:05 smithi183 ceph-mon[35580]: Reconfiguring daemon mon.a on smithi067 2024-03-28T15:53:05.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:05 smithi067 ceph-mon[31707]: Reconfiguring mon.a (unknown last config time)... 2024-03-28T15:53:05.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:05 smithi067 ceph-mon[31707]: Reconfiguring daemon mon.a on smithi067 2024-03-28T15:53:06.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:05 smithi150 ceph-mon[34825]: Reconfiguring mon.a (unknown last config time)... 2024-03-28T15:53:06.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:05 smithi150 ceph-mon[34825]: Reconfiguring daemon mon.a on smithi067 2024-03-28T15:53:06.141 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T15:53:06.846 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:06 smithi183 ceph-mon[35580]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:06.847 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:06 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:06.847 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:06 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:06.847 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:06 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T15:53:06.847 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:06 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T15:53:06.847 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:06 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:53:07.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:06 smithi150 ceph-mon[34825]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:07.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:06 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:07.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:06 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:07.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:06 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T15:53:07.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:06 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T15:53:07.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:06 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:53:07.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:06 smithi067 ceph-mon[31707]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:07.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:06 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:07.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:06 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:07.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:06 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T15:53:07.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:06 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T15:53:07.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:06 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:53:07.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:07 smithi183 ceph-mon[35580]: Reconfiguring mon.b (monmap changed)... 2024-03-28T15:53:07.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:07 smithi183 ceph-mon[35580]: Reconfiguring daemon mon.b on smithi150 2024-03-28T15:53:07.998 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:07 smithi150 ceph-mon[34825]: Reconfiguring mon.b (monmap changed)... 2024-03-28T15:53:07.998 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:07 smithi150 ceph-mon[34825]: Reconfiguring daemon mon.b on smithi150 2024-03-28T15:53:08.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:07 smithi067 ceph-mon[31707]: Reconfiguring mon.b (monmap changed)... 2024-03-28T15:53:08.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:07 smithi067 ceph-mon[31707]: Reconfiguring daemon mon.b on smithi150 2024-03-28T15:53:08.305 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:08.304+0000 7f7131200700 1 -- 172.21.15.183:0/2003813896 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f712c101cc0 msgr2=0x7f712c1020e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:53:08.305 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:08.305+0000 7f7131200700 1 --2- 172.21.15.183:0/2003813896 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f712c101cc0 0x7f712c1020e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f711c009910 tx=0x7f711c009c20 comp rx=0 tx=0).stop 2024-03-28T15:53:08.305 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:08.305+0000 7f7131200700 1 -- 172.21.15.183:0/2003813896 shutdown_connections 2024-03-28T15:53:08.306 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:08.305+0000 7f7131200700 1 --2- 172.21.15.183:0/2003813896 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f712c101cc0 0x7f712c1020e0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:08.306 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:08.305+0000 7f7131200700 1 -- 172.21.15.183:0/2003813896 >> 172.21.15.183:0/2003813896 conn(0x7f712c0fd260 msgr2=0x7f712c0ff6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:53:08.306 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:08.305+0000 7f7131200700 1 -- 172.21.15.183:0/2003813896 shutdown_connections 2024-03-28T15:53:08.306 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:08.305+0000 7f7131200700 1 -- 172.21.15.183:0/2003813896 wait complete. 2024-03-28T15:53:08.307 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:08.306+0000 7f7131200700 1 Processor -- start 2024-03-28T15:53:08.307 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:08.306+0000 7f7131200700 1 -- start start 2024-03-28T15:53:08.307 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:08.306+0000 7f7131200700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f712c101cc0 0x7f712c199650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:08.307 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:08.306+0000 7f7131200700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f712c199b90 0x7f712c19ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:08.307 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:08.306+0000 7f7131200700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f712c19a010 0x7f712c1a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:08.307 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:08.306+0000 7f7131200700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f712c107c00 con 0x7f712c199b90 2024-03-28T15:53:08.307 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:08.306+0000 7f7131200700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f712c107a80 con 0x7f712c19a010 2024-03-28T15:53:08.308 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:08.306+0000 7f7131200700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f712c107d80 con 0x7f712c101cc0 2024-03-28T15:53:08.308 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:08.307+0000 7f7129e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f712c101cc0 0x7f712c199650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:08.308 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:08.307+0000 7f7129e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f712c101cc0 0x7f712c199650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.183:43450/0 (socket says 172.21.15.183:43450) 2024-03-28T15:53:08.308 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:08.307+0000 7f7129e00700 1 -- 172.21.15.183:0/2516603823 learned_addr learned my addr 172.21.15.183:0/2516603823 (peer_addr_for_me v2:172.21.15.183:0/0) 2024-03-28T15:53:08.308 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:08.307+0000 7f7121400700 1 --2- 172.21.15.183:0/2516603823 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f712c199b90 0x7f712c19ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:08.308 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:08.307+0000 7f7129e00700 1 -- 172.21.15.183:0/2516603823 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f712c19a010 msgr2=0x7f712c1a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:53:08.308 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:08.307+0000 7f7129e00700 1 --2- 172.21.15.183:0/2516603823 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f712c19a010 0x7f712c1a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:08.308 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:08.307+0000 7f7129e00700 1 -- 172.21.15.183:0/2516603823 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f712c199b90 msgr2=0x7f712c19ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:53:08.308 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:08.307+0000 7f7129e00700 1 --2- 172.21.15.183:0/2516603823 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f712c199b90 0x7f712c19ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:08.308 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:08.307+0000 7f7129e00700 1 -- 172.21.15.183:0/2516603823 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f712c1a1a60 con 0x7f712c101cc0 2024-03-28T15:53:08.309 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:08.308+0000 7f7129e00700 1 --2- 172.21.15.183:0/2516603823 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f712c101cc0 0x7f712c199650 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f711c005e00 tx=0x7f711c005030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:53:08.310 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:08.308+0000 7f7123400700 1 -- 172.21.15.183:0/2516603823 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f711c027070 con 0x7f712c101cc0 2024-03-28T15:53:08.310 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:08.308+0000 7f7123400700 1 -- 172.21.15.183:0/2516603823 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f711c00be10 con 0x7f712c101cc0 2024-03-28T15:53:08.310 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:08.308+0000 7f7131200700 1 -- 172.21.15.183:0/2516603823 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f712c1a1cf0 con 0x7f712c101cc0 2024-03-28T15:53:08.310 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:08.308+0000 7f7123400700 1 -- 172.21.15.183:0/2516603823 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f711c00f460 con 0x7f712c101cc0 2024-03-28T15:53:08.310 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:08.308+0000 7f7131200700 1 -- 172.21.15.183:0/2516603823 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f712c1a2200 con 0x7f712c101cc0 2024-03-28T15:53:08.310 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:08.309+0000 7f7123400700 1 -- 172.21.15.183:0/2516603823 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f711c02a070 con 0x7f712c101cc0 2024-03-28T15:53:08.310 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:08.309+0000 7f7131200700 1 -- 172.21.15.183:0/2516603823 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f712c0637c0 con 0x7f712c101cc0 2024-03-28T15:53:08.310 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:08.310+0000 7f7123400700 1 --2- 172.21.15.183:0/2516603823 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7110039450 0x7f711003b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:08.310 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:08.310+0000 7f7123400700 1 -- 172.21.15.183:0/2516603823 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f711c04d390 con 0x7f712c101cc0 2024-03-28T15:53:08.311 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:08.310+0000 7f7121400700 1 --2- 172.21.15.183:0/2516603823 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7110039450 0x7f711003b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:08.311 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:08.311+0000 7f7121400700 1 --2- 172.21.15.183:0/2516603823 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7110039450 0x7f711003b910 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7114009850 tx=0x7f7114008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:53:08.316 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:08.316+0000 7f7123400700 1 -- 172.21.15.183:0/2516603823 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f711c013070 con 0x7f712c101cc0 2024-03-28T15:53:08.556 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:08.556+0000 7f7131200700 1 -- 172.21.15.183:0/2516603823 --> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi067=x;smithi150=y;smithi183=z", "target": ["mon-mgr", ""]}) v1 -- 0x7f712c062660 con 0x7f7110039450 2024-03-28T15:53:08.573 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:08.571+0000 7f7123400700 1 -- 172.21.15.183:0/2516603823 <== mgr.14152 v2:172.21.15.67:6800/208726654 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f712c062660 con 0x7f7110039450 2024-03-28T15:53:08.573 INFO:teuthology.orchestra.run.smithi183.stdout:Scheduled mgr update... 2024-03-28T15:53:08.574 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:08.574+0000 7f7131200700 1 -- 172.21.15.183:0/2516603823 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7110039450 msgr2=0x7f711003b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:53:08.574 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:08.574+0000 7f7131200700 1 --2- 172.21.15.183:0/2516603823 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7110039450 0x7f711003b910 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7114009850 tx=0x7f7114008040 comp rx=0 tx=0).stop 2024-03-28T15:53:08.574 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:08.574+0000 7f7131200700 1 -- 172.21.15.183:0/2516603823 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f712c101cc0 msgr2=0x7f712c199650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:53:08.574 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:08.574+0000 7f7131200700 1 --2- 172.21.15.183:0/2516603823 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f712c101cc0 0x7f712c199650 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f711c005e00 tx=0x7f711c005030 comp rx=0 tx=0).stop 2024-03-28T15:53:08.574 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:08.574+0000 7f7131200700 1 -- 172.21.15.183:0/2516603823 shutdown_connections 2024-03-28T15:53:08.574 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:08.574+0000 7f7131200700 1 --2- 172.21.15.183:0/2516603823 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7110039450 0x7f711003b910 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:08.574 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:08.574+0000 7f7131200700 1 --2- 172.21.15.183:0/2516603823 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f712c19a010 0x7f712c1a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:08.574 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:08.574+0000 7f7131200700 1 --2- 172.21.15.183:0/2516603823 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f712c101cc0 0x7f712c199650 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:08.574 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:08.574+0000 7f7131200700 1 --2- 172.21.15.183:0/2516603823 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f712c199b90 0x7f712c19ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:08.574 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:08.574+0000 7f7131200700 1 -- 172.21.15.183:0/2516603823 >> 172.21.15.183:0/2516603823 conn(0x7f712c0fd260 msgr2=0x7f712c0ff6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:53:08.575 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:08.574+0000 7f7131200700 1 -- 172.21.15.183:0/2516603823 shutdown_connections 2024-03-28T15:53:08.575 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:53:08.574+0000 7f7131200700 1 -- 172.21.15.183:0/2516603823 wait complete. 2024-03-28T15:53:08.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:08 smithi183 ceph-mon[35580]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:08.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:08 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:08.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:08 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:08.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:08 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T15:53:08.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:08 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T15:53:08.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:08 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:53:08.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:08 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:09.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:08 smithi150 ceph-mon[34825]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:09.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:08 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:09.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:08 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:09.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:08 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T15:53:09.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:08 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T15:53:09.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:08 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:53:09.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:08 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:09.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:08 smithi067 ceph-mon[31707]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:09.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:08 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:09.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:08 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:09.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:08 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T15:53:09.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:08 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T15:53:09.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:08 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:53:09.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:08 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:09.335 DEBUG:teuthology.orchestra.run.smithi150:mgr.y> sudo journalctl -f -n 0 -u ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@mgr.y.service 2024-03-28T15:53:09.343 DEBUG:teuthology.orchestra.run.smithi183:mgr.z> sudo journalctl -f -n 0 -u ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@mgr.z.service 2024-03-28T15:53:09.349 INFO:tasks.cephadm:Deploying OSDs... 2024-03-28T15:53:09.349 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-28T15:53:09.349 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/scratch_devs of=/dev/stdout 2024-03-28T15:53:09.409 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-28T15:53:09.409 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_1 2024-03-28T15:53:09.469 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-28T15:53:09.469 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T15:53:09.470 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 627 Links: 1 2024-03-28T15:53:09.470 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T15:53:09.470 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T15:53:09.470 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-03-28 15:51:55.393988553 +0000 2024-03-28T15:53:09.470 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-03-28 15:45:46.023738056 +0000 2024-03-28T15:53:09.470 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-03-28 15:45:46.023738056 +0000 2024-03-28T15:53:09.470 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: 2024-03-28 15:45:46.023738056 +0000 2024-03-28T15:53:09.470 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-28T15:53:09.567 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-03-28T15:53:09.568 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-03-28T15:53:09.568 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000520896 s, 983 kB/s 2024-03-28T15:53:09.572 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-28T15:53:09.635 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_2 2024-03-28T15:53:09.688 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-28T15:53:09.688 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T15:53:09.688 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 645 Links: 1 2024-03-28T15:53:09.688 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T15:53:09.688 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T15:53:09.688 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-03-28 15:51:55.466986933 +0000 2024-03-28T15:53:09.688 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-03-28 15:45:46.617724908 +0000 2024-03-28T15:53:09.688 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-03-28 15:45:46.617724908 +0000 2024-03-28T15:53:09.688 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: 2024-03-28 15:45:46.617724908 +0000 2024-03-28T15:53:09.689 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-28T15:53:09.779 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-03-28T15:53:09.779 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-03-28T15:53:09.779 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000341466 s, 1.5 MB/s 2024-03-28T15:53:09.784 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-28T15:53:09.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:09 smithi150 ceph-mon[34825]: Reconfiguring mon.c (monmap changed)... 2024-03-28T15:53:09.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:09 smithi150 ceph-mon[34825]: Reconfiguring daemon mon.c on smithi183 2024-03-28T15:53:09.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:09 smithi150 ceph-mon[34825]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi067=x;smithi150=y;smithi183=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:53:09.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:09 smithi150 ceph-mon[34825]: Saving service mgr spec with placement smithi067=x;smithi150=y;smithi183=z;count:3 2024-03-28T15:53:09.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:09 smithi067 ceph-mon[31707]: Reconfiguring mon.c (monmap changed)... 2024-03-28T15:53:09.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:09 smithi067 ceph-mon[31707]: Reconfiguring daemon mon.c on smithi183 2024-03-28T15:53:09.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:09 smithi067 ceph-mon[31707]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi067=x;smithi150=y;smithi183=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:53:09.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:09 smithi067 ceph-mon[31707]: Saving service mgr spec with placement smithi067=x;smithi150=y;smithi183=z;count:3 2024-03-28T15:53:09.842 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_3 2024-03-28T15:53:09.940 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-28T15:53:09.940 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T15:53:09.940 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-28T15:53:09.940 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T15:53:09.940 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T15:53:09.940 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-03-28 15:51:55.549985091 +0000 2024-03-28T15:53:09.941 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-03-28 15:45:47.218711605 +0000 2024-03-28T15:53:09.941 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-03-28 15:45:47.218711605 +0000 2024-03-28T15:53:09.941 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: 2024-03-28 15:45:47.218711605 +0000 2024-03-28T15:53:09.941 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-28T15:53:10.037 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-03-28T15:53:10.037 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-03-28T15:53:10.037 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000369479 s, 1.4 MB/s 2024-03-28T15:53:10.041 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-28T15:53:10.100 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_4 2024-03-28T15:53:10.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:09 smithi183 ceph-mon[35580]: Reconfiguring mon.c (monmap changed)... 2024-03-28T15:53:10.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:09 smithi183 ceph-mon[35580]: Reconfiguring daemon mon.c on smithi183 2024-03-28T15:53:10.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:09 smithi183 ceph-mon[35580]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi067=x;smithi150=y;smithi183=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:53:10.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:09 smithi183 ceph-mon[35580]: Saving service mgr spec with placement smithi067=x;smithi150=y;smithi183=z;count:3 2024-03-28T15:53:10.153 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-28T15:53:10.153 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T15:53:10.153 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-28T15:53:10.153 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T15:53:10.153 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T15:53:10.153 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-03-28 15:51:55.623983449 +0000 2024-03-28T15:53:10.153 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-03-28 15:45:47.803698656 +0000 2024-03-28T15:53:10.153 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-03-28 15:45:47.803698656 +0000 2024-03-28T15:53:10.153 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: 2024-03-28 15:45:47.803698656 +0000 2024-03-28T15:53:10.154 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-28T15:53:10.244 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-03-28T15:53:10.244 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-03-28T15:53:10.244 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000386301 s, 1.3 MB/s 2024-03-28T15:53:10.248 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-28T15:53:10.305 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-28T15:53:10.305 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/scratch_devs of=/dev/stdout 2024-03-28T15:53:10.365 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-28T15:53:10.366 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_1 2024-03-28T15:53:10.420 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-28T15:53:10.420 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T15:53:10.420 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 5h/5d Inode: 631 Links: 1 2024-03-28T15:53:10.420 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T15:53:10.420 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T15:53:10.420 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2024-03-28 15:52:53.476535932 +0000 2024-03-28T15:53:10.420 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2024-03-28 15:45:26.315857047 +0000 2024-03-28T15:53:10.420 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2024-03-28 15:45:26.315857047 +0000 2024-03-28T15:53:10.420 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: 2024-03-28 15:45:26.315857047 +0000 2024-03-28T15:53:10.421 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-28T15:53:10.510 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2024-03-28T15:53:10.510 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2024-03-28T15:53:10.510 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000485412 s, 1.1 MB/s 2024-03-28T15:53:10.514 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-28T15:53:10.569 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_2 2024-03-28T15:53:10.617 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-28T15:53:10.617 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T15:53:10.617 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 5h/5d Inode: 652 Links: 1 2024-03-28T15:53:10.617 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T15:53:10.617 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T15:53:10.617 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2024-03-28 15:52:53.477535911 +0000 2024-03-28T15:53:10.618 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2024-03-28 15:45:26.916844434 +0000 2024-03-28T15:53:10.618 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2024-03-28 15:45:26.916844434 +0000 2024-03-28T15:53:10.618 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: 2024-03-28 15:45:26.916844434 +0000 2024-03-28T15:53:10.618 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-28T15:53:10.705 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2024-03-28T15:53:10.705 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2024-03-28T15:53:10.705 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.00033094 s, 1.5 MB/s 2024-03-28T15:53:10.709 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-28T15:53:10.766 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_3 2024-03-28T15:53:10.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:10 smithi150 ceph-mon[34825]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:10.819 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-28T15:53:10.819 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T15:53:10.819 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 5h/5d Inode: 670 Links: 1 2024-03-28T15:53:10.819 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T15:53:10.819 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T15:53:10.819 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2024-03-28 15:52:53.477535911 +0000 2024-03-28T15:53:10.819 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2024-03-28 15:45:27.501832157 +0000 2024-03-28T15:53:10.819 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2024-03-28 15:45:27.501832157 +0000 2024-03-28T15:53:10.820 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: 2024-03-28 15:45:27.501832157 +0000 2024-03-28T15:53:10.820 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-28T15:53:10.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:10 smithi183 ceph-mon[35580]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:10.954 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2024-03-28T15:53:10.954 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2024-03-28T15:53:10.954 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000355217 s, 1.4 MB/s 2024-03-28T15:53:10.958 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-28T15:53:11.015 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_4 2024-03-28T15:53:11.065 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-28T15:53:11.065 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T15:53:11.066 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 5h/5d Inode: 689 Links: 1 2024-03-28T15:53:11.066 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T15:53:11.066 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T15:53:11.066 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2024-03-28 15:52:53.478535890 +0000 2024-03-28T15:53:11.066 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2024-03-28 15:45:28.078820048 +0000 2024-03-28T15:53:11.066 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2024-03-28 15:45:28.078820048 +0000 2024-03-28T15:53:11.066 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: 2024-03-28 15:45:28.078820048 +0000 2024-03-28T15:53:11.066 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-28T15:53:11.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:10 smithi067 ceph-mon[31707]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:11.152 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2024-03-28T15:53:11.152 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2024-03-28T15:53:11.152 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000325526 s, 1.6 MB/s 2024-03-28T15:53:11.156 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-28T15:53:11.210 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-28T15:53:11.210 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/scratch_devs of=/dev/stdout 2024-03-28T15:53:11.262 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-28T15:53:11.262 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_1 2024-03-28T15:53:11.308 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-28T15:53:11.308 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T15:53:11.309 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 633 Links: 1 2024-03-28T15:53:11.309 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T15:53:11.309 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T15:53:11.309 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-03-28 15:52:42.548147548 +0000 2024-03-28T15:53:11.309 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-03-28 15:45:26.329348537 +0000 2024-03-28T15:53:11.309 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-03-28 15:45:26.329348537 +0000 2024-03-28T15:53:11.309 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-03-28 15:45:26.329348537 +0000 2024-03-28T15:53:11.309 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-28T15:53:11.385 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-03-28T15:53:11.385 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-03-28T15:53:11.385 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000331702 s, 1.5 MB/s 2024-03-28T15:53:11.389 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-28T15:53:11.439 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_2 2024-03-28T15:53:11.482 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-28T15:53:11.482 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T15:53:11.482 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 651 Links: 1 2024-03-28T15:53:11.482 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T15:53:11.482 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T15:53:11.482 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-03-28 15:52:42.548147548 +0000 2024-03-28T15:53:11.482 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-03-28 15:45:26.903333196 +0000 2024-03-28T15:53:11.483 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-03-28 15:45:26.903333196 +0000 2024-03-28T15:53:11.483 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-03-28 15:45:26.903333196 +0000 2024-03-28T15:53:11.483 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-28T15:53:11.598 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-03-28T15:53:11.598 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-03-28T15:53:11.598 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000291128 s, 1.8 MB/s 2024-03-28T15:53:11.601 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-28T15:53:11.642 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_3 2024-03-28T15:53:11.724 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-28T15:53:11.724 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T15:53:11.724 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 669 Links: 1 2024-03-28T15:53:11.724 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T15:53:11.724 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T15:53:11.725 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-03-28 15:52:42.549147521 +0000 2024-03-28T15:53:11.725 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-03-28 15:45:27.500317241 +0000 2024-03-28T15:53:11.725 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-03-28 15:45:27.500317241 +0000 2024-03-28T15:53:11.725 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-03-28 15:45:27.500317241 +0000 2024-03-28T15:53:11.725 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-28T15:53:11.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:11 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:11.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:11 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:11.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:11 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:53:11.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:11 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:53:11.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:11 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:53:11.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:11 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:11.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:11 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T15:53:11.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:11 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-28T15:53:11.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:11 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T15:53:11.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:11 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:53:11.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:11 smithi150 ceph-mon[34825]: Deploying daemon mgr.z on smithi183 2024-03-28T15:53:11.854 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-03-28T15:53:11.855 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-03-28T15:53:11.855 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.00033045 s, 1.5 MB/s 2024-03-28T15:53:11.859 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-28T15:53:11.916 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_4 2024-03-28T15:53:11.959 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-28T15:53:11.959 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T15:53:11.959 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 688 Links: 1 2024-03-28T15:53:11.960 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T15:53:11.960 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T15:53:11.960 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-03-28 15:52:42.550147494 +0000 2024-03-28T15:53:11.960 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-03-28 15:45:28.068302061 +0000 2024-03-28T15:53:11.960 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-03-28 15:45:28.068302061 +0000 2024-03-28T15:53:11.960 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-03-28 15:45:28.068302061 +0000 2024-03-28T15:53:11.960 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-28T15:53:12.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:11 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:12.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:11 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:12.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:11 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:53:12.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:11 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:53:12.071 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:11 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:53:12.071 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:11 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:12.071 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:11 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T15:53:12.071 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:11 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-28T15:53:12.071 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:11 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T15:53:12.071 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:11 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:53:12.072 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:11 smithi067 ceph-mon[31707]: Deploying daemon mgr.z on smithi183 2024-03-28T15:53:12.077 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:11 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:12.077 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:11 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:12.078 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:11 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:53:12.078 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:11 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:53:12.078 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:11 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:53:12.078 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:11 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:12.078 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:11 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T15:53:12.079 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:11 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-28T15:53:12.079 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:11 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T15:53:12.079 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:11 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:53:12.079 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:11 smithi183 ceph-mon[35580]: Deploying daemon mgr.z on smithi183 2024-03-28T15:53:12.087 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-03-28T15:53:12.087 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-03-28T15:53:12.087 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000489864 s, 1.0 MB/s 2024-03-28T15:53:12.091 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-28T15:53:12.148 INFO:tasks.cephadm:Deploying osd.0 on smithi067 with /dev/vg_nvme/lv_4... 2024-03-28T15:53:12.148 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-28T15:53:12.610 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:53:13.042 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:12 smithi067 ceph-mon[31707]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:13.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:12 smithi150 ceph-mon[34825]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:13.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:12 smithi183 ceph-mon[35580]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:13.963 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 28 15:53:13 smithi183 systemd[1]: Starting Ceph mgr.z for b5f8225a-ed1a-11ee-95d0-87774f69a715... 2024-03-28T15:53:14.264 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 28 15:53:13 smithi183 podman[37442]: 2024-03-28 15:53:13.869148747 +0000 UTC m=+0.034921266 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T15:53:14.265 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 28 15:53:13 smithi183 podman[37442]: 2024-03-28 15:53:13.976803423 +0000 UTC m=+0.142575973 container create 66d2d9e37125865ae3365ce21d6946736aa01c8cc1b7d63eff25beec9d7f5c02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-z, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-28T15:53:14.265 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 28 15:53:14 smithi183 podman[37442]: 2024-03-28 15:53:14.263704694 +0000 UTC m=+0.429477245 container init 66d2d9e37125865ae3365ce21d6946736aa01c8cc1b7d63eff25beec9d7f5c02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-z, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-28T15:53:14.600 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 28 15:53:14 smithi183 podman[37442]: 2024-03-28 15:53:14.273947899 +0000 UTC m=+0.439720450 container start 66d2d9e37125865ae3365ce21d6946736aa01c8cc1b7d63eff25beec9d7f5c02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-z, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240319) 2024-03-28T15:53:14.600 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 28 15:53:14 smithi183 ceph-mgr[37461]: -- 172.21.15.183:0/4212562925 <== mon.1 v2:172.21.15.183:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ce068065a0 con 0x55ce067dac00 2024-03-28T15:53:14.600 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 28 15:53:14 smithi183 bash[37442]: 66d2d9e37125865ae3365ce21d6946736aa01c8cc1b7d63eff25beec9d7f5c02 2024-03-28T15:53:14.601 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 28 15:53:14 smithi183 systemd[1]: Started Ceph mgr.z for b5f8225a-ed1a-11ee-95d0-87774f69a715. 2024-03-28T15:53:15.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:14 smithi150 ceph-mon[34825]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:15.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:14 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:15.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:14 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:15.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:14 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:15.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:14 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T15:53:15.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:14 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-28T15:53:15.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:14 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T15:53:15.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:14 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:53:15.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:14 smithi067 ceph-mon[31707]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:15.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:14 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:15.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:14 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:15.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:14 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:15.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:14 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T15:53:15.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:14 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-28T15:53:15.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:14 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T15:53:15.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:14 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:53:15.100 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 28 15:53:15 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-z[37457]: 2024-03-28T15:53:15.024+0000 7f67cea1f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T15:53:15.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:14 smithi183 ceph-mon[35580]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:15.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:14 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:15.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:14 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:15.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:14 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:15.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:14 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T15:53:15.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:14 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-28T15:53:15.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:14 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T15:53:15.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:14 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:53:15.306 INFO:teuthology.orchestra.run.smithi067.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-28T15:53:15.306 INFO:teuthology.orchestra.run.smithi067.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-28T15:53:15.306 INFO:teuthology.orchestra.run.smithi067.stderr: stderr: 10+0 records in 2024-03-28T15:53:15.306 INFO:teuthology.orchestra.run.smithi067.stderr:10+0 records out 2024-03-28T15:53:15.306 INFO:teuthology.orchestra.run.smithi067.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0380006 s, 276 MB/s 2024-03-28T15:53:15.306 INFO:teuthology.orchestra.run.smithi067.stderr:--> Zapping successful for: 2024-03-28T15:53:15.600 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 28 15:53:15 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-z[37457]: 2024-03-28T15:53:15.142+0000 7f67cea1f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T15:53:15.958 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:15 smithi067 ceph-mon[31707]: Deploying daemon mgr.y on smithi150 2024-03-28T15:53:16.018 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi067:vg_nvme/lv_4 2024-03-28T15:53:16.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:15 smithi183 ceph-mon[35580]: Deploying daemon mgr.y on smithi150 2024-03-28T15:53:16.100 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 28 15:53:15 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-z[37457]: 2024-03-28T15:53:15.927+0000 7f67cea1f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T15:53:16.100 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 28 15:53:16 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-z[37457]: 2024-03-28T15:53:16.047+0000 7f67cea1f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T15:53:16.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:15 smithi150 ceph-mon[34825]: Deploying daemon mgr.y on smithi150 2024-03-28T15:53:16.472 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:53:16.600 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 28 15:53:16 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-z[37457]: 2024-03-28T15:53:16.230+0000 7f67cea1f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T15:53:16.600 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 28 15:53:16 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-z[37457]: 2024-03-28T15:53:16.352+0000 7f67cea1f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T15:53:16.600 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 28 15:53:16 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-z[37457]: 2024-03-28T15:53:16.460+0000 7f67cea1f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T15:53:16.929 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:16 smithi067 ceph-mon[31707]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:17.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:16 smithi150 ceph-mon[34825]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:17.053 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:16 smithi183 ceph-mon[35580]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:17.053 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 28 15:53:16 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-z[37457]: 2024-03-28T15:53:16.800+0000 7f67cea1f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T15:53:17.053 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 28 15:53:16 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-z[37457]: 2024-03-28T15:53:16.942+0000 7f67cea1f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T15:53:17.350 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 28 15:53:17 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-z[37457]: 2024-03-28T15:53:17.052+0000 7f67cea1f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T15:53:17.350 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 28 15:53:17 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-z[37457]: 2024-03-28T15:53:17.175+0000 7f67cea1f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T15:53:17.850 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 28 15:53:17 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-z[37457]: 2024-03-28T15:53:17.389+0000 7f67cea1f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T15:53:17.929 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Mar 28 15:53:17 smithi150 systemd[1]: Starting Ceph mgr.y for b5f8225a-ed1a-11ee-95d0-87774f69a715... 2024-03-28T15:53:18.224 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Mar 28 15:53:17 smithi150 podman[36192]: 2024-03-28 15:53:17.834372068 +0000 UTC m=+0.037578085 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T15:53:18.224 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Mar 28 15:53:17 smithi150 podman[36192]: 2024-03-28 15:53:17.939492813 +0000 UTC m=+0.142698866 container create d5b1a132ba2eba3a4b5e425b8b3f8e5cb60775039278a57d6d87af6f6e22667f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-y, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-28T15:53:18.225 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Mar 28 15:53:18 smithi150 podman[36192]: 2024-03-28 15:53:18.167873557 +0000 UTC m=+0.371079616 container init d5b1a132ba2eba3a4b5e425b8b3f8e5cb60775039278a57d6d87af6f6e22667f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-y, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T15:53:18.225 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Mar 28 15:53:18 smithi150 podman[36192]: 2024-03-28 15:53:18.177208207 +0000 UTC m=+0.380414229 container start d5b1a132ba2eba3a4b5e425b8b3f8e5cb60775039278a57d6d87af6f6e22667f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-y, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_BRANCH=HEAD) 2024-03-28T15:53:18.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:18.362+0000 7f6d4ce00700 1 -- 172.21.15.67:0/865492879 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d48101ec0 msgr2=0x7f6d48108770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:53:18.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:18.362+0000 7f6d4ce00700 1 --2- 172.21.15.67:0/865492879 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d48101ec0 0x7f6d48108770 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f6d4000b0a0 tx=0x7f6d4000b3b0 comp rx=0 tx=0).stop 2024-03-28T15:53:18.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:18.362+0000 7f6d4ce00700 1 -- 172.21.15.67:0/865492879 shutdown_connections 2024-03-28T15:53:18.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:18.362+0000 7f6d4ce00700 1 --2- 172.21.15.67:0/865492879 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d48101ec0 0x7f6d48108770 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:18.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:18.362+0000 7f6d4ce00700 1 --2- 172.21.15.67:0/865492879 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6d48100320 0x7f6d48100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:18.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:18.362+0000 7f6d4ce00700 1 --2- 172.21.15.67:0/865492879 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d48101520 0x7f6d48101980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:18.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:18.362+0000 7f6d4ce00700 1 -- 172.21.15.67:0/865492879 >> 172.21.15.67:0/865492879 conn(0x7f6d480fb900 msgr2=0x7f6d480fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:53:18.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:18.362+0000 7f6d4ce00700 1 -- 172.21.15.67:0/865492879 shutdown_connections 2024-03-28T15:53:18.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:18.362+0000 7f6d4ce00700 1 -- 172.21.15.67:0/865492879 wait complete. 2024-03-28T15:53:18.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:18.363+0000 7f6d4ce00700 1 Processor -- start 2024-03-28T15:53:18.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:18.364+0000 7f6d4ce00700 1 -- start start 2024-03-28T15:53:18.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:18.364+0000 7f6d4ce00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d48100320 0x7f6d48198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:18.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:18.364+0000 7f6d4ce00700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d48101520 0x7f6d481986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:18.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:18.364+0000 7f6d4ce00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6d48101ec0 0x7f6d4819f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:18.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:18.364+0000 7f6d4ce00700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d4810cab0 con 0x7f6d48100320 2024-03-28T15:53:18.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:18.364+0000 7f6d4ce00700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d4810c930 con 0x7f6d48101520 2024-03-28T15:53:18.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:18.364+0000 7f6d4ce00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d4810cc30 con 0x7f6d48101ec0 2024-03-28T15:53:18.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:18.364+0000 7f6d47400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d48100320 0x7f6d48198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:18.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:18.364+0000 7f6d46a00700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d48101520 0x7f6d481986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:18.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:18.364+0000 7f6d47400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d48100320 0x7f6d48198170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38472/0 (socket says 172.21.15.67:38472) 2024-03-28T15:53:18.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:18.364+0000 7f6d47400700 1 -- 172.21.15.67:0/2921054664 learned_addr learned my addr 172.21.15.67:0/2921054664 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:53:18.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:18.365+0000 7f6d47e00700 1 --2- 172.21.15.67:0/2921054664 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6d48101ec0 0x7f6d4819f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:18.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:18.365+0000 7f6d46a00700 1 -- 172.21.15.67:0/2921054664 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6d48101ec0 msgr2=0x7f6d4819f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:53:18.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:18.365+0000 7f6d46a00700 1 --2- 172.21.15.67:0/2921054664 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6d48101ec0 0x7f6d4819f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:18.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:18.365+0000 7f6d46a00700 1 -- 172.21.15.67:0/2921054664 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d48100320 msgr2=0x7f6d48198170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:53:18.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:18.365+0000 7f6d46a00700 1 --2- 172.21.15.67:0/2921054664 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d48100320 0x7f6d48198170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:18.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:18.365+0000 7f6d46a00700 1 -- 172.21.15.67:0/2921054664 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d4819fdd0 con 0x7f6d48101520 2024-03-28T15:53:18.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:18.365+0000 7f6d47400700 1 --2- 172.21.15.67:0/2921054664 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d48100320 0x7f6d48198170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:53:18.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:18.365+0000 7f6d46a00700 1 --2- 172.21.15.67:0/2921054664 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d48101520 0x7f6d481986b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6d3400e9f0 tx=0x7f6d3400ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:53:18.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:18.366+0000 7f6d3fe00700 1 -- 172.21.15.67:0/2921054664 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d3400c9c0 con 0x7f6d48101520 2024-03-28T15:53:18.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:18.366+0000 7f6d4ce00700 1 -- 172.21.15.67:0/2921054664 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d481a00c0 con 0x7f6d48101520 2024-03-28T15:53:18.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:18.366+0000 7f6d3fe00700 1 -- 172.21.15.67:0/2921054664 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6d3400cb60 con 0x7f6d48101520 2024-03-28T15:53:18.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:18.366+0000 7f6d3fe00700 1 -- 172.21.15.67:0/2921054664 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d34018680 con 0x7f6d48101520 2024-03-28T15:53:18.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:18.366+0000 7f6d4ce00700 1 -- 172.21.15.67:0/2921054664 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d481a0650 con 0x7f6d48101520 2024-03-28T15:53:18.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:18.367+0000 7f6d4ce00700 1 -- 172.21.15.67:0/2921054664 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d48068e50 con 0x7f6d48101520 2024-03-28T15:53:18.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:18.367+0000 7f6d3fe00700 1 -- 172.21.15.67:0/2921054664 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f6d34010430 con 0x7f6d48101520 2024-03-28T15:53:18.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:18.368+0000 7f6d3fe00700 1 --2- 172.21.15.67:0/2921054664 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f6d2c039200 0x7f6d2c03b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:18.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:18.368+0000 7f6d3fe00700 1 -- 172.21.15.67:0/2921054664 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f6d3404c320 con 0x7f6d48101520 2024-03-28T15:53:18.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:18.368+0000 7f6d47400700 1 --2- 172.21.15.67:0/2921054664 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f6d2c039200 0x7f6d2c03b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:18.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:18.369+0000 7f6d47400700 1 --2- 172.21.15.67:0/2921054664 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f6d2c039200 0x7f6d2c03b6c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6d380105c0 tx=0x7f6d3800b4a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:53:18.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:18.378+0000 7f6d3fe00700 1 -- 172.21.15.67:0/2921054664 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6d34010720 con 0x7f6d48101520 2024-03-28T15:53:18.552 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Mar 28 15:53:18 smithi150 ceph-mgr[36211]: -- 172.21.15.150:0/2268910623 <== mon.2 v2:172.21.15.150:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5608c364a5a0 con 0x5608c361f000 2024-03-28T15:53:18.552 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Mar 28 15:53:18 smithi150 bash[36192]: d5b1a132ba2eba3a4b5e425b8b3f8e5cb60775039278a57d6d87af6f6e22667f 2024-03-28T15:53:18.553 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Mar 28 15:53:18 smithi150 systemd[1]: Started Ceph mgr.y for b5f8225a-ed1a-11ee-95d0-87774f69a715. 2024-03-28T15:53:18.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:18.628+0000 7f6d4ce00700 1 -- 172.21.15.67:0/2921054664 --> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f6d480fda60 con 0x7f6d2c039200 2024-03-28T15:53:19.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:18 smithi067 ceph-mon[31707]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:19.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:18 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:19.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:18 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:19.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:18 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:19.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:18 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:19.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:18 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:53:19.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:18 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:53:19.071 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:18 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:53:19.072 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:18 smithi150 ceph-mon[34825]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:19.072 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:18 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:19.072 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:18 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:19.072 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:18 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:19.072 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:18 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:19.072 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:18 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:53:19.072 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:18 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:53:19.072 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:18 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:53:19.073 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Mar 28 15:53:18 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-y[36207]: 2024-03-28T15:53:18.946+0000 7f9ffb063200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T15:53:19.073 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Mar 28 15:53:19 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-y[36207]: 2024-03-28T15:53:19.070+0000 7f9ffb063200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T15:53:19.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:18 smithi183 ceph-mon[35580]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:19.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:18 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:19.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:18 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:19.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:18 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:19.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:18 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:19.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:18 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:53:19.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:18 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:53:19.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:18 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:53:19.101 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 28 15:53:18 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-z[37457]: 2024-03-28T15:53:18.680+0000 7f67cea1f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T15:53:19.101 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 28 15:53:18 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-z[37457]: 2024-03-28T15:53:18.857+0000 7f67cea1f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T15:53:19.600 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 28 15:53:19 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-z[37457]: 2024-03-28T15:53:19.291+0000 7f67cea1f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T15:53:19.600 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 28 15:53:19 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-z[37457]: 2024-03-28T15:53:19.414+0000 7f67cea1f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T15:53:19.905 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:19 smithi067 ceph-mon[31707]: from='client.34115 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:53:19.905 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:19 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:53:19.905 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:19 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:19.935 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 28 15:53:19 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-z[37457]: 2024-03-28T15:53:19.684+0000 7f67cea1f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T15:53:19.935 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:19 smithi183 ceph-mon[35580]: from='client.34115 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:53:19.935 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:19 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:53:19.935 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:19 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:20.160 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:19 smithi150 ceph-mon[34825]: from='client.34115 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:53:20.160 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:19 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:53:20.160 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:19 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:20.161 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Mar 28 15:53:19 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-y[36207]: 2024-03-28T15:53:19.914+0000 7f9ffb063200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T15:53:20.161 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Mar 28 15:53:20 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-y[36207]: 2024-03-28T15:53:20.039+0000 7f9ffb063200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T15:53:20.195 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 28 15:53:19 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-z[37457]: 2024-03-28T15:53:19.934+0000 7f67cea1f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T15:53:20.463 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Mar 28 15:53:20 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-y[36207]: 2024-03-28T15:53:20.224+0000 7f9ffb063200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T15:53:20.463 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Mar 28 15:53:20 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-y[36207]: 2024-03-28T15:53:20.350+0000 7f9ffb063200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T15:53:20.463 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Mar 28 15:53:20 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-y[36207]: 2024-03-28T15:53:20.461+0000 7f9ffb063200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T15:53:20.600 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 28 15:53:20 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-z[37457]: 2024-03-28T15:53:20.330+0000 7f67cea1f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T15:53:20.971 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 28 15:53:20 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-z[37457]: 2024-03-28T15:53:20.671+0000 7f67cea1f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T15:53:20.971 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:20 smithi183 ceph-mon[35580]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:20.971 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:20 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:20.971 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:20 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:20.971 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:20 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:20.971 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:20 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:20.972 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:20 smithi067 ceph-mon[31707]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:20.972 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:20 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:20.972 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:20 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:20.972 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:20 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:20.972 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:20 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:21.102 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:20 smithi150 ceph-mon[34825]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:21.103 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:20 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:21.103 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:20 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:21.103 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:20 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:21.103 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:20 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:21.103 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Mar 28 15:53:20 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-y[36207]: 2024-03-28T15:53:20.824+0000 7f9ffb063200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T15:53:21.103 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Mar 28 15:53:20 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-y[36207]: 2024-03-28T15:53:20.980+0000 7f9ffb063200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T15:53:21.350 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 28 15:53:20 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-z[37457]: 2024-03-28T15:53:20.969+0000 7f67cea1f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T15:53:21.457 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Mar 28 15:53:21 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-y[36207]: 2024-03-28T15:53:21.100+0000 7f9ffb063200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T15:53:21.457 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Mar 28 15:53:21 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-y[36207]: 2024-03-28T15:53:21.228+0000 7f9ffb063200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T15:53:21.802 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Mar 28 15:53:21 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-y[36207]: 2024-03-28T15:53:21.455+0000 7f9ffb063200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T15:53:22.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:21 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:53:22.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:21 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:53:22.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:21 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:22.101 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 28 15:53:21 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-z[37457]: 2024-03-28T15:53:21.985+0000 7f67cea1f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T15:53:22.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:21 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:53:22.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:21 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:53:22.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:21 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:22.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:21 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:53:22.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:21 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:53:22.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:21 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:22.795 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 28 15:53:22 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-z[37457]: 2024-03-28T15:53:22.710+0000 7f67cea1f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T15:53:23.052 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Mar 28 15:53:22 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-y[36207]: 2024-03-28T15:53:22.790+0000 7f9ffb063200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T15:53:23.052 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Mar 28 15:53:22 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-y[36207]: 2024-03-28T15:53:22.960+0000 7f9ffb063200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T15:53:23.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:22 smithi150 ceph-mon[34825]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:23.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:22 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T15:53:23.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:22 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T15:53:23.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:22 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:53:23.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:22 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/3492420308' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7c1791b6-9c0e-4512-a327-cad166bb5581"}]: dispatch 2024-03-28T15:53:23.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:22 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/3492420308' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7c1791b6-9c0e-4512-a327-cad166bb5581"}]': finished 2024-03-28T15:53:23.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:22 smithi150 ceph-mon[34825]: osdmap e5: 1 total, 0 up, 1 in 2024-03-28T15:53:23.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:22 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T15:53:23.100 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 28 15:53:22 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-z[37457]: 2024-03-28T15:53:22.845+0000 7f67cea1f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T15:53:23.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:22 smithi183 ceph-mon[35580]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:23.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:22 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T15:53:23.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:22 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T15:53:23.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:22 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:53:23.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:22 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/3492420308' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7c1791b6-9c0e-4512-a327-cad166bb5581"}]: dispatch 2024-03-28T15:53:23.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:22 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/3492420308' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7c1791b6-9c0e-4512-a327-cad166bb5581"}]': finished 2024-03-28T15:53:23.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:22 smithi183 ceph-mon[35580]: osdmap e5: 1 total, 0 up, 1 in 2024-03-28T15:53:23.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:22 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T15:53:23.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:22 smithi067 ceph-mon[31707]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:23.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:22 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T15:53:23.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:22 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T15:53:23.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:22 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:53:23.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:22 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/3492420308' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7c1791b6-9c0e-4512-a327-cad166bb5581"}]: dispatch 2024-03-28T15:53:23.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:22 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/3492420308' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7c1791b6-9c0e-4512-a327-cad166bb5581"}]': finished 2024-03-28T15:53:23.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:22 smithi067 ceph-mon[31707]: osdmap e5: 1 total, 0 up, 1 in 2024-03-28T15:53:23.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:22 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T15:53:23.600 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 28 15:53:23 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-z[37457]: 2024-03-28T15:53:23.159+0000 7f67cea1f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T15:53:23.750 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Mar 28 15:53:23 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-y[36207]: 2024-03-28T15:53:23.376+0000 7f9ffb063200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T15:53:23.750 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Mar 28 15:53:23 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-y[36207]: 2024-03-28T15:53:23.494+0000 7f9ffb063200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T15:53:23.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:23.922+0000 7f6d3fe00700 1 -- 172.21.15.67:0/2921054664 <== mon.2 v2:172.21.15.150:3300/0 7 ==== mgrmap(e 14) v1 ==== 91698+0+0 (secure 0 0 0) 0x7f6d34014230 con 0x7f6d48101520 2024-03-28T15:53:24.055 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Mar 28 15:53:23 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-y[36207]: 2024-03-28T15:53:23.748+0000 7f9ffb063200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T15:53:24.055 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Mar 28 15:53:23 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-y[36207]: 2024-03-28T15:53:23.992+0000 7f9ffb063200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T15:53:24.055 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:23 smithi150 ceph-mon[34825]: Reconfiguring mgr.x (unknown last config time)... 2024-03-28T15:53:24.055 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:23 smithi150 ceph-mon[34825]: Reconfiguring daemon mgr.x on smithi067 2024-03-28T15:53:24.055 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:23 smithi150 ceph-mon[34825]: Standby manager daemon z started 2024-03-28T15:53:24.056 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:23 smithi150 ceph-mon[34825]: from='mgr.? 172.21.15.183:0/2457178131' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-28T15:53:24.056 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:23 smithi150 ceph-mon[34825]: from='mgr.? 172.21.15.183:0/2457178131' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T15:53:24.056 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:23 smithi150 ceph-mon[34825]: from='mgr.? 172.21.15.183:0/2457178131' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-28T15:53:24.056 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:23 smithi150 ceph-mon[34825]: from='mgr.? 172.21.15.183:0/2457178131' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T15:53:24.056 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:23 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/3821548371' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:53:24.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:23 smithi183 ceph-mon[35580]: Reconfiguring mgr.x (unknown last config time)... 2024-03-28T15:53:24.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:23 smithi183 ceph-mon[35580]: Reconfiguring daemon mgr.x on smithi067 2024-03-28T15:53:24.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:23 smithi183 ceph-mon[35580]: Standby manager daemon z started 2024-03-28T15:53:24.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:23 smithi183 ceph-mon[35580]: from='mgr.? 172.21.15.183:0/2457178131' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-28T15:53:24.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:23 smithi183 ceph-mon[35580]: from='mgr.? 172.21.15.183:0/2457178131' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T15:53:24.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:23 smithi183 ceph-mon[35580]: from='mgr.? 172.21.15.183:0/2457178131' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-28T15:53:24.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:23 smithi183 ceph-mon[35580]: from='mgr.? 172.21.15.183:0/2457178131' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T15:53:24.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:23 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/3821548371' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:53:24.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:23 smithi067 ceph-mon[31707]: Reconfiguring mgr.x (unknown last config time)... 2024-03-28T15:53:24.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:23 smithi067 ceph-mon[31707]: Reconfiguring daemon mgr.x on smithi067 2024-03-28T15:53:24.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:23 smithi067 ceph-mon[31707]: Standby manager daemon z started 2024-03-28T15:53:24.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:23 smithi067 ceph-mon[31707]: from='mgr.? 172.21.15.183:0/2457178131' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-28T15:53:24.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:23 smithi067 ceph-mon[31707]: from='mgr.? 172.21.15.183:0/2457178131' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T15:53:24.321 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:23 smithi067 ceph-mon[31707]: from='mgr.? 172.21.15.183:0/2457178131' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-28T15:53:24.321 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:23 smithi067 ceph-mon[31707]: from='mgr.? 172.21.15.183:0/2457178131' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T15:53:24.321 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:23 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/3821548371' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:53:24.679 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Mar 28 15:53:24 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-y[36207]: 2024-03-28T15:53:24.360+0000 7f9ffb063200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T15:53:24.980 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Mar 28 15:53:24 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-y[36207]: 2024-03-28T15:53:24.677+0000 7f9ffb063200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T15:53:24.980 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:24 smithi150 ceph-mon[34825]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:24.980 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:24 smithi150 ceph-mon[34825]: mgrmap e14: x(active, since 105s), standbys: z 2024-03-28T15:53:24.980 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:24 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-28T15:53:25.302 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Mar 28 15:53:24 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-y[36207]: 2024-03-28T15:53:24.978+0000 7f9ffb063200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T15:53:25.340 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:24 smithi067 ceph-mon[31707]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:25.340 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:24 smithi067 ceph-mon[31707]: mgrmap e14: x(active, since 105s), standbys: z 2024-03-28T15:53:25.340 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:24 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-28T15:53:25.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:24 smithi183 ceph-mon[35580]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:25.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:24 smithi183 ceph-mon[35580]: mgrmap e14: x(active, since 105s), standbys: z 2024-03-28T15:53:25.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:24 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-28T15:53:26.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:25 smithi150 ceph-mon[34825]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:26.307 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Mar 28 15:53:25 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-y[36207]: 2024-03-28T15:53:25.991+0000 7f9ffb063200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T15:53:26.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:25 smithi067 ceph-mon[31707]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:26.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:25 smithi183 ceph-mon[35580]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:27.052 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Mar 28 15:53:26 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-y[36207]: 2024-03-28T15:53:26.684+0000 7f9ffb063200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T15:53:27.052 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Mar 28 15:53:26 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-y[36207]: 2024-03-28T15:53:26.813+0000 7f9ffb063200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T15:53:27.552 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Mar 28 15:53:27 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-y[36207]: 2024-03-28T15:53:27.090+0000 7f9ffb063200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T15:53:27.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:27.994+0000 7f6d3fe00700 1 -- 172.21.15.67:0/2921054664 <== mon.2 v2:172.21.15.150:3300/0 8 ==== mgrmap(e 15) v1 ==== 137234+0+0 (secure 0 0 0) 0x7f6d3401d210 con 0x7f6d48101520 2024-03-28T15:53:28.233 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:27 smithi067 ceph-mon[31707]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:28.233 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:27 smithi067 ceph-mon[31707]: Standby manager daemon y started 2024-03-28T15:53:28.233 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:27 smithi067 ceph-mon[31707]: from='mgr.? 172.21.15.150:0/3348625401' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-28T15:53:28.233 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:27 smithi067 ceph-mon[31707]: from='mgr.? 172.21.15.150:0/3348625401' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T15:53:28.233 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:27 smithi067 ceph-mon[31707]: from='mgr.? 172.21.15.150:0/3348625401' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-28T15:53:28.233 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:27 smithi067 ceph-mon[31707]: from='mgr.? 172.21.15.150:0/3348625401' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T15:53:28.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:27 smithi150 ceph-mon[34825]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:28.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:27 smithi150 ceph-mon[34825]: Standby manager daemon y started 2024-03-28T15:53:28.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:27 smithi150 ceph-mon[34825]: from='mgr.? 172.21.15.150:0/3348625401' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-28T15:53:28.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:27 smithi150 ceph-mon[34825]: from='mgr.? 172.21.15.150:0/3348625401' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T15:53:28.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:27 smithi150 ceph-mon[34825]: from='mgr.? 172.21.15.150:0/3348625401' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-28T15:53:28.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:27 smithi150 ceph-mon[34825]: from='mgr.? 172.21.15.150:0/3348625401' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T15:53:28.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:27 smithi183 ceph-mon[35580]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:28.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:27 smithi183 ceph-mon[35580]: Standby manager daemon y started 2024-03-28T15:53:28.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:27 smithi183 ceph-mon[35580]: from='mgr.? 172.21.15.150:0/3348625401' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-28T15:53:28.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:27 smithi183 ceph-mon[35580]: from='mgr.? 172.21.15.150:0/3348625401' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T15:53:28.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:27 smithi183 ceph-mon[35580]: from='mgr.? 172.21.15.150:0/3348625401' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-28T15:53:28.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:27 smithi183 ceph-mon[35580]: from='mgr.? 172.21.15.150:0/3348625401' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T15:53:29.304 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:28 smithi150 ceph-mon[34825]: mgrmap e15: x(active, since 109s), standbys: z, y 2024-03-28T15:53:29.305 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:28 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-28T15:53:29.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:28 smithi067 ceph-mon[31707]: mgrmap e15: x(active, since 109s), standbys: z, y 2024-03-28T15:53:29.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:28 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-28T15:53:29.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:28 smithi183 ceph-mon[35580]: mgrmap e15: x(active, since 109s), standbys: z, y 2024-03-28T15:53:29.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:28 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-28T15:53:30.667 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:30 smithi067 ceph-mon[31707]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:30.667 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:30 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:30.667 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:30 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:30.667 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:30 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:53:30.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:30 smithi150 ceph-mon[34825]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:30.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:30 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:30.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:30 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:30.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:30 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:53:30.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:30 smithi183 ceph-mon[35580]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:30.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:30 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:30.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:30 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:30.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:30 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:53:31.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:31 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:53:31.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:31 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:53:31.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:31 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:31.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:31 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:53:31.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:31 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:53:31.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:31 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:31.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:31 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:53:31.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:31 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:53:31.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:31 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:32.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:32 smithi067 ceph-mon[31707]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:32.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:32 smithi183 ceph-mon[35580]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:33.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:32 smithi150 ceph-mon[34825]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:33.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:33 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-28T15:53:33.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:33 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:53:33.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:33 smithi067 ceph-mon[31707]: Deploying daemon osd.0 on smithi067 2024-03-28T15:53:33.843 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:33 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-28T15:53:33.843 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:33 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:53:33.843 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:33 smithi183 ceph-mon[35580]: Deploying daemon osd.0 on smithi067 2024-03-28T15:53:34.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:33 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-28T15:53:34.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:33 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:53:34.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:33 smithi150 ceph-mon[34825]: Deploying daemon osd.0 on smithi067 2024-03-28T15:53:34.757 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:34 smithi067 ceph-mon[31707]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:34.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:34 smithi183 ceph-mon[35580]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:35.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:34 smithi150 ceph-mon[34825]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:36.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:36 smithi183 ceph-mon[35580]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:37.017 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:36 smithi067 ceph-mon[31707]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:37.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:36 smithi150 ceph-mon[34825]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:38.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:38 smithi183 ceph-mon[35580]: pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:39.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:38 smithi150 ceph-mon[34825]: pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:39.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:38 smithi067 ceph-mon[31707]: pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:40.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:40 smithi067 ceph-mon[31707]: pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:40.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:40 smithi183 ceph-mon[35580]: pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:41.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:40 smithi150 ceph-mon[34825]: pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:42.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:41 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:42.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:41 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:42.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:41 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:53:42.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:41 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:42.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:41 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:42.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:41 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:53:42.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:41 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:42.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:41 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:42.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:41 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:53:43.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:42 smithi067 ceph-mon[31707]: pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:43.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:42 smithi150 ceph-mon[34825]: pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:43.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:42 smithi183 ceph-mon[35580]: pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:45.003 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:44 smithi067 ceph-mon[31707]: pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:45.003 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:44 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:45.003 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:44 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:45.003 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:44 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:53:45.004 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:44 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:53:45.004 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:44 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:45.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:44 smithi150 ceph-mon[34825]: pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:45.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:44 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:45.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:44 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:45.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:44 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:53:45.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:44 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:53:45.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:44 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:45.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:44 smithi183 ceph-mon[35580]: pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:45.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:44 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:45.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:44 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:45.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:44 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:53:45.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:44 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:53:45.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:44 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:45.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:45.541+0000 7f6d3fe00700 1 -- 172.21.15.67:0/2921054664 <== mgr.14152 v2:172.21.15.67:6800/208726654 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f6d480fda60 con 0x7f6d2c039200 2024-03-28T15:53:45.542 INFO:teuthology.orchestra.run.smithi067.stdout:Created osd(s) 0 on host 'smithi067' 2024-03-28T15:53:45.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:45.546+0000 7f6d4ce00700 1 -- 172.21.15.67:0/2921054664 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f6d2c039200 msgr2=0x7f6d2c03b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:53:45.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:45.546+0000 7f6d4ce00700 1 --2- 172.21.15.67:0/2921054664 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f6d2c039200 0x7f6d2c03b6c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6d380105c0 tx=0x7f6d3800b4a0 comp rx=0 tx=0).stop 2024-03-28T15:53:45.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:45.546+0000 7f6d4ce00700 1 -- 172.21.15.67:0/2921054664 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d48101520 msgr2=0x7f6d481986b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:53:45.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:45.546+0000 7f6d4ce00700 1 --2- 172.21.15.67:0/2921054664 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d48101520 0x7f6d481986b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6d3400e9f0 tx=0x7f6d3400ed00 comp rx=0 tx=0).stop 2024-03-28T15:53:45.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:45.547+0000 7f6d4ce00700 1 -- 172.21.15.67:0/2921054664 shutdown_connections 2024-03-28T15:53:45.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:45.547+0000 7f6d4ce00700 1 --2- 172.21.15.67:0/2921054664 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f6d2c039200 0x7f6d2c03b6c0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:45.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:45.547+0000 7f6d4ce00700 1 --2- 172.21.15.67:0/2921054664 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6d48101ec0 0x7f6d4819f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:45.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:45.547+0000 7f6d4ce00700 1 --2- 172.21.15.67:0/2921054664 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d48100320 0x7f6d48198170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:45.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:45.547+0000 7f6d4ce00700 1 --2- 172.21.15.67:0/2921054664 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d48101520 0x7f6d481986b0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:45.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:45.547+0000 7f6d4ce00700 1 -- 172.21.15.67:0/2921054664 >> 172.21.15.67:0/2921054664 conn(0x7f6d480fb900 msgr2=0x7f6d480fd350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:53:45.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:45.547+0000 7f6d4ce00700 1 -- 172.21.15.67:0/2921054664 shutdown_connections 2024-03-28T15:53:45.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:53:45.547+0000 7f6d4ce00700 1 -- 172.21.15.67:0/2921054664 wait complete. 2024-03-28T15:53:45.685 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:45 smithi067 ceph-mon[31707]: from='osd.0 [v2:172.21.15.67:6802/1894681701,v1:172.21.15.67:6803/1894681701]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-28T15:53:45.685 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:45 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:45.685 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:45 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:45.685 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:45 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:53:46.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:45 smithi150 ceph-mon[34825]: from='osd.0 [v2:172.21.15.67:6802/1894681701,v1:172.21.15.67:6803/1894681701]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-28T15:53:46.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:45 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:46.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:45 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:46.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:45 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:53:46.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:45 smithi183 ceph-mon[35580]: from='osd.0 [v2:172.21.15.67:6802/1894681701,v1:172.21.15.67:6803/1894681701]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-28T15:53:46.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:45 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:46.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:45 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:46.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:45 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:53:46.315 DEBUG:teuthology.orchestra.run.smithi067:osd.0> sudo journalctl -f -n 0 -u ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@osd.0.service 2024-03-28T15:53:46.319 INFO:tasks.cephadm:Deploying osd.1 on smithi150 with /dev/vg_nvme/lv_4... 2024-03-28T15:53:46.319 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-28T15:53:46.777 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T15:53:46.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:46 smithi067 ceph-mon[31707]: pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:46.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:46 smithi067 ceph-mon[31707]: from='osd.0 [v2:172.21.15.67:6802/1894681701,v1:172.21.15.67:6803/1894681701]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-28T15:53:46.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:46 smithi067 ceph-mon[31707]: osdmap e6: 1 total, 0 up, 1 in 2024-03-28T15:53:46.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:46 smithi067 ceph-mon[31707]: from='osd.0 [v2:172.21.15.67:6802/1894681701,v1:172.21.15.67:6803/1894681701]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-03-28T15:53:46.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:46 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T15:53:46.937 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:46 smithi150 ceph-mon[34825]: pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:46.937 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:46 smithi150 ceph-mon[34825]: from='osd.0 [v2:172.21.15.67:6802/1894681701,v1:172.21.15.67:6803/1894681701]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-28T15:53:46.938 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:46 smithi150 ceph-mon[34825]: osdmap e6: 1 total, 0 up, 1 in 2024-03-28T15:53:46.938 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:46 smithi150 ceph-mon[34825]: from='osd.0 [v2:172.21.15.67:6802/1894681701,v1:172.21.15.67:6803/1894681701]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-03-28T15:53:46.938 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:46 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T15:53:47.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:46 smithi183 ceph-mon[35580]: pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:47.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:46 smithi183 ceph-mon[35580]: from='osd.0 [v2:172.21.15.67:6802/1894681701,v1:172.21.15.67:6803/1894681701]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-28T15:53:47.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:46 smithi183 ceph-mon[35580]: osdmap e6: 1 total, 0 up, 1 in 2024-03-28T15:53:47.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:46 smithi183 ceph-mon[35580]: from='osd.0 [v2:172.21.15.67:6802/1894681701,v1:172.21.15.67:6803/1894681701]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-03-28T15:53:47.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:46 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T15:53:47.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:47 smithi067 ceph-mon[31707]: from='osd.0 [v2:172.21.15.67:6802/1894681701,v1:172.21.15.67:6803/1894681701]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-03-28T15:53:47.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:47 smithi067 ceph-mon[31707]: osdmap e7: 1 total, 0 up, 1 in 2024-03-28T15:53:47.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:47 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T15:53:47.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:47 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T15:53:47.947 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:47 smithi150 ceph-mon[34825]: from='osd.0 [v2:172.21.15.67:6802/1894681701,v1:172.21.15.67:6803/1894681701]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-03-28T15:53:47.947 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:47 smithi150 ceph-mon[34825]: osdmap e7: 1 total, 0 up, 1 in 2024-03-28T15:53:47.947 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:47 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T15:53:47.947 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:47 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T15:53:48.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:47 smithi183 ceph-mon[35580]: from='osd.0 [v2:172.21.15.67:6802/1894681701,v1:172.21.15.67:6803/1894681701]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-03-28T15:53:48.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:47 smithi183 ceph-mon[35580]: osdmap e7: 1 total, 0 up, 1 in 2024-03-28T15:53:48.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:47 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T15:53:48.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:47 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T15:53:48.189 INFO:journalctl@ceph.osd.0.smithi067.stdout:Mar 28 15:53:47 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-0[46432]: 2024-03-28T15:53:47.873+0000 7fd057e00700 -1 osd.0 0 waiting for initial osdmap 2024-03-28T15:53:48.190 INFO:journalctl@ceph.osd.0.smithi067.stdout:Mar 28 15:53:47 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-0[46432]: 2024-03-28T15:53:47.903+0000 7fd04dc00700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T15:53:49.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:48 smithi150 ceph-mon[34825]: purged_snaps scrub starts 2024-03-28T15:53:49.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:48 smithi150 ceph-mon[34825]: purged_snaps scrub ok 2024-03-28T15:53:49.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:48 smithi150 ceph-mon[34825]: pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:49.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:48 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T15:53:49.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:48 smithi150 ceph-mon[34825]: from='osd.0 [v2:172.21.15.67:6802/1894681701,v1:172.21.15.67:6803/1894681701]' entity='osd.0' 2024-03-28T15:53:49.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:48 smithi067 ceph-mon[31707]: purged_snaps scrub starts 2024-03-28T15:53:49.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:48 smithi067 ceph-mon[31707]: purged_snaps scrub ok 2024-03-28T15:53:49.071 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:48 smithi067 ceph-mon[31707]: pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:49.071 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:48 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T15:53:49.071 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:48 smithi067 ceph-mon[31707]: from='osd.0 [v2:172.21.15.67:6802/1894681701,v1:172.21.15.67:6803/1894681701]' entity='osd.0' 2024-03-28T15:53:49.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:48 smithi183 ceph-mon[35580]: purged_snaps scrub starts 2024-03-28T15:53:49.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:48 smithi183 ceph-mon[35580]: purged_snaps scrub ok 2024-03-28T15:53:49.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:48 smithi183 ceph-mon[35580]: pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:49.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:48 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T15:53:49.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:48 smithi183 ceph-mon[35580]: from='osd.0 [v2:172.21.15.67:6802/1894681701,v1:172.21.15.67:6803/1894681701]' entity='osd.0' 2024-03-28T15:53:49.421 INFO:teuthology.orchestra.run.smithi150.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-28T15:53:49.421 INFO:teuthology.orchestra.run.smithi150.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-28T15:53:49.421 INFO:teuthology.orchestra.run.smithi150.stderr: stderr: 10+0 records in 2024-03-28T15:53:49.421 INFO:teuthology.orchestra.run.smithi150.stderr:10+0 records out 2024-03-28T15:53:49.421 INFO:teuthology.orchestra.run.smithi150.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0340334 s, 308 MB/s 2024-03-28T15:53:49.421 INFO:teuthology.orchestra.run.smithi150.stderr:--> Zapping successful for: 2024-03-28T15:53:49.951 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:49 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T15:53:49.951 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:49 smithi067 ceph-mon[31707]: osd.0 [v2:172.21.15.67:6802/1894681701,v1:172.21.15.67:6803/1894681701] boot 2024-03-28T15:53:49.951 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:49 smithi067 ceph-mon[31707]: osdmap e8: 1 total, 1 up, 1 in 2024-03-28T15:53:49.951 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:49 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T15:53:50.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:49 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T15:53:50.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:49 smithi150 ceph-mon[34825]: osd.0 [v2:172.21.15.67:6802/1894681701,v1:172.21.15.67:6803/1894681701] boot 2024-03-28T15:53:50.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:49 smithi150 ceph-mon[34825]: osdmap e8: 1 total, 1 up, 1 in 2024-03-28T15:53:50.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:49 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T15:53:50.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:49 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T15:53:50.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:49 smithi183 ceph-mon[35580]: osd.0 [v2:172.21.15.67:6802/1894681701,v1:172.21.15.67:6803/1894681701] boot 2024-03-28T15:53:50.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:49 smithi183 ceph-mon[35580]: osdmap e8: 1 total, 1 up, 1 in 2024-03-28T15:53:50.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:49 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T15:53:50.259 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi150:vg_nvme/lv_4 2024-03-28T15:53:50.678 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T15:53:50.978 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:50 smithi150 ceph-mon[34825]: pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:51.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:50 smithi067 ceph-mon[31707]: pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:51.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:50 smithi183 ceph-mon[35580]: pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:53:51.939 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:51 smithi150 ceph-mon[34825]: osdmap e9: 1 total, 1 up, 1 in 2024-03-28T15:53:51.939 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:51 smithi150 ceph-mon[34825]: Detected new or changed devices on smithi067 2024-03-28T15:53:51.939 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:51 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:51.939 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:51 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:51.939 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:51 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:51.939 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:51 smithi150 ceph-mon[34825]: Adjusting osd_memory_target on smithi067 to 17091M 2024-03-28T15:53:51.939 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:51 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:51.940 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:51 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:53:51.940 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:51 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:53:51.940 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:51 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:52.002 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:51 smithi183 ceph-mon[35580]: osdmap e9: 1 total, 1 up, 1 in 2024-03-28T15:53:52.002 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:51 smithi183 ceph-mon[35580]: Detected new or changed devices on smithi067 2024-03-28T15:53:52.002 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:51 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:52.002 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:51 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:52.002 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:51 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:52.002 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:51 smithi183 ceph-mon[35580]: Adjusting osd_memory_target on smithi067 to 17091M 2024-03-28T15:53:52.003 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:51 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:52.003 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:51 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:53:52.003 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:51 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:53:52.003 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:51 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:52.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:51 smithi067 ceph-mon[31707]: osdmap e9: 1 total, 1 up, 1 in 2024-03-28T15:53:52.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:51 smithi067 ceph-mon[31707]: Detected new or changed devices on smithi067 2024-03-28T15:53:52.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:51 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:52.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:51 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:52.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:51 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:52.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:51 smithi067 ceph-mon[31707]: Adjusting osd_memory_target on smithi067 to 17091M 2024-03-28T15:53:52.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:51 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:52.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:51 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:53:52.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:51 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:53:52.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:51 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:53:52.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:53:52.669+0000 7fb9fe600700 1 -- 172.21.15.150:0/1853112274 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9f80761c0 msgr2=0x7fb9f8074620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:53:52.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:53:52.669+0000 7fb9fe600700 1 --2- 172.21.15.150:0/1853112274 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9f80761c0 0x7fb9f8074620 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb9e800b0a0 tx=0x7fb9e800b3b0 comp rx=0 tx=0).stop 2024-03-28T15:53:52.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:53:52.669+0000 7fb9fe600700 1 -- 172.21.15.150:0/1853112274 shutdown_connections 2024-03-28T15:53:52.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:53:52.669+0000 7fb9fe600700 1 --2- 172.21.15.150:0/1853112274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9f810a0a0 0x7fb9f810c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:52.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:53:52.669+0000 7fb9fe600700 1 --2- 172.21.15.150:0/1853112274 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9f80761c0 0x7fb9f8074620 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:52.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:53:52.669+0000 7fb9fe600700 1 --2- 172.21.15.150:0/1853112274 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb9f8074bf0 0x7fb9f8075070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:52.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:53:52.669+0000 7fb9fe600700 1 -- 172.21.15.150:0/1853112274 >> 172.21.15.150:0/1853112274 conn(0x7fb9f80fd730 msgr2=0x7fb9f80ffbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:53:52.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:53:52.669+0000 7fb9fe600700 1 -- 172.21.15.150:0/1853112274 shutdown_connections 2024-03-28T15:53:52.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:53:52.669+0000 7fb9fe600700 1 -- 172.21.15.150:0/1853112274 wait complete. 2024-03-28T15:53:52.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:53:52.671+0000 7fb9fe600700 1 Processor -- start 2024-03-28T15:53:52.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:53:52.671+0000 7fb9fe600700 1 -- start start 2024-03-28T15:53:52.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:53:52.671+0000 7fb9fe600700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9f8074bf0 0x7fb9f819a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:52.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:53:52.671+0000 7fb9fe600700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9f80761c0 0x7fb9f819ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:52.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:53:52.671+0000 7fb9fe600700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb9f810a0a0 0x7fb9f81a1be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:52.674 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:53:52.671+0000 7fb9fe600700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f810ff30 con 0x7fb9f8074bf0 2024-03-28T15:53:52.674 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:53:52.671+0000 7fb9fe600700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f810fdb0 con 0x7fb9f80761c0 2024-03-28T15:53:52.674 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:53:52.671+0000 7fb9fe600700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f81100b0 con 0x7fb9f810a0a0 2024-03-28T15:53:52.674 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:53:52.672+0000 7fb9f7400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9f8074bf0 0x7fb9f819a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:52.674 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:53:52.672+0000 7fb9f7400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9f8074bf0 0x7fb9f819a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.150:45032/0 (socket says 172.21.15.150:45032) 2024-03-28T15:53:52.674 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:53:52.672+0000 7fb9f7400700 1 -- 172.21.15.150:0/747603598 learned_addr learned my addr 172.21.15.150:0/747603598 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-28T15:53:52.674 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:53:52.672+0000 7fb9f7e00700 1 --2- 172.21.15.150:0/747603598 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb9f810a0a0 0x7fb9f81a1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:52.674 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:53:52.672+0000 7fb9eea00700 1 --2- 172.21.15.150:0/747603598 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9f80761c0 0x7fb9f819ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:52.674 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:53:52.672+0000 7fb9eea00700 1 -- 172.21.15.150:0/747603598 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb9f810a0a0 msgr2=0x7fb9f81a1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:53:52.674 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:53:52.672+0000 7fb9eea00700 1 --2- 172.21.15.150:0/747603598 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb9f810a0a0 0x7fb9f81a1be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:52.674 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:53:52.672+0000 7fb9eea00700 1 -- 172.21.15.150:0/747603598 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9f8074bf0 msgr2=0x7fb9f819a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:53:52.674 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:53:52.672+0000 7fb9eea00700 1 --2- 172.21.15.150:0/747603598 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9f8074bf0 0x7fb9f819a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:52.674 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:53:52.672+0000 7fb9eea00700 1 -- 172.21.15.150:0/747603598 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9f81a2280 con 0x7fb9f80761c0 2024-03-28T15:53:52.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:53:52.673+0000 7fb9eea00700 1 --2- 172.21.15.150:0/747603598 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9f80761c0 0x7fb9f819ab60 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb9f8075ef0 tx=0x7fb9e000b730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:53:52.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:53:52.673+0000 7fb9f4c00700 1 -- 172.21.15.150:0/747603598 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9e0015020 con 0x7fb9f80761c0 2024-03-28T15:53:52.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:53:52.673+0000 7fb9f4c00700 1 -- 172.21.15.150:0/747603598 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb9e000be30 con 0x7fb9f80761c0 2024-03-28T15:53:52.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:53:52.673+0000 7fb9fe600700 1 -- 172.21.15.150:0/747603598 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9f81a2570 con 0x7fb9f80761c0 2024-03-28T15:53:52.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:53:52.673+0000 7fb9f4c00700 1 -- 172.21.15.150:0/747603598 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9e0010be0 con 0x7fb9f80761c0 2024-03-28T15:53:52.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:53:52.673+0000 7fb9fe600700 1 -- 172.21.15.150:0/747603598 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9f81a2b80 con 0x7fb9f80761c0 2024-03-28T15:53:52.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:53:52.674+0000 7fb9fe600700 1 -- 172.21.15.150:0/747603598 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9f8068e50 con 0x7fb9f80761c0 2024-03-28T15:53:52.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:53:52.676+0000 7fb9f4c00700 1 -- 172.21.15.150:0/747603598 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 15) v1 ==== 137234+0+0 (secure 0 0 0) 0x7fb9e0018020 con 0x7fb9f80761c0 2024-03-28T15:53:52.678 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:53:52.678+0000 7fb9f4c00700 1 --2- 172.21.15.150:0/747603598 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb9dc0a2880 0x7fb9dc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:52.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:53:52.679+0000 7fb9f4c00700 1 -- 172.21.15.150:0/747603598 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1767+0+0 (secure 0 0 0) 0x7fb9e00cc570 con 0x7fb9f80761c0 2024-03-28T15:53:52.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:53:52.679+0000 7fb9f7400700 1 --2- 172.21.15.150:0/747603598 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb9dc0a2880 0x7fb9dc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:52.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:53:52.680+0000 7fb9f7400700 1 --2- 172.21.15.150:0/747603598 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb9dc0a2880 0x7fb9dc0a4d40 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb9e8009fd0 tx=0x7fb9e800b7f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:53:52.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:53:52.684+0000 7fb9f4c00700 1 -- 172.21.15.150:0/747603598 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb9e001f020 con 0x7fb9f80761c0 2024-03-28T15:53:52.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:53:52.920+0000 7fb9fe600700 1 -- 172.21.15.150:0/747603598 --> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fb9f80625d0 con 0x7fb9dc0a2880 2024-03-28T15:53:53.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:52 smithi067 ceph-mon[31707]: pgmap v65: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:53:53.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:52 smithi183 ceph-mon[35580]: pgmap v65: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:53:53.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:52 smithi150 ceph-mon[34825]: pgmap v65: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:53:54.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:53 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:53:54.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:53 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:53:54.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:53 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:53:54.245 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:53 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:53:54.246 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:53 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:53:54.246 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:53 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:53:54.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:53 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:53:54.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:53 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:53:54.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:53 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:53:54.922 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:54 smithi150 ceph-mon[34825]: pgmap v66: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:53:54.922 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:54 smithi150 ceph-mon[34825]: from='client.34139 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:53:55.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:54 smithi183 ceph-mon[35580]: pgmap v66: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:53:55.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:54 smithi183 ceph-mon[35580]: from='client.34139 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:53:55.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:54 smithi067 ceph-mon[31707]: pgmap v66: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:53:55.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:54 smithi067 ceph-mon[31707]: from='client.34139 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:53:57.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:56 smithi183 ceph-mon[35580]: pgmap v67: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:53:57.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:56 smithi183 ceph-mon[35580]: from='client.? 172.21.15.150:0/1589393929' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0cdda7f3-d00c-4436-bcc1-49fb8072b2e3"}]: dispatch 2024-03-28T15:53:57.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:56 smithi183 ceph-mon[35580]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0cdda7f3-d00c-4436-bcc1-49fb8072b2e3"}]: dispatch 2024-03-28T15:53:57.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:56 smithi183 ceph-mon[35580]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0cdda7f3-d00c-4436-bcc1-49fb8072b2e3"}]': finished 2024-03-28T15:53:57.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:56 smithi183 ceph-mon[35580]: osdmap e10: 2 total, 1 up, 2 in 2024-03-28T15:53:57.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:56 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T15:53:57.213 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:56 smithi150 ceph-mon[34825]: pgmap v67: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:53:57.213 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:56 smithi150 ceph-mon[34825]: from='client.? 172.21.15.150:0/1589393929' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0cdda7f3-d00c-4436-bcc1-49fb8072b2e3"}]: dispatch 2024-03-28T15:53:57.213 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:56 smithi150 ceph-mon[34825]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0cdda7f3-d00c-4436-bcc1-49fb8072b2e3"}]: dispatch 2024-03-28T15:53:57.213 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:56 smithi150 ceph-mon[34825]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0cdda7f3-d00c-4436-bcc1-49fb8072b2e3"}]': finished 2024-03-28T15:53:57.214 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:56 smithi150 ceph-mon[34825]: osdmap e10: 2 total, 1 up, 2 in 2024-03-28T15:53:57.214 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:56 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T15:53:57.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:56 smithi067 ceph-mon[31707]: pgmap v67: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:53:57.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:56 smithi067 ceph-mon[31707]: from='client.? 172.21.15.150:0/1589393929' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0cdda7f3-d00c-4436-bcc1-49fb8072b2e3"}]: dispatch 2024-03-28T15:53:57.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:56 smithi067 ceph-mon[31707]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0cdda7f3-d00c-4436-bcc1-49fb8072b2e3"}]: dispatch 2024-03-28T15:53:57.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:56 smithi067 ceph-mon[31707]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0cdda7f3-d00c-4436-bcc1-49fb8072b2e3"}]': finished 2024-03-28T15:53:57.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:56 smithi067 ceph-mon[31707]: osdmap e10: 2 total, 1 up, 2 in 2024-03-28T15:53:57.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:56 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T15:53:58.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:57 smithi150 ceph-mon[34825]: from='client.? 172.21.15.150:0/2993356988' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:53:58.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:57 smithi067 ceph-mon[31707]: from='client.? 172.21.15.150:0/2993356988' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:53:58.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:57 smithi183 ceph-mon[35580]: from='client.? 172.21.15.150:0/2993356988' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:53:59.204 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:58 smithi150 ceph-mon[34825]: pgmap v69: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:53:59.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:58 smithi067 ceph-mon[31707]: pgmap v69: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:53:59.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:58 smithi183 ceph-mon[35580]: pgmap v69: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:54:00.305 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:53:59 smithi150 ceph-mon[34825]: pgmap v70: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:54:00.322 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:53:59 smithi067 ceph-mon[31707]: pgmap v70: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:54:00.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:53:59 smithi183 ceph-mon[35580]: pgmap v70: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:54:02.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:01 smithi150 ceph-mon[34825]: pgmap v71: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:54:02.321 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:01 smithi067 ceph-mon[31707]: pgmap v71: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:54:02.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:01 smithi183 ceph-mon[35580]: pgmap v71: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:54:04.291 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:03 smithi150 ceph-mon[34825]: pgmap v72: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:54:04.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:03 smithi067 ceph-mon[31707]: pgmap v72: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:54:04.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:03 smithi183 ceph-mon[35580]: pgmap v72: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:54:06.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:05 smithi150 ceph-mon[34825]: pgmap v73: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:54:06.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:05 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-28T15:54:06.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:05 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:54:06.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:05 smithi150 ceph-mon[34825]: Deploying daemon osd.1 on smithi150 2024-03-28T15:54:06.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:05 smithi067 ceph-mon[31707]: pgmap v73: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:54:06.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:05 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-28T15:54:06.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:05 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:54:06.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:05 smithi067 ceph-mon[31707]: Deploying daemon osd.1 on smithi150 2024-03-28T15:54:06.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:05 smithi183 ceph-mon[35580]: pgmap v73: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:54:06.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:05 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-28T15:54:06.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:05 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:54:06.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:05 smithi183 ceph-mon[35580]: Deploying daemon osd.1 on smithi150 2024-03-28T15:54:08.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:07 smithi150 ceph-mon[34825]: pgmap v74: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:54:08.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:07 smithi067 ceph-mon[31707]: pgmap v74: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:54:08.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:07 smithi183 ceph-mon[35580]: pgmap v74: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:54:10.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:09 smithi150 ceph-mon[34825]: pgmap v75: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:54:10.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:09 smithi067 ceph-mon[31707]: pgmap v75: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:54:10.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:09 smithi183 ceph-mon[35580]: pgmap v75: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:54:12.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:11 smithi150 ceph-mon[34825]: pgmap v76: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:54:12.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:11 smithi067 ceph-mon[31707]: pgmap v76: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:54:12.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:11 smithi183 ceph-mon[35580]: pgmap v76: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:54:14.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:13 smithi150 ceph-mon[34825]: pgmap v77: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:54:14.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:13 smithi067 ceph-mon[31707]: pgmap v77: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:54:14.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:13 smithi183 ceph-mon[35580]: pgmap v77: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:54:15.220 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:14 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:54:15.221 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:14 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:54:15.221 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:14 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:54:15.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:14 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:54:15.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:14 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:54:15.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:14 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:54:15.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:14 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:54:15.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:14 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:54:15.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:14 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:54:16.284 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:16 smithi150 ceph-mon[34825]: pgmap v78: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:54:16.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:16 smithi067 ceph-mon[31707]: pgmap v78: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:54:16.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:16 smithi183 ceph-mon[35580]: pgmap v78: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:54:18.744 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:18 smithi150 ceph-mon[34825]: pgmap v79: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:54:18.744 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:18 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:54:18.744 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:18 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:54:18.744 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:18 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:54:18.744 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:18 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:54:18.744 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:18 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:54:18.744 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:18 smithi150 ceph-mon[34825]: from='osd.1 [v2:172.21.15.150:6800/1734465762,v1:172.21.15.150:6801/1734465762]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-28T15:54:18.744 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:18 smithi150 ceph-mon[34825]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-28T15:54:18.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:18 smithi067 ceph-mon[31707]: pgmap v79: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:54:18.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:18 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:54:18.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:18 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:54:18.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:18 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:54:18.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:18 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:54:18.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:18 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:54:18.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:18 smithi067 ceph-mon[31707]: from='osd.1 [v2:172.21.15.150:6800/1734465762,v1:172.21.15.150:6801/1734465762]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-28T15:54:18.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:18 smithi067 ceph-mon[31707]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-28T15:54:18.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:54:18.837+0000 7fb9f4c00700 1 -- 172.21.15.150:0/747603598 <== mgr.14152 v2:172.21.15.67:6800/208726654 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fb9f80625d0 con 0x7fb9dc0a2880 2024-03-28T15:54:18.838 INFO:teuthology.orchestra.run.smithi150.stdout:Created osd(s) 1 on host 'smithi150' 2024-03-28T15:54:18.841 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:54:18.840+0000 7fb9fe600700 1 -- 172.21.15.150:0/747603598 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb9dc0a2880 msgr2=0x7fb9dc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:54:18.841 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:54:18.840+0000 7fb9fe600700 1 --2- 172.21.15.150:0/747603598 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb9dc0a2880 0x7fb9dc0a4d40 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb9e8009fd0 tx=0x7fb9e800b7f0 comp rx=0 tx=0).stop 2024-03-28T15:54:18.841 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:54:18.840+0000 7fb9fe600700 1 -- 172.21.15.150:0/747603598 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9f80761c0 msgr2=0x7fb9f819ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:54:18.841 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:54:18.840+0000 7fb9fe600700 1 --2- 172.21.15.150:0/747603598 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9f80761c0 0x7fb9f819ab60 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb9f8075ef0 tx=0x7fb9e000b730 comp rx=0 tx=0).stop 2024-03-28T15:54:18.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:54:18.840+0000 7fb9fe600700 1 -- 172.21.15.150:0/747603598 shutdown_connections 2024-03-28T15:54:18.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:54:18.840+0000 7fb9fe600700 1 --2- 172.21.15.150:0/747603598 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb9dc0a2880 0x7fb9dc0a4d40 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:18.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:54:18.840+0000 7fb9fe600700 1 --2- 172.21.15.150:0/747603598 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb9f810a0a0 0x7fb9f81a1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:18.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:54:18.840+0000 7fb9fe600700 1 --2- 172.21.15.150:0/747603598 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9f8074bf0 0x7fb9f819a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:18.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:54:18.840+0000 7fb9fe600700 1 --2- 172.21.15.150:0/747603598 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9f80761c0 0x7fb9f819ab60 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:18.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:54:18.840+0000 7fb9fe600700 1 -- 172.21.15.150:0/747603598 >> 172.21.15.150:0/747603598 conn(0x7fb9f80fd730 msgr2=0x7fb9f8105fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:54:18.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:54:18.842+0000 7fb9fe600700 1 -- 172.21.15.150:0/747603598 shutdown_connections 2024-03-28T15:54:18.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:54:18.842+0000 7fb9fe600700 1 -- 172.21.15.150:0/747603598 wait complete. 2024-03-28T15:54:18.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:18 smithi183 ceph-mon[35580]: pgmap v79: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:54:18.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:18 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:54:18.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:18 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:54:18.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:18 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:54:18.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:18 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:54:18.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:18 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:54:18.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:18 smithi183 ceph-mon[35580]: from='osd.1 [v2:172.21.15.150:6800/1734465762,v1:172.21.15.150:6801/1734465762]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-28T15:54:18.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:18 smithi183 ceph-mon[35580]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-28T15:54:19.682 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:19 smithi150 ceph-mon[34825]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-28T15:54:19.683 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:19 smithi150 ceph-mon[34825]: osdmap e11: 2 total, 1 up, 2 in 2024-03-28T15:54:19.683 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:19 smithi150 ceph-mon[34825]: from='osd.1 [v2:172.21.15.150:6800/1734465762,v1:172.21.15.150:6801/1734465762]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-03-28T15:54:19.683 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:19 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T15:54:19.683 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:19 smithi150 ceph-mon[34825]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-03-28T15:54:19.683 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:19 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:54:19.683 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:19 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:54:19.683 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:19 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:54:19.732 DEBUG:teuthology.orchestra.run.smithi150:osd.1> sudo journalctl -f -n 0 -u ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@osd.1.service 2024-03-28T15:54:19.737 INFO:tasks.cephadm:Deploying osd.2 on smithi183 with /dev/vg_nvme/lv_4... 2024-03-28T15:54:19.737 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-28T15:54:19.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:19 smithi067 ceph-mon[31707]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-28T15:54:19.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:19 smithi067 ceph-mon[31707]: osdmap e11: 2 total, 1 up, 2 in 2024-03-28T15:54:19.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:19 smithi067 ceph-mon[31707]: from='osd.1 [v2:172.21.15.150:6800/1734465762,v1:172.21.15.150:6801/1734465762]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-03-28T15:54:19.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:19 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T15:54:19.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:19 smithi067 ceph-mon[31707]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-03-28T15:54:19.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:19 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:54:19.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:19 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:54:19.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:19 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:54:19.826 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:19 smithi183 ceph-mon[35580]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-28T15:54:19.826 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:19 smithi183 ceph-mon[35580]: osdmap e11: 2 total, 1 up, 2 in 2024-03-28T15:54:19.826 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:19 smithi183 ceph-mon[35580]: from='osd.1 [v2:172.21.15.150:6800/1734465762,v1:172.21.15.150:6801/1734465762]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-03-28T15:54:19.826 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:19 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T15:54:19.826 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:19 smithi183 ceph-mon[35580]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-03-28T15:54:19.826 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:19 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:54:19.826 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:19 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:54:19.826 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:19 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:54:20.176 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T15:54:20.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:20 smithi183 ceph-mon[35580]: pgmap v81: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:54:20.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:20 smithi183 ceph-mon[35580]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-03-28T15:54:20.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:20 smithi183 ceph-mon[35580]: osdmap e12: 2 total, 1 up, 2 in 2024-03-28T15:54:20.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:20 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T15:54:20.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:20 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T15:54:20.815 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:20 smithi150 ceph-mon[34825]: pgmap v81: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:54:20.815 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:20 smithi150 ceph-mon[34825]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-03-28T15:54:20.815 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:20 smithi150 ceph-mon[34825]: osdmap e12: 2 total, 1 up, 2 in 2024-03-28T15:54:20.815 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:20 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T15:54:20.815 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:20 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T15:54:20.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:20 smithi067 ceph-mon[31707]: pgmap v81: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:54:20.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:20 smithi067 ceph-mon[31707]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-03-28T15:54:20.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:20 smithi067 ceph-mon[31707]: osdmap e12: 2 total, 1 up, 2 in 2024-03-28T15:54:20.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:20 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T15:54:20.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:20 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T15:54:21.702 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:21 smithi150 ceph-mon[34825]: purged_snaps scrub starts 2024-03-28T15:54:21.702 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:21 smithi150 ceph-mon[34825]: purged_snaps scrub ok 2024-03-28T15:54:21.702 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:21 smithi150 ceph-mon[34825]: osd.1 [v2:172.21.15.150:6800/1734465762,v1:172.21.15.150:6801/1734465762] boot 2024-03-28T15:54:21.702 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:21 smithi150 ceph-mon[34825]: osdmap e13: 2 total, 2 up, 2 in 2024-03-28T15:54:21.702 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:21 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T15:54:21.753 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:21 smithi183 ceph-mon[35580]: purged_snaps scrub starts 2024-03-28T15:54:21.754 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:21 smithi183 ceph-mon[35580]: purged_snaps scrub ok 2024-03-28T15:54:21.754 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:21 smithi183 ceph-mon[35580]: osd.1 [v2:172.21.15.150:6800/1734465762,v1:172.21.15.150:6801/1734465762] boot 2024-03-28T15:54:21.754 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:21 smithi183 ceph-mon[35580]: osdmap e13: 2 total, 2 up, 2 in 2024-03-28T15:54:21.754 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:21 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T15:54:22.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:21 smithi067 ceph-mon[31707]: purged_snaps scrub starts 2024-03-28T15:54:22.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:21 smithi067 ceph-mon[31707]: purged_snaps scrub ok 2024-03-28T15:54:22.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:21 smithi067 ceph-mon[31707]: osd.1 [v2:172.21.15.150:6800/1734465762,v1:172.21.15.150:6801/1734465762] boot 2024-03-28T15:54:22.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:21 smithi067 ceph-mon[31707]: osdmap e13: 2 total, 2 up, 2 in 2024-03-28T15:54:22.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:21 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T15:54:22.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:22 smithi183 ceph-mon[35580]: pgmap v84: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:54:22.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:22 smithi183 ceph-mon[35580]: osdmap e14: 2 total, 2 up, 2 in 2024-03-28T15:54:22.893 INFO:teuthology.orchestra.run.smithi183.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-28T15:54:22.893 INFO:teuthology.orchestra.run.smithi183.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-28T15:54:22.893 INFO:teuthology.orchestra.run.smithi183.stderr: stderr: 10+0 records in 2024-03-28T15:54:22.893 INFO:teuthology.orchestra.run.smithi183.stderr:10+0 records out 2024-03-28T15:54:22.893 INFO:teuthology.orchestra.run.smithi183.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0356892 s, 294 MB/s 2024-03-28T15:54:22.894 INFO:teuthology.orchestra.run.smithi183.stderr: stderr: 2024-03-28T15:54:22.894 INFO:teuthology.orchestra.run.smithi183.stderr:--> Zapping successful for: 2024-03-28T15:54:23.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:22 smithi150 ceph-mon[34825]: pgmap v84: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:54:23.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:22 smithi150 ceph-mon[34825]: osdmap e14: 2 total, 2 up, 2 in 2024-03-28T15:54:23.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:22 smithi067 ceph-mon[31707]: pgmap v84: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:54:23.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:22 smithi067 ceph-mon[31707]: osdmap e14: 2 total, 2 up, 2 in 2024-03-28T15:54:23.582 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi183:vg_nvme/lv_4 2024-03-28T15:54:23.992 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T15:54:24.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:24 smithi183 ceph-mon[35580]: pgmap v86: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:54:24.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:24 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:54:24.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:24 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:54:24.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:24 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:54:24.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:24 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:54:24.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:24 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:54:24.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:24 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:54:24.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:24 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:54:25.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:24 smithi150 ceph-mon[34825]: pgmap v86: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:54:25.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:24 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:54:25.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:24 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:54:25.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:24 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:54:25.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:24 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:54:25.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:24 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:54:25.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:24 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:54:25.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:24 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:54:25.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:24 smithi067 ceph-mon[31707]: pgmap v86: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:54:25.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:24 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:54:25.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:24 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:54:25.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:24 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:54:25.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:24 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:54:25.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:24 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:54:25.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:24 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:54:25.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:24 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:54:25.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:25 smithi183 ceph-mon[35580]: Detected new or changed devices on smithi150 2024-03-28T15:54:25.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:25 smithi183 ceph-mon[35580]: Adjusting osd_memory_target on smithi150 to 17091M 2024-03-28T15:54:26.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:25 smithi150 ceph-mon[34825]: Detected new or changed devices on smithi150 2024-03-28T15:54:26.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:25 smithi150 ceph-mon[34825]: Adjusting osd_memory_target on smithi150 to 17091M 2024-03-28T15:54:26.067 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:54:26.066+0000 7f5b7de00700 1 -- 172.21.15.183:0/3296823520 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5b78108810 msgr2=0x7f5b78108c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:54:26.067 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:54:26.066+0000 7f5b7de00700 1 --2- 172.21.15.183:0/3296823520 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5b78108810 0x7f5b78108c30 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5b68009990 tx=0x7f5b68009ca0 comp rx=0 tx=0).stop 2024-03-28T15:54:26.067 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:54:26.066+0000 7f5b7de00700 1 -- 172.21.15.183:0/3296823520 shutdown_connections 2024-03-28T15:54:26.067 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:54:26.066+0000 7f5b7de00700 1 --2- 172.21.15.183:0/3296823520 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b7810a3b0 0x7f5b78110c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:26.067 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:54:26.066+0000 7f5b7de00700 1 --2- 172.21.15.183:0/3296823520 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5b78108810 0x7f5b78108c30 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:26.067 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:54:26.066+0000 7f5b7de00700 1 --2- 172.21.15.183:0/3296823520 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b78109a10 0x7f5b78109e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:26.067 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:54:26.066+0000 7f5b7de00700 1 -- 172.21.15.183:0/3296823520 >> 172.21.15.183:0/3296823520 conn(0x7f5b780777e0 msgr2=0x7f5b78079c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:54:26.067 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:54:26.066+0000 7f5b7de00700 1 -- 172.21.15.183:0/3296823520 shutdown_connections 2024-03-28T15:54:26.067 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:54:26.066+0000 7f5b7de00700 1 -- 172.21.15.183:0/3296823520 wait complete. 2024-03-28T15:54:26.068 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:54:26.067+0000 7f5b7de00700 1 Processor -- start 2024-03-28T15:54:26.068 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:54:26.068+0000 7f5b7de00700 1 -- start start 2024-03-28T15:54:26.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:25 smithi067 ceph-mon[31707]: Detected new or changed devices on smithi150 2024-03-28T15:54:26.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:25 smithi067 ceph-mon[31707]: Adjusting osd_memory_target on smithi150 to 17091M 2024-03-28T15:54:26.070 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:54:26.068+0000 7f5b7de00700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b78108810 0x7f5b7819e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:26.070 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:54:26.068+0000 7f5b7de00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5b78109a10 0x7f5b7819ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:26.070 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:54:26.068+0000 7f5b7de00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b7810a3b0 0x7f5b781a5f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:26.071 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:54:26.068+0000 7f5b7de00700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b781142f0 con 0x7f5b7810a3b0 2024-03-28T15:54:26.071 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:54:26.068+0000 7f5b7de00700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b78114170 con 0x7f5b78108810 2024-03-28T15:54:26.071 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:54:26.068+0000 7f5b7de00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b78114470 con 0x7f5b78109a10 2024-03-28T15:54:26.071 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:54:26.068+0000 7f5b6fe00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5b78109a10 0x7f5b7819ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:26.071 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:54:26.068+0000 7f5b6fe00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5b78109a10 0x7f5b7819ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.183:56012/0 (socket says 172.21.15.183:56012) 2024-03-28T15:54:26.071 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:54:26.068+0000 7f5b6fe00700 1 -- 172.21.15.183:0/3304017255 learned_addr learned my addr 172.21.15.183:0/3304017255 (peer_addr_for_me v2:172.21.15.183:0/0) 2024-03-28T15:54:26.071 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:54:26.069+0000 7f5b76a00700 1 --2- 172.21.15.183:0/3304017255 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b78108810 0x7f5b7819e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:26.073 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:54:26.069+0000 7f5b6fe00700 1 -- 172.21.15.183:0/3304017255 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b78108810 msgr2=0x7f5b7819e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:54:26.073 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:54:26.069+0000 7f5b6fe00700 1 --2- 172.21.15.183:0/3304017255 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b78108810 0x7f5b7819e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:26.073 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:54:26.069+0000 7f5b6fe00700 1 -- 172.21.15.183:0/3304017255 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b7810a3b0 msgr2=0x7f5b781a5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:54:26.073 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:54:26.069+0000 7f5b77400700 1 --2- 172.21.15.183:0/3304017255 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b7810a3b0 0x7f5b781a5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:26.073 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:54:26.069+0000 7f5b6fe00700 1 --2- 172.21.15.183:0/3304017255 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b7810a3b0 0x7f5b781a5f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:26.073 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:54:26.069+0000 7f5b6fe00700 1 -- 172.21.15.183:0/3304017255 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b781a6620 con 0x7f5b78109a10 2024-03-28T15:54:26.074 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:54:26.069+0000 7f5b6fe00700 1 --2- 172.21.15.183:0/3304017255 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5b78109a10 0x7f5b7819ef00 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5b6000d9b0 tx=0x7f5b6000dd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:54:26.074 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:54:26.069+0000 7f5b7ca00700 1 -- 172.21.15.183:0/3304017255 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b60017070 con 0x7f5b78109a10 2024-03-28T15:54:26.074 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:54:26.069+0000 7f5b7ca00700 1 -- 172.21.15.183:0/3304017255 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5b60004830 con 0x7f5b78109a10 2024-03-28T15:54:26.074 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:54:26.070+0000 7f5b7de00700 1 -- 172.21.15.183:0/3304017255 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b781a6910 con 0x7f5b78109a10 2024-03-28T15:54:26.074 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:54:26.070+0000 7f5b7ca00700 1 -- 172.21.15.183:0/3304017255 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b600053a0 con 0x7f5b78109a10 2024-03-28T15:54:26.074 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:54:26.070+0000 7f5b7de00700 1 -- 172.21.15.183:0/3304017255 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b781a6e00 con 0x7f5b78109a10 2024-03-28T15:54:26.075 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:54:26.071+0000 7f5b7de00700 1 -- 172.21.15.183:0/3304017255 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b78068e50 con 0x7f5b78109a10 2024-03-28T15:54:26.075 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:54:26.073+0000 7f5b7ca00700 1 -- 172.21.15.183:0/3304017255 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 15) v1 ==== 137234+0+0 (secure 0 0 0) 0x7f5b600049e0 con 0x7f5b78109a10 2024-03-28T15:54:26.075 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:54:26.076+0000 7f5b7ca00700 1 --2- 172.21.15.183:0/3304017255 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f5b5c0a2800 0x7f5b5c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:26.076 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:54:26.076+0000 7f5b7ca00700 1 -- 172.21.15.183:0/3304017255 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2309+0+0 (secure 0 0 0) 0x7f5b600cd5e0 con 0x7f5b78109a10 2024-03-28T15:54:26.076 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:54:26.076+0000 7f5b76a00700 1 --2- 172.21.15.183:0/3304017255 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f5b5c0a2800 0x7f5b5c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:26.079 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:54:26.077+0000 7f5b76a00700 1 --2- 172.21.15.183:0/3304017255 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f5b5c0a2800 0x7f5b5c0a4cc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5b6800b5c0 tx=0x7f5b68005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:54:26.079 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:54:26.079+0000 7f5b7ca00700 1 -- 172.21.15.183:0/3304017255 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5b60020020 con 0x7f5b78109a10 2024-03-28T15:54:26.313 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:54:26.312+0000 7f5b7de00700 1 -- 172.21.15.183:0/3304017255 --> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f5b780625d0 con 0x7f5b5c0a2800 2024-03-28T15:54:26.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:26 smithi183 ceph-mon[35580]: pgmap v87: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:54:26.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:26 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:54:26.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:26 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:54:26.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:26 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:54:27.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:26 smithi150 ceph-mon[34825]: pgmap v87: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:54:27.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:26 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:54:27.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:26 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:54:27.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:26 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:54:27.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:26 smithi067 ceph-mon[31707]: pgmap v87: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:54:27.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:26 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:54:27.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:26 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:54:27.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:26 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:54:27.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:27 smithi183 ceph-mon[35580]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:54:28.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:27 smithi150 ceph-mon[34825]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:54:28.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:27 smithi067 ceph-mon[31707]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:54:28.829 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:28 smithi183 ceph-mon[35580]: pgmap v88: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:54:29.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:28 smithi150 ceph-mon[34825]: pgmap v88: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:54:29.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:28 smithi067 ceph-mon[31707]: pgmap v88: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:54:30.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:30 smithi150 ceph-mon[34825]: pgmap v89: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:54:30.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:30 smithi150 ceph-mon[34825]: from='client.? 172.21.15.183:0/2399388946' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a55b9a02-c3fd-4061-95c0-53a24a17ade0"}]: dispatch 2024-03-28T15:54:30.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:30 smithi150 ceph-mon[34825]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a55b9a02-c3fd-4061-95c0-53a24a17ade0"}]: dispatch 2024-03-28T15:54:30.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:30 smithi150 ceph-mon[34825]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a55b9a02-c3fd-4061-95c0-53a24a17ade0"}]': finished 2024-03-28T15:54:30.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:30 smithi150 ceph-mon[34825]: osdmap e15: 3 total, 2 up, 3 in 2024-03-28T15:54:30.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:30 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T15:54:31.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:30 smithi067 ceph-mon[31707]: pgmap v89: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:54:31.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:30 smithi067 ceph-mon[31707]: from='client.? 172.21.15.183:0/2399388946' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a55b9a02-c3fd-4061-95c0-53a24a17ade0"}]: dispatch 2024-03-28T15:54:31.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:30 smithi067 ceph-mon[31707]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a55b9a02-c3fd-4061-95c0-53a24a17ade0"}]: dispatch 2024-03-28T15:54:31.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:30 smithi067 ceph-mon[31707]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a55b9a02-c3fd-4061-95c0-53a24a17ade0"}]': finished 2024-03-28T15:54:31.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:30 smithi067 ceph-mon[31707]: osdmap e15: 3 total, 2 up, 3 in 2024-03-28T15:54:31.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:30 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T15:54:31.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:30 smithi183 ceph-mon[35580]: pgmap v89: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:54:31.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:30 smithi183 ceph-mon[35580]: from='client.? 172.21.15.183:0/2399388946' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a55b9a02-c3fd-4061-95c0-53a24a17ade0"}]: dispatch 2024-03-28T15:54:31.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:30 smithi183 ceph-mon[35580]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a55b9a02-c3fd-4061-95c0-53a24a17ade0"}]: dispatch 2024-03-28T15:54:31.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:30 smithi183 ceph-mon[35580]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a55b9a02-c3fd-4061-95c0-53a24a17ade0"}]': finished 2024-03-28T15:54:31.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:30 smithi183 ceph-mon[35580]: osdmap e15: 3 total, 2 up, 3 in 2024-03-28T15:54:31.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:30 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T15:54:32.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:31 smithi150 ceph-mon[34825]: from='client.? 172.21.15.183:0/2341128070' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:54:32.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:31 smithi067 ceph-mon[31707]: from='client.? 172.21.15.183:0/2341128070' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:54:32.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:31 smithi183 ceph-mon[35580]: from='client.? 172.21.15.183:0/2341128070' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:54:33.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:32 smithi150 ceph-mon[34825]: pgmap v91: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:54:33.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:32 smithi067 ceph-mon[31707]: pgmap v91: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:54:33.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:32 smithi183 ceph-mon[35580]: pgmap v91: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:54:34.894 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:34 smithi183 ceph-mon[35580]: pgmap v92: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:54:34.907 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:34 smithi067 ceph-mon[31707]: pgmap v92: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:54:35.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:34 smithi150 ceph-mon[34825]: pgmap v92: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:54:36.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:36 smithi067 ceph-mon[31707]: pgmap v93: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:54:36.945 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:36 smithi183 ceph-mon[35580]: pgmap v93: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:54:37.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:36 smithi150 ceph-mon[34825]: pgmap v93: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:54:39.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:38 smithi150 ceph-mon[34825]: pgmap v94: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:54:39.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:38 smithi067 ceph-mon[31707]: pgmap v94: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:54:39.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:38 smithi183 ceph-mon[35580]: pgmap v94: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:54:39.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:39 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-28T15:54:39.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:39 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:54:40.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:39 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-28T15:54:40.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:39 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:54:40.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:39 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-28T15:54:40.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:39 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:54:40.784 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:40 smithi183 ceph-mon[35580]: pgmap v95: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:54:40.784 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:40 smithi183 ceph-mon[35580]: Deploying daemon osd.2 on smithi183 2024-03-28T15:54:40.927 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:40 smithi150 ceph-mon[34825]: pgmap v95: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:54:40.927 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:40 smithi150 ceph-mon[34825]: Deploying daemon osd.2 on smithi183 2024-03-28T15:54:41.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:40 smithi067 ceph-mon[31707]: pgmap v95: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:54:41.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:40 smithi067 ceph-mon[31707]: Deploying daemon osd.2 on smithi183 2024-03-28T15:54:42.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:42 smithi183 ceph-mon[35580]: pgmap v96: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:54:43.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:42 smithi150 ceph-mon[34825]: pgmap v96: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:54:43.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:42 smithi067 ceph-mon[31707]: pgmap v96: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:54:44.844 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:44 smithi183 ceph-mon[35580]: pgmap v97: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:54:45.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:44 smithi150 ceph-mon[34825]: pgmap v97: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:54:45.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:44 smithi067 ceph-mon[31707]: pgmap v97: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:54:46.968 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:46 smithi183 ceph-mon[35580]: pgmap v98: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:54:47.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:46 smithi150 ceph-mon[34825]: pgmap v98: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:54:47.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:46 smithi067 ceph-mon[31707]: pgmap v98: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:54:48.477 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:48 smithi183 ceph-mon[35580]: pgmap v99: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:54:48.477 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:48 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:54:48.477 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:48 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:54:48.477 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:48 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:54:48.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:48 smithi150 ceph-mon[34825]: pgmap v99: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:54:48.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:48 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:54:48.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:48 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:54:48.553 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:48 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:54:48.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:48 smithi067 ceph-mon[31707]: pgmap v99: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:54:48.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:48 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:54:48.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:48 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:54:48.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:48 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:54:50.507 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:50 smithi183 ceph-mon[35580]: pgmap v100: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:54:50.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:50 smithi150 ceph-mon[34825]: pgmap v100: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:54:50.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:50 smithi067 ceph-mon[31707]: pgmap v100: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:54:51.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:51 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:54:51.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:51 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:54:51.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:51 smithi150 ceph-mon[34825]: from='osd.2 [v2:172.21.15.183:6800/3398095931,v1:172.21.15.183:6801/3398095931]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-28T15:54:51.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:51 smithi150 ceph-mon[34825]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-28T15:54:51.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:51 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:54:51.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:51 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:54:51.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:51 smithi067 ceph-mon[31707]: from='osd.2 [v2:172.21.15.183:6800/3398095931,v1:172.21.15.183:6801/3398095931]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-28T15:54:51.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:51 smithi067 ceph-mon[31707]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-28T15:54:51.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:51 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:54:51.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:51 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:54:51.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:51 smithi183 ceph-mon[35580]: from='osd.2 [v2:172.21.15.183:6800/3398095931,v1:172.21.15.183:6801/3398095931]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-28T15:54:51.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:51 smithi183 ceph-mon[35580]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-28T15:54:52.079 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:54:52.077+0000 7f5b7ca00700 1 -- 172.21.15.183:0/3304017255 <== mgr.14152 v2:172.21.15.67:6800/208726654 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f5b780625d0 con 0x7f5b5c0a2800 2024-03-28T15:54:52.079 INFO:teuthology.orchestra.run.smithi183.stdout:Created osd(s) 2 on host 'smithi183' 2024-03-28T15:54:52.082 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:54:52.080+0000 7f5b7de00700 1 -- 172.21.15.183:0/3304017255 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f5b5c0a2800 msgr2=0x7f5b5c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:54:52.082 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:54:52.080+0000 7f5b7de00700 1 --2- 172.21.15.183:0/3304017255 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f5b5c0a2800 0x7f5b5c0a4cc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5b6800b5c0 tx=0x7f5b68005fd0 comp rx=0 tx=0).stop 2024-03-28T15:54:52.082 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:54:52.080+0000 7f5b7de00700 1 -- 172.21.15.183:0/3304017255 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5b78109a10 msgr2=0x7f5b7819ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:54:52.083 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:54:52.080+0000 7f5b7de00700 1 --2- 172.21.15.183:0/3304017255 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5b78109a10 0x7f5b7819ef00 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5b6000d9b0 tx=0x7f5b6000dd70 comp rx=0 tx=0).stop 2024-03-28T15:54:52.083 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:54:52.080+0000 7f5b7de00700 1 -- 172.21.15.183:0/3304017255 shutdown_connections 2024-03-28T15:54:52.083 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:54:52.080+0000 7f5b7de00700 1 --2- 172.21.15.183:0/3304017255 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f5b5c0a2800 0x7f5b5c0a4cc0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:52.083 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:54:52.081+0000 7f5b7de00700 1 --2- 172.21.15.183:0/3304017255 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b7810a3b0 0x7f5b781a5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:52.083 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:54:52.081+0000 7f5b7de00700 1 --2- 172.21.15.183:0/3304017255 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b78108810 0x7f5b7819e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:52.083 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:54:52.081+0000 7f5b7de00700 1 --2- 172.21.15.183:0/3304017255 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5b78109a10 0x7f5b7819ef00 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:52.083 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:54:52.081+0000 7f5b7de00700 1 -- 172.21.15.183:0/3304017255 >> 172.21.15.183:0/3304017255 conn(0x7f5b780777e0 msgr2=0x7f5b7810cd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:54:52.083 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:54:52.081+0000 7f5b7de00700 1 -- 172.21.15.183:0/3304017255 shutdown_connections 2024-03-28T15:54:52.083 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:54:52.081+0000 7f5b7de00700 1 -- 172.21.15.183:0/3304017255 wait complete. 2024-03-28T15:54:52.556 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:52 smithi150 ceph-mon[34825]: pgmap v101: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:54:52.556 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:52 smithi150 ceph-mon[34825]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-28T15:54:52.556 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:52 smithi150 ceph-mon[34825]: from='osd.2 [v2:172.21.15.183:6800/3398095931,v1:172.21.15.183:6801/3398095931]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-03-28T15:54:52.556 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:52 smithi150 ceph-mon[34825]: osdmap e16: 3 total, 2 up, 3 in 2024-03-28T15:54:52.557 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:52 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T15:54:52.557 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:52 smithi150 ceph-mon[34825]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-03-28T15:54:52.557 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:52 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:54:52.557 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:52 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:54:52.557 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:52 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:54:52.557 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:52 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:54:52.557 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:52 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:54:52.557 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:52 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:54:52.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:52 smithi067 ceph-mon[31707]: pgmap v101: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:54:52.571 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:52 smithi067 ceph-mon[31707]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-28T15:54:52.571 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:52 smithi067 ceph-mon[31707]: from='osd.2 [v2:172.21.15.183:6800/3398095931,v1:172.21.15.183:6801/3398095931]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-03-28T15:54:52.571 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:52 smithi067 ceph-mon[31707]: osdmap e16: 3 total, 2 up, 3 in 2024-03-28T15:54:52.571 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:52 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T15:54:52.571 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:52 smithi067 ceph-mon[31707]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-03-28T15:54:52.571 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:52 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:54:52.571 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:52 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:54:52.571 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:52 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:54:52.571 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:52 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:54:52.571 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:52 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:54:52.571 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:52 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:54:52.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:52 smithi183 ceph-mon[35580]: pgmap v101: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:54:52.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:52 smithi183 ceph-mon[35580]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-28T15:54:52.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:52 smithi183 ceph-mon[35580]: from='osd.2 [v2:172.21.15.183:6800/3398095931,v1:172.21.15.183:6801/3398095931]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-03-28T15:54:52.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:52 smithi183 ceph-mon[35580]: osdmap e16: 3 total, 2 up, 3 in 2024-03-28T15:54:52.602 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:52 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T15:54:52.602 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:52 smithi183 ceph-mon[35580]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-03-28T15:54:52.602 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:52 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:54:52.602 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:52 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:54:52.602 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:52 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:54:52.602 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:52 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:54:52.602 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:52 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:54:52.602 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:52 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:54:52.850 DEBUG:teuthology.orchestra.run.smithi183:osd.2> sudo journalctl -f -n 0 -u ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@osd.2.service 2024-03-28T15:54:52.855 INFO:tasks.cephadm:Deploying osd.3 on smithi067 with /dev/vg_nvme/lv_3... 2024-03-28T15:54:52.855 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-28T15:54:53.321 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:54:53.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:53 smithi183 ceph-mon[35580]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-03-28T15:54:53.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:53 smithi183 ceph-mon[35580]: osdmap e17: 3 total, 2 up, 3 in 2024-03-28T15:54:53.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:53 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T15:54:53.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:53 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T15:54:53.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:53 smithi150 ceph-mon[34825]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-03-28T15:54:53.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:53 smithi150 ceph-mon[34825]: osdmap e17: 3 total, 2 up, 3 in 2024-03-28T15:54:53.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:53 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T15:54:53.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:53 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T15:54:53.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:53 smithi067 ceph-mon[31707]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-03-28T15:54:53.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:53 smithi067 ceph-mon[31707]: osdmap e17: 3 total, 2 up, 3 in 2024-03-28T15:54:53.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:53 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T15:54:53.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:53 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T15:54:54.496 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:54 smithi067 ceph-mon[31707]: purged_snaps scrub starts 2024-03-28T15:54:54.496 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:54 smithi067 ceph-mon[31707]: purged_snaps scrub ok 2024-03-28T15:54:54.497 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:54 smithi067 ceph-mon[31707]: pgmap v104: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:54:54.497 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:54 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T15:54:54.497 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:54 smithi067 ceph-mon[31707]: osd.2 [v2:172.21.15.183:6800/3398095931,v1:172.21.15.183:6801/3398095931] boot 2024-03-28T15:54:54.497 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:54 smithi067 ceph-mon[31707]: osdmap e18: 3 total, 3 up, 3 in 2024-03-28T15:54:54.497 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:54 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T15:54:54.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:54 smithi150 ceph-mon[34825]: purged_snaps scrub starts 2024-03-28T15:54:54.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:54 smithi150 ceph-mon[34825]: purged_snaps scrub ok 2024-03-28T15:54:54.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:54 smithi150 ceph-mon[34825]: pgmap v104: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:54:54.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:54 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T15:54:54.553 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:54 smithi150 ceph-mon[34825]: osd.2 [v2:172.21.15.183:6800/3398095931,v1:172.21.15.183:6801/3398095931] boot 2024-03-28T15:54:54.553 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:54 smithi150 ceph-mon[34825]: osdmap e18: 3 total, 3 up, 3 in 2024-03-28T15:54:54.553 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:54 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T15:54:54.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:54 smithi183 ceph-mon[35580]: purged_snaps scrub starts 2024-03-28T15:54:54.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:54 smithi183 ceph-mon[35580]: purged_snaps scrub ok 2024-03-28T15:54:54.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:54 smithi183 ceph-mon[35580]: pgmap v104: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:54:54.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:54 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T15:54:54.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:54 smithi183 ceph-mon[35580]: osd.2 [v2:172.21.15.183:6800/3398095931,v1:172.21.15.183:6801/3398095931] boot 2024-03-28T15:54:54.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:54 smithi183 ceph-mon[35580]: osdmap e18: 3 total, 3 up, 3 in 2024-03-28T15:54:54.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:54 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T15:54:55.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:55 smithi183 ceph-mon[35580]: osdmap e19: 3 total, 3 up, 3 in 2024-03-28T15:54:55.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:55 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-28T15:54:56.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:55 smithi150 ceph-mon[34825]: osdmap e19: 3 total, 3 up, 3 in 2024-03-28T15:54:56.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:55 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-28T15:54:56.067 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:55 smithi067 ceph-mon[31707]: osdmap e19: 3 total, 3 up, 3 in 2024-03-28T15:54:56.068 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:55 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-28T15:54:56.068 INFO:teuthology.orchestra.run.smithi067.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-28T15:54:56.068 INFO:teuthology.orchestra.run.smithi067.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-28T15:54:56.068 INFO:teuthology.orchestra.run.smithi067.stderr: stderr: 10+0 records in 2024-03-28T15:54:56.068 INFO:teuthology.orchestra.run.smithi067.stderr:10+0 records out 2024-03-28T15:54:56.068 INFO:teuthology.orchestra.run.smithi067.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0377065 s, 278 MB/s 2024-03-28T15:54:56.068 INFO:teuthology.orchestra.run.smithi067.stderr:--> Zapping successful for: 2024-03-28T15:54:56.789 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi067:vg_nvme/lv_3 2024-03-28T15:54:57.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:56 smithi150 ceph-mon[34825]: pgmap v107: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:54:57.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:56 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-28T15:54:57.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:56 smithi150 ceph-mon[34825]: osdmap e20: 3 total, 3 up, 3 in 2024-03-28T15:54:57.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:56 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-28T15:54:57.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:56 smithi067 ceph-mon[31707]: pgmap v107: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:54:57.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:56 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-28T15:54:57.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:56 smithi067 ceph-mon[31707]: osdmap e20: 3 total, 3 up, 3 in 2024-03-28T15:54:57.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:56 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-28T15:54:57.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:56 smithi183 ceph-mon[35580]: pgmap v107: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:54:57.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:56 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-28T15:54:57.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:56 smithi183 ceph-mon[35580]: osdmap e20: 3 total, 3 up, 3 in 2024-03-28T15:54:57.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:56 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-28T15:54:57.249 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:54:57.659 INFO:journalctl@ceph.osd.0.smithi067.stdout:Mar 28 15:54:57 smithi067 sudo[50636]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-28T15:54:57.659 INFO:journalctl@ceph.osd.0.smithi067.stdout:Mar 28 15:54:57 smithi067 sudo[50636]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T15:54:57.660 INFO:journalctl@ceph.osd.0.smithi067.stdout:Mar 28 15:54:57 smithi067 sudo[50636]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T15:54:57.660 INFO:journalctl@ceph.osd.0.smithi067.stdout:Mar 28 15:54:57 smithi067 sudo[50636]: pam_unix(sudo:session): session closed for user root 2024-03-28T15:54:58.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:57 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-28T15:54:58.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:57 smithi150 ceph-mon[34825]: osdmap e21: 3 total, 3 up, 3 in 2024-03-28T15:54:58.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:57 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:54:58.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:57 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:54:58.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:57 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:54:58.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:57 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:54:58.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:57 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:54:58.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:57 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:54:58.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:57 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:54:58.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:57 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-28T15:54:58.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:57 smithi067 ceph-mon[31707]: osdmap e21: 3 total, 3 up, 3 in 2024-03-28T15:54:58.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:57 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:54:58.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:57 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:54:58.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:57 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:54:58.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:57 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:54:58.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:57 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:54:58.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:57 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:54:58.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:57 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:54:58.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:57 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-28T15:54:58.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:57 smithi183 ceph-mon[35580]: osdmap e21: 3 total, 3 up, 3 in 2024-03-28T15:54:58.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:57 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:54:58.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:57 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:54:58.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:57 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:54:58.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:57 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:54:58.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:57 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:54:58.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:57 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:54:58.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:57 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:54:58.435 INFO:journalctl@ceph.osd.0.smithi067.stdout:Mar 28 15:54:58 smithi067 sudo[50660]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-28T15:54:58.435 INFO:journalctl@ceph.osd.0.smithi067.stdout:Mar 28 15:54:58 smithi067 sudo[50660]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T15:54:58.435 INFO:journalctl@ceph.osd.0.smithi067.stdout:Mar 28 15:54:58 smithi067 sudo[50660]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T15:54:58.435 INFO:journalctl@ceph.osd.0.smithi067.stdout:Mar 28 15:54:58 smithi067 sudo[50660]: pam_unix(sudo:session): session closed for user root 2024-03-28T15:54:58.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:58 smithi067 ceph-mon[31707]: pgmap v110: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:54:58.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:58 smithi067 ceph-mon[31707]: Detected new or changed devices on smithi183 2024-03-28T15:54:58.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:58 smithi067 ceph-mon[31707]: Adjusting osd_memory_target on smithi183 to 17091M 2024-03-28T15:54:58.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:58 smithi067 ceph-mon[31707]: mgrmap e16: x(active, since 3m), standbys: z, y 2024-03-28T15:54:58.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:58 smithi067 ceph-mon[31707]: osdmap e22: 3 total, 3 up, 3 in 2024-03-28T15:54:58.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:54:58 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:54:59.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:58 smithi150 ceph-mon[34825]: pgmap v110: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:54:59.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:58 smithi150 ceph-mon[34825]: Detected new or changed devices on smithi183 2024-03-28T15:54:59.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:58 smithi150 ceph-mon[34825]: Adjusting osd_memory_target on smithi183 to 17091M 2024-03-28T15:54:59.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:58 smithi150 ceph-mon[34825]: mgrmap e16: x(active, since 3m), standbys: z, y 2024-03-28T15:54:59.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:58 smithi150 ceph-mon[34825]: osdmap e22: 3 total, 3 up, 3 in 2024-03-28T15:54:59.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:54:58 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:54:59.053 INFO:journalctl@ceph.osd.1.smithi150.stdout:Mar 28 15:54:58 smithi150 sudo[43667]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-28T15:54:59.053 INFO:journalctl@ceph.osd.1.smithi150.stdout:Mar 28 15:54:58 smithi150 sudo[43667]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T15:54:59.053 INFO:journalctl@ceph.osd.1.smithi150.stdout:Mar 28 15:54:58 smithi150 sudo[43667]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T15:54:59.053 INFO:journalctl@ceph.osd.1.smithi150.stdout:Mar 28 15:54:58 smithi150 sudo[43667]: pam_unix(sudo:session): session closed for user root 2024-03-28T15:54:59.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:58 smithi183 ceph-mon[35580]: pgmap v110: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:54:59.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:58 smithi183 ceph-mon[35580]: Detected new or changed devices on smithi183 2024-03-28T15:54:59.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:58 smithi183 ceph-mon[35580]: Adjusting osd_memory_target on smithi183 to 17091M 2024-03-28T15:54:59.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:58 smithi183 ceph-mon[35580]: mgrmap e16: x(active, since 3m), standbys: z, y 2024-03-28T15:54:59.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:58 smithi183 ceph-mon[35580]: osdmap e22: 3 total, 3 up, 3 in 2024-03-28T15:54:59.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:54:58 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:54:59.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:54:59.287+0000 7f1ce6200700 1 -- 172.21.15.67:0/2794829488 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ce0105610 msgr2=0x7f1ce0105a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:54:59.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:54:59.287+0000 7f1ce6200700 1 --2- 172.21.15.67:0/2794829488 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ce0105610 0x7f1ce0105a70 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1cc80099c0 tx=0x7f1cc8009cd0 comp rx=0 tx=0).stop 2024-03-28T15:54:59.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:54:59.287+0000 7f1ce6200700 1 -- 172.21.15.67:0/2794829488 shutdown_connections 2024-03-28T15:54:59.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:54:59.287+0000 7f1ce6200700 1 --2- 172.21.15.67:0/2794829488 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1ce0105fb0 0x7f1ce010c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:59.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:54:59.287+0000 7f1ce6200700 1 --2- 172.21.15.67:0/2794829488 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1ce0104410 0x7f1ce0104830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:59.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:54:59.287+0000 7f1ce6200700 1 --2- 172.21.15.67:0/2794829488 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ce0105610 0x7f1ce0105a70 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:59.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:54:59.287+0000 7f1ce6200700 1 -- 172.21.15.67:0/2794829488 >> 172.21.15.67:0/2794829488 conn(0x7f1ce00ff9b0 msgr2=0x7f1ce0101df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:54:59.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:54:59.287+0000 7f1ce6200700 1 -- 172.21.15.67:0/2794829488 shutdown_connections 2024-03-28T15:54:59.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:54:59.287+0000 7f1ce6200700 1 -- 172.21.15.67:0/2794829488 wait complete. 2024-03-28T15:54:59.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:54:59.288+0000 7f1ce6200700 1 Processor -- start 2024-03-28T15:54:59.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:54:59.288+0000 7f1ce6200700 1 -- start start 2024-03-28T15:54:59.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:54:59.289+0000 7f1ce6200700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1ce0104410 0x7f1ce019a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:59.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:54:59.289+0000 7f1ce6200700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1ce0105610 0x7f1ce019aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:59.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:54:59.289+0000 7f1ce6200700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ce0105fb0 0x7f1ce01a1a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:59.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:54:59.289+0000 7f1ce6200700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce010feb0 con 0x7f1ce0105fb0 2024-03-28T15:54:59.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:54:59.289+0000 7f1ce6200700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce010fd30 con 0x7f1ce0104410 2024-03-28T15:54:59.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:54:59.289+0000 7f1ce6200700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce0110030 con 0x7f1ce0105610 2024-03-28T15:54:59.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:54:59.289+0000 7f1cdfe00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ce0105fb0 0x7f1ce01a1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:59.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:54:59.289+0000 7f1cdfe00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ce0105fb0 0x7f1ce01a1a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59650/0 (socket says 172.21.15.67:59650) 2024-03-28T15:54:59.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:54:59.289+0000 7f1cdfe00700 1 -- 172.21.15.67:0/1011856713 learned_addr learned my addr 172.21.15.67:0/1011856713 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:54:59.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:54:59.289+0000 7f1cdf400700 1 --2- 172.21.15.67:0/1011856713 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1ce0104410 0x7f1ce019a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:59.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:54:59.289+0000 7f1cd6a00700 1 --2- 172.21.15.67:0/1011856713 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1ce0105610 0x7f1ce019aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:59.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:54:59.290+0000 7f1cdfe00700 1 -- 172.21.15.67:0/1011856713 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1ce0105610 msgr2=0x7f1ce019aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:54:59.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:54:59.290+0000 7f1cdfe00700 1 --2- 172.21.15.67:0/1011856713 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1ce0105610 0x7f1ce019aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:59.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:54:59.290+0000 7f1cdfe00700 1 -- 172.21.15.67:0/1011856713 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1ce0104410 msgr2=0x7f1ce019a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:54:59.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:54:59.290+0000 7f1cdfe00700 1 --2- 172.21.15.67:0/1011856713 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1ce0104410 0x7f1ce019a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:59.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:54:59.290+0000 7f1cdfe00700 1 -- 172.21.15.67:0/1011856713 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ce01a2120 con 0x7f1ce0105fb0 2024-03-28T15:54:59.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:54:59.290+0000 7f1cdfe00700 1 --2- 172.21.15.67:0/1011856713 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ce0105fb0 0x7f1ce01a1a80 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1cd800ef60 tx=0x7f1cd800ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:54:59.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:54:59.290+0000 7f1cdcc00700 1 -- 172.21.15.67:0/1011856713 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cd80092e0 con 0x7f1ce0105fb0 2024-03-28T15:54:59.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:54:59.290+0000 7f1cdcc00700 1 -- 172.21.15.67:0/1011856713 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1cd8004bb0 con 0x7f1ce0105fb0 2024-03-28T15:54:59.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:54:59.290+0000 7f1ce6200700 1 -- 172.21.15.67:0/1011856713 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ce01a23b0 con 0x7f1ce0105fb0 2024-03-28T15:54:59.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:54:59.290+0000 7f1cdcc00700 1 -- 172.21.15.67:0/1011856713 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cd8004280 con 0x7f1ce0105fb0 2024-03-28T15:54:59.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:54:59.291+0000 7f1ce6200700 1 -- 172.21.15.67:0/1011856713 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ce01a28f0 con 0x7f1ce0105fb0 2024-03-28T15:54:59.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:54:59.292+0000 7f1ce6200700 1 -- 172.21.15.67:0/1011856713 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ce0068e50 con 0x7f1ce0105fb0 2024-03-28T15:54:59.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:54:59.294+0000 7f1cdcc00700 1 -- 172.21.15.67:0/1011856713 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f1cd801b030 con 0x7f1ce0105fb0 2024-03-28T15:54:59.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:54:59.295+0000 7f1cdcc00700 1 --2- 172.21.15.67:0/1011856713 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f1cc40a2890 0x7f1cc40a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:59.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:54:59.295+0000 7f1cdcc00700 1 -- 172.21.15.67:0/1011856713 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 3237+0+0 (secure 0 0 0) 0x7f1cd80cd1b0 con 0x7f1ce0105fb0 2024-03-28T15:54:59.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:54:59.295+0000 7f1cdf400700 1 --2- 172.21.15.67:0/1011856713 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f1cc40a2890 0x7f1cc40a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:59.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:54:59.296+0000 7f1cdf400700 1 --2- 172.21.15.67:0/1011856713 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f1cc40a2890 0x7f1cc40a4d50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1ce0105470 tx=0x7f1cd000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:54:59.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:54:59.300+0000 7f1cdcc00700 1 -- 172.21.15.67:0/1011856713 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1cd8022020 con 0x7f1ce0105fb0 2024-03-28T15:54:59.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:54:59.548+0000 7f1ce6200700 1 -- 172.21.15.67:0/1011856713 --> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f1ce0062690 con 0x7f1cc40a2890 2024-03-28T15:54:59.802 INFO:journalctl@ceph.osd.1.smithi150.stdout:Mar 28 15:54:59 smithi150 sudo[43670]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-28T15:54:59.802 INFO:journalctl@ceph.osd.1.smithi150.stdout:Mar 28 15:54:59 smithi150 sudo[43670]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T15:54:59.802 INFO:journalctl@ceph.osd.1.smithi150.stdout:Mar 28 15:54:59 smithi150 sudo[43670]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T15:54:59.802 INFO:journalctl@ceph.osd.1.smithi150.stdout:Mar 28 15:54:59 smithi150 sudo[43670]: pam_unix(sudo:session): session closed for user root 2024-03-28T15:55:00.350 INFO:journalctl@ceph.osd.2.smithi183.stdout:Mar 28 15:54:59 smithi183 sudo[44574]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-28T15:55:00.350 INFO:journalctl@ceph.osd.2.smithi183.stdout:Mar 28 15:54:59 smithi183 sudo[44574]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T15:55:00.350 INFO:journalctl@ceph.osd.2.smithi183.stdout:Mar 28 15:54:59 smithi183 sudo[44574]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T15:55:00.351 INFO:journalctl@ceph.osd.2.smithi183.stdout:Mar 28 15:54:59 smithi183 sudo[44574]: pam_unix(sudo:session): session closed for user root 2024-03-28T15:55:00.632 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:00 smithi067 ceph-mon[31707]: pgmap v112: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:55:00.633 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:00 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:55:00.633 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:00 smithi067 ceph-mon[31707]: from='client.14292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:55:00.633 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:00 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:55:00.633 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:00 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:55:00.633 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:00 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:55:00.705 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:00 smithi150 ceph-mon[34825]: pgmap v112: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:55:00.705 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:00 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:55:00.705 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:00 smithi150 ceph-mon[34825]: from='client.14292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:55:00.705 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:00 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:55:00.705 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:00 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:55:00.705 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:00 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:55:00.850 INFO:journalctl@ceph.osd.2.smithi183.stdout:Mar 28 15:55:00 smithi183 sudo[44577]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-28T15:55:00.850 INFO:journalctl@ceph.osd.2.smithi183.stdout:Mar 28 15:55:00 smithi183 sudo[44577]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T15:55:00.850 INFO:journalctl@ceph.osd.2.smithi183.stdout:Mar 28 15:55:00 smithi183 sudo[44577]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T15:55:00.850 INFO:journalctl@ceph.osd.2.smithi183.stdout:Mar 28 15:55:00 smithi183 sudo[44577]: pam_unix(sudo:session): session closed for user root 2024-03-28T15:55:00.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:00 smithi183 ceph-mon[35580]: pgmap v112: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:55:00.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:00 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:55:00.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:00 smithi183 ceph-mon[35580]: from='client.14292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:55:00.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:00 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:55:00.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:00 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:55:00.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:00 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:55:01.487 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:01 smithi067 sudo[50854]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-28T15:55:01.488 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:01 smithi067 sudo[50854]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T15:55:01.488 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:01 smithi067 sudo[50854]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T15:55:01.488 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:01 smithi067 sudo[50854]: pam_unix(sudo:session): session closed for user root 2024-03-28T15:55:02.027 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:01 smithi067 sudo[50878]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-28T15:55:02.027 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:01 smithi067 sudo[50878]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T15:55:02.028 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:01 smithi067 sudo[50878]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T15:55:02.028 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:01 smithi067 sudo[50878]: pam_unix(sudo:session): session closed for user root 2024-03-28T15:55:02.028 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:01 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:55:02.028 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:01 smithi067 ceph-mon[31707]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T15:55:02.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:01 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:55:02.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:01 smithi150 ceph-mon[34825]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T15:55:02.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:01 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:55:02.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:01 smithi183 ceph-mon[35580]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T15:55:02.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:02 smithi183 sudo[44580]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-28T15:55:02.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:02 smithi183 sudo[44580]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T15:55:02.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:02 smithi183 sudo[44580]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T15:55:02.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:02 smithi183 sudo[44580]: pam_unix(sudo:session): session closed for user root 2024-03-28T15:55:03.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:02 smithi150 ceph-mon[34825]: pgmap v113: 1 pgs: 1 creating+peering; 0 B data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:55:03.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:02 smithi150 ceph-mon[34825]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T15:55:03.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:02 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T15:55:03.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:02 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T15:55:03.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:02 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T15:55:03.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:02 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T15:55:03.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:02 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T15:55:03.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:02 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T15:55:03.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:02 smithi067 ceph-mon[31707]: pgmap v113: 1 pgs: 1 creating+peering; 0 B data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:55:03.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:02 smithi067 ceph-mon[31707]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T15:55:03.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:02 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T15:55:03.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:02 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T15:55:03.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:02 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T15:55:03.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:02 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T15:55:03.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:02 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T15:55:03.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:02 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T15:55:03.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:02 smithi183 sudo[44583]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-28T15:55:03.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:02 smithi183 sudo[44583]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T15:55:03.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:02 smithi183 sudo[44583]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T15:55:03.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:02 smithi183 sudo[44583]: pam_unix(sudo:session): session closed for user root 2024-03-28T15:55:03.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:02 smithi183 ceph-mon[35580]: pgmap v113: 1 pgs: 1 creating+peering; 0 B data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:55:03.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:02 smithi183 ceph-mon[35580]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T15:55:03.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:02 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T15:55:03.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:02 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T15:55:03.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:02 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T15:55:03.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:02 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T15:55:03.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:02 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T15:55:03.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:02 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T15:55:03.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:03 smithi150 sudo[43673]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-28T15:55:03.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:03 smithi150 sudo[43673]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T15:55:03.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:03 smithi150 sudo[43673]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T15:55:03.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:03 smithi150 sudo[43673]: pam_unix(sudo:session): session closed for user root 2024-03-28T15:55:04.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:03 smithi150 sudo[43676]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-28T15:55:04.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:03 smithi150 sudo[43676]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T15:55:04.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:03 smithi150 sudo[43676]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T15:55:04.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:03 smithi150 sudo[43676]: pam_unix(sudo:session): session closed for user root 2024-03-28T15:55:04.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:04 smithi150 ceph-mon[34825]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T15:55:04.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:04 smithi150 ceph-mon[34825]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T15:55:04.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:04 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T15:55:04.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:04 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T15:55:04.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:04 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T15:55:04.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:04 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/1717156777' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d34940c0-5ce9-4b8b-aa36-4872414a043f"}]: dispatch 2024-03-28T15:55:04.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:04 smithi067 ceph-mon[31707]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T15:55:04.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:04 smithi067 ceph-mon[31707]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T15:55:04.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:04 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T15:55:04.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:04 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T15:55:04.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:04 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T15:55:04.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:04 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/1717156777' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d34940c0-5ce9-4b8b-aa36-4872414a043f"}]: dispatch 2024-03-28T15:55:04.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:04 smithi183 ceph-mon[35580]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T15:55:04.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:04 smithi183 ceph-mon[35580]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T15:55:04.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:04 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T15:55:04.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:04 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T15:55:04.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:04 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T15:55:04.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:04 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/1717156777' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d34940c0-5ce9-4b8b-aa36-4872414a043f"}]: dispatch 2024-03-28T15:55:05.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:05 smithi150 ceph-mon[34825]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T15:55:05.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:05 smithi150 ceph-mon[34825]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:55:05.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:05 smithi150 ceph-mon[34825]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T15:55:05.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:05 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/1717156777' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d34940c0-5ce9-4b8b-aa36-4872414a043f"}]': finished 2024-03-28T15:55:05.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:05 smithi150 ceph-mon[34825]: osdmap e23: 4 total, 3 up, 4 in 2024-03-28T15:55:05.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:05 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T15:55:05.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:05 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/1979201842' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:55:05.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:05 smithi067 ceph-mon[31707]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T15:55:05.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:05 smithi067 ceph-mon[31707]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:55:05.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:05 smithi067 ceph-mon[31707]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T15:55:05.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:05 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/1717156777' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d34940c0-5ce9-4b8b-aa36-4872414a043f"}]': finished 2024-03-28T15:55:05.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:05 smithi067 ceph-mon[31707]: osdmap e23: 4 total, 3 up, 4 in 2024-03-28T15:55:05.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:05 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T15:55:05.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:05 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/1979201842' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:55:05.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:05 smithi183 ceph-mon[35580]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T15:55:05.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:05 smithi183 ceph-mon[35580]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:55:05.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:05 smithi183 ceph-mon[35580]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T15:55:05.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:05 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/1717156777' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d34940c0-5ce9-4b8b-aa36-4872414a043f"}]': finished 2024-03-28T15:55:05.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:05 smithi183 ceph-mon[35580]: osdmap e23: 4 total, 3 up, 4 in 2024-03-28T15:55:05.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:05 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T15:55:05.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:05 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/1979201842' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:55:06.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:06 smithi150 ceph-mon[34825]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:55:06.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:06 smithi067 ceph-mon[31707]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:55:06.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:06 smithi183 ceph-mon[35580]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:55:08.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:08 smithi150 ceph-mon[34825]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 65 KiB/s wr, 5 op/s 2024-03-28T15:55:08.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:08 smithi067 ceph-mon[31707]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 65 KiB/s wr, 5 op/s 2024-03-28T15:55:08.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:08 smithi183 ceph-mon[35580]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 65 KiB/s wr, 5 op/s 2024-03-28T15:55:10.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:10 smithi150 ceph-mon[34825]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2024-03-28T15:55:10.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:10 smithi067 ceph-mon[31707]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2024-03-28T15:55:10.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:10 smithi183 ceph-mon[35580]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2024-03-28T15:55:12.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:12 smithi150 ceph-mon[34825]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2024-03-28T15:55:12.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:12 smithi067 ceph-mon[31707]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2024-03-28T15:55:12.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:12 smithi183 ceph-mon[35580]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2024-03-28T15:55:14.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:14 smithi150 ceph-mon[34825]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2024-03-28T15:55:14.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:14 smithi067 ceph-mon[31707]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2024-03-28T15:55:14.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:14 smithi183 ceph-mon[35580]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2024-03-28T15:55:15.318 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:15 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-28T15:55:15.318 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:15 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:55:15.318 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:15 smithi067 ceph-mon[31707]: Deploying daemon osd.3 on smithi067 2024-03-28T15:55:15.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:15 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-28T15:55:15.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:15 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:55:15.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:15 smithi183 ceph-mon[35580]: Deploying daemon osd.3 on smithi067 2024-03-28T15:55:15.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:15 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-28T15:55:15.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:15 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:55:15.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:15 smithi150 ceph-mon[34825]: Deploying daemon osd.3 on smithi067 2024-03-28T15:55:16.196 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:16 smithi067 ceph-mon[31707]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 843 B/s rd, 24 KiB/s wr, 2 op/s 2024-03-28T15:55:16.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:16 smithi183 ceph-mon[35580]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 843 B/s rd, 24 KiB/s wr, 2 op/s 2024-03-28T15:55:16.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:16 smithi150 ceph-mon[34825]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 843 B/s rd, 24 KiB/s wr, 2 op/s 2024-03-28T15:55:18.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:18 smithi067 ceph-mon[31707]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-28T15:55:18.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:18 smithi183 ceph-mon[35580]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-28T15:55:18.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:18 smithi150 ceph-mon[34825]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-28T15:55:20.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:20 smithi067 ceph-mon[31707]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:55:20.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:20 smithi183 ceph-mon[35580]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:55:20.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:20 smithi150 ceph-mon[34825]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:55:22.309 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:22 smithi067 ceph-mon[31707]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:55:22.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:22 smithi183 ceph-mon[35580]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:55:22.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:22 smithi150 ceph-mon[34825]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:55:23.729 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:23 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:55:23.729 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:23 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:55:23.729 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:23 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:55:23.755 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:23 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:55:23.755 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:23 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:55:23.755 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:23 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:55:23.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:23 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:55:23.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:23 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:55:23.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:23 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:55:24.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:24 smithi150 ceph-mon[34825]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:55:24.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:24 smithi067 ceph-mon[31707]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:55:24.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:24 smithi183 ceph-mon[35580]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:55:27.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:26 smithi150 ceph-mon[34825]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:55:27.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:26 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:55:27.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:26 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:55:27.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:26 smithi150 ceph-mon[34825]: from='osd.3 [v2:172.21.15.67:6810/1933541875,v1:172.21.15.67:6811/1933541875]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-28T15:55:27.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:26 smithi067 ceph-mon[31707]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:55:27.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:26 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:55:27.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:26 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:55:27.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:26 smithi067 ceph-mon[31707]: from='osd.3 [v2:172.21.15.67:6810/1933541875,v1:172.21.15.67:6811/1933541875]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-28T15:55:27.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:26 smithi183 ceph-mon[35580]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:55:27.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:26 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:55:27.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:26 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:55:27.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:26 smithi183 ceph-mon[35580]: from='osd.3 [v2:172.21.15.67:6810/1933541875,v1:172.21.15.67:6811/1933541875]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-28T15:55:27.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:55:27.550+0000 7f1cdcc00700 1 -- 172.21.15.67:0/1011856713 <== mgr.14152 v2:172.21.15.67:6800/208726654 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f1ce0062690 con 0x7f1cc40a2890 2024-03-28T15:55:27.551 INFO:teuthology.orchestra.run.smithi067.stdout:Created osd(s) 3 on host 'smithi067' 2024-03-28T15:55:27.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:55:27.554+0000 7f1ce6200700 1 -- 172.21.15.67:0/1011856713 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f1cc40a2890 msgr2=0x7f1cc40a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:55:27.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:55:27.554+0000 7f1ce6200700 1 --2- 172.21.15.67:0/1011856713 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f1cc40a2890 0x7f1cc40a4d50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1ce0105470 tx=0x7f1cd000b410 comp rx=0 tx=0).stop 2024-03-28T15:55:27.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:55:27.554+0000 7f1ce6200700 1 -- 172.21.15.67:0/1011856713 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ce0105fb0 msgr2=0x7f1ce01a1a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:55:27.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:55:27.554+0000 7f1ce6200700 1 --2- 172.21.15.67:0/1011856713 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ce0105fb0 0x7f1ce01a1a80 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1cd800ef60 tx=0x7f1cd800ef90 comp rx=0 tx=0).stop 2024-03-28T15:55:27.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:55:27.554+0000 7f1ce6200700 1 -- 172.21.15.67:0/1011856713 shutdown_connections 2024-03-28T15:55:27.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:55:27.554+0000 7f1ce6200700 1 --2- 172.21.15.67:0/1011856713 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f1cc40a2890 0x7f1cc40a4d50 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:27.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:55:27.554+0000 7f1ce6200700 1 --2- 172.21.15.67:0/1011856713 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ce0105fb0 0x7f1ce01a1a80 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:27.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:55:27.554+0000 7f1ce6200700 1 --2- 172.21.15.67:0/1011856713 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1ce0104410 0x7f1ce019a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:27.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:55:27.554+0000 7f1ce6200700 1 --2- 172.21.15.67:0/1011856713 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1ce0105610 0x7f1ce019aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:27.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:55:27.554+0000 7f1ce6200700 1 -- 172.21.15.67:0/1011856713 >> 172.21.15.67:0/1011856713 conn(0x7f1ce00ff9b0 msgr2=0x7f1ce0108910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:55:27.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:55:27.555+0000 7f1ce6200700 1 -- 172.21.15.67:0/1011856713 shutdown_connections 2024-03-28T15:55:27.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:55:27.555+0000 7f1ce6200700 1 -- 172.21.15.67:0/1011856713 wait complete. 2024-03-28T15:55:28.244 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:27 smithi067 ceph-mon[31707]: from='osd.3 [v2:172.21.15.67:6810/1933541875,v1:172.21.15.67:6811/1933541875]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-28T15:55:28.245 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:27 smithi067 ceph-mon[31707]: osdmap e24: 4 total, 3 up, 4 in 2024-03-28T15:55:28.245 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:27 smithi067 ceph-mon[31707]: from='osd.3 [v2:172.21.15.67:6810/1933541875,v1:172.21.15.67:6811/1933541875]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-03-28T15:55:28.245 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:27 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T15:55:28.245 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:27 smithi067 ceph-mon[31707]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:55:28.245 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:27 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:55:28.245 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:27 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:55:28.245 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:27 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:55:28.245 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:27 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:55:28.245 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:27 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:55:28.245 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:27 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:55:28.296 DEBUG:teuthology.orchestra.run.smithi067:osd.3> sudo journalctl -f -n 0 -u ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@osd.3.service 2024-03-28T15:55:28.301 INFO:tasks.cephadm:Deploying osd.4 on smithi150 with /dev/vg_nvme/lv_3... 2024-03-28T15:55:28.302 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-28T15:55:28.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:27 smithi150 ceph-mon[34825]: from='osd.3 [v2:172.21.15.67:6810/1933541875,v1:172.21.15.67:6811/1933541875]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-28T15:55:28.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:27 smithi150 ceph-mon[34825]: osdmap e24: 4 total, 3 up, 4 in 2024-03-28T15:55:28.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:27 smithi150 ceph-mon[34825]: from='osd.3 [v2:172.21.15.67:6810/1933541875,v1:172.21.15.67:6811/1933541875]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-03-28T15:55:28.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:27 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T15:55:28.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:27 smithi150 ceph-mon[34825]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:55:28.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:27 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:55:28.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:27 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:55:28.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:27 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:55:28.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:27 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:55:28.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:27 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:55:28.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:27 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:55:28.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:27 smithi183 ceph-mon[35580]: from='osd.3 [v2:172.21.15.67:6810/1933541875,v1:172.21.15.67:6811/1933541875]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-28T15:55:28.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:27 smithi183 ceph-mon[35580]: osdmap e24: 4 total, 3 up, 4 in 2024-03-28T15:55:28.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:27 smithi183 ceph-mon[35580]: from='osd.3 [v2:172.21.15.67:6810/1933541875,v1:172.21.15.67:6811/1933541875]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-03-28T15:55:28.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:27 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T15:55:28.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:27 smithi183 ceph-mon[35580]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:55:28.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:27 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:55:28.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:27 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:55:28.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:27 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:55:28.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:27 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:55:28.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:27 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:55:28.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:27 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:55:28.748 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T15:55:29.183 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:28 smithi150 ceph-mon[34825]: from='osd.3 [v2:172.21.15.67:6810/1933541875,v1:172.21.15.67:6811/1933541875]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-03-28T15:55:29.183 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:28 smithi150 ceph-mon[34825]: osdmap e25: 4 total, 3 up, 4 in 2024-03-28T15:55:29.183 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:28 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T15:55:29.183 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:28 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T15:55:29.184 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:28 smithi150 ceph-mon[34825]: osdmap e26: 4 total, 3 up, 4 in 2024-03-28T15:55:29.184 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:28 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T15:55:29.224 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:28 smithi067 ceph-mon[31707]: from='osd.3 [v2:172.21.15.67:6810/1933541875,v1:172.21.15.67:6811/1933541875]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-03-28T15:55:29.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:28 smithi067 ceph-mon[31707]: osdmap e25: 4 total, 3 up, 4 in 2024-03-28T15:55:29.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:28 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T15:55:29.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:28 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T15:55:29.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:28 smithi067 ceph-mon[31707]: osdmap e26: 4 total, 3 up, 4 in 2024-03-28T15:55:29.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:28 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T15:55:29.225 INFO:journalctl@ceph.osd.3.smithi067.stdout:Mar 28 15:55:28 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-3[52980]: 2024-03-28T15:55:28.946+0000 7fc679200700 -1 osd.3 0 waiting for initial osdmap 2024-03-28T15:55:29.225 INFO:journalctl@ceph.osd.3.smithi067.stdout:Mar 28 15:55:28 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-3[52980]: 2024-03-28T15:55:28.976+0000 7fc66f000700 -1 osd.3 26 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T15:55:29.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:28 smithi183 ceph-mon[35580]: from='osd.3 [v2:172.21.15.67:6810/1933541875,v1:172.21.15.67:6811/1933541875]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-03-28T15:55:29.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:28 smithi183 ceph-mon[35580]: osdmap e25: 4 total, 3 up, 4 in 2024-03-28T15:55:29.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:28 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T15:55:29.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:28 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T15:55:29.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:28 smithi183 ceph-mon[35580]: osdmap e26: 4 total, 3 up, 4 in 2024-03-28T15:55:29.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:28 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T15:55:30.018 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:29 smithi150 ceph-mon[34825]: purged_snaps scrub starts 2024-03-28T15:55:30.019 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:29 smithi150 ceph-mon[34825]: purged_snaps scrub ok 2024-03-28T15:55:30.019 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:29 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T15:55:30.019 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:29 smithi150 ceph-mon[34825]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:55:30.019 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:29 smithi150 ceph-mon[34825]: from='osd.3 [v2:172.21.15.67:6810/1933541875,v1:172.21.15.67:6811/1933541875]' entity='osd.3' 2024-03-28T15:55:30.173 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:29 smithi067 ceph-mon[31707]: purged_snaps scrub starts 2024-03-28T15:55:30.173 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:29 smithi067 ceph-mon[31707]: purged_snaps scrub ok 2024-03-28T15:55:30.173 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:29 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T15:55:30.173 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:29 smithi067 ceph-mon[31707]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:55:30.173 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:29 smithi067 ceph-mon[31707]: from='osd.3 [v2:172.21.15.67:6810/1933541875,v1:172.21.15.67:6811/1933541875]' entity='osd.3' 2024-03-28T15:55:30.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:29 smithi183 ceph-mon[35580]: purged_snaps scrub starts 2024-03-28T15:55:30.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:29 smithi183 ceph-mon[35580]: purged_snaps scrub ok 2024-03-28T15:55:30.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:29 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T15:55:30.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:29 smithi183 ceph-mon[35580]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:55:30.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:29 smithi183 ceph-mon[35580]: from='osd.3 [v2:172.21.15.67:6810/1933541875,v1:172.21.15.67:6811/1933541875]' entity='osd.3' 2024-03-28T15:55:31.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:30 smithi150 ceph-mon[34825]: osd.3 [v2:172.21.15.67:6810/1933541875,v1:172.21.15.67:6811/1933541875] boot 2024-03-28T15:55:31.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:30 smithi150 ceph-mon[34825]: osdmap e27: 4 total, 4 up, 4 in 2024-03-28T15:55:31.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:30 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T15:55:31.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:30 smithi150 ceph-mon[34825]: osdmap e28: 4 total, 4 up, 4 in 2024-03-28T15:55:31.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:30 smithi067 ceph-mon[31707]: osd.3 [v2:172.21.15.67:6810/1933541875,v1:172.21.15.67:6811/1933541875] boot 2024-03-28T15:55:31.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:30 smithi067 ceph-mon[31707]: osdmap e27: 4 total, 4 up, 4 in 2024-03-28T15:55:31.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:30 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T15:55:31.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:30 smithi067 ceph-mon[31707]: osdmap e28: 4 total, 4 up, 4 in 2024-03-28T15:55:31.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:30 smithi183 ceph-mon[35580]: osd.3 [v2:172.21.15.67:6810/1933541875,v1:172.21.15.67:6811/1933541875] boot 2024-03-28T15:55:31.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:30 smithi183 ceph-mon[35580]: osdmap e27: 4 total, 4 up, 4 in 2024-03-28T15:55:31.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:30 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T15:55:31.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:30 smithi183 ceph-mon[35580]: osdmap e28: 4 total, 4 up, 4 in 2024-03-28T15:55:31.605 INFO:teuthology.orchestra.run.smithi150.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-28T15:55:31.605 INFO:teuthology.orchestra.run.smithi150.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-28T15:55:31.605 INFO:teuthology.orchestra.run.smithi150.stderr: stderr: 10+0 records in 2024-03-28T15:55:31.605 INFO:teuthology.orchestra.run.smithi150.stderr:10+0 records out 2024-03-28T15:55:31.605 INFO:teuthology.orchestra.run.smithi150.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0382416 s, 274 MB/s 2024-03-28T15:55:31.605 INFO:teuthology.orchestra.run.smithi150.stderr:--> Zapping successful for: 2024-03-28T15:55:32.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:31 smithi150 ceph-mon[34825]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-28T15:55:32.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:31 smithi150 ceph-mon[34825]: osdmap e29: 4 total, 4 up, 4 in 2024-03-28T15:55:32.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:31 smithi067 ceph-mon[31707]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-28T15:55:32.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:31 smithi067 ceph-mon[31707]: osdmap e29: 4 total, 4 up, 4 in 2024-03-28T15:55:32.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:31 smithi183 ceph-mon[35580]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-28T15:55:32.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:31 smithi183 ceph-mon[35580]: osdmap e29: 4 total, 4 up, 4 in 2024-03-28T15:55:32.397 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi150:vg_nvme/lv_3 2024-03-28T15:55:32.827 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T15:55:34.037 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:33 smithi150 ceph-mon[34825]: pgmap v137: 1 pgs: 1 remapped+peering; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-28T15:55:34.038 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:33 smithi150 ceph-mon[34825]: osdmap e30: 4 total, 4 up, 4 in 2024-03-28T15:55:34.038 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:33 smithi150 ceph-mon[34825]: Detected new or changed devices on smithi067 2024-03-28T15:55:34.038 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:33 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:55:34.038 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:33 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:55:34.038 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:33 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:55:34.038 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:33 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:55:34.038 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:33 smithi150 ceph-mon[34825]: Adjusting osd_memory_target on smithi067 to 8545M 2024-03-28T15:55:34.038 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:33 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:55:34.038 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:33 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:55:34.038 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:33 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:55:34.038 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:33 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:55:34.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:33 smithi183 ceph-mon[35580]: pgmap v137: 1 pgs: 1 remapped+peering; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-28T15:55:34.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:33 smithi183 ceph-mon[35580]: osdmap e30: 4 total, 4 up, 4 in 2024-03-28T15:55:34.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:33 smithi183 ceph-mon[35580]: Detected new or changed devices on smithi067 2024-03-28T15:55:34.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:33 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:55:34.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:33 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:55:34.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:33 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:55:34.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:33 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:55:34.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:33 smithi183 ceph-mon[35580]: Adjusting osd_memory_target on smithi067 to 8545M 2024-03-28T15:55:34.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:33 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:55:34.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:33 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:55:34.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:33 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:55:34.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:33 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:55:34.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:33 smithi067 ceph-mon[31707]: pgmap v137: 1 pgs: 1 remapped+peering; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-28T15:55:34.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:33 smithi067 ceph-mon[31707]: osdmap e30: 4 total, 4 up, 4 in 2024-03-28T15:55:34.223 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:33 smithi067 ceph-mon[31707]: Detected new or changed devices on smithi067 2024-03-28T15:55:34.223 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:33 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:55:34.223 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:33 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:55:34.223 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:33 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:55:34.223 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:33 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:55:34.223 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:33 smithi067 ceph-mon[31707]: Adjusting osd_memory_target on smithi067 to 8545M 2024-03-28T15:55:34.223 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:33 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:55:34.223 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:33 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:55:34.223 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:33 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:55:34.223 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:33 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:55:34.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:55:34.855+0000 7fe527800700 1 -- 172.21.15.150:0/526673974 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5200761c0 msgr2=0x7fe520074620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:55:34.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:55:34.855+0000 7fe527800700 1 --2- 172.21.15.150:0/526673974 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5200761c0 0x7fe520074620 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fe514009990 tx=0x7fe514009ca0 comp rx=0 tx=0).stop 2024-03-28T15:55:34.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:55:34.855+0000 7fe527800700 1 -- 172.21.15.150:0/526673974 shutdown_connections 2024-03-28T15:55:34.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:55:34.855+0000 7fe527800700 1 --2- 172.21.15.150:0/526673974 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe52010a090 0x7fe52010c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:34.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:55:34.855+0000 7fe527800700 1 --2- 172.21.15.150:0/526673974 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5200761c0 0x7fe520074620 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:34.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:55:34.855+0000 7fe527800700 1 --2- 172.21.15.150:0/526673974 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe520074bf0 0x7fe520075070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:34.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:55:34.855+0000 7fe527800700 1 -- 172.21.15.150:0/526673974 >> 172.21.15.150:0/526673974 conn(0x7fe5200fd7a0 msgr2=0x7fe5200ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:55:34.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:55:34.855+0000 7fe527800700 1 -- 172.21.15.150:0/526673974 shutdown_connections 2024-03-28T15:55:34.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:55:34.856+0000 7fe527800700 1 -- 172.21.15.150:0/526673974 wait complete. 2024-03-28T15:55:34.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:55:34.857+0000 7fe527800700 1 Processor -- start 2024-03-28T15:55:34.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:55:34.857+0000 7fe527800700 1 -- start start 2024-03-28T15:55:34.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:55:34.857+0000 7fe527800700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe520074bf0 0x7fe52019a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:55:34.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:55:34.857+0000 7fe527800700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe5200761c0 0x7fe52019ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:55:34.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:55:34.857+0000 7fe527800700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe52010a090 0x7fe5201a1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:55:34.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:55:34.857+0000 7fe527800700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe52010ff20 con 0x7fe52010a090 2024-03-28T15:55:34.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:55:34.857+0000 7fe527800700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe52010fda0 con 0x7fe520074bf0 2024-03-28T15:55:34.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:55:34.857+0000 7fe527800700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5201100a0 con 0x7fe5200761c0 2024-03-28T15:55:34.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:55:34.857+0000 7fe524e00700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe520074bf0 0x7fe52019a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:55:34.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:55:34.857+0000 7fe524e00700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe520074bf0 0x7fe52019a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45278/0 (socket says 172.21.15.150:45278) 2024-03-28T15:55:34.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:55:34.858+0000 7fe524e00700 1 -- 172.21.15.150:0/2657584120 learned_addr learned my addr 172.21.15.150:0/2657584120 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-28T15:55:34.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:55:34.858+0000 7fe51fe00700 1 --2- 172.21.15.150:0/2657584120 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe5200761c0 0x7fe52019ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:55:34.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:55:34.858+0000 7fe525800700 1 --2- 172.21.15.150:0/2657584120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe52010a090 0x7fe5201a1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:55:34.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:55:34.858+0000 7fe524e00700 1 -- 172.21.15.150:0/2657584120 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe5200761c0 msgr2=0x7fe52019ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:55:34.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:55:34.858+0000 7fe524e00700 1 --2- 172.21.15.150:0/2657584120 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe5200761c0 0x7fe52019ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:34.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:55:34.858+0000 7fe524e00700 1 -- 172.21.15.150:0/2657584120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe52010a090 msgr2=0x7fe5201a1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:55:34.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:55:34.858+0000 7fe524e00700 1 --2- 172.21.15.150:0/2657584120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe52010a090 0x7fe5201a1bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:34.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:55:34.858+0000 7fe524e00700 1 -- 172.21.15.150:0/2657584120 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5201a2260 con 0x7fe520074bf0 2024-03-28T15:55:34.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:55:34.858+0000 7fe524e00700 1 --2- 172.21.15.150:0/2657584120 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe520074bf0 0x7fe52019a600 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe514003780 tx=0x7fe5140047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:55:34.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:55:34.859+0000 7fe51d600700 1 -- 172.21.15.150:0/2657584120 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe514027070 con 0x7fe520074bf0 2024-03-28T15:55:34.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:55:34.859+0000 7fe51d600700 1 -- 172.21.15.150:0/2657584120 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe514022470 con 0x7fe520074bf0 2024-03-28T15:55:34.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:55:34.859+0000 7fe51d600700 1 -- 172.21.15.150:0/2657584120 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe51400f6c0 con 0x7fe520074bf0 2024-03-28T15:55:34.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:55:34.859+0000 7fe527800700 1 -- 172.21.15.150:0/2657584120 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5201a24f0 con 0x7fe520074bf0 2024-03-28T15:55:34.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:55:34.859+0000 7fe527800700 1 -- 172.21.15.150:0/2657584120 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5201a29a0 con 0x7fe520074bf0 2024-03-28T15:55:34.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:55:34.860+0000 7fe527800700 1 -- 172.21.15.150:0/2657584120 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe520068e50 con 0x7fe520074bf0 2024-03-28T15:55:34.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:55:34.862+0000 7fe51d600700 1 -- 172.21.15.150:0/2657584120 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fe51402a070 con 0x7fe520074bf0 2024-03-28T15:55:34.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:55:34.863+0000 7fe51d600700 1 --2- 172.21.15.150:0/2657584120 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fe5080a2910 0x7fe5080a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:55:34.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:55:34.864+0000 7fe51d600700 1 -- 172.21.15.150:0/2657584120 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 3669+0+0 (secure 0 0 0) 0x7fe5140cebb0 con 0x7fe520074bf0 2024-03-28T15:55:34.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:55:34.864+0000 7fe51fe00700 1 --2- 172.21.15.150:0/2657584120 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fe5080a2910 0x7fe5080a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:55:34.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:55:34.865+0000 7fe51fe00700 1 --2- 172.21.15.150:0/2657584120 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fe5080a2910 0x7fe5080a4dd0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe50c005950 tx=0x7fe50c00a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:55:34.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:55:34.867+0000 7fe51d600700 1 -- 172.21.15.150:0/2657584120 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe514013070 con 0x7fe520074bf0 2024-03-28T15:55:35.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:55:35.098+0000 7fe527800700 1 -- 172.21.15.150:0/2657584120 --> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fe5200625d0 con 0x7fe5080a2910 2024-03-28T15:55:35.244 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:34 smithi067 ceph-mon[31707]: osdmap e31: 4 total, 4 up, 4 in 2024-03-28T15:55:35.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:34 smithi150 ceph-mon[34825]: osdmap e31: 4 total, 4 up, 4 in 2024-03-28T15:55:35.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:34 smithi183 ceph-mon[35580]: osdmap e31: 4 total, 4 up, 4 in 2024-03-28T15:55:36.240 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:35 smithi150 ceph-mon[34825]: pgmap v139: 1 pgs: 1 remapped+peering; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-28T15:55:36.241 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:35 smithi150 ceph-mon[34825]: from='client.34208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:55:36.241 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:35 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:55:36.241 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:35 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:55:36.241 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:35 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:55:36.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:35 smithi067 ceph-mon[31707]: pgmap v139: 1 pgs: 1 remapped+peering; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-28T15:55:36.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:35 smithi067 ceph-mon[31707]: from='client.34208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:55:36.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:35 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:55:36.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:35 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:55:36.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:35 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:55:36.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:35 smithi183 ceph-mon[35580]: pgmap v139: 1 pgs: 1 remapped+peering; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-28T15:55:36.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:35 smithi183 ceph-mon[35580]: from='client.34208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:55:36.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:35 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:55:36.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:35 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:55:36.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:35 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:55:38.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:37 smithi150 ceph-mon[34825]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T15:55:38.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:37 smithi067 ceph-mon[31707]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T15:55:38.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:37 smithi183 ceph-mon[35580]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T15:55:39.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:38 smithi150 ceph-mon[34825]: from='client.? 172.21.15.150:0/1600463452' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5cc3e260-83a3-4a4d-aa6a-34a3b12b2b5f"}]: dispatch 2024-03-28T15:55:39.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:38 smithi150 ceph-mon[34825]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5cc3e260-83a3-4a4d-aa6a-34a3b12b2b5f"}]: dispatch 2024-03-28T15:55:39.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:38 smithi150 ceph-mon[34825]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5cc3e260-83a3-4a4d-aa6a-34a3b12b2b5f"}]': finished 2024-03-28T15:55:39.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:38 smithi150 ceph-mon[34825]: osdmap e32: 5 total, 4 up, 5 in 2024-03-28T15:55:39.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:38 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T15:55:39.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:38 smithi067 ceph-mon[31707]: from='client.? 172.21.15.150:0/1600463452' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5cc3e260-83a3-4a4d-aa6a-34a3b12b2b5f"}]: dispatch 2024-03-28T15:55:39.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:38 smithi067 ceph-mon[31707]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5cc3e260-83a3-4a4d-aa6a-34a3b12b2b5f"}]: dispatch 2024-03-28T15:55:39.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:38 smithi067 ceph-mon[31707]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5cc3e260-83a3-4a4d-aa6a-34a3b12b2b5f"}]': finished 2024-03-28T15:55:39.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:38 smithi067 ceph-mon[31707]: osdmap e32: 5 total, 4 up, 5 in 2024-03-28T15:55:39.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:38 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T15:55:39.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:38 smithi183 ceph-mon[35580]: from='client.? 172.21.15.150:0/1600463452' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5cc3e260-83a3-4a4d-aa6a-34a3b12b2b5f"}]: dispatch 2024-03-28T15:55:39.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:38 smithi183 ceph-mon[35580]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5cc3e260-83a3-4a4d-aa6a-34a3b12b2b5f"}]: dispatch 2024-03-28T15:55:39.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:38 smithi183 ceph-mon[35580]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5cc3e260-83a3-4a4d-aa6a-34a3b12b2b5f"}]': finished 2024-03-28T15:55:39.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:38 smithi183 ceph-mon[35580]: osdmap e32: 5 total, 4 up, 5 in 2024-03-28T15:55:39.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:38 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T15:55:40.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:40 smithi150 ceph-mon[34825]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T15:55:40.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:40 smithi150 ceph-mon[34825]: from='client.? 172.21.15.150:0/2323687053' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:55:40.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:39 smithi067 ceph-mon[31707]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T15:55:40.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:39 smithi067 ceph-mon[31707]: from='client.? 172.21.15.150:0/2323687053' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:55:40.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:39 smithi183 ceph-mon[35580]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T15:55:40.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:39 smithi183 ceph-mon[35580]: from='client.? 172.21.15.150:0/2323687053' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:55:42.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:41 smithi150 ceph-mon[34825]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-28T15:55:42.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:41 smithi067 ceph-mon[31707]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-28T15:55:42.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:41 smithi183 ceph-mon[35580]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-28T15:55:44.277 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:44 smithi150 ceph-mon[34825]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-03-28T15:55:44.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:44 smithi067 ceph-mon[31707]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-03-28T15:55:44.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:44 smithi183 ceph-mon[35580]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-03-28T15:55:46.264 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:46 smithi150 ceph-mon[34825]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-28T15:55:46.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:46 smithi067 ceph-mon[31707]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-28T15:55:46.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:46 smithi183 ceph-mon[35580]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-28T15:55:48.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:48 smithi150 ceph-mon[34825]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-28T15:55:48.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:48 smithi067 ceph-mon[31707]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-28T15:55:48.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:48 smithi183 ceph-mon[35580]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-28T15:55:49.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:49 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-28T15:55:49.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:49 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:55:49.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:49 smithi150 ceph-mon[34825]: Deploying daemon osd.4 on smithi150 2024-03-28T15:55:49.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:49 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-28T15:55:49.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:49 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:55:49.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:49 smithi067 ceph-mon[31707]: Deploying daemon osd.4 on smithi150 2024-03-28T15:55:49.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:49 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-28T15:55:49.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:49 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:55:49.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:49 smithi183 ceph-mon[35580]: Deploying daemon osd.4 on smithi150 2024-03-28T15:55:50.238 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:50 smithi150 ceph-mon[34825]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-28T15:55:50.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:50 smithi067 ceph-mon[31707]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-28T15:55:50.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:50 smithi183 ceph-mon[35580]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-28T15:55:52.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:52 smithi150 ceph-mon[34825]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-28T15:55:52.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:52 smithi067 ceph-mon[31707]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-28T15:55:52.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:52 smithi183 ceph-mon[35580]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-28T15:55:54.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:54 smithi150 ceph-mon[34825]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-28T15:55:54.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:54 smithi067 ceph-mon[31707]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-28T15:55:54.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:54 smithi183 ceph-mon[35580]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-28T15:55:56.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:56 smithi150 ceph-mon[34825]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-28T15:55:56.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:56 smithi067 ceph-mon[31707]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-28T15:55:56.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:56 smithi183 ceph-mon[35580]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-28T15:55:57.466 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:57 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:55:57.466 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:57 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:55:57.466 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:57 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:55:57.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:57 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:55:57.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:57 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:55:57.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:57 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:55:57.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:57 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:55:57.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:57 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:55:57.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:57 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:55:58.429 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:55:58 smithi150 ceph-mon[34825]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-28T15:55:58.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:55:58 smithi067 ceph-mon[31707]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-28T15:55:58.816 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:55:58 smithi183 ceph-mon[35580]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-28T15:56:00.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:00 smithi150 ceph-mon[34825]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-28T15:56:00.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:00 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:56:00.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:00 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:56:00.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:00 smithi150 ceph-mon[34825]: from='osd.4 [v2:172.21.15.150:6808/3775269993,v1:172.21.15.150:6809/3775269993]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-28T15:56:00.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:00 smithi150 ceph-mon[34825]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-28T15:56:00.971 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:56:00.968+0000 7fe51d600700 1 -- 172.21.15.150:0/2657584120 <== mgr.14152 v2:172.21.15.67:6800/208726654 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fe5200625d0 con 0x7fe5080a2910 2024-03-28T15:56:00.971 INFO:teuthology.orchestra.run.smithi150.stdout:Created osd(s) 4 on host 'smithi150' 2024-03-28T15:56:00.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:56:00.971+0000 7fe527800700 1 -- 172.21.15.150:0/2657584120 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fe5080a2910 msgr2=0x7fe5080a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:56:00.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:56:00.971+0000 7fe527800700 1 --2- 172.21.15.150:0/2657584120 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fe5080a2910 0x7fe5080a4dd0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe50c005950 tx=0x7fe50c00a300 comp rx=0 tx=0).stop 2024-03-28T15:56:00.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:56:00.971+0000 7fe527800700 1 -- 172.21.15.150:0/2657584120 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe520074bf0 msgr2=0x7fe52019a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:56:00.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:56:00.971+0000 7fe527800700 1 --2- 172.21.15.150:0/2657584120 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe520074bf0 0x7fe52019a600 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe514003780 tx=0x7fe5140047b0 comp rx=0 tx=0).stop 2024-03-28T15:56:00.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:56:00.972+0000 7fe527800700 1 -- 172.21.15.150:0/2657584120 shutdown_connections 2024-03-28T15:56:00.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:56:00.972+0000 7fe527800700 1 --2- 172.21.15.150:0/2657584120 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fe5080a2910 0x7fe5080a4dd0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:56:00.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:56:00.972+0000 7fe527800700 1 --2- 172.21.15.150:0/2657584120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe52010a090 0x7fe5201a1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:56:00.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:56:00.972+0000 7fe527800700 1 --2- 172.21.15.150:0/2657584120 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe520074bf0 0x7fe52019a600 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:56:00.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:56:00.972+0000 7fe527800700 1 --2- 172.21.15.150:0/2657584120 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe5200761c0 0x7fe52019ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:56:00.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:56:00.972+0000 7fe527800700 1 -- 172.21.15.150:0/2657584120 >> 172.21.15.150:0/2657584120 conn(0x7fe5200fd7a0 msgr2=0x7fe520105f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:56:00.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:56:00.972+0000 7fe527800700 1 -- 172.21.15.150:0/2657584120 shutdown_connections 2024-03-28T15:56:00.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:56:00.972+0000 7fe527800700 1 -- 172.21.15.150:0/2657584120 wait complete. 2024-03-28T15:56:01.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:00 smithi067 ceph-mon[31707]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-28T15:56:01.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:00 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:56:01.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:00 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:56:01.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:00 smithi067 ceph-mon[31707]: from='osd.4 [v2:172.21.15.150:6808/3775269993,v1:172.21.15.150:6809/3775269993]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-28T15:56:01.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:00 smithi067 ceph-mon[31707]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-28T15:56:01.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:00 smithi183 ceph-mon[35580]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-28T15:56:01.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:00 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:56:01.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:00 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:56:01.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:00 smithi183 ceph-mon[35580]: from='osd.4 [v2:172.21.15.150:6808/3775269993,v1:172.21.15.150:6809/3775269993]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-28T15:56:01.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:00 smithi183 ceph-mon[35580]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-28T15:56:01.800 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:01 smithi150 ceph-mon[34825]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-28T15:56:01.800 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:01 smithi150 ceph-mon[34825]: from='osd.4 [v2:172.21.15.150:6808/3775269993,v1:172.21.15.150:6809/3775269993]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-03-28T15:56:01.800 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:01 smithi150 ceph-mon[34825]: osdmap e33: 5 total, 4 up, 5 in 2024-03-28T15:56:01.800 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:01 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T15:56:01.800 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:01 smithi150 ceph-mon[34825]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-03-28T15:56:01.800 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:01 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:56:01.801 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:01 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:56:01.868 DEBUG:teuthology.orchestra.run.smithi150:osd.4> sudo journalctl -f -n 0 -u ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@osd.4.service 2024-03-28T15:56:01.873 INFO:tasks.cephadm:Deploying osd.5 on smithi183 with /dev/vg_nvme/lv_3... 2024-03-28T15:56:01.873 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-28T15:56:01.962 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:01 smithi183 ceph-mon[35580]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-28T15:56:01.962 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:01 smithi183 ceph-mon[35580]: from='osd.4 [v2:172.21.15.150:6808/3775269993,v1:172.21.15.150:6809/3775269993]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-03-28T15:56:01.963 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:01 smithi183 ceph-mon[35580]: osdmap e33: 5 total, 4 up, 5 in 2024-03-28T15:56:01.963 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:01 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T15:56:01.963 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:01 smithi183 ceph-mon[35580]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-03-28T15:56:01.963 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:01 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:56:01.963 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:01 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:56:02.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:01 smithi067 ceph-mon[31707]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-28T15:56:02.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:01 smithi067 ceph-mon[31707]: from='osd.4 [v2:172.21.15.150:6808/3775269993,v1:172.21.15.150:6809/3775269993]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-03-28T15:56:02.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:01 smithi067 ceph-mon[31707]: osdmap e33: 5 total, 4 up, 5 in 2024-03-28T15:56:02.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:01 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T15:56:02.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:01 smithi067 ceph-mon[31707]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-03-28T15:56:02.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:01 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:56:02.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:01 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:56:02.313 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T15:56:02.708 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:02 smithi150 ceph-mon[34825]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-28T15:56:02.708 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:02 smithi150 ceph-mon[34825]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-03-28T15:56:02.708 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:02 smithi150 ceph-mon[34825]: osdmap e34: 5 total, 4 up, 5 in 2024-03-28T15:56:02.708 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:02 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T15:56:02.709 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:02 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T15:56:02.718 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:02 smithi183 ceph-mon[35580]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-28T15:56:02.719 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:02 smithi183 ceph-mon[35580]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-03-28T15:56:02.719 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:02 smithi183 ceph-mon[35580]: osdmap e34: 5 total, 4 up, 5 in 2024-03-28T15:56:02.719 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:02 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T15:56:02.719 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:02 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T15:56:03.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:02 smithi067 ceph-mon[31707]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-28T15:56:03.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:02 smithi067 ceph-mon[31707]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-03-28T15:56:03.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:02 smithi067 ceph-mon[31707]: osdmap e34: 5 total, 4 up, 5 in 2024-03-28T15:56:03.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:02 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T15:56:03.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:02 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T15:56:03.955 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:03 smithi150 ceph-mon[34825]: purged_snaps scrub starts 2024-03-28T15:56:03.956 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:03 smithi150 ceph-mon[34825]: purged_snaps scrub ok 2024-03-28T15:56:03.956 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:03 smithi150 ceph-mon[34825]: osd.4 [v2:172.21.15.150:6808/3775269993,v1:172.21.15.150:6809/3775269993] boot 2024-03-28T15:56:03.956 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:03 smithi150 ceph-mon[34825]: osdmap e35: 5 total, 5 up, 5 in 2024-03-28T15:56:03.956 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:03 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T15:56:04.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:03 smithi067 ceph-mon[31707]: purged_snaps scrub starts 2024-03-28T15:56:04.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:03 smithi067 ceph-mon[31707]: purged_snaps scrub ok 2024-03-28T15:56:04.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:03 smithi067 ceph-mon[31707]: osd.4 [v2:172.21.15.150:6808/3775269993,v1:172.21.15.150:6809/3775269993] boot 2024-03-28T15:56:04.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:03 smithi067 ceph-mon[31707]: osdmap e35: 5 total, 5 up, 5 in 2024-03-28T15:56:04.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:03 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T15:56:04.072 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:03 smithi183 ceph-mon[35580]: purged_snaps scrub starts 2024-03-28T15:56:04.072 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:03 smithi183 ceph-mon[35580]: purged_snaps scrub ok 2024-03-28T15:56:04.072 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:03 smithi183 ceph-mon[35580]: osd.4 [v2:172.21.15.150:6808/3775269993,v1:172.21.15.150:6809/3775269993] boot 2024-03-28T15:56:04.072 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:03 smithi183 ceph-mon[35580]: osdmap e35: 5 total, 5 up, 5 in 2024-03-28T15:56:04.072 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:03 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T15:56:04.852 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:04 smithi183 ceph-mon[35580]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-28T15:56:04.852 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:04 smithi183 ceph-mon[35580]: osdmap e36: 5 total, 5 up, 5 in 2024-03-28T15:56:04.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:04 smithi067 ceph-mon[31707]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-28T15:56:04.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:04 smithi067 ceph-mon[31707]: osdmap e36: 5 total, 5 up, 5 in 2024-03-28T15:56:05.055 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:04 smithi150 ceph-mon[34825]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-28T15:56:05.055 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:04 smithi150 ceph-mon[34825]: osdmap e36: 5 total, 5 up, 5 in 2024-03-28T15:56:05.294 INFO:teuthology.orchestra.run.smithi183.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-28T15:56:05.294 INFO:teuthology.orchestra.run.smithi183.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-28T15:56:05.294 INFO:teuthology.orchestra.run.smithi183.stderr: stderr: 10+0 records in 2024-03-28T15:56:05.294 INFO:teuthology.orchestra.run.smithi183.stderr:10+0 records out 2024-03-28T15:56:05.294 INFO:teuthology.orchestra.run.smithi183.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0370907 s, 283 MB/s 2024-03-28T15:56:05.294 INFO:teuthology.orchestra.run.smithi183.stderr:--> Zapping successful for: 2024-03-28T15:56:06.184 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi183:vg_nvme/lv_3 2024-03-28T15:56:06.616 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T15:56:06.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:06 smithi150 ceph-mon[34825]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:56:06.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:06 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:56:06.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:06 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:56:06.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:06 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:56:06.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:06 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:56:06.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:06 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:56:06.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:06 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:56:06.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:06 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:56:06.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:06 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:56:06.921 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:06 smithi183 ceph-mon[35580]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:56:06.921 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:06 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:56:06.921 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:06 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:56:06.921 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:06 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:56:06.921 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:06 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:56:06.921 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:06 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:56:06.922 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:06 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:56:06.922 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:06 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:56:06.922 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:06 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:56:07.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:06 smithi067 ceph-mon[31707]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:56:07.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:06 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:56:07.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:06 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:56:07.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:06 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:56:07.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:06 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:56:07.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:06 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:56:07.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:06 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:56:07.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:06 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:56:07.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:06 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:56:07.764 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:07 smithi183 ceph-mon[35580]: Detected new or changed devices on smithi150 2024-03-28T15:56:07.764 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:07 smithi183 ceph-mon[35580]: Adjusting osd_memory_target on smithi150 to 8545M 2024-03-28T15:56:07.764 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:07 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:56:07.764 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:07 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:56:07.765 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:07 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:56:07.765 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:07 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:56:08.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:07 smithi150 ceph-mon[34825]: Detected new or changed devices on smithi150 2024-03-28T15:56:08.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:07 smithi150 ceph-mon[34825]: Adjusting osd_memory_target on smithi150 to 8545M 2024-03-28T15:56:08.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:07 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:56:08.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:07 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:56:08.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:07 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:56:08.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:07 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:56:08.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:07 smithi067 ceph-mon[31707]: Detected new or changed devices on smithi150 2024-03-28T15:56:08.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:07 smithi067 ceph-mon[31707]: Adjusting osd_memory_target on smithi150 to 8545M 2024-03-28T15:56:08.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:07 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:56:08.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:07 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:56:08.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:07 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:56:08.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:07 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:56:08.657 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:56:08.655+0000 7f8320a00700 1 -- 172.21.15.183:0/3946085696 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f831c104410 msgr2=0x7f831c104830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:56:08.657 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:56:08.655+0000 7f8320a00700 1 --2- 172.21.15.183:0/3946085696 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f831c104410 0x7f831c104830 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f830c009990 tx=0x7f830c009ca0 comp rx=0 tx=0).stop 2024-03-28T15:56:08.657 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:56:08.656+0000 7f8320a00700 1 -- 172.21.15.183:0/3946085696 shutdown_connections 2024-03-28T15:56:08.658 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:56:08.656+0000 7f8320a00700 1 --2- 172.21.15.183:0/3946085696 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f831c105fb0 0x7f831c10c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:56:08.658 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:56:08.656+0000 7f8320a00700 1 --2- 172.21.15.183:0/3946085696 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f831c104410 0x7f831c104830 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:56:08.658 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:56:08.656+0000 7f8320a00700 1 --2- 172.21.15.183:0/3946085696 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f831c105610 0x7f831c105a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:56:08.658 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:56:08.656+0000 7f8320a00700 1 -- 172.21.15.183:0/3946085696 >> 172.21.15.183:0/3946085696 conn(0x7f831c0ff9b0 msgr2=0x7f831c101df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:56:08.658 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:56:08.656+0000 7f8320a00700 1 -- 172.21.15.183:0/3946085696 shutdown_connections 2024-03-28T15:56:08.658 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:56:08.656+0000 7f8320a00700 1 -- 172.21.15.183:0/3946085696 wait complete. 2024-03-28T15:56:08.660 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:56:08.657+0000 7f8320a00700 1 Processor -- start 2024-03-28T15:56:08.660 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:56:08.657+0000 7f8320a00700 1 -- start start 2024-03-28T15:56:08.660 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:56:08.657+0000 7f8320a00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f831c104410 0x7f831c19a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:56:08.660 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:56:08.657+0000 7f8320a00700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f831c105610 0x7f831c19aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:56:08.660 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:56:08.657+0000 7f8320a00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f831c105fb0 0x7f831c1a1b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:56:08.660 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:56:08.657+0000 7f8320a00700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f831c10ff00 con 0x7f831c104410 2024-03-28T15:56:08.660 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:56:08.657+0000 7f8320a00700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f831c10fd80 con 0x7f831c105610 2024-03-28T15:56:08.660 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:56:08.657+0000 7f8320a00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f831c110080 con 0x7f831c105fb0 2024-03-28T15:56:08.660 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:56:08.658+0000 7f831a800700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f831c105fb0 0x7f831c1a1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:56:08.661 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:56:08.658+0000 7f831a800700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f831c105fb0 0x7f831c1a1b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.183:60738/0 (socket says 172.21.15.183:60738) 2024-03-28T15:56:08.661 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:56:08.658+0000 7f831a800700 1 -- 172.21.15.183:0/3335596108 learned_addr learned my addr 172.21.15.183:0/3335596108 (peer_addr_for_me v2:172.21.15.183:0/0) 2024-03-28T15:56:08.661 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:56:08.658+0000 7f8319e00700 1 --2- 172.21.15.183:0/3335596108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f831c104410 0x7f831c19a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:56:08.661 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:56:08.658+0000 7f8319400700 1 --2- 172.21.15.183:0/3335596108 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f831c105610 0x7f831c19aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:56:08.661 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:56:08.658+0000 7f831a800700 1 -- 172.21.15.183:0/3335596108 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f831c105610 msgr2=0x7f831c19aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:56:08.661 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:56:08.658+0000 7f831a800700 1 --2- 172.21.15.183:0/3335596108 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f831c105610 0x7f831c19aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:56:08.661 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:56:08.658+0000 7f831a800700 1 -- 172.21.15.183:0/3335596108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f831c104410 msgr2=0x7f831c19a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:56:08.661 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:56:08.658+0000 7f831a800700 1 --2- 172.21.15.183:0/3335596108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f831c104410 0x7f831c19a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:56:08.661 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:56:08.658+0000 7f831a800700 1 -- 172.21.15.183:0/3335596108 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f831c1a21a0 con 0x7f831c105fb0 2024-03-28T15:56:08.661 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:56:08.658+0000 7f831a800700 1 --2- 172.21.15.183:0/3335596108 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f831c105fb0 0x7f831c1a1b00 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f831400add0 tx=0x7f831400c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:56:08.661 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:56:08.659+0000 7f8312a00700 1 -- 172.21.15.183:0/3335596108 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8314017020 con 0x7f831c105fb0 2024-03-28T15:56:08.661 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:56:08.659+0000 7f8312a00700 1 -- 172.21.15.183:0/3335596108 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f831400cdf0 con 0x7f831c105fb0 2024-03-28T15:56:08.661 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:56:08.659+0000 7f8320a00700 1 -- 172.21.15.183:0/3335596108 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f831c1a2490 con 0x7f831c105fb0 2024-03-28T15:56:08.661 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:56:08.659+0000 7f8312a00700 1 -- 172.21.15.183:0/3335596108 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8314005460 con 0x7f831c105fb0 2024-03-28T15:56:08.662 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:56:08.659+0000 7f8320a00700 1 -- 172.21.15.183:0/3335596108 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f831c1a29d0 con 0x7f831c105fb0 2024-03-28T15:56:08.662 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:56:08.660+0000 7f8320a00700 1 -- 172.21.15.183:0/3335596108 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f831c068e50 con 0x7f831c105fb0 2024-03-28T15:56:08.663 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:56:08.662+0000 7f8312a00700 1 -- 172.21.15.183:0/3335596108 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f8314004ad0 con 0x7f831c105fb0 2024-03-28T15:56:08.664 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:56:08.663+0000 7f8312a00700 1 --2- 172.21.15.183:0/3335596108 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f83000a2890 0x7f83000a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:56:08.664 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:56:08.664+0000 7f8312a00700 1 -- 172.21.15.183:0/3335596108 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4101+0+0 (secure 0 0 0) 0x7f83140cd1c0 con 0x7f831c105fb0 2024-03-28T15:56:08.665 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:56:08.664+0000 7f8319e00700 1 --2- 172.21.15.183:0/3335596108 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f83000a2890 0x7f83000a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:56:08.666 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:56:08.665+0000 7f8319e00700 1 --2- 172.21.15.183:0/3335596108 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f83000a2890 0x7f83000a4d50 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f830c00b5c0 tx=0x7f830c0038e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:56:08.669 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:56:08.668+0000 7f8312a00700 1 -- 172.21.15.183:0/3335596108 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8314021020 con 0x7f831c105fb0 2024-03-28T15:56:08.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:08 smithi183 ceph-mon[35580]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:56:08.901 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:56:08.900+0000 7f8320a00700 1 -- 172.21.15.183:0/3335596108 --> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f831c0625d0 con 0x7f83000a2890 2024-03-28T15:56:09.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:08 smithi150 ceph-mon[34825]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:56:09.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:08 smithi067 ceph-mon[31707]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:56:09.963 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:09 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:56:09.963 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:09 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:56:09.963 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:09 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:56:10.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:09 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:56:10.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:09 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:56:10.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:09 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:56:10.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:09 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:56:10.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:09 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:56:10.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:09 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:56:10.834 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:10 smithi183 ceph-mon[35580]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:56:10.835 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:10 smithi183 ceph-mon[35580]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:56:11.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:10 smithi150 ceph-mon[34825]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:56:11.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:10 smithi150 ceph-mon[34825]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:56:11.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:10 smithi067 ceph-mon[31707]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:56:11.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:10 smithi067 ceph-mon[31707]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:56:13.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:12 smithi150 ceph-mon[34825]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:56:13.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:12 smithi150 ceph-mon[34825]: from='client.? 172.21.15.183:0/246564790' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a2542b21-d6ac-413e-8b9b-4a6548b20840"}]: dispatch 2024-03-28T15:56:13.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:12 smithi150 ceph-mon[34825]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a2542b21-d6ac-413e-8b9b-4a6548b20840"}]: dispatch 2024-03-28T15:56:13.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:12 smithi150 ceph-mon[34825]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a2542b21-d6ac-413e-8b9b-4a6548b20840"}]': finished 2024-03-28T15:56:13.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:12 smithi150 ceph-mon[34825]: osdmap e37: 6 total, 5 up, 6 in 2024-03-28T15:56:13.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:12 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T15:56:13.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:12 smithi067 ceph-mon[31707]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:56:13.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:12 smithi067 ceph-mon[31707]: from='client.? 172.21.15.183:0/246564790' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a2542b21-d6ac-413e-8b9b-4a6548b20840"}]: dispatch 2024-03-28T15:56:13.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:12 smithi067 ceph-mon[31707]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a2542b21-d6ac-413e-8b9b-4a6548b20840"}]: dispatch 2024-03-28T15:56:13.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:12 smithi067 ceph-mon[31707]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a2542b21-d6ac-413e-8b9b-4a6548b20840"}]': finished 2024-03-28T15:56:13.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:12 smithi067 ceph-mon[31707]: osdmap e37: 6 total, 5 up, 6 in 2024-03-28T15:56:13.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:12 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T15:56:13.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:12 smithi183 ceph-mon[35580]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:56:13.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:12 smithi183 ceph-mon[35580]: from='client.? 172.21.15.183:0/246564790' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a2542b21-d6ac-413e-8b9b-4a6548b20840"}]: dispatch 2024-03-28T15:56:13.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:12 smithi183 ceph-mon[35580]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a2542b21-d6ac-413e-8b9b-4a6548b20840"}]: dispatch 2024-03-28T15:56:13.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:12 smithi183 ceph-mon[35580]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a2542b21-d6ac-413e-8b9b-4a6548b20840"}]': finished 2024-03-28T15:56:13.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:12 smithi183 ceph-mon[35580]: osdmap e37: 6 total, 5 up, 6 in 2024-03-28T15:56:13.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:12 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T15:56:14.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:13 smithi150 ceph-mon[34825]: from='client.? 172.21.15.183:0/1737915235' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:56:14.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:13 smithi067 ceph-mon[31707]: from='client.? 172.21.15.183:0/1737915235' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:56:14.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:13 smithi183 ceph-mon[35580]: from='client.? 172.21.15.183:0/1737915235' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:56:15.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:14 smithi150 ceph-mon[34825]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:56:15.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:14 smithi067 ceph-mon[31707]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:56:15.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:14 smithi183 ceph-mon[35580]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:56:17.046 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:16 smithi183 ceph-mon[35580]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:56:17.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:16 smithi150 ceph-mon[34825]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:56:17.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:16 smithi067 ceph-mon[31707]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:56:19.020 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:18 smithi183 ceph-mon[35580]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:56:19.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:18 smithi150 ceph-mon[34825]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:56:19.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:18 smithi067 ceph-mon[31707]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:56:21.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:20 smithi150 ceph-mon[34825]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:56:21.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:20 smithi067 ceph-mon[31707]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:56:21.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:20 smithi183 ceph-mon[35580]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:56:22.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:21 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-28T15:56:22.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:21 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:56:22.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:21 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-28T15:56:22.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:21 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:56:22.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:21 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-28T15:56:22.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:21 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:56:22.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:22 smithi183 ceph-mon[35580]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:56:22.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:22 smithi183 ceph-mon[35580]: Deploying daemon osd.5 on smithi183 2024-03-28T15:56:23.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:22 smithi150 ceph-mon[34825]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:56:23.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:22 smithi150 ceph-mon[34825]: Deploying daemon osd.5 on smithi183 2024-03-28T15:56:23.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:22 smithi067 ceph-mon[31707]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:56:23.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:22 smithi067 ceph-mon[31707]: Deploying daemon osd.5 on smithi183 2024-03-28T15:56:25.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:24 smithi150 ceph-mon[34825]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:56:25.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:24 smithi067 ceph-mon[31707]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:56:25.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:24 smithi183 ceph-mon[35580]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:56:27.049 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:26 smithi183 ceph-mon[35580]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:56:27.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:26 smithi150 ceph-mon[34825]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:56:27.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:26 smithi067 ceph-mon[31707]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:56:29.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:28 smithi150 ceph-mon[34825]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:56:29.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:28 smithi067 ceph-mon[31707]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:56:29.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:28 smithi183 ceph-mon[35580]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:56:31.014 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:30 smithi183 ceph-mon[35580]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:56:31.014 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:30 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:56:31.014 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:30 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:56:31.014 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:30 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:56:31.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:30 smithi150 ceph-mon[34825]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:56:31.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:30 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:56:31.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:30 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:56:31.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:30 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:56:31.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:30 smithi067 ceph-mon[31707]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:56:31.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:30 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:56:31.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:30 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:56:31.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:30 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:56:32.937 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:32 smithi183 ceph-mon[35580]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:56:33.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:32 smithi067 ceph-mon[31707]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:56:33.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:32 smithi150 ceph-mon[34825]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:56:34.306 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:33 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:56:34.306 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:33 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:56:34.306 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:33 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:56:34.306 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:33 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:56:34.306 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:33 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:56:34.306 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:33 smithi150 ceph-mon[34825]: from='osd.5 [v2:172.21.15.183:6808/164284137,v1:172.21.15.183:6809/164284137]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-28T15:56:34.307 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:33 smithi150 ceph-mon[34825]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-28T15:56:34.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:33 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:56:34.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:33 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:56:34.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:33 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:56:34.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:33 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:56:34.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:33 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:56:34.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:33 smithi067 ceph-mon[31707]: from='osd.5 [v2:172.21.15.183:6808/164284137,v1:172.21.15.183:6809/164284137]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-28T15:56:34.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:33 smithi067 ceph-mon[31707]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-28T15:56:34.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:33 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:56:34.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:33 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:56:34.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:33 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:56:34.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:33 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:56:34.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:33 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:56:34.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:33 smithi183 ceph-mon[35580]: from='osd.5 [v2:172.21.15.183:6808/164284137,v1:172.21.15.183:6809/164284137]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-28T15:56:34.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:33 smithi183 ceph-mon[35580]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-28T15:56:34.618 INFO:teuthology.orchestra.run.smithi183.stdout:Created osd(s) 5 on host 'smithi183' 2024-03-28T15:56:34.618 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:56:34.615+0000 7f8312a00700 1 -- 172.21.15.183:0/3335596108 <== mgr.14152 v2:172.21.15.67:6800/208726654 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f831c0625d0 con 0x7f83000a2890 2024-03-28T15:56:34.621 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:56:34.620+0000 7f8320a00700 1 -- 172.21.15.183:0/3335596108 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f83000a2890 msgr2=0x7f83000a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:56:34.621 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:56:34.620+0000 7f8320a00700 1 --2- 172.21.15.183:0/3335596108 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f83000a2890 0x7f83000a4d50 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f830c00b5c0 tx=0x7f830c0038e0 comp rx=0 tx=0).stop 2024-03-28T15:56:34.622 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:56:34.620+0000 7f8320a00700 1 -- 172.21.15.183:0/3335596108 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f831c105fb0 msgr2=0x7f831c1a1b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:56:34.622 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:56:34.620+0000 7f8320a00700 1 --2- 172.21.15.183:0/3335596108 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f831c105fb0 0x7f831c1a1b00 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f831400add0 tx=0x7f831400c330 comp rx=0 tx=0).stop 2024-03-28T15:56:34.622 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:56:34.621+0000 7f8320a00700 1 -- 172.21.15.183:0/3335596108 shutdown_connections 2024-03-28T15:56:34.622 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:56:34.621+0000 7f8320a00700 1 --2- 172.21.15.183:0/3335596108 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f83000a2890 0x7f83000a4d50 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:56:34.622 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:56:34.621+0000 7f8320a00700 1 --2- 172.21.15.183:0/3335596108 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f831c105fb0 0x7f831c1a1b00 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:56:34.622 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:56:34.621+0000 7f8320a00700 1 --2- 172.21.15.183:0/3335596108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f831c104410 0x7f831c19a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:56:34.622 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:56:34.621+0000 7f8320a00700 1 --2- 172.21.15.183:0/3335596108 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f831c105610 0x7f831c19aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:56:34.623 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:56:34.621+0000 7f8320a00700 1 -- 172.21.15.183:0/3335596108 >> 172.21.15.183:0/3335596108 conn(0x7f831c0ff9b0 msgr2=0x7f831c108910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:56:34.623 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:56:34.621+0000 7f8320a00700 1 -- 172.21.15.183:0/3335596108 shutdown_connections 2024-03-28T15:56:34.623 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:56:34.621+0000 7f8320a00700 1 -- 172.21.15.183:0/3335596108 wait complete. 2024-03-28T15:56:35.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:34 smithi067 ceph-mon[31707]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:56:35.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:34 smithi067 ceph-mon[31707]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-28T15:56:35.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:34 smithi067 ceph-mon[31707]: osdmap e38: 6 total, 5 up, 6 in 2024-03-28T15:56:35.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:34 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T15:56:35.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:34 smithi067 ceph-mon[31707]: from='osd.5 [v2:172.21.15.183:6808/164284137,v1:172.21.15.183:6809/164284137]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-03-28T15:56:35.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:34 smithi067 ceph-mon[31707]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-03-28T15:56:35.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:34 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:56:35.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:34 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:56:35.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:34 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:56:35.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:34 smithi150 ceph-mon[34825]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:56:35.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:34 smithi150 ceph-mon[34825]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-28T15:56:35.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:34 smithi150 ceph-mon[34825]: osdmap e38: 6 total, 5 up, 6 in 2024-03-28T15:56:35.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:34 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T15:56:35.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:34 smithi150 ceph-mon[34825]: from='osd.5 [v2:172.21.15.183:6808/164284137,v1:172.21.15.183:6809/164284137]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-03-28T15:56:35.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:34 smithi150 ceph-mon[34825]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-03-28T15:56:35.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:34 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:56:35.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:34 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:56:35.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:34 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:56:35.326 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:34 smithi183 ceph-mon[35580]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:56:35.326 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:34 smithi183 ceph-mon[35580]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-28T15:56:35.326 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:34 smithi183 ceph-mon[35580]: osdmap e38: 6 total, 5 up, 6 in 2024-03-28T15:56:35.326 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:34 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T15:56:35.327 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:34 smithi183 ceph-mon[35580]: from='osd.5 [v2:172.21.15.183:6808/164284137,v1:172.21.15.183:6809/164284137]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-03-28T15:56:35.327 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:34 smithi183 ceph-mon[35580]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-03-28T15:56:35.327 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:34 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:56:35.327 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:34 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:56:35.327 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:34 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:56:35.380 DEBUG:teuthology.orchestra.run.smithi183:osd.5> sudo journalctl -f -n 0 -u ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@osd.5.service 2024-03-28T15:56:35.386 INFO:tasks.cephadm:Deploying osd.6 on smithi067 with /dev/vg_nvme/lv_2... 2024-03-28T15:56:35.386 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-28T15:56:35.849 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:56:36.202 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:35 smithi067 ceph-mon[31707]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-03-28T15:56:36.202 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:35 smithi067 ceph-mon[31707]: osdmap e39: 6 total, 5 up, 6 in 2024-03-28T15:56:36.202 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:35 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T15:56:36.202 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:35 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T15:56:36.202 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:35 smithi067 ceph-mon[31707]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:56:36.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:35 smithi150 ceph-mon[34825]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-03-28T15:56:36.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:35 smithi150 ceph-mon[34825]: osdmap e39: 6 total, 5 up, 6 in 2024-03-28T15:56:36.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:35 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T15:56:36.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:35 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T15:56:36.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:35 smithi150 ceph-mon[34825]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:56:36.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:35 smithi183 ceph-mon[35580]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-03-28T15:56:36.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:35 smithi183 ceph-mon[35580]: osdmap e39: 6 total, 5 up, 6 in 2024-03-28T15:56:36.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:35 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T15:56:36.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:35 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T15:56:36.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:35 smithi183 ceph-mon[35580]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:56:36.351 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 28 15:56:36 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-5[47486]: 2024-03-28T15:56:36.037+0000 7fd7c8000700 -1 osd.5 0 waiting for initial osdmap 2024-03-28T15:56:36.351 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 28 15:56:36 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-5[47486]: 2024-03-28T15:56:36.068+0000 7fd7bde00700 -1 osd.5 40 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T15:56:37.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:36 smithi183 ceph-mon[35580]: purged_snaps scrub starts 2024-03-28T15:56:37.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:36 smithi183 ceph-mon[35580]: purged_snaps scrub ok 2024-03-28T15:56:37.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:36 smithi183 ceph-mon[35580]: osdmap e40: 6 total, 5 up, 6 in 2024-03-28T15:56:37.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:36 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T15:56:37.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:36 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T15:56:37.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:36 smithi183 ceph-mon[35580]: from='osd.5 ' entity='osd.5' 2024-03-28T15:56:37.169 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:36 smithi067 ceph-mon[31707]: purged_snaps scrub starts 2024-03-28T15:56:37.169 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:36 smithi067 ceph-mon[31707]: purged_snaps scrub ok 2024-03-28T15:56:37.169 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:36 smithi067 ceph-mon[31707]: osdmap e40: 6 total, 5 up, 6 in 2024-03-28T15:56:37.169 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:36 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T15:56:37.169 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:36 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T15:56:37.169 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:36 smithi067 ceph-mon[31707]: from='osd.5 ' entity='osd.5' 2024-03-28T15:56:37.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:36 smithi150 ceph-mon[34825]: purged_snaps scrub starts 2024-03-28T15:56:37.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:36 smithi150 ceph-mon[34825]: purged_snaps scrub ok 2024-03-28T15:56:37.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:36 smithi150 ceph-mon[34825]: osdmap e40: 6 total, 5 up, 6 in 2024-03-28T15:56:37.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:36 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T15:56:37.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:36 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T15:56:37.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:36 smithi150 ceph-mon[34825]: from='osd.5 ' entity='osd.5' 2024-03-28T15:56:38.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:37 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T15:56:38.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:37 smithi150 ceph-mon[34825]: pgmap v180: 1 pgs: 1 remapped+peering; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:56:38.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:37 smithi150 ceph-mon[34825]: osd.5 [v2:172.21.15.183:6808/164284137,v1:172.21.15.183:6809/164284137] boot 2024-03-28T15:56:38.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:37 smithi150 ceph-mon[34825]: osdmap e41: 6 total, 6 up, 6 in 2024-03-28T15:56:38.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:37 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T15:56:38.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:37 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T15:56:38.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:37 smithi067 ceph-mon[31707]: pgmap v180: 1 pgs: 1 remapped+peering; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:56:38.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:37 smithi067 ceph-mon[31707]: osd.5 [v2:172.21.15.183:6808/164284137,v1:172.21.15.183:6809/164284137] boot 2024-03-28T15:56:38.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:37 smithi067 ceph-mon[31707]: osdmap e41: 6 total, 6 up, 6 in 2024-03-28T15:56:38.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:37 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T15:56:38.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:37 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T15:56:38.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:37 smithi183 ceph-mon[35580]: pgmap v180: 1 pgs: 1 remapped+peering; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:56:38.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:37 smithi183 ceph-mon[35580]: osd.5 [v2:172.21.15.183:6808/164284137,v1:172.21.15.183:6809/164284137] boot 2024-03-28T15:56:38.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:37 smithi183 ceph-mon[35580]: osdmap e41: 6 total, 6 up, 6 in 2024-03-28T15:56:38.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:37 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T15:56:38.812 INFO:teuthology.orchestra.run.smithi067.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-28T15:56:38.812 INFO:teuthology.orchestra.run.smithi067.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-28T15:56:38.812 INFO:teuthology.orchestra.run.smithi067.stderr: stderr: 10+0 records in 2024-03-28T15:56:38.812 INFO:teuthology.orchestra.run.smithi067.stderr:10+0 records out 2024-03-28T15:56:38.812 INFO:teuthology.orchestra.run.smithi067.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0399962 s, 262 MB/s 2024-03-28T15:56:38.812 INFO:teuthology.orchestra.run.smithi067.stderr:--> Zapping successful for: 2024-03-28T15:56:39.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:38 smithi067 ceph-mon[31707]: osdmap e42: 6 total, 6 up, 6 in 2024-03-28T15:56:39.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:38 smithi150 ceph-mon[34825]: osdmap e42: 6 total, 6 up, 6 in 2024-03-28T15:56:39.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:38 smithi183 ceph-mon[35580]: osdmap e42: 6 total, 6 up, 6 in 2024-03-28T15:56:39.622 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi067:vg_nvme/lv_2 2024-03-28T15:56:40.091 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:56:40.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:39 smithi183 ceph-mon[35580]: pgmap v182: 1 pgs: 1 remapped+peering; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-03-28T15:56:40.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:39 smithi183 ceph-mon[35580]: osdmap e43: 6 total, 6 up, 6 in 2024-03-28T15:56:40.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:39 smithi183 ceph-mon[35580]: Detected new or changed devices on smithi183 2024-03-28T15:56:40.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:39 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:56:40.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:39 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:56:40.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:39 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:56:40.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:39 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:56:40.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:39 smithi183 ceph-mon[35580]: Adjusting osd_memory_target on smithi183 to 8545M 2024-03-28T15:56:40.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:39 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:56:40.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:39 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:56:40.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:39 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:56:40.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:39 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:56:40.241 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:39 smithi067 ceph-mon[31707]: pgmap v182: 1 pgs: 1 remapped+peering; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-03-28T15:56:40.241 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:39 smithi067 ceph-mon[31707]: osdmap e43: 6 total, 6 up, 6 in 2024-03-28T15:56:40.241 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:39 smithi067 ceph-mon[31707]: Detected new or changed devices on smithi183 2024-03-28T15:56:40.241 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:39 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:56:40.241 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:39 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:56:40.241 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:39 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:56:40.241 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:39 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:56:40.241 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:39 smithi067 ceph-mon[31707]: Adjusting osd_memory_target on smithi183 to 8545M 2024-03-28T15:56:40.241 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:39 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:56:40.241 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:39 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:56:40.241 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:39 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:56:40.241 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:39 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:56:40.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:39 smithi150 ceph-mon[34825]: pgmap v182: 1 pgs: 1 remapped+peering; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-03-28T15:56:40.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:39 smithi150 ceph-mon[34825]: osdmap e43: 6 total, 6 up, 6 in 2024-03-28T15:56:40.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:39 smithi150 ceph-mon[34825]: Detected new or changed devices on smithi183 2024-03-28T15:56:40.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:39 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:56:40.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:39 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:56:40.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:39 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:56:40.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:39 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:56:40.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:39 smithi150 ceph-mon[34825]: Adjusting osd_memory_target on smithi183 to 8545M 2024-03-28T15:56:40.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:39 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:56:40.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:39 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:56:40.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:39 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:56:40.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:39 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:56:41.280 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:40 smithi067 ceph-mon[31707]: osdmap e44: 6 total, 6 up, 6 in 2024-03-28T15:56:41.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:40 smithi150 ceph-mon[34825]: osdmap e44: 6 total, 6 up, 6 in 2024-03-28T15:56:41.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:40 smithi183 ceph-mon[35580]: osdmap e44: 6 total, 6 up, 6 in 2024-03-28T15:56:42.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:56:42.119+0000 7f41cb600700 1 -- 172.21.15.67:0/4163419971 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41c4105610 msgr2=0x7f41c4105a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:56:42.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:56:42.119+0000 7f41cb600700 1 --2- 172.21.15.67:0/4163419971 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41c4105610 0x7f41c4105a70 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f41b00099c0 tx=0x7f41b0009cd0 comp rx=0 tx=0).stop 2024-03-28T15:56:42.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:56:42.119+0000 7f41cb600700 1 -- 172.21.15.67:0/4163419971 shutdown_connections 2024-03-28T15:56:42.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:56:42.119+0000 7f41cb600700 1 --2- 172.21.15.67:0/4163419971 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f41c4105fb0 0x7f41c410c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:56:42.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:56:42.119+0000 7f41cb600700 1 --2- 172.21.15.67:0/4163419971 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41c4104410 0x7f41c4104830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:56:42.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:56:42.119+0000 7f41cb600700 1 --2- 172.21.15.67:0/4163419971 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41c4105610 0x7f41c4105a70 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:56:42.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:56:42.119+0000 7f41cb600700 1 -- 172.21.15.67:0/4163419971 >> 172.21.15.67:0/4163419971 conn(0x7f41c40ff9b0 msgr2=0x7f41c4101df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:56:42.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:56:42.119+0000 7f41cb600700 1 -- 172.21.15.67:0/4163419971 shutdown_connections 2024-03-28T15:56:42.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:56:42.119+0000 7f41cb600700 1 -- 172.21.15.67:0/4163419971 wait complete. 2024-03-28T15:56:42.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:56:42.120+0000 7f41cb600700 1 Processor -- start 2024-03-28T15:56:42.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:56:42.120+0000 7f41cb600700 1 -- start start 2024-03-28T15:56:42.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:56:42.121+0000 7f41cb600700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41c4104410 0x7f41c419a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:56:42.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:56:42.121+0000 7f41cb600700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41c4105610 0x7f41c419aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:56:42.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:56:42.121+0000 7f41cb600700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f41c4105fb0 0x7f41c41a1a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:56:42.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:56:42.121+0000 7f41cb600700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41c410feb0 con 0x7f41c4104410 2024-03-28T15:56:42.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:56:42.121+0000 7f41cb600700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41c410fd30 con 0x7f41c4105610 2024-03-28T15:56:42.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:56:42.121+0000 7f41cb600700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41c4110030 con 0x7f41c4105fb0 2024-03-28T15:56:42.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:56:42.122+0000 7f41c3e00700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41c4105610 0x7f41c419aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:56:42.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:56:42.122+0000 7f41c8c00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41c4104410 0x7f41c419a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:56:42.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:56:42.122+0000 7f41c9600700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f41c4105fb0 0x7f41c41a1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:56:42.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:56:42.122+0000 7f41c8c00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41c4104410 0x7f41c419a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47216/0 (socket says 172.21.15.67:47216) 2024-03-28T15:56:42.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:56:42.122+0000 7f41c8c00700 1 -- 172.21.15.67:0/1140326144 learned_addr learned my addr 172.21.15.67:0/1140326144 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:56:42.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:56:42.122+0000 7f41c3e00700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41c4105610 0x7f41c419aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.67:52180/0 (socket says 172.21.15.67:52180) 2024-03-28T15:56:42.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:56:42.122+0000 7f41c8c00700 1 -- 172.21.15.67:0/1140326144 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f41c4105fb0 msgr2=0x7f41c41a1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:56:42.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:56:42.122+0000 7f41c8c00700 1 --2- 172.21.15.67:0/1140326144 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f41c4105fb0 0x7f41c41a1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:56:42.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:56:42.122+0000 7f41c8c00700 1 -- 172.21.15.67:0/1140326144 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41c4105610 msgr2=0x7f41c419aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:56:42.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:56:42.122+0000 7f41c8c00700 1 --2- 172.21.15.67:0/1140326144 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41c4105610 0x7f41c419aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:56:42.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:56:42.122+0000 7f41c8c00700 1 -- 172.21.15.67:0/1140326144 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41c41a2130 con 0x7f41c4104410 2024-03-28T15:56:42.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:56:42.122+0000 7f41c8c00700 1 --2- 172.21.15.67:0/1140326144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41c4104410 0x7f41c419a4d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f41b800d9b0 tx=0x7f41b800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:56:42.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:56:42.123+0000 7f41c1600700 1 -- 172.21.15.67:0/1140326144 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41b8017030 con 0x7f41c4104410 2024-03-28T15:56:42.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:56:42.123+0000 7f41c1600700 1 -- 172.21.15.67:0/1140326144 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f41b80044d0 con 0x7f41c4104410 2024-03-28T15:56:42.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:56:42.123+0000 7f41c1600700 1 -- 172.21.15.67:0/1140326144 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41b80038e0 con 0x7f41c4104410 2024-03-28T15:56:42.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:56:42.123+0000 7f41cb600700 1 -- 172.21.15.67:0/1140326144 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41c41a2420 con 0x7f41c4104410 2024-03-28T15:56:42.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:56:42.123+0000 7f41cb600700 1 -- 172.21.15.67:0/1140326144 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41c41a29b0 con 0x7f41c4104410 2024-03-28T15:56:42.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:56:42.124+0000 7f41cb600700 1 -- 172.21.15.67:0/1140326144 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41c4068e50 con 0x7f41c4104410 2024-03-28T15:56:42.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:56:42.126+0000 7f41c1600700 1 -- 172.21.15.67:0/1140326144 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f41b801a070 con 0x7f41c4104410 2024-03-28T15:56:42.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:56:42.128+0000 7f41c1600700 1 --2- 172.21.15.67:0/1140326144 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f41ac0a2910 0x7f41ac0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:56:42.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:56:42.128+0000 7f41c1600700 1 -- 172.21.15.67:0/1140326144 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 4533+0+0 (secure 0 0 0) 0x7f41b80cd800 con 0x7f41c4104410 2024-03-28T15:56:42.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:56:42.128+0000 7f41c3e00700 1 --2- 172.21.15.67:0/1140326144 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f41ac0a2910 0x7f41ac0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:56:42.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:56:42.129+0000 7f41c3e00700 1 --2- 172.21.15.67:0/1140326144 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f41ac0a2910 0x7f41ac0a4dd0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f41b0000c00 tx=0x7f41b0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:56:42.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:56:42.131+0000 7f41c1600700 1 -- 172.21.15.67:0/1140326144 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f41b809b740 con 0x7f41c4104410 2024-03-28T15:56:42.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:42 smithi150 ceph-mon[34825]: pgmap v185: 1 pgs: 1 remapped+peering; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-28T15:56:42.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:42 smithi150 ceph-mon[34825]: osdmap e45: 6 total, 6 up, 6 in 2024-03-28T15:56:42.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:42 smithi067 ceph-mon[31707]: pgmap v185: 1 pgs: 1 remapped+peering; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-28T15:56:42.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:42 smithi067 ceph-mon[31707]: osdmap e45: 6 total, 6 up, 6 in 2024-03-28T15:56:42.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:42 smithi183 ceph-mon[35580]: pgmap v185: 1 pgs: 1 remapped+peering; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-28T15:56:42.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:42 smithi183 ceph-mon[35580]: osdmap e45: 6 total, 6 up, 6 in 2024-03-28T15:56:42.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:56:42.369+0000 7f41cb600700 1 -- 172.21.15.67:0/1140326144 --> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f41c40625d0 con 0x7f41ac0a2910 2024-03-28T15:56:43.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:43 smithi150 ceph-mon[34825]: from='client.14349 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:56:43.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:43 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:56:43.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:43 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:56:43.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:43 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:56:43.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:43 smithi067 ceph-mon[31707]: from='client.14349 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:56:43.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:43 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:56:43.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:43 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:56:43.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:43 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:56:43.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:43 smithi183 ceph-mon[35580]: from='client.14349 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:56:43.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:43 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:56:43.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:43 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:56:43.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:43 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:56:44.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:44 smithi150 ceph-mon[34825]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-03-28T15:56:44.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:44 smithi067 ceph-mon[31707]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-03-28T15:56:44.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:44 smithi183 ceph-mon[35580]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-03-28T15:56:46.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:46 smithi150 ceph-mon[34825]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T15:56:46.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:46 smithi067 ceph-mon[31707]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T15:56:46.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:46 smithi183 ceph-mon[35580]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T15:56:47.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:47 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/2863228968' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8905a2d3-0f1b-4853-a218-22da2635150a"}]: dispatch 2024-03-28T15:56:47.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:47 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/2863228968' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8905a2d3-0f1b-4853-a218-22da2635150a"}]': finished 2024-03-28T15:56:47.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:47 smithi150 ceph-mon[34825]: osdmap e46: 7 total, 6 up, 7 in 2024-03-28T15:56:47.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:47 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T15:56:47.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:47 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/2863228968' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8905a2d3-0f1b-4853-a218-22da2635150a"}]: dispatch 2024-03-28T15:56:47.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:47 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/2863228968' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8905a2d3-0f1b-4853-a218-22da2635150a"}]': finished 2024-03-28T15:56:47.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:47 smithi067 ceph-mon[31707]: osdmap e46: 7 total, 6 up, 7 in 2024-03-28T15:56:47.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:47 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T15:56:47.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:47 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/2863228968' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8905a2d3-0f1b-4853-a218-22da2635150a"}]: dispatch 2024-03-28T15:56:47.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:47 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/2863228968' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8905a2d3-0f1b-4853-a218-22da2635150a"}]': finished 2024-03-28T15:56:47.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:47 smithi183 ceph-mon[35580]: osdmap e46: 7 total, 6 up, 7 in 2024-03-28T15:56:47.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:47 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T15:56:48.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:48 smithi150 ceph-mon[34825]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-03-28T15:56:48.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:48 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/529877791' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:56:48.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:48 smithi067 ceph-mon[31707]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-03-28T15:56:48.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:48 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/529877791' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:56:48.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:48 smithi183 ceph-mon[35580]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-03-28T15:56:48.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:48 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/529877791' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:56:50.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:50 smithi150 ceph-mon[34825]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-28T15:56:50.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:50 smithi067 ceph-mon[31707]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-28T15:56:50.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:50 smithi183 ceph-mon[35580]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-28T15:56:52.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:52 smithi150 ceph-mon[34825]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-28T15:56:52.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:52 smithi067 ceph-mon[31707]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-28T15:56:52.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:52 smithi183 ceph-mon[35580]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-28T15:56:54.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:54 smithi067 ceph-mon[31707]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-28T15:56:54.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:54 smithi183 ceph-mon[35580]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-28T15:56:54.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:54 smithi150 ceph-mon[34825]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-28T15:56:56.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:56 smithi067 ceph-mon[31707]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-28T15:56:56.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:56 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-28T15:56:56.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:56 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:56:56.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:56 smithi067 ceph-mon[31707]: Deploying daemon osd.6 on smithi067 2024-03-28T15:56:56.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:56 smithi183 ceph-mon[35580]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-28T15:56:56.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:56 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-28T15:56:56.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:56 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:56:56.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:56 smithi183 ceph-mon[35580]: Deploying daemon osd.6 on smithi067 2024-03-28T15:56:56.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:56 smithi150 ceph-mon[34825]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-28T15:56:56.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:56 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-28T15:56:56.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:56 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:56:56.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:56 smithi150 ceph-mon[34825]: Deploying daemon osd.6 on smithi067 2024-03-28T15:56:58.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:56:58 smithi067 ceph-mon[31707]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-28T15:56:58.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:56:58 smithi183 ceph-mon[35580]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-28T15:56:58.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:56:58 smithi150 ceph-mon[34825]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-28T15:57:00.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:00 smithi067 ceph-mon[31707]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-28T15:57:00.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:00 smithi183 ceph-mon[35580]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-28T15:57:00.355 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:00 smithi150 ceph-mon[34825]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-28T15:57:02.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:02 smithi183 ceph-mon[35580]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-28T15:57:02.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:02 smithi150 ceph-mon[34825]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-28T15:57:02.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:02 smithi067 ceph-mon[31707]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-28T15:57:04.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:04 smithi183 ceph-mon[35580]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-28T15:57:04.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:04 smithi150 ceph-mon[34825]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-28T15:57:04.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:04 smithi067 ceph-mon[31707]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-28T15:57:05.443 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:05 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:57:05.444 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:05 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:57:05.444 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:05 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:57:05.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:05 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:57:05.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:05 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:57:05.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:05 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:57:05.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:05 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:57:05.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:05 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:57:05.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:05 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:57:06.485 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:06 smithi067 ceph-mon[31707]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-28T15:57:06.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:06 smithi183 ceph-mon[35580]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-28T15:57:06.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:06 smithi150 ceph-mon[34825]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-28T15:57:08.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:08 smithi067 ceph-mon[31707]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-28T15:57:08.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:08 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:57:08.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:08 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:57:08.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:08 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:57:08.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:08 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:57:08.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:08 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:57:08.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:08 smithi067 ceph-mon[31707]: from='osd.6 [v2:172.21.15.67:6818/3565528013,v1:172.21.15.67:6819/3565528013]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-28T15:57:08.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:08 smithi183 ceph-mon[35580]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-28T15:57:08.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:08 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:57:08.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:08 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:57:08.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:08 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:57:08.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:08 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:57:08.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:08 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:57:08.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:08 smithi183 ceph-mon[35580]: from='osd.6 [v2:172.21.15.67:6818/3565528013,v1:172.21.15.67:6819/3565528013]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-28T15:57:08.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:08 smithi150 ceph-mon[34825]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-28T15:57:08.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:08 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:57:08.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:08 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:57:08.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:08 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:57:08.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:08 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:57:08.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:08 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:57:08.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:08 smithi150 ceph-mon[34825]: from='osd.6 [v2:172.21.15.67:6818/3565528013,v1:172.21.15.67:6819/3565528013]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-28T15:57:09.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:57:09.260+0000 7f41c1600700 1 -- 172.21.15.67:0/1140326144 <== mgr.14152 v2:172.21.15.67:6800/208726654 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f41c40625d0 con 0x7f41ac0a2910 2024-03-28T15:57:09.262 INFO:teuthology.orchestra.run.smithi067.stdout:Created osd(s) 6 on host 'smithi067' 2024-03-28T15:57:09.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:57:09.263+0000 7f41cb600700 1 -- 172.21.15.67:0/1140326144 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f41ac0a2910 msgr2=0x7f41ac0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:57:09.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:57:09.263+0000 7f41cb600700 1 --2- 172.21.15.67:0/1140326144 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f41ac0a2910 0x7f41ac0a4dd0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f41b0000c00 tx=0x7f41b0005fb0 comp rx=0 tx=0).stop 2024-03-28T15:57:09.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:57:09.263+0000 7f41cb600700 1 -- 172.21.15.67:0/1140326144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41c4104410 msgr2=0x7f41c419a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:57:09.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:57:09.263+0000 7f41cb600700 1 --2- 172.21.15.67:0/1140326144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41c4104410 0x7f41c419a4d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f41b800d9b0 tx=0x7f41b800dcc0 comp rx=0 tx=0).stop 2024-03-28T15:57:09.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:57:09.264+0000 7f41cb600700 1 -- 172.21.15.67:0/1140326144 shutdown_connections 2024-03-28T15:57:09.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:57:09.264+0000 7f41cb600700 1 --2- 172.21.15.67:0/1140326144 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f41ac0a2910 0x7f41ac0a4dd0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:57:09.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:57:09.264+0000 7f41cb600700 1 --2- 172.21.15.67:0/1140326144 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f41c4105fb0 0x7f41c41a1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:57:09.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:57:09.264+0000 7f41cb600700 1 --2- 172.21.15.67:0/1140326144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41c4104410 0x7f41c419a4d0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:57:09.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:57:09.264+0000 7f41cb600700 1 --2- 172.21.15.67:0/1140326144 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41c4105610 0x7f41c419aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:57:09.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:57:09.264+0000 7f41cb600700 1 -- 172.21.15.67:0/1140326144 >> 172.21.15.67:0/1140326144 conn(0x7f41c40ff9b0 msgr2=0x7f41c4108910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:57:09.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:57:09.264+0000 7f41cb600700 1 -- 172.21.15.67:0/1140326144 shutdown_connections 2024-03-28T15:57:09.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:57:09.264+0000 7f41cb600700 1 -- 172.21.15.67:0/1140326144 wait complete. 2024-03-28T15:57:10.180 DEBUG:teuthology.orchestra.run.smithi067:osd.6> sudo journalctl -f -n 0 -u ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@osd.6.service 2024-03-28T15:57:10.185 INFO:tasks.cephadm:Deploying osd.7 on smithi150 with /dev/vg_nvme/lv_2... 2024-03-28T15:57:10.186 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-28T15:57:10.269 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:10 smithi150 ceph-mon[34825]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-28T15:57:10.269 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:10 smithi150 ceph-mon[34825]: from='osd.6 [v2:172.21.15.67:6818/3565528013,v1:172.21.15.67:6819/3565528013]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-28T15:57:10.269 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:10 smithi150 ceph-mon[34825]: osdmap e47: 7 total, 6 up, 7 in 2024-03-28T15:57:10.269 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:10 smithi150 ceph-mon[34825]: from='osd.6 [v2:172.21.15.67:6818/3565528013,v1:172.21.15.67:6819/3565528013]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-03-28T15:57:10.270 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:10 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T15:57:10.270 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:10 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:57:10.270 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:10 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:57:10.270 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:10 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:57:10.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:10 smithi067 ceph-mon[31707]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-28T15:57:10.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:10 smithi067 ceph-mon[31707]: from='osd.6 [v2:172.21.15.67:6818/3565528013,v1:172.21.15.67:6819/3565528013]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-28T15:57:10.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:10 smithi067 ceph-mon[31707]: osdmap e47: 7 total, 6 up, 7 in 2024-03-28T15:57:10.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:10 smithi067 ceph-mon[31707]: from='osd.6 [v2:172.21.15.67:6818/3565528013,v1:172.21.15.67:6819/3565528013]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-03-28T15:57:10.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:10 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T15:57:10.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:10 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:57:10.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:10 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:57:10.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:10 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:57:10.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:10 smithi183 ceph-mon[35580]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-28T15:57:10.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:10 smithi183 ceph-mon[35580]: from='osd.6 [v2:172.21.15.67:6818/3565528013,v1:172.21.15.67:6819/3565528013]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-28T15:57:10.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:10 smithi183 ceph-mon[35580]: osdmap e47: 7 total, 6 up, 7 in 2024-03-28T15:57:10.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:10 smithi183 ceph-mon[35580]: from='osd.6 [v2:172.21.15.67:6818/3565528013,v1:172.21.15.67:6819/3565528013]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-03-28T15:57:10.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:10 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T15:57:10.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:10 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:57:10.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:10 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:57:10.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:10 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:57:10.624 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T15:57:11.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:11 smithi150 ceph-mon[34825]: from='osd.6 [v2:172.21.15.67:6818/3565528013,v1:172.21.15.67:6819/3565528013]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-03-28T15:57:11.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:11 smithi150 ceph-mon[34825]: osdmap e48: 7 total, 6 up, 7 in 2024-03-28T15:57:11.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:11 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T15:57:11.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:11 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T15:57:11.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:11 smithi067 ceph-mon[31707]: from='osd.6 [v2:172.21.15.67:6818/3565528013,v1:172.21.15.67:6819/3565528013]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-03-28T15:57:11.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:11 smithi067 ceph-mon[31707]: osdmap e48: 7 total, 6 up, 7 in 2024-03-28T15:57:11.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:11 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T15:57:11.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:11 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T15:57:11.320 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 15:57:11 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T15:57:11.093+0000 7ff6e5e00700 -1 osd.6 0 waiting for initial osdmap 2024-03-28T15:57:11.321 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 15:57:11 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T15:57:11.140+0000 7ff6dbc00700 -1 osd.6 48 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T15:57:11.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:11 smithi183 ceph-mon[35580]: from='osd.6 [v2:172.21.15.67:6818/3565528013,v1:172.21.15.67:6819/3565528013]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-03-28T15:57:11.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:11 smithi183 ceph-mon[35580]: osdmap e48: 7 total, 6 up, 7 in 2024-03-28T15:57:11.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:11 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T15:57:11.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:11 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T15:57:12.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:12 smithi150 ceph-mon[34825]: purged_snaps scrub starts 2024-03-28T15:57:12.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:12 smithi150 ceph-mon[34825]: purged_snaps scrub ok 2024-03-28T15:57:12.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:12 smithi150 ceph-mon[34825]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-28T15:57:12.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:12 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T15:57:12.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:12 smithi150 ceph-mon[34825]: from='osd.6 [v2:172.21.15.67:6818/3565528013,v1:172.21.15.67:6819/3565528013]' entity='osd.6' 2024-03-28T15:57:12.117 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:12 smithi067 ceph-mon[31707]: purged_snaps scrub starts 2024-03-28T15:57:12.117 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:12 smithi067 ceph-mon[31707]: purged_snaps scrub ok 2024-03-28T15:57:12.117 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:12 smithi067 ceph-mon[31707]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-28T15:57:12.117 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:12 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T15:57:12.117 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:12 smithi067 ceph-mon[31707]: from='osd.6 [v2:172.21.15.67:6818/3565528013,v1:172.21.15.67:6819/3565528013]' entity='osd.6' 2024-03-28T15:57:12.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:12 smithi183 ceph-mon[35580]: purged_snaps scrub starts 2024-03-28T15:57:12.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:12 smithi183 ceph-mon[35580]: purged_snaps scrub ok 2024-03-28T15:57:12.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:12 smithi183 ceph-mon[35580]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-28T15:57:12.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:12 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T15:57:12.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:12 smithi183 ceph-mon[35580]: from='osd.6 [v2:172.21.15.67:6818/3565528013,v1:172.21.15.67:6819/3565528013]' entity='osd.6' 2024-03-28T15:57:13.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:13 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T15:57:13.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:13 smithi150 ceph-mon[34825]: osd.6 [v2:172.21.15.67:6818/3565528013,v1:172.21.15.67:6819/3565528013] boot 2024-03-28T15:57:13.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:13 smithi150 ceph-mon[34825]: osdmap e49: 7 total, 7 up, 7 in 2024-03-28T15:57:13.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:13 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T15:57:13.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:13 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T15:57:13.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:13 smithi067 ceph-mon[31707]: osd.6 [v2:172.21.15.67:6818/3565528013,v1:172.21.15.67:6819/3565528013] boot 2024-03-28T15:57:13.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:13 smithi067 ceph-mon[31707]: osdmap e49: 7 total, 7 up, 7 in 2024-03-28T15:57:13.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:13 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T15:57:13.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:13 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T15:57:13.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:13 smithi183 ceph-mon[35580]: osd.6 [v2:172.21.15.67:6818/3565528013,v1:172.21.15.67:6819/3565528013] boot 2024-03-28T15:57:13.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:13 smithi183 ceph-mon[35580]: osdmap e49: 7 total, 7 up, 7 in 2024-03-28T15:57:13.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:13 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T15:57:13.427 INFO:teuthology.orchestra.run.smithi150.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-28T15:57:13.428 INFO:teuthology.orchestra.run.smithi150.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-28T15:57:13.428 INFO:teuthology.orchestra.run.smithi150.stderr: stderr: 10+0 records in 2024-03-28T15:57:13.428 INFO:teuthology.orchestra.run.smithi150.stderr:10+0 records out 2024-03-28T15:57:13.428 INFO:teuthology.orchestra.run.smithi150.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0336862 s, 311 MB/s 2024-03-28T15:57:13.428 INFO:teuthology.orchestra.run.smithi150.stderr: stderr: 2024-03-28T15:57:13.428 INFO:teuthology.orchestra.run.smithi150.stderr:--> Zapping successful for: 2024-03-28T15:57:14.384 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:14 smithi150 ceph-mon[34825]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-28T15:57:14.384 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:14 smithi150 ceph-mon[34825]: osdmap e50: 7 total, 7 up, 7 in 2024-03-28T15:57:14.449 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi150:vg_nvme/lv_2 2024-03-28T15:57:14.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:14 smithi067 ceph-mon[31707]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-28T15:57:14.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:14 smithi067 ceph-mon[31707]: osdmap e50: 7 total, 7 up, 7 in 2024-03-28T15:57:14.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:14 smithi183 ceph-mon[35580]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-28T15:57:14.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:14 smithi183 ceph-mon[35580]: osdmap e50: 7 total, 7 up, 7 in 2024-03-28T15:57:14.863 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T15:57:16.399 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:16 smithi150 ceph-mon[34825]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:57:16.399 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:16 smithi150 ceph-mon[34825]: Detected new or changed devices on smithi067 2024-03-28T15:57:16.400 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:16 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:57:16.400 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:16 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:57:16.400 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:16 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:57:16.400 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:16 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:57:16.400 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:16 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:57:16.400 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:16 smithi150 ceph-mon[34825]: Adjusting osd_memory_target on smithi067 to 5697M 2024-03-28T15:57:16.400 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:16 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:57:16.400 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:16 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:57:16.400 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:16 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:57:16.400 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:16 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:57:16.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:16 smithi067 ceph-mon[31707]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:57:16.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:16 smithi067 ceph-mon[31707]: Detected new or changed devices on smithi067 2024-03-28T15:57:16.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:16 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:57:16.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:16 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:57:16.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:16 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:57:16.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:16 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:57:16.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:16 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:57:16.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:16 smithi067 ceph-mon[31707]: Adjusting osd_memory_target on smithi067 to 5697M 2024-03-28T15:57:16.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:16 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:57:16.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:16 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:57:16.571 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:16 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:57:16.571 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:16 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:57:16.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:16 smithi183 ceph-mon[35580]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:57:16.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:16 smithi183 ceph-mon[35580]: Detected new or changed devices on smithi067 2024-03-28T15:57:16.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:16 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:57:16.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:16 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:57:16.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:16 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:57:16.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:16 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:57:16.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:16 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:57:16.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:16 smithi183 ceph-mon[35580]: Adjusting osd_memory_target on smithi067 to 5697M 2024-03-28T15:57:16.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:16 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:57:16.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:16 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:57:16.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:16 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:57:16.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:16 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:57:16.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:57:16.879+0000 7f7cf4400700 1 -- 172.21.15.150:0/2913913205 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7cec10a3b0 msgr2=0x7f7cec110c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:57:16.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:57:16.879+0000 7f7cf4400700 1 --2- 172.21.15.150:0/2913913205 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7cec10a3b0 0x7f7cec110c60 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7ce400b0d0 tx=0x7f7ce400b3e0 comp rx=0 tx=0).stop 2024-03-28T15:57:16.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:57:16.879+0000 7f7cf4400700 1 -- 172.21.15.150:0/2913913205 shutdown_connections 2024-03-28T15:57:16.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:57:16.879+0000 7f7cf4400700 1 --2- 172.21.15.150:0/2913913205 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7cec10a3b0 0x7f7cec110c60 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:57:16.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:57:16.879+0000 7f7cf4400700 1 --2- 172.21.15.150:0/2913913205 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7cec108810 0x7f7cec108c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:57:16.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:57:16.879+0000 7f7cf4400700 1 --2- 172.21.15.150:0/2913913205 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cec109a10 0x7f7cec109e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:57:16.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:57:16.879+0000 7f7cf4400700 1 -- 172.21.15.150:0/2913913205 >> 172.21.15.150:0/2913913205 conn(0x7f7cec0777e0 msgr2=0x7f7cec079c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:57:16.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:57:16.880+0000 7f7cf4400700 1 -- 172.21.15.150:0/2913913205 shutdown_connections 2024-03-28T15:57:16.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:57:16.880+0000 7f7cf4400700 1 -- 172.21.15.150:0/2913913205 wait complete. 2024-03-28T15:57:16.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:57:16.881+0000 7f7cf4400700 1 Processor -- start 2024-03-28T15:57:16.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:57:16.881+0000 7f7cf4400700 1 -- start start 2024-03-28T15:57:16.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:57:16.881+0000 7f7cf4400700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7cec108810 0x7f7cec19e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:57:16.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:57:16.882+0000 7f7cf4400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7cec109a10 0x7f7cec19ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:57:16.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:57:16.882+0000 7f7cf4400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cec10a3b0 0x7f7cec1a5f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:57:16.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:57:16.882+0000 7f7cf4400700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cec114370 con 0x7f7cec10a3b0 2024-03-28T15:57:16.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:57:16.882+0000 7f7cf4400700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cec1141f0 con 0x7f7cec108810 2024-03-28T15:57:16.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:57:16.882+0000 7f7cf4400700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cec1144f0 con 0x7f7cec109a10 2024-03-28T15:57:16.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:57:16.882+0000 7f7cf1a00700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7cec108810 0x7f7cec19e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:57:16.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:57:16.882+0000 7f7cf1a00700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7cec108810 0x7f7cec19e950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50738/0 (socket says 172.21.15.150:50738) 2024-03-28T15:57:16.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:57:16.882+0000 7f7cf1a00700 1 -- 172.21.15.150:0/3915184065 learned_addr learned my addr 172.21.15.150:0/3915184065 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-28T15:57:16.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:57:16.882+0000 7f7cf1000700 1 --2- 172.21.15.150:0/3915184065 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7cec109a10 0x7f7cec19ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:57:16.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:57:16.882+0000 7f7cf2400700 1 --2- 172.21.15.150:0/3915184065 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cec10a3b0 0x7f7cec1a5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:57:16.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:57:16.882+0000 7f7cf1a00700 1 -- 172.21.15.150:0/3915184065 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7cec109a10 msgr2=0x7f7cec19ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:57:16.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:57:16.882+0000 7f7cf1a00700 1 --2- 172.21.15.150:0/3915184065 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7cec109a10 0x7f7cec19ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:57:16.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:57:16.882+0000 7f7cf1a00700 1 -- 172.21.15.150:0/3915184065 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cec10a3b0 msgr2=0x7f7cec1a5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:57:16.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:57:16.883+0000 7f7cf1a00700 1 --2- 172.21.15.150:0/3915184065 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cec10a3b0 0x7f7cec1a5f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:57:16.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:57:16.883+0000 7f7cf1a00700 1 -- 172.21.15.150:0/3915184065 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cec1a65b0 con 0x7f7cec108810 2024-03-28T15:57:16.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:57:16.883+0000 7f7cf2400700 1 --2- 172.21.15.150:0/3915184065 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cec10a3b0 0x7f7cec1a5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:57:16.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:57:16.883+0000 7f7cf1000700 1 --2- 172.21.15.150:0/3915184065 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7cec109a10 0x7f7cec19ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:57:16.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:57:16.883+0000 7f7cf1a00700 1 --2- 172.21.15.150:0/3915184065 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7cec108810 0x7f7cec19e950 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f7ce000b500 tx=0x7f7ce000b8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:57:16.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:57:16.883+0000 7f7cea000700 1 -- 172.21.15.150:0/3915184065 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce0015020 con 0x7f7cec108810 2024-03-28T15:57:16.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:57:16.883+0000 7f7cea000700 1 -- 172.21.15.150:0/3915184065 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7ce0011ce0 con 0x7f7cec108810 2024-03-28T15:57:16.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:57:16.883+0000 7f7cf4400700 1 -- 172.21.15.150:0/3915184065 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cec1a68a0 con 0x7f7cec108810 2024-03-28T15:57:16.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:57:16.883+0000 7f7cea000700 1 -- 172.21.15.150:0/3915184065 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce0010d90 con 0x7f7cec108810 2024-03-28T15:57:16.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:57:16.883+0000 7f7cf4400700 1 -- 172.21.15.150:0/3915184065 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cec1a6eb0 con 0x7f7cec108810 2024-03-28T15:57:16.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:57:16.885+0000 7f7cf4400700 1 -- 172.21.15.150:0/3915184065 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cec068e50 con 0x7f7cec108810 2024-03-28T15:57:16.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:57:16.887+0000 7f7cea000700 1 -- 172.21.15.150:0/3915184065 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f7ce0018020 con 0x7f7cec108810 2024-03-28T15:57:16.889 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:57:16.888+0000 7f7cea000700 1 --2- 172.21.15.150:0/3915184065 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7cd40a2910 0x7f7cd40a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:57:16.889 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:57:16.888+0000 7f7cea000700 1 -- 172.21.15.150:0/3915184065 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4965+0+0 (secure 0 0 0) 0x7f7ce00cdaf0 con 0x7f7cec108810 2024-03-28T15:57:16.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:57:16.889+0000 7f7cf1000700 1 --2- 172.21.15.150:0/3915184065 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7cd40a2910 0x7f7cd40a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:57:16.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:57:16.890+0000 7f7cf1000700 1 --2- 172.21.15.150:0/3915184065 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7cd40a2910 0x7f7cd40a4dd0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7cd8005950 tx=0x7f7cd8005f50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:57:16.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:57:16.891+0000 7f7cea000700 1 -- 172.21.15.150:0/3915184065 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7ce001f020 con 0x7f7cec108810 2024-03-28T15:57:17.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:57:17.128+0000 7f7cf4400700 1 -- 172.21.15.150:0/3915184065 --> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f7cec0625d0 con 0x7f7cd40a2910 2024-03-28T15:57:18.405 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:18 smithi150 ceph-mon[34825]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:57:18.406 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:18 smithi150 ceph-mon[34825]: from='client.34274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:57:18.406 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:18 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:57:18.406 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:18 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:57:18.406 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:18 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:57:18.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:18 smithi067 ceph-mon[31707]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:57:18.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:18 smithi067 ceph-mon[31707]: from='client.34274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:57:18.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:18 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:57:18.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:18 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:57:18.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:18 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:57:18.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:18 smithi183 ceph-mon[35580]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:57:18.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:18 smithi183 ceph-mon[35580]: from='client.34274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:57:18.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:18 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:57:18.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:18 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:57:18.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:18 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:57:20.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:20 smithi150 ceph-mon[34825]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:57:20.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:20 smithi067 ceph-mon[31707]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:57:20.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:20 smithi183 ceph-mon[35580]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:57:21.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:21 smithi150 ceph-mon[34825]: from='client.? 172.21.15.150:0/919402827' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b275055b-b451-4180-91b1-e6e61a828c48"}]: dispatch 2024-03-28T15:57:21.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:21 smithi150 ceph-mon[34825]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b275055b-b451-4180-91b1-e6e61a828c48"}]: dispatch 2024-03-28T15:57:21.553 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:21 smithi150 ceph-mon[34825]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b275055b-b451-4180-91b1-e6e61a828c48"}]': finished 2024-03-28T15:57:21.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:21 smithi067 ceph-mon[31707]: from='client.? 172.21.15.150:0/919402827' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b275055b-b451-4180-91b1-e6e61a828c48"}]: dispatch 2024-03-28T15:57:21.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:21 smithi067 ceph-mon[31707]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b275055b-b451-4180-91b1-e6e61a828c48"}]: dispatch 2024-03-28T15:57:21.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:21 smithi067 ceph-mon[31707]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b275055b-b451-4180-91b1-e6e61a828c48"}]': finished 2024-03-28T15:57:21.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:21 smithi183 ceph-mon[35580]: from='client.? 172.21.15.150:0/919402827' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b275055b-b451-4180-91b1-e6e61a828c48"}]: dispatch 2024-03-28T15:57:21.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:21 smithi183 ceph-mon[35580]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b275055b-b451-4180-91b1-e6e61a828c48"}]: dispatch 2024-03-28T15:57:21.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:21 smithi183 ceph-mon[35580]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b275055b-b451-4180-91b1-e6e61a828c48"}]': finished 2024-03-28T15:57:22.502 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:22 smithi183 ceph-mon[35580]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:57:22.502 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:22 smithi183 ceph-mon[35580]: osdmap e51: 8 total, 7 up, 8 in 2024-03-28T15:57:22.502 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:22 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T15:57:22.502 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:22 smithi183 ceph-mon[35580]: from='client.? 172.21.15.150:0/1302964300' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:57:22.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:22 smithi150 ceph-mon[34825]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:57:22.553 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:22 smithi150 ceph-mon[34825]: osdmap e51: 8 total, 7 up, 8 in 2024-03-28T15:57:22.553 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:22 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T15:57:22.553 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:22 smithi150 ceph-mon[34825]: from='client.? 172.21.15.150:0/1302964300' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:57:22.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:22 smithi067 ceph-mon[31707]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:57:22.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:22 smithi067 ceph-mon[31707]: osdmap e51: 8 total, 7 up, 8 in 2024-03-28T15:57:22.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:22 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T15:57:22.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:22 smithi067 ceph-mon[31707]: from='client.? 172.21.15.150:0/1302964300' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:57:24.500 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:24 smithi150 ceph-mon[34825]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:57:24.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:24 smithi067 ceph-mon[31707]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:57:24.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:24 smithi183 ceph-mon[35580]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:57:26.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:26 smithi150 ceph-mon[34825]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:57:26.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:26 smithi067 ceph-mon[31707]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:57:26.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:26 smithi183 ceph-mon[35580]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:57:28.357 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:28 smithi150 ceph-mon[34825]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:57:28.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:28 smithi067 ceph-mon[31707]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:57:28.889 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:28 smithi183 ceph-mon[35580]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:57:30.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:30 smithi150 ceph-mon[34825]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:57:30.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:30 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-28T15:57:30.553 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:30 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:57:30.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:30 smithi067 ceph-mon[31707]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:57:30.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:30 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-28T15:57:30.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:30 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:57:30.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:30 smithi183 ceph-mon[35580]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:57:30.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:30 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-28T15:57:30.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:30 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:57:31.407 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:31 smithi150 ceph-mon[34825]: Deploying daemon osd.7 on smithi150 2024-03-28T15:57:31.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:31 smithi067 ceph-mon[31707]: Deploying daemon osd.7 on smithi150 2024-03-28T15:57:31.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:31 smithi183 ceph-mon[35580]: Deploying daemon osd.7 on smithi150 2024-03-28T15:57:32.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:32 smithi150 ceph-mon[34825]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:57:32.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:32 smithi067 ceph-mon[31707]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:57:32.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:32 smithi183 ceph-mon[35580]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:57:34.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:34 smithi150 ceph-mon[34825]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:57:34.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:34 smithi067 ceph-mon[31707]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:57:34.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:34 smithi183 ceph-mon[35580]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:57:36.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:36 smithi150 ceph-mon[34825]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:57:36.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:36 smithi067 ceph-mon[31707]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:57:36.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:36 smithi183 ceph-mon[35580]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:57:38.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:38 smithi150 ceph-mon[34825]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:57:38.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:38 smithi067 ceph-mon[31707]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:57:38.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:38 smithi183 ceph-mon[35580]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:57:39.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:39 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:57:39.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:39 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:57:39.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:39 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:57:39.475 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:39 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:57:39.475 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:39 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:57:39.475 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:39 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:57:39.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:39 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:57:39.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:39 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:57:39.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:39 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:57:40.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:40 smithi150 ceph-mon[34825]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:57:40.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:40 smithi067 ceph-mon[31707]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:57:40.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:40 smithi183 ceph-mon[35580]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:57:43.005 INFO:teuthology.orchestra.run.smithi150.stdout:Created osd(s) 7 on host 'smithi150' 2024-03-28T15:57:43.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:57:43.005+0000 7f7cea000700 1 -- 172.21.15.150:0/3915184065 <== mgr.14152 v2:172.21.15.67:6800/208726654 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f7cec0625d0 con 0x7f7cd40a2910 2024-03-28T15:57:43.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:57:43.008+0000 7f7cf4400700 1 -- 172.21.15.150:0/3915184065 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7cd40a2910 msgr2=0x7f7cd40a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:57:43.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:57:43.008+0000 7f7cf4400700 1 --2- 172.21.15.150:0/3915184065 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7cd40a2910 0x7f7cd40a4dd0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7cd8005950 tx=0x7f7cd8005f50 comp rx=0 tx=0).stop 2024-03-28T15:57:43.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:57:43.008+0000 7f7cf4400700 1 -- 172.21.15.150:0/3915184065 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7cec108810 msgr2=0x7f7cec19e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:57:43.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:57:43.008+0000 7f7cf4400700 1 --2- 172.21.15.150:0/3915184065 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7cec108810 0x7f7cec19e950 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f7ce000b500 tx=0x7f7ce000b8c0 comp rx=0 tx=0).stop 2024-03-28T15:57:43.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:57:43.008+0000 7f7cf4400700 1 -- 172.21.15.150:0/3915184065 shutdown_connections 2024-03-28T15:57:43.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:57:43.009+0000 7f7cf4400700 1 --2- 172.21.15.150:0/3915184065 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7cd40a2910 0x7f7cd40a4dd0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:57:43.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:57:43.009+0000 7f7cf4400700 1 --2- 172.21.15.150:0/3915184065 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cec10a3b0 0x7f7cec1a5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:57:43.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:57:43.009+0000 7f7cf4400700 1 --2- 172.21.15.150:0/3915184065 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7cec108810 0x7f7cec19e950 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:57:43.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:57:43.009+0000 7f7cf4400700 1 --2- 172.21.15.150:0/3915184065 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7cec109a10 0x7f7cec19ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:57:43.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:57:43.009+0000 7f7cf4400700 1 -- 172.21.15.150:0/3915184065 >> 172.21.15.150:0/3915184065 conn(0x7f7cec0777e0 msgr2=0x7f7cec10cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:57:43.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:57:43.009+0000 7f7cf4400700 1 -- 172.21.15.150:0/3915184065 shutdown_connections 2024-03-28T15:57:43.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:57:43.009+0000 7f7cf4400700 1 -- 172.21.15.150:0/3915184065 wait complete. 2024-03-28T15:57:43.133 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:42 smithi150 ceph-mon[34825]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:57:43.133 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:42 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:57:43.133 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:42 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:57:43.133 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:42 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:57:43.134 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:42 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:57:43.134 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:42 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:57:43.134 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:42 smithi150 ceph-mon[34825]: from='osd.7 [v2:172.21.15.150:6816/3735115340,v1:172.21.15.150:6817/3735115340]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-28T15:57:43.134 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:42 smithi150 ceph-mon[34825]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-28T15:57:43.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:42 smithi067 ceph-mon[31707]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:57:43.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:42 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:57:43.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:42 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:57:43.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:42 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:57:43.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:42 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:57:43.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:42 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:57:43.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:42 smithi067 ceph-mon[31707]: from='osd.7 [v2:172.21.15.150:6816/3735115340,v1:172.21.15.150:6817/3735115340]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-28T15:57:43.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:42 smithi067 ceph-mon[31707]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-28T15:57:43.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:42 smithi183 ceph-mon[35580]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:57:43.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:42 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:57:43.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:42 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:57:43.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:42 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:57:43.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:42 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:57:43.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:42 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:57:43.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:42 smithi183 ceph-mon[35580]: from='osd.7 [v2:172.21.15.150:6816/3735115340,v1:172.21.15.150:6817/3735115340]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-28T15:57:43.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:42 smithi183 ceph-mon[35580]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-28T15:57:43.843 DEBUG:teuthology.orchestra.run.smithi150:osd.7> sudo journalctl -f -n 0 -u ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@osd.7.service 2024-03-28T15:57:43.848 INFO:tasks.cephadm:Deploying osd.8 on smithi183 with /dev/vg_nvme/lv_2... 2024-03-28T15:57:43.849 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-28T15:57:44.037 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:43 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:57:44.037 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:43 smithi150 ceph-mon[34825]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-28T15:57:44.037 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:43 smithi150 ceph-mon[34825]: osdmap e52: 8 total, 7 up, 8 in 2024-03-28T15:57:44.037 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:43 smithi150 ceph-mon[34825]: from='osd.7 [v2:172.21.15.150:6816/3735115340,v1:172.21.15.150:6817/3735115340]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-03-28T15:57:44.037 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:43 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T15:57:44.037 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:43 smithi150 ceph-mon[34825]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-03-28T15:57:44.037 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:43 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:57:44.037 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:43 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:57:44.037 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:43 smithi150 ceph-mon[34825]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:57:44.288 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T15:57:44.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:43 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:57:44.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:43 smithi067 ceph-mon[31707]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-28T15:57:44.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:43 smithi067 ceph-mon[31707]: osdmap e52: 8 total, 7 up, 8 in 2024-03-28T15:57:44.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:43 smithi067 ceph-mon[31707]: from='osd.7 [v2:172.21.15.150:6816/3735115340,v1:172.21.15.150:6817/3735115340]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-03-28T15:57:44.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:43 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T15:57:44.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:43 smithi067 ceph-mon[31707]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-03-28T15:57:44.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:43 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:57:44.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:43 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:57:44.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:43 smithi067 ceph-mon[31707]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:57:44.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:43 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:57:44.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:43 smithi183 ceph-mon[35580]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-28T15:57:44.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:43 smithi183 ceph-mon[35580]: osdmap e52: 8 total, 7 up, 8 in 2024-03-28T15:57:44.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:43 smithi183 ceph-mon[35580]: from='osd.7 [v2:172.21.15.150:6816/3735115340,v1:172.21.15.150:6817/3735115340]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-03-28T15:57:44.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:43 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T15:57:44.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:43 smithi183 ceph-mon[35580]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-03-28T15:57:44.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:43 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:57:44.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:43 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:57:44.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:43 smithi183 ceph-mon[35580]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:57:45.005 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:44 smithi150 ceph-mon[34825]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-03-28T15:57:45.005 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:44 smithi150 ceph-mon[34825]: osdmap e53: 8 total, 7 up, 8 in 2024-03-28T15:57:45.005 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:44 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T15:57:45.005 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:44 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T15:57:45.005 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:44 smithi150 ceph-mon[34825]: osdmap e54: 8 total, 7 up, 8 in 2024-03-28T15:57:45.005 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:44 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T15:57:45.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:44 smithi183 ceph-mon[35580]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-03-28T15:57:45.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:44 smithi183 ceph-mon[35580]: osdmap e53: 8 total, 7 up, 8 in 2024-03-28T15:57:45.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:44 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T15:57:45.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:44 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T15:57:45.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:44 smithi183 ceph-mon[35580]: osdmap e54: 8 total, 7 up, 8 in 2024-03-28T15:57:45.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:44 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T15:57:45.302 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 15:57:45 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T15:57:45.003+0000 7f425da00700 -1 osd.7 0 waiting for initial osdmap 2024-03-28T15:57:45.302 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 15:57:45 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T15:57:45.047+0000 7f4253800700 -1 osd.7 54 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T15:57:45.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:44 smithi067 ceph-mon[31707]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-03-28T15:57:45.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:44 smithi067 ceph-mon[31707]: osdmap e53: 8 total, 7 up, 8 in 2024-03-28T15:57:45.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:44 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T15:57:45.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:44 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T15:57:45.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:44 smithi067 ceph-mon[31707]: osdmap e54: 8 total, 7 up, 8 in 2024-03-28T15:57:45.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:44 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T15:57:46.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:45 smithi150 ceph-mon[34825]: purged_snaps scrub starts 2024-03-28T15:57:46.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:45 smithi150 ceph-mon[34825]: purged_snaps scrub ok 2024-03-28T15:57:46.054 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:45 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T15:57:46.054 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:45 smithi150 ceph-mon[34825]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:57:46.054 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:45 smithi150 ceph-mon[34825]: from='osd.7 ' entity='osd.7' 2024-03-28T15:57:46.054 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:45 smithi150 ceph-mon[34825]: osd.7 [v2:172.21.15.150:6816/3735115340,v1:172.21.15.150:6817/3735115340] boot 2024-03-28T15:57:46.054 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:45 smithi150 ceph-mon[34825]: osdmap e55: 8 total, 8 up, 8 in 2024-03-28T15:57:46.054 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:45 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T15:57:46.281 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:45 smithi183 ceph-mon[35580]: purged_snaps scrub starts 2024-03-28T15:57:46.281 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:45 smithi183 ceph-mon[35580]: purged_snaps scrub ok 2024-03-28T15:57:46.281 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:45 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T15:57:46.281 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:45 smithi183 ceph-mon[35580]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:57:46.281 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:45 smithi183 ceph-mon[35580]: from='osd.7 ' entity='osd.7' 2024-03-28T15:57:46.281 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:45 smithi183 ceph-mon[35580]: osd.7 [v2:172.21.15.150:6816/3735115340,v1:172.21.15.150:6817/3735115340] boot 2024-03-28T15:57:46.282 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:45 smithi183 ceph-mon[35580]: osdmap e55: 8 total, 8 up, 8 in 2024-03-28T15:57:46.282 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:45 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T15:57:46.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:45 smithi067 ceph-mon[31707]: purged_snaps scrub starts 2024-03-28T15:57:46.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:45 smithi067 ceph-mon[31707]: purged_snaps scrub ok 2024-03-28T15:57:46.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:45 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T15:57:46.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:45 smithi067 ceph-mon[31707]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:57:46.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:45 smithi067 ceph-mon[31707]: from='osd.7 ' entity='osd.7' 2024-03-28T15:57:46.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:45 smithi067 ceph-mon[31707]: osd.7 [v2:172.21.15.150:6816/3735115340,v1:172.21.15.150:6817/3735115340] boot 2024-03-28T15:57:46.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:45 smithi067 ceph-mon[31707]: osdmap e55: 8 total, 8 up, 8 in 2024-03-28T15:57:46.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:45 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T15:57:47.310 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:47 smithi183 ceph-mon[35580]: osdmap e56: 8 total, 8 up, 8 in 2024-03-28T15:57:47.313 INFO:teuthology.orchestra.run.smithi183.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-28T15:57:47.313 INFO:teuthology.orchestra.run.smithi183.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-28T15:57:47.313 INFO:teuthology.orchestra.run.smithi183.stderr: stderr: 10+0 records in 2024-03-28T15:57:47.313 INFO:teuthology.orchestra.run.smithi183.stderr:10+0 records out 2024-03-28T15:57:47.313 INFO:teuthology.orchestra.run.smithi183.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0363168 s, 289 MB/s 2024-03-28T15:57:47.313 INFO:teuthology.orchestra.run.smithi183.stderr:--> Zapping successful for: 2024-03-28T15:57:47.430 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:47 smithi150 ceph-mon[34825]: osdmap e56: 8 total, 8 up, 8 in 2024-03-28T15:57:47.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:47 smithi067 ceph-mon[31707]: osdmap e56: 8 total, 8 up, 8 in 2024-03-28T15:57:48.081 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi183:vg_nvme/lv_2 2024-03-28T15:57:48.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:48 smithi183 ceph-mon[35580]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:57:48.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:48 smithi183 ceph-mon[35580]: osdmap e57: 8 total, 8 up, 8 in 2024-03-28T15:57:48.497 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:48 smithi067 ceph-mon[31707]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:57:48.498 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:48 smithi067 ceph-mon[31707]: osdmap e57: 8 total, 8 up, 8 in 2024-03-28T15:57:48.516 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T15:57:48.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:48 smithi150 ceph-mon[34825]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:57:48.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:48 smithi150 ceph-mon[34825]: osdmap e57: 8 total, 8 up, 8 in 2024-03-28T15:57:49.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:49 smithi183 ceph-mon[35580]: osdmap e58: 8 total, 8 up, 8 in 2024-03-28T15:57:49.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:49 smithi183 ceph-mon[35580]: Detected new or changed devices on smithi150 2024-03-28T15:57:49.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:49 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:57:49.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:49 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:57:49.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:49 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:57:49.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:49 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:57:49.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:49 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:57:49.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:49 smithi183 ceph-mon[35580]: Adjusting osd_memory_target on smithi150 to 5697M 2024-03-28T15:57:49.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:49 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:57:49.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:49 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:57:49.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:49 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:57:49.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:49 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:57:49.553 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:49 smithi150 ceph-mon[34825]: osdmap e58: 8 total, 8 up, 8 in 2024-03-28T15:57:49.553 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:49 smithi150 ceph-mon[34825]: Detected new or changed devices on smithi150 2024-03-28T15:57:49.553 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:49 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:57:49.554 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:49 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:57:49.554 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:49 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:57:49.554 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:49 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:57:49.554 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:49 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:57:49.554 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:49 smithi150 ceph-mon[34825]: Adjusting osd_memory_target on smithi150 to 5697M 2024-03-28T15:57:49.554 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:49 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:57:49.555 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:49 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:57:49.555 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:49 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:57:49.555 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:49 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:57:49.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:49 smithi067 ceph-mon[31707]: osdmap e58: 8 total, 8 up, 8 in 2024-03-28T15:57:49.571 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:49 smithi067 ceph-mon[31707]: Detected new or changed devices on smithi150 2024-03-28T15:57:49.571 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:49 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:57:49.571 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:49 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:57:49.571 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:49 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:57:49.571 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:49 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:57:49.571 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:49 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:57:49.571 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:49 smithi067 ceph-mon[31707]: Adjusting osd_memory_target on smithi150 to 5697M 2024-03-28T15:57:49.571 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:49 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:57:49.571 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:49 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:57:49.572 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:49 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:57:49.572 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:49 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:57:50.333 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:50 smithi183 ceph-mon[35580]: pgmap v233: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:57:50.333 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:50 smithi183 ceph-mon[35580]: osdmap e59: 8 total, 8 up, 8 in 2024-03-28T15:57:50.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:50 smithi150 ceph-mon[34825]: pgmap v233: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:57:50.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:50 smithi150 ceph-mon[34825]: osdmap e59: 8 total, 8 up, 8 in 2024-03-28T15:57:50.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:50 smithi067 ceph-mon[31707]: pgmap v233: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:57:50.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:50 smithi067 ceph-mon[31707]: osdmap e59: 8 total, 8 up, 8 in 2024-03-28T15:57:50.581 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:57:50.579+0000 7ff7f8800700 1 -- 172.21.15.183:0/1083217752 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff7f0103d60 msgr2=0x7ff7f010a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:57:50.582 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:57:50.579+0000 7ff7f8800700 1 --2- 172.21.15.183:0/1083217752 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff7f0103d60 0x7ff7f010a610 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec00b0a0 tx=0x7ff7ec00b3b0 comp rx=0 tx=0).stop 2024-03-28T15:57:50.582 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:57:50.579+0000 7ff7f8800700 1 -- 172.21.15.183:0/1083217752 shutdown_connections 2024-03-28T15:57:50.583 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:57:50.579+0000 7ff7f8800700 1 --2- 172.21.15.183:0/1083217752 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff7f0103d60 0x7ff7f010a610 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:57:50.585 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:57:50.579+0000 7ff7f8800700 1 --2- 172.21.15.183:0/1083217752 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7f01021c0 0x7ff7f01025e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:57:50.585 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:57:50.579+0000 7ff7f8800700 1 --2- 172.21.15.183:0/1083217752 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7f01033c0 0x7ff7f0103820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:57:50.586 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:57:50.579+0000 7ff7f8800700 1 -- 172.21.15.183:0/1083217752 >> 172.21.15.183:0/1083217752 conn(0x7ff7f00fd7a0 msgr2=0x7ff7f00ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:57:50.586 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:57:50.579+0000 7ff7f8800700 1 -- 172.21.15.183:0/1083217752 shutdown_connections 2024-03-28T15:57:50.586 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:57:50.580+0000 7ff7f8800700 1 -- 172.21.15.183:0/1083217752 wait complete. 2024-03-28T15:57:50.586 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:57:50.580+0000 7ff7f8800700 1 Processor -- start 2024-03-28T15:57:50.586 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:57:50.581+0000 7ff7f8800700 1 -- start start 2024-03-28T15:57:50.587 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:57:50.581+0000 7ff7f8800700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff7f01021c0 0x7ff7f019a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:57:50.587 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:57:50.581+0000 7ff7f8800700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7f01033c0 0x7ff7f019a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:57:50.588 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:57:50.581+0000 7ff7f8800700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7f0103d60 0x7ff7f01a19d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:57:50.588 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:57:50.581+0000 7ff7f8800700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7f010dc50 con 0x7ff7f01033c0 2024-03-28T15:57:50.588 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:57:50.581+0000 7ff7f8800700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7f010dad0 con 0x7ff7f0103d60 2024-03-28T15:57:50.588 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:57:50.581+0000 7ff7f8800700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7f010ddd0 con 0x7ff7f01021c0 2024-03-28T15:57:50.588 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:57:50.581+0000 7ff7f5e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff7f01021c0 0x7ff7f019a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:57:50.588 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:57:50.581+0000 7ff7f5e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff7f01021c0 0x7ff7f019a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.183:57740/0 (socket says 172.21.15.183:57740) 2024-03-28T15:57:50.588 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:57:50.581+0000 7ff7f5e00700 1 -- 172.21.15.183:0/4286654288 learned_addr learned my addr 172.21.15.183:0/4286654288 (peer_addr_for_me v2:172.21.15.183:0/0) 2024-03-28T15:57:50.588 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:57:50.582+0000 7ff7f5400700 1 --2- 172.21.15.183:0/4286654288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7f01033c0 0x7ff7f019a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:57:50.588 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:57:50.582+0000 7ff7f5e00700 1 -- 172.21.15.183:0/4286654288 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7f0103d60 msgr2=0x7ff7f01a19d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:57:50.588 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:57:50.582+0000 7ff7f6800700 1 --2- 172.21.15.183:0/4286654288 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7f0103d60 0x7ff7f01a19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:57:50.588 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:57:50.582+0000 7ff7f5e00700 1 --2- 172.21.15.183:0/4286654288 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7f0103d60 0x7ff7f01a19d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:57:50.588 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:57:50.582+0000 7ff7f5e00700 1 -- 172.21.15.183:0/4286654288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7f01033c0 msgr2=0x7ff7f019a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:57:50.589 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:57:50.582+0000 7ff7f5e00700 1 --2- 172.21.15.183:0/4286654288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7f01033c0 0x7ff7f019a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:57:50.589 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:57:50.582+0000 7ff7f5e00700 1 -- 172.21.15.183:0/4286654288 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7f01a2070 con 0x7ff7f01021c0 2024-03-28T15:57:50.589 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:57:50.582+0000 7ff7f5e00700 1 --2- 172.21.15.183:0/4286654288 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff7f01021c0 0x7ff7f019a410 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff7e400c9a0 tx=0x7ff7e400cd60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:57:50.589 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:57:50.582+0000 7ff7eaa00700 1 -- 172.21.15.183:0/4286654288 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7e4004220 con 0x7ff7f01021c0 2024-03-28T15:57:50.589 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:57:50.582+0000 7ff7eaa00700 1 -- 172.21.15.183:0/4286654288 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff7e4007d10 con 0x7ff7f01021c0 2024-03-28T15:57:50.589 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:57:50.582+0000 7ff7f8800700 1 -- 172.21.15.183:0/4286654288 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7f01a2360 con 0x7ff7f01021c0 2024-03-28T15:57:50.589 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:57:50.583+0000 7ff7eaa00700 1 -- 172.21.15.183:0/4286654288 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7e4003e10 con 0x7ff7f01021c0 2024-03-28T15:57:50.589 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:57:50.583+0000 7ff7f8800700 1 -- 172.21.15.183:0/4286654288 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7f01a28f0 con 0x7ff7f01021c0 2024-03-28T15:57:50.589 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:57:50.584+0000 7ff7f8800700 1 -- 172.21.15.183:0/4286654288 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7f0068e50 con 0x7ff7f01021c0 2024-03-28T15:57:50.589 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:57:50.586+0000 7ff7eaa00700 1 -- 172.21.15.183:0/4286654288 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ff7e4007860 con 0x7ff7f01021c0 2024-03-28T15:57:50.590 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:57:50.587+0000 7ff7eaa00700 1 --2- 172.21.15.183:0/4286654288 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7ff7d80a2910 0x7ff7d80a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:57:50.590 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:57:50.588+0000 7ff7eaa00700 1 -- 172.21.15.183:0/4286654288 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5397+0+0 (secure 0 0 0) 0x7ff7e40cde90 con 0x7ff7f01021c0 2024-03-28T15:57:50.590 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:57:50.588+0000 7ff7f5400700 1 --2- 172.21.15.183:0/4286654288 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7ff7d80a2910 0x7ff7d80a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:57:50.590 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:57:50.589+0000 7ff7f5400700 1 --2- 172.21.15.183:0/4286654288 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7ff7d80a2910 0x7ff7d80a4dd0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ff7e000ba60 tx=0x7ff7e0005d50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:57:50.591 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:57:50.590+0000 7ff7eaa00700 1 -- 172.21.15.183:0/4286654288 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff7e4014030 con 0x7ff7f01021c0 2024-03-28T15:57:50.820 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:57:50.819+0000 7ff7f8800700 1 -- 172.21.15.183:0/4286654288 --> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7ff7f0075ae0 con 0x7ff7d80a2910 2024-03-28T15:57:51.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:51 smithi150 ceph-mon[34825]: from='client.24307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:57:51.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:51 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:57:51.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:51 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:57:51.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:51 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:57:51.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:51 smithi067 ceph-mon[31707]: from='client.24307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:57:51.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:51 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:57:51.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:51 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:57:51.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:51 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:57:51.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:51 smithi183 ceph-mon[35580]: from='client.24307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:57:51.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:51 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:57:51.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:51 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:57:51.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:51 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:57:52.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:52 smithi183 ceph-mon[35580]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:57:52.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:52 smithi150 ceph-mon[34825]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:57:52.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:52 smithi067 ceph-mon[31707]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:57:54.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:54 smithi150 ceph-mon[34825]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:57:54.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:54 smithi067 ceph-mon[31707]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:57:54.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:54 smithi183 ceph-mon[35580]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:57:55.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:55 smithi150 ceph-mon[34825]: from='client.? 172.21.15.183:0/1869676105' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b6abf7ad-eb34-4f60-a4eb-348a92d7195b"}]: dispatch 2024-03-28T15:57:55.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:55 smithi150 ceph-mon[34825]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b6abf7ad-eb34-4f60-a4eb-348a92d7195b"}]: dispatch 2024-03-28T15:57:55.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:55 smithi150 ceph-mon[34825]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b6abf7ad-eb34-4f60-a4eb-348a92d7195b"}]': finished 2024-03-28T15:57:55.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:55 smithi150 ceph-mon[34825]: osdmap e60: 9 total, 8 up, 9 in 2024-03-28T15:57:55.553 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:55 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T15:57:55.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:55 smithi067 ceph-mon[31707]: from='client.? 172.21.15.183:0/1869676105' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b6abf7ad-eb34-4f60-a4eb-348a92d7195b"}]: dispatch 2024-03-28T15:57:55.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:55 smithi067 ceph-mon[31707]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b6abf7ad-eb34-4f60-a4eb-348a92d7195b"}]: dispatch 2024-03-28T15:57:55.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:55 smithi067 ceph-mon[31707]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b6abf7ad-eb34-4f60-a4eb-348a92d7195b"}]': finished 2024-03-28T15:57:55.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:55 smithi067 ceph-mon[31707]: osdmap e60: 9 total, 8 up, 9 in 2024-03-28T15:57:55.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:55 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T15:57:55.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:55 smithi183 ceph-mon[35580]: from='client.? 172.21.15.183:0/1869676105' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b6abf7ad-eb34-4f60-a4eb-348a92d7195b"}]: dispatch 2024-03-28T15:57:55.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:55 smithi183 ceph-mon[35580]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b6abf7ad-eb34-4f60-a4eb-348a92d7195b"}]: dispatch 2024-03-28T15:57:55.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:55 smithi183 ceph-mon[35580]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b6abf7ad-eb34-4f60-a4eb-348a92d7195b"}]': finished 2024-03-28T15:57:55.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:55 smithi183 ceph-mon[35580]: osdmap e60: 9 total, 8 up, 9 in 2024-03-28T15:57:55.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:55 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T15:57:56.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:56 smithi150 ceph-mon[34825]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:57:56.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:56 smithi150 ceph-mon[34825]: from='client.? 172.21.15.183:0/748834822' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:57:56.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:56 smithi067 ceph-mon[31707]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:57:56.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:56 smithi067 ceph-mon[31707]: from='client.? 172.21.15.183:0/748834822' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:57:56.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:56 smithi183 ceph-mon[35580]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:57:56.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:56 smithi183 ceph-mon[35580]: from='client.? 172.21.15.183:0/748834822' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:57:58.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:57:58 smithi150 ceph-mon[34825]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:57:58.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:57:58 smithi067 ceph-mon[31707]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:57:58.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:57:58 smithi183 ceph-mon[35580]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:58:00.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:00 smithi150 ceph-mon[34825]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:58:00.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:00 smithi067 ceph-mon[31707]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:58:00.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:00 smithi183 ceph-mon[35580]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:58:02.342 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:02 smithi183 ceph-mon[35580]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:58:02.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:02 smithi150 ceph-mon[34825]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:58:02.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:02 smithi067 ceph-mon[31707]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:58:04.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:04 smithi150 ceph-mon[34825]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:58:04.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:04 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-28T15:58:04.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:04 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:58:04.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:04 smithi150 ceph-mon[34825]: Deploying daemon osd.8 on smithi183 2024-03-28T15:58:04.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:04 smithi067 ceph-mon[31707]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:58:04.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:04 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-28T15:58:04.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:04 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:58:04.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:04 smithi067 ceph-mon[31707]: Deploying daemon osd.8 on smithi183 2024-03-28T15:58:04.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:04 smithi183 ceph-mon[35580]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:58:04.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:04 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-28T15:58:04.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:04 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:58:04.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:04 smithi183 ceph-mon[35580]: Deploying daemon osd.8 on smithi183 2024-03-28T15:58:06.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:06 smithi183 ceph-mon[35580]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:58:06.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:06 smithi150 ceph-mon[34825]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:58:06.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:06 smithi067 ceph-mon[31707]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:58:08.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:08 smithi067 ceph-mon[31707]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:58:08.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:08 smithi150 ceph-mon[34825]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:58:08.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:08 smithi183 ceph-mon[35580]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:58:10.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:10 smithi150 ceph-mon[34825]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:58:10.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:10 smithi067 ceph-mon[31707]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:58:10.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:10 smithi183 ceph-mon[35580]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:58:12.306 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:12 smithi150 ceph-mon[34825]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:58:12.307 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:12 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:58:12.307 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:12 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:58:12.307 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:12 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:58:12.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:12 smithi067 ceph-mon[31707]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:58:12.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:12 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:58:12.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:12 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:58:12.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:12 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:58:12.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:12 smithi183 ceph-mon[35580]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:58:12.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:12 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:58:12.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:12 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:58:12.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:12 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:58:14.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:14 smithi150 ceph-mon[34825]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:58:14.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:14 smithi067 ceph-mon[31707]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:58:14.597 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:14 smithi183 ceph-mon[35580]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:58:16.835 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:58:16.834+0000 7ff7eaa00700 1 -- 172.21.15.183:0/4286654288 <== mgr.14152 v2:172.21.15.67:6800/208726654 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ff7f0075ae0 con 0x7ff7d80a2910 2024-03-28T15:58:16.836 INFO:teuthology.orchestra.run.smithi183.stdout:Created osd(s) 8 on host 'smithi183' 2024-03-28T15:58:16.838 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:58:16.838+0000 7ff7f8800700 1 -- 172.21.15.183:0/4286654288 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7ff7d80a2910 msgr2=0x7ff7d80a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:58:16.838 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:58:16.838+0000 7ff7f8800700 1 --2- 172.21.15.183:0/4286654288 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7ff7d80a2910 0x7ff7d80a4dd0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ff7e000ba60 tx=0x7ff7e0005d50 comp rx=0 tx=0).stop 2024-03-28T15:58:16.838 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:58:16.838+0000 7ff7f8800700 1 -- 172.21.15.183:0/4286654288 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff7f01021c0 msgr2=0x7ff7f019a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:58:16.838 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:58:16.838+0000 7ff7f8800700 1 --2- 172.21.15.183:0/4286654288 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff7f01021c0 0x7ff7f019a410 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff7e400c9a0 tx=0x7ff7e400cd60 comp rx=0 tx=0).stop 2024-03-28T15:58:16.838 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:58:16.838+0000 7ff7f8800700 1 -- 172.21.15.183:0/4286654288 shutdown_connections 2024-03-28T15:58:16.838 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:58:16.838+0000 7ff7f8800700 1 --2- 172.21.15.183:0/4286654288 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7ff7d80a2910 0x7ff7d80a4dd0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:58:16.838 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:58:16.838+0000 7ff7f8800700 1 --2- 172.21.15.183:0/4286654288 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7f0103d60 0x7ff7f01a19d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:58:16.839 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:58:16.838+0000 7ff7f8800700 1 --2- 172.21.15.183:0/4286654288 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff7f01021c0 0x7ff7f019a410 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:58:16.839 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:58:16.838+0000 7ff7f8800700 1 --2- 172.21.15.183:0/4286654288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7f01033c0 0x7ff7f019a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:58:16.839 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:58:16.838+0000 7ff7f8800700 1 -- 172.21.15.183:0/4286654288 >> 172.21.15.183:0/4286654288 conn(0x7ff7f00fd7a0 msgr2=0x7ff7f00743c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:58:16.839 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:58:16.839+0000 7ff7f8800700 1 -- 172.21.15.183:0/4286654288 shutdown_connections 2024-03-28T15:58:16.839 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:58:16.839+0000 7ff7f8800700 1 -- 172.21.15.183:0/4286654288 wait complete. 2024-03-28T15:58:17.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:16 smithi150 ceph-mon[34825]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:58:17.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:16 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:58:17.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:16 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:58:17.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:16 smithi150 ceph-mon[34825]: from='osd.8 [v2:172.21.15.183:6816/1025400095,v1:172.21.15.183:6817/1025400095]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-28T15:58:17.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:16 smithi150 ceph-mon[34825]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-28T15:58:17.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:16 smithi067 ceph-mon[31707]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:58:17.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:16 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:58:17.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:16 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:58:17.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:16 smithi067 ceph-mon[31707]: from='osd.8 [v2:172.21.15.183:6816/1025400095,v1:172.21.15.183:6817/1025400095]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-28T15:58:17.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:16 smithi067 ceph-mon[31707]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-28T15:58:17.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:16 smithi183 ceph-mon[35580]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:58:17.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:16 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:58:17.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:16 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:58:17.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:16 smithi183 ceph-mon[35580]: from='osd.8 [v2:172.21.15.183:6816/1025400095,v1:172.21.15.183:6817/1025400095]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-28T15:58:17.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:16 smithi183 ceph-mon[35580]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-28T15:58:17.620 DEBUG:teuthology.orchestra.run.smithi183:osd.8> sudo journalctl -f -n 0 -u ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@osd.8.service 2024-03-28T15:58:17.624 INFO:tasks.cephadm:Deploying osd.9 on smithi067 with /dev/vg_nvme/lv_1... 2024-03-28T15:58:17.624 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-28T15:58:17.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:17 smithi067 ceph-mon[31707]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-28T15:58:17.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:17 smithi067 ceph-mon[31707]: osdmap e61: 9 total, 8 up, 9 in 2024-03-28T15:58:17.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:17 smithi067 ceph-mon[31707]: from='osd.8 [v2:172.21.15.183:6816/1025400095,v1:172.21.15.183:6817/1025400095]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-03-28T15:58:17.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:17 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T15:58:17.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:17 smithi067 ceph-mon[31707]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-03-28T15:58:17.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:17 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:58:17.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:17 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:58:17.850 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 15:58:17 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T15:58:17.701+0000 7f76aea00700 -1 osd.8 62 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T15:58:17.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:17 smithi183 ceph-mon[35580]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-28T15:58:17.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:17 smithi183 ceph-mon[35580]: osdmap e61: 9 total, 8 up, 9 in 2024-03-28T15:58:17.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:17 smithi183 ceph-mon[35580]: from='osd.8 [v2:172.21.15.183:6816/1025400095,v1:172.21.15.183:6817/1025400095]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-03-28T15:58:17.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:17 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T15:58:17.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:17 smithi183 ceph-mon[35580]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-03-28T15:58:17.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:17 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:58:17.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:17 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:58:18.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:17 smithi150 ceph-mon[34825]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-28T15:58:18.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:17 smithi150 ceph-mon[34825]: osdmap e61: 9 total, 8 up, 9 in 2024-03-28T15:58:18.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:17 smithi150 ceph-mon[34825]: from='osd.8 [v2:172.21.15.183:6816/1025400095,v1:172.21.15.183:6817/1025400095]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-03-28T15:58:18.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:17 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T15:58:18.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:17 smithi150 ceph-mon[34825]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-03-28T15:58:18.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:17 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:58:18.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:17 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:58:18.088 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:58:19.012 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:18 smithi183 ceph-mon[35580]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:58:19.013 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:18 smithi183 ceph-mon[35580]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-03-28T15:58:19.013 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:18 smithi183 ceph-mon[35580]: osdmap e62: 9 total, 8 up, 9 in 2024-03-28T15:58:19.013 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:18 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T15:58:19.013 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:18 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T15:58:19.037 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:18 smithi067 ceph-mon[31707]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:58:19.037 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:18 smithi067 ceph-mon[31707]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-03-28T15:58:19.038 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:18 smithi067 ceph-mon[31707]: osdmap e62: 9 total, 8 up, 9 in 2024-03-28T15:58:19.038 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:18 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T15:58:19.038 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:18 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T15:58:19.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:18 smithi150 ceph-mon[34825]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:58:19.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:18 smithi150 ceph-mon[34825]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-03-28T15:58:19.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:18 smithi150 ceph-mon[34825]: osdmap e62: 9 total, 8 up, 9 in 2024-03-28T15:58:19.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:18 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T15:58:19.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:18 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T15:58:19.967 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:19 smithi067 ceph-mon[31707]: purged_snaps scrub starts 2024-03-28T15:58:19.967 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:19 smithi067 ceph-mon[31707]: purged_snaps scrub ok 2024-03-28T15:58:19.968 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:19 smithi067 ceph-mon[31707]: osd.8 [v2:172.21.15.183:6816/1025400095,v1:172.21.15.183:6817/1025400095] boot 2024-03-28T15:58:19.968 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:19 smithi067 ceph-mon[31707]: osdmap e63: 9 total, 9 up, 9 in 2024-03-28T15:58:19.968 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:19 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T15:58:19.968 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:19 smithi067 ceph-mon[31707]: osdmap e64: 9 total, 9 up, 9 in 2024-03-28T15:58:20.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:19 smithi150 ceph-mon[34825]: purged_snaps scrub starts 2024-03-28T15:58:20.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:19 smithi150 ceph-mon[34825]: purged_snaps scrub ok 2024-03-28T15:58:20.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:19 smithi150 ceph-mon[34825]: osd.8 [v2:172.21.15.183:6816/1025400095,v1:172.21.15.183:6817/1025400095] boot 2024-03-28T15:58:20.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:19 smithi150 ceph-mon[34825]: osdmap e63: 9 total, 9 up, 9 in 2024-03-28T15:58:20.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:19 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T15:58:20.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:19 smithi150 ceph-mon[34825]: osdmap e64: 9 total, 9 up, 9 in 2024-03-28T15:58:20.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:19 smithi183 ceph-mon[35580]: purged_snaps scrub starts 2024-03-28T15:58:20.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:19 smithi183 ceph-mon[35580]: purged_snaps scrub ok 2024-03-28T15:58:20.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:19 smithi183 ceph-mon[35580]: osd.8 [v2:172.21.15.183:6816/1025400095,v1:172.21.15.183:6817/1025400095] boot 2024-03-28T15:58:20.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:19 smithi183 ceph-mon[35580]: osdmap e63: 9 total, 9 up, 9 in 2024-03-28T15:58:20.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:19 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T15:58:20.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:19 smithi183 ceph-mon[35580]: osdmap e64: 9 total, 9 up, 9 in 2024-03-28T15:58:20.986 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:20 smithi067 ceph-mon[31707]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:58:20.986 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:20 smithi067 ceph-mon[31707]: osdmap e65: 9 total, 9 up, 9 in 2024-03-28T15:58:20.986 INFO:teuthology.orchestra.run.smithi067.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-28T15:58:20.986 INFO:teuthology.orchestra.run.smithi067.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-28T15:58:20.986 INFO:teuthology.orchestra.run.smithi067.stderr: stderr: 10+0 records in 2024-03-28T15:58:20.986 INFO:teuthology.orchestra.run.smithi067.stderr:10+0 records out 2024-03-28T15:58:20.986 INFO:teuthology.orchestra.run.smithi067.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0519653 s, 202 MB/s 2024-03-28T15:58:20.986 INFO:teuthology.orchestra.run.smithi067.stderr:--> Zapping successful for: 2024-03-28T15:58:21.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:20 smithi150 ceph-mon[34825]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:58:21.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:20 smithi150 ceph-mon[34825]: osdmap e65: 9 total, 9 up, 9 in 2024-03-28T15:58:21.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:20 smithi183 ceph-mon[35580]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:58:21.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:20 smithi183 ceph-mon[35580]: osdmap e65: 9 total, 9 up, 9 in 2024-03-28T15:58:21.827 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:21 smithi183 ceph-mon[35580]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-03-28T15:58:21.827 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:21 smithi183 ceph-mon[35580]: osdmap e66: 9 total, 9 up, 9 in 2024-03-28T15:58:21.906 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi067:vg_nvme/lv_1 2024-03-28T15:58:22.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:21 smithi150 ceph-mon[34825]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-03-28T15:58:22.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:21 smithi150 ceph-mon[34825]: osdmap e66: 9 total, 9 up, 9 in 2024-03-28T15:58:22.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:21 smithi067 ceph-mon[31707]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-03-28T15:58:22.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:21 smithi067 ceph-mon[31707]: osdmap e66: 9 total, 9 up, 9 in 2024-03-28T15:58:22.355 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:58:23.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:22 smithi150 ceph-mon[34825]: pgmap v256: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 281 MiB used, 804 GiB / 805 GiB avail; 2/6 objects degraded (33.333%) 2024-03-28T15:58:23.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:22 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:58:23.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:22 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:58:23.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:22 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:58:23.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:22 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:58:23.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:22 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:58:23.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:22 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:58:23.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:22 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:58:23.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:22 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:58:23.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:22 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:58:23.054 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:22 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:58:23.054 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:22 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:58:23.054 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:22 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:58:23.054 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:22 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:58:23.054 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:22 smithi150 ceph-mon[34825]: osdmap e67: 9 total, 9 up, 9 in 2024-03-28T15:58:23.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:22 smithi067 ceph-mon[31707]: pgmap v256: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 281 MiB used, 804 GiB / 805 GiB avail; 2/6 objects degraded (33.333%) 2024-03-28T15:58:23.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:22 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:58:23.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:22 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:58:23.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:22 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:58:23.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:22 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:58:23.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:22 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:58:23.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:22 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:58:23.071 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:22 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:58:23.071 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:22 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:58:23.071 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:22 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:58:23.071 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:22 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:58:23.071 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:22 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:58:23.071 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:22 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:58:23.071 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:22 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:58:23.071 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:22 smithi067 ceph-mon[31707]: osdmap e67: 9 total, 9 up, 9 in 2024-03-28T15:58:23.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:22 smithi183 ceph-mon[35580]: pgmap v256: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 281 MiB used, 804 GiB / 805 GiB avail; 2/6 objects degraded (33.333%) 2024-03-28T15:58:23.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:22 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:58:23.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:22 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:58:23.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:22 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:58:23.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:22 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:58:23.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:22 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:58:23.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:22 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:58:23.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:22 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:58:23.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:22 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:58:23.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:22 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:58:23.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:22 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:58:23.102 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:22 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:58:23.102 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:22 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:58:23.102 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:22 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:58:23.102 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:22 smithi183 ceph-mon[35580]: osdmap e67: 9 total, 9 up, 9 in 2024-03-28T15:58:23.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:23 smithi067 ceph-mon[31707]: Detected new or changed devices on smithi183 2024-03-28T15:58:23.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:23 smithi067 ceph-mon[31707]: Adjusting osd_memory_target on smithi183 to 5697M 2024-03-28T15:58:24.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:23 smithi150 ceph-mon[34825]: Detected new or changed devices on smithi183 2024-03-28T15:58:24.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:23 smithi150 ceph-mon[34825]: Adjusting osd_memory_target on smithi183 to 5697M 2024-03-28T15:58:24.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:23 smithi183 ceph-mon[35580]: Detected new or changed devices on smithi183 2024-03-28T15:58:24.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:23 smithi183 ceph-mon[35580]: Adjusting osd_memory_target on smithi183 to 5697M 2024-03-28T15:58:24.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:58:24.410+0000 7fb0cf200700 1 -- 172.21.15.67:0/128612397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0c8109fd0 msgr2=0x7fb0c810c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:58:24.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:58:24.410+0000 7fb0cf200700 1 --2- 172.21.15.67:0/128612397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0c8109fd0 0x7fb0c810c460 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb0c400b0a0 tx=0x7fb0c400b3b0 comp rx=0 tx=0).stop 2024-03-28T15:58:24.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:58:24.410+0000 7fb0cf200700 1 -- 172.21.15.67:0/128612397 shutdown_connections 2024-03-28T15:58:24.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:58:24.410+0000 7fb0cf200700 1 --2- 172.21.15.67:0/128612397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0c8109fd0 0x7fb0c810c460 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:58:24.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:58:24.410+0000 7fb0cf200700 1 --2- 172.21.15.67:0/128612397 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb0c80761c0 0x7fb0c8074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:58:24.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:58:24.410+0000 7fb0cf200700 1 --2- 172.21.15.67:0/128612397 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0c8074bf0 0x7fb0c8075070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:58:24.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:58:24.410+0000 7fb0cf200700 1 -- 172.21.15.67:0/128612397 >> 172.21.15.67:0/128612397 conn(0x7fb0c80fd6a0 msgr2=0x7fb0c80ffae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:58:24.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:58:24.410+0000 7fb0cf200700 1 -- 172.21.15.67:0/128612397 shutdown_connections 2024-03-28T15:58:24.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:58:24.410+0000 7fb0cf200700 1 -- 172.21.15.67:0/128612397 wait complete. 2024-03-28T15:58:24.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:58:24.412+0000 7fb0cf200700 1 Processor -- start 2024-03-28T15:58:24.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:58:24.412+0000 7fb0cf200700 1 -- start start 2024-03-28T15:58:24.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:58:24.412+0000 7fb0cf200700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0c8074bf0 0x7fb0c819a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:58:24.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:58:24.412+0000 7fb0cf200700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb0c80761c0 0x7fb0c819a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:58:24.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:58:24.412+0000 7fb0cf200700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0c8109fd0 0x7fb0c81a19a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:58:24.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:58:24.412+0000 7fb0cf200700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c810fe00 con 0x7fb0c8074bf0 2024-03-28T15:58:24.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:58:24.412+0000 7fb0cf200700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c810fc80 con 0x7fb0c8109fd0 2024-03-28T15:58:24.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:58:24.412+0000 7fb0cf200700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c810ff80 con 0x7fb0c80761c0 2024-03-28T15:58:24.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:58:24.413+0000 7fb0cde00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0c8074bf0 0x7fb0c819a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:58:24.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:58:24.413+0000 7fb0cde00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0c8074bf0 0x7fb0c819a3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48826/0 (socket says 172.21.15.67:48826) 2024-03-28T15:58:24.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:58:24.413+0000 7fb0cde00700 1 -- 172.21.15.67:0/1826630996 learned_addr learned my addr 172.21.15.67:0/1826630996 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T15:58:24.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:58:24.413+0000 7fb0cd400700 1 --2- 172.21.15.67:0/1826630996 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb0c80761c0 0x7fb0c819a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:58:24.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:58:24.413+0000 7fb0ce800700 1 --2- 172.21.15.67:0/1826630996 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0c8109fd0 0x7fb0c81a19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:58:24.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:58:24.413+0000 7fb0cde00700 1 -- 172.21.15.67:0/1826630996 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb0c80761c0 msgr2=0x7fb0c819a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:58:24.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:58:24.413+0000 7fb0cde00700 1 --2- 172.21.15.67:0/1826630996 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb0c80761c0 0x7fb0c819a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:58:24.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:58:24.413+0000 7fb0cde00700 1 -- 172.21.15.67:0/1826630996 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0c8109fd0 msgr2=0x7fb0c81a19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:58:24.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:58:24.413+0000 7fb0cde00700 1 --2- 172.21.15.67:0/1826630996 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0c8109fd0 0x7fb0c81a19a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:58:24.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:58:24.413+0000 7fb0cde00700 1 -- 172.21.15.67:0/1826630996 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0c81a2040 con 0x7fb0c8074bf0 2024-03-28T15:58:24.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:58:24.414+0000 7fb0cde00700 1 --2- 172.21.15.67:0/1826630996 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0c8074bf0 0x7fb0c819a3e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc00ea20 tx=0x7fb0bc00ede0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:58:24.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:58:24.414+0000 7fb0c2a00700 1 -- 172.21.15.67:0/1826630996 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0bc00cbd0 con 0x7fb0c8074bf0 2024-03-28T15:58:24.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:58:24.414+0000 7fb0c2a00700 1 -- 172.21.15.67:0/1826630996 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb0bc00cd70 con 0x7fb0c8074bf0 2024-03-28T15:58:24.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:58:24.414+0000 7fb0cf200700 1 -- 172.21.15.67:0/1826630996 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0c81a2330 con 0x7fb0c8074bf0 2024-03-28T15:58:24.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:58:24.414+0000 7fb0c2a00700 1 -- 172.21.15.67:0/1826630996 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0bc010430 con 0x7fb0c8074bf0 2024-03-28T15:58:24.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:58:24.414+0000 7fb0cf200700 1 -- 172.21.15.67:0/1826630996 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0c81a2820 con 0x7fb0c8074bf0 2024-03-28T15:58:24.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:58:24.415+0000 7fb0cf200700 1 -- 172.21.15.67:0/1826630996 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0c8068e50 con 0x7fb0c8074bf0 2024-03-28T15:58:24.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:58:24.417+0000 7fb0c2a00700 1 -- 172.21.15.67:0/1826630996 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fb0bc004750 con 0x7fb0c8074bf0 2024-03-28T15:58:24.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:58:24.420+0000 7fb0c2a00700 1 --2- 172.21.15.67:0/1826630996 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb0b00a2890 0x7fb0b00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:58:24.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:58:24.420+0000 7fb0c2a00700 1 -- 172.21.15.67:0/1826630996 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5829+0+0 (secure 0 0 0) 0x7fb0bc0ce760 con 0x7fb0c8074bf0 2024-03-28T15:58:24.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:58:24.420+0000 7fb0cd400700 1 --2- 172.21.15.67:0/1826630996 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb0b00a2890 0x7fb0b00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:58:24.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:58:24.421+0000 7fb0cd400700 1 --2- 172.21.15.67:0/1826630996 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb0b00a2890 0x7fb0b00a4d50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb0c8075ef0 tx=0x7fb0b8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:58:24.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:58:24.423+0000 7fb0c2a00700 1 -- 172.21.15.67:0/1826630996 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb0bc053740 con 0x7fb0c8074bf0 2024-03-28T15:58:24.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:58:24.664+0000 7fb0cf200700 1 -- 172.21.15.67:0/1826630996 --> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fb0c80625d0 con 0x7fb0b00a2890 2024-03-28T15:58:25.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:24 smithi150 ceph-mon[34825]: pgmap v259: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 281 MiB used, 804 GiB / 805 GiB avail; 2/6 objects degraded (33.333%) 2024-03-28T15:58:25.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:24 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:58:25.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:24 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:58:25.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:24 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:58:25.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:24 smithi067 ceph-mon[31707]: pgmap v259: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 281 MiB used, 804 GiB / 805 GiB avail; 2/6 objects degraded (33.333%) 2024-03-28T15:58:25.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:24 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:58:25.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:24 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:58:25.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:24 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:58:25.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:24 smithi183 ceph-mon[35580]: pgmap v259: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 281 MiB used, 804 GiB / 805 GiB avail; 2/6 objects degraded (33.333%) 2024-03-28T15:58:25.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:24 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:58:25.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:24 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:58:25.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:24 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:58:25.937 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:25 smithi067 ceph-mon[31707]: from='client.14403 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:58:26.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:25 smithi150 ceph-mon[34825]: from='client.14403 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:58:26.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:25 smithi183 ceph-mon[35580]: from='client.14403 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:58:27.022 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:26 smithi067 ceph-mon[31707]: pgmap v260: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 282 MiB used, 804 GiB / 805 GiB avail; 2/6 objects degraded (33.333%) 2024-03-28T15:58:27.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:26 smithi150 ceph-mon[34825]: pgmap v260: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 282 MiB used, 804 GiB / 805 GiB avail; 2/6 objects degraded (33.333%) 2024-03-28T15:58:27.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:26 smithi183 ceph-mon[35580]: pgmap v260: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 282 MiB used, 804 GiB / 805 GiB avail; 2/6 objects degraded (33.333%) 2024-03-28T15:58:28.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:27 smithi183 ceph-mon[35580]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-03-28T15:58:28.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:27 smithi183 ceph-mon[35580]: Cluster is now healthy 2024-03-28T15:58:28.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:27 smithi150 ceph-mon[34825]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-03-28T15:58:28.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:27 smithi150 ceph-mon[34825]: Cluster is now healthy 2024-03-28T15:58:28.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:27 smithi067 ceph-mon[31707]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-03-28T15:58:28.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:27 smithi067 ceph-mon[31707]: Cluster is now healthy 2024-03-28T15:58:29.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:28 smithi183 ceph-mon[35580]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 804 GiB / 805 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-03-28T15:58:29.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:28 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/4160189311' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "28a78e3a-0657-40aa-816f-16a6fadadcbb"}]: dispatch 2024-03-28T15:58:29.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:28 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/4160189311' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "28a78e3a-0657-40aa-816f-16a6fadadcbb"}]': finished 2024-03-28T15:58:29.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:28 smithi183 ceph-mon[35580]: osdmap e68: 10 total, 9 up, 10 in 2024-03-28T15:58:29.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:28 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T15:58:29.140 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:28 smithi067 ceph-mon[31707]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 804 GiB / 805 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-03-28T15:58:29.141 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:28 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/4160189311' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "28a78e3a-0657-40aa-816f-16a6fadadcbb"}]: dispatch 2024-03-28T15:58:29.141 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:28 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/4160189311' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "28a78e3a-0657-40aa-816f-16a6fadadcbb"}]': finished 2024-03-28T15:58:29.141 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:28 smithi067 ceph-mon[31707]: osdmap e68: 10 total, 9 up, 10 in 2024-03-28T15:58:29.141 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:28 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T15:58:29.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:28 smithi150 ceph-mon[34825]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 804 GiB / 805 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-03-28T15:58:29.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:28 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/4160189311' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "28a78e3a-0657-40aa-816f-16a6fadadcbb"}]: dispatch 2024-03-28T15:58:29.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:28 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/4160189311' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "28a78e3a-0657-40aa-816f-16a6fadadcbb"}]': finished 2024-03-28T15:58:29.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:28 smithi150 ceph-mon[34825]: osdmap e68: 10 total, 9 up, 10 in 2024-03-28T15:58:29.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:28 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T15:58:30.204 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:29 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/3780203129' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:58:30.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:29 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/3780203129' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:58:30.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:29 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/3780203129' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:58:31.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:30 smithi150 ceph-mon[34825]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 804 GiB / 805 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-28T15:58:31.821 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:30 smithi183 ceph-mon[35580]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 804 GiB / 805 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-28T15:58:31.821 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:30 smithi067 ceph-mon[31707]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 804 GiB / 805 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-28T15:58:33.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:32 smithi150 ceph-mon[34825]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 804 GiB / 805 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-03-28T15:58:33.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:32 smithi067 ceph-mon[31707]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 804 GiB / 805 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-03-28T15:58:33.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:32 smithi183 ceph-mon[35580]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 804 GiB / 805 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-03-28T15:58:35.230 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:34 smithi067 ceph-mon[31707]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-28T15:58:35.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:34 smithi150 ceph-mon[34825]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-28T15:58:35.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:34 smithi183 ceph-mon[35580]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-28T15:58:37.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:36 smithi067 ceph-mon[31707]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-28T15:58:37.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:36 smithi150 ceph-mon[34825]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-28T15:58:37.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:36 smithi183 ceph-mon[35580]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-28T15:58:38.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:37 smithi150 ceph-mon[34825]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 804 GiB / 805 GiB avail 2024-03-28T15:58:38.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:37 smithi067 ceph-mon[31707]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 804 GiB / 805 GiB avail 2024-03-28T15:58:38.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:37 smithi183 ceph-mon[35580]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 804 GiB / 805 GiB avail 2024-03-28T15:58:39.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:38 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-28T15:58:39.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:38 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:58:39.225 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:38 smithi067 ceph-mon[31707]: Deploying daemon osd.9 on smithi067 2024-03-28T15:58:39.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:38 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-28T15:58:39.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:38 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:58:39.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:38 smithi150 ceph-mon[34825]: Deploying daemon osd.9 on smithi067 2024-03-28T15:58:39.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:38 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-28T15:58:39.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:38 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:58:39.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:38 smithi183 ceph-mon[35580]: Deploying daemon osd.9 on smithi067 2024-03-28T15:58:40.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:39 smithi067 ceph-mon[31707]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 804 GiB / 805 GiB avail 2024-03-28T15:58:40.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:39 smithi150 ceph-mon[34825]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 804 GiB / 805 GiB avail 2024-03-28T15:58:40.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:39 smithi183 ceph-mon[35580]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 804 GiB / 805 GiB avail 2024-03-28T15:58:42.273 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:42 smithi067 ceph-mon[31707]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 804 GiB / 805 GiB avail 2024-03-28T15:58:42.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:42 smithi150 ceph-mon[34825]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 804 GiB / 805 GiB avail 2024-03-28T15:58:42.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:42 smithi183 ceph-mon[35580]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 804 GiB / 805 GiB avail 2024-03-28T15:58:44.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:44 smithi067 ceph-mon[31707]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 804 GiB / 805 GiB avail 2024-03-28T15:58:44.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:44 smithi183 ceph-mon[35580]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 804 GiB / 805 GiB avail 2024-03-28T15:58:44.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:44 smithi150 ceph-mon[34825]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 804 GiB / 805 GiB avail 2024-03-28T15:58:46.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:46 smithi067 ceph-mon[31707]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 804 GiB / 805 GiB avail 2024-03-28T15:58:46.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:46 smithi183 ceph-mon[35580]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 804 GiB / 805 GiB avail 2024-03-28T15:58:46.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:46 smithi150 ceph-mon[34825]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 804 GiB / 805 GiB avail 2024-03-28T15:58:47.382 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:47 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:58:47.382 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:47 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:58:47.382 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:47 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:58:47.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:47 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:58:47.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:47 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:58:47.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:47 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:58:47.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:47 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:58:47.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:47 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:58:47.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:47 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:58:48.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:48 smithi067 ceph-mon[31707]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 804 GiB / 805 GiB avail 2024-03-28T15:58:48.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:48 smithi150 ceph-mon[34825]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 804 GiB / 805 GiB avail 2024-03-28T15:58:48.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:48 smithi183 ceph-mon[35580]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 804 GiB / 805 GiB avail 2024-03-28T15:58:50.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:50 smithi067 ceph-mon[31707]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 804 GiB / 805 GiB avail 2024-03-28T15:58:50.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:50 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:58:50.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:50 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:58:50.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:50 smithi067 ceph-mon[31707]: from='osd.9 [v2:172.21.15.67:6826/76885409,v1:172.21.15.67:6827/76885409]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-28T15:58:51.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:50 smithi150 ceph-mon[34825]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 804 GiB / 805 GiB avail 2024-03-28T15:58:51.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:50 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:58:51.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:50 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:58:51.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:50 smithi150 ceph-mon[34825]: from='osd.9 [v2:172.21.15.67:6826/76885409,v1:172.21.15.67:6827/76885409]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-28T15:58:51.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:58:51.060+0000 7fb0c2a00700 1 -- 172.21.15.67:0/1826630996 <== mgr.14152 v2:172.21.15.67:6800/208726654 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fb0c80625d0 con 0x7fb0b00a2890 2024-03-28T15:58:51.061 INFO:teuthology.orchestra.run.smithi067.stdout:Created osd(s) 9 on host 'smithi067' 2024-03-28T15:58:51.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:58:51.064+0000 7fb0cf200700 1 -- 172.21.15.67:0/1826630996 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb0b00a2890 msgr2=0x7fb0b00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:58:51.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:58:51.064+0000 7fb0cf200700 1 --2- 172.21.15.67:0/1826630996 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb0b00a2890 0x7fb0b00a4d50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb0c8075ef0 tx=0x7fb0b8009260 comp rx=0 tx=0).stop 2024-03-28T15:58:51.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:58:51.064+0000 7fb0cf200700 1 -- 172.21.15.67:0/1826630996 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0c8074bf0 msgr2=0x7fb0c819a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:58:51.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:58:51.064+0000 7fb0cf200700 1 --2- 172.21.15.67:0/1826630996 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0c8074bf0 0x7fb0c819a3e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc00ea20 tx=0x7fb0bc00ede0 comp rx=0 tx=0).stop 2024-03-28T15:58:51.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:58:51.064+0000 7fb0cf200700 1 -- 172.21.15.67:0/1826630996 shutdown_connections 2024-03-28T15:58:51.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:58:51.064+0000 7fb0cf200700 1 --2- 172.21.15.67:0/1826630996 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb0b00a2890 0x7fb0b00a4d50 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:58:51.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:58:51.064+0000 7fb0cf200700 1 --2- 172.21.15.67:0/1826630996 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0c8109fd0 0x7fb0c81a19a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:58:51.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:58:51.065+0000 7fb0cf200700 1 --2- 172.21.15.67:0/1826630996 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0c8074bf0 0x7fb0c819a3e0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:58:51.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:58:51.065+0000 7fb0cf200700 1 --2- 172.21.15.67:0/1826630996 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb0c80761c0 0x7fb0c819a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:58:51.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:58:51.065+0000 7fb0cf200700 1 -- 172.21.15.67:0/1826630996 >> 172.21.15.67:0/1826630996 conn(0x7fb0c80fd6a0 msgr2=0x7fb0c8105ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:58:51.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:58:51.065+0000 7fb0cf200700 1 -- 172.21.15.67:0/1826630996 shutdown_connections 2024-03-28T15:58:51.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T15:58:51.065+0000 7fb0cf200700 1 -- 172.21.15.67:0/1826630996 wait complete. 2024-03-28T15:58:51.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:50 smithi183 ceph-mon[35580]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 804 GiB / 805 GiB avail 2024-03-28T15:58:51.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:50 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:58:51.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:50 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:58:51.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:50 smithi183 ceph-mon[35580]: from='osd.9 [v2:172.21.15.67:6826/76885409,v1:172.21.15.67:6827/76885409]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-28T15:58:51.818 DEBUG:teuthology.orchestra.run.smithi067:osd.9> sudo journalctl -f -n 0 -u ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@osd.9.service 2024-03-28T15:58:51.825 INFO:tasks.cephadm:Deploying osd.10 on smithi150 with /dev/vg_nvme/lv_1... 2024-03-28T15:58:51.825 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-28T15:58:52.003 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:51 smithi183 ceph-mon[35580]: from='osd.9 [v2:172.21.15.67:6826/76885409,v1:172.21.15.67:6827/76885409]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-28T15:58:52.003 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:51 smithi183 ceph-mon[35580]: osdmap e69: 10 total, 9 up, 10 in 2024-03-28T15:58:52.003 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:51 smithi183 ceph-mon[35580]: from='osd.9 [v2:172.21.15.67:6826/76885409,v1:172.21.15.67:6827/76885409]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-03-28T15:58:52.004 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:51 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T15:58:52.004 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:51 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:58:52.004 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:51 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:58:52.004 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:51 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:58:52.004 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:51 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:58:52.004 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:51 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:58:52.004 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:51 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:58:52.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:51 smithi150 ceph-mon[34825]: from='osd.9 [v2:172.21.15.67:6826/76885409,v1:172.21.15.67:6827/76885409]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-28T15:58:52.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:51 smithi150 ceph-mon[34825]: osdmap e69: 10 total, 9 up, 10 in 2024-03-28T15:58:52.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:51 smithi150 ceph-mon[34825]: from='osd.9 [v2:172.21.15.67:6826/76885409,v1:172.21.15.67:6827/76885409]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-03-28T15:58:52.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:51 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T15:58:52.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:51 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:58:52.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:51 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:58:52.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:51 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:58:52.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:51 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:58:52.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:51 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:58:52.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:51 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:58:52.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:51 smithi067 ceph-mon[31707]: from='osd.9 [v2:172.21.15.67:6826/76885409,v1:172.21.15.67:6827/76885409]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-28T15:58:52.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:51 smithi067 ceph-mon[31707]: osdmap e69: 10 total, 9 up, 10 in 2024-03-28T15:58:52.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:51 smithi067 ceph-mon[31707]: from='osd.9 [v2:172.21.15.67:6826/76885409,v1:172.21.15.67:6827/76885409]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-03-28T15:58:52.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:51 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T15:58:52.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:51 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:58:52.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:51 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:58:52.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:51 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:58:52.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:51 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:58:52.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:51 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:58:52.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:51 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:58:52.270 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T15:58:52.991 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:52 smithi067 ceph-mon[31707]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 804 GiB / 805 GiB avail 2024-03-28T15:58:52.991 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:52 smithi067 ceph-mon[31707]: from='osd.9 [v2:172.21.15.67:6826/76885409,v1:172.21.15.67:6827/76885409]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-03-28T15:58:52.991 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:52 smithi067 ceph-mon[31707]: osdmap e70: 10 total, 9 up, 10 in 2024-03-28T15:58:52.991 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:52 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T15:58:52.991 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:52 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T15:58:53.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:52 smithi150 ceph-mon[34825]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 804 GiB / 805 GiB avail 2024-03-28T15:58:53.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:52 smithi150 ceph-mon[34825]: from='osd.9 [v2:172.21.15.67:6826/76885409,v1:172.21.15.67:6827/76885409]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-03-28T15:58:53.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:52 smithi150 ceph-mon[34825]: osdmap e70: 10 total, 9 up, 10 in 2024-03-28T15:58:53.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:52 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T15:58:53.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:52 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T15:58:53.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:52 smithi183 ceph-mon[35580]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 804 GiB / 805 GiB avail 2024-03-28T15:58:53.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:52 smithi183 ceph-mon[35580]: from='osd.9 [v2:172.21.15.67:6826/76885409,v1:172.21.15.67:6827/76885409]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-03-28T15:58:53.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:52 smithi183 ceph-mon[35580]: osdmap e70: 10 total, 9 up, 10 in 2024-03-28T15:58:53.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:52 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T15:58:53.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:52 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T15:58:53.900 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:53 smithi067 ceph-mon[31707]: purged_snaps scrub starts 2024-03-28T15:58:53.900 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:53 smithi067 ceph-mon[31707]: purged_snaps scrub ok 2024-03-28T15:58:53.900 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:53 smithi067 ceph-mon[31707]: osd.9 [v2:172.21.15.67:6826/76885409,v1:172.21.15.67:6827/76885409] boot 2024-03-28T15:58:53.900 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:53 smithi067 ceph-mon[31707]: osdmap e71: 10 total, 10 up, 10 in 2024-03-28T15:58:53.900 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:53 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T15:58:53.925 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:53 smithi150 ceph-mon[34825]: purged_snaps scrub starts 2024-03-28T15:58:53.926 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:53 smithi150 ceph-mon[34825]: purged_snaps scrub ok 2024-03-28T15:58:53.926 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:53 smithi150 ceph-mon[34825]: osd.9 [v2:172.21.15.67:6826/76885409,v1:172.21.15.67:6827/76885409] boot 2024-03-28T15:58:53.926 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:53 smithi150 ceph-mon[34825]: osdmap e71: 10 total, 10 up, 10 in 2024-03-28T15:58:53.926 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:53 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T15:58:54.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:53 smithi183 ceph-mon[35580]: purged_snaps scrub starts 2024-03-28T15:58:54.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:53 smithi183 ceph-mon[35580]: purged_snaps scrub ok 2024-03-28T15:58:54.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:53 smithi183 ceph-mon[35580]: osd.9 [v2:172.21.15.67:6826/76885409,v1:172.21.15.67:6827/76885409] boot 2024-03-28T15:58:54.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:53 smithi183 ceph-mon[35580]: osdmap e71: 10 total, 10 up, 10 in 2024-03-28T15:58:54.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:53 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T15:58:55.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:54 smithi150 ceph-mon[34825]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:58:55.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:54 smithi150 ceph-mon[34825]: osdmap e72: 10 total, 10 up, 10 in 2024-03-28T15:58:55.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:54 smithi067 ceph-mon[31707]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:58:55.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:54 smithi067 ceph-mon[31707]: osdmap e72: 10 total, 10 up, 10 in 2024-03-28T15:58:55.080 INFO:teuthology.orchestra.run.smithi150.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-28T15:58:55.080 INFO:teuthology.orchestra.run.smithi150.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-28T15:58:55.081 INFO:teuthology.orchestra.run.smithi150.stderr: stderr: 10+0 records in 2024-03-28T15:58:55.081 INFO:teuthology.orchestra.run.smithi150.stderr:10+0 records out 2024-03-28T15:58:55.081 INFO:teuthology.orchestra.run.smithi150.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0323634 s, 324 MB/s 2024-03-28T15:58:55.081 INFO:teuthology.orchestra.run.smithi150.stderr:--> Zapping successful for: 2024-03-28T15:58:55.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:54 smithi183 ceph-mon[35580]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:58:55.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:54 smithi183 ceph-mon[35580]: osdmap e72: 10 total, 10 up, 10 in 2024-03-28T15:58:55.926 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi150:vg_nvme/lv_1 2024-03-28T15:58:56.357 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T15:58:57.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:56 smithi150 ceph-mon[34825]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:58:57.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:56 smithi067 ceph-mon[31707]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:58:57.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:56 smithi183 ceph-mon[35580]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:58:57.898 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:57 smithi150 ceph-mon[34825]: Detected new or changed devices on smithi067 2024-03-28T15:58:57.898 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:57 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:58:57.898 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:57 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:58:57.898 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:57 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:58:58.182 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:57 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:58:58.182 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:57 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:58:58.182 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:57 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:58:58.182 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:57 smithi150 ceph-mon[34825]: Adjusting osd_memory_target on smithi067 to 4272M 2024-03-28T15:58:58.182 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:57 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:58:58.183 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:57 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:58:58.183 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:57 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:58:58.183 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:57 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:58:58.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:57 smithi067 ceph-mon[31707]: Detected new or changed devices on smithi067 2024-03-28T15:58:58.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:57 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:58:58.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:57 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:58:58.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:57 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:58:58.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:57 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:58:58.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:57 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:58:58.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:57 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:58:58.321 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:57 smithi067 ceph-mon[31707]: Adjusting osd_memory_target on smithi067 to 4272M 2024-03-28T15:58:58.321 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:57 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:58:58.321 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:57 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:58:58.321 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:57 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:58:58.321 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:57 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:58:58.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:57 smithi183 ceph-mon[35580]: Detected new or changed devices on smithi067 2024-03-28T15:58:58.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:57 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:58:58.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:57 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:58:58.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:57 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:58:58.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:57 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:58:58.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:57 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:58:58.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:57 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:58:58.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:57 smithi183 ceph-mon[35580]: Adjusting osd_memory_target on smithi067 to 4272M 2024-03-28T15:58:58.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:57 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:58:58.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:57 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:58:58.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:57 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:58:58.352 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:57 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:58:58.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:58:58.521+0000 7ff7afe00700 1 -- 172.21.15.150:0/1073863922 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7b0109fe0 msgr2=0x7ff7b010c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:58:58.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:58:58.521+0000 7ff7afe00700 1 --2- 172.21.15.150:0/1073863922 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7b0109fe0 0x7ff7b010c470 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff7a800b0a0 tx=0x7ff7a800b3b0 comp rx=0 tx=0).stop 2024-03-28T15:58:58.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:58:58.522+0000 7ff7afe00700 1 -- 172.21.15.150:0/1073863922 shutdown_connections 2024-03-28T15:58:58.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:58:58.522+0000 7ff7afe00700 1 --2- 172.21.15.150:0/1073863922 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7b0109fe0 0x7ff7b010c470 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:58:58.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:58:58.522+0000 7ff7afe00700 1 --2- 172.21.15.150:0/1073863922 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff7b0076160 0x7ff7b00745c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:58:58.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:58:58.522+0000 7ff7afe00700 1 --2- 172.21.15.150:0/1073863922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7b0074b90 0x7ff7b0075010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:58:58.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:58:58.522+0000 7ff7afe00700 1 -- 172.21.15.150:0/1073863922 >> 172.21.15.150:0/1073863922 conn(0x7ff7b00fd6a0 msgr2=0x7ff7b00ffae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:58:58.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:58:58.522+0000 7ff7afe00700 1 -- 172.21.15.150:0/1073863922 shutdown_connections 2024-03-28T15:58:58.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:58:58.522+0000 7ff7afe00700 1 -- 172.21.15.150:0/1073863922 wait complete. 2024-03-28T15:58:58.526 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:58:58.523+0000 7ff7afe00700 1 Processor -- start 2024-03-28T15:58:58.526 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:58:58.523+0000 7ff7afe00700 1 -- start start 2024-03-28T15:58:58.526 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:58:58.524+0000 7ff7afe00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff7b0074b90 0x7ff7b019a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:58:58.526 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:58:58.524+0000 7ff7afe00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7b0076160 0x7ff7b019a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:58:58.526 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:58:58.524+0000 7ff7afe00700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7b0109fe0 0x7ff7b01a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:58:58.526 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:58:58.524+0000 7ff7afe00700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b010fd60 con 0x7ff7b0076160 2024-03-28T15:58:58.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:58:58.524+0000 7ff7afe00700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b010fbe0 con 0x7ff7b0109fe0 2024-03-28T15:58:58.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:58:58.524+0000 7ff7afe00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b010fee0 con 0x7ff7b0074b90 2024-03-28T15:58:58.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:58:58.524+0000 7ff7af400700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7b0109fe0 0x7ff7b01a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:58:58.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:58:58.524+0000 7ff7af400700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7b0109fe0 0x7ff7b01a1870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45384/0 (socket says 172.21.15.150:45384) 2024-03-28T15:58:58.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:58:58.524+0000 7ff7af400700 1 -- 172.21.15.150:0/4291542974 learned_addr learned my addr 172.21.15.150:0/4291542974 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-28T15:58:58.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:58:58.524+0000 7ff7aea00700 1 --2- 172.21.15.150:0/4291542974 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff7b0074b90 0x7ff7b019a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:58:58.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:58:58.524+0000 7ff7ae000700 1 --2- 172.21.15.150:0/4291542974 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7b0076160 0x7ff7b019a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:58:58.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:58:58.524+0000 7ff7af400700 1 -- 172.21.15.150:0/4291542974 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff7b0074b90 msgr2=0x7ff7b019a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:58:58.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:58:58.524+0000 7ff7af400700 1 --2- 172.21.15.150:0/4291542974 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff7b0074b90 0x7ff7b019a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:58:58.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:58:58.524+0000 7ff7af400700 1 -- 172.21.15.150:0/4291542974 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7b0076160 msgr2=0x7ff7b019a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:58:58.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:58:58.524+0000 7ff7af400700 1 --2- 172.21.15.150:0/4291542974 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7b0076160 0x7ff7b019a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:58:58.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:58:58.524+0000 7ff7af400700 1 -- 172.21.15.150:0/4291542974 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7b01a1f10 con 0x7ff7b0109fe0 2024-03-28T15:58:58.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:58:58.524+0000 7ff7ae000700 1 --2- 172.21.15.150:0/4291542974 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7b0076160 0x7ff7b019a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T15:58:58.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:58:58.525+0000 7ff7af400700 1 --2- 172.21.15.150:0/4291542974 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7b0109fe0 0x7ff7b01a1870 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff7a800ced0 tx=0x7ff7a800cfb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:58:58.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:58:58.525+0000 7ff7a7400700 1 -- 172.21.15.150:0/4291542974 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a8021070 con 0x7ff7b0109fe0 2024-03-28T15:58:58.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:58:58.525+0000 7ff7a7400700 1 -- 172.21.15.150:0/4291542974 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff7a801d900 con 0x7ff7b0109fe0 2024-03-28T15:58:58.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:58:58.525+0000 7ff7afe00700 1 -- 172.21.15.150:0/4291542974 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7b01a21a0 con 0x7ff7b0109fe0 2024-03-28T15:58:58.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:58:58.525+0000 7ff7a7400700 1 -- 172.21.15.150:0/4291542974 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a8005420 con 0x7ff7b0109fe0 2024-03-28T15:58:58.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:58:58.525+0000 7ff7afe00700 1 -- 172.21.15.150:0/4291542974 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7b01a26e0 con 0x7ff7b0109fe0 2024-03-28T15:58:58.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:58:58.527+0000 7ff7afe00700 1 -- 172.21.15.150:0/4291542974 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7b0068e50 con 0x7ff7b0109fe0 2024-03-28T15:58:58.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:58:58.530+0000 7ff7a7400700 1 -- 172.21.15.150:0/4291542974 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ff7a8024070 con 0x7ff7b0109fe0 2024-03-28T15:58:58.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:58:58.531+0000 7ff7a7400700 1 --2- 172.21.15.150:0/4291542974 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7ff7940a2890 0x7ff7940a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:58:58.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:58:58.531+0000 7ff7a7400700 1 -- 172.21.15.150:0/4291542974 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6261+0+0 (secure 0 0 0) 0x7ff7a800f020 con 0x7ff7b0109fe0 2024-03-28T15:58:58.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:58:58.532+0000 7ff7aea00700 1 --2- 172.21.15.150:0/4291542974 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7ff7940a2890 0x7ff7940a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:58:58.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:58:58.533+0000 7ff7aea00700 1 --2- 172.21.15.150:0/4291542974 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7ff7940a2890 0x7ff7940a4d50 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff7a0005ea0 tx=0x7ff7a0005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:58:58.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:58:58.535+0000 7ff7a7400700 1 -- 172.21.15.150:0/4291542974 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff7a802b020 con 0x7ff7b0109fe0 2024-03-28T15:58:58.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:58:58.766+0000 7ff7afe00700 1 -- 172.21.15.150:0/4291542974 --> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7ff7b00625d0 con 0x7ff7940a2890 2024-03-28T15:58:59.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:58 smithi150 ceph-mon[34825]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:58:59.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:58 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:58:59.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:58 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:58:59.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:58 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:58:59.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:58 smithi067 ceph-mon[31707]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:58:59.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:58 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:58:59.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:58 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:58:59.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:58 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:58:59.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:58 smithi183 ceph-mon[35580]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:58:59.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:58 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:58:59.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:58 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:58:59.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:58 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:59:00.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:58:59 smithi150 ceph-mon[34825]: from='client.34334 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:59:00.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:58:59 smithi067 ceph-mon[31707]: from='client.34334 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:59:00.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:58:59 smithi183 ceph-mon[35580]: from='client.34334 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:59:01.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:00 smithi150 ceph-mon[34825]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:59:01.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:00 smithi067 ceph-mon[31707]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:59:01.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:00 smithi183 ceph-mon[35580]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:59:03.235 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:02 smithi150 ceph-mon[34825]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:59:03.235 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:02 smithi150 ceph-mon[34825]: from='client.? 172.21.15.150:0/1291785611' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d287907e-424f-4b49-a06e-4613c6de0474"}]: dispatch 2024-03-28T15:59:03.235 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:02 smithi150 ceph-mon[34825]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d287907e-424f-4b49-a06e-4613c6de0474"}]: dispatch 2024-03-28T15:59:03.235 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:02 smithi150 ceph-mon[34825]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d287907e-424f-4b49-a06e-4613c6de0474"}]': finished 2024-03-28T15:59:03.235 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:02 smithi150 ceph-mon[34825]: osdmap e73: 11 total, 10 up, 11 in 2024-03-28T15:59:03.235 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:02 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T15:59:03.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:02 smithi067 ceph-mon[31707]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:59:03.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:02 smithi067 ceph-mon[31707]: from='client.? 172.21.15.150:0/1291785611' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d287907e-424f-4b49-a06e-4613c6de0474"}]: dispatch 2024-03-28T15:59:03.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:02 smithi067 ceph-mon[31707]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d287907e-424f-4b49-a06e-4613c6de0474"}]: dispatch 2024-03-28T15:59:03.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:02 smithi067 ceph-mon[31707]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d287907e-424f-4b49-a06e-4613c6de0474"}]': finished 2024-03-28T15:59:03.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:02 smithi067 ceph-mon[31707]: osdmap e73: 11 total, 10 up, 11 in 2024-03-28T15:59:03.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:02 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T15:59:03.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:02 smithi183 ceph-mon[35580]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:59:03.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:02 smithi183 ceph-mon[35580]: from='client.? 172.21.15.150:0/1291785611' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d287907e-424f-4b49-a06e-4613c6de0474"}]: dispatch 2024-03-28T15:59:03.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:02 smithi183 ceph-mon[35580]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d287907e-424f-4b49-a06e-4613c6de0474"}]: dispatch 2024-03-28T15:59:03.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:02 smithi183 ceph-mon[35580]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d287907e-424f-4b49-a06e-4613c6de0474"}]': finished 2024-03-28T15:59:03.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:02 smithi183 ceph-mon[35580]: osdmap e73: 11 total, 10 up, 11 in 2024-03-28T15:59:03.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:02 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T15:59:04.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:03 smithi067 ceph-mon[31707]: from='client.? 172.21.15.150:0/4164923450' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:59:04.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:03 smithi150 ceph-mon[34825]: from='client.? 172.21.15.150:0/4164923450' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:59:04.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:03 smithi183 ceph-mon[35580]: from='client.? 172.21.15.150:0/4164923450' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:59:05.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:04 smithi150 ceph-mon[34825]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:59:05.322 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:04 smithi067 ceph-mon[31707]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:59:05.354 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:04 smithi183 ceph-mon[35580]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:59:06.245 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:05 smithi067 ceph-mon[31707]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:59:06.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:05 smithi150 ceph-mon[34825]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:59:06.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:05 smithi183 ceph-mon[35580]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:59:08.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:08 smithi150 ceph-mon[34825]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:59:08.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:08 smithi067 ceph-mon[31707]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:59:08.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:08 smithi183 ceph-mon[35580]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:59:10.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:10 smithi150 ceph-mon[34825]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:59:10.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:10 smithi067 ceph-mon[31707]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:59:10.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:10 smithi183 ceph-mon[35580]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:59:12.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:12 smithi150 ceph-mon[34825]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:59:12.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:12 smithi183 ceph-mon[35580]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:59:12.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:12 smithi067 ceph-mon[31707]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:59:13.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:13 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-28T15:59:13.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:13 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:59:13.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:13 smithi183 ceph-mon[35580]: Deploying daemon osd.10 on smithi150 2024-03-28T15:59:13.352 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:13 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-28T15:59:13.352 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:13 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:59:13.352 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:13 smithi150 ceph-mon[34825]: Deploying daemon osd.10 on smithi150 2024-03-28T15:59:13.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:13 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-28T15:59:13.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:13 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:59:13.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:13 smithi067 ceph-mon[31707]: Deploying daemon osd.10 on smithi150 2024-03-28T15:59:14.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:14 smithi150 ceph-mon[34825]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:59:14.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:14 smithi183 ceph-mon[35580]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:59:14.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:14 smithi067 ceph-mon[31707]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:59:16.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:16 smithi183 ceph-mon[35580]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:59:16.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:16 smithi150 ceph-mon[34825]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:59:16.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:16 smithi067 ceph-mon[31707]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:59:18.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:18 smithi150 ceph-mon[34825]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:59:18.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:18 smithi067 ceph-mon[31707]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:59:18.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:18 smithi183 ceph-mon[35580]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:59:20.444 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:20 smithi150 ceph-mon[34825]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:59:20.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:20 smithi067 ceph-mon[31707]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:59:20.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:20 smithi183 ceph-mon[35580]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:59:21.976 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:21 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:59:21.976 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:21 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:59:21.976 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:21 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:59:22.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:21 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:59:22.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:21 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:59:22.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:21 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:59:22.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:21 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:59:22.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:21 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:59:22.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:21 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:59:22.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:22 smithi150 ceph-mon[34825]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:59:23.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:22 smithi067 ceph-mon[31707]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:59:23.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:22 smithi183 ceph-mon[35580]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:59:25.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:24 smithi150 ceph-mon[34825]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:59:25.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:24 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:59:25.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:24 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:59:25.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:24 smithi150 ceph-mon[34825]: from='osd.10 [v2:172.21.15.150:6824/3489982080,v1:172.21.15.150:6825/3489982080]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-28T15:59:25.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:24 smithi150 ceph-mon[34825]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-28T15:59:25.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:24 smithi067 ceph-mon[31707]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:59:25.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:24 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:59:25.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:24 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:59:25.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:24 smithi067 ceph-mon[31707]: from='osd.10 [v2:172.21.15.150:6824/3489982080,v1:172.21.15.150:6825/3489982080]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-28T15:59:25.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:24 smithi067 ceph-mon[31707]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-28T15:59:25.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:24 smithi183 ceph-mon[35580]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:59:25.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:24 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:59:25.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:24 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:59:25.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:24 smithi183 ceph-mon[35580]: from='osd.10 [v2:172.21.15.150:6824/3489982080,v1:172.21.15.150:6825/3489982080]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-28T15:59:25.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:24 smithi183 ceph-mon[35580]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-28T15:59:25.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:59:25.757+0000 7ff7a7400700 1 -- 172.21.15.150:0/4291542974 <== mgr.14152 v2:172.21.15.67:6800/208726654 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7ff7b00625d0 con 0x7ff7940a2890 2024-03-28T15:59:25.759 INFO:teuthology.orchestra.run.smithi150.stdout:Created osd(s) 10 on host 'smithi150' 2024-03-28T15:59:25.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:59:25.761+0000 7ff7afe00700 1 -- 172.21.15.150:0/4291542974 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7ff7940a2890 msgr2=0x7ff7940a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:59:25.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:59:25.761+0000 7ff7afe00700 1 --2- 172.21.15.150:0/4291542974 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7ff7940a2890 0x7ff7940a4d50 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff7a0005ea0 tx=0x7ff7a0005e30 comp rx=0 tx=0).stop 2024-03-28T15:59:25.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:59:25.761+0000 7ff7afe00700 1 -- 172.21.15.150:0/4291542974 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7b0109fe0 msgr2=0x7ff7b01a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:59:25.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:59:25.761+0000 7ff7afe00700 1 --2- 172.21.15.150:0/4291542974 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7b0109fe0 0x7ff7b01a1870 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff7a800ced0 tx=0x7ff7a800cfb0 comp rx=0 tx=0).stop 2024-03-28T15:59:25.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:59:25.761+0000 7ff7afe00700 1 -- 172.21.15.150:0/4291542974 shutdown_connections 2024-03-28T15:59:25.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:59:25.761+0000 7ff7afe00700 1 --2- 172.21.15.150:0/4291542974 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7ff7940a2890 0x7ff7940a4d50 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:59:25.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:59:25.761+0000 7ff7afe00700 1 --2- 172.21.15.150:0/4291542974 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7b0109fe0 0x7ff7b01a1870 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:59:25.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:59:25.761+0000 7ff7afe00700 1 --2- 172.21.15.150:0/4291542974 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff7b0074b90 0x7ff7b019a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:59:25.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:59:25.761+0000 7ff7afe00700 1 --2- 172.21.15.150:0/4291542974 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7b0076160 0x7ff7b019a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:59:25.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:59:25.761+0000 7ff7afe00700 1 -- 172.21.15.150:0/4291542974 >> 172.21.15.150:0/4291542974 conn(0x7ff7b00fd6a0 msgr2=0x7ff7b0105f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:59:25.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:59:25.761+0000 7ff7afe00700 1 -- 172.21.15.150:0/4291542974 shutdown_connections 2024-03-28T15:59:25.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-28T15:59:25.761+0000 7ff7afe00700 1 -- 172.21.15.150:0/4291542974 wait complete. 2024-03-28T15:59:26.579 DEBUG:teuthology.orchestra.run.smithi150:osd.10> sudo journalctl -f -n 0 -u ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@osd.10.service 2024-03-28T15:59:26.584 INFO:tasks.cephadm:Deploying osd.11 on smithi183 with /dev/vg_nvme/lv_1... 2024-03-28T15:59:26.584 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-28T15:59:26.767 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:26 smithi150 ceph-mon[34825]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:59:26.768 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:26 smithi150 ceph-mon[34825]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-28T15:59:26.768 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:26 smithi150 ceph-mon[34825]: osdmap e74: 11 total, 10 up, 11 in 2024-03-28T15:59:26.768 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:26 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T15:59:26.768 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:26 smithi150 ceph-mon[34825]: from='osd.10 [v2:172.21.15.150:6824/3489982080,v1:172.21.15.150:6825/3489982080]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-03-28T15:59:26.768 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:26 smithi150 ceph-mon[34825]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-03-28T15:59:26.768 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:26 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:59:26.768 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:26 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:59:26.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:26 smithi067 ceph-mon[31707]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:59:26.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:26 smithi067 ceph-mon[31707]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-28T15:59:26.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:26 smithi067 ceph-mon[31707]: osdmap e74: 11 total, 10 up, 11 in 2024-03-28T15:59:26.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:26 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T15:59:26.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:26 smithi067 ceph-mon[31707]: from='osd.10 [v2:172.21.15.150:6824/3489982080,v1:172.21.15.150:6825/3489982080]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-03-28T15:59:26.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:26 smithi067 ceph-mon[31707]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-03-28T15:59:26.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:26 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:59:26.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:26 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:59:26.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:26 smithi183 ceph-mon[35580]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:59:26.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:26 smithi183 ceph-mon[35580]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-28T15:59:26.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:26 smithi183 ceph-mon[35580]: osdmap e74: 11 total, 10 up, 11 in 2024-03-28T15:59:26.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:26 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T15:59:26.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:26 smithi183 ceph-mon[35580]: from='osd.10 [v2:172.21.15.150:6824/3489982080,v1:172.21.15.150:6825/3489982080]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-03-28T15:59:26.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:26 smithi183 ceph-mon[35580]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-03-28T15:59:26.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:26 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:59:26.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:26 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:59:27.027 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T15:59:27.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:27 smithi150 ceph-mon[34825]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-03-28T15:59:27.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:27 smithi150 ceph-mon[34825]: osdmap e75: 11 total, 10 up, 11 in 2024-03-28T15:59:27.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:27 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T15:59:27.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:27 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T15:59:27.803 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 15:59:27 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T15:59:27.524+0000 7f6b23e00700 -1 osd.10 0 waiting for initial osdmap 2024-03-28T15:59:27.803 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 15:59:27 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T15:59:27.595+0000 7f6b19c00700 -1 osd.10 76 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T15:59:27.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:27 smithi067 ceph-mon[31707]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-03-28T15:59:27.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:27 smithi067 ceph-mon[31707]: osdmap e75: 11 total, 10 up, 11 in 2024-03-28T15:59:27.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:27 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T15:59:27.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:27 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T15:59:27.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:27 smithi183 ceph-mon[35580]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-03-28T15:59:27.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:27 smithi183 ceph-mon[35580]: osdmap e75: 11 total, 10 up, 11 in 2024-03-28T15:59:27.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:27 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T15:59:27.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:27 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T15:59:28.704 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:28 smithi183 ceph-mon[35580]: purged_snaps scrub starts 2024-03-28T15:59:28.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:28 smithi183 ceph-mon[35580]: purged_snaps scrub ok 2024-03-28T15:59:28.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:28 smithi183 ceph-mon[35580]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:59:28.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:28 smithi183 ceph-mon[35580]: osdmap e76: 11 total, 10 up, 11 in 2024-03-28T15:59:28.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:28 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T15:59:28.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:28 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T15:59:28.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:28 smithi183 ceph-mon[35580]: from='osd.10 ' entity='osd.10' 2024-03-28T15:59:28.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:28 smithi150 ceph-mon[34825]: purged_snaps scrub starts 2024-03-28T15:59:28.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:28 smithi150 ceph-mon[34825]: purged_snaps scrub ok 2024-03-28T15:59:28.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:28 smithi150 ceph-mon[34825]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:59:28.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:28 smithi150 ceph-mon[34825]: osdmap e76: 11 total, 10 up, 11 in 2024-03-28T15:59:28.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:28 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T15:59:28.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:28 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T15:59:28.804 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:28 smithi150 ceph-mon[34825]: from='osd.10 ' entity='osd.10' 2024-03-28T15:59:28.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:28 smithi067 ceph-mon[31707]: purged_snaps scrub starts 2024-03-28T15:59:28.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:28 smithi067 ceph-mon[31707]: purged_snaps scrub ok 2024-03-28T15:59:28.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:28 smithi067 ceph-mon[31707]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:59:28.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:28 smithi067 ceph-mon[31707]: osdmap e76: 11 total, 10 up, 11 in 2024-03-28T15:59:28.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:28 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T15:59:28.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:28 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T15:59:28.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:28 smithi067 ceph-mon[31707]: from='osd.10 ' entity='osd.10' 2024-03-28T15:59:29.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:29 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T15:59:29.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:29 smithi150 ceph-mon[34825]: osd.10 [v2:172.21.15.150:6824/3489982080,v1:172.21.15.150:6825/3489982080] boot 2024-03-28T15:59:29.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:29 smithi150 ceph-mon[34825]: osdmap e77: 11 total, 11 up, 11 in 2024-03-28T15:59:29.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:29 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T15:59:29.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:29 smithi150 ceph-mon[34825]: osdmap e78: 11 total, 11 up, 11 in 2024-03-28T15:59:29.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:29 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T15:59:29.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:29 smithi067 ceph-mon[31707]: osd.10 [v2:172.21.15.150:6824/3489982080,v1:172.21.15.150:6825/3489982080] boot 2024-03-28T15:59:29.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:29 smithi067 ceph-mon[31707]: osdmap e77: 11 total, 11 up, 11 in 2024-03-28T15:59:29.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:29 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T15:59:29.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:29 smithi067 ceph-mon[31707]: osdmap e78: 11 total, 11 up, 11 in 2024-03-28T15:59:29.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:29 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T15:59:29.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:29 smithi183 ceph-mon[35580]: osd.10 [v2:172.21.15.150:6824/3489982080,v1:172.21.15.150:6825/3489982080] boot 2024-03-28T15:59:29.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:29 smithi183 ceph-mon[35580]: osdmap e77: 11 total, 11 up, 11 in 2024-03-28T15:59:29.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:29 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T15:59:29.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:29 smithi183 ceph-mon[35580]: osdmap e78: 11 total, 11 up, 11 in 2024-03-28T15:59:29.858 INFO:teuthology.orchestra.run.smithi183.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-28T15:59:29.858 INFO:teuthology.orchestra.run.smithi183.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-28T15:59:29.858 INFO:teuthology.orchestra.run.smithi183.stderr: stderr: 10+0 records in 2024-03-28T15:59:29.858 INFO:teuthology.orchestra.run.smithi183.stderr:10+0 records out 2024-03-28T15:59:29.858 INFO:teuthology.orchestra.run.smithi183.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0374925 s, 280 MB/s 2024-03-28T15:59:29.858 INFO:teuthology.orchestra.run.smithi183.stderr:--> Zapping successful for: 2024-03-28T15:59:30.610 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi183:vg_nvme/lv_1 2024-03-28T15:59:30.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:30 smithi067 ceph-mon[31707]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:59:30.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:30 smithi067 ceph-mon[31707]: osdmap e79: 11 total, 11 up, 11 in 2024-03-28T15:59:30.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:30 smithi183 ceph-mon[35580]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:59:30.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:30 smithi183 ceph-mon[35580]: osdmap e79: 11 total, 11 up, 11 in 2024-03-28T15:59:31.009 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:30 smithi150 ceph-mon[34825]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:59:31.009 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:30 smithi150 ceph-mon[34825]: osdmap e79: 11 total, 11 up, 11 in 2024-03-28T15:59:31.115 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T15:59:32.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:32 smithi183 ceph-mon[35580]: pgmap v305: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-28T15:59:32.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:32 smithi183 ceph-mon[35580]: Detected new or changed devices on smithi150 2024-03-28T15:59:32.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:32 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:59:32.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:32 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:59:32.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:32 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:59:32.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:32 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:59:32.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:32 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:59:32.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:32 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:59:32.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:32 smithi183 ceph-mon[35580]: Adjusting osd_memory_target on smithi150 to 4272M 2024-03-28T15:59:32.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:32 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:59:32.352 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:32 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:59:32.352 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:32 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:59:32.352 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:32 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:59:32.352 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:32 smithi183 ceph-mon[35580]: osdmap e80: 11 total, 11 up, 11 in 2024-03-28T15:59:32.352 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:32 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:59:32.352 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:32 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:59:32.352 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:32 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:59:32.352 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:32 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:59:32.553 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:32 smithi150 ceph-mon[34825]: pgmap v305: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-28T15:59:32.553 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:32 smithi150 ceph-mon[34825]: Detected new or changed devices on smithi150 2024-03-28T15:59:32.553 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:32 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:59:32.553 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:32 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:59:32.553 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:32 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:59:32.553 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:32 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:59:32.553 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:32 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:59:32.553 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:32 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:59:32.553 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:32 smithi150 ceph-mon[34825]: Adjusting osd_memory_target on smithi150 to 4272M 2024-03-28T15:59:32.553 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:32 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:59:32.553 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:32 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:59:32.553 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:32 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:59:32.554 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:32 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:59:32.554 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:32 smithi150 ceph-mon[34825]: osdmap e80: 11 total, 11 up, 11 in 2024-03-28T15:59:32.554 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:32 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:59:32.554 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:32 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:59:32.554 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:32 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:59:32.554 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:32 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:59:32.571 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:32 smithi067 ceph-mon[31707]: pgmap v305: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-28T15:59:32.571 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:32 smithi067 ceph-mon[31707]: Detected new or changed devices on smithi150 2024-03-28T15:59:32.571 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:32 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:59:32.571 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:32 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:59:32.572 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:32 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:59:32.572 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:32 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:59:32.572 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:32 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:59:32.572 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:32 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:59:32.572 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:32 smithi067 ceph-mon[31707]: Adjusting osd_memory_target on smithi150 to 4272M 2024-03-28T15:59:32.572 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:32 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:59:32.572 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:32 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:59:32.572 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:32 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:59:32.572 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:32 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:59:32.572 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:32 smithi067 ceph-mon[31707]: osdmap e80: 11 total, 11 up, 11 in 2024-03-28T15:59:32.572 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:32 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:59:32.572 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:32 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:59:32.572 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:32 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:59:32.572 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:32 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:59:33.325 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:59:33.324+0000 7ff8ab000700 1 -- 172.21.15.183:0/747816657 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff8a4105fb0 msgr2=0x7ff8a410c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:59:33.325 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:59:33.324+0000 7ff8ab000700 1 --2- 172.21.15.183:0/747816657 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff8a4105fb0 0x7ff8a410c860 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff89c00b0a0 tx=0x7ff89c00b3b0 comp rx=0 tx=0).stop 2024-03-28T15:59:33.325 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:59:33.324+0000 7ff8ab000700 1 -- 172.21.15.183:0/747816657 shutdown_connections 2024-03-28T15:59:33.325 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:59:33.324+0000 7ff8ab000700 1 --2- 172.21.15.183:0/747816657 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff8a4105fb0 0x7ff8a410c860 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:59:33.325 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:59:33.324+0000 7ff8ab000700 1 --2- 172.21.15.183:0/747816657 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8a4104410 0x7ff8a4104830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:59:33.325 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:59:33.324+0000 7ff8ab000700 1 --2- 172.21.15.183:0/747816657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8a4105610 0x7ff8a4105a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:59:33.325 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:59:33.324+0000 7ff8ab000700 1 -- 172.21.15.183:0/747816657 >> 172.21.15.183:0/747816657 conn(0x7ff8a40ff9b0 msgr2=0x7ff8a4101df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:59:33.325 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:59:33.324+0000 7ff8ab000700 1 -- 172.21.15.183:0/747816657 shutdown_connections 2024-03-28T15:59:33.325 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:59:33.324+0000 7ff8ab000700 1 -- 172.21.15.183:0/747816657 wait complete. 2024-03-28T15:59:33.328 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:59:33.325+0000 7ff8ab000700 1 Processor -- start 2024-03-28T15:59:33.328 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:59:33.325+0000 7ff8ab000700 1 -- start start 2024-03-28T15:59:33.328 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:59:33.326+0000 7ff8ab000700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8a4104410 0x7ff8a419a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:59:33.328 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:59:33.326+0000 7ff8ab000700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff8a4105610 0x7ff8a419aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:59:33.328 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:59:33.326+0000 7ff8ab000700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8a4105fb0 0x7ff8a41a1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:59:33.328 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:59:33.326+0000 7ff8ab000700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a410ff00 con 0x7ff8a4105fb0 2024-03-28T15:59:33.328 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:59:33.326+0000 7ff8ab000700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a410fd80 con 0x7ff8a4104410 2024-03-28T15:59:33.329 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:59:33.326+0000 7ff8ab000700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a4110080 con 0x7ff8a4105610 2024-03-28T15:59:33.329 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:59:33.326+0000 7ff8a3400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff8a4105610 0x7ff8a419aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:59:33.329 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:59:33.326+0000 7ff8a3400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff8a4105610 0x7ff8a419aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.183:59306/0 (socket says 172.21.15.183:59306) 2024-03-28T15:59:33.329 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:59:33.326+0000 7ff8a3400700 1 -- 172.21.15.183:0/3219602896 learned_addr learned my addr 172.21.15.183:0/3219602896 (peer_addr_for_me v2:172.21.15.183:0/0) 2024-03-28T15:59:33.329 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:59:33.326+0000 7ff8a3e00700 1 --2- 172.21.15.183:0/3219602896 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8a4104410 0x7ff8a419a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:59:33.329 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:59:33.326+0000 7ff8a9000700 1 --2- 172.21.15.183:0/3219602896 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8a4105fb0 0x7ff8a41a1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:59:33.329 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:59:33.327+0000 7ff8a3400700 1 -- 172.21.15.183:0/3219602896 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8a4104410 msgr2=0x7ff8a419a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:59:33.329 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:59:33.327+0000 7ff8a3400700 1 --2- 172.21.15.183:0/3219602896 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8a4104410 0x7ff8a419a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:59:33.329 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:59:33.327+0000 7ff8a3400700 1 -- 172.21.15.183:0/3219602896 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8a4105fb0 msgr2=0x7ff8a41a1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:59:33.329 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:59:33.327+0000 7ff8a3400700 1 --2- 172.21.15.183:0/3219602896 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8a4105fb0 0x7ff8a41a1aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:59:33.329 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:59:33.327+0000 7ff8a3400700 1 -- 172.21.15.183:0/3219602896 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8a41a2140 con 0x7ff8a4105610 2024-03-28T15:59:33.329 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:59:33.327+0000 7ff8a3400700 1 --2- 172.21.15.183:0/3219602896 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff8a4105610 0x7ff8a419aa20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff89000d980 tx=0x7ff89000dd40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:59:33.329 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:59:33.327+0000 7ff8a0c00700 1 -- 172.21.15.183:0/3219602896 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff890016070 con 0x7ff8a4105610 2024-03-28T15:59:33.329 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:59:33.327+0000 7ff8a0c00700 1 -- 172.21.15.183:0/3219602896 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff890010460 con 0x7ff8a4105610 2024-03-28T15:59:33.330 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:59:33.327+0000 7ff8ab000700 1 -- 172.21.15.183:0/3219602896 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8a41a2430 con 0x7ff8a4105610 2024-03-28T15:59:33.330 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:59:33.327+0000 7ff8a0c00700 1 -- 172.21.15.183:0/3219602896 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff89000f620 con 0x7ff8a4105610 2024-03-28T15:59:33.330 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:59:33.328+0000 7ff8ab000700 1 -- 172.21.15.183:0/3219602896 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8a41a29c0 con 0x7ff8a4105610 2024-03-28T15:59:33.330 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:59:33.329+0000 7ff8ab000700 1 -- 172.21.15.183:0/3219602896 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8a4068e50 con 0x7ff8a4105610 2024-03-28T15:59:33.330 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:59:33.331+0000 7ff8a0c00700 1 -- 172.21.15.183:0/3219602896 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ff890019070 con 0x7ff8a4105610 2024-03-28T15:59:33.332 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:59:33.332+0000 7ff8a0c00700 1 --2- 172.21.15.183:0/3219602896 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7ff8880a2910 0x7ff8880a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:59:33.332 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:59:33.332+0000 7ff8a0c00700 1 -- 172.21.15.183:0/3219602896 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6694+0+0 (secure 0 0 0) 0x7ff8900cdfe0 con 0x7ff8a4105610 2024-03-28T15:59:33.332 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:59:33.333+0000 7ff8a3e00700 1 --2- 172.21.15.183:0/3219602896 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7ff8880a2910 0x7ff8880a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:59:33.333 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:59:33.334+0000 7ff8a3e00700 1 --2- 172.21.15.183:0/3219602896 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7ff8880a2910 0x7ff8880a4dd0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ff894005950 tx=0x7ff8940058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:59:33.337 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:59:33.337+0000 7ff8a0c00700 1 -- 172.21.15.183:0/3219602896 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff89009b630 con 0x7ff8a4105610 2024-03-28T15:59:33.572 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T15:59:33.571+0000 7ff8ab000700 1 -- 172.21.15.183:0/3219602896 --> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7ff8a40625d0 con 0x7ff8880a2910 2024-03-28T15:59:33.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:33 smithi183 ceph-mon[35580]: osdmap e81: 11 total, 11 up, 11 in 2024-03-28T15:59:34.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:33 smithi150 ceph-mon[34825]: osdmap e81: 11 total, 11 up, 11 in 2024-03-28T15:59:34.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:33 smithi067 ceph-mon[31707]: osdmap e81: 11 total, 11 up, 11 in 2024-03-28T15:59:34.840 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:34 smithi183 ceph-mon[35580]: pgmap v308: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-28T15:59:34.841 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:34 smithi183 ceph-mon[35580]: from='client.24376 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:59:34.841 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:34 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:59:34.841 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:34 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:59:34.841 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:34 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:59:35.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:34 smithi150 ceph-mon[34825]: pgmap v308: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-28T15:59:35.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:34 smithi150 ceph-mon[34825]: from='client.24376 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:59:35.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:34 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:59:35.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:34 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:59:35.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:34 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:59:35.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:34 smithi067 ceph-mon[31707]: pgmap v308: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-28T15:59:35.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:34 smithi067 ceph-mon[31707]: from='client.24376 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:59:35.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:34 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:59:35.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:34 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:59:35.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:34 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:59:36.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:36 smithi183 ceph-mon[35580]: pgmap v309: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-03-28T15:59:36.885 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:36 smithi067 ceph-mon[31707]: pgmap v309: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-03-28T15:59:37.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:36 smithi150 ceph-mon[34825]: pgmap v309: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-03-28T15:59:37.900 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:37 smithi067 ceph-mon[31707]: from='client.? 172.21.15.183:0/392978831' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a3c29c47-a228-4f42-a23c-fbdb0c093699"}]: dispatch 2024-03-28T15:59:37.900 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:37 smithi067 ceph-mon[31707]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a3c29c47-a228-4f42-a23c-fbdb0c093699"}]: dispatch 2024-03-28T15:59:37.900 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:37 smithi067 ceph-mon[31707]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a3c29c47-a228-4f42-a23c-fbdb0c093699"}]': finished 2024-03-28T15:59:37.900 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:37 smithi067 ceph-mon[31707]: osdmap e82: 12 total, 11 up, 12 in 2024-03-28T15:59:37.901 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:37 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T15:59:38.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:37 smithi150 ceph-mon[34825]: from='client.? 172.21.15.183:0/392978831' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a3c29c47-a228-4f42-a23c-fbdb0c093699"}]: dispatch 2024-03-28T15:59:38.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:37 smithi150 ceph-mon[34825]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a3c29c47-a228-4f42-a23c-fbdb0c093699"}]: dispatch 2024-03-28T15:59:38.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:37 smithi150 ceph-mon[34825]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a3c29c47-a228-4f42-a23c-fbdb0c093699"}]': finished 2024-03-28T15:59:38.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:37 smithi150 ceph-mon[34825]: osdmap e82: 12 total, 11 up, 12 in 2024-03-28T15:59:38.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:37 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T15:59:38.090 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:37 smithi183 ceph-mon[35580]: from='client.? 172.21.15.183:0/392978831' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a3c29c47-a228-4f42-a23c-fbdb0c093699"}]: dispatch 2024-03-28T15:59:38.091 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:37 smithi183 ceph-mon[35580]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a3c29c47-a228-4f42-a23c-fbdb0c093699"}]: dispatch 2024-03-28T15:59:38.091 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:37 smithi183 ceph-mon[35580]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a3c29c47-a228-4f42-a23c-fbdb0c093699"}]': finished 2024-03-28T15:59:38.091 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:37 smithi183 ceph-mon[35580]: osdmap e82: 12 total, 11 up, 12 in 2024-03-28T15:59:38.091 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:37 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T15:59:39.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:38 smithi150 ceph-mon[34825]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 983 GiB / 983 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-03-28T15:59:39.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:38 smithi150 ceph-mon[34825]: from='client.? 172.21.15.183:0/3805203518' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:59:39.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:38 smithi067 ceph-mon[31707]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 983 GiB / 983 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-03-28T15:59:39.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:38 smithi067 ceph-mon[31707]: from='client.? 172.21.15.183:0/3805203518' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:59:39.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:38 smithi183 ceph-mon[35580]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 983 GiB / 983 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-03-28T15:59:39.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:38 smithi183 ceph-mon[35580]: from='client.? 172.21.15.183:0/3805203518' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:59:41.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:40 smithi150 ceph-mon[34825]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 983 GiB / 983 GiB avail 2024-03-28T15:59:41.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:40 smithi067 ceph-mon[31707]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 983 GiB / 983 GiB avail 2024-03-28T15:59:41.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:40 smithi183 ceph-mon[35580]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 983 GiB / 983 GiB avail 2024-03-28T15:59:42.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:42 smithi183 ceph-mon[35580]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 983 GiB / 983 GiB avail 2024-03-28T15:59:42.966 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:42 smithi150 ceph-mon[34825]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 983 GiB / 983 GiB avail 2024-03-28T15:59:43.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:42 smithi067 ceph-mon[31707]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 983 GiB / 983 GiB avail 2024-03-28T15:59:45.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:44 smithi150 ceph-mon[34825]: pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 983 GiB / 983 GiB avail 2024-03-28T15:59:45.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:44 smithi067 ceph-mon[31707]: pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 983 GiB / 983 GiB avail 2024-03-28T15:59:45.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:44 smithi183 ceph-mon[35580]: pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 983 GiB / 983 GiB avail 2024-03-28T15:59:46.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:46 smithi183 ceph-mon[35580]: pgmap v315: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 983 GiB / 983 GiB avail 2024-03-28T15:59:47.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:46 smithi150 ceph-mon[34825]: pgmap v315: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 983 GiB / 983 GiB avail 2024-03-28T15:59:47.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:46 smithi067 ceph-mon[31707]: pgmap v315: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 983 GiB / 983 GiB avail 2024-03-28T15:59:48.041 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:47 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-28T15:59:48.041 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:47 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:59:48.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:47 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-28T15:59:48.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:47 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:59:48.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:47 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-28T15:59:48.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:47 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:59:48.970 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:48 smithi067 ceph-mon[31707]: pgmap v316: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 983 GiB / 983 GiB avail 2024-03-28T15:59:48.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:48 smithi067 ceph-mon[31707]: Deploying daemon osd.11 on smithi183 2024-03-28T15:59:49.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:48 smithi150 ceph-mon[34825]: pgmap v316: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 983 GiB / 983 GiB avail 2024-03-28T15:59:49.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:48 smithi150 ceph-mon[34825]: Deploying daemon osd.11 on smithi183 2024-03-28T15:59:49.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:48 smithi183 ceph-mon[35580]: pgmap v316: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 983 GiB / 983 GiB avail 2024-03-28T15:59:49.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:48 smithi183 ceph-mon[35580]: Deploying daemon osd.11 on smithi183 2024-03-28T15:59:51.022 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:50 smithi183 ceph-mon[35580]: pgmap v317: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 983 GiB / 983 GiB avail 2024-03-28T15:59:51.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:50 smithi150 ceph-mon[34825]: pgmap v317: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 983 GiB / 983 GiB avail 2024-03-28T15:59:51.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:50 smithi067 ceph-mon[31707]: pgmap v317: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 983 GiB / 983 GiB avail 2024-03-28T15:59:52.781 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:52 smithi183 ceph-mon[35580]: pgmap v318: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 983 GiB / 983 GiB avail 2024-03-28T15:59:53.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:52 smithi150 ceph-mon[34825]: pgmap v318: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 983 GiB / 983 GiB avail 2024-03-28T15:59:53.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:52 smithi067 ceph-mon[31707]: pgmap v318: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 983 GiB / 983 GiB avail 2024-03-28T15:59:54.985 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:54 smithi183 ceph-mon[35580]: pgmap v319: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 983 GiB / 983 GiB avail 2024-03-28T15:59:55.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:54 smithi150 ceph-mon[34825]: pgmap v319: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 983 GiB / 983 GiB avail 2024-03-28T15:59:55.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:54 smithi067 ceph-mon[31707]: pgmap v319: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 983 GiB / 983 GiB avail 2024-03-28T15:59:56.534 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:56 smithi183 ceph-mon[35580]: pgmap v320: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 983 GiB / 983 GiB avail 2024-03-28T15:59:56.534 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:56 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:59:56.534 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:56 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:59:56.534 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:56 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:59:56.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:56 smithi067 ceph-mon[31707]: pgmap v320: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 983 GiB / 983 GiB avail 2024-03-28T15:59:56.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:56 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:59:56.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:56 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:59:56.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:56 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:59:56.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:56 smithi150 ceph-mon[34825]: pgmap v320: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 983 GiB / 983 GiB avail 2024-03-28T15:59:56.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:56 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:59:56.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:56 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:59:56.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:56 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:59:58.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:58 smithi067 ceph-mon[31707]: pgmap v321: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 983 GiB / 983 GiB avail 2024-03-28T15:59:58.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:58 smithi183 ceph-mon[35580]: pgmap v321: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 983 GiB / 983 GiB avail 2024-03-28T15:59:58.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:58 smithi150 ceph-mon[34825]: pgmap v321: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 983 GiB / 983 GiB avail 2024-03-28T15:59:59.528 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:59 smithi183 ceph-mon[35580]: from='osd.11 [v2:172.21.15.183:6824/1892639849,v1:172.21.15.183:6825/1892639849]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-28T15:59:59.528 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:59 smithi183 ceph-mon[35580]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-28T15:59:59.528 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:59 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:59:59.528 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:59 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:59:59.528 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:59 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:59:59.528 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 15:59:59 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:59:59.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:59 smithi067 ceph-mon[31707]: from='osd.11 [v2:172.21.15.183:6824/1892639849,v1:172.21.15.183:6825/1892639849]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-28T15:59:59.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:59 smithi067 ceph-mon[31707]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-28T15:59:59.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:59 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:59:59.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:59 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:59:59.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:59 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:59:59.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 15:59:59 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:59:59.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:59 smithi150 ceph-mon[34825]: from='osd.11 [v2:172.21.15.183:6824/1892639849,v1:172.21.15.183:6825/1892639849]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-28T15:59:59.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:59 smithi150 ceph-mon[34825]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-28T15:59:59.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:59 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:59:59.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:59 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T15:59:59.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:59 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:59:59.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 15:59:59 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T16:00:00.402 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T16:00:00.400+0000 7ff8a0c00700 1 -- 172.21.15.183:0/3219602896 <== mgr.14152 v2:172.21.15.67:6800/208726654 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7ff8a40625d0 con 0x7ff8880a2910 2024-03-28T16:00:00.402 INFO:teuthology.orchestra.run.smithi183.stdout:Created osd(s) 11 on host 'smithi183' 2024-03-28T16:00:00.407 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T16:00:00.404+0000 7ff8ab000700 1 -- 172.21.15.183:0/3219602896 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7ff8880a2910 msgr2=0x7ff8880a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:00.407 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T16:00:00.404+0000 7ff8ab000700 1 --2- 172.21.15.183:0/3219602896 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7ff8880a2910 0x7ff8880a4dd0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ff894005950 tx=0x7ff8940058e0 comp rx=0 tx=0).stop 2024-03-28T16:00:00.407 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T16:00:00.404+0000 7ff8ab000700 1 -- 172.21.15.183:0/3219602896 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff8a4105610 msgr2=0x7ff8a419aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:00.407 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T16:00:00.404+0000 7ff8ab000700 1 --2- 172.21.15.183:0/3219602896 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff8a4105610 0x7ff8a419aa20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff89000d980 tx=0x7ff89000dd40 comp rx=0 tx=0).stop 2024-03-28T16:00:00.407 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T16:00:00.405+0000 7ff8ab000700 1 -- 172.21.15.183:0/3219602896 shutdown_connections 2024-03-28T16:00:00.407 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T16:00:00.405+0000 7ff8ab000700 1 --2- 172.21.15.183:0/3219602896 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7ff8880a2910 0x7ff8880a4dd0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:00.407 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T16:00:00.405+0000 7ff8ab000700 1 --2- 172.21.15.183:0/3219602896 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8a4105fb0 0x7ff8a41a1aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:00.407 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T16:00:00.405+0000 7ff8ab000700 1 --2- 172.21.15.183:0/3219602896 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8a4104410 0x7ff8a419a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:00.407 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T16:00:00.405+0000 7ff8ab000700 1 --2- 172.21.15.183:0/3219602896 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff8a4105610 0x7ff8a419aa20 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:00.407 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T16:00:00.405+0000 7ff8ab000700 1 -- 172.21.15.183:0/3219602896 >> 172.21.15.183:0/3219602896 conn(0x7ff8a40ff9b0 msgr2=0x7ff8a4108910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:00:00.408 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T16:00:00.405+0000 7ff8ab000700 1 -- 172.21.15.183:0/3219602896 shutdown_connections 2024-03-28T16:00:00.408 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T16:00:00.406+0000 7ff8ab000700 1 -- 172.21.15.183:0/3219602896 wait complete. 2024-03-28T16:00:00.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:00 smithi183 ceph-mon[35580]: pgmap v322: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 983 GiB / 983 GiB avail 2024-03-28T16:00:00.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:00 smithi183 ceph-mon[35580]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-28T16:00:00.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:00 smithi183 ceph-mon[35580]: osdmap e83: 12 total, 11 up, 12 in 2024-03-28T16:00:00.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:00 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T16:00:00.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:00 smithi183 ceph-mon[35580]: from='osd.11 [v2:172.21.15.183:6824/1892639849,v1:172.21.15.183:6825/1892639849]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-03-28T16:00:00.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:00 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:00.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:00 smithi183 ceph-mon[35580]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-03-28T16:00:00.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:00 smithi183 ceph-mon[35580]: overall HEALTH_OK 2024-03-28T16:00:00.705 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:00 smithi150 ceph-mon[34825]: pgmap v322: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 983 GiB / 983 GiB avail 2024-03-28T16:00:00.705 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:00 smithi150 ceph-mon[34825]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-28T16:00:00.705 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:00 smithi150 ceph-mon[34825]: osdmap e83: 12 total, 11 up, 12 in 2024-03-28T16:00:00.705 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:00 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T16:00:00.705 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:00 smithi150 ceph-mon[34825]: from='osd.11 [v2:172.21.15.183:6824/1892639849,v1:172.21.15.183:6825/1892639849]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-03-28T16:00:00.705 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:00 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:00.705 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:00 smithi150 ceph-mon[34825]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-03-28T16:00:00.706 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:00 smithi150 ceph-mon[34825]: overall HEALTH_OK 2024-03-28T16:00:00.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:00 smithi067 ceph-mon[31707]: pgmap v322: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 983 GiB / 983 GiB avail 2024-03-28T16:00:00.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:00 smithi067 ceph-mon[31707]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-28T16:00:00.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:00 smithi067 ceph-mon[31707]: osdmap e83: 12 total, 11 up, 12 in 2024-03-28T16:00:00.821 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:00 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T16:00:00.821 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:00 smithi067 ceph-mon[31707]: from='osd.11 [v2:172.21.15.183:6824/1892639849,v1:172.21.15.183:6825/1892639849]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-03-28T16:00:00.821 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:00 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:00.821 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:00 smithi067 ceph-mon[31707]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-03-28T16:00:00.821 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:00 smithi067 ceph-mon[31707]: overall HEALTH_OK 2024-03-28T16:00:01.285 DEBUG:teuthology.orchestra.run.smithi183:osd.11> sudo journalctl -f -n 0 -u ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@osd.11.service 2024-03-28T16:00:01.298 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-03-28T16:00:01.298 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph osd stat -f json 2024-03-28T16:00:01.484 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:01 smithi183 ceph-mon[35580]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-03-28T16:00:01.484 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:01 smithi183 ceph-mon[35580]: osdmap e84: 12 total, 11 up, 12 in 2024-03-28T16:00:01.484 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:01 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T16:00:01.484 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:01 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T16:00:01.484 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:01 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T16:00:01.484 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:01 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:01.485 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:01 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:01.785 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T16:00:01.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:01 smithi150 ceph-mon[34825]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-03-28T16:00:01.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:01 smithi150 ceph-mon[34825]: osdmap e84: 12 total, 11 up, 12 in 2024-03-28T16:00:01.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:01 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T16:00:01.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:01 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T16:00:01.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:01 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T16:00:01.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:01 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:01.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:01 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:01.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:01 smithi067 ceph-mon[31707]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-03-28T16:00:01.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:01 smithi067 ceph-mon[31707]: osdmap e84: 12 total, 11 up, 12 in 2024-03-28T16:00:01.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:01 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T16:00:01.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:01 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T16:00:01.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:01 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T16:00:01.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:01 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:01.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:01 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:02.557 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:02 smithi183 ceph-mon[35580]: purged_snaps scrub starts 2024-03-28T16:00:02.557 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:02 smithi183 ceph-mon[35580]: purged_snaps scrub ok 2024-03-28T16:00:02.557 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:02 smithi183 ceph-mon[35580]: pgmap v325: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 983 GiB / 983 GiB avail 2024-03-28T16:00:02.557 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:02 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T16:00:02.557 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:02 smithi183 ceph-mon[35580]: osd.11 [v2:172.21.15.183:6824/1892639849,v1:172.21.15.183:6825/1892639849] boot 2024-03-28T16:00:02.557 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:02 smithi183 ceph-mon[35580]: osdmap e85: 12 total, 12 up, 12 in 2024-03-28T16:00:02.558 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:02 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T16:00:02.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:02 smithi067 ceph-mon[31707]: purged_snaps scrub starts 2024-03-28T16:00:02.571 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:02 smithi067 ceph-mon[31707]: purged_snaps scrub ok 2024-03-28T16:00:02.571 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:02 smithi067 ceph-mon[31707]: pgmap v325: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 983 GiB / 983 GiB avail 2024-03-28T16:00:02.571 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:02 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T16:00:02.571 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:02 smithi067 ceph-mon[31707]: osd.11 [v2:172.21.15.183:6824/1892639849,v1:172.21.15.183:6825/1892639849] boot 2024-03-28T16:00:02.571 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:02 smithi067 ceph-mon[31707]: osdmap e85: 12 total, 12 up, 12 in 2024-03-28T16:00:02.571 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:02 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T16:00:02.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:02 smithi150 ceph-mon[34825]: purged_snaps scrub starts 2024-03-28T16:00:02.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:02 smithi150 ceph-mon[34825]: purged_snaps scrub ok 2024-03-28T16:00:02.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:02 smithi150 ceph-mon[34825]: pgmap v325: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 983 GiB / 983 GiB avail 2024-03-28T16:00:02.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:02 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T16:00:02.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:02 smithi150 ceph-mon[34825]: osd.11 [v2:172.21.15.183:6824/1892639849,v1:172.21.15.183:6825/1892639849] boot 2024-03-28T16:00:02.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:02 smithi150 ceph-mon[34825]: osdmap e85: 12 total, 12 up, 12 in 2024-03-28T16:00:02.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:02 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T16:00:03.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:03 smithi183 ceph-mon[35580]: osdmap e86: 12 total, 12 up, 12 in 2024-03-28T16:00:03.754 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:03 smithi067 ceph-mon[31707]: osdmap e86: 12 total, 12 up, 12 in 2024-03-28T16:00:03.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:03 smithi150 ceph-mon[34825]: osdmap e86: 12 total, 12 up, 12 in 2024-03-28T16:00:04.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:04.153+0000 7f8e75600700 1 -- 172.21.15.67:0/2852923698 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e70103d60 msgr2=0x7f8e7010a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:04.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:04.153+0000 7f8e75600700 1 --2- 172.21.15.67:0/2852923698 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e70103d60 0x7f8e7010a610 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f8e6800b0d0 tx=0x7f8e6800b3e0 comp rx=0 tx=0).stop 2024-03-28T16:00:04.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:04.153+0000 7f8e75600700 1 -- 172.21.15.67:0/2852923698 shutdown_connections 2024-03-28T16:00:04.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:04.153+0000 7f8e75600700 1 --2- 172.21.15.67:0/2852923698 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e70103d60 0x7f8e7010a610 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:04.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:04.153+0000 7f8e75600700 1 --2- 172.21.15.67:0/2852923698 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8e701021c0 0x7f8e701025e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:04.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:04.153+0000 7f8e75600700 1 --2- 172.21.15.67:0/2852923698 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e701033c0 0x7f8e70103820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:04.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:04.153+0000 7f8e75600700 1 -- 172.21.15.67:0/2852923698 >> 172.21.15.67:0/2852923698 conn(0x7f8e700fd7a0 msgr2=0x7f8e700ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:00:04.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:04.154+0000 7f8e75600700 1 -- 172.21.15.67:0/2852923698 shutdown_connections 2024-03-28T16:00:04.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:04.154+0000 7f8e75600700 1 -- 172.21.15.67:0/2852923698 wait complete. 2024-03-28T16:00:04.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:04.155+0000 7f8e75600700 1 Processor -- start 2024-03-28T16:00:04.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:04.155+0000 7f8e75600700 1 -- start start 2024-03-28T16:00:04.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:04.156+0000 7f8e75600700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e701021c0 0x7f8e70075f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:04.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:04.156+0000 7f8e75600700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e701033c0 0x7f8e70074580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:04.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:04.156+0000 7f8e75600700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8e70103d60 0x7f8e70074ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:04.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:04.156+0000 7f8e75600700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e7010dc50 con 0x7f8e701021c0 2024-03-28T16:00:04.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:04.156+0000 7f8e75600700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e7010dad0 con 0x7f8e701033c0 2024-03-28T16:00:04.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:04.156+0000 7f8e75600700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e7010ddd0 con 0x7f8e70103d60 2024-03-28T16:00:04.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:04.156+0000 7f8e6ea00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e701021c0 0x7f8e70075f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:04.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:04.156+0000 7f8e6ea00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e701021c0 0x7f8e70075f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39858/0 (socket says 172.21.15.67:39858) 2024-03-28T16:00:04.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:04.156+0000 7f8e6ea00700 1 -- 172.21.15.67:0/3631514775 learned_addr learned my addr 172.21.15.67:0/3631514775 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:00:04.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:04.156+0000 7f8e6f400700 1 --2- 172.21.15.67:0/3631514775 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8e70103d60 0x7f8e70074ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:04.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:04.156+0000 7f8e6e000700 1 --2- 172.21.15.67:0/3631514775 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e701033c0 0x7f8e70074580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:04.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:04.157+0000 7f8e6ea00700 1 -- 172.21.15.67:0/3631514775 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8e70103d60 msgr2=0x7f8e70074ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:04.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:04.157+0000 7f8e6ea00700 1 --2- 172.21.15.67:0/3631514775 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8e70103d60 0x7f8e70074ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:04.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:04.157+0000 7f8e6ea00700 1 -- 172.21.15.67:0/3631514775 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e701033c0 msgr2=0x7f8e70074580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:04.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:04.157+0000 7f8e6ea00700 1 --2- 172.21.15.67:0/3631514775 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e701033c0 0x7f8e70074580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:04.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:04.157+0000 7f8e6ea00700 1 -- 172.21.15.67:0/3631514775 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e70075370 con 0x7f8e701021c0 2024-03-28T16:00:04.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:04.157+0000 7f8e6e000700 1 --2- 172.21.15.67:0/3631514775 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e701033c0 0x7f8e70074580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T16:00:04.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:04.157+0000 7f8e6ea00700 1 --2- 172.21.15.67:0/3631514775 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e701021c0 0x7f8e70075f30 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f8e6000d980 tx=0x7f8e6000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:00:04.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:04.157+0000 7f8e67400700 1 -- 172.21.15.67:0/3631514775 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e60017030 con 0x7f8e701021c0 2024-03-28T16:00:04.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:04.157+0000 7f8e67400700 1 -- 172.21.15.67:0/3631514775 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8e600044d0 con 0x7f8e701021c0 2024-03-28T16:00:04.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:04.157+0000 7f8e67400700 1 -- 172.21.15.67:0/3631514775 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e600039e0 con 0x7f8e701021c0 2024-03-28T16:00:04.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:04.157+0000 7f8e75600700 1 -- 172.21.15.67:0/3631514775 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e701a2430 con 0x7f8e701021c0 2024-03-28T16:00:04.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:04.158+0000 7f8e75600700 1 -- 172.21.15.67:0/3631514775 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e701a2970 con 0x7f8e701021c0 2024-03-28T16:00:04.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:04.159+0000 7f8e75600700 1 -- 172.21.15.67:0/3631514775 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e70068e50 con 0x7f8e701021c0 2024-03-28T16:00:04.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:04.161+0000 7f8e67400700 1 -- 172.21.15.67:0/3631514775 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f8e6001a070 con 0x7f8e701021c0 2024-03-28T16:00:04.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:04.162+0000 7f8e67400700 1 --2- 172.21.15.67:0/3631514775 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f8e540a2910 0x7f8e540a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:04.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:04.162+0000 7f8e67400700 1 -- 172.21.15.67:0/3631514775 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f8e600cdf10 con 0x7f8e701021c0 2024-03-28T16:00:04.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:04.163+0000 7f8e6e000700 1 --2- 172.21.15.67:0/3631514775 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f8e540a2910 0x7f8e540a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:04.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:04.163+0000 7f8e6e000700 1 --2- 172.21.15.67:0/3631514775 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f8e540a2910 0x7f8e540a4dd0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c005950 tx=0x7f8e5c0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:00:04.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:04.167+0000 7f8e67400700 1 -- 172.21.15.67:0/3631514775 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8e600232f0 con 0x7f8e701021c0 2024-03-28T16:00:04.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:04.391+0000 7f8e75600700 1 -- 172.21.15.67:0/3631514775 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f8e70062da0 con 0x7f8e701021c0 2024-03-28T16:00:04.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:04.392+0000 7f8e67400700 1 -- 172.21.15.67:0/3631514775 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v86) v1 ==== 74+0+133 (secure 0 0 0) 0x7f8e6009b3b0 con 0x7f8e701021c0 2024-03-28T16:00:04.393 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:00:04.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:04.395+0000 7f8e75600700 1 -- 172.21.15.67:0/3631514775 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f8e540a2910 msgr2=0x7f8e540a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:04.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:04.395+0000 7f8e75600700 1 --2- 172.21.15.67:0/3631514775 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f8e540a2910 0x7f8e540a4dd0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c005950 tx=0x7f8e5c0058e0 comp rx=0 tx=0).stop 2024-03-28T16:00:04.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:04.395+0000 7f8e75600700 1 -- 172.21.15.67:0/3631514775 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e701021c0 msgr2=0x7f8e70075f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:04.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:04.395+0000 7f8e75600700 1 --2- 172.21.15.67:0/3631514775 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e701021c0 0x7f8e70075f30 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f8e6000d980 tx=0x7f8e6000dc90 comp rx=0 tx=0).stop 2024-03-28T16:00:04.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:04.395+0000 7f8e75600700 1 -- 172.21.15.67:0/3631514775 shutdown_connections 2024-03-28T16:00:04.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:04.395+0000 7f8e75600700 1 --2- 172.21.15.67:0/3631514775 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f8e540a2910 0x7f8e540a4dd0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:04.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:04.395+0000 7f8e75600700 1 --2- 172.21.15.67:0/3631514775 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8e70103d60 0x7f8e70074ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:04.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:04.395+0000 7f8e75600700 1 --2- 172.21.15.67:0/3631514775 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e701021c0 0x7f8e70075f30 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:04.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:04.395+0000 7f8e75600700 1 --2- 172.21.15.67:0/3631514775 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e701033c0 0x7f8e70074580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:04.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:04.395+0000 7f8e75600700 1 -- 172.21.15.67:0/3631514775 >> 172.21.15.67:0/3631514775 conn(0x7f8e700fd7a0 msgr2=0x7f8e70106740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:00:04.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:04.396+0000 7f8e75600700 1 -- 172.21.15.67:0/3631514775 shutdown_connections 2024-03-28T16:00:04.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:04.396+0000 7f8e75600700 1 -- 172.21.15.67:0/3631514775 wait complete. 2024-03-28T16:00:04.798 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:04 smithi067 ceph-mon[31707]: pgmap v328: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:04.798 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:04 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/3631514775' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-28T16:00:04.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:04 smithi150 ceph-mon[34825]: pgmap v328: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:04.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:04 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/3631514775' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-28T16:00:04.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:04 smithi183 ceph-mon[35580]: pgmap v328: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:04.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:04 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/3631514775' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-28T16:00:05.196 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":86,"num_osds":12,"num_up_osds":12,"osd_up_since":1711641601,"num_in_osds":12,"osd_in_since":1711641577,"num_remapped_pgs":0} 2024-03-28T16:00:05.197 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-28T16:00:05.669 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T16:00:07.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:06 smithi067 ceph-mon[31707]: pgmap v329: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:07.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:06 smithi067 ceph-mon[31707]: Detected new or changed devices on smithi183 2024-03-28T16:00:07.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:06 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:07.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:06 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:07.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:06 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:00:07.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:06 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:00:07.071 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:06 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:00:07.071 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:06 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:00:07.071 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:06 smithi067 ceph-mon[31707]: Adjusting osd_memory_target on smithi183 to 4272M 2024-03-28T16:00:07.071 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:06 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:07.071 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:06 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:00:07.071 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:06 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T16:00:07.071 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:06 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:07.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:06 smithi183 ceph-mon[35580]: pgmap v329: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:07.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:06 smithi183 ceph-mon[35580]: Detected new or changed devices on smithi183 2024-03-28T16:00:07.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:06 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:07.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:06 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:07.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:06 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:00:07.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:06 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:00:07.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:06 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:00:07.102 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:06 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:00:07.102 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:06 smithi183 ceph-mon[35580]: Adjusting osd_memory_target on smithi183 to 4272M 2024-03-28T16:00:07.102 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:06 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:07.102 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:06 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:00:07.102 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:06 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T16:00:07.102 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:06 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:07.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:06 smithi150 ceph-mon[34825]: pgmap v329: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:07.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:06 smithi150 ceph-mon[34825]: Detected new or changed devices on smithi183 2024-03-28T16:00:07.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:06 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:07.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:06 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:07.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:06 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:00:07.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:06 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:00:07.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:06 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:00:07.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:06 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:00:07.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:06 smithi150 ceph-mon[34825]: Adjusting osd_memory_target on smithi183 to 4272M 2024-03-28T16:00:07.304 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:06 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:07.304 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:06 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:00:07.304 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:06 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T16:00:07.304 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:06 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:07.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:07.773+0000 7f4519c00700 1 -- 172.21.15.67:0/2660400866 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f451406a760 msgr2=0x7f451410e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:07.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:07.773+0000 7f4519c00700 1 --2- 172.21.15.67:0/2660400866 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f451406a760 0x7f451410e700 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f44fc009990 tx=0x7f44fc009ca0 comp rx=0 tx=0).stop 2024-03-28T16:00:07.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:07.774+0000 7f4519c00700 1 -- 172.21.15.67:0/2660400866 shutdown_connections 2024-03-28T16:00:07.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:07.774+0000 7f4519c00700 1 --2- 172.21.15.67:0/2660400866 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f451410ec40 0x7f4514111030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:07.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:07.774+0000 7f4519c00700 1 --2- 172.21.15.67:0/2660400866 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4514069e40 0x7f451406a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:07.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:07.774+0000 7f4519c00700 1 --2- 172.21.15.67:0/2660400866 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f451406a760 0x7f451410e700 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:07.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:07.774+0000 7f4519c00700 1 -- 172.21.15.67:0/2660400866 >> 172.21.15.67:0/2660400866 conn(0x7f4514077fc0 msgr2=0x7f45140783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:00:07.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:07.774+0000 7f4519c00700 1 -- 172.21.15.67:0/2660400866 shutdown_connections 2024-03-28T16:00:07.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:07.774+0000 7f4519c00700 1 -- 172.21.15.67:0/2660400866 wait complete. 2024-03-28T16:00:07.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:07.775+0000 7f4519c00700 1 Processor -- start 2024-03-28T16:00:07.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:07.775+0000 7f4519c00700 1 -- start start 2024-03-28T16:00:07.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:07.776+0000 7f4519c00700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4514069e40 0x7f45141a0760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:07.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:07.776+0000 7f4519c00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f451406a760 0x7f45141a0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:07.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:07.776+0000 7f4519c00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f451410ec40 0x7f451419a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:07.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:07.776+0000 7f4519c00700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45141145d0 con 0x7f451410ec40 2024-03-28T16:00:07.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:07.776+0000 7f4519c00700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4514114450 con 0x7f4514069e40 2024-03-28T16:00:07.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:07.776+0000 7f4519c00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4514114750 con 0x7f451406a760 2024-03-28T16:00:07.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:07.776+0000 7f4513400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f451410ec40 0x7f451419a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:07.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:07.776+0000 7f4513400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f451410ec40 0x7f451419a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39868/0 (socket says 172.21.15.67:39868) 2024-03-28T16:00:07.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:07.776+0000 7f4513400700 1 -- 172.21.15.67:0/2025938028 learned_addr learned my addr 172.21.15.67:0/2025938028 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:00:07.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:07.776+0000 7f450a000700 1 --2- 172.21.15.67:0/2025938028 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f451406a760 0x7f45141a0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:07.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:07.776+0000 7f4512a00700 1 --2- 172.21.15.67:0/2025938028 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4514069e40 0x7f45141a0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:07.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:07.776+0000 7f4513400700 1 -- 172.21.15.67:0/2025938028 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f451406a760 msgr2=0x7f45141a0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:07.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:07.776+0000 7f4513400700 1 --2- 172.21.15.67:0/2025938028 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f451406a760 0x7f45141a0ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:07.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:07.776+0000 7f4513400700 1 -- 172.21.15.67:0/2025938028 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4514069e40 msgr2=0x7f45141a0760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:07.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:07.777+0000 7f4513400700 1 --2- 172.21.15.67:0/2025938028 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4514069e40 0x7f45141a0760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:07.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:07.777+0000 7f4513400700 1 -- 172.21.15.67:0/2025938028 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f451419b0d0 con 0x7f451410ec40 2024-03-28T16:00:07.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:07.777+0000 7f4513400700 1 --2- 172.21.15.67:0/2025938028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f451410ec40 0x7f451419a970 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f450c00b480 tx=0x7f450c00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:00:07.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:07.777+0000 7f450be00700 1 -- 172.21.15.67:0/2025938028 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f450c015020 con 0x7f451410ec40 2024-03-28T16:00:07.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:07.777+0000 7f450be00700 1 -- 172.21.15.67:0/2025938028 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f450c011ce0 con 0x7f451410ec40 2024-03-28T16:00:07.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:07.777+0000 7f450be00700 1 -- 172.21.15.67:0/2025938028 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f450c010e60 con 0x7f451410ec40 2024-03-28T16:00:07.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:07.777+0000 7f4519c00700 1 -- 172.21.15.67:0/2025938028 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f451419b3c0 con 0x7f451410ec40 2024-03-28T16:00:07.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:07.777+0000 7f4519c00700 1 -- 172.21.15.67:0/2025938028 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45141a7630 con 0x7f451410ec40 2024-03-28T16:00:07.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:07.779+0000 7f4519c00700 1 -- 172.21.15.67:0/2025938028 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4514068e50 con 0x7f451410ec40 2024-03-28T16:00:07.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:07.781+0000 7f450be00700 1 -- 172.21.15.67:0/2025938028 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f450c018020 con 0x7f451410ec40 2024-03-28T16:00:07.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:07.782+0000 7f450be00700 1 --2- 172.21.15.67:0/2025938028 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f44f80a2910 0x7f44f80a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:07.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:07.782+0000 7f450be00700 1 -- 172.21.15.67:0/2025938028 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f450c0cde90 con 0x7f451410ec40 2024-03-28T16:00:07.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:07.782+0000 7f4512a00700 1 --2- 172.21.15.67:0/2025938028 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f44f80a2910 0x7f44f80a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:07.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:07.783+0000 7f4512a00700 1 --2- 172.21.15.67:0/2025938028 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f44f80a2910 0x7f44f80a4dd0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f4514100db0 tx=0x7f4504009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:00:07.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:07.787+0000 7f450be00700 1 -- 172.21.15.67:0/2025938028 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f450c01f020 con 0x7f451410ec40 2024-03-28T16:00:08.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:08.006+0000 7f4519c00700 1 -- 172.21.15.67:0/2025938028 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f451402da30 con 0x7f451410ec40 2024-03-28T16:00:08.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:08.010+0000 7f450be00700 1 -- 172.21.15.67:0/2025938028 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+18458 (secure 0 0 0) 0x7f450c09b330 con 0x7f451410ec40 2024-03-28T16:00:08.011 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:00:08.011 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":86,"fsid":"b5f8225a-ed1a-11ee-95d0-87774f69a715","created":"2024-03-28T15:50:21.471924+0000","modified":"2024-03-28T16:00:02.385208+0000","last_up_change":"2024-03-28T16:00:01.369123+0000","last_in_change":"2024-03-28T15:59:37.515483+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T15:54:55.588007+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7c1791b6-9c0e-4512-a327-cad166bb5581","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6803","nonce":1894681701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6805","nonce":1894681701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6809","nonce":1894681701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6807","nonce":1894681701}]},"public_addr":"172.21.15.67:6803/1894681701","cluster_addr":"172.21.15.67:6805/1894681701","heartbeat_back_addr":"172.21.15.67:6809/1894681701","heartbeat_front_addr":"172.21.15.67:6807/1894681701","state":["exists","up"]},{"osd":1,"uuid":"0cdda7f3-d00c-4436-bcc1-49fb8072b2e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6801","nonce":1734465762}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6803","nonce":1734465762}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6807","nonce":1734465762}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6805","nonce":1734465762}]},"public_addr":"172.21.15.150:6801/1734465762","cluster_addr":"172.21.15.150:6803/1734465762","heartbeat_back_addr":"172.21.15.150:6807/1734465762","heartbeat_front_addr":"172.21.15.150:6805/1734465762","state":["exists","up"]},{"osd":2,"uuid":"a55b9a02-c3fd-4061-95c0-53a24a17ade0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6801","nonce":3398095931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6803","nonce":3398095931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6807","nonce":3398095931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6805","nonce":3398095931}]},"public_addr":"172.21.15.183:6801/3398095931","cluster_addr":"172.21.15.183:6803/3398095931","heartbeat_back_addr":"172.21.15.183:6807/3398095931","heartbeat_front_addr":"172.21.15.183:6805/3398095931","state":["exists","up"]},{"osd":3,"uuid":"d34940c0-5ce9-4b8b-aa36-4872414a043f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6811","nonce":1933541875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6813","nonce":1933541875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6817","nonce":1933541875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6815","nonce":1933541875}]},"public_addr":"172.21.15.67:6811/1933541875","cluster_addr":"172.21.15.67:6813/1933541875","heartbeat_back_addr":"172.21.15.67:6817/1933541875","heartbeat_front_addr":"172.21.15.67:6815/1933541875","state":["exists","up"]},{"osd":4,"uuid":"5cc3e260-83a3-4a4d-aa6a-34a3b12b2b5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6809","nonce":3775269993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6811","nonce":3775269993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6815","nonce":3775269993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6813","nonce":3775269993}]},"public_addr":"172.21.15.150:6809/3775269993","cluster_addr":"172.21.15.150:6811/3775269993","heartbeat_back_addr":"172.21.15.150:6815/3775269993","heartbeat_front_addr":"172.21.15.150:6813/3775269993","state":["exists","up"]},{"osd":5,"uuid":"a2542b21-d6ac-413e-8b9b-4a6548b20840","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6809","nonce":164284137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6811","nonce":164284137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6815","nonce":164284137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6813","nonce":164284137}]},"public_addr":"172.21.15.183:6809/164284137","cluster_addr":"172.21.15.183:6811/164284137","heartbeat_back_addr":"172.21.15.183:6815/164284137","heartbeat_front_addr":"172.21.15.183:6813/164284137","state":["exists","up"]},{"osd":6,"uuid":"8905a2d3-0f1b-4853-a218-22da2635150a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6819","nonce":3565528013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6821","nonce":3565528013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6825","nonce":3565528013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6823","nonce":3565528013}]},"public_addr":"172.21.15.67:6819/3565528013","cluster_addr":"172.21.15.67:6821/3565528013","heartbeat_back_addr":"172.21.15.67:6825/3565528013","heartbeat_front_addr":"172.21.15.67:6823/3565528013","state":["exists","up"]},{"osd":7,"uuid":"b275055b-b451-4180-91b1-e6e61a828c48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6817","nonce":3735115340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6819","nonce":3735115340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6823","nonce":3735115340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6821","nonce":3735115340}]},"public_addr":"172.21.15.150:6817/3735115340","cluster_addr":"172.21.15.150:6819/3735115340","heartbeat_back_addr":"172.21.15.150:6823/3735115340","heartbeat_front_addr":"172.21.15.150:6821/3735115340","state":["exists","up"]},{"osd":8,"uuid":"b6abf7ad-eb34-4f60-a4eb-348a92d7195b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6817","nonce":1025400095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6819","nonce":1025400095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6823","nonce":1025400095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6821","nonce":1025400095}]},"public_addr":"172.21.15.183:6817/1025400095","cluster_addr":"172.21.15.183:6819/1025400095","heartbeat_back_addr":"172.21.15.183:6823/1025400095","heartbeat_front_addr":"172.21.15.183:6821/1025400095","state":["exists","up"]},{"osd":9,"uuid":"28a78e3a-0657-40aa-816f-16a6fadadcbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6827","nonce":76885409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6829","nonce":76885409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6833","nonce":76885409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6831","nonce":76885409}]},"public_addr":"172.21.15.67:6827/76885409","cluster_addr":"172.21.15.67:6829/76885409","heartbeat_back_addr":"172.21.15.67:6833/76885409","heartbeat_front_addr":"172.21.15.67:6831/76885409","state":["exists","up"]},{"osd":10,"uuid":"d287907e-424f-4b49-a06e-4613c6de0474","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6825","nonce":3489982080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6827","nonce":3489982080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6831","nonce":3489982080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6829","nonce":3489982080}]},"public_addr":"172.21.15.150:6825/3489982080","cluster_addr":"172.21.15.150:6827/3489982080","heartbeat_back_addr":"172.21.15.150:6831/3489982080","heartbeat_front_addr":"172.21.15.150:6829/3489982080","state":["exists","up"]},{"osd":11,"uuid":"a3c29c47-a228-4f42-a23c-fbdb0c093699","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":85,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6825","nonce":1892639849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6826","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6827","nonce":1892639849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6831","nonce":1892639849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6829","nonce":1892639849}]},"public_addr":"172.21.15.183:6825/1892639849","cluster_addr":"172.21.15.183:6827/1892639849","heartbeat_back_addr":"172.21.15.183:6831/1892639849","heartbeat_front_addr":"172.21.15.183:6829/1892639849","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:53:45.646211+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:54:18.956590+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:54:52.161646+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:55:27.502822+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:56:01.162932+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:56:34.674452+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:57:09.298679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:57:43.230406+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:58:17.003821+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:58:51.085863+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:59:25.665065+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T16:00:00.263457+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6801/1631001968":"2024-03-29T15:51:38.875334+0000","172.21.15.67:6800/1631001968":"2024-03-29T15:51:38.875334+0000","172.21.15.67:0/1533056310":"2024-03-29T15:51:38.875334+0000","172.21.15.67:6801/3580491071":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/2030412458":"2024-03-29T15:51:03.859740+0000","172.21.15.67:6800/3580491071":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/2905307533":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/3763759988":"2024-03-29T15:51:38.875334+0000","172.21.15.67:0/4089399880":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/4239108661":"2024-03-29T15:51:38.875334+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T16:00:08.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:08.012+0000 7f4519c00700 1 -- 172.21.15.67:0/2025938028 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f44f80a2910 msgr2=0x7f44f80a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:08.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:08.012+0000 7f4519c00700 1 --2- 172.21.15.67:0/2025938028 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f44f80a2910 0x7f44f80a4dd0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f4514100db0 tx=0x7f4504009040 comp rx=0 tx=0).stop 2024-03-28T16:00:08.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:08.013+0000 7f4519c00700 1 -- 172.21.15.67:0/2025938028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f451410ec40 msgr2=0x7f451419a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:08.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:08.013+0000 7f4519c00700 1 --2- 172.21.15.67:0/2025938028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f451410ec40 0x7f451419a970 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f450c00b480 tx=0x7f450c00b840 comp rx=0 tx=0).stop 2024-03-28T16:00:08.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:08.013+0000 7f4519c00700 1 -- 172.21.15.67:0/2025938028 shutdown_connections 2024-03-28T16:00:08.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:08.013+0000 7f4519c00700 1 --2- 172.21.15.67:0/2025938028 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f44f80a2910 0x7f44f80a4dd0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:08.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:08.013+0000 7f4519c00700 1 --2- 172.21.15.67:0/2025938028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f451410ec40 0x7f451419a970 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:08.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:08.013+0000 7f4519c00700 1 --2- 172.21.15.67:0/2025938028 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4514069e40 0x7f45141a0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:08.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:08.013+0000 7f4519c00700 1 --2- 172.21.15.67:0/2025938028 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f451406a760 0x7f45141a0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:08.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:08.013+0000 7f4519c00700 1 -- 172.21.15.67:0/2025938028 >> 172.21.15.67:0/2025938028 conn(0x7f4514077fc0 msgr2=0x7f4514110130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:00:08.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:08.013+0000 7f4519c00700 1 -- 172.21.15.67:0/2025938028 shutdown_connections 2024-03-28T16:00:08.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:08.013+0000 7f4519c00700 1 -- 172.21.15.67:0/2025938028 wait complete. 2024-03-28T16:00:08.768 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T15:54:55.588007+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T16:00:08.769 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-28T16:00:09.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:08 smithi067 ceph-mon[31707]: pgmap v330: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:09.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:08 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/2025938028' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:00:09.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:08 smithi183 ceph-mon[35580]: pgmap v330: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:09.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:08 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/2025938028' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:00:09.247 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T16:00:09.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:08 smithi150 ceph-mon[34825]: pgmap v330: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:09.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:08 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/2025938028' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:00:10.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:10 smithi067 ceph-mon[31707]: pgmap v331: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:11.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:10 smithi183 ceph-mon[35580]: pgmap v331: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:11.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:10 smithi150 ceph-mon[34825]: pgmap v331: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:11.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:11.379+0000 7f9b2fe00700 1 -- 172.21.15.67:0/426134401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b300744b0 msgr2=0x7f9b30074890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:11.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:11.379+0000 7f9b2fe00700 1 --2- 172.21.15.67:0/426134401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b300744b0 0x7f9b30074890 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9b20009990 tx=0x7f9b20009ca0 comp rx=0 tx=0).stop 2024-03-28T16:00:11.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:11.379+0000 7f9b2fe00700 1 -- 172.21.15.67:0/426134401 shutdown_connections 2024-03-28T16:00:11.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:11.379+0000 7f9b2fe00700 1 --2- 172.21.15.67:0/426134401 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b3010e940 0x7f9b30110d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:11.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:11.379+0000 7f9b2fe00700 1 --2- 172.21.15.67:0/426134401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b300744b0 0x7f9b30074890 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:11.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:11.379+0000 7f9b2fe00700 1 --2- 172.21.15.67:0/426134401 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9b30074dd0 0x7f9b3010e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:11.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:11.379+0000 7f9b2fe00700 1 -- 172.21.15.67:0/426134401 >> 172.21.15.67:0/426134401 conn(0x7f9b300fdb90 msgr2=0x7f9b300fffb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:00:11.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:11.380+0000 7f9b2fe00700 1 -- 172.21.15.67:0/426134401 shutdown_connections 2024-03-28T16:00:11.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:11.380+0000 7f9b2fe00700 1 -- 172.21.15.67:0/426134401 wait complete. 2024-03-28T16:00:11.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:11.381+0000 7f9b2fe00700 1 Processor -- start 2024-03-28T16:00:11.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:11.381+0000 7f9b2fe00700 1 -- start start 2024-03-28T16:00:11.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:11.381+0000 7f9b2fe00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9b300744b0 0x7f9b301a06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:11.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:11.381+0000 7f9b2fe00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b30074dd0 0x7f9b301a0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:11.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:11.381+0000 7f9b2fe00700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b3010e940 0x7f9b3019a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:11.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:11.381+0000 7f9b2fe00700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b30114500 con 0x7f9b30074dd0 2024-03-28T16:00:11.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:11.381+0000 7f9b2fe00700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b30114380 con 0x7f9b3010e940 2024-03-28T16:00:11.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:11.381+0000 7f9b2fe00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b30114680 con 0x7f9b300744b0 2024-03-28T16:00:11.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:11.382+0000 7f9b2e000700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b30074dd0 0x7f9b301a0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:11.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:11.382+0000 7f9b2e000700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b30074dd0 0x7f9b301a0be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48564/0 (socket says 172.21.15.67:48564) 2024-03-28T16:00:11.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:11.382+0000 7f9b2e000700 1 -- 172.21.15.67:0/1310796475 learned_addr learned my addr 172.21.15.67:0/1310796475 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:00:11.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:11.382+0000 7f9b2ea00700 1 --2- 172.21.15.67:0/1310796475 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9b300744b0 0x7f9b301a06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:11.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:11.382+0000 7f9b2f400700 1 --2- 172.21.15.67:0/1310796475 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b3010e940 0x7f9b3019a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:11.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:11.382+0000 7f9b2e000700 1 -- 172.21.15.67:0/1310796475 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9b300744b0 msgr2=0x7f9b301a06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:11.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:11.382+0000 7f9b2e000700 1 --2- 172.21.15.67:0/1310796475 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9b300744b0 0x7f9b301a06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:11.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:11.382+0000 7f9b2e000700 1 -- 172.21.15.67:0/1310796475 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b3010e940 msgr2=0x7f9b3019a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:11.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:11.382+0000 7f9b2e000700 1 --2- 172.21.15.67:0/1310796475 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b3010e940 0x7f9b3019a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:11.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:11.382+0000 7f9b2e000700 1 -- 172.21.15.67:0/1310796475 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b3019afc0 con 0x7f9b30074dd0 2024-03-28T16:00:11.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:11.383+0000 7f9b2e000700 1 --2- 172.21.15.67:0/1310796475 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b30074dd0 0x7f9b301a0be0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9b1c00b530 tx=0x7f9b1c00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:00:11.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:11.383+0000 7f9b27400700 1 -- 172.21.15.67:0/1310796475 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b1c015020 con 0x7f9b30074dd0 2024-03-28T16:00:11.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:11.383+0000 7f9b27400700 1 -- 172.21.15.67:0/1310796475 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9b1c011ce0 con 0x7f9b30074dd0 2024-03-28T16:00:11.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:11.383+0000 7f9b2fe00700 1 -- 172.21.15.67:0/1310796475 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b3019b2b0 con 0x7f9b30074dd0 2024-03-28T16:00:11.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:11.383+0000 7f9b27400700 1 -- 172.21.15.67:0/1310796475 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b1c010e40 con 0x7f9b30074dd0 2024-03-28T16:00:11.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:11.383+0000 7f9b2fe00700 1 -- 172.21.15.67:0/1310796475 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b301a7570 con 0x7f9b30074dd0 2024-03-28T16:00:11.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:11.384+0000 7f9b2fe00700 1 -- 172.21.15.67:0/1310796475 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b30068e50 con 0x7f9b30074dd0 2024-03-28T16:00:11.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:11.386+0000 7f9b27400700 1 -- 172.21.15.67:0/1310796475 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f9b1c018020 con 0x7f9b30074dd0 2024-03-28T16:00:11.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:11.388+0000 7f9b27400700 1 --2- 172.21.15.67:0/1310796475 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f9b140a2910 0x7f9b140a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:11.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:11.388+0000 7f9b27400700 1 -- 172.21.15.67:0/1310796475 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f9b1c0cdee0 con 0x7f9b30074dd0 2024-03-28T16:00:11.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:11.388+0000 7f9b2ea00700 1 --2- 172.21.15.67:0/1310796475 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f9b140a2910 0x7f9b140a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:11.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:11.389+0000 7f9b2ea00700 1 --2- 172.21.15.67:0/1310796475 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f9b140a2910 0x7f9b140a4dd0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f9b3019b9a0 tx=0x7f9b2000b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:00:11.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:11.393+0000 7f9b27400700 1 -- 172.21.15.67:0/1310796475 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9b1c01f020 con 0x7f9b30074dd0 2024-03-28T16:00:11.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:11.619+0000 7f9b2fe00700 1 -- 172.21.15.67:0/1310796475 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9b3019bd90 con 0x7f9b30074dd0 2024-03-28T16:00:11.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:11.620+0000 7f9b27400700 1 -- 172.21.15.67:0/1310796475 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v86) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9b1c010380 con 0x7f9b30074dd0 2024-03-28T16:00:11.622 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-03-28T16:00:11.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:11.623+0000 7f9b2fe00700 1 -- 172.21.15.67:0/1310796475 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f9b140a2910 msgr2=0x7f9b140a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:11.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:11.623+0000 7f9b2fe00700 1 --2- 172.21.15.67:0/1310796475 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f9b140a2910 0x7f9b140a4dd0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f9b3019b9a0 tx=0x7f9b2000b540 comp rx=0 tx=0).stop 2024-03-28T16:00:11.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:11.623+0000 7f9b2fe00700 1 -- 172.21.15.67:0/1310796475 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b30074dd0 msgr2=0x7f9b301a0be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:11.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:11.623+0000 7f9b2fe00700 1 --2- 172.21.15.67:0/1310796475 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b30074dd0 0x7f9b301a0be0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9b1c00b530 tx=0x7f9b1c00b840 comp rx=0 tx=0).stop 2024-03-28T16:00:11.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:11.623+0000 7f9b2fe00700 1 -- 172.21.15.67:0/1310796475 shutdown_connections 2024-03-28T16:00:11.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:11.623+0000 7f9b2fe00700 1 --2- 172.21.15.67:0/1310796475 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f9b140a2910 0x7f9b140a4dd0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:11.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:11.623+0000 7f9b2fe00700 1 --2- 172.21.15.67:0/1310796475 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b3010e940 0x7f9b3019a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:11.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:11.623+0000 7f9b2fe00700 1 --2- 172.21.15.67:0/1310796475 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9b300744b0 0x7f9b301a06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:11.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:11.623+0000 7f9b2fe00700 1 --2- 172.21.15.67:0/1310796475 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b30074dd0 0x7f9b301a0be0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:11.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:11.623+0000 7f9b2fe00700 1 -- 172.21.15.67:0/1310796475 >> 172.21.15.67:0/1310796475 conn(0x7f9b300fdb90 msgr2=0x7f9b300fe900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:00:11.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:11.623+0000 7f9b2fe00700 1 -- 172.21.15.67:0/1310796475 shutdown_connections 2024-03-28T16:00:11.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:11.624+0000 7f9b2fe00700 1 -- 172.21.15.67:0/1310796475 wait complete. 2024-03-28T16:00:11.986 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:11 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/1310796475' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T16:00:12.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:11 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/1310796475' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T16:00:12.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:11 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/1310796475' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T16:00:12.437 INFO:tasks.cephadm:Adding mds.a on smithi067 2024-03-28T16:00:12.437 INFO:tasks.cephadm:Adding mds.d on smithi067 2024-03-28T16:00:12.437 INFO:tasks.cephadm:Adding mds.g on smithi067 2024-03-28T16:00:12.437 INFO:tasks.cephadm:Adding mds.j on smithi067 2024-03-28T16:00:12.437 INFO:tasks.cephadm:Adding mds.b on smithi150 2024-03-28T16:00:12.438 INFO:tasks.cephadm:Adding mds.e on smithi150 2024-03-28T16:00:12.438 INFO:tasks.cephadm:Adding mds.h on smithi150 2024-03-28T16:00:12.438 INFO:tasks.cephadm:Adding mds.k on smithi150 2024-03-28T16:00:12.438 INFO:tasks.cephadm:Adding mds.c on smithi183 2024-03-28T16:00:12.438 INFO:tasks.cephadm:Adding mds.f on smithi183 2024-03-28T16:00:12.438 INFO:tasks.cephadm:Adding mds.i on smithi183 2024-03-28T16:00:12.438 INFO:tasks.cephadm:Adding mds.l on smithi183 2024-03-28T16:00:12.438 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph orch apply mds all '12;smithi067=a;smithi067=d;smithi067=g;smithi067=j;smithi150=b;smithi150=e;smithi150=h;smithi150=k;smithi183=c;smithi183=f;smithi183=i;smithi183=l' 2024-03-28T16:00:12.883 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T16:00:13.079 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:12 smithi183 ceph-mon[35580]: pgmap v332: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:13.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:12 smithi150 ceph-mon[34825]: pgmap v332: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:13.321 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:12 smithi067 ceph-mon[31707]: pgmap v332: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:14.866 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T16:00:14.865+0000 7f12ee000700 1 -- 172.21.15.183:0/2518821651 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f12e8076160 msgr2=0x7f12e80745c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:14.867 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T16:00:14.865+0000 7f12ee000700 1 --2- 172.21.15.183:0/2518821651 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f12e8076160 0x7f12e80745c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f12d8009990 tx=0x7f12d8009ca0 comp rx=0 tx=0).stop 2024-03-28T16:00:14.867 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T16:00:14.865+0000 7f12ee000700 1 -- 172.21.15.183:0/2518821651 shutdown_connections 2024-03-28T16:00:14.867 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T16:00:14.865+0000 7f12ee000700 1 --2- 172.21.15.183:0/2518821651 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12e810a0a0 0x7f12e810c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:14.867 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T16:00:14.865+0000 7f12ee000700 1 --2- 172.21.15.183:0/2518821651 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f12e8076160 0x7f12e80745c0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:14.867 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T16:00:14.865+0000 7f12ee000700 1 --2- 172.21.15.183:0/2518821651 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12e8074b90 0x7f12e8075010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:14.867 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T16:00:14.865+0000 7f12ee000700 1 -- 172.21.15.183:0/2518821651 >> 172.21.15.183:0/2518821651 conn(0x7f12e80fd7a0 msgr2=0x7f12e80ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:00:14.867 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T16:00:14.865+0000 7f12ee000700 1 -- 172.21.15.183:0/2518821651 shutdown_connections 2024-03-28T16:00:14.867 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T16:00:14.865+0000 7f12ee000700 1 -- 172.21.15.183:0/2518821651 wait complete. 2024-03-28T16:00:14.869 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T16:00:14.866+0000 7f12ee000700 1 Processor -- start 2024-03-28T16:00:14.870 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T16:00:14.866+0000 7f12ee000700 1 -- start start 2024-03-28T16:00:14.870 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T16:00:14.867+0000 7f12ee000700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12e8074b90 0x7f12e819a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:14.870 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T16:00:14.867+0000 7f12ee000700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12e8076160 0x7f12e819aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:14.870 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T16:00:14.867+0000 7f12ee000700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f12e810a0a0 0x7f12e81a1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:14.870 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T16:00:14.867+0000 7f12ee000700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12e810fe80 con 0x7f12e8074b90 2024-03-28T16:00:14.870 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T16:00:14.867+0000 7f12ee000700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12e810fd00 con 0x7f12e8076160 2024-03-28T16:00:14.870 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T16:00:14.867+0000 7f12ee000700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12e8110000 con 0x7f12e810a0a0 2024-03-28T16:00:14.870 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T16:00:14.867+0000 7f12e7e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f12e810a0a0 0x7f12e81a1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:14.870 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T16:00:14.867+0000 7f12e7e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f12e810a0a0 0x7f12e81a1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.183:60252/0 (socket says 172.21.15.183:60252) 2024-03-28T16:00:14.870 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T16:00:14.867+0000 7f12e7e00700 1 -- 172.21.15.183:0/1287689869 learned_addr learned my addr 172.21.15.183:0/1287689869 (peer_addr_for_me v2:172.21.15.183:0/0) 2024-03-28T16:00:14.871 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T16:00:14.867+0000 7f12e7400700 1 --2- 172.21.15.183:0/1287689869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12e8074b90 0x7f12e819a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:14.871 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T16:00:14.867+0000 7f12e6a00700 1 --2- 172.21.15.183:0/1287689869 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12e8076160 0x7f12e819aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:14.871 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T16:00:14.868+0000 7f12e7e00700 1 -- 172.21.15.183:0/1287689869 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12e8076160 msgr2=0x7f12e819aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:14.871 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T16:00:14.868+0000 7f12e7e00700 1 --2- 172.21.15.183:0/1287689869 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12e8076160 0x7f12e819aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:14.871 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T16:00:14.868+0000 7f12e7e00700 1 -- 172.21.15.183:0/1287689869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12e8074b90 msgr2=0x7f12e819a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:14.871 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T16:00:14.868+0000 7f12e7e00700 1 --2- 172.21.15.183:0/1287689869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12e8074b90 0x7f12e819a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:14.871 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T16:00:14.868+0000 7f12e7e00700 1 -- 172.21.15.183:0/1287689869 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12e81a2190 con 0x7f12e810a0a0 2024-03-28T16:00:14.871 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T16:00:14.868+0000 7f12e7e00700 1 --2- 172.21.15.183:0/1287689869 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f12e810a0a0 0x7f12e81a1af0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f12e000add0 tx=0x7f12e000c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:00:14.871 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T16:00:14.868+0000 7f12dfe00700 1 -- 172.21.15.183:0/1287689869 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e0017020 con 0x7f12e810a0a0 2024-03-28T16:00:14.871 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T16:00:14.868+0000 7f12ee000700 1 -- 172.21.15.183:0/1287689869 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12e81a2480 con 0x7f12e810a0a0 2024-03-28T16:00:14.871 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T16:00:14.868+0000 7f12dfe00700 1 -- 172.21.15.183:0/1287689869 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f12e000cdf0 con 0x7f12e810a0a0 2024-03-28T16:00:14.871 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T16:00:14.868+0000 7f12dfe00700 1 -- 172.21.15.183:0/1287689869 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e0005460 con 0x7f12e810a0a0 2024-03-28T16:00:14.871 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T16:00:14.868+0000 7f12ee000700 1 -- 172.21.15.183:0/1287689869 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12e81a29c0 con 0x7f12e810a0a0 2024-03-28T16:00:14.872 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T16:00:14.870+0000 7f12ee000700 1 -- 172.21.15.183:0/1287689869 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12e8068e50 con 0x7f12e810a0a0 2024-03-28T16:00:14.872 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T16:00:14.872+0000 7f12dfe00700 1 -- 172.21.15.183:0/1287689869 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f12e0004ad0 con 0x7f12e810a0a0 2024-03-28T16:00:14.874 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T16:00:14.873+0000 7f12dfe00700 1 --2- 172.21.15.183:0/1287689869 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f12cc0a2890 0x7f12cc0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:14.874 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T16:00:14.873+0000 7f12dfe00700 1 -- 172.21.15.183:0/1287689869 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f12e00cde50 con 0x7f12e810a0a0 2024-03-28T16:00:14.875 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T16:00:14.874+0000 7f12e7400700 1 --2- 172.21.15.183:0/1287689869 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f12cc0a2890 0x7f12cc0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:14.876 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T16:00:14.875+0000 7f12e7400700 1 --2- 172.21.15.183:0/1287689869 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f12cc0a2890 0x7f12cc0a4d50 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f12d8006010 tx=0x7f12d8003870 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:00:14.879 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T16:00:14.878+0000 7f12dfe00700 1 -- 172.21.15.183:0/1287689869 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f12e0021020 con 0x7f12e810a0a0 2024-03-28T16:00:15.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:14 smithi183 ceph-mon[35580]: pgmap v333: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:15.116 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T16:00:15.115+0000 7f12ee000700 1 -- 172.21.15.183:0/1287689869 --> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi067=a;smithi067=d;smithi067=g;smithi067=j;smithi150=b;smithi150=e;smithi150=h;smithi150=k;smithi183=c;smithi183=f;smithi183=i;smithi183=l", "target": ["mon-mgr", ""]}) v1 -- 0x7f12e8062890 con 0x7f12cc0a2890 2024-03-28T16:00:15.135 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T16:00:15.134+0000 7f12dfe00700 1 -- 172.21.15.183:0/1287689869 <== mgr.14152 v2:172.21.15.67:6800/208726654 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f12e8062890 con 0x7f12cc0a2890 2024-03-28T16:00:15.135 INFO:teuthology.orchestra.run.smithi183.stdout:Scheduled mds.all update... 2024-03-28T16:00:15.138 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T16:00:15.137+0000 7f12ee000700 1 -- 172.21.15.183:0/1287689869 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f12cc0a2890 msgr2=0x7f12cc0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:15.138 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T16:00:15.137+0000 7f12ee000700 1 --2- 172.21.15.183:0/1287689869 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f12cc0a2890 0x7f12cc0a4d50 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f12d8006010 tx=0x7f12d8003870 comp rx=0 tx=0).stop 2024-03-28T16:00:15.139 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T16:00:15.137+0000 7f12ee000700 1 -- 172.21.15.183:0/1287689869 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f12e810a0a0 msgr2=0x7f12e81a1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:15.139 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T16:00:15.137+0000 7f12ee000700 1 --2- 172.21.15.183:0/1287689869 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f12e810a0a0 0x7f12e81a1af0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f12e000add0 tx=0x7f12e000c330 comp rx=0 tx=0).stop 2024-03-28T16:00:15.139 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T16:00:15.137+0000 7f12ee000700 1 -- 172.21.15.183:0/1287689869 shutdown_connections 2024-03-28T16:00:15.139 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T16:00:15.137+0000 7f12ee000700 1 --2- 172.21.15.183:0/1287689869 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f12cc0a2890 0x7f12cc0a4d50 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:15.139 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T16:00:15.137+0000 7f12ee000700 1 --2- 172.21.15.183:0/1287689869 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f12e810a0a0 0x7f12e81a1af0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:15.139 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T16:00:15.137+0000 7f12ee000700 1 --2- 172.21.15.183:0/1287689869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12e8074b90 0x7f12e819a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:15.139 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T16:00:15.137+0000 7f12ee000700 1 --2- 172.21.15.183:0/1287689869 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12e8076160 0x7f12e819aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:15.139 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T16:00:15.137+0000 7f12ee000700 1 -- 172.21.15.183:0/1287689869 >> 172.21.15.183:0/1287689869 conn(0x7f12e80fd7a0 msgr2=0x7f12e8105ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:00:15.139 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T16:00:15.137+0000 7f12ee000700 1 -- 172.21.15.183:0/1287689869 shutdown_connections 2024-03-28T16:00:15.139 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T16:00:15.137+0000 7f12ee000700 1 -- 172.21.15.183:0/1287689869 wait complete. 2024-03-28T16:00:15.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:14 smithi150 ceph-mon[34825]: pgmap v333: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:15.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:14 smithi067 ceph-mon[31707]: pgmap v333: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:15.857 DEBUG:teuthology.orchestra.run.smithi067:mds.a> sudo journalctl -f -n 0 -u ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@mds.a.service 2024-03-28T16:00:15.865 DEBUG:teuthology.orchestra.run.smithi067:mds.d> sudo journalctl -f -n 0 -u ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@mds.d.service 2024-03-28T16:00:15.913 DEBUG:teuthology.orchestra.run.smithi067:mds.g> sudo journalctl -f -n 0 -u ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@mds.g.service 2024-03-28T16:00:15.958 DEBUG:teuthology.orchestra.run.smithi067:mds.j> sudo journalctl -f -n 0 -u ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@mds.j.service 2024-03-28T16:00:16.006 DEBUG:teuthology.orchestra.run.smithi150:mds.b> sudo journalctl -f -n 0 -u ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@mds.b.service 2024-03-28T16:00:16.015 DEBUG:teuthology.orchestra.run.smithi150:mds.e> sudo journalctl -f -n 0 -u ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@mds.e.service 2024-03-28T16:00:16.060 DEBUG:teuthology.orchestra.run.smithi150:mds.h> sudo journalctl -f -n 0 -u ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@mds.h.service 2024-03-28T16:00:16.107 DEBUG:teuthology.orchestra.run.smithi150:mds.k> sudo journalctl -f -n 0 -u ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@mds.k.service 2024-03-28T16:00:16.152 DEBUG:teuthology.orchestra.run.smithi183:mds.c> sudo journalctl -f -n 0 -u ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@mds.c.service 2024-03-28T16:00:16.160 DEBUG:teuthology.orchestra.run.smithi183:mds.f> sudo journalctl -f -n 0 -u ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@mds.f.service 2024-03-28T16:00:16.206 DEBUG:teuthology.orchestra.run.smithi183:mds.i> sudo journalctl -f -n 0 -u ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@mds.i.service 2024-03-28T16:00:16.252 DEBUG:teuthology.orchestra.run.smithi183:mds.l> sudo journalctl -f -n 0 -u ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@mds.l.service 2024-03-28T16:00:16.298 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-03-28T16:00:16.299 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T16:00:16.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:16 smithi067 ceph-mon[31707]: pgmap v334: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:16.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:16 smithi067 ceph-mon[31707]: from='client.24415 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi067=a;smithi067=d;smithi067=g;smithi067=j;smithi150=b;smithi150=e;smithi150=h;smithi150=k;smithi183=c;smithi183=f;smithi183=i;smithi183=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T16:00:16.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:16 smithi067 ceph-mon[31707]: Saving service mds.all spec with placement smithi067=a;smithi067=d;smithi067=g;smithi067=j;smithi150=b;smithi150=e;smithi150=h;smithi150=k;smithi183=c;smithi183=f;smithi183=i;smithi183=l;count:12 2024-03-28T16:00:16.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:16 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:16.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:16 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T16:00:16.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:16 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:00:16.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:16 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T16:00:16.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:16 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:16.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:16 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T16:00:16.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:16 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T16:00:16.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:16 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:00:16.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:16 smithi067 ceph-mon[31707]: Deploying daemon mds.l on smithi183 2024-03-28T16:00:16.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:16.545+0000 7fc4dc600640 1 -- 172.21.15.67:0/3298659739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4d4105490 msgr2=0x7fc4d410bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:16.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:16.545+0000 7fc4dc600640 1 --2- 172.21.15.67:0/3298659739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4d4105490 0x7fc4d410bd20 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc4cc009ca0 tx=0x7fc4cc02c0f0 comp rx=0 tx=0).stop 2024-03-28T16:00:16.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:16.545+0000 7fc4dc600640 1 -- 172.21.15.67:0/3298659739 shutdown_connections 2024-03-28T16:00:16.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:16.545+0000 7fc4dc600640 1 --2- 172.21.15.67:0/3298659739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4d4105490 0x7fc4d410bd20 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:16.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:16.545+0000 7fc4dc600640 1 --2- 172.21.15.67:0/3298659739 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4d4104ad0 0x7fc4d4104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:16.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:16.545+0000 7fc4dc600640 1 --2- 172.21.15.67:0/3298659739 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc4d41038e0 0x7fc4d4103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:16.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:16.545+0000 7fc4dc600640 1 -- 172.21.15.67:0/3298659739 >> 172.21.15.67:0/3298659739 conn(0x7fc4d40fee90 msgr2=0x7fc4d41012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:00:16.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:16.546+0000 7fc4dc600640 1 -- 172.21.15.67:0/3298659739 shutdown_connections 2024-03-28T16:00:16.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:16.546+0000 7fc4dc600640 1 -- 172.21.15.67:0/3298659739 wait complete. 2024-03-28T16:00:16.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:16.547+0000 7fc4dc600640 1 Processor -- start 2024-03-28T16:00:16.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:16.547+0000 7fc4dc600640 1 -- start start 2024-03-28T16:00:16.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:16.547+0000 7fc4dc600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4d41038e0 0x7fc4d4199a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:16.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:16.547+0000 7fc4dc600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc4d4104ad0 0x7fc4d4199fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:16.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:16.547+0000 7fc4dc600640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4d4105490 0x7fc4d41a1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:16.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:16.547+0000 7fc4dc600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d410f3f0 con 0x7fc4d41038e0 2024-03-28T16:00:16.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:16.547+0000 7fc4dc600640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d410f270 con 0x7fc4d4105490 2024-03-28T16:00:16.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:16.547+0000 7fc4dc600640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d410f570 con 0x7fc4d4104ad0 2024-03-28T16:00:16.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:16.547+0000 7fc4d9c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4d41038e0 0x7fc4d4199a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:16.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:16.547+0000 7fc4d9c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4d41038e0 0x7fc4d4199a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48586/0 (socket says 172.21.15.67:48586) 2024-03-28T16:00:16.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:16.547+0000 7fc4d9c00640 1 -- 172.21.15.67:0/688063871 learned_addr learned my addr 172.21.15.67:0/688063871 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:00:16.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:16.548+0000 7fc4d9200640 1 --2- 172.21.15.67:0/688063871 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc4d4104ad0 0x7fc4d4199fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:16.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:16.548+0000 7fc4da600640 1 --2- 172.21.15.67:0/688063871 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4d4105490 0x7fc4d41a1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:16.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:16.548+0000 7fc4d9c00640 1 -- 172.21.15.67:0/688063871 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc4d4104ad0 msgr2=0x7fc4d4199fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:16.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:16.548+0000 7fc4d9c00640 1 --2- 172.21.15.67:0/688063871 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc4d4104ad0 0x7fc4d4199fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:16.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:16.548+0000 7fc4d9c00640 1 -- 172.21.15.67:0/688063871 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4d4105490 msgr2=0x7fc4d41a1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:16.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:16.548+0000 7fc4d9c00640 1 --2- 172.21.15.67:0/688063871 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4d4105490 0x7fc4d41a1040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:16.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:16.548+0000 7fc4d9c00640 1 -- 172.21.15.67:0/688063871 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4d41a16b0 con 0x7fc4d41038e0 2024-03-28T16:00:16.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:16.548+0000 7fc4d9c00640 1 --2- 172.21.15.67:0/688063871 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4d41038e0 0x7fc4d4199a90 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fc4c800d770 tx=0x7fc4c800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:00:16.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:16.548+0000 7fc4d2000640 1 -- 172.21.15.67:0/688063871 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c8017070 con 0x7fc4d41038e0 2024-03-28T16:00:16.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:16.548+0000 7fc4d2000640 1 -- 172.21.15.67:0/688063871 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc4c8009c30 con 0x7fc4d41038e0 2024-03-28T16:00:16.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:16.548+0000 7fc4dc600640 1 -- 172.21.15.67:0/688063871 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4d41a19a0 con 0x7fc4d41038e0 2024-03-28T16:00:16.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:16.548+0000 7fc4d2000640 1 -- 172.21.15.67:0/688063871 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c8003d00 con 0x7fc4d41038e0 2024-03-28T16:00:16.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:16.549+0000 7fc4dc600640 1 -- 172.21.15.67:0/688063871 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4d41a1ef0 con 0x7fc4d41038e0 2024-03-28T16:00:16.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:16.550+0000 7fc4dc600640 1 -- 172.21.15.67:0/688063871 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4d40688e0 con 0x7fc4d41038e0 2024-03-28T16:00:16.555 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:16 smithi150 ceph-mon[34825]: pgmap v334: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:16.555 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:16 smithi150 ceph-mon[34825]: from='client.24415 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi067=a;smithi067=d;smithi067=g;smithi067=j;smithi150=b;smithi150=e;smithi150=h;smithi150=k;smithi183=c;smithi183=f;smithi183=i;smithi183=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T16:00:16.555 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:16 smithi150 ceph-mon[34825]: Saving service mds.all spec with placement smithi067=a;smithi067=d;smithi067=g;smithi067=j;smithi150=b;smithi150=e;smithi150=h;smithi150=k;smithi183=c;smithi183=f;smithi183=i;smithi183=l;count:12 2024-03-28T16:00:16.555 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:16 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:16.555 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:16 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T16:00:16.555 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:16 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:00:16.555 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:16 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T16:00:16.556 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:16 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:16.556 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:16 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T16:00:16.556 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:16 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T16:00:16.556 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:16 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:00:16.556 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:16 smithi150 ceph-mon[34825]: Deploying daemon mds.l on smithi183 2024-03-28T16:00:16.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:16.552+0000 7fc4d2000640 1 -- 172.21.15.67:0/688063871 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fc4c8003ea0 con 0x7fc4d41038e0 2024-03-28T16:00:16.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:16.553+0000 7fc4d2000640 1 --2- 172.21.15.67:0/688063871 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fc4bc0a2800 0x7fc4bc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:16.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:16.553+0000 7fc4d2000640 1 -- 172.21.15.67:0/688063871 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fc4c80ce440 con 0x7fc4d41038e0 2024-03-28T16:00:16.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:16.553+0000 7fc4d9200640 1 --2- 172.21.15.67:0/688063871 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fc4bc0a2800 0x7fc4bc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:16.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:16.554+0000 7fc4d9200640 1 --2- 172.21.15.67:0/688063871 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fc4bc0a2800 0x7fc4bc0a4cc0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc4c4005fd0 tx=0x7fc4c4005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:00:16.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:16.558+0000 7fc4d2000640 1 -- 172.21.15.67:0/688063871 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc4c809b8e0 con 0x7fc4d41038e0 2024-03-28T16:00:16.570 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:16 smithi183 ceph-mon[35580]: pgmap v334: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:16.570 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:16 smithi183 ceph-mon[35580]: from='client.24415 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi067=a;smithi067=d;smithi067=g;smithi067=j;smithi150=b;smithi150=e;smithi150=h;smithi150=k;smithi183=c;smithi183=f;smithi183=i;smithi183=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T16:00:16.570 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:16 smithi183 ceph-mon[35580]: Saving service mds.all spec with placement smithi067=a;smithi067=d;smithi067=g;smithi067=j;smithi150=b;smithi150=e;smithi150=h;smithi150=k;smithi183=c;smithi183=f;smithi183=i;smithi183=l;count:12 2024-03-28T16:00:16.570 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:16 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:16.570 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:16 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T16:00:16.570 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:16 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:00:16.570 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:16 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T16:00:16.570 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:16 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:16.570 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:16 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T16:00:16.570 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:16 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T16:00:16.570 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:16 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:00:16.571 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:16 smithi183 ceph-mon[35580]: Deploying daemon mds.l on smithi183 2024-03-28T16:00:16.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:16.762+0000 7fc4dc600640 1 -- 172.21.15.67:0/688063871 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc4d4062360 con 0x7fc4d41038e0 2024-03-28T16:00:16.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:16.764+0000 7fc4d2000640 1 -- 172.21.15.67:0/688063871 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+18458 (secure 0 0 0) 0x7fc4c8009de0 con 0x7fc4d41038e0 2024-03-28T16:00:16.765 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:00:16.766 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":86,"fsid":"b5f8225a-ed1a-11ee-95d0-87774f69a715","created":"2024-03-28T15:50:21.471924+0000","modified":"2024-03-28T16:00:02.385208+0000","last_up_change":"2024-03-28T16:00:01.369123+0000","last_in_change":"2024-03-28T15:59:37.515483+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T15:54:55.588007+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7c1791b6-9c0e-4512-a327-cad166bb5581","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6803","nonce":1894681701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6805","nonce":1894681701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6809","nonce":1894681701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6807","nonce":1894681701}]},"public_addr":"172.21.15.67:6803/1894681701","cluster_addr":"172.21.15.67:6805/1894681701","heartbeat_back_addr":"172.21.15.67:6809/1894681701","heartbeat_front_addr":"172.21.15.67:6807/1894681701","state":["exists","up"]},{"osd":1,"uuid":"0cdda7f3-d00c-4436-bcc1-49fb8072b2e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6801","nonce":1734465762}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6803","nonce":1734465762}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6807","nonce":1734465762}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6805","nonce":1734465762}]},"public_addr":"172.21.15.150:6801/1734465762","cluster_addr":"172.21.15.150:6803/1734465762","heartbeat_back_addr":"172.21.15.150:6807/1734465762","heartbeat_front_addr":"172.21.15.150:6805/1734465762","state":["exists","up"]},{"osd":2,"uuid":"a55b9a02-c3fd-4061-95c0-53a24a17ade0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6801","nonce":3398095931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6803","nonce":3398095931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6807","nonce":3398095931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6805","nonce":3398095931}]},"public_addr":"172.21.15.183:6801/3398095931","cluster_addr":"172.21.15.183:6803/3398095931","heartbeat_back_addr":"172.21.15.183:6807/3398095931","heartbeat_front_addr":"172.21.15.183:6805/3398095931","state":["exists","up"]},{"osd":3,"uuid":"d34940c0-5ce9-4b8b-aa36-4872414a043f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6811","nonce":1933541875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6813","nonce":1933541875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6817","nonce":1933541875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6815","nonce":1933541875}]},"public_addr":"172.21.15.67:6811/1933541875","cluster_addr":"172.21.15.67:6813/1933541875","heartbeat_back_addr":"172.21.15.67:6817/1933541875","heartbeat_front_addr":"172.21.15.67:6815/1933541875","state":["exists","up"]},{"osd":4,"uuid":"5cc3e260-83a3-4a4d-aa6a-34a3b12b2b5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6809","nonce":3775269993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6811","nonce":3775269993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6815","nonce":3775269993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6813","nonce":3775269993}]},"public_addr":"172.21.15.150:6809/3775269993","cluster_addr":"172.21.15.150:6811/3775269993","heartbeat_back_addr":"172.21.15.150:6815/3775269993","heartbeat_front_addr":"172.21.15.150:6813/3775269993","state":["exists","up"]},{"osd":5,"uuid":"a2542b21-d6ac-413e-8b9b-4a6548b20840","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6809","nonce":164284137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6811","nonce":164284137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6815","nonce":164284137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6813","nonce":164284137}]},"public_addr":"172.21.15.183:6809/164284137","cluster_addr":"172.21.15.183:6811/164284137","heartbeat_back_addr":"172.21.15.183:6815/164284137","heartbeat_front_addr":"172.21.15.183:6813/164284137","state":["exists","up"]},{"osd":6,"uuid":"8905a2d3-0f1b-4853-a218-22da2635150a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6819","nonce":3565528013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6821","nonce":3565528013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6825","nonce":3565528013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6823","nonce":3565528013}]},"public_addr":"172.21.15.67:6819/3565528013","cluster_addr":"172.21.15.67:6821/3565528013","heartbeat_back_addr":"172.21.15.67:6825/3565528013","heartbeat_front_addr":"172.21.15.67:6823/3565528013","state":["exists","up"]},{"osd":7,"uuid":"b275055b-b451-4180-91b1-e6e61a828c48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6817","nonce":3735115340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6819","nonce":3735115340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6823","nonce":3735115340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6821","nonce":3735115340}]},"public_addr":"172.21.15.150:6817/3735115340","cluster_addr":"172.21.15.150:6819/3735115340","heartbeat_back_addr":"172.21.15.150:6823/3735115340","heartbeat_front_addr":"172.21.15.150:6821/3735115340","state":["exists","up"]},{"osd":8,"uuid":"b6abf7ad-eb34-4f60-a4eb-348a92d7195b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6817","nonce":1025400095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6819","nonce":1025400095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6823","nonce":1025400095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6821","nonce":1025400095}]},"public_addr":"172.21.15.183:6817/1025400095","cluster_addr":"172.21.15.183:6819/1025400095","heartbeat_back_addr":"172.21.15.183:6823/1025400095","heartbeat_front_addr":"172.21.15.183:6821/1025400095","state":["exists","up"]},{"osd":9,"uuid":"28a78e3a-0657-40aa-816f-16a6fadadcbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6827","nonce":76885409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6829","nonce":76885409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6833","nonce":76885409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6831","nonce":76885409}]},"public_addr":"172.21.15.67:6827/76885409","cluster_addr":"172.21.15.67:6829/76885409","heartbeat_back_addr":"172.21.15.67:6833/76885409","heartbeat_front_addr":"172.21.15.67:6831/76885409","state":["exists","up"]},{"osd":10,"uuid":"d287907e-424f-4b49-a06e-4613c6de0474","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6825","nonce":3489982080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6827","nonce":3489982080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6831","nonce":3489982080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6829","nonce":3489982080}]},"public_addr":"172.21.15.150:6825/3489982080","cluster_addr":"172.21.15.150:6827/3489982080","heartbeat_back_addr":"172.21.15.150:6831/3489982080","heartbeat_front_addr":"172.21.15.150:6829/3489982080","state":["exists","up"]},{"osd":11,"uuid":"a3c29c47-a228-4f42-a23c-fbdb0c093699","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":85,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6825","nonce":1892639849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6826","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6827","nonce":1892639849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6831","nonce":1892639849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6829","nonce":1892639849}]},"public_addr":"172.21.15.183:6825/1892639849","cluster_addr":"172.21.15.183:6827/1892639849","heartbeat_back_addr":"172.21.15.183:6831/1892639849","heartbeat_front_addr":"172.21.15.183:6829/1892639849","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:53:45.646211+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:54:18.956590+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:54:52.161646+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:55:27.502822+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:56:01.162932+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:56:34.674452+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:57:09.298679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:57:43.230406+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:58:17.003821+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:58:51.085863+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:59:25.665065+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T16:00:00.263457+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6801/1631001968":"2024-03-29T15:51:38.875334+0000","172.21.15.67:6800/1631001968":"2024-03-29T15:51:38.875334+0000","172.21.15.67:0/1533056310":"2024-03-29T15:51:38.875334+0000","172.21.15.67:6801/3580491071":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/2030412458":"2024-03-29T15:51:03.859740+0000","172.21.15.67:6800/3580491071":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/2905307533":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/3763759988":"2024-03-29T15:51:38.875334+0000","172.21.15.67:0/4089399880":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/4239108661":"2024-03-29T15:51:38.875334+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T16:00:16.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:16.766+0000 7fc4dc600640 1 -- 172.21.15.67:0/688063871 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fc4bc0a2800 msgr2=0x7fc4bc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:16.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:16.766+0000 7fc4dc600640 1 --2- 172.21.15.67:0/688063871 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fc4bc0a2800 0x7fc4bc0a4cc0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc4c4005fd0 tx=0x7fc4c4005850 comp rx=0 tx=0).stop 2024-03-28T16:00:16.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:16.766+0000 7fc4dc600640 1 -- 172.21.15.67:0/688063871 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4d41038e0 msgr2=0x7fc4d4199a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:16.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:16.766+0000 7fc4dc600640 1 --2- 172.21.15.67:0/688063871 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4d41038e0 0x7fc4d4199a90 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fc4c800d770 tx=0x7fc4c800dc30 comp rx=0 tx=0).stop 2024-03-28T16:00:16.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:16.767+0000 7fc4dc600640 1 -- 172.21.15.67:0/688063871 shutdown_connections 2024-03-28T16:00:16.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:16.767+0000 7fc4dc600640 1 --2- 172.21.15.67:0/688063871 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fc4bc0a2800 0x7fc4bc0a4cc0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:16.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:16.767+0000 7fc4dc600640 1 --2- 172.21.15.67:0/688063871 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4d4105490 0x7fc4d41a1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:16.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:16.767+0000 7fc4dc600640 1 --2- 172.21.15.67:0/688063871 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc4d4104ad0 0x7fc4d4199fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:16.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:16.767+0000 7fc4dc600640 1 --2- 172.21.15.67:0/688063871 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4d41038e0 0x7fc4d4199a90 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:16.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:16.767+0000 7fc4dc600640 1 -- 172.21.15.67:0/688063871 >> 172.21.15.67:0/688063871 conn(0x7fc4d40fee90 msgr2=0x7fc4d4107e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:00:16.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:16.767+0000 7fc4dc600640 1 -- 172.21.15.67:0/688063871 shutdown_connections 2024-03-28T16:00:16.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:16.767+0000 7fc4dc600640 1 -- 172.21.15.67:0/688063871 wait complete. 2024-03-28T16:00:16.791 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T15:54:55.588007+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T16:00:16.792 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T16:00:17.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.064+0000 7fb123000640 1 -- 172.21.15.67:0/3105788495 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb11c1015a0 msgr2=0x7fb11c1019c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:17.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.064+0000 7fb123000640 1 --2- 172.21.15.67:0/3105788495 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb11c1015a0 0x7fb11c1019c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb110009900 tx=0x7fb11002bc30 comp rx=0 tx=0).stop 2024-03-28T16:00:17.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.064+0000 7fb123000640 1 -- 172.21.15.67:0/3105788495 shutdown_connections 2024-03-28T16:00:17.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.064+0000 7fb123000640 1 --2- 172.21.15.67:0/3105788495 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb11c103150 0x7fb11c1099e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:17.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.064+0000 7fb123000640 1 --2- 172.21.15.67:0/3105788495 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb11c102790 0x7fb11c102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:17.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.064+0000 7fb123000640 1 --2- 172.21.15.67:0/3105788495 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb11c1015a0 0x7fb11c1019c0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:17.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.064+0000 7fb123000640 1 -- 172.21.15.67:0/3105788495 >> 172.21.15.67:0/3105788495 conn(0x7fb11c0fcb10 msgr2=0x7fb11c0fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:00:17.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.064+0000 7fb123000640 1 -- 172.21.15.67:0/3105788495 shutdown_connections 2024-03-28T16:00:17.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.065+0000 7fb123000640 1 -- 172.21.15.67:0/3105788495 wait complete. 2024-03-28T16:00:17.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.065+0000 7fb123000640 1 Processor -- start 2024-03-28T16:00:17.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.066+0000 7fb123000640 1 -- start start 2024-03-28T16:00:17.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.066+0000 7fb123000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb11c1015a0 0x7fb11c199a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:17.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.066+0000 7fb123000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb11c102790 0x7fb11c199f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:17.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.066+0000 7fb123000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb11c103150 0x7fb11c1a0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:17.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.066+0000 7fb123000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb11c10d170 con 0x7fb11c103150 2024-03-28T16:00:17.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.066+0000 7fb123000640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb11c10cff0 con 0x7fb11c1015a0 2024-03-28T16:00:17.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.066+0000 7fb123000640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb11c10d2f0 con 0x7fb11c102790 2024-03-28T16:00:17.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.066+0000 7fb122600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb11c103150 0x7fb11c1a0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:17.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.066+0000 7fb122600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb11c103150 0x7fb11c1a0fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48600/0 (socket says 172.21.15.67:48600) 2024-03-28T16:00:17.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.066+0000 7fb122600640 1 -- 172.21.15.67:0/582065381 learned_addr learned my addr 172.21.15.67:0/582065381 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:00:17.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.066+0000 7fb121200640 1 --2- 172.21.15.67:0/582065381 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb11c102790 0x7fb11c199f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:17.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.067+0000 7fb121c00640 1 --2- 172.21.15.67:0/582065381 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb11c1015a0 0x7fb11c199a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:17.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.067+0000 7fb122600640 1 -- 172.21.15.67:0/582065381 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb11c102790 msgr2=0x7fb11c199f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:17.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.067+0000 7fb122600640 1 --2- 172.21.15.67:0/582065381 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb11c102790 0x7fb11c199f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:17.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.067+0000 7fb122600640 1 -- 172.21.15.67:0/582065381 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb11c1015a0 msgr2=0x7fb11c199a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:17.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.067+0000 7fb122600640 1 --2- 172.21.15.67:0/582065381 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb11c1015a0 0x7fb11c199a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:17.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.067+0000 7fb122600640 1 -- 172.21.15.67:0/582065381 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb11c1a1650 con 0x7fb11c103150 2024-03-28T16:00:17.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.067+0000 7fb121c00640 1 --2- 172.21.15.67:0/582065381 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb11c1015a0 0x7fb11c199a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:00:17.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.067+0000 7fb122600640 1 --2- 172.21.15.67:0/582065381 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb11c103150 0x7fb11c1a0fe0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb11400eaa0 tx=0x7fb11400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:00:17.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.067+0000 7fb11a000640 1 -- 172.21.15.67:0/582065381 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb11400cdb0 con 0x7fb11c103150 2024-03-28T16:00:17.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.067+0000 7fb11a000640 1 -- 172.21.15.67:0/582065381 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb114005020 con 0x7fb11c103150 2024-03-28T16:00:17.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.067+0000 7fb11a000640 1 -- 172.21.15.67:0/582065381 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1140116d0 con 0x7fb11c103150 2024-03-28T16:00:17.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.067+0000 7fb123000640 1 -- 172.21.15.67:0/582065381 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb11c1a1940 con 0x7fb11c103150 2024-03-28T16:00:17.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.068+0000 7fb123000640 1 -- 172.21.15.67:0/582065381 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb11c1a1ec0 con 0x7fb11c103150 2024-03-28T16:00:17.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.069+0000 7fb123000640 1 -- 172.21.15.67:0/582065381 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb11c1a2150 con 0x7fb11c103150 2024-03-28T16:00:17.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.071+0000 7fb11a000640 1 -- 172.21.15.67:0/582065381 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fb1140051d0 con 0x7fb11c103150 2024-03-28T16:00:17.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.072+0000 7fb11a000640 1 --2- 172.21.15.67:0/582065381 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb1040a2800 0x7fb1040a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:17.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.072+0000 7fb11a000640 1 -- 172.21.15.67:0/582065381 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fb1140ce880 con 0x7fb11c103150 2024-03-28T16:00:17.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.072+0000 7fb121c00640 1 --2- 172.21.15.67:0/582065381 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb1040a2800 0x7fb1040a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:17.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.073+0000 7fb121c00640 1 --2- 172.21.15.67:0/582065381 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb1040a2800 0x7fb1040a4cc0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fb110036040 tx=0x7fb110005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:00:17.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.076+0000 7fb11a000640 1 -- 172.21.15.67:0/582065381 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb114016030 con 0x7fb11c103150 2024-03-28T16:00:17.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.284+0000 7fb123000640 1 -- 172.21.15.67:0/582065381 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb11c1a2320 con 0x7fb11c103150 2024-03-28T16:00:17.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.285+0000 7fb11a000640 1 -- 172.21.15.67:0/582065381 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v86) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb11409bdc0 con 0x7fb11c103150 2024-03-28T16:00:17.286 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-03-28T16:00:17.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.288+0000 7fb123000640 1 -- 172.21.15.67:0/582065381 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb1040a2800 msgr2=0x7fb1040a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:17.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.288+0000 7fb123000640 1 --2- 172.21.15.67:0/582065381 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb1040a2800 0x7fb1040a4cc0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fb110036040 tx=0x7fb110005fd0 comp rx=0 tx=0).stop 2024-03-28T16:00:17.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.288+0000 7fb123000640 1 -- 172.21.15.67:0/582065381 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb11c103150 msgr2=0x7fb11c1a0fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:17.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.288+0000 7fb123000640 1 --2- 172.21.15.67:0/582065381 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb11c103150 0x7fb11c1a0fe0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb11400eaa0 tx=0x7fb11400ef60 comp rx=0 tx=0).stop 2024-03-28T16:00:17.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.288+0000 7fb123000640 1 -- 172.21.15.67:0/582065381 shutdown_connections 2024-03-28T16:00:17.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.288+0000 7fb123000640 1 --2- 172.21.15.67:0/582065381 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb1040a2800 0x7fb1040a4cc0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:17.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.288+0000 7fb123000640 1 --2- 172.21.15.67:0/582065381 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb11c103150 0x7fb11c1a0fe0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:17.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.288+0000 7fb123000640 1 --2- 172.21.15.67:0/582065381 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb11c102790 0x7fb11c199f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:17.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.288+0000 7fb123000640 1 --2- 172.21.15.67:0/582065381 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb11c1015a0 0x7fb11c199a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:17.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.288+0000 7fb123000640 1 -- 172.21.15.67:0/582065381 >> 172.21.15.67:0/582065381 conn(0x7fb11c0fcb10 msgr2=0x7fb11c107f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:00:17.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.288+0000 7fb123000640 1 -- 172.21.15.67:0/582065381 shutdown_connections 2024-03-28T16:00:17.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.288+0000 7fb123000640 1 -- 172.21.15.67:0/582065381 wait complete. 2024-03-28T16:00:17.311 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T16:00:17.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:17 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/688063871' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:00:17.397 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:17 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/688063871' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:00:17.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.530+0000 7fc082c00640 1 -- 172.21.15.67:0/523868121 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc07c1048e0 msgr2=0x7fc07c104d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:17.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.530+0000 7fc082c00640 1 --2- 172.21.15.67:0/523868121 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc07c1048e0 0x7fc07c104d60 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc068009900 tx=0x7fc06802bc30 comp rx=0 tx=0).stop 2024-03-28T16:00:17.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.530+0000 7fc082c00640 1 -- 172.21.15.67:0/523868121 shutdown_connections 2024-03-28T16:00:17.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.530+0000 7fc082c00640 1 --2- 172.21.15.67:0/523868121 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc07c1052a0 0x7fc07c10bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:17.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.530+0000 7fc082c00640 1 --2- 172.21.15.67:0/523868121 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc07c1048e0 0x7fc07c104d60 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:17.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.530+0000 7fc082c00640 1 --2- 172.21.15.67:0/523868121 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc07c1036f0 0x7fc07c103b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:17.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.530+0000 7fc082c00640 1 -- 172.21.15.67:0/523868121 >> 172.21.15.67:0/523868121 conn(0x7fc07c0fec80 msgr2=0x7fc07c1010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:00:17.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.530+0000 7fc082c00640 1 -- 172.21.15.67:0/523868121 shutdown_connections 2024-03-28T16:00:17.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.530+0000 7fc082c00640 1 -- 172.21.15.67:0/523868121 wait complete. 2024-03-28T16:00:17.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.531+0000 7fc082c00640 1 Processor -- start 2024-03-28T16:00:17.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.531+0000 7fc082c00640 1 -- start start 2024-03-28T16:00:17.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.531+0000 7fc082c00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc07c1036f0 0x7fc07c199900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:17.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.531+0000 7fc082c00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc07c1048e0 0x7fc07c199e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:17.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.531+0000 7fc082c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc07c1052a0 0x7fc07c1a0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:17.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.531+0000 7fc082c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc07c10f250 con 0x7fc07c1052a0 2024-03-28T16:00:17.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.531+0000 7fc082c00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc07c10f0d0 con 0x7fc07c1048e0 2024-03-28T16:00:17.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.531+0000 7fc082c00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc07c10f3d0 con 0x7fc07c1036f0 2024-03-28T16:00:17.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.532+0000 7fc082200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc07c1052a0 0x7fc07c1a0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:17.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.532+0000 7fc082200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc07c1052a0 0x7fc07c1a0eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48612/0 (socket says 172.21.15.67:48612) 2024-03-28T16:00:17.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.532+0000 7fc082200640 1 -- 172.21.15.67:0/1717682902 learned_addr learned my addr 172.21.15.67:0/1717682902 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:00:17.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.532+0000 7fc081800640 1 --2- 172.21.15.67:0/1717682902 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc07c1036f0 0x7fc07c199900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:17.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.532+0000 7fc080e00640 1 --2- 172.21.15.67:0/1717682902 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc07c1048e0 0x7fc07c199e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:17.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.532+0000 7fc082200640 1 -- 172.21.15.67:0/1717682902 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc07c1036f0 msgr2=0x7fc07c199900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:17.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.532+0000 7fc082200640 1 --2- 172.21.15.67:0/1717682902 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc07c1036f0 0x7fc07c199900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:17.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.532+0000 7fc082200640 1 -- 172.21.15.67:0/1717682902 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc07c1048e0 msgr2=0x7fc07c199e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:17.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.532+0000 7fc082200640 1 --2- 172.21.15.67:0/1717682902 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc07c1048e0 0x7fc07c199e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:17.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.532+0000 7fc082200640 1 -- 172.21.15.67:0/1717682902 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc07c1a1520 con 0x7fc07c1052a0 2024-03-28T16:00:17.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.532+0000 7fc080e00640 1 --2- 172.21.15.67:0/1717682902 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc07c1048e0 0x7fc07c199e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:00:17.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.532+0000 7fc082200640 1 --2- 172.21.15.67:0/1717682902 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc07c1052a0 0x7fc07c1a0eb0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fc07400eaa0 tx=0x7fc07400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:00:17.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.533+0000 7fc07a000640 1 -- 172.21.15.67:0/1717682902 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc07400cdb0 con 0x7fc07c1052a0 2024-03-28T16:00:17.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.533+0000 7fc07a000640 1 -- 172.21.15.67:0/1717682902 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc074005020 con 0x7fc07c1052a0 2024-03-28T16:00:17.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.533+0000 7fc07a000640 1 -- 172.21.15.67:0/1717682902 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0740116d0 con 0x7fc07c1052a0 2024-03-28T16:00:17.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.533+0000 7fc082c00640 1 -- 172.21.15.67:0/1717682902 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc07c1a1810 con 0x7fc07c1052a0 2024-03-28T16:00:17.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.533+0000 7fc082c00640 1 -- 172.21.15.67:0/1717682902 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc07c1a1d60 con 0x7fc07c1052a0 2024-03-28T16:00:17.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.534+0000 7fc082c00640 1 -- 172.21.15.67:0/1717682902 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc07c10aa90 con 0x7fc07c1052a0 2024-03-28T16:00:17.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.536+0000 7fc07a000640 1 -- 172.21.15.67:0/1717682902 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fc0740051d0 con 0x7fc07c1052a0 2024-03-28T16:00:17.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.537+0000 7fc07a000640 1 --2- 172.21.15.67:0/1717682902 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fc0640a2880 0x7fc0640a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:17.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.537+0000 7fc07a000640 1 -- 172.21.15.67:0/1717682902 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fc0740ce8b0 con 0x7fc07c1052a0 2024-03-28T16:00:17.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.538+0000 7fc081800640 1 --2- 172.21.15.67:0/1717682902 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fc0640a2880 0x7fc0640a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:17.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.538+0000 7fc081800640 1 --2- 172.21.15.67:0/1717682902 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fc0640a2880 0x7fc0640a4d40 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fc070005fd0 tx=0x7fc070005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:00:17.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.542+0000 7fc07a000640 1 -- 172.21.15.67:0/1717682902 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc074016030 con 0x7fc07c1052a0 2024-03-28T16:00:17.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:17 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/688063871' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:00:17.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.788+0000 7fc082c00640 1 -- 172.21.15.67:0/1717682902 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc07c02d070 con 0x7fc07c1052a0 2024-03-28T16:00:17.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.789+0000 7fc07a000640 1 -- 172.21.15.67:0/1717682902 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) v1 ==== 93+0+535 (secure 0 0 0) 0x7fc07409bd50 con 0x7fc07c1052a0 2024-03-28T16:00:17.791 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:00:17.791 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-28T16:00:17.791 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 1 2024-03-28T16:00:17.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.792+0000 7fc082c00640 1 -- 172.21.15.67:0/1717682902 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fc0640a2880 msgr2=0x7fc0640a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:17.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.792+0000 7fc082c00640 1 --2- 172.21.15.67:0/1717682902 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fc0640a2880 0x7fc0640a4d40 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fc070005fd0 tx=0x7fc070005850 comp rx=0 tx=0).stop 2024-03-28T16:00:17.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.792+0000 7fc082c00640 1 -- 172.21.15.67:0/1717682902 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc07c1052a0 msgr2=0x7fc07c1a0eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:17.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.792+0000 7fc082c00640 1 --2- 172.21.15.67:0/1717682902 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc07c1052a0 0x7fc07c1a0eb0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fc07400eaa0 tx=0x7fc07400ef60 comp rx=0 tx=0).stop 2024-03-28T16:00:17.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.792+0000 7fc082c00640 1 -- 172.21.15.67:0/1717682902 shutdown_connections 2024-03-28T16:00:17.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.792+0000 7fc082c00640 1 --2- 172.21.15.67:0/1717682902 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fc0640a2880 0x7fc0640a4d40 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:17.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.792+0000 7fc082c00640 1 --2- 172.21.15.67:0/1717682902 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc07c1052a0 0x7fc07c1a0eb0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:17.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.792+0000 7fc082c00640 1 --2- 172.21.15.67:0/1717682902 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc07c1048e0 0x7fc07c199e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:17.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.792+0000 7fc082c00640 1 --2- 172.21.15.67:0/1717682902 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc07c1036f0 0x7fc07c199900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:17.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.792+0000 7fc082c00640 1 -- 172.21.15.67:0/1717682902 >> 172.21.15.67:0/1717682902 conn(0x7fc07c0fec80 msgr2=0x7fc07c107c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:00:17.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.792+0000 7fc082c00640 1 -- 172.21.15.67:0/1717682902 shutdown_connections 2024-03-28T16:00:17.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:17.792+0000 7fc082c00640 1 -- 172.21.15.67:0/1717682902 wait complete. 2024-03-28T16:00:18.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:18 smithi150 ceph-mon[34825]: pgmap v335: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:18.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:18 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/582065381' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T16:00:18.553 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:18 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/1717682902' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:00:18.557 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:18 smithi183 ceph-mon[35580]: pgmap v335: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:18.557 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:18 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/582065381' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T16:00:18.557 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:18 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/1717682902' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:00:18.557 INFO:journalctl@ceph.mds.l.smithi183.stdout:Mar 28 16:00:18 smithi183 systemd[1]: Starting Ceph mds.l for b5f8225a-ed1a-11ee-95d0-87774f69a715... 2024-03-28T16:00:18.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:18 smithi067 ceph-mon[31707]: pgmap v335: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:18.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:18 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/582065381' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T16:00:18.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:18 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/1717682902' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:00:18.829 INFO:journalctl@ceph.mds.l.smithi183.stdout:Mar 28 16:00:18 smithi183 podman[65559]: 2024-03-28 16:00:18.460174367 +0000 UTC m=+0.033615974 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T16:00:18.829 INFO:journalctl@ceph.mds.l.smithi183.stdout:Mar 28 16:00:18 smithi183 podman[65559]: 2024-03-28 16:00:18.590585962 +0000 UTC m=+0.164027564 container create 4489b0bde9085bba82870b23a47a0018f73b0e52b526dc8d98a72242ea6be629 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-l, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-28T16:00:18.829 INFO:journalctl@ceph.mds.l.smithi183.stdout:Mar 28 16:00:18 smithi183 podman[65559]: 2024-03-28 16:00:18.82801087 +0000 UTC m=+0.401452572 container init 4489b0bde9085bba82870b23a47a0018f73b0e52b526dc8d98a72242ea6be629 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-l, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=) 2024-03-28T16:00:19.101 INFO:journalctl@ceph.mds.l.smithi183.stdout:Mar 28 16:00:18 smithi183 podman[65559]: 2024-03-28 16:00:18.838229154 +0000 UTC m=+0.411670783 container start 4489b0bde9085bba82870b23a47a0018f73b0e52b526dc8d98a72242ea6be629 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-l, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T16:00:19.101 INFO:journalctl@ceph.mds.l.smithi183.stdout:Mar 28 16:00:18 smithi183 ceph-mds[65578]: -- 172.21.15.183:0/4208162600 <== mon.1 v2:172.21.15.183:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x564e9a651a40 con 0x564e9a625000 2024-03-28T16:00:19.101 INFO:journalctl@ceph.mds.l.smithi183.stdout:Mar 28 16:00:18 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-l[65574]: starting mds.l at 2024-03-28T16:00:19.101 INFO:journalctl@ceph.mds.l.smithi183.stdout:Mar 28 16:00:18 smithi183 bash[65559]: 4489b0bde9085bba82870b23a47a0018f73b0e52b526dc8d98a72242ea6be629 2024-03-28T16:00:19.102 INFO:journalctl@ceph.mds.l.smithi183.stdout:Mar 28 16:00:18 smithi183 systemd[1]: Started Ceph mds.l for b5f8225a-ed1a-11ee-95d0-87774f69a715. 2024-03-28T16:00:19.815 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T16:00:20.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:20.052+0000 7ffbee200640 1 -- 172.21.15.67:0/3553261431 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbe81052a0 msgr2=0x7ffbe810bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:20.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:20.052+0000 7ffbee200640 1 --2- 172.21.15.67:0/3553261431 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbe81052a0 0x7ffbe810bb30 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ffbe0009ca0 tx=0x7ffbe002c0f0 comp rx=0 tx=0).stop 2024-03-28T16:00:20.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:20.052+0000 7ffbee200640 1 -- 172.21.15.67:0/3553261431 shutdown_connections 2024-03-28T16:00:20.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:20.052+0000 7ffbee200640 1 --2- 172.21.15.67:0/3553261431 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbe81052a0 0x7ffbe810bb30 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:20.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:20.052+0000 7ffbee200640 1 --2- 172.21.15.67:0/3553261431 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffbe81048e0 0x7ffbe8104d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:20.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:20.052+0000 7ffbee200640 1 --2- 172.21.15.67:0/3553261431 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffbe81036f0 0x7ffbe8103b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:20.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:20.052+0000 7ffbee200640 1 -- 172.21.15.67:0/3553261431 >> 172.21.15.67:0/3553261431 conn(0x7ffbe80fec80 msgr2=0x7ffbe81010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:00:20.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:20.052+0000 7ffbee200640 1 -- 172.21.15.67:0/3553261431 shutdown_connections 2024-03-28T16:00:20.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:20.052+0000 7ffbee200640 1 -- 172.21.15.67:0/3553261431 wait complete. 2024-03-28T16:00:20.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:20.053+0000 7ffbee200640 1 Processor -- start 2024-03-28T16:00:20.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:20.053+0000 7ffbee200640 1 -- start start 2024-03-28T16:00:20.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:20.053+0000 7ffbee200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffbe81036f0 0x7ffbe81998c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:20.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:20.053+0000 7ffbee200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbe81048e0 0x7ffbe8199e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:20.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:20.053+0000 7ffbee200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffbe81052a0 0x7ffbe81a0e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:20.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:20.053+0000 7ffbee200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe810f220 con 0x7ffbe81048e0 2024-03-28T16:00:20.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:20.053+0000 7ffbee200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe810f0a0 con 0x7ffbe81036f0 2024-03-28T16:00:20.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:20.053+0000 7ffbee200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe810f3a0 con 0x7ffbe81052a0 2024-03-28T16:00:20.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:20.054+0000 7ffbe7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbe81048e0 0x7ffbe8199e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:20.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:20.054+0000 7ffbe7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbe81048e0 0x7ffbe8199e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59966/0 (socket says 172.21.15.67:59966) 2024-03-28T16:00:20.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:20.054+0000 7ffbe7e00640 1 -- 172.21.15.67:0/4086079813 learned_addr learned my addr 172.21.15.67:0/4086079813 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:00:20.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:20.054+0000 7ffbece00640 1 --2- 172.21.15.67:0/4086079813 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffbe81036f0 0x7ffbe81998c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:20.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:20.054+0000 7ffbed800640 1 --2- 172.21.15.67:0/4086079813 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffbe81052a0 0x7ffbe81a0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:20.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:20.054+0000 7ffbe7e00640 1 -- 172.21.15.67:0/4086079813 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffbe81052a0 msgr2=0x7ffbe81a0e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:20.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:20.054+0000 7ffbe7e00640 1 --2- 172.21.15.67:0/4086079813 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffbe81052a0 0x7ffbe81a0e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:20.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:20.054+0000 7ffbe7e00640 1 -- 172.21.15.67:0/4086079813 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffbe81036f0 msgr2=0x7ffbe81998c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:20.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:20.054+0000 7ffbe7e00640 1 --2- 172.21.15.67:0/4086079813 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffbe81036f0 0x7ffbe81998c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:20.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:20.054+0000 7ffbe7e00640 1 -- 172.21.15.67:0/4086079813 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbe81a14e0 con 0x7ffbe81048e0 2024-03-28T16:00:20.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:20.055+0000 7ffbe7e00640 1 --2- 172.21.15.67:0/4086079813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbe81048e0 0x7ffbe8199e00 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ffbd800e9f0 tx=0x7ffbd800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:00:20.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:20.055+0000 7ffbe5600640 1 -- 172.21.15.67:0/4086079813 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd800cd80 con 0x7ffbe81048e0 2024-03-28T16:00:20.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:20.055+0000 7ffbe5600640 1 -- 172.21.15.67:0/4086079813 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ffbd80108e0 con 0x7ffbe81048e0 2024-03-28T16:00:20.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:20.055+0000 7ffbe5600640 1 -- 172.21.15.67:0/4086079813 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd8018920 con 0x7ffbe81048e0 2024-03-28T16:00:20.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:20.055+0000 7ffbee200640 1 -- 172.21.15.67:0/4086079813 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbe81a17d0 con 0x7ffbe81048e0 2024-03-28T16:00:20.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:20.055+0000 7ffbee200640 1 -- 172.21.15.67:0/4086079813 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbe81a1d70 con 0x7ffbe81048e0 2024-03-28T16:00:20.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:20.056+0000 7ffbee200640 1 -- 172.21.15.67:0/4086079813 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbe810aa10 con 0x7ffbe81048e0 2024-03-28T16:00:20.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:20.058+0000 7ffbe5600640 1 -- 172.21.15.67:0/4086079813 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ffbd8010a90 con 0x7ffbe81048e0 2024-03-28T16:00:20.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:20.059+0000 7ffbe5600640 1 --2- 172.21.15.67:0/4086079813 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7ffbd00a2880 0x7ffbd00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:20.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:20.060+0000 7ffbe5600640 1 -- 172.21.15.67:0/4086079813 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ffbd80ceea0 con 0x7ffbe81048e0 2024-03-28T16:00:20.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:20.060+0000 7ffbece00640 1 --2- 172.21.15.67:0/4086079813 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7ffbd00a2880 0x7ffbd00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:20.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:20.061+0000 7ffbece00640 1 --2- 172.21.15.67:0/4086079813 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7ffbd00a2880 0x7ffbd00a4d40 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc005fd0 tx=0x7ffbdc005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:00:20.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:20.063+0000 7ffbe5600640 1 -- 172.21.15.67:0/4086079813 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ffbd809c340 con 0x7ffbe81048e0 2024-03-28T16:00:20.173 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:19 smithi150 ceph-mon[34825]: mds.? [v2:172.21.15.183:6832/2129665423,v1:172.21.15.183:6833/2129665423] up:boot 2024-03-28T16:00:20.174 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:19 smithi150 ceph-mon[34825]: fsmap 1 up:standby 2024-03-28T16:00:20.174 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:19 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T16:00:20.174 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:19 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:20.174 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:19 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:20.174 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:19 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:20.174 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:19 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T16:00:20.174 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:19 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T16:00:20.174 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:19 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:00:20.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:20.302+0000 7ffbee200640 1 -- 172.21.15.67:0/4086079813 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffbe802d070 con 0x7ffbe81048e0 2024-03-28T16:00:20.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:20.303+0000 7ffbe5600640 1 -- 172.21.15.67:0/4086079813 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1328 (secure 0 0 0) 0x7ffbd809bc40 con 0x7ffbe81048e0 2024-03-28T16:00:20.304 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:00:20.304 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2129665423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2129665423},{"type":"v1","addr":"172.21.15.183:6833","nonce":2129665423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-28T16:00:20.304 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 2 2024-03-28T16:00:20.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:20.305+0000 7ffbee200640 1 -- 172.21.15.67:0/4086079813 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7ffbd00a2880 msgr2=0x7ffbd00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:20.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:20.305+0000 7ffbee200640 1 --2- 172.21.15.67:0/4086079813 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7ffbd00a2880 0x7ffbd00a4d40 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc005fd0 tx=0x7ffbdc005850 comp rx=0 tx=0).stop 2024-03-28T16:00:20.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:20.305+0000 7ffbee200640 1 -- 172.21.15.67:0/4086079813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbe81048e0 msgr2=0x7ffbe8199e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:20.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:20.305+0000 7ffbee200640 1 --2- 172.21.15.67:0/4086079813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbe81048e0 0x7ffbe8199e00 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ffbd800e9f0 tx=0x7ffbd800eeb0 comp rx=0 tx=0).stop 2024-03-28T16:00:20.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:20.305+0000 7ffbee200640 1 -- 172.21.15.67:0/4086079813 shutdown_connections 2024-03-28T16:00:20.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:20.305+0000 7ffbee200640 1 --2- 172.21.15.67:0/4086079813 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7ffbd00a2880 0x7ffbd00a4d40 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:20.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:20.305+0000 7ffbee200640 1 --2- 172.21.15.67:0/4086079813 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffbe81052a0 0x7ffbe81a0e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:20.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:20.305+0000 7ffbee200640 1 --2- 172.21.15.67:0/4086079813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbe81048e0 0x7ffbe8199e00 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:20.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:20.305+0000 7ffbee200640 1 --2- 172.21.15.67:0/4086079813 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffbe81036f0 0x7ffbe81998c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:20.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:20.305+0000 7ffbee200640 1 -- 172.21.15.67:0/4086079813 >> 172.21.15.67:0/4086079813 conn(0x7ffbe80fec80 msgr2=0x7ffbe8107be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:00:20.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:20.305+0000 7ffbee200640 1 -- 172.21.15.67:0/4086079813 shutdown_connections 2024-03-28T16:00:20.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:20.305+0000 7ffbee200640 1 -- 172.21.15.67:0/4086079813 wait complete. 2024-03-28T16:00:20.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:19 smithi067 ceph-mon[31707]: mds.? [v2:172.21.15.183:6832/2129665423,v1:172.21.15.183:6833/2129665423] up:boot 2024-03-28T16:00:20.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:19 smithi067 ceph-mon[31707]: fsmap 1 up:standby 2024-03-28T16:00:20.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:19 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T16:00:20.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:19 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:20.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:19 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:20.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:19 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:20.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:19 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T16:00:20.321 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:19 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T16:00:20.321 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:19 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:00:20.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:19 smithi183 ceph-mon[35580]: mds.? [v2:172.21.15.183:6832/2129665423,v1:172.21.15.183:6833/2129665423] up:boot 2024-03-28T16:00:20.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:19 smithi183 ceph-mon[35580]: fsmap 1 up:standby 2024-03-28T16:00:20.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:19 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T16:00:20.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:19 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:20.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:19 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:20.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:19 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:20.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:19 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T16:00:20.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:19 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T16:00:20.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:19 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:00:21.129 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:20 smithi150 ceph-mon[34825]: pgmap v336: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:21.129 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:20 smithi150 ceph-mon[34825]: Deploying daemon mds.k on smithi150 2024-03-28T16:00:21.129 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:20 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/4086079813' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:00:21.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:20 smithi067 ceph-mon[31707]: pgmap v336: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:21.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:20 smithi067 ceph-mon[31707]: Deploying daemon mds.k on smithi150 2024-03-28T16:00:21.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:20 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/4086079813' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:00:21.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:20 smithi183 ceph-mon[35580]: pgmap v336: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:21.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:20 smithi183 ceph-mon[35580]: Deploying daemon mds.k on smithi150 2024-03-28T16:00:21.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:20 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/4086079813' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:00:22.303 INFO:journalctl@ceph.mds.k.smithi150.stdout:Mar 28 16:00:21 smithi150 systemd[1]: Starting Ceph mds.k for b5f8225a-ed1a-11ee-95d0-87774f69a715... 2024-03-28T16:00:22.326 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T16:00:22.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:22.558+0000 7f97c7200640 1 -- 172.21.15.67:0/3584025909 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97c8103090 msgr2=0x7f97c8109920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:22.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:22.558+0000 7f97c7200640 1 --2- 172.21.15.67:0/3584025909 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97c8103090 0x7f97c8109920 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f97c0009ca0 tx=0x7f97c002c100 comp rx=0 tx=0).stop 2024-03-28T16:00:22.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:22.559+0000 7f97c7200640 1 -- 172.21.15.67:0/3584025909 shutdown_connections 2024-03-28T16:00:22.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:22.559+0000 7f97c7200640 1 --2- 172.21.15.67:0/3584025909 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97c8103090 0x7f97c8109920 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:22.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:22.559+0000 7f97c7200640 1 --2- 172.21.15.67:0/3584025909 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f97c81026d0 0x7f97c8102b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:22.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:22.559+0000 7f97c7200640 1 --2- 172.21.15.67:0/3584025909 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97c81014e0 0x7f97c8101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:22.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:22.559+0000 7f97c7200640 1 -- 172.21.15.67:0/3584025909 >> 172.21.15.67:0/3584025909 conn(0x7f97c80fca30 msgr2=0x7f97c80fee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:00:22.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:22.559+0000 7f97c7200640 1 -- 172.21.15.67:0/3584025909 shutdown_connections 2024-03-28T16:00:22.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:22.559+0000 7f97c7200640 1 -- 172.21.15.67:0/3584025909 wait complete. 2024-03-28T16:00:22.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:22.560+0000 7f97c7200640 1 Processor -- start 2024-03-28T16:00:22.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:22.560+0000 7f97c7200640 1 -- start start 2024-03-28T16:00:22.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:22.560+0000 7f97c7200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f97c81014e0 0x7f97c8199960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:22.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:22.560+0000 7f97c7200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97c81026d0 0x7f97c8199ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:22.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:22.560+0000 7f97c7200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97c8103090 0x7f97c81a0e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:22.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:22.560+0000 7f97c7200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97c810cff0 con 0x7f97c8103090 2024-03-28T16:00:22.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:22.560+0000 7f97c7200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97c810ce70 con 0x7f97c81026d0 2024-03-28T16:00:22.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:22.560+0000 7f97c7200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97c810d170 con 0x7f97c81014e0 2024-03-28T16:00:22.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:22.561+0000 7f97c6800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97c8103090 0x7f97c81a0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:22.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:22.561+0000 7f97c6800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97c8103090 0x7f97c81a0e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59984/0 (socket says 172.21.15.67:59984) 2024-03-28T16:00:22.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:22.561+0000 7f97c6800640 1 -- 172.21.15.67:0/2198800405 learned_addr learned my addr 172.21.15.67:0/2198800405 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:00:22.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:22.561+0000 7f97bfe00640 1 --2- 172.21.15.67:0/2198800405 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97c81026d0 0x7f97c8199ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:22.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:22.561+0000 7f97c5e00640 1 --2- 172.21.15.67:0/2198800405 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f97c81014e0 0x7f97c8199960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:22.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:22.561+0000 7f97c6800640 1 -- 172.21.15.67:0/2198800405 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f97c81014e0 msgr2=0x7f97c8199960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:22.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:22.561+0000 7f97c6800640 1 --2- 172.21.15.67:0/2198800405 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f97c81014e0 0x7f97c8199960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:22.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:22.561+0000 7f97c6800640 1 -- 172.21.15.67:0/2198800405 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97c81026d0 msgr2=0x7f97c8199ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:22.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:22.561+0000 7f97c6800640 1 --2- 172.21.15.67:0/2198800405 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97c81026d0 0x7f97c8199ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:22.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:22.561+0000 7f97c6800640 1 -- 172.21.15.67:0/2198800405 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97c81a14e0 con 0x7f97c8103090 2024-03-28T16:00:22.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:22.561+0000 7f97bfe00640 1 --2- 172.21.15.67:0/2198800405 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97c81026d0 0x7f97c8199ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T16:00:22.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:22.561+0000 7f97c6800640 1 --2- 172.21.15.67:0/2198800405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97c8103090 0x7f97c81a0e70 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f97c0032520 tx=0x7f97c0034880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:00:22.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:22.562+0000 7f97bea00640 1 -- 172.21.15.67:0/2198800405 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97c0039040 con 0x7f97c8103090 2024-03-28T16:00:22.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:22.562+0000 7f97bea00640 1 -- 172.21.15.67:0/2198800405 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f97c0044070 con 0x7f97c8103090 2024-03-28T16:00:22.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:22.562+0000 7f97c7200640 1 -- 172.21.15.67:0/2198800405 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97c81a1770 con 0x7f97c8103090 2024-03-28T16:00:22.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:22.562+0000 7f97bea00640 1 -- 172.21.15.67:0/2198800405 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97c00043c0 con 0x7f97c8103090 2024-03-28T16:00:22.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:22.562+0000 7f97c7200640 1 -- 172.21.15.67:0/2198800405 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97c81a1cb0 con 0x7f97c8103090 2024-03-28T16:00:22.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:22.563+0000 7f97c7200640 1 -- 172.21.15.67:0/2198800405 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97c81a1f10 con 0x7f97c8103090 2024-03-28T16:00:22.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:22.565+0000 7f97bea00640 1 -- 172.21.15.67:0/2198800405 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f97c0004560 con 0x7f97c8103090 2024-03-28T16:00:22.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:22.566+0000 7f97bea00640 1 --2- 172.21.15.67:0/2198800405 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f97ac0a2880 0x7f97ac0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:22.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:22.566+0000 7f97bea00640 1 -- 172.21.15.67:0/2198800405 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f97c00f0d20 con 0x7f97c8103090 2024-03-28T16:00:22.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:22.566+0000 7f97c5e00640 1 --2- 172.21.15.67:0/2198800405 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f97ac0a2880 0x7f97ac0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:22.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:22.567+0000 7f97c5e00640 1 --2- 172.21.15.67:0/2198800405 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f97ac0a2880 0x7f97ac0a4d40 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f97b8005fd0 tx=0x7f97b8005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:00:22.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:22.571+0000 7f97bea00640 1 -- 172.21.15.67:0/2198800405 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f97c00333a0 con 0x7f97c8103090 2024-03-28T16:00:22.686 INFO:journalctl@ceph.mds.k.smithi150.stdout:Mar 28 16:00:22 smithi150 podman[64594]: 2024-03-28 16:00:22.278494832 +0000 UTC m=+0.039658033 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T16:00:22.686 INFO:journalctl@ceph.mds.k.smithi150.stdout:Mar 28 16:00:22 smithi150 podman[64594]: 2024-03-28 16:00:22.431094648 +0000 UTC m=+0.192257873 container create 8b52dd296b4cdc7ec44a3b6050ca01fec8636133bc39934a4e0c9523dff8ffac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-k, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-28T16:00:22.686 INFO:journalctl@ceph.mds.k.smithi150.stdout:Mar 28 16:00:22 smithi150 podman[64594]: 2024-03-28 16:00:22.684785009 +0000 UTC m=+0.445948241 container init 8b52dd296b4cdc7ec44a3b6050ca01fec8636133bc39934a4e0c9523dff8ffac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-k, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T16:00:22.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:22.817+0000 7f97c7200640 1 -- 172.21.15.67:0/2198800405 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f97c81a2180 con 0x7f97c8103090 2024-03-28T16:00:22.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:22.818+0000 7f97bea00640 1 -- 172.21.15.67:0/2198800405 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1328 (secure 0 0 0) 0x7f97c00be1c0 con 0x7f97c8103090 2024-03-28T16:00:22.819 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:00:22.819 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2129665423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2129665423},{"type":"v1","addr":"172.21.15.183:6833","nonce":2129665423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-28T16:00:22.819 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 2 2024-03-28T16:00:22.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:22.821+0000 7f97c7200640 1 -- 172.21.15.67:0/2198800405 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f97ac0a2880 msgr2=0x7f97ac0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:22.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:22.821+0000 7f97c7200640 1 --2- 172.21.15.67:0/2198800405 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f97ac0a2880 0x7f97ac0a4d40 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f97b8005fd0 tx=0x7f97b8005850 comp rx=0 tx=0).stop 2024-03-28T16:00:22.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:22.821+0000 7f97c7200640 1 -- 172.21.15.67:0/2198800405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97c8103090 msgr2=0x7f97c81a0e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:22.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:22.821+0000 7f97c7200640 1 --2- 172.21.15.67:0/2198800405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97c8103090 0x7f97c81a0e70 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f97c0032520 tx=0x7f97c0034880 comp rx=0 tx=0).stop 2024-03-28T16:00:22.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:22.821+0000 7f97c7200640 1 -- 172.21.15.67:0/2198800405 shutdown_connections 2024-03-28T16:00:22.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:22.821+0000 7f97c7200640 1 --2- 172.21.15.67:0/2198800405 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f97ac0a2880 0x7f97ac0a4d40 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:22.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:22.821+0000 7f97c7200640 1 --2- 172.21.15.67:0/2198800405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97c8103090 0x7f97c81a0e70 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:22.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:22.821+0000 7f97c7200640 1 --2- 172.21.15.67:0/2198800405 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97c81026d0 0x7f97c8199ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:22.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:22.821+0000 7f97c7200640 1 --2- 172.21.15.67:0/2198800405 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f97c81014e0 0x7f97c8199960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:22.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:22.821+0000 7f97c7200640 1 -- 172.21.15.67:0/2198800405 >> 172.21.15.67:0/2198800405 conn(0x7f97c80fca30 msgr2=0x7f97c80738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:00:22.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:22.821+0000 7f97c7200640 1 -- 172.21.15.67:0/2198800405 shutdown_connections 2024-03-28T16:00:22.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:22.821+0000 7f97c7200640 1 -- 172.21.15.67:0/2198800405 wait complete. 2024-03-28T16:00:22.939 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:22 smithi150 ceph-mon[34825]: pgmap v337: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:22.939 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:22 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/2198800405' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:00:22.940 INFO:journalctl@ceph.mds.k.smithi150.stdout:Mar 28 16:00:22 smithi150 podman[64594]: 2024-03-28 16:00:22.695479636 +0000 UTC m=+0.456642824 container start 8b52dd296b4cdc7ec44a3b6050ca01fec8636133bc39934a4e0c9523dff8ffac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-28T16:00:22.940 INFO:journalctl@ceph.mds.k.smithi150.stdout:Mar 28 16:00:22 smithi150 ceph-mds[64614]: -- 172.21.15.150:0/3263014128 <== mon.2 v2:172.21.15.150:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ece1143a40 con 0x55ece1117000 2024-03-28T16:00:22.940 INFO:journalctl@ceph.mds.k.smithi150.stdout:Mar 28 16:00:22 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-k[64609]: starting mds.k at 2024-03-28T16:00:22.940 INFO:journalctl@ceph.mds.k.smithi150.stdout:Mar 28 16:00:22 smithi150 bash[64594]: 8b52dd296b4cdc7ec44a3b6050ca01fec8636133bc39934a4e0c9523dff8ffac 2024-03-28T16:00:22.940 INFO:journalctl@ceph.mds.k.smithi150.stdout:Mar 28 16:00:22 smithi150 systemd[1]: Started Ceph mds.k for b5f8225a-ed1a-11ee-95d0-87774f69a715. 2024-03-28T16:00:23.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:22 smithi183 ceph-mon[35580]: pgmap v337: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:23.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:22 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/2198800405' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:00:23.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:22 smithi067 ceph-mon[31707]: pgmap v337: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:23.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:22 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/2198800405' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:00:24.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:23 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:24.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:23 smithi183 ceph-mon[35580]: mds.? [v2:172.21.15.150:6832/1008335521,v1:172.21.15.150:6833/1008335521] up:boot 2024-03-28T16:00:24.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:23 smithi183 ceph-mon[35580]: fsmap 2 up:standby 2024-03-28T16:00:24.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:23 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T16:00:24.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:23 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:24.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:23 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:24.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:23 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T16:00:24.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:23 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T16:00:24.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:23 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:00:24.193 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:23 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:24.193 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:23 smithi067 ceph-mon[31707]: mds.? [v2:172.21.15.150:6832/1008335521,v1:172.21.15.150:6833/1008335521] up:boot 2024-03-28T16:00:24.193 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:23 smithi067 ceph-mon[31707]: fsmap 2 up:standby 2024-03-28T16:00:24.193 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:23 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T16:00:24.193 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:23 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:24.193 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:23 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:24.193 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:23 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T16:00:24.193 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:23 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T16:00:24.193 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:23 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:00:24.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:23 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:24.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:23 smithi150 ceph-mon[34825]: mds.? [v2:172.21.15.150:6832/1008335521,v1:172.21.15.150:6833/1008335521] up:boot 2024-03-28T16:00:24.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:23 smithi150 ceph-mon[34825]: fsmap 2 up:standby 2024-03-28T16:00:24.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:23 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T16:00:24.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:23 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:24.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:23 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:24.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:23 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T16:00:24.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:23 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T16:00:24.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:23 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:00:24.843 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T16:00:25.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:24 smithi067 ceph-mon[31707]: Deploying daemon mds.j on smithi067 2024-03-28T16:00:25.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:24 smithi067 ceph-mon[31707]: pgmap v338: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:25.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:25.079+0000 7f7f64a00640 1 -- 172.21.15.67:0/934330576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f60108cf0 msgr2=0x7f7f60109170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:25.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:25.079+0000 7f7f64a00640 1 --2- 172.21.15.67:0/934330576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f60108cf0 0x7f7f60109170 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f7f48009900 tx=0x7f7f4802bc30 comp rx=0 tx=0).stop 2024-03-28T16:00:25.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:25.079+0000 7f7f64a00640 1 -- 172.21.15.67:0/934330576 shutdown_connections 2024-03-28T16:00:25.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:25.079+0000 7f7f64a00640 1 --2- 172.21.15.67:0/934330576 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7f601096b0 0x7f7f6010ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:25.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:25.079+0000 7f7f64a00640 1 --2- 172.21.15.67:0/934330576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f60108cf0 0x7f7f60109170 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:25.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:25.079+0000 7f7f64a00640 1 --2- 172.21.15.67:0/934330576 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f60107b00 0x7f7f60107f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:25.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:25.079+0000 7f7f64a00640 1 -- 172.21.15.67:0/934330576 >> 172.21.15.67:0/934330576 conn(0x7f7f60076ab0 msgr2=0x7f7f60078ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:00:25.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:25.079+0000 7f7f64a00640 1 -- 172.21.15.67:0/934330576 shutdown_connections 2024-03-28T16:00:25.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:25.079+0000 7f7f64a00640 1 -- 172.21.15.67:0/934330576 wait complete. 2024-03-28T16:00:25.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:25.080+0000 7f7f64a00640 1 Processor -- start 2024-03-28T16:00:25.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:25.080+0000 7f7f64a00640 1 -- start start 2024-03-28T16:00:25.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:25.081+0000 7f7f64a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f60107b00 0x7f7f6019dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:25.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:25.081+0000 7f7f64a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f60108cf0 0x7f7f6019e250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:25.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:25.081+0000 7f7f64a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7f601096b0 0x7f7f601a5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:25.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:25.081+0000 7f7f64a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f60113680 con 0x7f7f60107b00 2024-03-28T16:00:25.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:25.081+0000 7f7f64a00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f60113500 con 0x7f7f60108cf0 2024-03-28T16:00:25.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:25.081+0000 7f7f64a00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f60113800 con 0x7f7f601096b0 2024-03-28T16:00:25.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:25.081+0000 7f7f5e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f60107b00 0x7f7f6019dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:25.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:25.081+0000 7f7f5e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f60107b00 0x7f7f6019dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60002/0 (socket says 172.21.15.67:60002) 2024-03-28T16:00:25.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:25.081+0000 7f7f5e800640 1 -- 172.21.15.67:0/771156568 learned_addr learned my addr 172.21.15.67:0/771156568 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:00:25.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:25.081+0000 7f7f5f200640 1 --2- 172.21.15.67:0/771156568 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7f601096b0 0x7f7f601a5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:25.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:25.081+0000 7f7f5e800640 1 -- 172.21.15.67:0/771156568 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7f601096b0 msgr2=0x7f7f601a5280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:25.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:25.081+0000 7f7f5e800640 1 --2- 172.21.15.67:0/771156568 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7f601096b0 0x7f7f601a5280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:25.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:25.081+0000 7f7f5e800640 1 -- 172.21.15.67:0/771156568 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f60108cf0 msgr2=0x7f7f6019e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:25.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:25.081+0000 7f7f5de00640 1 --2- 172.21.15.67:0/771156568 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f60108cf0 0x7f7f6019e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:25.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:25.081+0000 7f7f5e800640 1 --2- 172.21.15.67:0/771156568 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f60108cf0 0x7f7f6019e250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:25.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:25.081+0000 7f7f5e800640 1 -- 172.21.15.67:0/771156568 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f601a5920 con 0x7f7f60107b00 2024-03-28T16:00:25.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:25.082+0000 7f7f5e800640 1 --2- 172.21.15.67:0/771156568 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f60107b00 0x7f7f6019dd10 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7f5000ea80 tx=0x7f7f5000ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:00:25.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:25.082+0000 7f7f57400640 1 -- 172.21.15.67:0/771156568 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f5000ce40 con 0x7f7f60107b00 2024-03-28T16:00:25.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:25.082+0000 7f7f57400640 1 -- 172.21.15.67:0/771156568 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7f50011920 con 0x7f7f60107b00 2024-03-28T16:00:25.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:25.082+0000 7f7f64a00640 1 -- 172.21.15.67:0/771156568 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f601a5c10 con 0x7f7f60107b00 2024-03-28T16:00:25.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:25.082+0000 7f7f57400640 1 -- 172.21.15.67:0/771156568 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f5001aaa0 con 0x7f7f60107b00 2024-03-28T16:00:25.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:25.082+0000 7f7f64a00640 1 -- 172.21.15.67:0/771156568 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f601a6160 con 0x7f7f60107b00 2024-03-28T16:00:25.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:25.083+0000 7f7f64a00640 1 -- 172.21.15.67:0/771156568 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f6010ee20 con 0x7f7f60107b00 2024-03-28T16:00:25.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:25.085+0000 7f7f57400640 1 -- 172.21.15.67:0/771156568 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f7f50011ad0 con 0x7f7f60107b00 2024-03-28T16:00:25.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:25.087+0000 7f7f57400640 1 --2- 172.21.15.67:0/771156568 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7f440a2800 0x7f7f440a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:25.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:25.087+0000 7f7f57400640 1 -- 172.21.15.67:0/771156568 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f7f500ced70 con 0x7f7f60107b00 2024-03-28T16:00:25.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:25.087+0000 7f7f5de00640 1 --2- 172.21.15.67:0/771156568 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7f440a2800 0x7f7f440a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:25.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:25.088+0000 7f7f5de00640 1 --2- 172.21.15.67:0/771156568 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7f440a2800 0x7f7f440a4cc0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7f48005c40 tx=0x7f7f48005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:00:25.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:25.091+0000 7f7f57400640 1 -- 172.21.15.67:0/771156568 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7f50016030 con 0x7f7f60107b00 2024-03-28T16:00:25.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:24 smithi183 ceph-mon[35580]: Deploying daemon mds.j on smithi067 2024-03-28T16:00:25.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:24 smithi183 ceph-mon[35580]: pgmap v338: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:25.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:24 smithi150 ceph-mon[34825]: Deploying daemon mds.j on smithi067 2024-03-28T16:00:25.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:24 smithi150 ceph-mon[34825]: pgmap v338: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:25.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:25.344+0000 7f7f64a00640 1 -- 172.21.15.67:0/771156568 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7f6002d070 con 0x7f7f60107b00 2024-03-28T16:00:25.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:25.345+0000 7f7f57400640 1 -- 172.21.15.67:0/771156568 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2122 (secure 0 0 0) 0x7f7f5009c210 con 0x7f7f60107b00 2024-03-28T16:00:25.346 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:00:25.346 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2129665423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2129665423},{"type":"v1","addr":"172.21.15.183:6833","nonce":2129665423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":34409,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1008335521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1008335521},{"type":"v1","addr":"172.21.15.150:6833","nonce":1008335521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-28T16:00:25.346 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 3 2024-03-28T16:00:25.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:25.347+0000 7f7f64a00640 1 -- 172.21.15.67:0/771156568 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7f440a2800 msgr2=0x7f7f440a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:25.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:25.347+0000 7f7f64a00640 1 --2- 172.21.15.67:0/771156568 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7f440a2800 0x7f7f440a4cc0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7f48005c40 tx=0x7f7f48005fd0 comp rx=0 tx=0).stop 2024-03-28T16:00:25.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:25.347+0000 7f7f64a00640 1 -- 172.21.15.67:0/771156568 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f60107b00 msgr2=0x7f7f6019dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:25.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:25.347+0000 7f7f64a00640 1 --2- 172.21.15.67:0/771156568 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f60107b00 0x7f7f6019dd10 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7f5000ea80 tx=0x7f7f5000ef40 comp rx=0 tx=0).stop 2024-03-28T16:00:25.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:25.348+0000 7f7f64a00640 1 -- 172.21.15.67:0/771156568 shutdown_connections 2024-03-28T16:00:25.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:25.348+0000 7f7f64a00640 1 --2- 172.21.15.67:0/771156568 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7f440a2800 0x7f7f440a4cc0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:25.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:25.348+0000 7f7f64a00640 1 --2- 172.21.15.67:0/771156568 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7f601096b0 0x7f7f601a5280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:25.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:25.348+0000 7f7f64a00640 1 --2- 172.21.15.67:0/771156568 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f60108cf0 0x7f7f6019e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:25.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:25.348+0000 7f7f64a00640 1 --2- 172.21.15.67:0/771156568 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f60107b00 0x7f7f6019dd10 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:25.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:25.348+0000 7f7f64a00640 1 -- 172.21.15.67:0/771156568 >> 172.21.15.67:0/771156568 conn(0x7f7f60076ab0 msgr2=0x7f7f6010bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:00:25.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:25.348+0000 7f7f64a00640 1 -- 172.21.15.67:0/771156568 shutdown_connections 2024-03-28T16:00:25.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:25.348+0000 7f7f64a00640 1 -- 172.21.15.67:0/771156568 wait complete. 2024-03-28T16:00:26.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:25 smithi150 ceph-mon[34825]: pgmap v339: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:26.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:25 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/771156568' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:00:26.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:25 smithi067 ceph-mon[31707]: pgmap v339: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:26.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:25 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/771156568' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:00:26.321 INFO:journalctl@ceph.mds.j.smithi067.stdout:Mar 28 16:00:25 smithi067 systemd[1]: Starting Ceph mds.j for b5f8225a-ed1a-11ee-95d0-87774f69a715... 2024-03-28T16:00:26.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:25 smithi183 ceph-mon[35580]: pgmap v339: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:26.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:25 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/771156568' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:00:26.599 INFO:journalctl@ceph.mds.j.smithi067.stdout:Mar 28 16:00:26 smithi067 podman[72264]: 2024-03-28 16:00:26.251484079 +0000 UTC m=+0.037111093 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T16:00:26.600 INFO:journalctl@ceph.mds.j.smithi067.stdout:Mar 28 16:00:26 smithi067 podman[72264]: 2024-03-28 16:00:26.35939154 +0000 UTC m=+0.145018574 container create dd754e78743777780c556beef45db1f136a9c16c7d60e16424c5c73cfbf5799c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-j, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T16:00:26.600 INFO:journalctl@ceph.mds.j.smithi067.stdout:Mar 28 16:00:26 smithi067 podman[72264]: 2024-03-28 16:00:26.597087821 +0000 UTC m=+0.382714909 container init dd754e78743777780c556beef45db1f136a9c16c7d60e16424c5c73cfbf5799c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-j, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-28T16:00:27.070 INFO:journalctl@ceph.mds.j.smithi067.stdout:Mar 28 16:00:26 smithi067 podman[72264]: 2024-03-28 16:00:26.60820697 +0000 UTC m=+0.393833976 container start dd754e78743777780c556beef45db1f136a9c16c7d60e16424c5c73cfbf5799c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-j, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319) 2024-03-28T16:00:27.070 INFO:journalctl@ceph.mds.j.smithi067.stdout:Mar 28 16:00:26 smithi067 ceph-mds[72283]: -- 172.21.15.67:0/940834676 <== mon.0 v2:172.21.15.67:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x563545519a40 con 0x5635454ed000 2024-03-28T16:00:27.070 INFO:journalctl@ceph.mds.j.smithi067.stdout:Mar 28 16:00:26 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-j[72279]: starting mds.j at 2024-03-28T16:00:27.070 INFO:journalctl@ceph.mds.j.smithi067.stdout:Mar 28 16:00:26 smithi067 bash[72264]: dd754e78743777780c556beef45db1f136a9c16c7d60e16424c5c73cfbf5799c 2024-03-28T16:00:27.070 INFO:journalctl@ceph.mds.j.smithi067.stdout:Mar 28 16:00:26 smithi067 systemd[1]: Started Ceph mds.j for b5f8225a-ed1a-11ee-95d0-87774f69a715. 2024-03-28T16:00:27.373 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T16:00:27.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:27.587+0000 7ff11e200640 1 -- 172.21.15.67:0/75580370 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1181016c0 msgr2=0x7ff118101ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:27.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:27.587+0000 7ff11e200640 1 --2- 172.21.15.67:0/75580370 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1181016c0 0x7ff118101ae0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ff108009900 tx=0x7ff10802bc30 comp rx=0 tx=0).stop 2024-03-28T16:00:27.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:27.587+0000 7ff11e200640 1 -- 172.21.15.67:0/75580370 shutdown_connections 2024-03-28T16:00:27.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:27.587+0000 7ff11e200640 1 --2- 172.21.15.67:0/75580370 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff118103270 0x7ff118109b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:27.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:27.587+0000 7ff11e200640 1 --2- 172.21.15.67:0/75580370 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1181028b0 0x7ff118102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:27.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:27.587+0000 7ff11e200640 1 --2- 172.21.15.67:0/75580370 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1181016c0 0x7ff118101ae0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:27.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:27.587+0000 7ff11e200640 1 -- 172.21.15.67:0/75580370 >> 172.21.15.67:0/75580370 conn(0x7ff1180fcc50 msgr2=0x7ff1180ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:00:27.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:27.587+0000 7ff11e200640 1 -- 172.21.15.67:0/75580370 shutdown_connections 2024-03-28T16:00:27.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:27.587+0000 7ff11e200640 1 -- 172.21.15.67:0/75580370 wait complete. 2024-03-28T16:00:27.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:27.588+0000 7ff11e200640 1 Processor -- start 2024-03-28T16:00:27.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:27.588+0000 7ff11e200640 1 -- start start 2024-03-28T16:00:27.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:27.589+0000 7ff11e200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1181016c0 0x7ff118075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:27.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:27.589+0000 7ff11e200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1181028b0 0x7ff118073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:27.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:27.589+0000 7ff11e200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff118103270 0x7ff118073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:27.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:27.589+0000 7ff11e200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff11810d210 con 0x7ff1181016c0 2024-03-28T16:00:27.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:27.589+0000 7ff11e200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff11810d090 con 0x7ff1181028b0 2024-03-28T16:00:27.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:27.589+0000 7ff11e200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff11810d390 con 0x7ff118103270 2024-03-28T16:00:27.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:27.589+0000 7ff117400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1181016c0 0x7ff118075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:27.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:27.589+0000 7ff117400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1181016c0 0x7ff118075450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60050/0 (socket says 172.21.15.67:60050) 2024-03-28T16:00:27.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:27.589+0000 7ff117400640 1 -- 172.21.15.67:0/2153068483 learned_addr learned my addr 172.21.15.67:0/2153068483 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:00:27.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:27.589+0000 7ff116a00640 1 --2- 172.21.15.67:0/2153068483 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1181028b0 0x7ff118073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:27.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:27.589+0000 7ff117400640 1 -- 172.21.15.67:0/2153068483 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff118103270 msgr2=0x7ff118073fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:27.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:27.589+0000 7ff117400640 1 --2- 172.21.15.67:0/2153068483 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff118103270 0x7ff118073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:27.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:27.590+0000 7ff117400640 1 -- 172.21.15.67:0/2153068483 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1181028b0 msgr2=0x7ff118073a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:27.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:27.590+0000 7ff117400640 1 --2- 172.21.15.67:0/2153068483 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1181028b0 0x7ff118073a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:27.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:27.590+0000 7ff117400640 1 -- 172.21.15.67:0/2153068483 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff118074860 con 0x7ff1181016c0 2024-03-28T16:00:27.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:27.590+0000 7ff117400640 1 --2- 172.21.15.67:0/2153068483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1181016c0 0x7ff118075450 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff108005ab0 tx=0x7ff1080042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:00:27.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:27.590+0000 7ff10fe00640 1 -- 172.21.15.67:0/2153068483 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff108043020 con 0x7ff1181016c0 2024-03-28T16:00:27.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:27.590+0000 7ff10fe00640 1 -- 172.21.15.67:0/2153068483 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff1080054c0 con 0x7ff1181016c0 2024-03-28T16:00:27.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:27.590+0000 7ff11e200640 1 -- 172.21.15.67:0/2153068483 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1181a1a60 con 0x7ff1181016c0 2024-03-28T16:00:27.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:27.590+0000 7ff10fe00640 1 -- 172.21.15.67:0/2153068483 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff108003d10 con 0x7ff1181016c0 2024-03-28T16:00:27.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:27.590+0000 7ff11e200640 1 -- 172.21.15.67:0/2153068483 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1181a2000 con 0x7ff1181016c0 2024-03-28T16:00:27.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:27.592+0000 7ff11e200640 1 -- 172.21.15.67:0/2153068483 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff118108a00 con 0x7ff1181016c0 2024-03-28T16:00:27.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:27.594+0000 7ff10fe00640 1 -- 172.21.15.67:0/2153068483 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ff108046020 con 0x7ff1181016c0 2024-03-28T16:00:27.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:27.596+0000 7ff10fe00640 1 --2- 172.21.15.67:0/2153068483 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7ff0fc0a2880 0x7ff0fc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:27.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:27.597+0000 7ff10fe00640 1 -- 172.21.15.67:0/2153068483 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ff1080f0df0 con 0x7ff1181016c0 2024-03-28T16:00:27.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:27.597+0000 7ff116a00640 1 --2- 172.21.15.67:0/2153068483 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7ff0fc0a2880 0x7ff0fc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:27.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:27.597+0000 7ff116a00640 1 --2- 172.21.15.67:0/2153068483 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7ff0fc0a2880 0x7ff0fc0a4d40 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff104005fd0 tx=0x7ff104005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:00:27.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:27.599+0000 7ff10fe00640 1 -- 172.21.15.67:0/2153068483 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff10804d020 con 0x7ff1181016c0 2024-03-28T16:00:27.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:27.883+0000 7ff11e200640 1 -- 172.21.15.67:0/2153068483 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff11802d460 con 0x7ff1181016c0 2024-03-28T16:00:27.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:27.884+0000 7ff10fe00640 1 -- 172.21.15.67:0/2153068483 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2913 (secure 0 0 0) 0x7ff1080be290 con 0x7ff1181016c0 2024-03-28T16:00:27.886 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:00:27.886 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2008297659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2008297659},{"type":"v1","addr":"172.21.15.67:6835","nonce":2008297659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2129665423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2129665423},{"type":"v1","addr":"172.21.15.183:6833","nonce":2129665423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":34409,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1008335521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1008335521},{"type":"v1","addr":"172.21.15.150:6833","nonce":1008335521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-28T16:00:27.886 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 4 2024-03-28T16:00:27.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:27.887+0000 7ff11e200640 1 -- 172.21.15.67:0/2153068483 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7ff0fc0a2880 msgr2=0x7ff0fc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:27.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:27.887+0000 7ff11e200640 1 --2- 172.21.15.67:0/2153068483 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7ff0fc0a2880 0x7ff0fc0a4d40 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff104005fd0 tx=0x7ff104005850 comp rx=0 tx=0).stop 2024-03-28T16:00:27.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:27.887+0000 7ff11e200640 1 -- 172.21.15.67:0/2153068483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1181016c0 msgr2=0x7ff118075450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:27.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:27.887+0000 7ff11e200640 1 --2- 172.21.15.67:0/2153068483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1181016c0 0x7ff118075450 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff108005ab0 tx=0x7ff1080042f0 comp rx=0 tx=0).stop 2024-03-28T16:00:27.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:27.887+0000 7ff11e200640 1 -- 172.21.15.67:0/2153068483 shutdown_connections 2024-03-28T16:00:27.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:27.887+0000 7ff11e200640 1 --2- 172.21.15.67:0/2153068483 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7ff0fc0a2880 0x7ff0fc0a4d40 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:27.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:27.887+0000 7ff11e200640 1 --2- 172.21.15.67:0/2153068483 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff118103270 0x7ff118073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:27.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:27.887+0000 7ff11e200640 1 --2- 172.21.15.67:0/2153068483 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1181028b0 0x7ff118073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:27.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:27.887+0000 7ff11e200640 1 --2- 172.21.15.67:0/2153068483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1181016c0 0x7ff118075450 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:27.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:27.887+0000 7ff11e200640 1 -- 172.21.15.67:0/2153068483 >> 172.21.15.67:0/2153068483 conn(0x7ff1180fcc50 msgr2=0x7ff118105bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:00:27.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:27.887+0000 7ff11e200640 1 -- 172.21.15.67:0/2153068483 shutdown_connections 2024-03-28T16:00:27.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:27.887+0000 7ff11e200640 1 -- 172.21.15.67:0/2153068483 wait complete. 2024-03-28T16:00:28.063 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:27 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:28.063 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:27 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:28.063 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:27 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:28.063 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:27 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T16:00:28.064 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:27 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T16:00:28.064 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:27 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:00:28.064 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:27 smithi067 ceph-mon[31707]: Deploying daemon mds.d on smithi067 2024-03-28T16:00:28.064 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:27 smithi067 ceph-mon[31707]: mds.? [v2:172.21.15.67:6834/2008297659,v1:172.21.15.67:6835/2008297659] up:boot 2024-03-28T16:00:28.064 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:27 smithi067 ceph-mon[31707]: fsmap 3 up:standby 2024-03-28T16:00:28.064 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:27 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T16:00:28.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:27 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:28.102 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:27 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:28.102 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:27 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:28.102 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:27 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T16:00:28.102 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:27 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T16:00:28.102 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:27 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:00:28.102 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:27 smithi183 ceph-mon[35580]: Deploying daemon mds.d on smithi067 2024-03-28T16:00:28.103 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:27 smithi183 ceph-mon[35580]: mds.? [v2:172.21.15.67:6834/2008297659,v1:172.21.15.67:6835/2008297659] up:boot 2024-03-28T16:00:28.103 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:27 smithi183 ceph-mon[35580]: fsmap 3 up:standby 2024-03-28T16:00:28.103 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:27 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T16:00:28.304 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:27 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:28.305 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:27 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:28.305 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:27 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:28.305 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:27 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T16:00:28.305 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:27 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T16:00:28.305 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:27 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:00:28.306 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:27 smithi150 ceph-mon[34825]: Deploying daemon mds.d on smithi067 2024-03-28T16:00:28.306 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:27 smithi150 ceph-mon[34825]: mds.? [v2:172.21.15.67:6834/2008297659,v1:172.21.15.67:6835/2008297659] up:boot 2024-03-28T16:00:28.306 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:27 smithi150 ceph-mon[34825]: fsmap 3 up:standby 2024-03-28T16:00:28.306 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:27 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T16:00:28.921 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:28 smithi067 ceph-mon[31707]: pgmap v340: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:28.921 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:28 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/2153068483' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:00:29.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:28 smithi183 ceph-mon[35580]: pgmap v340: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:29.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:28 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/2153068483' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:00:29.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:28 smithi150 ceph-mon[34825]: pgmap v340: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:29.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:28 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/2153068483' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:00:29.912 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T16:00:30.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:30.132+0000 7f7f62200640 1 -- 172.21.15.67:0/3547547414 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f5c1038e0 msgr2=0x7f7f5c103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:30.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:30.132+0000 7f7f62200640 1 --2- 172.21.15.67:0/3547547414 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f5c1038e0 0x7f7f5c103d00 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c009ca0 tx=0x7f7f4c02c0f0 comp rx=0 tx=0).stop 2024-03-28T16:00:30.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:30.132+0000 7f7f62200640 1 -- 172.21.15.67:0/3547547414 shutdown_connections 2024-03-28T16:00:30.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:30.132+0000 7f7f62200640 1 --2- 172.21.15.67:0/3547547414 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7f5c105490 0x7f7f5c10bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:30.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:30.132+0000 7f7f62200640 1 --2- 172.21.15.67:0/3547547414 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f5c104ad0 0x7f7f5c104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:30.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:30.132+0000 7f7f62200640 1 --2- 172.21.15.67:0/3547547414 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f5c1038e0 0x7f7f5c103d00 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:30.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:30.132+0000 7f7f62200640 1 -- 172.21.15.67:0/3547547414 >> 172.21.15.67:0/3547547414 conn(0x7f7f5c0fee90 msgr2=0x7f7f5c1012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:00:30.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:30.132+0000 7f7f62200640 1 -- 172.21.15.67:0/3547547414 shutdown_connections 2024-03-28T16:00:30.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:30.132+0000 7f7f62200640 1 -- 172.21.15.67:0/3547547414 wait complete. 2024-03-28T16:00:30.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:30.133+0000 7f7f62200640 1 Processor -- start 2024-03-28T16:00:30.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:30.133+0000 7f7f62200640 1 -- start start 2024-03-28T16:00:30.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:30.133+0000 7f7f62200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f5c1038e0 0x7f7f5c199b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:30.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:30.133+0000 7f7f62200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7f5c104ad0 0x7f7f5c19a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:30.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:30.134+0000 7f7f62200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f5c105490 0x7f7f5c1a10c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:30.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:30.134+0000 7f7f62200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f5c10f410 con 0x7f7f5c1038e0 2024-03-28T16:00:30.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:30.134+0000 7f7f62200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f5c10f290 con 0x7f7f5c105490 2024-03-28T16:00:30.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:30.134+0000 7f7f62200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f5c10f590 con 0x7f7f5c104ad0 2024-03-28T16:00:30.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:30.134+0000 7f7f5b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f5c1038e0 0x7f7f5c199b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:30.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:30.134+0000 7f7f5b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f5c1038e0 0x7f7f5c199b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41100/0 (socket says 172.21.15.67:41100) 2024-03-28T16:00:30.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:30.134+0000 7f7f5b400640 1 -- 172.21.15.67:0/3201492169 learned_addr learned my addr 172.21.15.67:0/3201492169 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:00:30.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:30.134+0000 7f7f5be00640 1 --2- 172.21.15.67:0/3201492169 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f5c105490 0x7f7f5c1a10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:30.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:30.134+0000 7f7f52a00640 1 --2- 172.21.15.67:0/3201492169 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7f5c104ad0 0x7f7f5c19a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:30.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:30.134+0000 7f7f5b400640 1 -- 172.21.15.67:0/3201492169 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7f5c104ad0 msgr2=0x7f7f5c19a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:30.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:30.134+0000 7f7f5b400640 1 --2- 172.21.15.67:0/3201492169 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7f5c104ad0 0x7f7f5c19a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:30.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:30.134+0000 7f7f5b400640 1 -- 172.21.15.67:0/3201492169 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f5c105490 msgr2=0x7f7f5c1a10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:30.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:30.134+0000 7f7f5b400640 1 --2- 172.21.15.67:0/3201492169 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f5c105490 0x7f7f5c1a10c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:30.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:30.134+0000 7f7f5b400640 1 -- 172.21.15.67:0/3201492169 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f5c1a1730 con 0x7f7f5c1038e0 2024-03-28T16:00:30.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:30.135+0000 7f7f5b400640 1 --2- 172.21.15.67:0/3201492169 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f5c1038e0 0x7f7f5c199b10 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c02c700 tx=0x7f7f4c0328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:00:30.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:30.135+0000 7f7f58c00640 1 -- 172.21.15.67:0/3201492169 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f4c041020 con 0x7f7f5c1038e0 2024-03-28T16:00:30.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:30.135+0000 7f7f58c00640 1 -- 172.21.15.67:0/3201492169 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7f4c003e20 con 0x7f7f5c1038e0 2024-03-28T16:00:30.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:30.135+0000 7f7f58c00640 1 -- 172.21.15.67:0/3201492169 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f4c039040 con 0x7f7f5c1038e0 2024-03-28T16:00:30.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:30.135+0000 7f7f62200640 1 -- 172.21.15.67:0/3201492169 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f5c1a19c0 con 0x7f7f5c1038e0 2024-03-28T16:00:30.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:30.135+0000 7f7f62200640 1 -- 172.21.15.67:0/3201492169 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f5c1a1f00 con 0x7f7f5c1038e0 2024-03-28T16:00:30.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:30.137+0000 7f7f62200640 1 -- 172.21.15.67:0/3201492169 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f5c10acd0 con 0x7f7f5c1038e0 2024-03-28T16:00:30.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:30.139+0000 7f7f58c00640 1 -- 172.21.15.67:0/3201492169 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f7f4c044020 con 0x7f7f5c1038e0 2024-03-28T16:00:30.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:30.140+0000 7f7f58c00640 1 --2- 172.21.15.67:0/3201492169 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7f400a2880 0x7f7f400a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:30.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:30.140+0000 7f7f58c00640 1 -- 172.21.15.67:0/3201492169 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f7f4c0f0980 con 0x7f7f5c1038e0 2024-03-28T16:00:30.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:30.141+0000 7f7f52a00640 1 --2- 172.21.15.67:0/3201492169 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7f400a2880 0x7f7f400a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:30.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:30.141+0000 7f7f52a00640 1 --2- 172.21.15.67:0/3201492169 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7f400a2880 0x7f7f400a4d40 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7f44005fd0 tx=0x7f7f44005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:00:30.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:30.145+0000 7f7f58c00640 1 -- 172.21.15.67:0/3201492169 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7f4c04b020 con 0x7f7f5c1038e0 2024-03-28T16:00:30.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:30.420+0000 7f7f62200640 1 -- 172.21.15.67:0/3201492169 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7f5c02d070 con 0x7f7f5c1038e0 2024-03-28T16:00:30.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:30.421+0000 7f7f58c00640 1 -- 172.21.15.67:0/3201492169 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2913 (secure 0 0 0) 0x7f7f4c0bde20 con 0x7f7f5c1038e0 2024-03-28T16:00:30.421 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:00:30.421 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2008297659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2008297659},{"type":"v1","addr":"172.21.15.67:6835","nonce":2008297659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2129665423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2129665423},{"type":"v1","addr":"172.21.15.183:6833","nonce":2129665423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":34409,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1008335521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1008335521},{"type":"v1","addr":"172.21.15.150:6833","nonce":1008335521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-28T16:00:30.422 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 4 2024-03-28T16:00:30.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:30.424+0000 7f7f62200640 1 -- 172.21.15.67:0/3201492169 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7f400a2880 msgr2=0x7f7f400a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:30.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:30.424+0000 7f7f62200640 1 --2- 172.21.15.67:0/3201492169 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7f400a2880 0x7f7f400a4d40 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7f44005fd0 tx=0x7f7f44005850 comp rx=0 tx=0).stop 2024-03-28T16:00:30.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:30.424+0000 7f7f62200640 1 -- 172.21.15.67:0/3201492169 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f5c1038e0 msgr2=0x7f7f5c199b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:30.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:30.424+0000 7f7f62200640 1 --2- 172.21.15.67:0/3201492169 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f5c1038e0 0x7f7f5c199b10 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c02c700 tx=0x7f7f4c0328a0 comp rx=0 tx=0).stop 2024-03-28T16:00:30.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:30.425+0000 7f7f62200640 1 -- 172.21.15.67:0/3201492169 shutdown_connections 2024-03-28T16:00:30.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:30.425+0000 7f7f62200640 1 --2- 172.21.15.67:0/3201492169 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7f400a2880 0x7f7f400a4d40 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:30.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:30.425+0000 7f7f62200640 1 --2- 172.21.15.67:0/3201492169 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f5c105490 0x7f7f5c1a10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:30.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:30.425+0000 7f7f62200640 1 --2- 172.21.15.67:0/3201492169 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7f5c104ad0 0x7f7f5c19a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:30.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:30.425+0000 7f7f62200640 1 --2- 172.21.15.67:0/3201492169 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f5c1038e0 0x7f7f5c199b10 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:30.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:30.425+0000 7f7f62200640 1 -- 172.21.15.67:0/3201492169 >> 172.21.15.67:0/3201492169 conn(0x7f7f5c0fee90 msgr2=0x7f7f5c107ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:00:30.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:30.425+0000 7f7f62200640 1 -- 172.21.15.67:0/3201492169 shutdown_connections 2024-03-28T16:00:30.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:30.425+0000 7f7f62200640 1 -- 172.21.15.67:0/3201492169 wait complete. 2024-03-28T16:00:30.554 INFO:journalctl@ceph.mds.d.smithi067.stdout:Mar 28 16:00:30 smithi067 podman[72644]: 2024-03-28 16:00:30.542499221 +0000 UTC m=+0.384503053 container init 92b9cad2c191477db5fbece171102b6ea0d7bef324555141fb330efa45681363 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-d, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-28T16:00:30.555 INFO:journalctl@ceph.mds.d.smithi067.stdout:Mar 28 16:00:30 smithi067 podman[72644]: 2024-03-28 16:00:30.553563353 +0000 UTC m=+0.395567082 container start 92b9cad2c191477db5fbece171102b6ea0d7bef324555141fb330efa45681363 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-d, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T16:00:30.820 INFO:journalctl@ceph.mds.d.smithi067.stdout:Mar 28 16:00:30 smithi067 ceph-mds[72665]: -- 172.21.15.67:0/1078075222 <== mon.0 v2:172.21.15.67:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556d9603fa40 con 0x556d96013000 2024-03-28T16:00:30.820 INFO:journalctl@ceph.mds.d.smithi067.stdout:Mar 28 16:00:30 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-d[72659]: starting mds.d at 2024-03-28T16:00:30.820 INFO:journalctl@ceph.mds.d.smithi067.stdout:Mar 28 16:00:30 smithi067 bash[72644]: 92b9cad2c191477db5fbece171102b6ea0d7bef324555141fb330efa45681363 2024-03-28T16:00:30.820 INFO:journalctl@ceph.mds.d.smithi067.stdout:Mar 28 16:00:30 smithi067 systemd[1]: Started Ceph mds.d for b5f8225a-ed1a-11ee-95d0-87774f69a715. 2024-03-28T16:00:31.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:30 smithi183 ceph-mon[35580]: pgmap v341: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:31.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:30 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/3201492169' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:00:31.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:30 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:31.105 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:30 smithi150 ceph-mon[34825]: pgmap v341: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:31.105 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:30 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/3201492169' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:00:31.105 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:30 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:31.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:30 smithi067 ceph-mon[31707]: pgmap v341: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:31.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:30 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/3201492169' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:00:31.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:30 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:32.088 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:31 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:32.088 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:31 smithi150 ceph-mon[34825]: mds.? [v2:172.21.15.67:6836/320295745,v1:172.21.15.67:6837/320295745] up:boot 2024-03-28T16:00:32.088 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:31 smithi150 ceph-mon[34825]: fsmap 4 up:standby 2024-03-28T16:00:32.088 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:31 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T16:00:32.088 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:31 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:32.088 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:31 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T16:00:32.088 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:31 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T16:00:32.088 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:31 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:00:32.088 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:31 smithi150 ceph-mon[34825]: Deploying daemon mds.e on smithi150 2024-03-28T16:00:32.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:31 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:32.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:31 smithi183 ceph-mon[35580]: mds.? [v2:172.21.15.67:6836/320295745,v1:172.21.15.67:6837/320295745] up:boot 2024-03-28T16:00:32.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:31 smithi183 ceph-mon[35580]: fsmap 4 up:standby 2024-03-28T16:00:32.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:31 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T16:00:32.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:31 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:32.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:31 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T16:00:32.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:31 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T16:00:32.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:31 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:00:32.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:31 smithi183 ceph-mon[35580]: Deploying daemon mds.e on smithi150 2024-03-28T16:00:32.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:31 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:32.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:31 smithi067 ceph-mon[31707]: mds.? [v2:172.21.15.67:6836/320295745,v1:172.21.15.67:6837/320295745] up:boot 2024-03-28T16:00:32.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:31 smithi067 ceph-mon[31707]: fsmap 4 up:standby 2024-03-28T16:00:32.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:31 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T16:00:32.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:31 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:32.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:31 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T16:00:32.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:31 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T16:00:32.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:31 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:00:32.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:31 smithi067 ceph-mon[31707]: Deploying daemon mds.e on smithi150 2024-03-28T16:00:32.444 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T16:00:32.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:32.673+0000 7fce79600640 1 -- 172.21.15.67:0/2716783912 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce74109890 msgr2=0x7fce74110120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:32.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:32.673+0000 7fce79600640 1 --2- 172.21.15.67:0/2716783912 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce74109890 0x7fce74110120 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fce6c009ca0 tx=0x7fce6c02c0f0 comp rx=0 tx=0).stop 2024-03-28T16:00:32.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:32.673+0000 7fce79600640 1 -- 172.21.15.67:0/2716783912 shutdown_connections 2024-03-28T16:00:32.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:32.673+0000 7fce79600640 1 --2- 172.21.15.67:0/2716783912 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce74109890 0x7fce74110120 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:32.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:32.673+0000 7fce79600640 1 --2- 172.21.15.67:0/2716783912 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce74108ed0 0x7fce74109350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:32.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:32.673+0000 7fce79600640 1 --2- 172.21.15.67:0/2716783912 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fce74107ce0 0x7fce74108100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:32.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:32.673+0000 7fce79600640 1 -- 172.21.15.67:0/2716783912 >> 172.21.15.67:0/2716783912 conn(0x7fce74076cb0 msgr2=0x7fce740790d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:00:32.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:32.673+0000 7fce79600640 1 -- 172.21.15.67:0/2716783912 shutdown_connections 2024-03-28T16:00:32.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:32.673+0000 7fce79600640 1 -- 172.21.15.67:0/2716783912 wait complete. 2024-03-28T16:00:32.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:32.674+0000 7fce79600640 1 Processor -- start 2024-03-28T16:00:32.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:32.674+0000 7fce79600640 1 -- start start 2024-03-28T16:00:32.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:32.674+0000 7fce79600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fce74107ce0 0x7fce7419dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:32.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:32.674+0000 7fce79600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce74108ed0 0x7fce7419e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:32.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:32.674+0000 7fce79600640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce74109890 0x7fce741a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:32.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:32.674+0000 7fce79600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce741137f0 con 0x7fce74108ed0 2024-03-28T16:00:32.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:32.674+0000 7fce79600640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce74113670 con 0x7fce74109890 2024-03-28T16:00:32.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:32.674+0000 7fce79600640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce74113970 con 0x7fce74107ce0 2024-03-28T16:00:32.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:32.675+0000 7fce72000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce74108ed0 0x7fce7419e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:32.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:32.675+0000 7fce72000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce74108ed0 0x7fce7419e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41128/0 (socket says 172.21.15.67:41128) 2024-03-28T16:00:32.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:32.675+0000 7fce72000640 1 -- 172.21.15.67:0/4067821147 learned_addr learned my addr 172.21.15.67:0/4067821147 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:00:32.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:32.675+0000 7fce72a00640 1 --2- 172.21.15.67:0/4067821147 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fce74107ce0 0x7fce7419dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:32.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:32.675+0000 7fce73400640 1 --2- 172.21.15.67:0/4067821147 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce74109890 0x7fce741a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:32.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:32.675+0000 7fce72000640 1 -- 172.21.15.67:0/4067821147 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fce74107ce0 msgr2=0x7fce7419dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:32.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:32.675+0000 7fce72000640 1 --2- 172.21.15.67:0/4067821147 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fce74107ce0 0x7fce7419dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:32.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:32.675+0000 7fce72000640 1 -- 172.21.15.67:0/4067821147 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce74109890 msgr2=0x7fce741a5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:32.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:32.675+0000 7fce72000640 1 --2- 172.21.15.67:0/4067821147 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce74109890 0x7fce741a5470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:32.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:32.675+0000 7fce72000640 1 -- 172.21.15.67:0/4067821147 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce741a5ae0 con 0x7fce74108ed0 2024-03-28T16:00:32.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:32.675+0000 7fce73400640 1 --2- 172.21.15.67:0/4067821147 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce74109890 0x7fce741a5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:00:32.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:32.676+0000 7fce72000640 1 --2- 172.21.15.67:0/4067821147 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce74108ed0 0x7fce7419e400 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fce6000e9f0 tx=0x7fce6000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:00:32.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:32.676+0000 7fce6b400640 1 -- 172.21.15.67:0/4067821147 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce6000cd80 con 0x7fce74108ed0 2024-03-28T16:00:32.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:32.676+0000 7fce6b400640 1 -- 172.21.15.67:0/4067821147 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fce600108e0 con 0x7fce74108ed0 2024-03-28T16:00:32.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:32.676+0000 7fce6b400640 1 -- 172.21.15.67:0/4067821147 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce60018920 con 0x7fce74108ed0 2024-03-28T16:00:32.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:32.676+0000 7fce79600640 1 -- 172.21.15.67:0/4067821147 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce741a5dd0 con 0x7fce74108ed0 2024-03-28T16:00:32.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:32.676+0000 7fce79600640 1 -- 172.21.15.67:0/4067821147 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce741a62e0 con 0x7fce74108ed0 2024-03-28T16:00:32.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:32.677+0000 7fce79600640 1 -- 172.21.15.67:0/4067821147 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce7410f0d0 con 0x7fce74108ed0 2024-03-28T16:00:32.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:32.680+0000 7fce6b400640 1 -- 172.21.15.67:0/4067821147 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fce60010a90 con 0x7fce74108ed0 2024-03-28T16:00:32.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:32.681+0000 7fce6b400640 1 --2- 172.21.15.67:0/4067821147 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fce580a2880 0x7fce580a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:32.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:32.681+0000 7fce6b400640 1 -- 172.21.15.67:0/4067821147 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fce600cea30 con 0x7fce74108ed0 2024-03-28T16:00:32.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:32.681+0000 7fce72a00640 1 --2- 172.21.15.67:0/4067821147 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fce580a2880 0x7fce580a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:32.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:32.682+0000 7fce72a00640 1 --2- 172.21.15.67:0/4067821147 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fce580a2880 0x7fce580a4d40 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fce64005fd0 tx=0x7fce64005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:00:32.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:32.686+0000 7fce6b400640 1 -- 172.21.15.67:0/4067821147 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fce6009bed0 con 0x7fce74108ed0 2024-03-28T16:00:32.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:32.934+0000 7fce79600640 1 -- 172.21.15.67:0/4067821147 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fce74062340 con 0x7fce74108ed0 2024-03-28T16:00:32.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:32.935+0000 7fce6b400640 1 -- 172.21.15.67:0/4067821147 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+3701 (secure 0 0 0) 0x7fce6009b7d0 con 0x7fce74108ed0 2024-03-28T16:00:32.935 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:00:32.935 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2008297659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2008297659},{"type":"v1","addr":"172.21.15.67:6835","nonce":2008297659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14532,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/320295745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":320295745},{"type":"v1","addr":"172.21.15.67:6837","nonce":320295745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2129665423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2129665423},{"type":"v1","addr":"172.21.15.183:6833","nonce":2129665423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":34409,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1008335521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1008335521},{"type":"v1","addr":"172.21.15.150:6833","nonce":1008335521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-28T16:00:32.936 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 5 2024-03-28T16:00:32.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:32.938+0000 7fce79600640 1 -- 172.21.15.67:0/4067821147 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fce580a2880 msgr2=0x7fce580a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:32.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:32.938+0000 7fce79600640 1 --2- 172.21.15.67:0/4067821147 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fce580a2880 0x7fce580a4d40 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fce64005fd0 tx=0x7fce64005850 comp rx=0 tx=0).stop 2024-03-28T16:00:32.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:32.938+0000 7fce79600640 1 -- 172.21.15.67:0/4067821147 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce74108ed0 msgr2=0x7fce7419e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:32.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:32.938+0000 7fce79600640 1 --2- 172.21.15.67:0/4067821147 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce74108ed0 0x7fce7419e400 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fce6000e9f0 tx=0x7fce6000eeb0 comp rx=0 tx=0).stop 2024-03-28T16:00:32.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:32.938+0000 7fce79600640 1 -- 172.21.15.67:0/4067821147 shutdown_connections 2024-03-28T16:00:32.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:32.938+0000 7fce79600640 1 --2- 172.21.15.67:0/4067821147 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fce580a2880 0x7fce580a4d40 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:32.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:32.938+0000 7fce79600640 1 --2- 172.21.15.67:0/4067821147 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce74109890 0x7fce741a5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:32.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:32.938+0000 7fce79600640 1 --2- 172.21.15.67:0/4067821147 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce74108ed0 0x7fce7419e400 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:32.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:32.938+0000 7fce79600640 1 --2- 172.21.15.67:0/4067821147 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fce74107ce0 0x7fce7419dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:32.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:32.938+0000 7fce79600640 1 -- 172.21.15.67:0/4067821147 >> 172.21.15.67:0/4067821147 conn(0x7fce74076cb0 msgr2=0x7fce7410c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:00:32.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:32.938+0000 7fce79600640 1 -- 172.21.15.67:0/4067821147 shutdown_connections 2024-03-28T16:00:32.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:32.938+0000 7fce79600640 1 -- 172.21.15.67:0/4067821147 wait complete. 2024-03-28T16:00:32.947 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:32 smithi150 ceph-mon[34825]: pgmap v342: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:33.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:32 smithi067 ceph-mon[31707]: pgmap v342: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:33.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:32 smithi183 ceph-mon[35580]: pgmap v342: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:33.980 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:33 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/4067821147' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:00:34.195 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:33 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/4067821147' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:00:34.303 INFO:journalctl@ceph.mds.e.smithi150.stdout:Mar 28 16:00:33 smithi150 systemd[1]: Starting Ceph mds.e for b5f8225a-ed1a-11ee-95d0-87774f69a715... 2024-03-28T16:00:34.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:33 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/4067821147' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:00:34.620 INFO:journalctl@ceph.mds.e.smithi150.stdout:Mar 28 16:00:34 smithi150 podman[64909]: 2024-03-28 16:00:34.328051203 +0000 UTC m=+0.119230709 container create d7898e35df8d35841269c2dae298d516b6034cedba45e277904e7d4ef5985454 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-e, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-28T16:00:34.620 INFO:journalctl@ceph.mds.e.smithi150.stdout:Mar 28 16:00:34 smithi150 podman[64909]: 2024-03-28 16:00:34.240152478 +0000 UTC m=+0.031331969 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T16:00:34.620 INFO:journalctl@ceph.mds.e.smithi150.stdout:Mar 28 16:00:34 smithi150 podman[64909]: 2024-03-28 16:00:34.61743134 +0000 UTC m=+0.408610862 container init d7898e35df8d35841269c2dae298d516b6034cedba45e277904e7d4ef5985454 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-e, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-28T16:00:34.907 INFO:journalctl@ceph.mds.e.smithi150.stdout:Mar 28 16:00:34 smithi150 podman[64909]: 2024-03-28 16:00:34.627017176 +0000 UTC m=+0.418196664 container start d7898e35df8d35841269c2dae298d516b6034cedba45e277904e7d4ef5985454 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-e, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-28T16:00:34.908 INFO:journalctl@ceph.mds.e.smithi150.stdout:Mar 28 16:00:34 smithi150 ceph-mds[64928]: -- 172.21.15.150:0/530203251 <== mon.2 v2:172.21.15.150:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x562f64c6fa40 con 0x562f64c42800 2024-03-28T16:00:34.908 INFO:journalctl@ceph.mds.e.smithi150.stdout:Mar 28 16:00:34 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-e[64924]: starting mds.e at 2024-03-28T16:00:34.908 INFO:journalctl@ceph.mds.e.smithi150.stdout:Mar 28 16:00:34 smithi150 bash[64909]: d7898e35df8d35841269c2dae298d516b6034cedba45e277904e7d4ef5985454 2024-03-28T16:00:34.908 INFO:journalctl@ceph.mds.e.smithi150.stdout:Mar 28 16:00:34 smithi150 systemd[1]: Started Ceph mds.e for b5f8225a-ed1a-11ee-95d0-87774f69a715. 2024-03-28T16:00:34.959 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T16:00:35.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:35.190+0000 7f6677e00640 1 -- 172.21.15.67:0/768041267 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66781052b0 msgr2=0x7f667810bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:35.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:35.190+0000 7f6677e00640 1 --2- 172.21.15.67:0/768041267 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66781052b0 0x7f667810bb40 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6670009900 tx=0x7f667002bc30 comp rx=0 tx=0).stop 2024-03-28T16:00:35.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:35.190+0000 7f6677e00640 1 -- 172.21.15.67:0/768041267 shutdown_connections 2024-03-28T16:00:35.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:35.190+0000 7f6677e00640 1 --2- 172.21.15.67:0/768041267 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66781052b0 0x7f667810bb40 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:35.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:35.190+0000 7f6677e00640 1 --2- 172.21.15.67:0/768041267 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f66781048f0 0x7f6678104d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:35.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:35.190+0000 7f6677e00640 1 --2- 172.21.15.67:0/768041267 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6678103700 0x7f6678103b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:35.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:35.190+0000 7f6677e00640 1 -- 172.21.15.67:0/768041267 >> 172.21.15.67:0/768041267 conn(0x7f66780fec70 msgr2=0x7f66781010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:00:35.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:35.190+0000 7f6677e00640 1 -- 172.21.15.67:0/768041267 shutdown_connections 2024-03-28T16:00:35.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:35.190+0000 7f6677e00640 1 -- 172.21.15.67:0/768041267 wait complete. 2024-03-28T16:00:35.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:35.191+0000 7f6677e00640 1 Processor -- start 2024-03-28T16:00:35.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:35.191+0000 7f6677e00640 1 -- start start 2024-03-28T16:00:35.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:35.192+0000 7f6677e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6678103700 0x7f66781998b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:35.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:35.192+0000 7f6677e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f66781048f0 0x7f6678199df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:35.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:35.192+0000 7f6677e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66781052b0 0x7f66781a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:35.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:35.192+0000 7f6677e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f667810f260 con 0x7f66781052b0 2024-03-28T16:00:35.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:35.192+0000 7f6677e00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f667810f0e0 con 0x7f6678103700 2024-03-28T16:00:35.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:35.192+0000 7f6677e00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f667810f3e0 con 0x7f66781048f0 2024-03-28T16:00:35.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:35.192+0000 7f6677400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66781052b0 0x7f66781a0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:35.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:35.192+0000 7f6676000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f66781048f0 0x7f6678199df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:35.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:35.192+0000 7f6677400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66781052b0 0x7f66781a0e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41154/0 (socket says 172.21.15.67:41154) 2024-03-28T16:00:35.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:35.192+0000 7f6677400640 1 -- 172.21.15.67:0/2550668019 learned_addr learned my addr 172.21.15.67:0/2550668019 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:00:35.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:35.192+0000 7f6676a00640 1 --2- 172.21.15.67:0/2550668019 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6678103700 0x7f66781998b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:35.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:35.193+0000 7f6677400640 1 -- 172.21.15.67:0/2550668019 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f66781048f0 msgr2=0x7f6678199df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:35.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:35.193+0000 7f6677400640 1 --2- 172.21.15.67:0/2550668019 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f66781048f0 0x7f6678199df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:35.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:35.193+0000 7f6677400640 1 -- 172.21.15.67:0/2550668019 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6678103700 msgr2=0x7f66781998b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:35.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:35.193+0000 7f6677400640 1 --2- 172.21.15.67:0/2550668019 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6678103700 0x7f66781998b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:35.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:35.193+0000 7f6677400640 1 -- 172.21.15.67:0/2550668019 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66781a14d0 con 0x7f66781052b0 2024-03-28T16:00:35.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:35.193+0000 7f6677400640 1 --2- 172.21.15.67:0/2550668019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66781052b0 0x7f66781a0e60 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f6670036040 tx=0x7f6670004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:00:35.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:35.193+0000 7f666f400640 1 -- 172.21.15.67:0/2550668019 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6670043020 con 0x7f66781052b0 2024-03-28T16:00:35.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:35.193+0000 7f6677e00640 1 -- 172.21.15.67:0/2550668019 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66781a1760 con 0x7f66781052b0 2024-03-28T16:00:35.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:35.193+0000 7f666f400640 1 -- 172.21.15.67:0/2550668019 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6670004490 con 0x7f66781052b0 2024-03-28T16:00:35.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:35.193+0000 7f666f400640 1 -- 172.21.15.67:0/2550668019 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6670003d10 con 0x7f66781052b0 2024-03-28T16:00:35.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:35.194+0000 7f6677e00640 1 -- 172.21.15.67:0/2550668019 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66781a1cb0 con 0x7f66781052b0 2024-03-28T16:00:35.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:35.195+0000 7f6677e00640 1 -- 172.21.15.67:0/2550668019 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f667810aa20 con 0x7f66781052b0 2024-03-28T16:00:35.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:35.197+0000 7f666f400640 1 -- 172.21.15.67:0/2550668019 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f6670046020 con 0x7f66781052b0 2024-03-28T16:00:35.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:35.198+0000 7f666f400640 1 --2- 172.21.15.67:0/2550668019 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f66580a2800 0x7f66580a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:35.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:35.198+0000 7f666f400640 1 -- 172.21.15.67:0/2550668019 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f66700f0a50 con 0x7f66781052b0 2024-03-28T16:00:35.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:35.198+0000 7f6676a00640 1 --2- 172.21.15.67:0/2550668019 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f66580a2800 0x7f66580a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:35.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:35.199+0000 7f6676a00640 1 --2- 172.21.15.67:0/2550668019 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f66580a2800 0x7f66580a4cc0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f6668005fd0 tx=0x7f6668005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:00:35.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:35.203+0000 7f666f400640 1 -- 172.21.15.67:0/2550668019 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f667004d020 con 0x7f66781052b0 2024-03-28T16:00:35.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:34 smithi150 ceph-mon[34825]: pgmap v343: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:35.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:34 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:35.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:34 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:35.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:34 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:35.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:34 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T16:00:35.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:34 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T16:00:35.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:34 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:00:35.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:34 smithi067 ceph-mon[31707]: pgmap v343: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:35.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:34 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:35.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:34 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:35.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:34 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:35.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:34 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T16:00:35.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:34 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T16:00:35.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:34 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:00:35.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:34 smithi183 ceph-mon[35580]: pgmap v343: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:35.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:34 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:35.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:34 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:35.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:34 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:35.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:34 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T16:00:35.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:34 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T16:00:35.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:34 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:00:35.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:35.447+0000 7f6677e00640 1 -- 172.21.15.67:0/2550668019 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f66780622a0 con 0x7f66781052b0 2024-03-28T16:00:35.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:35.448+0000 7f666f400640 1 -- 172.21.15.67:0/2550668019 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4495 (secure 0 0 0) 0x7f66700be010 con 0x7f66781052b0 2024-03-28T16:00:35.448 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:00:35.448 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2008297659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2008297659},{"type":"v1","addr":"172.21.15.67:6835","nonce":2008297659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14532,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/320295745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":320295745},{"type":"v1","addr":"172.21.15.67:6837","nonce":320295745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2129665423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2129665423},{"type":"v1","addr":"172.21.15.183:6833","nonce":2129665423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":34409,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1008335521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1008335521},{"type":"v1","addr":"172.21.15.150:6833","nonce":1008335521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":34445,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3269935695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3269935695},{"type":"v1","addr":"172.21.15.150:6835","nonce":3269935695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[]} 2024-03-28T16:00:35.449 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 6 2024-03-28T16:00:35.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:35.451+0000 7f6677e00640 1 -- 172.21.15.67:0/2550668019 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f66580a2800 msgr2=0x7f66580a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:35.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:35.451+0000 7f6677e00640 1 --2- 172.21.15.67:0/2550668019 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f66580a2800 0x7f66580a4cc0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f6668005fd0 tx=0x7f6668005850 comp rx=0 tx=0).stop 2024-03-28T16:00:35.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:35.451+0000 7f6677e00640 1 -- 172.21.15.67:0/2550668019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66781052b0 msgr2=0x7f66781a0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:35.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:35.451+0000 7f6677e00640 1 --2- 172.21.15.67:0/2550668019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66781052b0 0x7f66781a0e60 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f6670036040 tx=0x7f6670004270 comp rx=0 tx=0).stop 2024-03-28T16:00:35.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:35.451+0000 7f6677e00640 1 -- 172.21.15.67:0/2550668019 shutdown_connections 2024-03-28T16:00:35.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:35.451+0000 7f6677e00640 1 --2- 172.21.15.67:0/2550668019 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f66580a2800 0x7f66580a4cc0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:35.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:35.451+0000 7f6677e00640 1 --2- 172.21.15.67:0/2550668019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66781052b0 0x7f66781a0e60 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:35.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:35.451+0000 7f6677e00640 1 --2- 172.21.15.67:0/2550668019 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f66781048f0 0x7f6678199df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:35.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:35.451+0000 7f6677e00640 1 --2- 172.21.15.67:0/2550668019 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6678103700 0x7f66781998b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:35.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:35.451+0000 7f6677e00640 1 -- 172.21.15.67:0/2550668019 >> 172.21.15.67:0/2550668019 conn(0x7f66780fec70 msgr2=0x7f6678107bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:00:35.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:35.452+0000 7f6677e00640 1 -- 172.21.15.67:0/2550668019 shutdown_connections 2024-03-28T16:00:35.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:35.452+0000 7f6677e00640 1 -- 172.21.15.67:0/2550668019 wait complete. 2024-03-28T16:00:36.165 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:35 smithi183 ceph-mon[35580]: Deploying daemon mds.f on smithi183 2024-03-28T16:00:36.165 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:35 smithi183 ceph-mon[35580]: mds.? [v2:172.21.15.150:6834/3269935695,v1:172.21.15.150:6835/3269935695] up:boot 2024-03-28T16:00:36.165 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:35 smithi183 ceph-mon[35580]: fsmap 5 up:standby 2024-03-28T16:00:36.165 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:35 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T16:00:36.165 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:35 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/2550668019' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:00:36.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:35 smithi150 ceph-mon[34825]: Deploying daemon mds.f on smithi183 2024-03-28T16:00:36.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:35 smithi150 ceph-mon[34825]: mds.? [v2:172.21.15.150:6834/3269935695,v1:172.21.15.150:6835/3269935695] up:boot 2024-03-28T16:00:36.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:35 smithi150 ceph-mon[34825]: fsmap 5 up:standby 2024-03-28T16:00:36.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:35 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T16:00:36.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:35 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/2550668019' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:00:36.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:35 smithi067 ceph-mon[31707]: Deploying daemon mds.f on smithi183 2024-03-28T16:00:36.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:35 smithi067 ceph-mon[31707]: mds.? [v2:172.21.15.150:6834/3269935695,v1:172.21.15.150:6835/3269935695] up:boot 2024-03-28T16:00:36.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:35 smithi067 ceph-mon[31707]: fsmap 5 up:standby 2024-03-28T16:00:36.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:35 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T16:00:36.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:35 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/2550668019' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:00:37.021 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:36 smithi067 ceph-mon[31707]: pgmap v344: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:37.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:36 smithi150 ceph-mon[34825]: pgmap v344: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:37.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:36 smithi183 ceph-mon[35580]: pgmap v344: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:37.473 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T16:00:37.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:37.687+0000 7f7cbf000640 1 -- 172.21.15.67:0/3425739105 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cb80ffd60 msgr2=0x7f7cb8102190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:37.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:37.687+0000 7f7cbf000640 1 --2- 172.21.15.67:0/3425739105 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cb80ffd60 0x7f7cb8102190 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f7cac009900 tx=0x7f7cac02bc30 comp rx=0 tx=0).stop 2024-03-28T16:00:37.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:37.687+0000 7f7cbf000640 1 -- 172.21.15.67:0/3425739105 shutdown_connections 2024-03-28T16:00:37.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:37.687+0000 7f7cbf000640 1 --2- 172.21.15.67:0/3425739105 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7cb8105080 0x7f7cb8105500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:37.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:37.687+0000 7f7cbf000640 1 --2- 172.21.15.67:0/3425739105 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7cb81026d0 0x7f7cb8104b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:37.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:37.687+0000 7f7cbf000640 1 --2- 172.21.15.67:0/3425739105 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cb80ffd60 0x7f7cb8102190 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:37.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:37.687+0000 7f7cbf000640 1 -- 172.21.15.67:0/3425739105 >> 172.21.15.67:0/3425739105 conn(0x7f7cb80fb980 msgr2=0x7f7cb80fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:00:37.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:37.687+0000 7f7cbf000640 1 -- 172.21.15.67:0/3425739105 shutdown_connections 2024-03-28T16:00:37.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:37.688+0000 7f7cbf000640 1 -- 172.21.15.67:0/3425739105 wait complete. 2024-03-28T16:00:37.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:37.689+0000 7f7cbf000640 1 Processor -- start 2024-03-28T16:00:37.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:37.689+0000 7f7cbf000640 1 -- start start 2024-03-28T16:00:37.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:37.689+0000 7f7cbf000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cb80ffd60 0x7f7cb819dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:37.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:37.689+0000 7f7cbf000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7cb81026d0 0x7f7cb819e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:37.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:37.689+0000 7f7cbf000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7cb8105080 0x7f7cb81a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:37.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:37.689+0000 7f7cbf000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb8113660 con 0x7f7cb80ffd60 2024-03-28T16:00:37.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:37.689+0000 7f7cbf000640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb81134e0 con 0x7f7cb8105080 2024-03-28T16:00:37.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:37.689+0000 7f7cbf000640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb81137e0 con 0x7f7cb81026d0 2024-03-28T16:00:37.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:37.689+0000 7f7cbdc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cb80ffd60 0x7f7cb819dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:37.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:37.689+0000 7f7cbdc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cb80ffd60 0x7f7cb819dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41162/0 (socket says 172.21.15.67:41162) 2024-03-28T16:00:37.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:37.689+0000 7f7cbdc00640 1 -- 172.21.15.67:0/3604420576 learned_addr learned my addr 172.21.15.67:0/3604420576 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:00:37.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:37.690+0000 7f7cbd200640 1 --2- 172.21.15.67:0/3604420576 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7cb81026d0 0x7f7cb819e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:37.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:37.690+0000 7f7cbdc00640 1 -- 172.21.15.67:0/3604420576 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7cb81026d0 msgr2=0x7f7cb819e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:37.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:37.690+0000 7f7cbdc00640 1 --2- 172.21.15.67:0/3604420576 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7cb81026d0 0x7f7cb819e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:37.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:37.690+0000 7f7cbdc00640 1 -- 172.21.15.67:0/3604420576 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7cb8105080 msgr2=0x7f7cb81a5240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:37.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:37.690+0000 7f7cbdc00640 1 --2- 172.21.15.67:0/3604420576 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7cb8105080 0x7f7cb81a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:37.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:37.690+0000 7f7cbdc00640 1 -- 172.21.15.67:0/3604420576 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cb81a58e0 con 0x7f7cb80ffd60 2024-03-28T16:00:37.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:37.690+0000 7f7cbd200640 1 --2- 172.21.15.67:0/3604420576 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7cb81026d0 0x7f7cb819e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:00:37.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:37.690+0000 7f7cbdc00640 1 --2- 172.21.15.67:0/3604420576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cb80ffd60 0x7f7cb819dca0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7cac000c00 tx=0x7f7cac0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:00:37.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:37.690+0000 7f7cb6000640 1 -- 172.21.15.67:0/3604420576 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cac043020 con 0x7f7cb80ffd60 2024-03-28T16:00:37.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:37.690+0000 7f7cb6000640 1 -- 172.21.15.67:0/3604420576 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7cac0054c0 con 0x7f7cb80ffd60 2024-03-28T16:00:37.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:37.690+0000 7f7cbf000640 1 -- 172.21.15.67:0/3604420576 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cb81a5b70 con 0x7f7cb80ffd60 2024-03-28T16:00:37.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:37.690+0000 7f7cb6000640 1 -- 172.21.15.67:0/3604420576 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cac003d40 con 0x7f7cb80ffd60 2024-03-28T16:00:37.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:37.691+0000 7f7cbf000640 1 -- 172.21.15.67:0/3604420576 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cb81a6110 con 0x7f7cb80ffd60 2024-03-28T16:00:37.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:37.692+0000 7f7cbf000640 1 -- 172.21.15.67:0/3604420576 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cb80fd470 con 0x7f7cb80ffd60 2024-03-28T16:00:37.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:37.694+0000 7f7cb6000640 1 -- 172.21.15.67:0/3604420576 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f7cac046020 con 0x7f7cb80ffd60 2024-03-28T16:00:37.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:37.695+0000 7f7cb6000640 1 --2- 172.21.15.67:0/3604420576 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7ca00a2880 0x7f7ca00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:37.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:37.695+0000 7f7cb6000640 1 -- 172.21.15.67:0/3604420576 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f7cac0f0c90 con 0x7f7cb80ffd60 2024-03-28T16:00:37.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:37.695+0000 7f7cbd200640 1 --2- 172.21.15.67:0/3604420576 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7ca00a2880 0x7f7ca00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:37.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:37.696+0000 7f7cbd200640 1 --2- 172.21.15.67:0/3604420576 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7ca00a2880 0x7f7ca00a4d40 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f7cb819f1a0 tx=0x7f7ca4005d90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:00:37.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:37.700+0000 7f7cb6000640 1 -- 172.21.15.67:0/3604420576 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7cac04d020 con 0x7f7cb80ffd60 2024-03-28T16:00:37.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:37.967+0000 7f7cbf000640 1 -- 172.21.15.67:0/3604420576 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7cb802d070 con 0x7f7cb80ffd60 2024-03-28T16:00:37.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:37.969+0000 7f7cb6000640 1 -- 172.21.15.67:0/3604420576 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4495 (secure 0 0 0) 0x7f7cac0be250 con 0x7f7cb80ffd60 2024-03-28T16:00:37.969 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:00:37.969 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2008297659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2008297659},{"type":"v1","addr":"172.21.15.67:6835","nonce":2008297659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14532,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/320295745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":320295745},{"type":"v1","addr":"172.21.15.67:6837","nonce":320295745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2129665423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2129665423},{"type":"v1","addr":"172.21.15.183:6833","nonce":2129665423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":34409,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1008335521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1008335521},{"type":"v1","addr":"172.21.15.150:6833","nonce":1008335521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":34445,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3269935695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3269935695},{"type":"v1","addr":"172.21.15.150:6835","nonce":3269935695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[]} 2024-03-28T16:00:37.969 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 6 2024-03-28T16:00:37.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:37.971+0000 7f7cbf000640 1 -- 172.21.15.67:0/3604420576 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7ca00a2880 msgr2=0x7f7ca00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:37.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:37.971+0000 7f7cbf000640 1 --2- 172.21.15.67:0/3604420576 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7ca00a2880 0x7f7ca00a4d40 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f7cb819f1a0 tx=0x7f7ca4005d90 comp rx=0 tx=0).stop 2024-03-28T16:00:37.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:37.971+0000 7f7cbf000640 1 -- 172.21.15.67:0/3604420576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cb80ffd60 msgr2=0x7f7cb819dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:37.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:37.971+0000 7f7cbf000640 1 --2- 172.21.15.67:0/3604420576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cb80ffd60 0x7f7cb819dca0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7cac000c00 tx=0x7f7cac0042f0 comp rx=0 tx=0).stop 2024-03-28T16:00:37.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:37.972+0000 7f7cbf000640 1 -- 172.21.15.67:0/3604420576 shutdown_connections 2024-03-28T16:00:37.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:37.972+0000 7f7cbf000640 1 --2- 172.21.15.67:0/3604420576 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7ca00a2880 0x7f7ca00a4d40 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:37.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:37.972+0000 7f7cbf000640 1 --2- 172.21.15.67:0/3604420576 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7cb8105080 0x7f7cb81a5240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:37.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:37.972+0000 7f7cbf000640 1 --2- 172.21.15.67:0/3604420576 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7cb81026d0 0x7f7cb819e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:37.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:37.972+0000 7f7cbf000640 1 --2- 172.21.15.67:0/3604420576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cb80ffd60 0x7f7cb819dca0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:37.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:37.972+0000 7f7cbf000640 1 -- 172.21.15.67:0/3604420576 >> 172.21.15.67:0/3604420576 conn(0x7f7cb80fb980 msgr2=0x7f7cb8103390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:00:37.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:37.972+0000 7f7cbf000640 1 -- 172.21.15.67:0/3604420576 shutdown_connections 2024-03-28T16:00:37.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:37.972+0000 7f7cbf000640 1 -- 172.21.15.67:0/3604420576 wait complete. 2024-03-28T16:00:38.060 INFO:journalctl@ceph.mds.f.smithi183.stdout:Mar 28 16:00:37 smithi183 systemd[1]: Starting Ceph mds.f for b5f8225a-ed1a-11ee-95d0-87774f69a715... 2024-03-28T16:00:38.321 INFO:journalctl@ceph.mds.f.smithi183.stdout:Mar 28 16:00:38 smithi183 podman[65880]: 2024-03-28 16:00:38.058902102 +0000 UTC m=+0.124970389 container create fc616700c00247e3cb65f7b0b5c6bb9b9bae245cf7eeae84bd0accf3158a14b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-f, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-28T16:00:38.322 INFO:journalctl@ceph.mds.f.smithi183.stdout:Mar 28 16:00:38 smithi183 podman[65880]: 2024-03-28 16:00:37.968410814 +0000 UTC m=+0.034479079 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T16:00:38.583 INFO:journalctl@ceph.mds.f.smithi183.stdout:Mar 28 16:00:38 smithi183 podman[65880]: 2024-03-28 16:00:38.320709814 +0000 UTC m=+0.386778123 container init fc616700c00247e3cb65f7b0b5c6bb9b9bae245cf7eeae84bd0accf3158a14b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-28T16:00:38.583 INFO:journalctl@ceph.mds.f.smithi183.stdout:Mar 28 16:00:38 smithi183 podman[65880]: 2024-03-28 16:00:38.333081497 +0000 UTC m=+0.399149770 container start fc616700c00247e3cb65f7b0b5c6bb9b9bae245cf7eeae84bd0accf3158a14b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-f, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-28T16:00:38.583 INFO:journalctl@ceph.mds.f.smithi183.stdout:Mar 28 16:00:38 smithi183 ceph-mds[65901]: -- 172.21.15.183:0/2355567863 <== mon.1 v2:172.21.15.183:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5638cd4bfa40 con 0x5638cd493000 2024-03-28T16:00:38.583 INFO:journalctl@ceph.mds.f.smithi183.stdout:Mar 28 16:00:38 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-f[65896]: starting mds.f at 2024-03-28T16:00:38.583 INFO:journalctl@ceph.mds.f.smithi183.stdout:Mar 28 16:00:38 smithi183 bash[65880]: fc616700c00247e3cb65f7b0b5c6bb9b9bae245cf7eeae84bd0accf3158a14b0 2024-03-28T16:00:38.583 INFO:journalctl@ceph.mds.f.smithi183.stdout:Mar 28 16:00:38 smithi183 systemd[1]: Started Ceph mds.f for b5f8225a-ed1a-11ee-95d0-87774f69a715. 2024-03-28T16:00:39.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:38 smithi150 ceph-mon[34825]: pgmap v345: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:39.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:38 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/3604420576' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:00:39.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:38 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:39.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:38 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:39.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:38 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:39.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:38 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T16:00:39.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:38 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T16:00:39.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:38 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:00:39.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:38 smithi067 ceph-mon[31707]: pgmap v345: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:39.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:38 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/3604420576' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:00:39.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:38 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:39.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:38 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:39.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:38 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:39.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:38 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T16:00:39.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:38 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T16:00:39.321 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:38 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:00:39.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:38 smithi183 ceph-mon[35580]: pgmap v345: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:39.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:38 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/3604420576' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:00:39.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:38 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:39.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:38 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:39.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:38 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:39.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:38 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T16:00:39.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:38 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T16:00:39.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:38 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:00:39.990 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T16:00:40.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:39 smithi183 ceph-mon[35580]: Deploying daemon mds.c on smithi183 2024-03-28T16:00:40.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:39 smithi183 ceph-mon[35580]: mds.? [v2:172.21.15.183:6834/1589784425,v1:172.21.15.183:6835/1589784425] up:boot 2024-03-28T16:00:40.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:39 smithi183 ceph-mon[35580]: fsmap 6 up:standby 2024-03-28T16:00:40.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:39 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T16:00:40.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:40.225+0000 7fe553200640 1 -- 172.21.15.67:0/28485194 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5541071c0 msgr2=0x7fe554109610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:40.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:40.225+0000 7fe553200640 1 --2- 172.21.15.67:0/28485194 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5541071c0 0x7fe554109610 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fe54c009ca0 tx=0x7fe54c02c0f0 comp rx=0 tx=0).stop 2024-03-28T16:00:40.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:40.226+0000 7fe553200640 1 -- 172.21.15.67:0/28485194 shutdown_connections 2024-03-28T16:00:40.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:40.226+0000 7fe553200640 1 --2- 172.21.15.67:0/28485194 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5541071c0 0x7fe554109610 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:40.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:40.226+0000 7fe553200640 1 --2- 172.21.15.67:0/28485194 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe554104890 0x7fe554106c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:40.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:40.226+0000 7fe553200640 1 --2- 172.21.15.67:0/28485194 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe554101f60 0x7fe554104350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:40.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:40.226+0000 7fe553200640 1 -- 172.21.15.67:0/28485194 >> 172.21.15.67:0/28485194 conn(0x7fe5540fb950 msgr2=0x7fe5540fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:00:40.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:40.226+0000 7fe553200640 1 -- 172.21.15.67:0/28485194 shutdown_connections 2024-03-28T16:00:40.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:40.226+0000 7fe553200640 1 -- 172.21.15.67:0/28485194 wait complete. 2024-03-28T16:00:40.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:40.227+0000 7fe553200640 1 Processor -- start 2024-03-28T16:00:40.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:40.227+0000 7fe553200640 1 -- start start 2024-03-28T16:00:40.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:40.227+0000 7fe553200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe554101f60 0x7fe5541976f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:40.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:40.227+0000 7fe553200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe554104890 0x7fe554197c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:40.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:40.227+0000 7fe553200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5541071c0 0x7fe55419eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:40.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:40.227+0000 7fe553200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe55410d020 con 0x7fe554101f60 2024-03-28T16:00:40.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:40.227+0000 7fe553200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe55410cea0 con 0x7fe5541071c0 2024-03-28T16:00:40.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:40.227+0000 7fe553200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe55410d1a0 con 0x7fe554104890 2024-03-28T16:00:40.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:40.228+0000 7fe551e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe554101f60 0x7fe5541976f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:40.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:40.228+0000 7fe551e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe554101f60 0x7fe5541976f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46878/0 (socket says 172.21.15.67:46878) 2024-03-28T16:00:40.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:40.228+0000 7fe551e00640 1 -- 172.21.15.67:0/1971231669 learned_addr learned my addr 172.21.15.67:0/1971231669 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:00:40.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:40.228+0000 7fe551400640 1 --2- 172.21.15.67:0/1971231669 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe554104890 0x7fe554197c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:40.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:40.228+0000 7fe552800640 1 --2- 172.21.15.67:0/1971231669 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5541071c0 0x7fe55419eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:40.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:40.228+0000 7fe551e00640 1 -- 172.21.15.67:0/1971231669 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe554104890 msgr2=0x7fe554197c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:40.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:40.228+0000 7fe551e00640 1 --2- 172.21.15.67:0/1971231669 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe554104890 0x7fe554197c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:40.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:40.228+0000 7fe551e00640 1 -- 172.21.15.67:0/1971231669 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5541071c0 msgr2=0x7fe55419eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:40.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:40.228+0000 7fe551e00640 1 --2- 172.21.15.67:0/1971231669 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5541071c0 0x7fe55419eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:40.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:40.228+0000 7fe551e00640 1 -- 172.21.15.67:0/1971231669 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe55419f310 con 0x7fe554101f60 2024-03-28T16:00:40.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:40.228+0000 7fe552800640 1 --2- 172.21.15.67:0/1971231669 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5541071c0 0x7fe55419eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:00:40.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:40.228+0000 7fe551e00640 1 --2- 172.21.15.67:0/1971231669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe554101f60 0x7fe5541976f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe54400d950 tx=0x7fe54400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:00:40.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:40.229+0000 7fe54aa00640 1 -- 172.21.15.67:0/1971231669 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe544017070 con 0x7fe554101f60 2024-03-28T16:00:40.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:40.229+0000 7fe54aa00640 1 -- 172.21.15.67:0/1971231669 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe544009c30 con 0x7fe554101f60 2024-03-28T16:00:40.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:40.229+0000 7fe54aa00640 1 -- 172.21.15.67:0/1971231669 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe54400f620 con 0x7fe554101f60 2024-03-28T16:00:40.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:40.229+0000 7fe553200640 1 -- 172.21.15.67:0/1971231669 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe55419f600 con 0x7fe554101f60 2024-03-28T16:00:40.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:40.229+0000 7fe553200640 1 -- 172.21.15.67:0/1971231669 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe55419fba0 con 0x7fe554101f60 2024-03-28T16:00:40.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:40.230+0000 7fe553200640 1 -- 172.21.15.67:0/1971231669 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5540fd440 con 0x7fe554101f60 2024-03-28T16:00:40.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:40.232+0000 7fe54aa00640 1 -- 172.21.15.67:0/1971231669 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fe5440109a0 con 0x7fe554101f60 2024-03-28T16:00:40.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:40.234+0000 7fe54aa00640 1 --2- 172.21.15.67:0/1971231669 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fe5380a2880 0x7fe5380a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:40.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:40.234+0000 7fe54aa00640 1 -- 172.21.15.67:0/1971231669 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fe5440ce2e0 con 0x7fe554101f60 2024-03-28T16:00:40.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:40.234+0000 7fe551400640 1 --2- 172.21.15.67:0/1971231669 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fe5380a2880 0x7fe5380a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:40.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:40.235+0000 7fe551400640 1 --2- 172.21.15.67:0/1971231669 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fe5380a2880 0x7fe5380a4d40 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe540005fd0 tx=0x7fe540005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:00:40.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:40.240+0000 7fe54aa00640 1 -- 172.21.15.67:0/1971231669 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe544009de0 con 0x7fe554101f60 2024-03-28T16:00:40.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:39 smithi150 ceph-mon[34825]: Deploying daemon mds.c on smithi183 2024-03-28T16:00:40.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:39 smithi150 ceph-mon[34825]: mds.? [v2:172.21.15.183:6834/1589784425,v1:172.21.15.183:6835/1589784425] up:boot 2024-03-28T16:00:40.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:39 smithi150 ceph-mon[34825]: fsmap 6 up:standby 2024-03-28T16:00:40.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:39 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T16:00:40.322 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:39 smithi067 ceph-mon[31707]: Deploying daemon mds.c on smithi183 2024-03-28T16:00:40.323 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:39 smithi067 ceph-mon[31707]: mds.? [v2:172.21.15.183:6834/1589784425,v1:172.21.15.183:6835/1589784425] up:boot 2024-03-28T16:00:40.323 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:39 smithi067 ceph-mon[31707]: fsmap 6 up:standby 2024-03-28T16:00:40.323 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:39 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T16:00:40.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:40.483+0000 7fe553200640 1 -- 172.21.15.67:0/1971231669 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe55402d070 con 0x7fe554101f60 2024-03-28T16:00:40.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:40.485+0000 7fe54aa00640 1 -- 172.21.15.67:0/1971231669 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+5289 (secure 0 0 0) 0x7fe54409b780 con 0x7fe554101f60 2024-03-28T16:00:40.485 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:00:40.485 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2008297659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2008297659},{"type":"v1","addr":"172.21.15.67:6835","nonce":2008297659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14532,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/320295745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":320295745},{"type":"v1","addr":"172.21.15.67:6837","nonce":320295745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2129665423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2129665423},{"type":"v1","addr":"172.21.15.183:6833","nonce":2129665423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24496,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/1589784425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":1589784425},{"type":"v1","addr":"172.21.15.183:6835","nonce":1589784425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":34409,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1008335521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1008335521},{"type":"v1","addr":"172.21.15.150:6833","nonce":1008335521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":34445,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3269935695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3269935695},{"type":"v1","addr":"172.21.15.150:6835","nonce":3269935695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[]} 2024-03-28T16:00:40.486 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 7 2024-03-28T16:00:40.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:40.488+0000 7fe553200640 1 -- 172.21.15.67:0/1971231669 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fe5380a2880 msgr2=0x7fe5380a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:40.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:40.488+0000 7fe553200640 1 --2- 172.21.15.67:0/1971231669 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fe5380a2880 0x7fe5380a4d40 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe540005fd0 tx=0x7fe540005850 comp rx=0 tx=0).stop 2024-03-28T16:00:40.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:40.488+0000 7fe553200640 1 -- 172.21.15.67:0/1971231669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe554101f60 msgr2=0x7fe5541976f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:40.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:40.488+0000 7fe553200640 1 --2- 172.21.15.67:0/1971231669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe554101f60 0x7fe5541976f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe54400d950 tx=0x7fe54400de10 comp rx=0 tx=0).stop 2024-03-28T16:00:40.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:40.488+0000 7fe553200640 1 -- 172.21.15.67:0/1971231669 shutdown_connections 2024-03-28T16:00:40.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:40.488+0000 7fe553200640 1 --2- 172.21.15.67:0/1971231669 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fe5380a2880 0x7fe5380a4d40 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:40.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:40.488+0000 7fe553200640 1 --2- 172.21.15.67:0/1971231669 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5541071c0 0x7fe55419eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:40.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:40.488+0000 7fe553200640 1 --2- 172.21.15.67:0/1971231669 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe554104890 0x7fe554197c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:40.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:40.488+0000 7fe553200640 1 --2- 172.21.15.67:0/1971231669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe554101f60 0x7fe5541976f0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:40.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:40.488+0000 7fe553200640 1 -- 172.21.15.67:0/1971231669 >> 172.21.15.67:0/1971231669 conn(0x7fe5540fb950 msgr2=0x7fe5541029a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:00:40.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:40.488+0000 7fe553200640 1 -- 172.21.15.67:0/1971231669 shutdown_connections 2024-03-28T16:00:40.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:40.488+0000 7fe553200640 1 -- 172.21.15.67:0/1971231669 wait complete. 2024-03-28T16:00:41.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:40 smithi150 ceph-mon[34825]: pgmap v346: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:41.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:40 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/1971231669' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:00:41.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:40 smithi067 ceph-mon[31707]: pgmap v346: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:41.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:40 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/1971231669' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:00:41.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:40 smithi183 ceph-mon[35580]: pgmap v346: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:41.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:40 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/1971231669' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:00:41.805 INFO:journalctl@ceph.mds.c.smithi183.stdout:Mar 28 16:00:41 smithi183 systemd[1]: Starting Ceph mds.c for b5f8225a-ed1a-11ee-95d0-87774f69a715... 2024-03-28T16:00:42.082 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:41 smithi183 ceph-mon[35580]: pgmap v347: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:42.083 INFO:journalctl@ceph.mds.c.smithi183.stdout:Mar 28 16:00:41 smithi183 podman[66198]: 2024-03-28 16:00:41.802672326 +0000 UTC m=+0.114719055 container create 2b20d7441f6a459f45a53251e72fb1a1158c8e056016982dddf223f28a09162c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-c, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, ceph=True) 2024-03-28T16:00:42.083 INFO:journalctl@ceph.mds.c.smithi183.stdout:Mar 28 16:00:41 smithi183 podman[66198]: 2024-03-28 16:00:41.721247978 +0000 UTC m=+0.033294681 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T16:00:42.083 INFO:journalctl@ceph.mds.c.smithi183.stdout:Mar 28 16:00:42 smithi183 podman[66198]: 2024-03-28 16:00:42.081394144 +0000 UTC m=+0.393440879 container init 2b20d7441f6a459f45a53251e72fb1a1158c8e056016982dddf223f28a09162c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-c, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T16:00:42.235 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:41 smithi150 ceph-mon[34825]: pgmap v347: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:42.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:41 smithi067 ceph-mon[31707]: pgmap v347: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:42.351 INFO:journalctl@ceph.mds.c.smithi183.stdout:Mar 28 16:00:42 smithi183 podman[66198]: 2024-03-28 16:00:42.091417025 +0000 UTC m=+0.403463727 container start 2b20d7441f6a459f45a53251e72fb1a1158c8e056016982dddf223f28a09162c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-c, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux ) 2024-03-28T16:00:42.351 INFO:journalctl@ceph.mds.c.smithi183.stdout:Mar 28 16:00:42 smithi183 ceph-mds[66217]: -- 172.21.15.183:0/2634728316 <== mon.1 v2:172.21.15.183:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c6d3621a40 con 0x55c6d35f4800 2024-03-28T16:00:42.351 INFO:journalctl@ceph.mds.c.smithi183.stdout:Mar 28 16:00:42 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-c[66213]: starting mds.c at 2024-03-28T16:00:42.351 INFO:journalctl@ceph.mds.c.smithi183.stdout:Mar 28 16:00:42 smithi183 bash[66198]: 2b20d7441f6a459f45a53251e72fb1a1158c8e056016982dddf223f28a09162c 2024-03-28T16:00:42.351 INFO:journalctl@ceph.mds.c.smithi183.stdout:Mar 28 16:00:42 smithi183 systemd[1]: Started Ceph mds.c for b5f8225a-ed1a-11ee-95d0-87774f69a715. 2024-03-28T16:00:42.511 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T16:00:42.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:42.745+0000 7ffbfb200640 1 -- 172.21.15.67:0/1056316322 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbf4074100 msgr2=0x7ffbf4074580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:42.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:42.745+0000 7ffbfb200640 1 --2- 172.21.15.67:0/1056316322 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbf4074100 0x7ffbf4074580 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ffbe0009930 tx=0x7ffbe002bc30 comp rx=0 tx=0).stop 2024-03-28T16:00:42.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:42.745+0000 7ffbfb200640 1 -- 172.21.15.67:0/1056316322 shutdown_connections 2024-03-28T16:00:42.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:42.745+0000 7ffbfb200640 1 --2- 172.21.15.67:0/1056316322 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffbf410dab0 0x7ffbf410fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:42.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:42.745+0000 7ffbfb200640 1 --2- 172.21.15.67:0/1056316322 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbf4074100 0x7ffbf4074580 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:42.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:42.745+0000 7ffbfb200640 1 --2- 172.21.15.67:0/1056316322 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffbf40756e0 0x7ffbf4073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:42.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:42.745+0000 7ffbfb200640 1 -- 172.21.15.67:0/1056316322 >> 172.21.15.67:0/1056316322 conn(0x7ffbf40fca30 msgr2=0x7ffbf40fee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:00:42.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:42.745+0000 7ffbfb200640 1 -- 172.21.15.67:0/1056316322 shutdown_connections 2024-03-28T16:00:42.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:42.745+0000 7ffbfb200640 1 -- 172.21.15.67:0/1056316322 wait complete. 2024-03-28T16:00:42.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:42.746+0000 7ffbfb200640 1 Processor -- start 2024-03-28T16:00:42.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:42.746+0000 7ffbfb200640 1 -- start start 2024-03-28T16:00:42.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:42.746+0000 7ffbfb200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffbf4074100 0x7ffbf419dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:42.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:42.746+0000 7ffbfb200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffbf40756e0 0x7ffbf419e250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:42.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:42.746+0000 7ffbfb200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbf410dab0 0x7ffbf41a5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:42.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:42.746+0000 7ffbfb200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbf4113630 con 0x7ffbf410dab0 2024-03-28T16:00:42.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:42.746+0000 7ffbfb200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbf41134b0 con 0x7ffbf4074100 2024-03-28T16:00:42.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:42.746+0000 7ffbfb200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbf41137b0 con 0x7ffbf40756e0 2024-03-28T16:00:42.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:42.747+0000 7ffbfa800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbf410dab0 0x7ffbf41a5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:42.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:42.747+0000 7ffbfa800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbf410dab0 0x7ffbf41a5280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46896/0 (socket says 172.21.15.67:46896) 2024-03-28T16:00:42.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:42.747+0000 7ffbfa800640 1 -- 172.21.15.67:0/625026271 learned_addr learned my addr 172.21.15.67:0/625026271 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:00:42.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:42.747+0000 7ffbf9e00640 1 --2- 172.21.15.67:0/625026271 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffbf4074100 0x7ffbf419dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:42.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:42.747+0000 7ffbf9400640 1 --2- 172.21.15.67:0/625026271 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffbf40756e0 0x7ffbf419e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:42.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:42.747+0000 7ffbfa800640 1 -- 172.21.15.67:0/625026271 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffbf40756e0 msgr2=0x7ffbf419e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:42.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:42.747+0000 7ffbfa800640 1 --2- 172.21.15.67:0/625026271 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffbf40756e0 0x7ffbf419e250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:42.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:42.747+0000 7ffbfa800640 1 -- 172.21.15.67:0/625026271 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffbf4074100 msgr2=0x7ffbf419dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:42.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:42.747+0000 7ffbfa800640 1 --2- 172.21.15.67:0/625026271 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffbf4074100 0x7ffbf419dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:42.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:42.747+0000 7ffbfa800640 1 -- 172.21.15.67:0/625026271 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbf41a5920 con 0x7ffbf410dab0 2024-03-28T16:00:42.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:42.748+0000 7ffbfa800640 1 --2- 172.21.15.67:0/625026271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbf410dab0 0x7ffbf41a5280 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ffbf000cce0 tx=0x7ffbf000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:00:42.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:42.748+0000 7ffbeea00640 1 -- 172.21.15.67:0/625026271 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbf000f450 con 0x7ffbf410dab0 2024-03-28T16:00:42.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:42.748+0000 7ffbeea00640 1 -- 172.21.15.67:0/625026271 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ffbf000faa0 con 0x7ffbf410dab0 2024-03-28T16:00:42.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:42.748+0000 7ffbfb200640 1 -- 172.21.15.67:0/625026271 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbf41a5c10 con 0x7ffbf410dab0 2024-03-28T16:00:42.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:42.748+0000 7ffbeea00640 1 -- 172.21.15.67:0/625026271 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbf0018b70 con 0x7ffbf410dab0 2024-03-28T16:00:42.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:42.748+0000 7ffbfb200640 1 -- 172.21.15.67:0/625026271 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbf41a6160 con 0x7ffbf410dab0 2024-03-28T16:00:42.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:42.749+0000 7ffbfb200640 1 -- 172.21.15.67:0/625026271 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbf4103c80 con 0x7ffbf410dab0 2024-03-28T16:00:42.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:42.751+0000 7ffbeea00640 1 -- 172.21.15.67:0/625026271 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ffbf0018d10 con 0x7ffbf410dab0 2024-03-28T16:00:42.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:42.753+0000 7ffbeea00640 1 --2- 172.21.15.67:0/625026271 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7ffbdc0a2880 0x7ffbdc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:42.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:42.753+0000 7ffbeea00640 1 -- 172.21.15.67:0/625026271 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ffbf00ceaa0 con 0x7ffbf410dab0 2024-03-28T16:00:42.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:42.753+0000 7ffbf9e00640 1 --2- 172.21.15.67:0/625026271 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7ffbdc0a2880 0x7ffbdc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:42.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:42.754+0000 7ffbf9e00640 1 --2- 172.21.15.67:0/625026271 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7ffbdc0a2880 0x7ffbdc0a4d40 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ffbf419f1e0 tx=0x7ffbe800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:00:42.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:42.757+0000 7ffbeea00640 1 -- 172.21.15.67:0/625026271 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ffbf0014030 con 0x7ffbf410dab0 2024-03-28T16:00:43.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:43.004+0000 7ffbfb200640 1 -- 172.21.15.67:0/625026271 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffbf402d070 con 0x7ffbf410dab0 2024-03-28T16:00:43.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:43.004+0000 7ffbeea00640 1 -- 172.21.15.67:0/625026271 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6083 (secure 0 0 0) 0x7ffbf009bf40 con 0x7ffbf410dab0 2024-03-28T16:00:43.005 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:00:43.005 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2008297659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2008297659},{"type":"v1","addr":"172.21.15.67:6835","nonce":2008297659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14532,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/320295745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":320295745},{"type":"v1","addr":"172.21.15.67:6837","nonce":320295745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2129665423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2129665423},{"type":"v1","addr":"172.21.15.183:6833","nonce":2129665423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24496,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/1589784425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":1589784425},{"type":"v1","addr":"172.21.15.183:6835","nonce":1589784425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24508,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/2185863835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":2185863835},{"type":"v1","addr":"172.21.15.183:6837","nonce":2185863835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":34409,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1008335521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1008335521},{"type":"v1","addr":"172.21.15.150:6833","nonce":1008335521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":34445,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3269935695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3269935695},{"type":"v1","addr":"172.21.15.150:6835","nonce":3269935695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[]} 2024-03-28T16:00:43.005 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 8 2024-03-28T16:00:43.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:43.007+0000 7ffbfb200640 1 -- 172.21.15.67:0/625026271 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7ffbdc0a2880 msgr2=0x7ffbdc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:43.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:43.007+0000 7ffbfb200640 1 --2- 172.21.15.67:0/625026271 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7ffbdc0a2880 0x7ffbdc0a4d40 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ffbf419f1e0 tx=0x7ffbe800b390 comp rx=0 tx=0).stop 2024-03-28T16:00:43.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:43.007+0000 7ffbfb200640 1 -- 172.21.15.67:0/625026271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbf410dab0 msgr2=0x7ffbf41a5280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:43.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:43.007+0000 7ffbfb200640 1 --2- 172.21.15.67:0/625026271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbf410dab0 0x7ffbf41a5280 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ffbf000cce0 tx=0x7ffbf000a620 comp rx=0 tx=0).stop 2024-03-28T16:00:43.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:43.007+0000 7ffbfb200640 1 -- 172.21.15.67:0/625026271 shutdown_connections 2024-03-28T16:00:43.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:43.007+0000 7ffbfb200640 1 --2- 172.21.15.67:0/625026271 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7ffbdc0a2880 0x7ffbdc0a4d40 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:43.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:43.007+0000 7ffbfb200640 1 --2- 172.21.15.67:0/625026271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbf410dab0 0x7ffbf41a5280 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:43.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:43.007+0000 7ffbfb200640 1 --2- 172.21.15.67:0/625026271 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffbf40756e0 0x7ffbf419e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:43.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:43.007+0000 7ffbfb200640 1 --2- 172.21.15.67:0/625026271 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffbf4074100 0x7ffbf419dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:43.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:43.007+0000 7ffbfb200640 1 -- 172.21.15.67:0/625026271 >> 172.21.15.67:0/625026271 conn(0x7ffbf40fca30 msgr2=0x7ffbf4100e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:00:43.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:43.008+0000 7ffbfb200640 1 -- 172.21.15.67:0/625026271 shutdown_connections 2024-03-28T16:00:43.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:43.008+0000 7ffbfb200640 1 -- 172.21.15.67:0/625026271 wait complete. 2024-03-28T16:00:43.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:43 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:43.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:43 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:43.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:43 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:43.571 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:43 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T16:00:43.571 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:43 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T16:00:43.571 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:43 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:00:43.571 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:43 smithi067 ceph-mon[31707]: Deploying daemon mds.h on smithi150 2024-03-28T16:00:43.571 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:43 smithi067 ceph-mon[31707]: mds.? [v2:172.21.15.183:6836/2185863835,v1:172.21.15.183:6837/2185863835] up:boot 2024-03-28T16:00:43.571 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:43 smithi067 ceph-mon[31707]: fsmap 7 up:standby 2024-03-28T16:00:43.571 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:43 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T16:00:43.571 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:43 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/625026271' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:00:43.581 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:43 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:43.581 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:43 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:43.581 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:43 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:43.582 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:43 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T16:00:43.582 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:43 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T16:00:43.582 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:43 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:00:43.582 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:43 smithi150 ceph-mon[34825]: Deploying daemon mds.h on smithi150 2024-03-28T16:00:43.582 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:43 smithi150 ceph-mon[34825]: mds.? [v2:172.21.15.183:6836/2185863835,v1:172.21.15.183:6837/2185863835] up:boot 2024-03-28T16:00:43.582 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:43 smithi150 ceph-mon[34825]: fsmap 7 up:standby 2024-03-28T16:00:43.582 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:43 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T16:00:43.582 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:43 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/625026271' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:00:43.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:43 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:43.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:43 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:43.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:43 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:43.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:43 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T16:00:43.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:43 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T16:00:43.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:43 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:00:43.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:43 smithi183 ceph-mon[35580]: Deploying daemon mds.h on smithi150 2024-03-28T16:00:43.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:43 smithi183 ceph-mon[35580]: mds.? [v2:172.21.15.183:6836/2185863835,v1:172.21.15.183:6837/2185863835] up:boot 2024-03-28T16:00:43.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:43 smithi183 ceph-mon[35580]: fsmap 7 up:standby 2024-03-28T16:00:43.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:43 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T16:00:43.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:43 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/625026271' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:00:44.582 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:44 smithi150 ceph-mon[34825]: pgmap v348: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:44.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:44 smithi183 ceph-mon[35580]: pgmap v348: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:44.823 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:44 smithi067 ceph-mon[31707]: pgmap v348: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:45.032 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T16:00:45.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:45.263+0000 7fb9e2400640 1 -- 172.21.15.67:0/3261999176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9dc102010 msgr2=0x7fb9dc104400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:45.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:45.263+0000 7fb9e2400640 1 --2- 172.21.15.67:0/3261999176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9dc102010 0x7fb9dc104400 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fb9c4009900 tx=0x7fb9c402bc30 comp rx=0 tx=0).stop 2024-03-28T16:00:45.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:45.264+0000 7fb9e2400640 1 -- 172.21.15.67:0/3261999176 shutdown_connections 2024-03-28T16:00:45.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:45.264+0000 7fb9e2400640 1 --2- 172.21.15.67:0/3261999176 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb9dc107270 0x7fb9dc109660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:45.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:45.264+0000 7fb9e2400640 1 --2- 172.21.15.67:0/3261999176 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9dc104940 0x7fb9dc106d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:45.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:45.264+0000 7fb9e2400640 1 --2- 172.21.15.67:0/3261999176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9dc102010 0x7fb9dc104400 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:45.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:45.264+0000 7fb9e2400640 1 -- 172.21.15.67:0/3261999176 >> 172.21.15.67:0/3261999176 conn(0x7fb9dc0fb940 msgr2=0x7fb9dc0fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:00:45.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:45.264+0000 7fb9e2400640 1 -- 172.21.15.67:0/3261999176 shutdown_connections 2024-03-28T16:00:45.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:45.264+0000 7fb9e2400640 1 -- 172.21.15.67:0/3261999176 wait complete. 2024-03-28T16:00:45.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:45.265+0000 7fb9e2400640 1 Processor -- start 2024-03-28T16:00:45.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:45.265+0000 7fb9e2400640 1 -- start start 2024-03-28T16:00:45.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:45.265+0000 7fb9e2400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9dc102010 0x7fb9dc199930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:45.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:45.265+0000 7fb9e2400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9dc104940 0x7fb9dc199e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:45.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:45.265+0000 7fb9e2400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb9dc107270 0x7fb9dc1a0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:45.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:45.265+0000 7fb9e2400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9dc10d0e0 con 0x7fb9dc104940 2024-03-28T16:00:45.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:45.265+0000 7fb9e2400640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9dc10cf60 con 0x7fb9dc102010 2024-03-28T16:00:45.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:45.265+0000 7fb9e2400640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9dc10d260 con 0x7fb9dc107270 2024-03-28T16:00:45.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:45.265+0000 7fb9dbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9dc104940 0x7fb9dc199e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:45.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:45.266+0000 7fb9dbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9dc104940 0x7fb9dc199e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46914/0 (socket says 172.21.15.67:46914) 2024-03-28T16:00:45.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:45.266+0000 7fb9dbe00640 1 -- 172.21.15.67:0/101646994 learned_addr learned my addr 172.21.15.67:0/101646994 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:00:45.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:45.266+0000 7fb9e1000640 1 --2- 172.21.15.67:0/101646994 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9dc102010 0x7fb9dc199930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:45.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:45.266+0000 7fb9e1a00640 1 --2- 172.21.15.67:0/101646994 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb9dc107270 0x7fb9dc1a0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:45.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:45.266+0000 7fb9dbe00640 1 -- 172.21.15.67:0/101646994 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb9dc107270 msgr2=0x7fb9dc1a0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:45.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:45.266+0000 7fb9dbe00640 1 --2- 172.21.15.67:0/101646994 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb9dc107270 0x7fb9dc1a0ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:45.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:45.266+0000 7fb9dbe00640 1 -- 172.21.15.67:0/101646994 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9dc102010 msgr2=0x7fb9dc199930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:45.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:45.266+0000 7fb9dbe00640 1 --2- 172.21.15.67:0/101646994 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9dc102010 0x7fb9dc199930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:45.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:45.266+0000 7fb9dbe00640 1 -- 172.21.15.67:0/101646994 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9dc1a1550 con 0x7fb9dc104940 2024-03-28T16:00:45.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:45.266+0000 7fb9dbe00640 1 --2- 172.21.15.67:0/101646994 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9dc104940 0x7fb9dc199e70 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc00ccb0 tx=0x7fb9cc00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:00:45.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:45.267+0000 7fb9d9600640 1 -- 172.21.15.67:0/101646994 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9cc00ce60 con 0x7fb9dc104940 2024-03-28T16:00:45.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:45.267+0000 7fb9d9600640 1 -- 172.21.15.67:0/101646994 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb9cc00fb00 con 0x7fb9dc104940 2024-03-28T16:00:45.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:45.267+0000 7fb9d9600640 1 -- 172.21.15.67:0/101646994 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9cc018bb0 con 0x7fb9dc104940 2024-03-28T16:00:45.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:45.267+0000 7fb9e2400640 1 -- 172.21.15.67:0/101646994 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9dc1a1840 con 0x7fb9dc104940 2024-03-28T16:00:45.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:45.267+0000 7fb9e2400640 1 -- 172.21.15.67:0/101646994 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9dc1a1dc0 con 0x7fb9dc104940 2024-03-28T16:00:45.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:45.268+0000 7fb9e2400640 1 -- 172.21.15.67:0/101646994 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9dc108d60 con 0x7fb9dc104940 2024-03-28T16:00:45.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:45.270+0000 7fb9d9600640 1 -- 172.21.15.67:0/101646994 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fb9cc00f650 con 0x7fb9dc104940 2024-03-28T16:00:45.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:45.272+0000 7fb9d9600640 1 --2- 172.21.15.67:0/101646994 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb9c00a2880 0x7fb9c00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:45.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:45.273+0000 7fb9d9600640 1 -- 172.21.15.67:0/101646994 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fb9cc0ced40 con 0x7fb9dc104940 2024-03-28T16:00:45.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:45.273+0000 7fb9e1000640 1 --2- 172.21.15.67:0/101646994 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb9c00a2880 0x7fb9c00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:45.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:45.273+0000 7fb9e1000640 1 --2- 172.21.15.67:0/101646994 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb9c00a2880 0x7fb9c00a4d40 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb9c4006010 tx=0x7fb9c4000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:00:45.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:45.276+0000 7fb9d9600640 1 -- 172.21.15.67:0/101646994 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb9cc014030 con 0x7fb9dc104940 2024-03-28T16:00:45.282 INFO:journalctl@ceph.mds.h.smithi150.stdout:Mar 28 16:00:45 smithi150 systemd[1]: Starting Ceph mds.h for b5f8225a-ed1a-11ee-95d0-87774f69a715... 2024-03-28T16:00:45.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:45.521+0000 7fb9e2400640 1 -- 172.21.15.67:0/101646994 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb9dc02d070 con 0x7fb9dc104940 2024-03-28T16:00:45.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:45.522+0000 7fb9d9600640 1 -- 172.21.15.67:0/101646994 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6083 (secure 0 0 0) 0x7fb9cc09c2f0 con 0x7fb9dc104940 2024-03-28T16:00:45.523 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:00:45.523 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2008297659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2008297659},{"type":"v1","addr":"172.21.15.67:6835","nonce":2008297659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14532,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/320295745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":320295745},{"type":"v1","addr":"172.21.15.67:6837","nonce":320295745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2129665423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2129665423},{"type":"v1","addr":"172.21.15.183:6833","nonce":2129665423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24496,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/1589784425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":1589784425},{"type":"v1","addr":"172.21.15.183:6835","nonce":1589784425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24508,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/2185863835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":2185863835},{"type":"v1","addr":"172.21.15.183:6837","nonce":2185863835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":34409,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1008335521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1008335521},{"type":"v1","addr":"172.21.15.150:6833","nonce":1008335521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":34445,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3269935695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3269935695},{"type":"v1","addr":"172.21.15.150:6835","nonce":3269935695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[]} 2024-03-28T16:00:45.523 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 8 2024-03-28T16:00:45.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:45.525+0000 7fb9e2400640 1 -- 172.21.15.67:0/101646994 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb9c00a2880 msgr2=0x7fb9c00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:45.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:45.525+0000 7fb9e2400640 1 --2- 172.21.15.67:0/101646994 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb9c00a2880 0x7fb9c00a4d40 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb9c4006010 tx=0x7fb9c4000bc0 comp rx=0 tx=0).stop 2024-03-28T16:00:45.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:45.525+0000 7fb9e2400640 1 -- 172.21.15.67:0/101646994 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9dc104940 msgr2=0x7fb9dc199e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:45.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:45.525+0000 7fb9e2400640 1 --2- 172.21.15.67:0/101646994 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9dc104940 0x7fb9dc199e70 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc00ccb0 tx=0x7fb9cc00a620 comp rx=0 tx=0).stop 2024-03-28T16:00:45.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:45.525+0000 7fb9e2400640 1 -- 172.21.15.67:0/101646994 shutdown_connections 2024-03-28T16:00:45.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:45.525+0000 7fb9e2400640 1 --2- 172.21.15.67:0/101646994 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb9c00a2880 0x7fb9c00a4d40 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:45.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:45.525+0000 7fb9e2400640 1 --2- 172.21.15.67:0/101646994 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb9dc107270 0x7fb9dc1a0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:45.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:45.525+0000 7fb9e2400640 1 --2- 172.21.15.67:0/101646994 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9dc104940 0x7fb9dc199e70 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:45.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:45.525+0000 7fb9e2400640 1 --2- 172.21.15.67:0/101646994 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9dc102010 0x7fb9dc199930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:45.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:45.525+0000 7fb9e2400640 1 -- 172.21.15.67:0/101646994 >> 172.21.15.67:0/101646994 conn(0x7fb9dc0fb940 msgr2=0x7fb9dc1017a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:00:45.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:45.525+0000 7fb9e2400640 1 -- 172.21.15.67:0/101646994 shutdown_connections 2024-03-28T16:00:45.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:45.525+0000 7fb9e2400640 1 -- 172.21.15.67:0/101646994 wait complete. 2024-03-28T16:00:45.903 INFO:journalctl@ceph.mds.h.smithi150.stdout:Mar 28 16:00:45 smithi150 podman[65214]: 2024-03-28 16:00:45.548745882 +0000 UTC m=+0.034044431 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T16:00:45.903 INFO:journalctl@ceph.mds.h.smithi150.stdout:Mar 28 16:00:45 smithi150 podman[65214]: 2024-03-28 16:00:45.647448719 +0000 UTC m=+0.132747306 container create b4dfaf989e0853cb88f1b39854334810b4a6e95e1976b7495a50ca4a87f5c040 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-h, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-28T16:00:45.903 INFO:journalctl@ceph.mds.h.smithi150.stdout:Mar 28 16:00:45 smithi150 podman[65214]: 2024-03-28 16:00:45.9015028 +0000 UTC m=+0.386801394 container init b4dfaf989e0853cb88f1b39854334810b4a6e95e1976b7495a50ca4a87f5c040 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-h, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319) 2024-03-28T16:00:46.303 INFO:journalctl@ceph.mds.h.smithi150.stdout:Mar 28 16:00:45 smithi150 podman[65214]: 2024-03-28 16:00:45.914066137 +0000 UTC m=+0.399364692 container start b4dfaf989e0853cb88f1b39854334810b4a6e95e1976b7495a50ca4a87f5c040 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-h, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True) 2024-03-28T16:00:46.303 INFO:journalctl@ceph.mds.h.smithi150.stdout:Mar 28 16:00:45 smithi150 ceph-mds[65234]: -- 172.21.15.150:0/2796008818 <== mon.2 v2:172.21.15.150:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55fa08d09a40 con 0x55fa08cdcc00 2024-03-28T16:00:46.303 INFO:journalctl@ceph.mds.h.smithi150.stdout:Mar 28 16:00:45 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-h[65230]: starting mds.h at 2024-03-28T16:00:46.303 INFO:journalctl@ceph.mds.h.smithi150.stdout:Mar 28 16:00:46 smithi150 bash[65214]: b4dfaf989e0853cb88f1b39854334810b4a6e95e1976b7495a50ca4a87f5c040 2024-03-28T16:00:46.303 INFO:journalctl@ceph.mds.h.smithi150.stdout:Mar 28 16:00:46 smithi150 systemd[1]: Started Ceph mds.h for b5f8225a-ed1a-11ee-95d0-87774f69a715. 2024-03-28T16:00:46.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:46 smithi183 ceph-mon[35580]: pgmap v349: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:46.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:46 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/101646994' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:00:46.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:46 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:46.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:46 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:46.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:46 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:46.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:46 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T16:00:46.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:46 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T16:00:46.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:46 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:00:46.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:46 smithi150 ceph-mon[34825]: pgmap v349: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:46.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:46 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/101646994' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:00:46.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:46 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:46.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:46 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:46.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:46 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:46.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:46 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T16:00:46.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:46 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T16:00:46.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:46 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:00:46.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:46 smithi067 ceph-mon[31707]: pgmap v349: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:46.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:46 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/101646994' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:00:46.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:46 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:46.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:46 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:46.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:46 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:46.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:46 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T16:00:46.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:46 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T16:00:46.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:46 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:00:47.460 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:47 smithi183 ceph-mon[35580]: Deploying daemon mds.i on smithi183 2024-03-28T16:00:47.461 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:47 smithi183 ceph-mon[35580]: mds.? [v2:172.21.15.150:6836/1112886277,v1:172.21.15.150:6837/1112886277] up:boot 2024-03-28T16:00:47.461 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:47 smithi183 ceph-mon[35580]: fsmap 8 up:standby 2024-03-28T16:00:47.461 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:47 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T16:00:47.547 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T16:00:47.635 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:47 smithi067 ceph-mon[31707]: Deploying daemon mds.i on smithi183 2024-03-28T16:00:47.635 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:47 smithi067 ceph-mon[31707]: mds.? [v2:172.21.15.150:6836/1112886277,v1:172.21.15.150:6837/1112886277] up:boot 2024-03-28T16:00:47.635 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:47 smithi067 ceph-mon[31707]: fsmap 8 up:standby 2024-03-28T16:00:47.635 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:47 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T16:00:47.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:47.776+0000 7f0932a00640 1 -- 172.21.15.67:0/1976618886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f092c1036f0 msgr2=0x7f092c103b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:47.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:47.776+0000 7f0932a00640 1 --2- 172.21.15.67:0/1976618886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f092c1036f0 0x7f092c103b10 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f0920009900 tx=0x7f092002bc30 comp rx=0 tx=0).stop 2024-03-28T16:00:47.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:47.776+0000 7f0932a00640 1 -- 172.21.15.67:0/1976618886 shutdown_connections 2024-03-28T16:00:47.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:47.776+0000 7f0932a00640 1 --2- 172.21.15.67:0/1976618886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f092c1052a0 0x7f092c10bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:47.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:47.776+0000 7f0932a00640 1 --2- 172.21.15.67:0/1976618886 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f092c1048e0 0x7f092c104d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:47.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:47.776+0000 7f0932a00640 1 --2- 172.21.15.67:0/1976618886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f092c1036f0 0x7f092c103b10 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:47.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:47.777+0000 7f0932a00640 1 -- 172.21.15.67:0/1976618886 >> 172.21.15.67:0/1976618886 conn(0x7f092c0fec80 msgr2=0x7f092c1010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:00:47.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:47.777+0000 7f0932a00640 1 -- 172.21.15.67:0/1976618886 shutdown_connections 2024-03-28T16:00:47.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:47.777+0000 7f0932a00640 1 -- 172.21.15.67:0/1976618886 wait complete. 2024-03-28T16:00:47.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:47.778+0000 7f0932a00640 1 Processor -- start 2024-03-28T16:00:47.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:47.778+0000 7f0932a00640 1 -- start start 2024-03-28T16:00:47.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:47.778+0000 7f0932a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f092c1036f0 0x7f092c199930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:47.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:47.778+0000 7f0932a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f092c1048e0 0x7f092c199e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:47.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:47.778+0000 7f0932a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f092c1052a0 0x7f092c1a0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:47.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:47.778+0000 7f0932a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f092c10f260 con 0x7f092c1036f0 2024-03-28T16:00:47.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:47.778+0000 7f0932a00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f092c10f0e0 con 0x7f092c1048e0 2024-03-28T16:00:47.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:47.778+0000 7f0932a00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f092c10f3e0 con 0x7f092c1052a0 2024-03-28T16:00:47.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:47.779+0000 7f0931600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f092c1036f0 0x7f092c199930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:47.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:47.779+0000 7f0931600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f092c1036f0 0x7f092c199930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46930/0 (socket says 172.21.15.67:46930) 2024-03-28T16:00:47.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:47.779+0000 7f0931600640 1 -- 172.21.15.67:0/1887075107 learned_addr learned my addr 172.21.15.67:0/1887075107 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:00:47.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:47.779+0000 7f0930c00640 1 --2- 172.21.15.67:0/1887075107 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f092c1048e0 0x7f092c199e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:47.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:47.779+0000 7f0932000640 1 --2- 172.21.15.67:0/1887075107 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f092c1052a0 0x7f092c1a0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:47.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:47.779+0000 7f0931600640 1 -- 172.21.15.67:0/1887075107 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f092c1052a0 msgr2=0x7f092c1a0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:47.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:47.779+0000 7f0931600640 1 --2- 172.21.15.67:0/1887075107 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f092c1052a0 0x7f092c1a0ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:47.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:47.779+0000 7f0931600640 1 -- 172.21.15.67:0/1887075107 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f092c1048e0 msgr2=0x7f092c199e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:47.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:47.779+0000 7f0931600640 1 --2- 172.21.15.67:0/1887075107 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f092c1048e0 0x7f092c199e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:47.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:47.779+0000 7f0931600640 1 -- 172.21.15.67:0/1887075107 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f092c1a1550 con 0x7f092c1036f0 2024-03-28T16:00:47.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:47.779+0000 7f0930c00640 1 --2- 172.21.15.67:0/1887075107 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f092c1048e0 0x7f092c199e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:00:47.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:47.779+0000 7f0932000640 1 --2- 172.21.15.67:0/1887075107 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f092c1052a0 0x7f092c1a0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:00:47.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:47.780+0000 7f0931600640 1 --2- 172.21.15.67:0/1887075107 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f092c1036f0 0x7f092c199930 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f0920005ab0 tx=0x7f09200042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:00:47.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:47.780+0000 7f092a000640 1 -- 172.21.15.67:0/1887075107 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0920043020 con 0x7f092c1036f0 2024-03-28T16:00:47.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:47.780+0000 7f092a000640 1 -- 172.21.15.67:0/1887075107 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f09200054c0 con 0x7f092c1036f0 2024-03-28T16:00:47.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:47.780+0000 7f0932a00640 1 -- 172.21.15.67:0/1887075107 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f092c1a17e0 con 0x7f092c1036f0 2024-03-28T16:00:47.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:47.780+0000 7f092a000640 1 -- 172.21.15.67:0/1887075107 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0920003d10 con 0x7f092c1036f0 2024-03-28T16:00:47.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:47.780+0000 7f0932a00640 1 -- 172.21.15.67:0/1887075107 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f092c1a1d30 con 0x7f092c1036f0 2024-03-28T16:00:47.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:47.781+0000 7f0932a00640 1 -- 172.21.15.67:0/1887075107 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f092c10aa90 con 0x7f092c1036f0 2024-03-28T16:00:47.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:47.783+0000 7f092a000640 1 -- 172.21.15.67:0/1887075107 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f0920046020 con 0x7f092c1036f0 2024-03-28T16:00:47.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:47.785+0000 7f092a000640 1 --2- 172.21.15.67:0/1887075107 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f09140a2800 0x7f09140a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:47.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:47.785+0000 7f092a000640 1 -- 172.21.15.67:0/1887075107 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f09200f0b40 con 0x7f092c1036f0 2024-03-28T16:00:47.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:47.785+0000 7f0930c00640 1 --2- 172.21.15.67:0/1887075107 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f09140a2800 0x7f09140a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:47.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:47.786+0000 7f0930c00640 1 --2- 172.21.15.67:0/1887075107 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f09140a2800 0x7f09140a4cc0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f092c19ae40 tx=0x7f0918005e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:00:47.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:47.789+0000 7f092a000640 1 -- 172.21.15.67:0/1887075107 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f092004d020 con 0x7f092c1036f0 2024-03-28T16:00:47.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:47 smithi150 ceph-mon[34825]: Deploying daemon mds.i on smithi183 2024-03-28T16:00:47.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:47 smithi150 ceph-mon[34825]: mds.? [v2:172.21.15.150:6836/1112886277,v1:172.21.15.150:6837/1112886277] up:boot 2024-03-28T16:00:47.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:47 smithi150 ceph-mon[34825]: fsmap 8 up:standby 2024-03-28T16:00:47.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:47 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T16:00:48.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:48.031+0000 7f0932a00640 1 -- 172.21.15.67:0/1887075107 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f092c02d070 con 0x7f092c1036f0 2024-03-28T16:00:48.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:48.032+0000 7f092a000640 1 -- 172.21.15.67:0/1887075107 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6877 (secure 0 0 0) 0x7f09200be100 con 0x7f092c1036f0 2024-03-28T16:00:48.033 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:00:48.033 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2008297659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2008297659},{"type":"v1","addr":"172.21.15.67:6835","nonce":2008297659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14532,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/320295745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":320295745},{"type":"v1","addr":"172.21.15.67:6837","nonce":320295745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2129665423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2129665423},{"type":"v1","addr":"172.21.15.183:6833","nonce":2129665423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24496,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/1589784425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":1589784425},{"type":"v1","addr":"172.21.15.183:6835","nonce":1589784425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24508,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/2185863835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":2185863835},{"type":"v1","addr":"172.21.15.183:6837","nonce":2185863835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":34409,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1008335521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1008335521},{"type":"v1","addr":"172.21.15.150:6833","nonce":1008335521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":34445,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3269935695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3269935695},{"type":"v1","addr":"172.21.15.150:6835","nonce":3269935695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":34475,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1112886277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":1112886277},{"type":"v1","addr":"172.21.15.150:6837","nonce":1112886277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[]} 2024-03-28T16:00:48.033 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 9 2024-03-28T16:00:48.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:48.035+0000 7f0932a00640 1 -- 172.21.15.67:0/1887075107 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f09140a2800 msgr2=0x7f09140a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:48.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:48.035+0000 7f0932a00640 1 --2- 172.21.15.67:0/1887075107 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f09140a2800 0x7f09140a4cc0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f092c19ae40 tx=0x7f0918005e50 comp rx=0 tx=0).stop 2024-03-28T16:00:48.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:48.035+0000 7f0932a00640 1 -- 172.21.15.67:0/1887075107 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f092c1036f0 msgr2=0x7f092c199930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:48.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:48.035+0000 7f0932a00640 1 --2- 172.21.15.67:0/1887075107 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f092c1036f0 0x7f092c199930 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f0920005ab0 tx=0x7f09200042f0 comp rx=0 tx=0).stop 2024-03-28T16:00:48.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:48.035+0000 7f0932a00640 1 -- 172.21.15.67:0/1887075107 shutdown_connections 2024-03-28T16:00:48.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:48.035+0000 7f0932a00640 1 --2- 172.21.15.67:0/1887075107 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f09140a2800 0x7f09140a4cc0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:48.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:48.035+0000 7f0932a00640 1 --2- 172.21.15.67:0/1887075107 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f092c1052a0 0x7f092c1a0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:48.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:48.035+0000 7f0932a00640 1 --2- 172.21.15.67:0/1887075107 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f092c1048e0 0x7f092c199e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:48.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:48.035+0000 7f0932a00640 1 --2- 172.21.15.67:0/1887075107 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f092c1036f0 0x7f092c199930 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:48.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:48.035+0000 7f0932a00640 1 -- 172.21.15.67:0/1887075107 >> 172.21.15.67:0/1887075107 conn(0x7f092c0fec80 msgr2=0x7f092c107c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:00:48.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:48.035+0000 7f0932a00640 1 -- 172.21.15.67:0/1887075107 shutdown_connections 2024-03-28T16:00:48.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:48.035+0000 7f0932a00640 1 -- 172.21.15.67:0/1887075107 wait complete. 2024-03-28T16:00:48.602 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:48 smithi183 ceph-mon[35580]: pgmap v350: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:48.602 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:48 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/1887075107' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:00:48.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:48 smithi150 ceph-mon[34825]: pgmap v350: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:48.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:48 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/1887075107' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:00:48.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:48 smithi067 ceph-mon[31707]: pgmap v350: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:48.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:48 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/1887075107' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:00:49.826 INFO:journalctl@ceph.mds.i.smithi183.stdout:Mar 28 16:00:49 smithi183 podman[66504]: 2024-03-28 16:00:49.471869054 +0000 UTC m=+0.031612268 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T16:00:49.826 INFO:journalctl@ceph.mds.i.smithi183.stdout:Mar 28 16:00:49 smithi183 podman[66504]: 2024-03-28 16:00:49.825148559 +0000 UTC m=+0.384891801 container init 999fdaab14075feb5960df431524dd27359a5e8a477a34b884dd583e3698ce82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-i, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True) 2024-03-28T16:00:50.058 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T16:00:50.101 INFO:journalctl@ceph.mds.i.smithi183.stdout:Mar 28 16:00:49 smithi183 podman[66504]: 2024-03-28 16:00:49.838728481 +0000 UTC m=+0.398471701 container start 999fdaab14075feb5960df431524dd27359a5e8a477a34b884dd583e3698ce82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-i, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-28T16:00:50.101 INFO:journalctl@ceph.mds.i.smithi183.stdout:Mar 28 16:00:49 smithi183 ceph-mds[66523]: -- 172.21.15.183:0/4038280867 <== mon.1 v2:172.21.15.183:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5645f5a39a40 con 0x5645f5a0c800 2024-03-28T16:00:50.101 INFO:journalctl@ceph.mds.i.smithi183.stdout:Mar 28 16:00:49 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-i[66519]: starting mds.i at 2024-03-28T16:00:50.101 INFO:journalctl@ceph.mds.i.smithi183.stdout:Mar 28 16:00:49 smithi183 bash[66504]: 999fdaab14075feb5960df431524dd27359a5e8a477a34b884dd583e3698ce82 2024-03-28T16:00:50.101 INFO:journalctl@ceph.mds.i.smithi183.stdout:Mar 28 16:00:49 smithi183 systemd[1]: Started Ceph mds.i for b5f8225a-ed1a-11ee-95d0-87774f69a715. 2024-03-28T16:00:50.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:50.271+0000 7f219b400640 1 -- 172.21.15.67:0/1808730432 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21940328b0 con 0x7f219c109560 2024-03-28T16:00:50.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:50.271+0000 7f21a3c00640 1 -- 172.21.15.67:0/1808730432 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f219c109560 msgr2=0x7f219c10b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:50.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:50.271+0000 7f21a3c00640 1 --2- 172.21.15.67:0/1808730432 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f219c109560 0x7f219c10b9d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f2194009ca0 tx=0x7f219402c0f0 comp rx=0 tx=0).stop 2024-03-28T16:00:50.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:50.271+0000 7f21a3c00640 1 -- 172.21.15.67:0/1808730432 shutdown_connections 2024-03-28T16:00:50.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:50.271+0000 7f21a3c00640 1 --2- 172.21.15.67:0/1808730432 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f219c109560 0x7f219c10b9d0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:50.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:50.271+0000 7f21a3c00640 1 --2- 172.21.15.67:0/1808730432 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f219c0740c0 0x7f219c074540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:50.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:50.271+0000 7f21a3c00640 1 --2- 172.21.15.67:0/1808730432 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f219c0756a0 0x7f219c073af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:50.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:50.271+0000 7f21a3c00640 1 -- 172.21.15.67:0/1808730432 >> 172.21.15.67:0/1808730432 conn(0x7f219c0fcc80 msgr2=0x7f219c0ff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:00:50.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:50.272+0000 7f21a3c00640 1 -- 172.21.15.67:0/1808730432 shutdown_connections 2024-03-28T16:00:50.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:50.272+0000 7f21a3c00640 1 -- 172.21.15.67:0/1808730432 wait complete. 2024-03-28T16:00:50.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:50.273+0000 7f21a3c00640 1 Processor -- start 2024-03-28T16:00:50.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:50.273+0000 7f21a3c00640 1 -- start start 2024-03-28T16:00:50.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:50.273+0000 7f21a3c00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f219c0740c0 0x7f219c199a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:50.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:50.273+0000 7f21a3c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f219c0756a0 0x7f219c199f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:50.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:50.273+0000 7f21a3c00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f219c109560 0x7f219c1a0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:50.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:50.273+0000 7f21a3c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f219c10f3c0 con 0x7f219c0756a0 2024-03-28T16:00:50.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:50.273+0000 7f21a3c00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f219c10f240 con 0x7f219c109560 2024-03-28T16:00:50.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:50.273+0000 7f21a3c00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f219c10f540 con 0x7f219c0740c0 2024-03-28T16:00:50.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:50.273+0000 7f219be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f219c0756a0 0x7f219c199f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:50.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:50.274+0000 7f219be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f219c0756a0 0x7f219c199f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57844/0 (socket says 172.21.15.67:57844) 2024-03-28T16:00:50.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:50.274+0000 7f219be00640 1 -- 172.21.15.67:0/3272041588 learned_addr learned my addr 172.21.15.67:0/3272041588 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:00:50.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:50.274+0000 7f21a1200640 1 --2- 172.21.15.67:0/3272041588 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f219c0740c0 0x7f219c199a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:50.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:50.274+0000 7f21a1c00640 1 --2- 172.21.15.67:0/3272041588 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f219c109560 0x7f219c1a0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:50.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:50.274+0000 7f219be00640 1 -- 172.21.15.67:0/3272041588 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f219c0740c0 msgr2=0x7f219c199a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:50.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:50.274+0000 7f219be00640 1 --2- 172.21.15.67:0/3272041588 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f219c0740c0 0x7f219c199a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:50.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:50.274+0000 7f219be00640 1 -- 172.21.15.67:0/3272041588 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f219c109560 msgr2=0x7f219c1a0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:50.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:50.274+0000 7f219be00640 1 --2- 172.21.15.67:0/3272041588 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f219c109560 0x7f219c1a0fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:50.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:50.274+0000 7f219be00640 1 -- 172.21.15.67:0/3272041588 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f219c1a1640 con 0x7f219c0756a0 2024-03-28T16:00:50.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:50.274+0000 7f21a1c00640 1 --2- 172.21.15.67:0/3272041588 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f219c109560 0x7f219c1a0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:00:50.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:50.274+0000 7f219be00640 1 --2- 172.21.15.67:0/3272041588 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f219c0756a0 0x7f219c199f60 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f218800d770 tx=0x7f218800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:00:50.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:50.275+0000 7f2199600640 1 -- 172.21.15.67:0/3272041588 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2188017070 con 0x7f219c0756a0 2024-03-28T16:00:50.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:50.275+0000 7f2199600640 1 -- 172.21.15.67:0/3272041588 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f21880054d0 con 0x7f219c0756a0 2024-03-28T16:00:50.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:50.275+0000 7f2199600640 1 -- 172.21.15.67:0/3272041588 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2188004cf0 con 0x7f219c0756a0 2024-03-28T16:00:50.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:50.275+0000 7f21a3c00640 1 -- 172.21.15.67:0/3272041588 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f219c1a1930 con 0x7f219c0756a0 2024-03-28T16:00:50.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:50.275+0000 7f21a3c00640 1 -- 172.21.15.67:0/3272041588 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f219c1a1e80 con 0x7f219c0756a0 2024-03-28T16:00:50.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:50.276+0000 7f21a3c00640 1 -- 172.21.15.67:0/3272041588 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f219c1082e0 con 0x7f219c0756a0 2024-03-28T16:00:50.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:50.278+0000 7f2199600640 1 -- 172.21.15.67:0/3272041588 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f2188004e90 con 0x7f219c0756a0 2024-03-28T16:00:50.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:50.281+0000 7f2199600640 1 --2- 172.21.15.67:0/3272041588 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f21840a2880 0x7f21840a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:50.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:50.281+0000 7f2199600640 1 -- 172.21.15.67:0/3272041588 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f21880ce0f0 con 0x7f219c0756a0 2024-03-28T16:00:50.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:50.281+0000 7f21a1200640 1 --2- 172.21.15.67:0/3272041588 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f21840a2880 0x7f21840a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:50.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:50.282+0000 7f21a1200640 1 --2- 172.21.15.67:0/3272041588 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f21840a2880 0x7f21840a4d40 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f2190005e10 tx=0x7f2190005da0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:00:50.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:50.286+0000 7f2199600640 1 -- 172.21.15.67:0/3272041588 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f218809b590 con 0x7f219c0756a0 2024-03-28T16:00:50.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:50.535+0000 7f21a3c00640 1 -- 172.21.15.67:0/3272041588 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f219c02d070 con 0x7f219c0756a0 2024-03-28T16:00:50.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:50.536+0000 7f2199600640 1 -- 172.21.15.67:0/3272041588 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7673 (secure 0 0 0) 0x7f218809ae90 con 0x7f219c0756a0 2024-03-28T16:00:50.537 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:00:50.537 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2008297659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2008297659},{"type":"v1","addr":"172.21.15.67:6835","nonce":2008297659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14532,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/320295745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":320295745},{"type":"v1","addr":"172.21.15.67:6837","nonce":320295745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2129665423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2129665423},{"type":"v1","addr":"172.21.15.183:6833","nonce":2129665423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24496,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/1589784425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":1589784425},{"type":"v1","addr":"172.21.15.183:6835","nonce":1589784425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24508,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/2185863835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":2185863835},{"type":"v1","addr":"172.21.15.183:6837","nonce":2185863835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24520,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6839/1242814398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":1242814398},{"type":"v1","addr":"172.21.15.183:6839","nonce":1242814398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":34409,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1008335521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1008335521},{"type":"v1","addr":"172.21.15.150:6833","nonce":1008335521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":34445,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3269935695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3269935695},{"type":"v1","addr":"172.21.15.150:6835","nonce":3269935695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":34475,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1112886277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":1112886277},{"type":"v1","addr":"172.21.15.150:6837","nonce":1112886277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[]} 2024-03-28T16:00:50.537 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 10 2024-03-28T16:00:50.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:50.539+0000 7f21a3c00640 1 -- 172.21.15.67:0/3272041588 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f21840a2880 msgr2=0x7f21840a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:50.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:50.539+0000 7f21a3c00640 1 --2- 172.21.15.67:0/3272041588 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f21840a2880 0x7f21840a4d40 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f2190005e10 tx=0x7f2190005da0 comp rx=0 tx=0).stop 2024-03-28T16:00:50.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:50.539+0000 7f21a3c00640 1 -- 172.21.15.67:0/3272041588 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f219c0756a0 msgr2=0x7f219c199f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:50.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:50.539+0000 7f21a3c00640 1 --2- 172.21.15.67:0/3272041588 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f219c0756a0 0x7f219c199f60 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f218800d770 tx=0x7f218800dc30 comp rx=0 tx=0).stop 2024-03-28T16:00:50.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:50.539+0000 7f21a3c00640 1 -- 172.21.15.67:0/3272041588 shutdown_connections 2024-03-28T16:00:50.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:50.539+0000 7f21a3c00640 1 --2- 172.21.15.67:0/3272041588 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f21840a2880 0x7f21840a4d40 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:50.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:50.539+0000 7f21a3c00640 1 --2- 172.21.15.67:0/3272041588 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f219c109560 0x7f219c1a0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:50.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:50.539+0000 7f21a3c00640 1 --2- 172.21.15.67:0/3272041588 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f219c0756a0 0x7f219c199f60 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:50.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:50.539+0000 7f21a3c00640 1 --2- 172.21.15.67:0/3272041588 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f219c0740c0 0x7f219c199a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:50.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:50.539+0000 7f21a3c00640 1 -- 172.21.15.67:0/3272041588 >> 172.21.15.67:0/3272041588 conn(0x7f219c0fcc80 msgr2=0x7f219c1054b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:00:50.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:50.539+0000 7f21a3c00640 1 -- 172.21.15.67:0/3272041588 shutdown_connections 2024-03-28T16:00:50.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:50.540+0000 7f21a3c00640 1 -- 172.21.15.67:0/3272041588 wait complete. 2024-03-28T16:00:50.557 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:50 smithi067 ceph-mon[31707]: pgmap v351: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:50.557 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:50 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:50.557 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:50 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:50.557 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:50 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:50.557 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:50 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T16:00:50.557 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:50 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T16:00:50.557 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:50 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:00:50.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:50 smithi150 ceph-mon[34825]: pgmap v351: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:50.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:50 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:50.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:50 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:50.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:50 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:50.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:50 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T16:00:50.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:50 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T16:00:50.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:50 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:00:50.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:50 smithi183 ceph-mon[35580]: pgmap v351: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:50.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:50 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:50.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:50 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:50.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:50 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:50.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:50 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T16:00:50.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:50 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T16:00:50.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:50 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:00:51.644 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:51 smithi067 ceph-mon[31707]: Deploying daemon mds.g on smithi067 2024-03-28T16:00:51.644 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:51 smithi067 ceph-mon[31707]: mds.? [v2:172.21.15.183:6838/1242814398,v1:172.21.15.183:6839/1242814398] up:boot 2024-03-28T16:00:51.645 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:51 smithi067 ceph-mon[31707]: fsmap 9 up:standby 2024-03-28T16:00:51.645 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:51 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T16:00:51.645 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:51 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/3272041588' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:00:51.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:51 smithi150 ceph-mon[34825]: Deploying daemon mds.g on smithi067 2024-03-28T16:00:51.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:51 smithi150 ceph-mon[34825]: mds.? [v2:172.21.15.183:6838/1242814398,v1:172.21.15.183:6839/1242814398] up:boot 2024-03-28T16:00:51.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:51 smithi150 ceph-mon[34825]: fsmap 9 up:standby 2024-03-28T16:00:51.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:51 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T16:00:51.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:51 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/3272041588' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:00:51.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:51 smithi183 ceph-mon[35580]: Deploying daemon mds.g on smithi067 2024-03-28T16:00:51.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:51 smithi183 ceph-mon[35580]: mds.? [v2:172.21.15.183:6838/1242814398,v1:172.21.15.183:6839/1242814398] up:boot 2024-03-28T16:00:51.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:51 smithi183 ceph-mon[35580]: fsmap 9 up:standby 2024-03-28T16:00:51.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:51 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T16:00:51.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:51 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/3272041588' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:00:52.561 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T16:00:52.653 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:52 smithi067 ceph-mon[31707]: pgmap v352: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:52.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:52 smithi150 ceph-mon[34825]: pgmap v352: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:52.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:52 smithi183 ceph-mon[35580]: pgmap v352: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:52.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:52.946+0000 7f2349600640 1 -- 172.21.15.67:0/3000529780 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23441016b0 msgr2=0x7f2344101ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:52.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:52.946+0000 7f2349600640 1 --2- 172.21.15.67:0/3000529780 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23441016b0 0x7f2344101ad0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f2334009950 tx=0x7f233402bcb0 comp rx=0 tx=0).stop 2024-03-28T16:00:52.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:52.946+0000 7f2349600640 1 -- 172.21.15.67:0/3000529780 shutdown_connections 2024-03-28T16:00:52.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:52.946+0000 7f2349600640 1 --2- 172.21.15.67:0/3000529780 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2344103260 0x7f2344109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:52.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:52.946+0000 7f2349600640 1 --2- 172.21.15.67:0/3000529780 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f23441028a0 0x7f2344102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:52.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:52.946+0000 7f2349600640 1 --2- 172.21.15.67:0/3000529780 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23441016b0 0x7f2344101ad0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:52.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:52.946+0000 7f2349600640 1 -- 172.21.15.67:0/3000529780 >> 172.21.15.67:0/3000529780 conn(0x7f23440fcc80 msgr2=0x7f23440ff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:00:52.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:52.947+0000 7f2349600640 1 -- 172.21.15.67:0/3000529780 shutdown_connections 2024-03-28T16:00:52.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:52.947+0000 7f2349600640 1 -- 172.21.15.67:0/3000529780 wait complete. 2024-03-28T16:00:52.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:52.948+0000 7f2349600640 1 Processor -- start 2024-03-28T16:00:52.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:52.948+0000 7f2349600640 1 -- start start 2024-03-28T16:00:52.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:52.949+0000 7f2349600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23441028a0 0x7f234419e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:52.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:52.949+0000 7f2349600640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2344103260 0x7f234419e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:52.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:52.949+0000 7f2349600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f23441a3770 0x7f23441a5b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:52.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:52.949+0000 7f2349600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f234410d2a0 con 0x7f23441028a0 2024-03-28T16:00:52.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:52.949+0000 7f2349600640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f234410d120 con 0x7f2344103260 2024-03-28T16:00:52.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:52.949+0000 7f2349600640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f234410d420 con 0x7f23441a3770 2024-03-28T16:00:52.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:52.949+0000 7f2342a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23441028a0 0x7f234419e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:52.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:52.950+0000 7f2342a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23441028a0 0x7f234419e1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57864/0 (socket says 172.21.15.67:57864) 2024-03-28T16:00:52.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:52.950+0000 7f2342a00640 1 -- 172.21.15.67:0/1800906275 learned_addr learned my addr 172.21.15.67:0/1800906275 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:00:52.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:52.950+0000 7f2343400640 1 --2- 172.21.15.67:0/1800906275 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f23441a3770 0x7f23441a5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:52.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:52.950+0000 7f2342000640 1 --2- 172.21.15.67:0/1800906275 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2344103260 0x7f234419e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:52.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:52.950+0000 7f2342a00640 1 -- 172.21.15.67:0/1800906275 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f23441a3770 msgr2=0x7f23441a5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:52.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:52.950+0000 7f2342a00640 1 --2- 172.21.15.67:0/1800906275 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f23441a3770 0x7f23441a5b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:52.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:52.950+0000 7f2342a00640 1 -- 172.21.15.67:0/1800906275 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2344103260 msgr2=0x7f234419e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:52.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:52.950+0000 7f2342a00640 1 --2- 172.21.15.67:0/1800906275 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2344103260 0x7f234419e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:52.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:52.950+0000 7f2342a00640 1 -- 172.21.15.67:0/1800906275 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23441a61d0 con 0x7f23441028a0 2024-03-28T16:00:52.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:52.951+0000 7f2342a00640 1 --2- 172.21.15.67:0/1800906275 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23441028a0 0x7f234419e1d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f23340098a0 tx=0x7f2334004e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:00:52.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:52.951+0000 7f233b400640 1 -- 172.21.15.67:0/1800906275 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2334043020 con 0x7f23441028a0 2024-03-28T16:00:52.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:52.951+0000 7f233b400640 1 -- 172.21.15.67:0/1800906275 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f23340044d0 con 0x7f23441028a0 2024-03-28T16:00:52.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:52.951+0000 7f233b400640 1 -- 172.21.15.67:0/1800906275 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2334003e20 con 0x7f23441028a0 2024-03-28T16:00:52.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:52.951+0000 7f2349600640 1 -- 172.21.15.67:0/1800906275 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23441a6460 con 0x7f23441028a0 2024-03-28T16:00:52.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:52.952+0000 7f2349600640 1 -- 172.21.15.67:0/1800906275 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23441a6930 con 0x7f23441028a0 2024-03-28T16:00:52.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:52.954+0000 7f2349600640 1 -- 172.21.15.67:0/1800906275 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23440688e0 con 0x7f23441028a0 2024-03-28T16:00:52.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:52.956+0000 7f233b400640 1 -- 172.21.15.67:0/1800906275 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f2334046020 con 0x7f23441028a0 2024-03-28T16:00:52.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:52.958+0000 7f233b400640 1 --2- 172.21.15.67:0/1800906275 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f23280a2880 0x7f23280a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:52.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:52.959+0000 7f233b400640 1 -- 172.21.15.67:0/1800906275 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f23340f0af0 con 0x7f23441028a0 2024-03-28T16:00:52.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:52.959+0000 7f2342000640 1 --2- 172.21.15.67:0/1800906275 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f23280a2880 0x7f23280a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:52.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:52.960+0000 7f2342000640 1 --2- 172.21.15.67:0/1800906275 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f23280a2880 0x7f23280a4d40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f234419f6e0 tx=0x7f23300090e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:00:52.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:52.966+0000 7f233b400640 1 -- 172.21.15.67:0/1800906275 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f233404d020 con 0x7f23441028a0 2024-03-28T16:00:53.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:53.220+0000 7f2349600640 1 -- 172.21.15.67:0/1800906275 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f234402d420 con 0x7f23441028a0 2024-03-28T16:00:53.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:53.221+0000 7f233b400640 1 -- 172.21.15.67:0/1800906275 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7673 (secure 0 0 0) 0x7f23340be020 con 0x7f23441028a0 2024-03-28T16:00:53.222 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:00:53.222 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2008297659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2008297659},{"type":"v1","addr":"172.21.15.67:6835","nonce":2008297659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14532,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/320295745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":320295745},{"type":"v1","addr":"172.21.15.67:6837","nonce":320295745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2129665423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2129665423},{"type":"v1","addr":"172.21.15.183:6833","nonce":2129665423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24496,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/1589784425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":1589784425},{"type":"v1","addr":"172.21.15.183:6835","nonce":1589784425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24508,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/2185863835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":2185863835},{"type":"v1","addr":"172.21.15.183:6837","nonce":2185863835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24520,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6839/1242814398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":1242814398},{"type":"v1","addr":"172.21.15.183:6839","nonce":1242814398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":34409,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1008335521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1008335521},{"type":"v1","addr":"172.21.15.150:6833","nonce":1008335521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":34445,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3269935695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3269935695},{"type":"v1","addr":"172.21.15.150:6835","nonce":3269935695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":34475,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1112886277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":1112886277},{"type":"v1","addr":"172.21.15.150:6837","nonce":1112886277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[]} 2024-03-28T16:00:53.222 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 10 2024-03-28T16:00:53.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:53.224+0000 7f2349600640 1 -- 172.21.15.67:0/1800906275 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f23280a2880 msgr2=0x7f23280a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:53.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:53.224+0000 7f2349600640 1 --2- 172.21.15.67:0/1800906275 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f23280a2880 0x7f23280a4d40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f234419f6e0 tx=0x7f23300090e0 comp rx=0 tx=0).stop 2024-03-28T16:00:53.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:53.224+0000 7f2349600640 1 -- 172.21.15.67:0/1800906275 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23441028a0 msgr2=0x7f234419e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:53.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:53.224+0000 7f2349600640 1 --2- 172.21.15.67:0/1800906275 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23441028a0 0x7f234419e1d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f23340098a0 tx=0x7f2334004e00 comp rx=0 tx=0).stop 2024-03-28T16:00:53.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:53.224+0000 7f2349600640 1 -- 172.21.15.67:0/1800906275 shutdown_connections 2024-03-28T16:00:53.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:53.224+0000 7f2349600640 1 --2- 172.21.15.67:0/1800906275 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f23280a2880 0x7f23280a4d40 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:53.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:53.224+0000 7f2349600640 1 --2- 172.21.15.67:0/1800906275 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f23441a3770 0x7f23441a5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:53.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:53.224+0000 7f2349600640 1 --2- 172.21.15.67:0/1800906275 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2344103260 0x7f234419e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:53.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:53.224+0000 7f2349600640 1 --2- 172.21.15.67:0/1800906275 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23441028a0 0x7f234419e1d0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:53.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:53.224+0000 7f2349600640 1 -- 172.21.15.67:0/1800906275 >> 172.21.15.67:0/1800906275 conn(0x7f23440fcc80 msgr2=0x7f2344103da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:00:53.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:53.224+0000 7f2349600640 1 -- 172.21.15.67:0/1800906275 shutdown_connections 2024-03-28T16:00:53.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:53.224+0000 7f2349600640 1 -- 172.21.15.67:0/1800906275 wait complete. 2024-03-28T16:00:53.565 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:53 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/1800906275' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:00:53.752 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:53 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/1800906275' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:00:53.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:53 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/1800906275' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:00:53.820 INFO:journalctl@ceph.mds.g.smithi067.stdout:Mar 28 16:00:53 smithi067 podman[73333]: 2024-03-28 16:00:53.599028521 +0000 UTC m=+0.166792955 container create a38fb47a48acf7de183e2f918df2ffb5714a08dbca0f7e3e6ff41eef037f214c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-g, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T16:00:54.320 INFO:journalctl@ceph.mds.g.smithi067.stdout:Mar 28 16:00:53 smithi067 podman[73333]: 2024-03-28 16:00:53.894457084 +0000 UTC m=+0.462221525 container init a38fb47a48acf7de183e2f918df2ffb5714a08dbca0f7e3e6ff41eef037f214c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-g, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True) 2024-03-28T16:00:54.320 INFO:journalctl@ceph.mds.g.smithi067.stdout:Mar 28 16:00:53 smithi067 podman[73333]: 2024-03-28 16:00:53.906188983 +0000 UTC m=+0.473953470 container start a38fb47a48acf7de183e2f918df2ffb5714a08dbca0f7e3e6ff41eef037f214c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-g, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-28T16:00:54.320 INFO:journalctl@ceph.mds.g.smithi067.stdout:Mar 28 16:00:53 smithi067 ceph-mds[73352]: -- 172.21.15.67:0/3359084005 <== mon.0 v2:172.21.15.67:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56464dbd3a40 con 0x56464dba7000 2024-03-28T16:00:54.320 INFO:journalctl@ceph.mds.g.smithi067.stdout:Mar 28 16:00:53 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-g[73348]: starting mds.g at 2024-03-28T16:00:54.320 INFO:journalctl@ceph.mds.g.smithi067.stdout:Mar 28 16:00:53 smithi067 bash[73333]: a38fb47a48acf7de183e2f918df2ffb5714a08dbca0f7e3e6ff41eef037f214c 2024-03-28T16:00:54.320 INFO:journalctl@ceph.mds.g.smithi067.stdout:Mar 28 16:00:53 smithi067 systemd[1]: Started Ceph mds.g for b5f8225a-ed1a-11ee-95d0-87774f69a715. 2024-03-28T16:00:54.781 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:54 smithi183 ceph-mon[35580]: pgmap v353: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:54.781 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:54 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:54.781 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:54 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:54.781 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:54 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:54.781 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:54 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T16:00:54.781 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:54 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T16:00:54.781 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:54 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:00:54.810 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:54 smithi150 ceph-mon[34825]: pgmap v353: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:54.810 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:54 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:54.810 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:54 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:54.811 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:54 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:54.811 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:54 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T16:00:54.811 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:54 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T16:00:54.811 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:54 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:00:54.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:54 smithi067 ceph-mon[31707]: pgmap v353: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:54.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:54 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:54.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:54 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:54.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:54 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:54.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:54 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T16:00:54.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:54 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T16:00:54.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:54 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:00:55.252 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T16:00:55.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:55.491+0000 7fd82de00640 1 -- 172.21.15.67:0/1307339764 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd828101060 msgr2=0x7fd8281014e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:55.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:55.491+0000 7fd82de00640 1 --2- 172.21.15.67:0/1307339764 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd828101060 0x7fd8281014e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd820009900 tx=0x7fd82002bc30 comp rx=0 tx=0).stop 2024-03-28T16:00:55.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:55.491+0000 7fd82de00640 1 -- 172.21.15.67:0/1307339764 shutdown_connections 2024-03-28T16:00:55.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:55.491+0000 7fd82de00640 1 --2- 172.21.15.67:0/1307339764 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd828101060 0x7fd8281014e0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:55.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:55.491+0000 7fd82de00640 1 --2- 172.21.15.67:0/1307339764 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8281006c0 0x7fd828100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:55.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:55.491+0000 7fd82de00640 1 --2- 172.21.15.67:0/1307339764 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd8280ffd60 0x7fd828100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:55.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:55.491+0000 7fd82de00640 1 -- 172.21.15.67:0/1307339764 >> 172.21.15.67:0/1307339764 conn(0x7fd8280fb980 msgr2=0x7fd8280fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:00:55.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:55.492+0000 7fd82de00640 1 -- 172.21.15.67:0/1307339764 shutdown_connections 2024-03-28T16:00:55.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:55.492+0000 7fd82de00640 1 -- 172.21.15.67:0/1307339764 wait complete. 2024-03-28T16:00:55.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:55.492+0000 7fd82de00640 1 Processor -- start 2024-03-28T16:00:55.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:55.493+0000 7fd82de00640 1 -- start start 2024-03-28T16:00:55.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:55.493+0000 7fd82de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8280ffd60 0x7fd828199890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:55.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:55.493+0000 7fd82de00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd8281006c0 0x7fd828199dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:55.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:55.493+0000 7fd82de00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd828101060 0x7fd8281a0e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:55.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:55.493+0000 7fd82de00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd82810f1f0 con 0x7fd8280ffd60 2024-03-28T16:00:55.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:55.493+0000 7fd82de00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd82810f070 con 0x7fd828101060 2024-03-28T16:00:55.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:55.493+0000 7fd82de00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd82810f370 con 0x7fd8281006c0 2024-03-28T16:00:55.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:55.493+0000 7fd82ca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8280ffd60 0x7fd828199890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:55.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:55.493+0000 7fd82ca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8280ffd60 0x7fd828199890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57908/0 (socket says 172.21.15.67:57908) 2024-03-28T16:00:55.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:55.493+0000 7fd82ca00640 1 -- 172.21.15.67:0/2328705633 learned_addr learned my addr 172.21.15.67:0/2328705633 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:00:55.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:55.493+0000 7fd827e00640 1 --2- 172.21.15.67:0/2328705633 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd8281006c0 0x7fd828199dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:55.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:55.494+0000 7fd82d400640 1 --2- 172.21.15.67:0/2328705633 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd828101060 0x7fd8281a0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:55.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:55.494+0000 7fd82ca00640 1 -- 172.21.15.67:0/2328705633 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd8281006c0 msgr2=0x7fd828199dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:55.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:55.494+0000 7fd82ca00640 1 --2- 172.21.15.67:0/2328705633 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd8281006c0 0x7fd828199dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:55.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:55.494+0000 7fd82ca00640 1 -- 172.21.15.67:0/2328705633 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd828101060 msgr2=0x7fd8281a0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:55.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:55.494+0000 7fd82ca00640 1 --2- 172.21.15.67:0/2328705633 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd828101060 0x7fd8281a0e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:55.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:55.494+0000 7fd82ca00640 1 -- 172.21.15.67:0/2328705633 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8281a14b0 con 0x7fd8280ffd60 2024-03-28T16:00:55.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:55.494+0000 7fd82ca00640 1 --2- 172.21.15.67:0/2328705633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8280ffd60 0x7fd828199890 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fd81800eab0 tx=0x7fd81800ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:00:55.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:55.494+0000 7fd825600640 1 -- 172.21.15.67:0/2328705633 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd81800ce40 con 0x7fd8280ffd60 2024-03-28T16:00:55.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:55.494+0000 7fd825600640 1 -- 172.21.15.67:0/2328705633 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd818011920 con 0x7fd8280ffd60 2024-03-28T16:00:55.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:55.494+0000 7fd825600640 1 -- 172.21.15.67:0/2328705633 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd818010a00 con 0x7fd8280ffd60 2024-03-28T16:00:55.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:55.494+0000 7fd82de00640 1 -- 172.21.15.67:0/2328705633 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8281a17a0 con 0x7fd8280ffd60 2024-03-28T16:00:55.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:55.494+0000 7fd82de00640 1 -- 172.21.15.67:0/2328705633 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8281a1d40 con 0x7fd8280ffd60 2024-03-28T16:00:55.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:55.496+0000 7fd82de00640 1 -- 172.21.15.67:0/2328705633 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd82810aab0 con 0x7fd8280ffd60 2024-03-28T16:00:55.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:55.497+0000 7fd825600640 1 -- 172.21.15.67:0/2328705633 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fd818011ad0 con 0x7fd8280ffd60 2024-03-28T16:00:55.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:55.500+0000 7fd825600640 1 --2- 172.21.15.67:0/2328705633 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fd80c0a2880 0x7fd80c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:55.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:55.500+0000 7fd825600640 1 -- 172.21.15.67:0/2328705633 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fd8180ced10 con 0x7fd8280ffd60 2024-03-28T16:00:55.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:55.500+0000 7fd827e00640 1 --2- 172.21.15.67:0/2328705633 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fd80c0a2880 0x7fd80c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:55.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:55.501+0000 7fd827e00640 1 --2- 172.21.15.67:0/2328705633 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fd80c0a2880 0x7fd80c0a4d40 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd810005fd0 tx=0x7fd810005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:00:55.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:55.505+0000 7fd825600640 1 -- 172.21.15.67:0/2328705633 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd8180d4020 con 0x7fd8280ffd60 2024-03-28T16:00:55.522 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:55 smithi067 ceph-mon[31707]: Deploying daemon mds.a on smithi067 2024-03-28T16:00:55.523 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:55 smithi067 ceph-mon[31707]: mds.? [v2:172.21.15.67:6838/691821273,v1:172.21.15.67:6839/691821273] up:boot 2024-03-28T16:00:55.523 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:55 smithi067 ceph-mon[31707]: fsmap 10 up:standby 2024-03-28T16:00:55.523 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:55 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T16:00:55.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:55.751+0000 7fd82de00640 1 -- 172.21.15.67:0/2328705633 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd82802d070 con 0x7fd8280ffd60 2024-03-28T16:00:55.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:55.753+0000 7fd825600640 1 -- 172.21.15.67:0/2328705633 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8462 (secure 0 0 0) 0x7fd818016030 con 0x7fd8280ffd60 2024-03-28T16:00:55.755 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:00:55.755 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2008297659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2008297659},{"type":"v1","addr":"172.21.15.67:6835","nonce":2008297659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14532,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/320295745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":320295745},{"type":"v1","addr":"172.21.15.67:6837","nonce":320295745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6839/691821273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":691821273},{"type":"v1","addr":"172.21.15.67:6839","nonce":691821273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2129665423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2129665423},{"type":"v1","addr":"172.21.15.183:6833","nonce":2129665423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24496,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/1589784425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":1589784425},{"type":"v1","addr":"172.21.15.183:6835","nonce":1589784425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24508,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/2185863835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":2185863835},{"type":"v1","addr":"172.21.15.183:6837","nonce":2185863835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24520,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6839/1242814398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":1242814398},{"type":"v1","addr":"172.21.15.183:6839","nonce":1242814398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":34409,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1008335521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1008335521},{"type":"v1","addr":"172.21.15.150:6833","nonce":1008335521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":34445,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3269935695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3269935695},{"type":"v1","addr":"172.21.15.150:6835","nonce":3269935695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":34475,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1112886277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":1112886277},{"type":"v1","addr":"172.21.15.150:6837","nonce":1112886277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[]} 2024-03-28T16:00:55.755 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 11 2024-03-28T16:00:55.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:55.756+0000 7fd82de00640 1 -- 172.21.15.67:0/2328705633 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fd80c0a2880 msgr2=0x7fd80c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:55.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:55.756+0000 7fd82de00640 1 --2- 172.21.15.67:0/2328705633 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fd80c0a2880 0x7fd80c0a4d40 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd810005fd0 tx=0x7fd810005850 comp rx=0 tx=0).stop 2024-03-28T16:00:55.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:55.756+0000 7fd82de00640 1 -- 172.21.15.67:0/2328705633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8280ffd60 msgr2=0x7fd828199890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:55.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:55.756+0000 7fd82de00640 1 --2- 172.21.15.67:0/2328705633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8280ffd60 0x7fd828199890 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fd81800eab0 tx=0x7fd81800ef70 comp rx=0 tx=0).stop 2024-03-28T16:00:55.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:55.756+0000 7fd82de00640 1 -- 172.21.15.67:0/2328705633 shutdown_connections 2024-03-28T16:00:55.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:55.756+0000 7fd82de00640 1 --2- 172.21.15.67:0/2328705633 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fd80c0a2880 0x7fd80c0a4d40 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:55.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:55.756+0000 7fd82de00640 1 --2- 172.21.15.67:0/2328705633 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd828101060 0x7fd8281a0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:55.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:55.756+0000 7fd82de00640 1 --2- 172.21.15.67:0/2328705633 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd8281006c0 0x7fd828199dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:55.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:55.756+0000 7fd82de00640 1 --2- 172.21.15.67:0/2328705633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8280ffd60 0x7fd828199890 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:55.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:55.756+0000 7fd82de00640 1 -- 172.21.15.67:0/2328705633 >> 172.21.15.67:0/2328705633 conn(0x7fd8280fb980 msgr2=0x7fd8280fd480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:00:55.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:55.756+0000 7fd82de00640 1 -- 172.21.15.67:0/2328705633 shutdown_connections 2024-03-28T16:00:55.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:55.756+0000 7fd82de00640 1 -- 172.21.15.67:0/2328705633 wait complete. 2024-03-28T16:00:55.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:55 smithi150 ceph-mon[34825]: Deploying daemon mds.a on smithi067 2024-03-28T16:00:55.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:55 smithi150 ceph-mon[34825]: mds.? [v2:172.21.15.67:6838/691821273,v1:172.21.15.67:6839/691821273] up:boot 2024-03-28T16:00:55.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:55 smithi150 ceph-mon[34825]: fsmap 10 up:standby 2024-03-28T16:00:55.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:55 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T16:00:55.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:55 smithi183 ceph-mon[35580]: Deploying daemon mds.a on smithi067 2024-03-28T16:00:55.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:55 smithi183 ceph-mon[35580]: mds.? [v2:172.21.15.67:6838/691821273,v1:172.21.15.67:6839/691821273] up:boot 2024-03-28T16:00:55.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:55 smithi183 ceph-mon[35580]: fsmap 10 up:standby 2024-03-28T16:00:55.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:55 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T16:00:56.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:56 smithi183 ceph-mon[35580]: pgmap v354: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:56.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:56 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/2328705633' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:00:56.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:56 smithi150 ceph-mon[34825]: pgmap v354: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:56.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:56 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/2328705633' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:00:56.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:56 smithi067 ceph-mon[31707]: pgmap v354: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:56.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:56 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/2328705633' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:00:57.556 INFO:journalctl@ceph.mds.a.smithi067.stdout:Mar 28 16:00:57 smithi067 systemd[1]: Starting Ceph mds.a for b5f8225a-ed1a-11ee-95d0-87774f69a715... 2024-03-28T16:00:57.780 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T16:00:57.819 INFO:journalctl@ceph.mds.a.smithi067.stdout:Mar 28 16:00:57 smithi067 podman[73680]: 2024-03-28 16:00:57.461820182 +0000 UTC m=+0.036163684 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T16:00:57.819 INFO:journalctl@ceph.mds.a.smithi067.stdout:Mar 28 16:00:57 smithi067 podman[73680]: 2024-03-28 16:00:57.560980105 +0000 UTC m=+0.135323641 container create 0d13172c8488f7ad865be21abfebfc4e6959d0b76871909e042f0d526032d664 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-a, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T16:00:58.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:58.010+0000 7f49bf400640 1 -- 172.21.15.67:0/2211428516 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49b81016c0 msgr2=0x7f49b8101ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:58.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:58.010+0000 7f49bf400640 1 --2- 172.21.15.67:0/2211428516 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49b81016c0 0x7f49b8101ae0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f49ac009900 tx=0x7f49ac02bc40 comp rx=0 tx=0).stop 2024-03-28T16:00:58.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:58.010+0000 7f49bf400640 1 -- 172.21.15.67:0/2211428516 shutdown_connections 2024-03-28T16:00:58.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:58.010+0000 7f49bf400640 1 --2- 172.21.15.67:0/2211428516 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f49b8103270 0x7f49b8109b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:58.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:58.010+0000 7f49bf400640 1 --2- 172.21.15.67:0/2211428516 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49b81028b0 0x7f49b8102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:58.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:58.010+0000 7f49bf400640 1 --2- 172.21.15.67:0/2211428516 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49b81016c0 0x7f49b8101ae0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:58.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:58.010+0000 7f49bf400640 1 -- 172.21.15.67:0/2211428516 >> 172.21.15.67:0/2211428516 conn(0x7f49b80fcc50 msgr2=0x7f49b80ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:00:58.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:58.010+0000 7f49bf400640 1 -- 172.21.15.67:0/2211428516 shutdown_connections 2024-03-28T16:00:58.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:58.010+0000 7f49bf400640 1 -- 172.21.15.67:0/2211428516 wait complete. 2024-03-28T16:00:58.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:58.011+0000 7f49bf400640 1 Processor -- start 2024-03-28T16:00:58.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:58.012+0000 7f49bf400640 1 -- start start 2024-03-28T16:00:58.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:58.012+0000 7f49bf400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49b81016c0 0x7f49b8199a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:58.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:58.012+0000 7f49bf400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49b81028b0 0x7f49b8199f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:58.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:58.012+0000 7f49bf400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f49b8103270 0x7f49b81a0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:58.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:58.012+0000 7f49bf400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49b810d1c0 con 0x7f49b81028b0 2024-03-28T16:00:58.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:58.012+0000 7f49bf400640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49b810d040 con 0x7f49b81016c0 2024-03-28T16:00:58.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:58.012+0000 7f49bf400640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49b810d340 con 0x7f49b8103270 2024-03-28T16:00:58.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:58.012+0000 7f49b7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49b81028b0 0x7f49b8199f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:58.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:58.012+0000 7f49b7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49b81028b0 0x7f49b8199f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57942/0 (socket says 172.21.15.67:57942) 2024-03-28T16:00:58.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:58.012+0000 7f49b7e00640 1 -- 172.21.15.67:0/3934734547 learned_addr learned my addr 172.21.15.67:0/3934734547 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:00:58.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:58.013+0000 7f49bca00640 1 --2- 172.21.15.67:0/3934734547 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49b81016c0 0x7f49b8199a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:58.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:58.013+0000 7f49bd400640 1 --2- 172.21.15.67:0/3934734547 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f49b8103270 0x7f49b81a0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:58.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:58.013+0000 7f49b7e00640 1 -- 172.21.15.67:0/3934734547 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f49b8103270 msgr2=0x7f49b81a0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:58.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:58.013+0000 7f49b7e00640 1 --2- 172.21.15.67:0/3934734547 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f49b8103270 0x7f49b81a0fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:58.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:58.013+0000 7f49b7e00640 1 -- 172.21.15.67:0/3934734547 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49b81016c0 msgr2=0x7f49b8199a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:58.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:58.013+0000 7f49b7e00640 1 --2- 172.21.15.67:0/3934734547 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49b81016c0 0x7f49b8199a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:58.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:58.013+0000 7f49b7e00640 1 -- 172.21.15.67:0/3934734547 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49b81a1620 con 0x7f49b81028b0 2024-03-28T16:00:58.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:58.013+0000 7f49bca00640 1 --2- 172.21.15.67:0/3934734547 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49b81016c0 0x7f49b8199a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:00:58.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:58.013+0000 7f49b7e00640 1 --2- 172.21.15.67:0/3934734547 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49b81028b0 0x7f49b8199f40 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f49a400c920 tx=0x7f49a400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:00:58.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:58.013+0000 7f49b5600640 1 -- 172.21.15.67:0/3934734547 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a4007c70 con 0x7f49b81028b0 2024-03-28T16:00:58.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:58.013+0000 7f49b5600640 1 -- 172.21.15.67:0/3934734547 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f49a4007e10 con 0x7f49b81028b0 2024-03-28T16:00:58.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:58.014+0000 7f49b5600640 1 -- 172.21.15.67:0/3934734547 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a4018820 con 0x7f49b81028b0 2024-03-28T16:00:58.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:58.014+0000 7f49bf400640 1 -- 172.21.15.67:0/3934734547 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49b81a1910 con 0x7f49b81028b0 2024-03-28T16:00:58.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:58.014+0000 7f49bf400640 1 -- 172.21.15.67:0/3934734547 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49b81a1e10 con 0x7f49b81028b0 2024-03-28T16:00:58.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:58.015+0000 7f49bf400640 1 -- 172.21.15.67:0/3934734547 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49b81a20a0 con 0x7f49b81028b0 2024-03-28T16:00:58.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:58.019+0000 7f49b5600640 1 -- 172.21.15.67:0/3934734547 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f49a40189c0 con 0x7f49b81028b0 2024-03-28T16:00:58.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:58.021+0000 7f49b5600640 1 --2- 172.21.15.67:0/3934734547 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f49a00a2880 0x7f49a00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:00:58.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:58.022+0000 7f49b5600640 1 -- 172.21.15.67:0/3934734547 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f49a40ce7e0 con 0x7f49b81028b0 2024-03-28T16:00:58.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:58.022+0000 7f49bca00640 1 --2- 172.21.15.67:0/3934734547 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f49a00a2880 0x7f49a00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:00:58.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:58.023+0000 7f49bca00640 1 --2- 172.21.15.67:0/3934734547 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f49a00a2880 0x7f49a00a4d40 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f49ac0096c0 tx=0x7f49ac000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:00:58.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:58.025+0000 7f49b5600640 1 -- 172.21.15.67:0/3934734547 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f49a4014030 con 0x7f49b81028b0 2024-03-28T16:00:58.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:58.276+0000 7f49bf400640 1 -- 172.21.15.67:0/3934734547 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f49b81a2310 con 0x7f49b81028b0 2024-03-28T16:00:58.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:58.277+0000 7f49b5600640 1 -- 172.21.15.67:0/3934734547 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8462 (secure 0 0 0) 0x7f49a409bc80 con 0x7f49b81028b0 2024-03-28T16:00:58.279 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:00:58.279 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2008297659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2008297659},{"type":"v1","addr":"172.21.15.67:6835","nonce":2008297659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14532,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/320295745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":320295745},{"type":"v1","addr":"172.21.15.67:6837","nonce":320295745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6839/691821273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":691821273},{"type":"v1","addr":"172.21.15.67:6839","nonce":691821273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2129665423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2129665423},{"type":"v1","addr":"172.21.15.183:6833","nonce":2129665423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24496,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/1589784425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":1589784425},{"type":"v1","addr":"172.21.15.183:6835","nonce":1589784425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24508,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/2185863835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":2185863835},{"type":"v1","addr":"172.21.15.183:6837","nonce":2185863835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24520,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6839/1242814398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":1242814398},{"type":"v1","addr":"172.21.15.183:6839","nonce":1242814398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":34409,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1008335521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1008335521},{"type":"v1","addr":"172.21.15.150:6833","nonce":1008335521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":34445,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3269935695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3269935695},{"type":"v1","addr":"172.21.15.150:6835","nonce":3269935695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":34475,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1112886277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":1112886277},{"type":"v1","addr":"172.21.15.150:6837","nonce":1112886277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[]} 2024-03-28T16:00:58.279 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 11 2024-03-28T16:00:58.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:58.280+0000 7f49bf400640 1 -- 172.21.15.67:0/3934734547 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f49a00a2880 msgr2=0x7f49a00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:58.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:58.280+0000 7f49bf400640 1 --2- 172.21.15.67:0/3934734547 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f49a00a2880 0x7f49a00a4d40 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f49ac0096c0 tx=0x7f49ac000bc0 comp rx=0 tx=0).stop 2024-03-28T16:00:58.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:58.280+0000 7f49bf400640 1 -- 172.21.15.67:0/3934734547 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49b81028b0 msgr2=0x7f49b8199f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:00:58.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:58.280+0000 7f49bf400640 1 --2- 172.21.15.67:0/3934734547 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49b81028b0 0x7f49b8199f40 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f49a400c920 tx=0x7f49a400cde0 comp rx=0 tx=0).stop 2024-03-28T16:00:58.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:58.281+0000 7f49bf400640 1 -- 172.21.15.67:0/3934734547 shutdown_connections 2024-03-28T16:00:58.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:58.281+0000 7f49bf400640 1 --2- 172.21.15.67:0/3934734547 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f49a00a2880 0x7f49a00a4d40 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:58.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:58.281+0000 7f49bf400640 1 --2- 172.21.15.67:0/3934734547 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f49b8103270 0x7f49b81a0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:58.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:58.281+0000 7f49bf400640 1 --2- 172.21.15.67:0/3934734547 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49b81028b0 0x7f49b8199f40 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:58.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:58.281+0000 7f49bf400640 1 --2- 172.21.15.67:0/3934734547 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49b81016c0 0x7f49b8199a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:00:58.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:58.281+0000 7f49bf400640 1 -- 172.21.15.67:0/3934734547 >> 172.21.15.67:0/3934734547 conn(0x7f49b80fcc50 msgr2=0x7f49b80738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:00:58.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:58.281+0000 7f49bf400640 1 -- 172.21.15.67:0/3934734547 shutdown_connections 2024-03-28T16:00:58.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:00:58.281+0000 7f49bf400640 1 -- 172.21.15.67:0/3934734547 wait complete. 2024-03-28T16:00:58.298 INFO:journalctl@ceph.mds.a.smithi067.stdout:Mar 28 16:00:57 smithi067 podman[73680]: 2024-03-28 16:00:57.847763443 +0000 UTC m=+0.422106950 container init 0d13172c8488f7ad865be21abfebfc4e6959d0b76871909e042f0d526032d664 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-a, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-28T16:00:58.299 INFO:journalctl@ceph.mds.a.smithi067.stdout:Mar 28 16:00:57 smithi067 podman[73680]: 2024-03-28 16:00:57.856410915 +0000 UTC m=+0.430754431 container start 0d13172c8488f7ad865be21abfebfc4e6959d0b76871909e042f0d526032d664 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-a, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, ceph=True) 2024-03-28T16:00:58.299 INFO:journalctl@ceph.mds.a.smithi067.stdout:Mar 28 16:00:57 smithi067 ceph-mds[73715]: -- 172.21.15.67:0/895360710 <== mon.0 v2:172.21.15.67:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f5bb2d1a40 con 0x55f5bb2a4c00 2024-03-28T16:00:58.299 INFO:journalctl@ceph.mds.a.smithi067.stdout:Mar 28 16:00:57 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-a[73695]: starting mds.a at 2024-03-28T16:00:58.299 INFO:journalctl@ceph.mds.a.smithi067.stdout:Mar 28 16:00:57 smithi067 bash[73680]: 0d13172c8488f7ad865be21abfebfc4e6959d0b76871909e042f0d526032d664 2024-03-28T16:00:58.299 INFO:journalctl@ceph.mds.a.smithi067.stdout:Mar 28 16:00:57 smithi067 systemd[1]: Started Ceph mds.a for b5f8225a-ed1a-11ee-95d0-87774f69a715. 2024-03-28T16:00:58.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:58 smithi067 ceph-mon[31707]: pgmap v355: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:58.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:58 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:58.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:58 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:58.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:58 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:58.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:58 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T16:00:58.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:58 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T16:00:58.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:58 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:00:58.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:58 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/3934734547' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:00:58.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:58 smithi150 ceph-mon[34825]: pgmap v355: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:58.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:58 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:58.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:58 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:58.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:58 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:58.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:58 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T16:00:58.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:58 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T16:00:58.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:58 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:00:58.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:58 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/3934734547' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:00:58.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:58 smithi183 ceph-mon[35580]: pgmap v355: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:00:58.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:58 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:58.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:58 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:58.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:58 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:00:58.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:58 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T16:00:58.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:58 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T16:00:58.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:58 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:00:58.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:58 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/3934734547' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:00:59.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:59 smithi150 ceph-mon[34825]: Deploying daemon mds.b on smithi150 2024-03-28T16:00:59.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:59 smithi150 ceph-mon[34825]: mds.? [v2:172.21.15.67:6840/3211254875,v1:172.21.15.67:6841/3211254875] up:boot 2024-03-28T16:00:59.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:59 smithi150 ceph-mon[34825]: fsmap 11 up:standby 2024-03-28T16:00:59.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:00:59 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T16:00:59.822 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:59 smithi067 ceph-mon[31707]: Deploying daemon mds.b on smithi150 2024-03-28T16:00:59.822 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:59 smithi067 ceph-mon[31707]: mds.? [v2:172.21.15.67:6840/3211254875,v1:172.21.15.67:6841/3211254875] up:boot 2024-03-28T16:00:59.822 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:59 smithi067 ceph-mon[31707]: fsmap 11 up:standby 2024-03-28T16:00:59.822 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:00:59 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T16:00:59.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:59 smithi183 ceph-mon[35580]: Deploying daemon mds.b on smithi150 2024-03-28T16:00:59.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:59 smithi183 ceph-mon[35580]: mds.? [v2:172.21.15.67:6840/3211254875,v1:172.21.15.67:6841/3211254875] up:boot 2024-03-28T16:00:59.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:59 smithi183 ceph-mon[35580]: fsmap 11 up:standby 2024-03-28T16:00:59.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:00:59 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T16:01:00.304 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T16:01:00.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:00.545+0000 7fc6ce200640 1 -- 172.21.15.67:0/4003720610 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6c81036f0 msgr2=0x7fc6c8103b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:00.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:00.545+0000 7fc6ce200640 1 --2- 172.21.15.67:0/4003720610 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6c81036f0 0x7fc6c8103b10 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fc6bc009900 tx=0x7fc6bc02bc30 comp rx=0 tx=0).stop 2024-03-28T16:01:00.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:00.545+0000 7fc6ce200640 1 -- 172.21.15.67:0/4003720610 shutdown_connections 2024-03-28T16:01:00.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:00.545+0000 7fc6ce200640 1 --2- 172.21.15.67:0/4003720610 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc6c81052a0 0x7fc6c810bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:00.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:00.545+0000 7fc6ce200640 1 --2- 172.21.15.67:0/4003720610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6c81048e0 0x7fc6c8104d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:00.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:00.545+0000 7fc6ce200640 1 --2- 172.21.15.67:0/4003720610 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6c81036f0 0x7fc6c8103b10 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:00.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:00.545+0000 7fc6ce200640 1 -- 172.21.15.67:0/4003720610 >> 172.21.15.67:0/4003720610 conn(0x7fc6c80fec80 msgr2=0x7fc6c81010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:00.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:00.545+0000 7fc6ce200640 1 -- 172.21.15.67:0/4003720610 shutdown_connections 2024-03-28T16:01:00.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:00.545+0000 7fc6ce200640 1 -- 172.21.15.67:0/4003720610 wait complete. 2024-03-28T16:01:00.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:00.546+0000 7fc6ce200640 1 Processor -- start 2024-03-28T16:01:00.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:00.547+0000 7fc6ce200640 1 -- start start 2024-03-28T16:01:00.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:00.547+0000 7fc6ce200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6c81036f0 0x7fc6c8199910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:00.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:00.547+0000 7fc6ce200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc6c81048e0 0x7fc6c8199e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:00.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:00.547+0000 7fc6ce200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6c81052a0 0x7fc6c81a0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:00.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:00.547+0000 7fc6ce200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6c810f270 con 0x7fc6c81052a0 2024-03-28T16:01:00.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:00.547+0000 7fc6ce200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6c810f0f0 con 0x7fc6c81036f0 2024-03-28T16:01:00.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:00.547+0000 7fc6ce200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6c810f3f0 con 0x7fc6c81048e0 2024-03-28T16:01:00.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:00.547+0000 7fc6cd800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6c81052a0 0x7fc6c81a0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:00.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:00.547+0000 7fc6cd800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6c81052a0 0x7fc6c81a0ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45430/0 (socket says 172.21.15.67:45430) 2024-03-28T16:01:00.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:00.547+0000 7fc6cd800640 1 -- 172.21.15.67:0/2547530717 learned_addr learned my addr 172.21.15.67:0/2547530717 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:01:00.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:00.548+0000 7fc6cce00640 1 --2- 172.21.15.67:0/2547530717 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6c81036f0 0x7fc6c8199910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:00.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:00.548+0000 7fc6c7e00640 1 --2- 172.21.15.67:0/2547530717 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc6c81048e0 0x7fc6c8199e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:00.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:00.548+0000 7fc6cd800640 1 -- 172.21.15.67:0/2547530717 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc6c81048e0 msgr2=0x7fc6c8199e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:00.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:00.548+0000 7fc6cd800640 1 --2- 172.21.15.67:0/2547530717 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc6c81048e0 0x7fc6c8199e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:00.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:00.548+0000 7fc6cd800640 1 -- 172.21.15.67:0/2547530717 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6c81036f0 msgr2=0x7fc6c8199910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:00.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:00.548+0000 7fc6cd800640 1 --2- 172.21.15.67:0/2547530717 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6c81036f0 0x7fc6c8199910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:00.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:00.548+0000 7fc6cd800640 1 -- 172.21.15.67:0/2547530717 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6c81a1530 con 0x7fc6c81052a0 2024-03-28T16:01:00.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:00.548+0000 7fc6cce00640 1 --2- 172.21.15.67:0/2547530717 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6c81036f0 0x7fc6c8199910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T16:01:00.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:00.548+0000 7fc6cd800640 1 --2- 172.21.15.67:0/2547530717 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6c81052a0 0x7fc6c81a0ec0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc6c000eaa0 tx=0x7fc6c000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:00.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:00.548+0000 7fc6c5600640 1 -- 172.21.15.67:0/2547530717 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c000cdb0 con 0x7fc6c81052a0 2024-03-28T16:01:00.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:00.549+0000 7fc6c5600640 1 -- 172.21.15.67:0/2547530717 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc6c0005020 con 0x7fc6c81052a0 2024-03-28T16:01:00.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:00.549+0000 7fc6ce200640 1 -- 172.21.15.67:0/2547530717 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6c81a1820 con 0x7fc6c81052a0 2024-03-28T16:01:00.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:00.549+0000 7fc6c5600640 1 -- 172.21.15.67:0/2547530717 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c00116d0 con 0x7fc6c81052a0 2024-03-28T16:01:00.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:00.549+0000 7fc6ce200640 1 -- 172.21.15.67:0/2547530717 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6c81a1d70 con 0x7fc6c81052a0 2024-03-28T16:01:00.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:00.550+0000 7fc6ce200640 1 -- 172.21.15.67:0/2547530717 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6c810aa10 con 0x7fc6c81052a0 2024-03-28T16:01:00.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:00.552+0000 7fc6c5600640 1 -- 172.21.15.67:0/2547530717 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fc6c00051d0 con 0x7fc6c81052a0 2024-03-28T16:01:00.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:00.553+0000 7fc6c5600640 1 --2- 172.21.15.67:0/2547530717 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fc6b00a2800 0x7fc6b00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:00.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:00.554+0000 7fc6c5600640 1 -- 172.21.15.67:0/2547530717 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fc6c00ce8c0 con 0x7fc6c81052a0 2024-03-28T16:01:00.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:00.554+0000 7fc6cce00640 1 --2- 172.21.15.67:0/2547530717 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fc6b00a2800 0x7fc6b00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:00.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:00.554+0000 7fc6cce00640 1 --2- 172.21.15.67:0/2547530717 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fc6b00a2800 0x7fc6b00a4cc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc6bc02c240 tx=0x7fc6bc005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:00.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:00.558+0000 7fc6c5600640 1 -- 172.21.15.67:0/2547530717 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc6c0016030 con 0x7fc6c81052a0 2024-03-28T16:01:00.742 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:00 smithi150 ceph-mon[34825]: pgmap v356: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:01:00.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:00.799+0000 7fc6ce200640 1 -- 172.21.15.67:0/2547530717 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc6c80622d0 con 0x7fc6c81052a0 2024-03-28T16:01:00.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:00.800+0000 7fc6c5600640 1 -- 172.21.15.67:0/2547530717 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+9254 (secure 0 0 0) 0x7fc6c009bd60 con 0x7fc6c81052a0 2024-03-28T16:01:00.802 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:01:00.802 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2008297659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2008297659},{"type":"v1","addr":"172.21.15.67:6835","nonce":2008297659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14532,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/320295745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":320295745},{"type":"v1","addr":"172.21.15.67:6837","nonce":320295745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6839/691821273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":691821273},{"type":"v1","addr":"172.21.15.67:6839","nonce":691821273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6841/3211254875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6840","nonce":3211254875},{"type":"v1","addr":"172.21.15.67:6841","nonce":3211254875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2129665423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2129665423},{"type":"v1","addr":"172.21.15.183:6833","nonce":2129665423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24496,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/1589784425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":1589784425},{"type":"v1","addr":"172.21.15.183:6835","nonce":1589784425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24508,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/2185863835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":2185863835},{"type":"v1","addr":"172.21.15.183:6837","nonce":2185863835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24520,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6839/1242814398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":1242814398},{"type":"v1","addr":"172.21.15.183:6839","nonce":1242814398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":34409,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1008335521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1008335521},{"type":"v1","addr":"172.21.15.150:6833","nonce":1008335521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":34445,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3269935695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3269935695},{"type":"v1","addr":"172.21.15.150:6835","nonce":3269935695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":34475,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1112886277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":1112886277},{"type":"v1","addr":"172.21.15.150:6837","nonce":1112886277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[]} 2024-03-28T16:01:00.802 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 12 2024-03-28T16:01:00.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:00.803+0000 7fc6ce200640 1 -- 172.21.15.67:0/2547530717 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fc6b00a2800 msgr2=0x7fc6b00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:00.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:00.803+0000 7fc6ce200640 1 --2- 172.21.15.67:0/2547530717 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fc6b00a2800 0x7fc6b00a4cc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc6bc02c240 tx=0x7fc6bc005fd0 comp rx=0 tx=0).stop 2024-03-28T16:01:00.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:00.803+0000 7fc6ce200640 1 -- 172.21.15.67:0/2547530717 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6c81052a0 msgr2=0x7fc6c81a0ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:00.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:00.803+0000 7fc6ce200640 1 --2- 172.21.15.67:0/2547530717 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6c81052a0 0x7fc6c81a0ec0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc6c000eaa0 tx=0x7fc6c000ef60 comp rx=0 tx=0).stop 2024-03-28T16:01:00.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:00.803+0000 7fc6ce200640 1 -- 172.21.15.67:0/2547530717 shutdown_connections 2024-03-28T16:01:00.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:00.803+0000 7fc6ce200640 1 --2- 172.21.15.67:0/2547530717 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fc6b00a2800 0x7fc6b00a4cc0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:00.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:00.803+0000 7fc6ce200640 1 --2- 172.21.15.67:0/2547530717 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6c81052a0 0x7fc6c81a0ec0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:00.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:00.803+0000 7fc6ce200640 1 --2- 172.21.15.67:0/2547530717 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc6c81048e0 0x7fc6c8199e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:00.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:00.803+0000 7fc6ce200640 1 --2- 172.21.15.67:0/2547530717 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6c81036f0 0x7fc6c8199910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:00.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:00.803+0000 7fc6ce200640 1 -- 172.21.15.67:0/2547530717 >> 172.21.15.67:0/2547530717 conn(0x7fc6c80fec80 msgr2=0x7fc6c8107be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:00.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:00.804+0000 7fc6ce200640 1 -- 172.21.15.67:0/2547530717 shutdown_connections 2024-03-28T16:01:00.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:00.804+0000 7fc6ce200640 1 -- 172.21.15.67:0/2547530717 wait complete. 2024-03-28T16:01:00.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:00 smithi067 ceph-mon[31707]: pgmap v356: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:01:00.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:00 smithi183 ceph-mon[35580]: pgmap v356: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:01:01.302 INFO:journalctl@ceph.mds.b.smithi150.stdout:Mar 28 16:01:01 smithi150 systemd[1]: Starting Ceph mds.b for b5f8225a-ed1a-11ee-95d0-87774f69a715... 2024-03-28T16:01:01.677 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:01 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/2547530717' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:01:01.677 INFO:journalctl@ceph.mds.b.smithi150.stdout:Mar 28 16:01:01 smithi150 podman[65517]: 2024-03-28 16:01:01.299801618 +0000 UTC m=+0.037756804 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T16:01:01.678 INFO:journalctl@ceph.mds.b.smithi150.stdout:Mar 28 16:01:01 smithi150 podman[65517]: 2024-03-28 16:01:01.400554572 +0000 UTC m=+0.138509765 container create b6cc8cd55de5de20837d9aab3cb242ab0d26a09a30e15aa846e4ab95c4357468 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-b, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-28T16:01:01.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:01 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/2547530717' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:01:01.843 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:01 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/2547530717' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:01:01.931 INFO:journalctl@ceph.mds.b.smithi150.stdout:Mar 28 16:01:01 smithi150 podman[65517]: 2024-03-28 16:01:01.688388567 +0000 UTC m=+0.426343746 container init b6cc8cd55de5de20837d9aab3cb242ab0d26a09a30e15aa846e4ab95c4357468 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-28T16:01:01.931 INFO:journalctl@ceph.mds.b.smithi150.stdout:Mar 28 16:01:01 smithi150 podman[65517]: 2024-03-28 16:01:01.696093794 +0000 UTC m=+0.434048949 container start b6cc8cd55de5de20837d9aab3cb242ab0d26a09a30e15aa846e4ab95c4357468 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-b, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2) 2024-03-28T16:01:01.931 INFO:journalctl@ceph.mds.b.smithi150.stdout:Mar 28 16:01:01 smithi150 ceph-mds[65543]: -- 172.21.15.150:0/3031992908 <== mon.2 v2:172.21.15.150:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c06f98fa40 con 0x55c06f962800 2024-03-28T16:01:01.931 INFO:journalctl@ceph.mds.b.smithi150.stdout:Mar 28 16:01:01 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-b[65534]: starting mds.b at 2024-03-28T16:01:01.932 INFO:journalctl@ceph.mds.b.smithi150.stdout:Mar 28 16:01:01 smithi150 bash[65517]: b6cc8cd55de5de20837d9aab3cb242ab0d26a09a30e15aa846e4ab95c4357468 2024-03-28T16:01:01.932 INFO:journalctl@ceph.mds.b.smithi150.stdout:Mar 28 16:01:01 smithi150 systemd[1]: Started Ceph mds.b for b5f8225a-ed1a-11ee-95d0-87774f69a715. 2024-03-28T16:01:02.824 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T16:01:03.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.044+0000 7f3d6d000640 1 -- 172.21.15.67:0/354331737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d681038e0 msgr2=0x7f3d68103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:03.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.044+0000 7f3d6d000640 1 --2- 172.21.15.67:0/354331737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d681038e0 0x7f3d68103d00 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f3d58009900 tx=0x7f3d5802bc30 comp rx=0 tx=0).stop 2024-03-28T16:01:03.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.044+0000 7f3d6d000640 1 -- 172.21.15.67:0/354331737 shutdown_connections 2024-03-28T16:01:03.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.044+0000 7f3d6d000640 1 --2- 172.21.15.67:0/354331737 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d68105490 0x7f3d6810bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:03.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.044+0000 7f3d6d000640 1 --2- 172.21.15.67:0/354331737 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3d68104ad0 0x7f3d68104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:03.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.044+0000 7f3d6d000640 1 --2- 172.21.15.67:0/354331737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d681038e0 0x7f3d68103d00 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:03.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.044+0000 7f3d6d000640 1 -- 172.21.15.67:0/354331737 >> 172.21.15.67:0/354331737 conn(0x7f3d680fee90 msgr2=0x7f3d681012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:03.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.044+0000 7f3d6d000640 1 -- 172.21.15.67:0/354331737 shutdown_connections 2024-03-28T16:01:03.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.044+0000 7f3d6d000640 1 -- 172.21.15.67:0/354331737 wait complete. 2024-03-28T16:01:03.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.046+0000 7f3d6d000640 1 Processor -- start 2024-03-28T16:01:03.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.046+0000 7f3d6d000640 1 -- start start 2024-03-28T16:01:03.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.046+0000 7f3d6d000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d681038e0 0x7f3d68199b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:03.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.046+0000 7f3d6d000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3d68104ad0 0x7f3d6819a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:03.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.046+0000 7f3d6d000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d68105490 0x7f3d681a1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:03.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.046+0000 7f3d6d000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d6810f450 con 0x7f3d68105490 2024-03-28T16:01:03.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.046+0000 7f3d6d000640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d6810f2d0 con 0x7f3d681038e0 2024-03-28T16:01:03.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.046+0000 7f3d6d000640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d6810f5d0 con 0x7f3d68104ad0 2024-03-28T16:01:03.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.047+0000 7f3d66800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d68105490 0x7f3d681a1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:03.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.047+0000 7f3d66800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d68105490 0x7f3d681a1100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45442/0 (socket says 172.21.15.67:45442) 2024-03-28T16:01:03.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.047+0000 7f3d66800640 1 -- 172.21.15.67:0/2122040762 learned_addr learned my addr 172.21.15.67:0/2122040762 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:01:03.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.047+0000 7f3d65e00640 1 --2- 172.21.15.67:0/2122040762 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d681038e0 0x7f3d68199b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:03.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.047+0000 7f3d65400640 1 --2- 172.21.15.67:0/2122040762 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3d68104ad0 0x7f3d6819a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:03.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.047+0000 7f3d66800640 1 -- 172.21.15.67:0/2122040762 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3d68104ad0 msgr2=0x7f3d6819a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:03.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.047+0000 7f3d66800640 1 --2- 172.21.15.67:0/2122040762 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3d68104ad0 0x7f3d6819a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:03.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.047+0000 7f3d66800640 1 -- 172.21.15.67:0/2122040762 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d681038e0 msgr2=0x7f3d68199b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:03.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.047+0000 7f3d66800640 1 --2- 172.21.15.67:0/2122040762 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d681038e0 0x7f3d68199b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:03.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.047+0000 7f3d66800640 1 -- 172.21.15.67:0/2122040762 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d681a1770 con 0x7f3d68105490 2024-03-28T16:01:03.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.048+0000 7f3d66800640 1 --2- 172.21.15.67:0/2122040762 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d68105490 0x7f3d681a1100 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f3d6000eaa0 tx=0x7f3d6000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:03.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.048+0000 7f3d5ea00640 1 -- 172.21.15.67:0/2122040762 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d6000cdb0 con 0x7f3d68105490 2024-03-28T16:01:03.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.048+0000 7f3d5ea00640 1 -- 172.21.15.67:0/2122040762 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3d60005020 con 0x7f3d68105490 2024-03-28T16:01:03.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.048+0000 7f3d6d000640 1 -- 172.21.15.67:0/2122040762 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d681a1a60 con 0x7f3d68105490 2024-03-28T16:01:03.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.048+0000 7f3d5ea00640 1 -- 172.21.15.67:0/2122040762 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d600116d0 con 0x7f3d68105490 2024-03-28T16:01:03.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.048+0000 7f3d6d000640 1 -- 172.21.15.67:0/2122040762 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d681a1fb0 con 0x7f3d68105490 2024-03-28T16:01:03.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.049+0000 7f3d6d000640 1 -- 172.21.15.67:0/2122040762 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d6810acd0 con 0x7f3d68105490 2024-03-28T16:01:03.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.051+0000 7f3d5ea00640 1 -- 172.21.15.67:0/2122040762 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f3d600051d0 con 0x7f3d68105490 2024-03-28T16:01:03.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.053+0000 7f3d5ea00640 1 --2- 172.21.15.67:0/2122040762 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f3d4c0a2880 0x7f3d4c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:03.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.054+0000 7f3d5ea00640 1 -- 172.21.15.67:0/2122040762 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f3d600ce740 con 0x7f3d68105490 2024-03-28T16:01:03.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.054+0000 7f3d65e00640 1 --2- 172.21.15.67:0/2122040762 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f3d4c0a2880 0x7f3d4c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:03.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.055+0000 7f3d65e00640 1 --2- 172.21.15.67:0/2122040762 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f3d4c0a2880 0x7f3d4c0a4d40 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3d5802c240 tx=0x7f3d5802c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:03.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.058+0000 7f3d5ea00640 1 -- 172.21.15.67:0/2122040762 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3d60016030 con 0x7f3d68105490 2024-03-28T16:01:03.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:02 smithi150 ceph-mon[34825]: pgmap v357: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:01:03.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:02 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:01:03.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:02 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:01:03.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:02 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:01:03.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:02 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:01:03.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:02 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:01:03.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:02 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T16:01:03.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:02 smithi150 ceph-mon[34825]: mds.? [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] up:boot 2024-03-28T16:01:03.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:02 smithi150 ceph-mon[34825]: fsmap 12 up:standby 2024-03-28T16:01:03.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:02 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T16:01:03.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.313+0000 7f3d6d000640 1 -- 172.21.15.67:0/2122040762 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d6802d070 con 0x7f3d68105490 2024-03-28T16:01:03.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.314+0000 7f3d5ea00640 1 -- 172.21.15.67:0/2122040762 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+10049 (secure 0 0 0) 0x7f3d6009b590 con 0x7f3d68105490 2024-03-28T16:01:03.316 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:01:03.316 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2008297659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2008297659},{"type":"v1","addr":"172.21.15.67:6835","nonce":2008297659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14532,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/320295745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":320295745},{"type":"v1","addr":"172.21.15.67:6837","nonce":320295745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6839/691821273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":691821273},{"type":"v1","addr":"172.21.15.67:6839","nonce":691821273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6841/3211254875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6840","nonce":3211254875},{"type":"v1","addr":"172.21.15.67:6841","nonce":3211254875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2129665423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2129665423},{"type":"v1","addr":"172.21.15.183:6833","nonce":2129665423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24496,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/1589784425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":1589784425},{"type":"v1","addr":"172.21.15.183:6835","nonce":1589784425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24508,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/2185863835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":2185863835},{"type":"v1","addr":"172.21.15.183:6837","nonce":2185863835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24520,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6839/1242814398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":1242814398},{"type":"v1","addr":"172.21.15.183:6839","nonce":1242814398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":34409,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1008335521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1008335521},{"type":"v1","addr":"172.21.15.150:6833","nonce":1008335521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":34445,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3269935695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3269935695},{"type":"v1","addr":"172.21.15.150:6835","nonce":3269935695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":34475,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1112886277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":1112886277},{"type":"v1","addr":"172.21.15.150:6837","nonce":1112886277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":34508,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/1211011614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":1211011614},{"type":"v1","addr":"172.21.15.150:6839","nonce":1211011614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[]} 2024-03-28T16:01:03.316 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 13 2024-03-28T16:01:03.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.317+0000 7f3d6d000640 1 -- 172.21.15.67:0/2122040762 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f3d4c0a2880 msgr2=0x7f3d4c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:03.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.317+0000 7f3d6d000640 1 --2- 172.21.15.67:0/2122040762 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f3d4c0a2880 0x7f3d4c0a4d40 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3d5802c240 tx=0x7f3d5802c7e0 comp rx=0 tx=0).stop 2024-03-28T16:01:03.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.317+0000 7f3d6d000640 1 -- 172.21.15.67:0/2122040762 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d68105490 msgr2=0x7f3d681a1100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:03.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.317+0000 7f3d6d000640 1 --2- 172.21.15.67:0/2122040762 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d68105490 0x7f3d681a1100 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f3d6000eaa0 tx=0x7f3d6000ef60 comp rx=0 tx=0).stop 2024-03-28T16:01:03.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.317+0000 7f3d6d000640 1 -- 172.21.15.67:0/2122040762 shutdown_connections 2024-03-28T16:01:03.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.317+0000 7f3d6d000640 1 --2- 172.21.15.67:0/2122040762 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f3d4c0a2880 0x7f3d4c0a4d40 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:03.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.317+0000 7f3d6d000640 1 --2- 172.21.15.67:0/2122040762 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d68105490 0x7f3d681a1100 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:03.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.317+0000 7f3d6d000640 1 --2- 172.21.15.67:0/2122040762 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3d68104ad0 0x7f3d6819a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:03.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.317+0000 7f3d6d000640 1 --2- 172.21.15.67:0/2122040762 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d681038e0 0x7f3d68199b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:03.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.317+0000 7f3d6d000640 1 -- 172.21.15.67:0/2122040762 >> 172.21.15.67:0/2122040762 conn(0x7f3d680fee90 msgr2=0x7f3d68107ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:03.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.317+0000 7f3d6d000640 1 -- 172.21.15.67:0/2122040762 shutdown_connections 2024-03-28T16:01:03.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.317+0000 7f3d6d000640 1 -- 172.21.15.67:0/2122040762 wait complete. 2024-03-28T16:01:03.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:02 smithi067 ceph-mon[31707]: pgmap v357: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:01:03.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:02 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:01:03.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:02 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:01:03.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:02 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:01:03.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:02 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:01:03.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:02 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:01:03.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:02 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T16:01:03.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:02 smithi067 ceph-mon[31707]: mds.? [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] up:boot 2024-03-28T16:01:03.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:02 smithi067 ceph-mon[31707]: fsmap 12 up:standby 2024-03-28T16:01:03.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:02 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T16:01:03.339 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T16:01:03.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:02 smithi183 ceph-mon[35580]: pgmap v357: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:01:03.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:02 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:01:03.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:02 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:01:03.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:02 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:01:03.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:02 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:01:03.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:02 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:01:03.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:02 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T16:01:03.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:02 smithi183 ceph-mon[35580]: mds.? [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] up:boot 2024-03-28T16:01:03.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:02 smithi183 ceph-mon[35580]: fsmap 12 up:standby 2024-03-28T16:01:03.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:02 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T16:01:03.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.552+0000 7f92a3c00640 1 -- 172.21.15.67:0/1289879367 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f929c104ad0 msgr2=0x7f929c104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:03.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.552+0000 7f92a3c00640 1 --2- 172.21.15.67:0/1289879367 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f929c104ad0 0x7f929c104f50 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9288009900 tx=0x7f928802bc30 comp rx=0 tx=0).stop 2024-03-28T16:01:03.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.552+0000 7f92a3c00640 1 -- 172.21.15.67:0/1289879367 shutdown_connections 2024-03-28T16:01:03.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.552+0000 7f92a3c00640 1 --2- 172.21.15.67:0/1289879367 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f929c105490 0x7f929c10bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:03.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.552+0000 7f92a3c00640 1 --2- 172.21.15.67:0/1289879367 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f929c104ad0 0x7f929c104f50 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:03.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.552+0000 7f92a3c00640 1 --2- 172.21.15.67:0/1289879367 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f929c1038e0 0x7f929c103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:03.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.552+0000 7f92a3c00640 1 -- 172.21.15.67:0/1289879367 >> 172.21.15.67:0/1289879367 conn(0x7f929c0fee90 msgr2=0x7f929c1012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:03.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.552+0000 7f92a3c00640 1 -- 172.21.15.67:0/1289879367 shutdown_connections 2024-03-28T16:01:03.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.552+0000 7f92a3c00640 1 -- 172.21.15.67:0/1289879367 wait complete. 2024-03-28T16:01:03.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.553+0000 7f92a3c00640 1 Processor -- start 2024-03-28T16:01:03.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.553+0000 7f92a3c00640 1 -- start start 2024-03-28T16:01:03.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.553+0000 7f92a3c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f929c1038e0 0x7f929c199b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:03.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.553+0000 7f92a3c00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f929c104ad0 0x7f929c19a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:03.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.553+0000 7f92a3c00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f929c105490 0x7f929c1a10e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:03.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.553+0000 7f92a3c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f929c10f460 con 0x7f929c1038e0 2024-03-28T16:01:03.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.553+0000 7f92a3c00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f929c10f2e0 con 0x7f929c104ad0 2024-03-28T16:01:03.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.553+0000 7f92a3c00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f929c10f5e0 con 0x7f929c105490 2024-03-28T16:01:03.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.554+0000 7f92a1200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f929c1038e0 0x7f929c199b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:03.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.554+0000 7f92a1200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f929c1038e0 0x7f929c199b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45462/0 (socket says 172.21.15.67:45462) 2024-03-28T16:01:03.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.554+0000 7f92a1200640 1 -- 172.21.15.67:0/870854563 learned_addr learned my addr 172.21.15.67:0/870854563 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:01:03.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.554+0000 7f92a1c00640 1 --2- 172.21.15.67:0/870854563 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f929c105490 0x7f929c1a10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:03.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.554+0000 7f929be00640 1 --2- 172.21.15.67:0/870854563 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f929c104ad0 0x7f929c19a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:03.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.554+0000 7f92a1200640 1 -- 172.21.15.67:0/870854563 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f929c105490 msgr2=0x7f929c1a10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:03.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.554+0000 7f92a1200640 1 --2- 172.21.15.67:0/870854563 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f929c105490 0x7f929c1a10e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:03.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.554+0000 7f92a1200640 1 -- 172.21.15.67:0/870854563 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f929c104ad0 msgr2=0x7f929c19a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:03.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.554+0000 7f92a1200640 1 --2- 172.21.15.67:0/870854563 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f929c104ad0 0x7f929c19a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:03.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.554+0000 7f92a1200640 1 -- 172.21.15.67:0/870854563 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f929c1a1750 con 0x7f929c1038e0 2024-03-28T16:01:03.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.554+0000 7f929be00640 1 --2- 172.21.15.67:0/870854563 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f929c104ad0 0x7f929c19a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:01:03.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.554+0000 7f92a1200640 1 --2- 172.21.15.67:0/870854563 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f929c1038e0 0x7f929c199b30 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f929000d950 tx=0x7f929000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:03.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.554+0000 7f9299600640 1 -- 172.21.15.67:0/870854563 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9290017020 con 0x7f929c1038e0 2024-03-28T16:01:03.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.555+0000 7f9299600640 1 -- 172.21.15.67:0/870854563 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9290009c30 con 0x7f929c1038e0 2024-03-28T16:01:03.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.555+0000 7f9299600640 1 -- 172.21.15.67:0/870854563 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f929000f6b0 con 0x7f929c1038e0 2024-03-28T16:01:03.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.555+0000 7f92a3c00640 1 -- 172.21.15.67:0/870854563 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f929c1a1a40 con 0x7f929c1038e0 2024-03-28T16:01:03.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.555+0000 7f92a3c00640 1 -- 172.21.15.67:0/870854563 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f929c1a1fe0 con 0x7f929c1038e0 2024-03-28T16:01:03.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.556+0000 7f92a3c00640 1 -- 172.21.15.67:0/870854563 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f929c10ac50 con 0x7f929c1038e0 2024-03-28T16:01:03.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.558+0000 7f9299600640 1 -- 172.21.15.67:0/870854563 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f929001a020 con 0x7f929c1038e0 2024-03-28T16:01:03.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.559+0000 7f9299600640 1 --2- 172.21.15.67:0/870854563 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f92840a2880 0x7f92840a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:03.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.559+0000 7f9299600640 1 -- 172.21.15.67:0/870854563 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f92900ce5c0 con 0x7f929c1038e0 2024-03-28T16:01:03.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.559+0000 7f929be00640 1 --2- 172.21.15.67:0/870854563 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f92840a2880 0x7f92840a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:03.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.560+0000 7f929be00640 1 --2- 172.21.15.67:0/870854563 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f92840a2880 0x7f92840a4d40 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f9288036040 tx=0x7f9288005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:03.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.563+0000 7f9299600640 1 -- 172.21.15.67:0/870854563 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9290021020 con 0x7f929c1038e0 2024-03-28T16:01:03.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.775+0000 7f92a3c00640 1 -- 172.21.15.67:0/870854563 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f929c02d070 con 0x7f929c1038e0 2024-03-28T16:01:03.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.778+0000 7f9299600640 1 -- 172.21.15.67:0/870854563 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+18458 (secure 0 0 0) 0x7f9290009de0 con 0x7f929c1038e0 2024-03-28T16:01:03.780 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:01:03.780 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":86,"fsid":"b5f8225a-ed1a-11ee-95d0-87774f69a715","created":"2024-03-28T15:50:21.471924+0000","modified":"2024-03-28T16:00:02.385208+0000","last_up_change":"2024-03-28T16:00:01.369123+0000","last_in_change":"2024-03-28T15:59:37.515483+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T15:54:55.588007+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7c1791b6-9c0e-4512-a327-cad166bb5581","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6803","nonce":1894681701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6805","nonce":1894681701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6809","nonce":1894681701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6807","nonce":1894681701}]},"public_addr":"172.21.15.67:6803/1894681701","cluster_addr":"172.21.15.67:6805/1894681701","heartbeat_back_addr":"172.21.15.67:6809/1894681701","heartbeat_front_addr":"172.21.15.67:6807/1894681701","state":["exists","up"]},{"osd":1,"uuid":"0cdda7f3-d00c-4436-bcc1-49fb8072b2e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6801","nonce":1734465762}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6803","nonce":1734465762}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6807","nonce":1734465762}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6805","nonce":1734465762}]},"public_addr":"172.21.15.150:6801/1734465762","cluster_addr":"172.21.15.150:6803/1734465762","heartbeat_back_addr":"172.21.15.150:6807/1734465762","heartbeat_front_addr":"172.21.15.150:6805/1734465762","state":["exists","up"]},{"osd":2,"uuid":"a55b9a02-c3fd-4061-95c0-53a24a17ade0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6801","nonce":3398095931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6803","nonce":3398095931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6807","nonce":3398095931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6805","nonce":3398095931}]},"public_addr":"172.21.15.183:6801/3398095931","cluster_addr":"172.21.15.183:6803/3398095931","heartbeat_back_addr":"172.21.15.183:6807/3398095931","heartbeat_front_addr":"172.21.15.183:6805/3398095931","state":["exists","up"]},{"osd":3,"uuid":"d34940c0-5ce9-4b8b-aa36-4872414a043f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6811","nonce":1933541875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6813","nonce":1933541875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6817","nonce":1933541875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6815","nonce":1933541875}]},"public_addr":"172.21.15.67:6811/1933541875","cluster_addr":"172.21.15.67:6813/1933541875","heartbeat_back_addr":"172.21.15.67:6817/1933541875","heartbeat_front_addr":"172.21.15.67:6815/1933541875","state":["exists","up"]},{"osd":4,"uuid":"5cc3e260-83a3-4a4d-aa6a-34a3b12b2b5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6809","nonce":3775269993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6811","nonce":3775269993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6815","nonce":3775269993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6813","nonce":3775269993}]},"public_addr":"172.21.15.150:6809/3775269993","cluster_addr":"172.21.15.150:6811/3775269993","heartbeat_back_addr":"172.21.15.150:6815/3775269993","heartbeat_front_addr":"172.21.15.150:6813/3775269993","state":["exists","up"]},{"osd":5,"uuid":"a2542b21-d6ac-413e-8b9b-4a6548b20840","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6809","nonce":164284137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6811","nonce":164284137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6815","nonce":164284137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6813","nonce":164284137}]},"public_addr":"172.21.15.183:6809/164284137","cluster_addr":"172.21.15.183:6811/164284137","heartbeat_back_addr":"172.21.15.183:6815/164284137","heartbeat_front_addr":"172.21.15.183:6813/164284137","state":["exists","up"]},{"osd":6,"uuid":"8905a2d3-0f1b-4853-a218-22da2635150a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6819","nonce":3565528013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6821","nonce":3565528013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6825","nonce":3565528013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6823","nonce":3565528013}]},"public_addr":"172.21.15.67:6819/3565528013","cluster_addr":"172.21.15.67:6821/3565528013","heartbeat_back_addr":"172.21.15.67:6825/3565528013","heartbeat_front_addr":"172.21.15.67:6823/3565528013","state":["exists","up"]},{"osd":7,"uuid":"b275055b-b451-4180-91b1-e6e61a828c48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6817","nonce":3735115340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6819","nonce":3735115340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6823","nonce":3735115340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6821","nonce":3735115340}]},"public_addr":"172.21.15.150:6817/3735115340","cluster_addr":"172.21.15.150:6819/3735115340","heartbeat_back_addr":"172.21.15.150:6823/3735115340","heartbeat_front_addr":"172.21.15.150:6821/3735115340","state":["exists","up"]},{"osd":8,"uuid":"b6abf7ad-eb34-4f60-a4eb-348a92d7195b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6817","nonce":1025400095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6819","nonce":1025400095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6823","nonce":1025400095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6821","nonce":1025400095}]},"public_addr":"172.21.15.183:6817/1025400095","cluster_addr":"172.21.15.183:6819/1025400095","heartbeat_back_addr":"172.21.15.183:6823/1025400095","heartbeat_front_addr":"172.21.15.183:6821/1025400095","state":["exists","up"]},{"osd":9,"uuid":"28a78e3a-0657-40aa-816f-16a6fadadcbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6827","nonce":76885409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6829","nonce":76885409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6833","nonce":76885409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6831","nonce":76885409}]},"public_addr":"172.21.15.67:6827/76885409","cluster_addr":"172.21.15.67:6829/76885409","heartbeat_back_addr":"172.21.15.67:6833/76885409","heartbeat_front_addr":"172.21.15.67:6831/76885409","state":["exists","up"]},{"osd":10,"uuid":"d287907e-424f-4b49-a06e-4613c6de0474","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6825","nonce":3489982080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6827","nonce":3489982080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6831","nonce":3489982080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6829","nonce":3489982080}]},"public_addr":"172.21.15.150:6825/3489982080","cluster_addr":"172.21.15.150:6827/3489982080","heartbeat_back_addr":"172.21.15.150:6831/3489982080","heartbeat_front_addr":"172.21.15.150:6829/3489982080","state":["exists","up"]},{"osd":11,"uuid":"a3c29c47-a228-4f42-a23c-fbdb0c093699","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":85,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6825","nonce":1892639849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6826","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6827","nonce":1892639849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6831","nonce":1892639849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6829","nonce":1892639849}]},"public_addr":"172.21.15.183:6825/1892639849","cluster_addr":"172.21.15.183:6827/1892639849","heartbeat_back_addr":"172.21.15.183:6831/1892639849","heartbeat_front_addr":"172.21.15.183:6829/1892639849","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:53:45.646211+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:54:18.956590+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:54:52.161646+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:55:27.502822+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:56:01.162932+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:56:34.674452+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:57:09.298679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:57:43.230406+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:58:17.003821+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:58:51.085863+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:59:25.665065+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T16:00:00.263457+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6801/1631001968":"2024-03-29T15:51:38.875334+0000","172.21.15.67:6800/1631001968":"2024-03-29T15:51:38.875334+0000","172.21.15.67:0/1533056310":"2024-03-29T15:51:38.875334+0000","172.21.15.67:6801/3580491071":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/2030412458":"2024-03-29T15:51:03.859740+0000","172.21.15.67:6800/3580491071":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/2905307533":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/3763759988":"2024-03-29T15:51:38.875334+0000","172.21.15.67:0/4089399880":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/4239108661":"2024-03-29T15:51:38.875334+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T16:01:03.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.781+0000 7f92a3c00640 1 -- 172.21.15.67:0/870854563 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f92840a2880 msgr2=0x7f92840a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:03.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.781+0000 7f92a3c00640 1 --2- 172.21.15.67:0/870854563 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f92840a2880 0x7f92840a4d40 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f9288036040 tx=0x7f9288005fd0 comp rx=0 tx=0).stop 2024-03-28T16:01:03.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.781+0000 7f92a3c00640 1 -- 172.21.15.67:0/870854563 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f929c1038e0 msgr2=0x7f929c199b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:03.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.781+0000 7f92a3c00640 1 --2- 172.21.15.67:0/870854563 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f929c1038e0 0x7f929c199b30 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f929000d950 tx=0x7f929000de10 comp rx=0 tx=0).stop 2024-03-28T16:01:03.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.781+0000 7f92a3c00640 1 -- 172.21.15.67:0/870854563 shutdown_connections 2024-03-28T16:01:03.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.781+0000 7f92a3c00640 1 --2- 172.21.15.67:0/870854563 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f92840a2880 0x7f92840a4d40 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:03.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.781+0000 7f92a3c00640 1 --2- 172.21.15.67:0/870854563 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f929c105490 0x7f929c1a10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:03.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.781+0000 7f92a3c00640 1 --2- 172.21.15.67:0/870854563 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f929c104ad0 0x7f929c19a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:03.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.781+0000 7f92a3c00640 1 --2- 172.21.15.67:0/870854563 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f929c1038e0 0x7f929c199b30 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:03.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.781+0000 7f92a3c00640 1 -- 172.21.15.67:0/870854563 >> 172.21.15.67:0/870854563 conn(0x7f929c0fee90 msgr2=0x7f929c107e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:03.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.781+0000 7f92a3c00640 1 -- 172.21.15.67:0/870854563 shutdown_connections 2024-03-28T16:01:03.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:03.781+0000 7f92a3c00640 1 -- 172.21.15.67:0/870854563 wait complete. 2024-03-28T16:01:03.801 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T15:54:55.588007+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T16:01:03.802 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T16:01:04.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.008+0000 7f7a18c00640 1 -- 172.21.15.67:0/350240352 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a14104ab0 msgr2=0x7f7a14104f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:04.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.008+0000 7f7a18c00640 1 --2- 172.21.15.67:0/350240352 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a14104ab0 0x7f7a14104f30 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f79fc009900 tx=0x7f79fc02bc30 comp rx=0 tx=0).stop 2024-03-28T16:01:04.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.008+0000 7f7a18c00640 1 -- 172.21.15.67:0/350240352 shutdown_connections 2024-03-28T16:01:04.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.008+0000 7f7a18c00640 1 --2- 172.21.15.67:0/350240352 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7a14105470 0x7f7a1410bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:04.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.008+0000 7f7a18c00640 1 --2- 172.21.15.67:0/350240352 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a14104ab0 0x7f7a14104f30 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:04.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.008+0000 7f7a18c00640 1 --2- 172.21.15.67:0/350240352 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a141038c0 0x7f7a14103ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:04.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.008+0000 7f7a18c00640 1 -- 172.21.15.67:0/350240352 >> 172.21.15.67:0/350240352 conn(0x7f7a140fee50 msgr2=0x7f7a14101270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:04.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.008+0000 7f7a18c00640 1 -- 172.21.15.67:0/350240352 shutdown_connections 2024-03-28T16:01:04.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.008+0000 7f7a18c00640 1 -- 172.21.15.67:0/350240352 wait complete. 2024-03-28T16:01:04.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.009+0000 7f7a18c00640 1 Processor -- start 2024-03-28T16:01:04.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.009+0000 7f7a18c00640 1 -- start start 2024-03-28T16:01:04.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.010+0000 7f7a18c00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7a141038c0 0x7f7a14199a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:04.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.010+0000 7f7a18c00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a14104ab0 0x7f7a14199fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:04.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.010+0000 7f7a18c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a14105470 0x7f7a141a1020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:04.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.010+0000 7f7a18c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1410f420 con 0x7f7a14105470 2024-03-28T16:01:04.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.010+0000 7f7a18c00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1410f2a0 con 0x7f7a14104ab0 2024-03-28T16:01:04.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.010+0000 7f7a18c00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1410f5a0 con 0x7f7a141038c0 2024-03-28T16:01:04.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.010+0000 7f7a12800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a14105470 0x7f7a141a1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:04.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.010+0000 7f7a12800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a14105470 0x7f7a141a1020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45476/0 (socket says 172.21.15.67:45476) 2024-03-28T16:01:04.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.010+0000 7f7a12800640 1 -- 172.21.15.67:0/945295127 learned_addr learned my addr 172.21.15.67:0/945295127 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:01:04.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.010+0000 7f7a11e00640 1 --2- 172.21.15.67:0/945295127 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7a141038c0 0x7f7a14199a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:04.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.010+0000 7f7a0be00640 1 --2- 172.21.15.67:0/945295127 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a14104ab0 0x7f7a14199fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:04.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.010+0000 7f7a12800640 1 -- 172.21.15.67:0/945295127 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7a141038c0 msgr2=0x7f7a14199a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:04.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.010+0000 7f7a12800640 1 --2- 172.21.15.67:0/945295127 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7a141038c0 0x7f7a14199a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:04.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.010+0000 7f7a12800640 1 -- 172.21.15.67:0/945295127 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a14104ab0 msgr2=0x7f7a14199fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:04.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.010+0000 7f7a12800640 1 --2- 172.21.15.67:0/945295127 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a14104ab0 0x7f7a14199fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:04.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.010+0000 7f7a12800640 1 -- 172.21.15.67:0/945295127 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a141a1690 con 0x7f7a14105470 2024-03-28T16:01:04.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.011+0000 7f7a12800640 1 --2- 172.21.15.67:0/945295127 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a14105470 0x7f7a141a1020 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c00cce0 tx=0x7f7a0c00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:04.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.011+0000 7f7a0aa00640 1 -- 172.21.15.67:0/945295127 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a0c00f450 con 0x7f7a14105470 2024-03-28T16:01:04.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.011+0000 7f7a0aa00640 1 -- 172.21.15.67:0/945295127 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7a0c00faa0 con 0x7f7a14105470 2024-03-28T16:01:04.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.011+0000 7f7a18c00640 1 -- 172.21.15.67:0/945295127 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a141a1980 con 0x7f7a14105470 2024-03-28T16:01:04.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.011+0000 7f7a0aa00640 1 -- 172.21.15.67:0/945295127 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a0c018b70 con 0x7f7a14105470 2024-03-28T16:01:04.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.011+0000 7f7a18c00640 1 -- 172.21.15.67:0/945295127 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a141a1ea0 con 0x7f7a14105470 2024-03-28T16:01:04.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.012+0000 7f7a18c00640 1 -- 172.21.15.67:0/945295127 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a1410abe0 con 0x7f7a14105470 2024-03-28T16:01:04.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.014+0000 7f7a0aa00640 1 -- 172.21.15.67:0/945295127 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f7a0c018d10 con 0x7f7a14105470 2024-03-28T16:01:04.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.015+0000 7f7a0aa00640 1 --2- 172.21.15.67:0/945295127 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f79f80a2880 0x7f79f80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:04.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.016+0000 7f7a0aa00640 1 -- 172.21.15.67:0/945295127 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f7a0c0ceac0 con 0x7f7a14105470 2024-03-28T16:01:04.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.016+0000 7f7a11e00640 1 --2- 172.21.15.67:0/945295127 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f79f80a2880 0x7f79f80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:04.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.017+0000 7f7a11e00640 1 --2- 172.21.15.67:0/945295127 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f79f80a2880 0x7f79f80a4d40 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7a04005fd0 tx=0x7f7a04005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:04.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.020+0000 7f7a0aa00640 1 -- 172.21.15.67:0/945295127 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7a0c014030 con 0x7f7a14105470 2024-03-28T16:01:04.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:03 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/2122040762' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:01:04.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:03 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/870854563' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:01:04.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:03 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/2122040762' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:01:04.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:03 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/870854563' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:01:04.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.231+0000 7f7a18c00640 1 -- 172.21.15.67:0/945295127 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7a1402d100 con 0x7f7a14105470 2024-03-28T16:01:04.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.232+0000 7f7a0aa00640 1 -- 172.21.15.67:0/945295127 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v86) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7a0c09bf60 con 0x7f7a14105470 2024-03-28T16:01:04.233 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-03-28T16:01:04.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.234+0000 7f7a18c00640 1 -- 172.21.15.67:0/945295127 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f79f80a2880 msgr2=0x7f79f80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:04.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.234+0000 7f7a18c00640 1 --2- 172.21.15.67:0/945295127 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f79f80a2880 0x7f79f80a4d40 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7a04005fd0 tx=0x7f7a04005850 comp rx=0 tx=0).stop 2024-03-28T16:01:04.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.234+0000 7f7a18c00640 1 -- 172.21.15.67:0/945295127 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a14105470 msgr2=0x7f7a141a1020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:04.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.234+0000 7f7a18c00640 1 --2- 172.21.15.67:0/945295127 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a14105470 0x7f7a141a1020 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c00cce0 tx=0x7f7a0c00a620 comp rx=0 tx=0).stop 2024-03-28T16:01:04.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.235+0000 7f7a18c00640 1 -- 172.21.15.67:0/945295127 shutdown_connections 2024-03-28T16:01:04.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.235+0000 7f7a18c00640 1 --2- 172.21.15.67:0/945295127 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f79f80a2880 0x7f79f80a4d40 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:04.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.235+0000 7f7a18c00640 1 --2- 172.21.15.67:0/945295127 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a14105470 0x7f7a141a1020 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:04.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.235+0000 7f7a18c00640 1 --2- 172.21.15.67:0/945295127 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a14104ab0 0x7f7a14199fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:04.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.235+0000 7f7a18c00640 1 --2- 172.21.15.67:0/945295127 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7a141038c0 0x7f7a14199a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:04.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.235+0000 7f7a18c00640 1 -- 172.21.15.67:0/945295127 >> 172.21.15.67:0/945295127 conn(0x7f7a140fee50 msgr2=0x7f7a14107db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:04.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.235+0000 7f7a18c00640 1 -- 172.21.15.67:0/945295127 shutdown_connections 2024-03-28T16:01:04.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.235+0000 7f7a18c00640 1 -- 172.21.15.67:0/945295127 wait complete. 2024-03-28T16:01:04.255 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-28T16:01:04.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:03 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/2122040762' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:01:04.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:03 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/870854563' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:01:04.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.458+0000 7f0aa9000640 1 -- 172.21.15.67:0/2620315922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0aa41038e0 msgr2=0x7f0aa4103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:04.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.458+0000 7f0aa9000640 1 --2- 172.21.15.67:0/2620315922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0aa41038e0 0x7f0aa4103d00 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f0a94009900 tx=0x7f0a9402bc30 comp rx=0 tx=0).stop 2024-03-28T16:01:04.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.458+0000 7f0aa9000640 1 -- 172.21.15.67:0/2620315922 shutdown_connections 2024-03-28T16:01:04.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.458+0000 7f0aa9000640 1 --2- 172.21.15.67:0/2620315922 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0aa4105490 0x7f0aa410bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:04.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.458+0000 7f0aa9000640 1 --2- 172.21.15.67:0/2620315922 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0aa4104ad0 0x7f0aa4104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:04.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.458+0000 7f0aa9000640 1 --2- 172.21.15.67:0/2620315922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0aa41038e0 0x7f0aa4103d00 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:04.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.458+0000 7f0aa9000640 1 -- 172.21.15.67:0/2620315922 >> 172.21.15.67:0/2620315922 conn(0x7f0aa40fee50 msgr2=0x7f0aa4101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:04.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.458+0000 7f0aa9000640 1 -- 172.21.15.67:0/2620315922 shutdown_connections 2024-03-28T16:01:04.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.458+0000 7f0aa9000640 1 -- 172.21.15.67:0/2620315922 wait complete. 2024-03-28T16:01:04.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.459+0000 7f0aa9000640 1 Processor -- start 2024-03-28T16:01:04.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.459+0000 7f0aa9000640 1 -- start start 2024-03-28T16:01:04.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.460+0000 7f0aa9000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0aa41038e0 0x7f0aa4199b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:04.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.460+0000 7f0aa9000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0aa4104ad0 0x7f0aa419a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:04.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.460+0000 7f0aa9000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0aa4105490 0x7f0aa41a10e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:04.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.460+0000 7f0aa9000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa410f460 con 0x7f0aa4104ad0 2024-03-28T16:01:04.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.460+0000 7f0aa9000640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa410f2e0 con 0x7f0aa41038e0 2024-03-28T16:01:04.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.460+0000 7f0aa9000640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa410f5e0 con 0x7f0aa4105490 2024-03-28T16:01:04.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.460+0000 7f0aa1400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0aa4104ad0 0x7f0aa419a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:04.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.460+0000 7f0aa1e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0aa41038e0 0x7f0aa4199b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:04.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.460+0000 7f0aa1400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0aa4104ad0 0x7f0aa419a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45502/0 (socket says 172.21.15.67:45502) 2024-03-28T16:01:04.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.460+0000 7f0aa1e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0aa41038e0 0x7f0aa4199b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.67:37014/0 (socket says 172.21.15.67:37014) 2024-03-28T16:01:04.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.460+0000 7f0aa1400640 1 -- 172.21.15.67:0/4213263330 learned_addr learned my addr 172.21.15.67:0/4213263330 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:01:04.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.460+0000 7f0aa2800640 1 --2- 172.21.15.67:0/4213263330 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0aa4105490 0x7f0aa41a10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:04.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.461+0000 7f0aa1400640 1 -- 172.21.15.67:0/4213263330 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0aa4105490 msgr2=0x7f0aa41a10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:04.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.461+0000 7f0aa1400640 1 --2- 172.21.15.67:0/4213263330 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0aa4105490 0x7f0aa41a10e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:04.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.461+0000 7f0aa1400640 1 -- 172.21.15.67:0/4213263330 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0aa41038e0 msgr2=0x7f0aa4199b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:04.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.461+0000 7f0aa1400640 1 --2- 172.21.15.67:0/4213263330 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0aa41038e0 0x7f0aa4199b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:04.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.461+0000 7f0aa1400640 1 -- 172.21.15.67:0/4213263330 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aa41a1750 con 0x7f0aa4104ad0 2024-03-28T16:01:04.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.461+0000 7f0aa1e00640 1 --2- 172.21.15.67:0/4213263330 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0aa41038e0 0x7f0aa4199b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T16:01:04.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.461+0000 7f0aa1400640 1 --2- 172.21.15.67:0/4213263330 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0aa4104ad0 0x7f0aa419a070 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f0a9000ea60 tx=0x7f0a9000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:04.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.461+0000 7f0a9aa00640 1 -- 172.21.15.67:0/4213263330 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a9000cde0 con 0x7f0aa4104ad0 2024-03-28T16:01:04.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.461+0000 7f0aa9000640 1 -- 172.21.15.67:0/4213263330 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0aa41a1a40 con 0x7f0aa4104ad0 2024-03-28T16:01:04.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.461+0000 7f0a9aa00640 1 -- 172.21.15.67:0/4213263330 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0a900035f0 con 0x7f0aa4104ad0 2024-03-28T16:01:04.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.461+0000 7f0a9aa00640 1 -- 172.21.15.67:0/4213263330 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a90010690 con 0x7f0aa4104ad0 2024-03-28T16:01:04.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.461+0000 7f0aa9000640 1 -- 172.21.15.67:0/4213263330 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0aa41a1fe0 con 0x7f0aa4104ad0 2024-03-28T16:01:04.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.463+0000 7f0aa9000640 1 -- 172.21.15.67:0/4213263330 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0aa410acd0 con 0x7f0aa4104ad0 2024-03-28T16:01:04.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.467+0000 7f0a9aa00640 1 -- 172.21.15.67:0/4213263330 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f0a900037a0 con 0x7f0aa4104ad0 2024-03-28T16:01:04.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.469+0000 7f0a9aa00640 1 --2- 172.21.15.67:0/4213263330 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f0a880a2800 0x7f0a880a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:04.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.469+0000 7f0a9aa00640 1 -- 172.21.15.67:0/4213263330 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f0a900ce960 con 0x7f0aa4104ad0 2024-03-28T16:01:04.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.469+0000 7f0aa1e00640 1 --2- 172.21.15.67:0/4213263330 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f0a880a2800 0x7f0a880a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:04.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.470+0000 7f0aa1e00640 1 --2- 172.21.15.67:0/4213263330 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f0a880a2800 0x7f0a880a4cc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0a9402c240 tx=0x7f0a9402c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:04.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.473+0000 7f0a9aa00640 1 -- 172.21.15.67:0/4213263330 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0a9009be40 con 0x7f0aa4104ad0 2024-03-28T16:01:04.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.680+0000 7f0aa9000640 1 -- 172.21.15.67:0/4213263330 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f0aa40622a0 con 0x7f0aa4104ad0 2024-03-28T16:01:04.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.680+0000 7f0a9aa00640 1 -- 172.21.15.67:0/4213263330 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v86) v1 ==== 84+0+65 (secure 0 0 0) 0x7f0a9009b740 con 0x7f0aa4104ad0 2024-03-28T16:01:04.681 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:01:04.681 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-03-28T16:01:04.682 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:01:04.682 INFO:teuthology.orchestra.run.smithi067.stdout: "poolnum": 1, 2024-03-28T16:01:04.682 INFO:teuthology.orchestra.run.smithi067.stdout: "poolname": ".mgr" 2024-03-28T16:01:04.682 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:01:04.682 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-03-28T16:01:04.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.683+0000 7f0aa9000640 1 -- 172.21.15.67:0/4213263330 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f0a880a2800 msgr2=0x7f0a880a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:04.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.683+0000 7f0aa9000640 1 --2- 172.21.15.67:0/4213263330 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f0a880a2800 0x7f0a880a4cc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0a9402c240 tx=0x7f0a9402c7e0 comp rx=0 tx=0).stop 2024-03-28T16:01:04.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.683+0000 7f0aa9000640 1 -- 172.21.15.67:0/4213263330 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0aa4104ad0 msgr2=0x7f0aa419a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:04.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.683+0000 7f0aa9000640 1 --2- 172.21.15.67:0/4213263330 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0aa4104ad0 0x7f0aa419a070 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f0a9000ea60 tx=0x7f0a9000ef20 comp rx=0 tx=0).stop 2024-03-28T16:01:04.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.683+0000 7f0aa9000640 1 -- 172.21.15.67:0/4213263330 shutdown_connections 2024-03-28T16:01:04.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.683+0000 7f0aa9000640 1 --2- 172.21.15.67:0/4213263330 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f0a880a2800 0x7f0a880a4cc0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:04.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.683+0000 7f0aa9000640 1 --2- 172.21.15.67:0/4213263330 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0aa4105490 0x7f0aa41a10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:04.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.683+0000 7f0aa9000640 1 --2- 172.21.15.67:0/4213263330 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0aa4104ad0 0x7f0aa419a070 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:04.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.683+0000 7f0aa9000640 1 --2- 172.21.15.67:0/4213263330 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0aa41038e0 0x7f0aa4199b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:04.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.683+0000 7f0aa9000640 1 -- 172.21.15.67:0/4213263330 >> 172.21.15.67:0/4213263330 conn(0x7f0aa40fee50 msgr2=0x7f0aa4107ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:04.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.683+0000 7f0aa9000640 1 -- 172.21.15.67:0/4213263330 shutdown_connections 2024-03-28T16:01:04.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.683+0000 7f0aa9000640 1 -- 172.21.15.67:0/4213263330 wait complete. 2024-03-28T16:01:04.706 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-28T16:01:04.708 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-28T16:01:04.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.900+0000 7f7f7f200640 1 -- 172.21.15.67:0/1665759585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f78075660 msgr2=0x7f7f78073ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:04.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.900+0000 7f7f7f200640 1 --2- 172.21.15.67:0/1665759585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f78075660 0x7f7f78073ab0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f7f68009900 tx=0x7f7f6802bcd0 comp rx=0 tx=0).stop 2024-03-28T16:01:04.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.901+0000 7f7f7f200640 1 -- 172.21.15.67:0/1665759585 shutdown_connections 2024-03-28T16:01:04.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.901+0000 7f7f7f200640 1 --2- 172.21.15.67:0/1665759585 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f78109580 0x7f7f7810ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:04.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.901+0000 7f7f7f200640 1 --2- 172.21.15.67:0/1665759585 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7f78074080 0x7f7f780744e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:04.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.901+0000 7f7f7f200640 1 --2- 172.21.15.67:0/1665759585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f78075660 0x7f7f78073ab0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:04.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.901+0000 7f7f7f200640 1 -- 172.21.15.67:0/1665759585 >> 172.21.15.67:0/1665759585 conn(0x7f7f780fcc40 msgr2=0x7f7f780ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:04.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.901+0000 7f7f7f200640 1 -- 172.21.15.67:0/1665759585 shutdown_connections 2024-03-28T16:01:04.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.901+0000 7f7f7f200640 1 -- 172.21.15.67:0/1665759585 wait complete. 2024-03-28T16:01:04.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.902+0000 7f7f7f200640 1 Processor -- start 2024-03-28T16:01:04.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.902+0000 7f7f7f200640 1 -- start start 2024-03-28T16:01:04.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.902+0000 7f7f7f200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7f78074080 0x7f7f78199b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:04.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.902+0000 7f7f7f200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f78075660 0x7f7f7819a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:04.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.902+0000 7f7f7f200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f78109580 0x7f7f781a10c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:04.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.902+0000 7f7f7f200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f7810f450 con 0x7f7f78109580 2024-03-28T16:01:04.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.902+0000 7f7f7f200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f7810f2d0 con 0x7f7f78075660 2024-03-28T16:01:04.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.902+0000 7f7f7f200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f7810f5d0 con 0x7f7f78074080 2024-03-28T16:01:04.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.902+0000 7f7f7d200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f78109580 0x7f7f781a10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:04.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.903+0000 7f7f7d200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f78109580 0x7f7f781a10c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45514/0 (socket says 172.21.15.67:45514) 2024-03-28T16:01:04.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.903+0000 7f7f7d200640 1 -- 172.21.15.67:0/836998732 learned_addr learned my addr 172.21.15.67:0/836998732 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:01:04.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.903+0000 7f7f77e00640 1 --2- 172.21.15.67:0/836998732 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7f78074080 0x7f7f78199b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:04.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.903+0000 7f7f7d200640 1 -- 172.21.15.67:0/836998732 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7f78074080 msgr2=0x7f7f78199b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:04.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.903+0000 7f7f7d200640 1 --2- 172.21.15.67:0/836998732 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7f78074080 0x7f7f78199b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:04.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.903+0000 7f7f77400640 1 --2- 172.21.15.67:0/836998732 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f78075660 0x7f7f7819a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:04.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.903+0000 7f7f7d200640 1 -- 172.21.15.67:0/836998732 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f78075660 msgr2=0x7f7f7819a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:04.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.903+0000 7f7f7d200640 1 --2- 172.21.15.67:0/836998732 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f78075660 0x7f7f7819a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:04.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.903+0000 7f7f7d200640 1 -- 172.21.15.67:0/836998732 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f781a1730 con 0x7f7f78109580 2024-03-28T16:01:04.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.903+0000 7f7f77400640 1 --2- 172.21.15.67:0/836998732 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f78075660 0x7f7f7819a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:01:04.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.903+0000 7f7f7d200640 1 --2- 172.21.15.67:0/836998732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f78109580 0x7f7f781a10c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7f7000eb10 tx=0x7f7f7000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:04.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.904+0000 7f7f74c00640 1 -- 172.21.15.67:0/836998732 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f70004b20 con 0x7f7f78109580 2024-03-28T16:01:04.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.904+0000 7f7f7f200640 1 -- 172.21.15.67:0/836998732 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f781a1a20 con 0x7f7f78109580 2024-03-28T16:01:04.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.904+0000 7f7f74c00640 1 -- 172.21.15.67:0/836998732 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7f70004cc0 con 0x7f7f78109580 2024-03-28T16:01:04.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.904+0000 7f7f74c00640 1 -- 172.21.15.67:0/836998732 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f70010690 con 0x7f7f78109580 2024-03-28T16:01:04.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.904+0000 7f7f7f200640 1 -- 172.21.15.67:0/836998732 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f781a1f70 con 0x7f7f78109580 2024-03-28T16:01:04.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.905+0000 7f7f7f200640 1 -- 172.21.15.67:0/836998732 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f78108340 con 0x7f7f78109580 2024-03-28T16:01:04.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.907+0000 7f7f74c00640 1 -- 172.21.15.67:0/836998732 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f7f70005020 con 0x7f7f78109580 2024-03-28T16:01:04.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.908+0000 7f7f74c00640 1 --2- 172.21.15.67:0/836998732 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7f5c0a2880 0x7f7f5c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:04.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.908+0000 7f7f74c00640 1 -- 172.21.15.67:0/836998732 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f7f700cec00 con 0x7f7f78109580 2024-03-28T16:01:04.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.908+0000 7f7f77e00640 1 --2- 172.21.15.67:0/836998732 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7f5c0a2880 0x7f7f5c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:04.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.909+0000 7f7f77e00640 1 --2- 172.21.15.67:0/836998732 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7f5c0a2880 0x7f7f5c0a4d40 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f7f68000c00 tx=0x7f7f6802c880 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:04.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:04.913+0000 7f7f74c00640 1 -- 172.21.15.67:0/836998732 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7f7009c0a0 con 0x7f7f78109580 2024-03-28T16:01:05.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:04 smithi067 ceph-mon[31707]: pgmap v358: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:01:05.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:04 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:01:05.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:04 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/945295127' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T16:01:05.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:04 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/4213263330' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-28T16:01:05.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:05.159+0000 7f7f7f200640 1 -- 172.21.15.67:0/836998732 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f7f7802d070 con 0x7f7f78109580 2024-03-28T16:01:05.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:04 smithi150 ceph-mon[34825]: pgmap v358: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:01:05.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:04 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:01:05.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:04 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/945295127' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T16:01:05.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:04 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/4213263330' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-28T16:01:05.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:04 smithi183 ceph-mon[35580]: pgmap v358: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:01:05.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:04 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:01:05.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:04 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/945295127' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T16:01:05.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:04 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/4213263330' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-28T16:01:06.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:06.162+0000 7f7f74c00640 1 -- 172.21.15.67:0/836998732 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v87) v1 ==== 138+0+0 (secure 0 0 0) 0x7f7f7009b9a0 con 0x7f7f78109580 2024-03-28T16:01:06.164 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' created 2024-03-28T16:01:06.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:06.167+0000 7f7f7f200640 1 -- 172.21.15.67:0/836998732 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7f5c0a2880 msgr2=0x7f7f5c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:06.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:06.167+0000 7f7f7f200640 1 --2- 172.21.15.67:0/836998732 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7f5c0a2880 0x7f7f5c0a4d40 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f7f68000c00 tx=0x7f7f6802c880 comp rx=0 tx=0).stop 2024-03-28T16:01:06.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:06.167+0000 7f7f7f200640 1 -- 172.21.15.67:0/836998732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f78109580 msgr2=0x7f7f781a10c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:06.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:06.167+0000 7f7f7f200640 1 --2- 172.21.15.67:0/836998732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f78109580 0x7f7f781a10c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7f7000eb10 tx=0x7f7f7000c2d0 comp rx=0 tx=0).stop 2024-03-28T16:01:06.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:06.168+0000 7f7f7f200640 1 -- 172.21.15.67:0/836998732 shutdown_connections 2024-03-28T16:01:06.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:06.168+0000 7f7f7f200640 1 --2- 172.21.15.67:0/836998732 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7f5c0a2880 0x7f7f5c0a4d40 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:06.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:06.168+0000 7f7f7f200640 1 --2- 172.21.15.67:0/836998732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f78109580 0x7f7f781a10c0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:06.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:06.168+0000 7f7f7f200640 1 --2- 172.21.15.67:0/836998732 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f78075660 0x7f7f7819a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:06.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:06.168+0000 7f7f7f200640 1 --2- 172.21.15.67:0/836998732 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7f78074080 0x7f7f78199b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:06.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:06.168+0000 7f7f7f200640 1 -- 172.21.15.67:0/836998732 >> 172.21.15.67:0/836998732 conn(0x7f7f780fcc40 msgr2=0x7f7f78105510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:06.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:06.168+0000 7f7f7f200640 1 -- 172.21.15.67:0/836998732 shutdown_connections 2024-03-28T16:01:06.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:06.169+0000 7f7f7f200640 1 -- 172.21.15.67:0/836998732 wait complete. 2024-03-28T16:01:06.197 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-28T16:01:06.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:06.433+0000 7fdd97200640 1 -- 172.21.15.67:0/1955680728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd901049b0 msgr2=0x7fdd90104e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:06.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:06.433+0000 7fdd97200640 1 --2- 172.21.15.67:0/1955680728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd901049b0 0x7fdd90104e30 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fdd80009900 tx=0x7fdd8002bc30 comp rx=0 tx=0).stop 2024-03-28T16:01:06.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:06.433+0000 7fdd97200640 1 -- 172.21.15.67:0/1955680728 shutdown_connections 2024-03-28T16:01:06.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:06.433+0000 7fdd97200640 1 --2- 172.21.15.67:0/1955680728 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd90105370 0x7fdd9010bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:06.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:06.433+0000 7fdd97200640 1 --2- 172.21.15.67:0/1955680728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd901049b0 0x7fdd90104e30 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:06.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:06.433+0000 7fdd97200640 1 --2- 172.21.15.67:0/1955680728 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd901037c0 0x7fdd90103be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:06.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:06.433+0000 7fdd97200640 1 -- 172.21.15.67:0/1955680728 >> 172.21.15.67:0/1955680728 conn(0x7fdd900fed50 msgr2=0x7fdd90101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:06.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:06.433+0000 7fdd97200640 1 -- 172.21.15.67:0/1955680728 shutdown_connections 2024-03-28T16:01:06.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:06.434+0000 7fdd97200640 1 -- 172.21.15.67:0/1955680728 wait complete. 2024-03-28T16:01:06.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:06.435+0000 7fdd97200640 1 Processor -- start 2024-03-28T16:01:06.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:06.435+0000 7fdd97200640 1 -- start start 2024-03-28T16:01:06.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:06.435+0000 7fdd97200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd901037c0 0x7fdd90079450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:06.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:06.435+0000 7fdd97200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd901049b0 0x7fdd90079990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:06.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:06.435+0000 7fdd97200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd90105370 0x7fdd90075fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:06.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:06.435+0000 7fdd97200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd9010f340 con 0x7fdd901049b0 2024-03-28T16:01:06.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:06.435+0000 7fdd97200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd9010f1c0 con 0x7fdd90105370 2024-03-28T16:01:06.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:06.435+0000 7fdd97200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd9010f4c0 con 0x7fdd901037c0 2024-03-28T16:01:06.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:06.435+0000 7fdd95400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd901049b0 0x7fdd90079990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:06.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:06.436+0000 7fdd95400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd901049b0 0x7fdd90079990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45524/0 (socket says 172.21.15.67:45524) 2024-03-28T16:01:06.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:06.436+0000 7fdd95400640 1 -- 172.21.15.67:0/2270803350 learned_addr learned my addr 172.21.15.67:0/2270803350 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:01:06.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:06.436+0000 7fdd95e00640 1 --2- 172.21.15.67:0/2270803350 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd901037c0 0x7fdd90079450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:06.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:06.436+0000 7fdd95400640 1 -- 172.21.15.67:0/2270803350 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd901037c0 msgr2=0x7fdd90079450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:06.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:06.436+0000 7fdd95400640 1 --2- 172.21.15.67:0/2270803350 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd901037c0 0x7fdd90079450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:06.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:06.436+0000 7fdd95400640 1 -- 172.21.15.67:0/2270803350 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd90105370 msgr2=0x7fdd90075fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:06.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:06.436+0000 7fdd95400640 1 --2- 172.21.15.67:0/2270803350 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd90105370 0x7fdd90075fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:06.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:06.436+0000 7fdd95400640 1 -- 172.21.15.67:0/2270803350 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd90076880 con 0x7fdd901049b0 2024-03-28T16:01:06.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:06.436+0000 7fdd95400640 1 --2- 172.21.15.67:0/2270803350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd901049b0 0x7fdd90079990 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fdd80000c00 tx=0x7fdd800042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:06.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:06.437+0000 7fdd8aa00640 1 -- 172.21.15.67:0/2270803350 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd80043020 con 0x7fdd901049b0 2024-03-28T16:01:06.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:06.437+0000 7fdd8aa00640 1 -- 172.21.15.67:0/2270803350 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fdd800054c0 con 0x7fdd901049b0 2024-03-28T16:01:06.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:06.437+0000 7fdd8aa00640 1 -- 172.21.15.67:0/2270803350 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd80003d70 con 0x7fdd901049b0 2024-03-28T16:01:06.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:06.437+0000 7fdd97200640 1 -- 172.21.15.67:0/2270803350 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd901a5e10 con 0x7fdd901049b0 2024-03-28T16:01:06.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:06.437+0000 7fdd97200640 1 -- 172.21.15.67:0/2270803350 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd901a61c0 con 0x7fdd901049b0 2024-03-28T16:01:06.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:06.438+0000 7fdd97200640 1 -- 172.21.15.67:0/2270803350 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd9010ab60 con 0x7fdd901049b0 2024-03-28T16:01:06.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:06.440+0000 7fdd8aa00640 1 -- 172.21.15.67:0/2270803350 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fdd80046020 con 0x7fdd901049b0 2024-03-28T16:01:06.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:06.441+0000 7fdd8aa00640 1 --2- 172.21.15.67:0/2270803350 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fdd780a2800 0x7fdd780a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:06.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:06.442+0000 7fdd8aa00640 1 -- 172.21.15.67:0/2270803350 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 7497+0+0 (secure 0 0 0) 0x7fdd800f0ff0 con 0x7fdd901049b0 2024-03-28T16:01:06.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:06.442+0000 7fdd95e00640 1 --2- 172.21.15.67:0/2270803350 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fdd780a2800 0x7fdd780a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:06.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:06.443+0000 7fdd95e00640 1 --2- 172.21.15.67:0/2270803350 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fdd780a2800 0x7fdd780a4cc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fdd90104810 tx=0x7fdd8400a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:06.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:06.445+0000 7fdd8aa00640 1 -- 172.21.15.67:0/2270803350 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fdd8004d020 con 0x7fdd901049b0 2024-03-28T16:01:06.451 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:06 smithi183 ceph-mon[35580]: pgmap v359: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:01:06.451 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:06 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:01:06.451 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:06 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:01:06.451 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:06 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:01:06.451 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:06 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:01:06.451 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:06 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/836998732' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-28T16:01:06.452 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:06 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:01:06.452 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:06 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:01:06.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:06 smithi150 ceph-mon[34825]: pgmap v359: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:01:06.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:06 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:01:06.553 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:06 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:01:06.553 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:06 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:01:06.553 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:06 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:01:06.553 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:06 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/836998732' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-28T16:01:06.553 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:06 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:01:06.553 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:06 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:01:06.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:06 smithi067 ceph-mon[31707]: pgmap v359: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:01:06.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:06 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:01:06.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:06 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:01:06.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:06 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:01:06.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:06 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:01:06.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:06 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/836998732' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-28T16:01:06.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:06 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:01:06.571 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:06 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:01:06.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:06.650+0000 7fdd97200640 1 -- 172.21.15.67:0/2270803350 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fdd9002d230 con 0x7fdd901049b0 2024-03-28T16:01:07.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:07.210+0000 7fdd8aa00640 1 -- 172.21.15.67:0/2270803350 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v88) v1 ==== 170+0+0 (secure 0 0 0) 0x7fdd800be360 con 0x7fdd901049b0 2024-03-28T16:01:07.212 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' created 2024-03-28T16:01:07.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:07.214+0000 7fdd97200640 1 -- 172.21.15.67:0/2270803350 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fdd780a2800 msgr2=0x7fdd780a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:07.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:07.214+0000 7fdd97200640 1 --2- 172.21.15.67:0/2270803350 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fdd780a2800 0x7fdd780a4cc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fdd90104810 tx=0x7fdd8400a2d0 comp rx=0 tx=0).stop 2024-03-28T16:01:07.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:07.214+0000 7fdd97200640 1 -- 172.21.15.67:0/2270803350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd901049b0 msgr2=0x7fdd90079990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:07.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:07.214+0000 7fdd97200640 1 --2- 172.21.15.67:0/2270803350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd901049b0 0x7fdd90079990 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fdd80000c00 tx=0x7fdd800042f0 comp rx=0 tx=0).stop 2024-03-28T16:01:07.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:07.215+0000 7fdd97200640 1 -- 172.21.15.67:0/2270803350 shutdown_connections 2024-03-28T16:01:07.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:07.215+0000 7fdd97200640 1 --2- 172.21.15.67:0/2270803350 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fdd780a2800 0x7fdd780a4cc0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:07.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:07.215+0000 7fdd97200640 1 --2- 172.21.15.67:0/2270803350 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd90105370 0x7fdd90075fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:07.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:07.215+0000 7fdd97200640 1 --2- 172.21.15.67:0/2270803350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd901049b0 0x7fdd90079990 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:07.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:07.215+0000 7fdd97200640 1 --2- 172.21.15.67:0/2270803350 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd901037c0 0x7fdd90079450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:07.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:07.215+0000 7fdd97200640 1 -- 172.21.15.67:0/2270803350 >> 172.21.15.67:0/2270803350 conn(0x7fdd900fed50 msgr2=0x7fdd90107d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:07.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:07.215+0000 7fdd97200640 1 -- 172.21.15.67:0/2270803350 shutdown_connections 2024-03-28T16:01:07.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:07.215+0000 7fdd97200640 1 -- 172.21.15.67:0/2270803350 wait complete. 2024-03-28T16:01:07.228 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:07 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/836998732' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-28T16:01:07.251 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-28T16:01:07.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:07.439+0000 7fb9e5800640 1 -- 172.21.15.67:0/255443968 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9e0101670 msgr2=0x7fb9e0101af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:07.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:07.439+0000 7fb9e5800640 1 --2- 172.21.15.67:0/255443968 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9e0101670 0x7fb9e0101af0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fb9c8009ca0 tx=0x7fb9c802c0f0 comp rx=0 tx=0).stop 2024-03-28T16:01:07.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:07.439+0000 7fb9e5800640 1 -- 172.21.15.67:0/255443968 shutdown_connections 2024-03-28T16:01:07.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:07.439+0000 7fb9e5800640 1 --2- 172.21.15.67:0/255443968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9e0102030 0x7fb9e01088c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:07.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:07.439+0000 7fb9e5800640 1 --2- 172.21.15.67:0/255443968 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9e0101670 0x7fb9e0101af0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:07.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:07.439+0000 7fb9e5800640 1 --2- 172.21.15.67:0/255443968 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb9e0100480 0x7fb9e01008a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:07.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:07.439+0000 7fb9e5800640 1 -- 172.21.15.67:0/255443968 >> 172.21.15.67:0/255443968 conn(0x7fb9e00fba10 msgr2=0x7fb9e00fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:07.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:07.439+0000 7fb9e5800640 1 -- 172.21.15.67:0/255443968 shutdown_connections 2024-03-28T16:01:07.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:07.440+0000 7fb9e5800640 1 -- 172.21.15.67:0/255443968 wait complete. 2024-03-28T16:01:07.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:07.440+0000 7fb9e5800640 1 Processor -- start 2024-03-28T16:01:07.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:07.441+0000 7fb9e5800640 1 -- start start 2024-03-28T16:01:07.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:07.441+0000 7fb9e5800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9e0100480 0x7fb9e0197740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:07.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:07.441+0000 7fb9e5800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb9e0101670 0x7fb9e0197c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:07.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:07.441+0000 7fb9e5800640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9e0102030 0x7fb9e019ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:07.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:07.441+0000 7fb9e5800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e010bfc0 con 0x7fb9e0100480 2024-03-28T16:01:07.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:07.441+0000 7fb9e5800640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e010be40 con 0x7fb9e0102030 2024-03-28T16:01:07.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:07.441+0000 7fb9e5800640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e010c140 con 0x7fb9e0101670 2024-03-28T16:01:07.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:07.441+0000 7fb9dfe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9e0100480 0x7fb9e0197740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:07.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:07.441+0000 7fb9dfe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9e0100480 0x7fb9e0197740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45544/0 (socket says 172.21.15.67:45544) 2024-03-28T16:01:07.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:07.441+0000 7fb9dfe00640 1 -- 172.21.15.67:0/2121311058 learned_addr learned my addr 172.21.15.67:0/2121311058 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:01:07.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:07.442+0000 7fb9dfe00640 1 -- 172.21.15.67:0/2121311058 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb9e0101670 msgr2=0x7fb9e0197c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:07.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:07.442+0000 7fb9e4e00640 1 --2- 172.21.15.67:0/2121311058 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9e0102030 0x7fb9e019ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:07.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:07.442+0000 7fb9dfe00640 1 --2- 172.21.15.67:0/2121311058 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb9e0101670 0x7fb9e0197c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:07.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:07.442+0000 7fb9dfe00640 1 -- 172.21.15.67:0/2121311058 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9e0102030 msgr2=0x7fb9e019ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:07.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:07.442+0000 7fb9dfe00640 1 --2- 172.21.15.67:0/2121311058 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9e0102030 0x7fb9e019ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:07.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:07.442+0000 7fb9dfe00640 1 -- 172.21.15.67:0/2121311058 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9e019f360 con 0x7fb9e0100480 2024-03-28T16:01:07.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:07.442+0000 7fb9e4e00640 1 --2- 172.21.15.67:0/2121311058 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9e0102030 0x7fb9e019ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:01:07.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:07.442+0000 7fb9dfe00640 1 --2- 172.21.15.67:0/2121311058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9e0100480 0x7fb9e0197740 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb9d800c970 tx=0x7fb9d800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:07.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:07.442+0000 7fb9dcc00640 1 -- 172.21.15.67:0/2121311058 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9d8007c10 con 0x7fb9e0100480 2024-03-28T16:01:07.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:07.442+0000 7fb9e5800640 1 -- 172.21.15.67:0/2121311058 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9e019f650 con 0x7fb9e0100480 2024-03-28T16:01:07.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:07.442+0000 7fb9dcc00640 1 -- 172.21.15.67:0/2121311058 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb9d8007db0 con 0x7fb9e0100480 2024-03-28T16:01:07.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:07.442+0000 7fb9dcc00640 1 -- 172.21.15.67:0/2121311058 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9d800f7f0 con 0x7fb9e0100480 2024-03-28T16:01:07.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:07.442+0000 7fb9e5800640 1 -- 172.21.15.67:0/2121311058 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9e019fbf0 con 0x7fb9e0100480 2024-03-28T16:01:07.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:07.443+0000 7fb9e5800640 1 -- 172.21.15.67:0/2121311058 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9e019feb0 con 0x7fb9e0100480 2024-03-28T16:01:07.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:07.445+0000 7fb9dcc00640 1 -- 172.21.15.67:0/2121311058 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fb9d8018480 con 0x7fb9e0100480 2024-03-28T16:01:07.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:07.447+0000 7fb9dcc00640 1 --2- 172.21.15.67:0/2121311058 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb9c40a2800 0x7fb9c40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:07.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:07.447+0000 7fb9dcc00640 1 -- 172.21.15.67:0/2121311058 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 7879+0+0 (secure 0 0 0) 0x7fb9d80ceff0 con 0x7fb9e0100480 2024-03-28T16:01:07.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:07.447+0000 7fb9df400640 1 --2- 172.21.15.67:0/2121311058 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb9c40a2800 0x7fb9c40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:07.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:07.448+0000 7fb9df400640 1 --2- 172.21.15.67:0/2121311058 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb9c40a2800 0x7fb9c40a4cc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fb9c8004af0 tx=0x7fb9c8032340 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:07.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:07.450+0000 7fb9dcc00640 1 -- 172.21.15.67:0/2121311058 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb9d809c2c0 con 0x7fb9e0100480 2024-03-28T16:01:07.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:07 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/836998732' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-28T16:01:07.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:07 smithi150 ceph-mon[34825]: osdmap e87: 12 total, 12 up, 12 in 2024-03-28T16:01:07.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:07 smithi067 ceph-mon[31707]: osdmap e87: 12 total, 12 up, 12 in 2024-03-28T16:01:07.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:07 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/836998732' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-28T16:01:07.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:07 smithi183 ceph-mon[35580]: osdmap e87: 12 total, 12 up, 12 in 2024-03-28T16:01:07.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:07.694+0000 7fb9e5800640 1 -- 172.21.15.67:0/2121311058 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fb9e01a02b0 con 0x7fb9e0100480 2024-03-28T16:01:08.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.268+0000 7fb9dcc00640 1 -- 172.21.15.67:0/2121311058 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7fb9d809bbc0 con 0x7fb9e0100480 2024-03-28T16:01:08.272 INFO:teuthology.orchestra.run.smithi067.stderr:new fs with metadata pool 2 and data pool 3 2024-03-28T16:01:08.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.273+0000 7fb9e5800640 1 -- 172.21.15.67:0/2121311058 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb9c40a2800 msgr2=0x7fb9c40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:08.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.273+0000 7fb9e5800640 1 --2- 172.21.15.67:0/2121311058 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb9c40a2800 0x7fb9c40a4cc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fb9c8004af0 tx=0x7fb9c8032340 comp rx=0 tx=0).stop 2024-03-28T16:01:08.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.273+0000 7fb9e5800640 1 -- 172.21.15.67:0/2121311058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9e0100480 msgr2=0x7fb9e0197740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:08.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.273+0000 7fb9e5800640 1 --2- 172.21.15.67:0/2121311058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9e0100480 0x7fb9e0197740 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb9d800c970 tx=0x7fb9d800ce30 comp rx=0 tx=0).stop 2024-03-28T16:01:08.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.274+0000 7fb9e5800640 1 -- 172.21.15.67:0/2121311058 shutdown_connections 2024-03-28T16:01:08.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.274+0000 7fb9e5800640 1 --2- 172.21.15.67:0/2121311058 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb9c40a2800 0x7fb9c40a4cc0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:08.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.274+0000 7fb9e5800640 1 --2- 172.21.15.67:0/2121311058 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9e0102030 0x7fb9e019ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:08.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.274+0000 7fb9e5800640 1 --2- 172.21.15.67:0/2121311058 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb9e0101670 0x7fb9e0197c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:08.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.274+0000 7fb9e5800640 1 --2- 172.21.15.67:0/2121311058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9e0100480 0x7fb9e0197740 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:08.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.274+0000 7fb9e5800640 1 -- 172.21.15.67:0/2121311058 >> 172.21.15.67:0/2121311058 conn(0x7fb9e00fba10 msgr2=0x7fb9e00fd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:08.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.274+0000 7fb9e5800640 1 -- 172.21.15.67:0/2121311058 shutdown_connections 2024-03-28T16:01:08.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.274+0000 7fb9e5800640 1 -- 172.21.15.67:0/2121311058 wait complete. 2024-03-28T16:01:08.299 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T16:01:08.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.494+0000 7fbd44c00640 1 -- 172.21.15.67:0/1577549069 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd40105490 msgr2=0x7fbd4010bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:08.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.494+0000 7fbd44c00640 1 --2- 172.21.15.67:0/1577549069 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd40105490 0x7fbd4010bd20 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fbd38009ca0 tx=0x7fbd3802c0f0 comp rx=0 tx=0).stop 2024-03-28T16:01:08.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.494+0000 7fbd44c00640 1 -- 172.21.15.67:0/1577549069 shutdown_connections 2024-03-28T16:01:08.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.494+0000 7fbd44c00640 1 --2- 172.21.15.67:0/1577549069 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd40105490 0x7fbd4010bd20 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:08.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.494+0000 7fbd44c00640 1 --2- 172.21.15.67:0/1577549069 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd40104ad0 0x7fbd40104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:08.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.494+0000 7fbd44c00640 1 --2- 172.21.15.67:0/1577549069 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbd401038e0 0x7fbd40103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:08.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.494+0000 7fbd44c00640 1 -- 172.21.15.67:0/1577549069 >> 172.21.15.67:0/1577549069 conn(0x7fbd400fee50 msgr2=0x7fbd40101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:08.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.494+0000 7fbd44c00640 1 -- 172.21.15.67:0/1577549069 shutdown_connections 2024-03-28T16:01:08.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.494+0000 7fbd44c00640 1 -- 172.21.15.67:0/1577549069 wait complete. 2024-03-28T16:01:08.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.495+0000 7fbd44c00640 1 Processor -- start 2024-03-28T16:01:08.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.495+0000 7fbd44c00640 1 -- start start 2024-03-28T16:01:08.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.495+0000 7fbd44c00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd401038e0 0x7fbd40199ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:08.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.495+0000 7fbd44c00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbd40104ad0 0x7fbd40199ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:08.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.495+0000 7fbd44c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd40105490 0x7fbd401a1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:08.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.495+0000 7fbd44c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd4010f410 con 0x7fbd40105490 2024-03-28T16:01:08.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.495+0000 7fbd44c00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd4010f290 con 0x7fbd401038e0 2024-03-28T16:01:08.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.495+0000 7fbd44c00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd4010f590 con 0x7fbd40104ad0 2024-03-28T16:01:08.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.496+0000 7fbd3e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd40105490 0x7fbd401a1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:08.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.496+0000 7fbd3e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd40105490 0x7fbd401a1060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45560/0 (socket says 172.21.15.67:45560) 2024-03-28T16:01:08.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.496+0000 7fbd3e800640 1 -- 172.21.15.67:0/533781036 learned_addr learned my addr 172.21.15.67:0/533781036 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:01:08.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.496+0000 7fbd3de00640 1 --2- 172.21.15.67:0/533781036 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd401038e0 0x7fbd40199ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:08.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.496+0000 7fbd3d400640 1 --2- 172.21.15.67:0/533781036 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbd40104ad0 0x7fbd40199ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:08.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.496+0000 7fbd3e800640 1 -- 172.21.15.67:0/533781036 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbd40104ad0 msgr2=0x7fbd40199ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:08.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.496+0000 7fbd3e800640 1 --2- 172.21.15.67:0/533781036 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbd40104ad0 0x7fbd40199ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:08.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.496+0000 7fbd3e800640 1 -- 172.21.15.67:0/533781036 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd401038e0 msgr2=0x7fbd40199ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:08.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.496+0000 7fbd3e800640 1 --2- 172.21.15.67:0/533781036 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd401038e0 0x7fbd40199ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:08.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.496+0000 7fbd3e800640 1 -- 172.21.15.67:0/533781036 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd401a16d0 con 0x7fbd40105490 2024-03-28T16:01:08.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.496+0000 7fbd3d400640 1 --2- 172.21.15.67:0/533781036 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbd40104ad0 0x7fbd40199ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T16:01:08.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.497+0000 7fbd3e800640 1 --2- 172.21.15.67:0/533781036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd40105490 0x7fbd401a1060 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fbd38032380 tx=0x7fbd38032820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:08.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.497+0000 7fbd36a00640 1 -- 172.21.15.67:0/533781036 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd38041020 con 0x7fbd40105490 2024-03-28T16:01:08.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.497+0000 7fbd44c00640 1 -- 172.21.15.67:0/533781036 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd401a1960 con 0x7fbd40105490 2024-03-28T16:01:08.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.497+0000 7fbd36a00640 1 -- 172.21.15.67:0/533781036 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbd38003e20 con 0x7fbd40105490 2024-03-28T16:01:08.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.497+0000 7fbd36a00640 1 -- 172.21.15.67:0/533781036 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd38039040 con 0x7fbd40105490 2024-03-28T16:01:08.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.497+0000 7fbd44c00640 1 -- 172.21.15.67:0/533781036 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd401a1ea0 con 0x7fbd40105490 2024-03-28T16:01:08.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.498+0000 7fbd44c00640 1 -- 172.21.15.67:0/533781036 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd4010acf0 con 0x7fbd40105490 2024-03-28T16:01:08.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.500+0000 7fbd36a00640 1 -- 172.21.15.67:0/533781036 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fbd38044020 con 0x7fbd40105490 2024-03-28T16:01:08.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.502+0000 7fbd36a00640 1 --2- 172.21.15.67:0/533781036 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fbd240a2800 0x7fbd240a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:08.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.502+0000 7fbd36a00640 1 -- 172.21.15.67:0/533781036 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fbd380f0a80 con 0x7fbd40105490 2024-03-28T16:01:08.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.503+0000 7fbd3de00640 1 --2- 172.21.15.67:0/533781036 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fbd240a2800 0x7fbd240a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:08.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.503+0000 7fbd3de00640 1 --2- 172.21.15.67:0/533781036 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fbd240a2800 0x7fbd240a4cc0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fbd40104930 tx=0x7fbd30005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:08.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.507+0000 7fbd36a00640 1 -- 172.21.15.67:0/533781036 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fbd380f2e20 con 0x7fbd40105490 2024-03-28T16:01:08.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:08 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/2270803350' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-28T16:01:08.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:08 smithi150 ceph-mon[34825]: pgmap v361: 33 pgs: 1 creating+peering, 1 active+clean, 31 unknown; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:01:08.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:08 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:01:08.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:08 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/2270803350' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-28T16:01:08.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:08 smithi150 ceph-mon[34825]: osdmap e88: 12 total, 12 up, 12 in 2024-03-28T16:01:08.553 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:08 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T16:01:08.553 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:08 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:01:08.553 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:08 smithi150 ceph-mon[34825]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-28T16:01:08.553 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:08 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T16:01:08.553 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:08 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/2121311058' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-28T16:01:08.553 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:08 smithi150 ceph-mon[34825]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-28T16:01:08.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:08 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/2270803350' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-28T16:01:08.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:08 smithi067 ceph-mon[31707]: pgmap v361: 33 pgs: 1 creating+peering, 1 active+clean, 31 unknown; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:01:08.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:08 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:01:08.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:08 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/2270803350' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-28T16:01:08.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:08 smithi067 ceph-mon[31707]: osdmap e88: 12 total, 12 up, 12 in 2024-03-28T16:01:08.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:08 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T16:01:08.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:08 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:01:08.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:08 smithi067 ceph-mon[31707]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-28T16:01:08.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:08 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T16:01:08.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:08 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/2121311058' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-28T16:01:08.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:08 smithi067 ceph-mon[31707]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-28T16:01:08.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:08 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/2270803350' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-28T16:01:08.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:08 smithi183 ceph-mon[35580]: pgmap v361: 33 pgs: 1 creating+peering, 1 active+clean, 31 unknown; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:01:08.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:08 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:01:08.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:08 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/2270803350' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-28T16:01:08.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:08 smithi183 ceph-mon[35580]: osdmap e88: 12 total, 12 up, 12 in 2024-03-28T16:01:08.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:08 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T16:01:08.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:08 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:01:08.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:08 smithi183 ceph-mon[35580]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-28T16:01:08.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:08 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T16:01:08.602 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:08 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/2121311058' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-28T16:01:08.602 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:08 smithi183 ceph-mon[35580]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-28T16:01:08.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.719+0000 7fbd44c00640 1 -- 172.21.15.67:0/533781036 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbd400622a0 con 0x7fbd40105490 2024-03-28T16:01:08.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.721+0000 7fbd36a00640 1 -- 172.21.15.67:0/533781036 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+22234 (secure 0 0 0) 0x7fbd3804b020 con 0x7fbd40105490 2024-03-28T16:01:08.723 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:01:08.724 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":89,"fsid":"b5f8225a-ed1a-11ee-95d0-87774f69a715","created":"2024-03-28T15:50:21.471924+0000","modified":"2024-03-28T16:01:07.696415+0000","last_up_change":"2024-03-28T16:00:01.369123+0000","last_in_change":"2024-03-28T15:59:37.515483+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T15:54:55.588007+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T16:01:05.719801+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T16:01:07.206303+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7c1791b6-9c0e-4512-a327-cad166bb5581","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6803","nonce":1894681701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6805","nonce":1894681701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6809","nonce":1894681701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6807","nonce":1894681701}]},"public_addr":"172.21.15.67:6803/1894681701","cluster_addr":"172.21.15.67:6805/1894681701","heartbeat_back_addr":"172.21.15.67:6809/1894681701","heartbeat_front_addr":"172.21.15.67:6807/1894681701","state":["exists","up"]},{"osd":1,"uuid":"0cdda7f3-d00c-4436-bcc1-49fb8072b2e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6801","nonce":1734465762}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6803","nonce":1734465762}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6807","nonce":1734465762}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6805","nonce":1734465762}]},"public_addr":"172.21.15.150:6801/1734465762","cluster_addr":"172.21.15.150:6803/1734465762","heartbeat_back_addr":"172.21.15.150:6807/1734465762","heartbeat_front_addr":"172.21.15.150:6805/1734465762","state":["exists","up"]},{"osd":2,"uuid":"a55b9a02-c3fd-4061-95c0-53a24a17ade0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6801","nonce":3398095931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6803","nonce":3398095931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6807","nonce":3398095931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6805","nonce":3398095931}]},"public_addr":"172.21.15.183:6801/3398095931","cluster_addr":"172.21.15.183:6803/3398095931","heartbeat_back_addr":"172.21.15.183:6807/3398095931","heartbeat_front_addr":"172.21.15.183:6805/3398095931","state":["exists","up"]},{"osd":3,"uuid":"d34940c0-5ce9-4b8b-aa36-4872414a043f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6811","nonce":1933541875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6813","nonce":1933541875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6817","nonce":1933541875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6815","nonce":1933541875}]},"public_addr":"172.21.15.67:6811/1933541875","cluster_addr":"172.21.15.67:6813/1933541875","heartbeat_back_addr":"172.21.15.67:6817/1933541875","heartbeat_front_addr":"172.21.15.67:6815/1933541875","state":["exists","up"]},{"osd":4,"uuid":"5cc3e260-83a3-4a4d-aa6a-34a3b12b2b5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6809","nonce":3775269993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6811","nonce":3775269993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6815","nonce":3775269993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6813","nonce":3775269993}]},"public_addr":"172.21.15.150:6809/3775269993","cluster_addr":"172.21.15.150:6811/3775269993","heartbeat_back_addr":"172.21.15.150:6815/3775269993","heartbeat_front_addr":"172.21.15.150:6813/3775269993","state":["exists","up"]},{"osd":5,"uuid":"a2542b21-d6ac-413e-8b9b-4a6548b20840","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6809","nonce":164284137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6811","nonce":164284137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6815","nonce":164284137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6813","nonce":164284137}]},"public_addr":"172.21.15.183:6809/164284137","cluster_addr":"172.21.15.183:6811/164284137","heartbeat_back_addr":"172.21.15.183:6815/164284137","heartbeat_front_addr":"172.21.15.183:6813/164284137","state":["exists","up"]},{"osd":6,"uuid":"8905a2d3-0f1b-4853-a218-22da2635150a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6819","nonce":3565528013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6821","nonce":3565528013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6825","nonce":3565528013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6823","nonce":3565528013}]},"public_addr":"172.21.15.67:6819/3565528013","cluster_addr":"172.21.15.67:6821/3565528013","heartbeat_back_addr":"172.21.15.67:6825/3565528013","heartbeat_front_addr":"172.21.15.67:6823/3565528013","state":["exists","up"]},{"osd":7,"uuid":"b275055b-b451-4180-91b1-e6e61a828c48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6817","nonce":3735115340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6819","nonce":3735115340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6823","nonce":3735115340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6821","nonce":3735115340}]},"public_addr":"172.21.15.150:6817/3735115340","cluster_addr":"172.21.15.150:6819/3735115340","heartbeat_back_addr":"172.21.15.150:6823/3735115340","heartbeat_front_addr":"172.21.15.150:6821/3735115340","state":["exists","up"]},{"osd":8,"uuid":"b6abf7ad-eb34-4f60-a4eb-348a92d7195b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6817","nonce":1025400095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6819","nonce":1025400095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6823","nonce":1025400095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6821","nonce":1025400095}]},"public_addr":"172.21.15.183:6817/1025400095","cluster_addr":"172.21.15.183:6819/1025400095","heartbeat_back_addr":"172.21.15.183:6823/1025400095","heartbeat_front_addr":"172.21.15.183:6821/1025400095","state":["exists","up"]},{"osd":9,"uuid":"28a78e3a-0657-40aa-816f-16a6fadadcbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6827","nonce":76885409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6829","nonce":76885409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6833","nonce":76885409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6831","nonce":76885409}]},"public_addr":"172.21.15.67:6827/76885409","cluster_addr":"172.21.15.67:6829/76885409","heartbeat_back_addr":"172.21.15.67:6833/76885409","heartbeat_front_addr":"172.21.15.67:6831/76885409","state":["exists","up"]},{"osd":10,"uuid":"d287907e-424f-4b49-a06e-4613c6de0474","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6825","nonce":3489982080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6827","nonce":3489982080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6831","nonce":3489982080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6829","nonce":3489982080}]},"public_addr":"172.21.15.150:6825/3489982080","cluster_addr":"172.21.15.150:6827/3489982080","heartbeat_back_addr":"172.21.15.150:6831/3489982080","heartbeat_front_addr":"172.21.15.150:6829/3489982080","state":["exists","up"]},{"osd":11,"uuid":"a3c29c47-a228-4f42-a23c-fbdb0c093699","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":85,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6825","nonce":1892639849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6826","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6827","nonce":1892639849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6831","nonce":1892639849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6829","nonce":1892639849}]},"public_addr":"172.21.15.183:6825/1892639849","cluster_addr":"172.21.15.183:6827/1892639849","heartbeat_back_addr":"172.21.15.183:6831/1892639849","heartbeat_front_addr":"172.21.15.183:6829/1892639849","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:53:45.646211+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:54:18.956590+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:54:52.161646+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:55:27.502822+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:56:01.162932+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:56:34.674452+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:57:09.298679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:57:43.230406+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:58:17.003821+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:58:51.085863+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:59:25.665065+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T16:00:00.263457+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6801/1631001968":"2024-03-29T15:51:38.875334+0000","172.21.15.67:6800/1631001968":"2024-03-29T15:51:38.875334+0000","172.21.15.67:0/1533056310":"2024-03-29T15:51:38.875334+0000","172.21.15.67:6801/3580491071":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/2030412458":"2024-03-29T15:51:03.859740+0000","172.21.15.67:6800/3580491071":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/2905307533":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/3763759988":"2024-03-29T15:51:38.875334+0000","172.21.15.67:0/4089399880":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/4239108661":"2024-03-29T15:51:38.875334+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T16:01:08.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.724+0000 7fbd44c00640 1 -- 172.21.15.67:0/533781036 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fbd240a2800 msgr2=0x7fbd240a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:08.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.724+0000 7fbd44c00640 1 --2- 172.21.15.67:0/533781036 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fbd240a2800 0x7fbd240a4cc0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fbd40104930 tx=0x7fbd30005f90 comp rx=0 tx=0).stop 2024-03-28T16:01:08.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.724+0000 7fbd44c00640 1 -- 172.21.15.67:0/533781036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd40105490 msgr2=0x7fbd401a1060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:08.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.724+0000 7fbd44c00640 1 --2- 172.21.15.67:0/533781036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd40105490 0x7fbd401a1060 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fbd38032380 tx=0x7fbd38032820 comp rx=0 tx=0).stop 2024-03-28T16:01:08.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.724+0000 7fbd44c00640 1 -- 172.21.15.67:0/533781036 shutdown_connections 2024-03-28T16:01:08.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.724+0000 7fbd44c00640 1 --2- 172.21.15.67:0/533781036 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fbd240a2800 0x7fbd240a4cc0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:08.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.724+0000 7fbd44c00640 1 --2- 172.21.15.67:0/533781036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd40105490 0x7fbd401a1060 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:08.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.724+0000 7fbd44c00640 1 --2- 172.21.15.67:0/533781036 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbd40104ad0 0x7fbd40199ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:08.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.724+0000 7fbd44c00640 1 --2- 172.21.15.67:0/533781036 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd401038e0 0x7fbd40199ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:08.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.724+0000 7fbd44c00640 1 -- 172.21.15.67:0/533781036 >> 172.21.15.67:0/533781036 conn(0x7fbd400fee50 msgr2=0x7fbd40107ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:08.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.724+0000 7fbd44c00640 1 -- 172.21.15.67:0/533781036 shutdown_connections 2024-03-28T16:01:08.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.724+0000 7fbd44c00640 1 -- 172.21.15.67:0/533781036 wait complete. 2024-03-28T16:01:08.744 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T16:01:08.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.960+0000 7f24f0a00640 1 -- 172.21.15.67:0/4131197282 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24ec103280 msgr2=0x7f24ec109b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:08.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.960+0000 7f24f0a00640 1 --2- 172.21.15.67:0/4131197282 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24ec103280 0x7f24ec109b10 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f24e4009ca0 tx=0x7f24e402c0f0 comp rx=0 tx=0).stop 2024-03-28T16:01:08.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.960+0000 7f24f0a00640 1 -- 172.21.15.67:0/4131197282 shutdown_connections 2024-03-28T16:01:08.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.960+0000 7f24f0a00640 1 --2- 172.21.15.67:0/4131197282 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24ec103280 0x7f24ec109b10 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:08.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.960+0000 7f24f0a00640 1 --2- 172.21.15.67:0/4131197282 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f24ec1028c0 0x7f24ec102d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:08.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.960+0000 7f24f0a00640 1 --2- 172.21.15.67:0/4131197282 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24ec1016d0 0x7f24ec101af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:08.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.960+0000 7f24f0a00640 1 -- 172.21.15.67:0/4131197282 >> 172.21.15.67:0/4131197282 conn(0x7f24ec0fcc40 msgr2=0x7f24ec0ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:08.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.960+0000 7f24f0a00640 1 -- 172.21.15.67:0/4131197282 shutdown_connections 2024-03-28T16:01:08.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.960+0000 7f24f0a00640 1 -- 172.21.15.67:0/4131197282 wait complete. 2024-03-28T16:01:08.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.961+0000 7f24f0a00640 1 Processor -- start 2024-03-28T16:01:08.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.961+0000 7f24f0a00640 1 -- start start 2024-03-28T16:01:08.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.961+0000 7f24f0a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f24ec1016d0 0x7f24ec075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:08.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.961+0000 7f24f0a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24ec1028c0 0x7f24ec073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:08.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.961+0000 7f24f0a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24ec103280 0x7f24ec073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:08.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.961+0000 7f24f0a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24ec10d200 con 0x7f24ec1028c0 2024-03-28T16:01:08.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.961+0000 7f24f0a00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24ec10d080 con 0x7f24ec103280 2024-03-28T16:01:08.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.961+0000 7f24f0a00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24ec10d380 con 0x7f24ec1016d0 2024-03-28T16:01:08.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.961+0000 7f24e9400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24ec1028c0 0x7f24ec073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:08.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.962+0000 7f24e9400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24ec1028c0 0x7f24ec073a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45576/0 (socket says 172.21.15.67:45576) 2024-03-28T16:01:08.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.962+0000 7f24e9400640 1 -- 172.21.15.67:0/59309820 learned_addr learned my addr 172.21.15.67:0/59309820 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:01:08.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.962+0000 7f24e9e00640 1 --2- 172.21.15.67:0/59309820 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f24ec1016d0 0x7f24ec075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:08.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.962+0000 7f24ea800640 1 --2- 172.21.15.67:0/59309820 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24ec103280 0x7f24ec073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:08.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.962+0000 7f24e9400640 1 -- 172.21.15.67:0/59309820 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f24ec1016d0 msgr2=0x7f24ec075450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:08.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.962+0000 7f24e9400640 1 --2- 172.21.15.67:0/59309820 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f24ec1016d0 0x7f24ec075450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:08.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.962+0000 7f24e9400640 1 -- 172.21.15.67:0/59309820 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24ec103280 msgr2=0x7f24ec073fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:08.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.962+0000 7f24e9400640 1 --2- 172.21.15.67:0/59309820 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24ec103280 0x7f24ec073fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:08.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.962+0000 7f24e9400640 1 -- 172.21.15.67:0/59309820 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24ec074860 con 0x7f24ec1028c0 2024-03-28T16:01:08.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.962+0000 7f24ea800640 1 --2- 172.21.15.67:0/59309820 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24ec103280 0x7f24ec073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T16:01:08.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.962+0000 7f24e9400640 1 --2- 172.21.15.67:0/59309820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24ec1028c0 0x7f24ec073a90 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f24d800d770 tx=0x7f24d800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:08.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.963+0000 7f24e2a00640 1 -- 172.21.15.67:0/59309820 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24d8017070 con 0x7f24ec1028c0 2024-03-28T16:01:08.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.963+0000 7f24e2a00640 1 -- 172.21.15.67:0/59309820 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f24d80054d0 con 0x7f24ec1028c0 2024-03-28T16:01:08.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.963+0000 7f24e2a00640 1 -- 172.21.15.67:0/59309820 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24d8004cf0 con 0x7f24ec1028c0 2024-03-28T16:01:08.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.963+0000 7f24f0a00640 1 -- 172.21.15.67:0/59309820 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24ec1a1970 con 0x7f24ec1028c0 2024-03-28T16:01:08.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.963+0000 7f24f0a00640 1 -- 172.21.15.67:0/59309820 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24ec1a1e70 con 0x7f24ec1028c0 2024-03-28T16:01:08.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.964+0000 7f24f0a00640 1 -- 172.21.15.67:0/59309820 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24ec1089f0 con 0x7f24ec1028c0 2024-03-28T16:01:08.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.968+0000 7f24e2a00640 1 -- 172.21.15.67:0/59309820 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f24d8004e90 con 0x7f24ec1028c0 2024-03-28T16:01:08.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.970+0000 7f24e2a00640 1 --2- 172.21.15.67:0/59309820 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f24d00a2800 0x7f24d00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:08.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.970+0000 7f24e2a00640 1 -- 172.21.15.67:0/59309820 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f24d80ce4c0 con 0x7f24ec1028c0 2024-03-28T16:01:08.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.970+0000 7f24e9e00640 1 --2- 172.21.15.67:0/59309820 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f24d00a2800 0x7f24d00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:08.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.971+0000 7f24e9e00640 1 --2- 172.21.15.67:0/59309820 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f24d00a2800 0x7f24d00a4cc0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f24dc0058c0 tx=0x7f24dc005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:08.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:08.973+0000 7f24e2a00640 1 -- 172.21.15.67:0/59309820 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f24d809b610 con 0x7f24ec1028c0 2024-03-28T16:01:09.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:09.174+0000 7f24f0a00640 1 -- 172.21.15.67:0/59309820 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f24ec0622d0 con 0x7f24ec1028c0 2024-03-28T16:01:09.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:09.178+0000 7f24e2a00640 1 -- 172.21.15.67:0/59309820 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+22234 (secure 0 0 0) 0x7f24d809af10 con 0x7f24ec1028c0 2024-03-28T16:01:09.180 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:01:09.180 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":89,"fsid":"b5f8225a-ed1a-11ee-95d0-87774f69a715","created":"2024-03-28T15:50:21.471924+0000","modified":"2024-03-28T16:01:07.696415+0000","last_up_change":"2024-03-28T16:00:01.369123+0000","last_in_change":"2024-03-28T15:59:37.515483+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T15:54:55.588007+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T16:01:05.719801+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T16:01:07.206303+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7c1791b6-9c0e-4512-a327-cad166bb5581","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6803","nonce":1894681701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6805","nonce":1894681701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6809","nonce":1894681701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6807","nonce":1894681701}]},"public_addr":"172.21.15.67:6803/1894681701","cluster_addr":"172.21.15.67:6805/1894681701","heartbeat_back_addr":"172.21.15.67:6809/1894681701","heartbeat_front_addr":"172.21.15.67:6807/1894681701","state":["exists","up"]},{"osd":1,"uuid":"0cdda7f3-d00c-4436-bcc1-49fb8072b2e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6801","nonce":1734465762}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6803","nonce":1734465762}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6807","nonce":1734465762}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6805","nonce":1734465762}]},"public_addr":"172.21.15.150:6801/1734465762","cluster_addr":"172.21.15.150:6803/1734465762","heartbeat_back_addr":"172.21.15.150:6807/1734465762","heartbeat_front_addr":"172.21.15.150:6805/1734465762","state":["exists","up"]},{"osd":2,"uuid":"a55b9a02-c3fd-4061-95c0-53a24a17ade0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6801","nonce":3398095931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6803","nonce":3398095931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6807","nonce":3398095931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6805","nonce":3398095931}]},"public_addr":"172.21.15.183:6801/3398095931","cluster_addr":"172.21.15.183:6803/3398095931","heartbeat_back_addr":"172.21.15.183:6807/3398095931","heartbeat_front_addr":"172.21.15.183:6805/3398095931","state":["exists","up"]},{"osd":3,"uuid":"d34940c0-5ce9-4b8b-aa36-4872414a043f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6811","nonce":1933541875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6813","nonce":1933541875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6817","nonce":1933541875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6815","nonce":1933541875}]},"public_addr":"172.21.15.67:6811/1933541875","cluster_addr":"172.21.15.67:6813/1933541875","heartbeat_back_addr":"172.21.15.67:6817/1933541875","heartbeat_front_addr":"172.21.15.67:6815/1933541875","state":["exists","up"]},{"osd":4,"uuid":"5cc3e260-83a3-4a4d-aa6a-34a3b12b2b5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6809","nonce":3775269993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6811","nonce":3775269993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6815","nonce":3775269993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6813","nonce":3775269993}]},"public_addr":"172.21.15.150:6809/3775269993","cluster_addr":"172.21.15.150:6811/3775269993","heartbeat_back_addr":"172.21.15.150:6815/3775269993","heartbeat_front_addr":"172.21.15.150:6813/3775269993","state":["exists","up"]},{"osd":5,"uuid":"a2542b21-d6ac-413e-8b9b-4a6548b20840","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6809","nonce":164284137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6811","nonce":164284137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6815","nonce":164284137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6813","nonce":164284137}]},"public_addr":"172.21.15.183:6809/164284137","cluster_addr":"172.21.15.183:6811/164284137","heartbeat_back_addr":"172.21.15.183:6815/164284137","heartbeat_front_addr":"172.21.15.183:6813/164284137","state":["exists","up"]},{"osd":6,"uuid":"8905a2d3-0f1b-4853-a218-22da2635150a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6819","nonce":3565528013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6821","nonce":3565528013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6825","nonce":3565528013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6823","nonce":3565528013}]},"public_addr":"172.21.15.67:6819/3565528013","cluster_addr":"172.21.15.67:6821/3565528013","heartbeat_back_addr":"172.21.15.67:6825/3565528013","heartbeat_front_addr":"172.21.15.67:6823/3565528013","state":["exists","up"]},{"osd":7,"uuid":"b275055b-b451-4180-91b1-e6e61a828c48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6817","nonce":3735115340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6819","nonce":3735115340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6823","nonce":3735115340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6821","nonce":3735115340}]},"public_addr":"172.21.15.150:6817/3735115340","cluster_addr":"172.21.15.150:6819/3735115340","heartbeat_back_addr":"172.21.15.150:6823/3735115340","heartbeat_front_addr":"172.21.15.150:6821/3735115340","state":["exists","up"]},{"osd":8,"uuid":"b6abf7ad-eb34-4f60-a4eb-348a92d7195b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6817","nonce":1025400095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6819","nonce":1025400095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6823","nonce":1025400095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6821","nonce":1025400095}]},"public_addr":"172.21.15.183:6817/1025400095","cluster_addr":"172.21.15.183:6819/1025400095","heartbeat_back_addr":"172.21.15.183:6823/1025400095","heartbeat_front_addr":"172.21.15.183:6821/1025400095","state":["exists","up"]},{"osd":9,"uuid":"28a78e3a-0657-40aa-816f-16a6fadadcbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6827","nonce":76885409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6829","nonce":76885409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6833","nonce":76885409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6831","nonce":76885409}]},"public_addr":"172.21.15.67:6827/76885409","cluster_addr":"172.21.15.67:6829/76885409","heartbeat_back_addr":"172.21.15.67:6833/76885409","heartbeat_front_addr":"172.21.15.67:6831/76885409","state":["exists","up"]},{"osd":10,"uuid":"d287907e-424f-4b49-a06e-4613c6de0474","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6825","nonce":3489982080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6827","nonce":3489982080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6831","nonce":3489982080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6829","nonce":3489982080}]},"public_addr":"172.21.15.150:6825/3489982080","cluster_addr":"172.21.15.150:6827/3489982080","heartbeat_back_addr":"172.21.15.150:6831/3489982080","heartbeat_front_addr":"172.21.15.150:6829/3489982080","state":["exists","up"]},{"osd":11,"uuid":"a3c29c47-a228-4f42-a23c-fbdb0c093699","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":85,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6825","nonce":1892639849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6826","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6827","nonce":1892639849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6831","nonce":1892639849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6829","nonce":1892639849}]},"public_addr":"172.21.15.183:6825/1892639849","cluster_addr":"172.21.15.183:6827/1892639849","heartbeat_back_addr":"172.21.15.183:6831/1892639849","heartbeat_front_addr":"172.21.15.183:6829/1892639849","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:53:45.646211+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:54:18.956590+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:54:52.161646+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:55:27.502822+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:56:01.162932+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:56:34.674452+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:57:09.298679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:57:43.230406+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:58:17.003821+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:58:51.085863+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:59:25.665065+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T16:00:00.263457+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6801/1631001968":"2024-03-29T15:51:38.875334+0000","172.21.15.67:6800/1631001968":"2024-03-29T15:51:38.875334+0000","172.21.15.67:0/1533056310":"2024-03-29T15:51:38.875334+0000","172.21.15.67:6801/3580491071":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/2030412458":"2024-03-29T15:51:03.859740+0000","172.21.15.67:6800/3580491071":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/2905307533":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/3763759988":"2024-03-29T15:51:38.875334+0000","172.21.15.67:0/4089399880":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/4239108661":"2024-03-29T15:51:38.875334+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T16:01:09.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:09.181+0000 7f24f0a00640 1 -- 172.21.15.67:0/59309820 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f24d00a2800 msgr2=0x7f24d00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:09.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:09.181+0000 7f24f0a00640 1 --2- 172.21.15.67:0/59309820 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f24d00a2800 0x7f24d00a4cc0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f24dc0058c0 tx=0x7f24dc005850 comp rx=0 tx=0).stop 2024-03-28T16:01:09.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:09.181+0000 7f24f0a00640 1 -- 172.21.15.67:0/59309820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24ec1028c0 msgr2=0x7f24ec073a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:09.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:09.181+0000 7f24f0a00640 1 --2- 172.21.15.67:0/59309820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24ec1028c0 0x7f24ec073a90 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f24d800d770 tx=0x7f24d800dc30 comp rx=0 tx=0).stop 2024-03-28T16:01:09.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:09.181+0000 7f24f0a00640 1 -- 172.21.15.67:0/59309820 shutdown_connections 2024-03-28T16:01:09.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:09.181+0000 7f24f0a00640 1 --2- 172.21.15.67:0/59309820 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f24d00a2800 0x7f24d00a4cc0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:09.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:09.181+0000 7f24f0a00640 1 --2- 172.21.15.67:0/59309820 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24ec103280 0x7f24ec073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:09.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:09.181+0000 7f24f0a00640 1 --2- 172.21.15.67:0/59309820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24ec1028c0 0x7f24ec073a90 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:09.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:09.181+0000 7f24f0a00640 1 --2- 172.21.15.67:0/59309820 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f24ec1016d0 0x7f24ec075450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:09.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:09.181+0000 7f24f0a00640 1 -- 172.21.15.67:0/59309820 >> 172.21.15.67:0/59309820 conn(0x7f24ec0fcc40 msgr2=0x7f24ec105bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:09.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:09.181+0000 7f24f0a00640 1 -- 172.21.15.67:0/59309820 shutdown_connections 2024-03-28T16:01:09.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:09.181+0000 7f24f0a00640 1 -- 172.21.15.67:0/59309820 wait complete. 2024-03-28T16:01:09.204 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-28T16:01:09.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:09.428+0000 7f75da000640 1 -- 172.21.15.67:0/2200120125 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75d4104a50 msgr2=0x7f75d4106e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:09.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:09.428+0000 7f75da000640 1 --2- 172.21.15.67:0/2200120125 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75d4104a50 0x7f75d4106e40 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f75c0009860 tx=0x7f75c002bb70 comp rx=0 tx=0).stop 2024-03-28T16:01:09.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:09.428+0000 7f75da000640 1 -- 172.21.15.67:0/2200120125 shutdown_connections 2024-03-28T16:01:09.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:09.428+0000 7f75da000640 1 --2- 172.21.15.67:0/2200120125 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f75d4107380 0x7f75d41097d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:09.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:09.428+0000 7f75da000640 1 --2- 172.21.15.67:0/2200120125 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75d4104a50 0x7f75d4106e40 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:09.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:09.428+0000 7f75da000640 1 --2- 172.21.15.67:0/2200120125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75d4102120 0x7f75d4104510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:09.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:09.428+0000 7f75da000640 1 -- 172.21.15.67:0/2200120125 >> 172.21.15.67:0/2200120125 conn(0x7f75d40fbb30 msgr2=0x7f75d40fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:09.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:09.428+0000 7f75da000640 1 -- 172.21.15.67:0/2200120125 shutdown_connections 2024-03-28T16:01:09.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:09.428+0000 7f75da000640 1 -- 172.21.15.67:0/2200120125 wait complete. 2024-03-28T16:01:09.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:09.429+0000 7f75da000640 1 Processor -- start 2024-03-28T16:01:09.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:09.429+0000 7f75da000640 1 -- start start 2024-03-28T16:01:09.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:09.430+0000 7f75da000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f75d4102120 0x7f75d4197940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:09.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:09.430+0000 7f75da000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75d4104a50 0x7f75d4197e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:09.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:09.430+0000 7f75da000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75d4107380 0x7f75d419eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:09.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:09.430+0000 7f75da000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75d410d1c0 con 0x7f75d4104a50 2024-03-28T16:01:09.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:09.430+0000 7f75da000640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75d410d040 con 0x7f75d4107380 2024-03-28T16:01:09.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:09.430+0000 7f75da000640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75d410d340 con 0x7f75d4102120 2024-03-28T16:01:09.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:09.430+0000 7f75d2a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75d4104a50 0x7f75d4197e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:09.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:09.430+0000 7f75d2a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75d4104a50 0x7f75d4197e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41160/0 (socket says 172.21.15.67:41160) 2024-03-28T16:01:09.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:09.430+0000 7f75d2a00640 1 -- 172.21.15.67:0/3021919954 learned_addr learned my addr 172.21.15.67:0/3021919954 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:01:09.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:09.430+0000 7f75d3e00640 1 --2- 172.21.15.67:0/3021919954 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75d4107380 0x7f75d419eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:09.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:09.430+0000 7f75d3400640 1 --2- 172.21.15.67:0/3021919954 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f75d4102120 0x7f75d4197940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:09.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:09.431+0000 7f75d2a00640 1 -- 172.21.15.67:0/3021919954 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f75d4102120 msgr2=0x7f75d4197940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:09.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:09.431+0000 7f75d2a00640 1 --2- 172.21.15.67:0/3021919954 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f75d4102120 0x7f75d4197940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:09.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:09.431+0000 7f75d2a00640 1 -- 172.21.15.67:0/3021919954 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75d4107380 msgr2=0x7f75d419eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:09.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:09.431+0000 7f75d2a00640 1 --2- 172.21.15.67:0/3021919954 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75d4107380 0x7f75d419eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:09.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:09.431+0000 7f75d2a00640 1 -- 172.21.15.67:0/3021919954 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75d419f560 con 0x7f75d4104a50 2024-03-28T16:01:09.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:09.431+0000 7f75d2a00640 1 --2- 172.21.15.67:0/3021919954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75d4104a50 0x7f75d4197e80 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f75c00057c0 tx=0x7f75c00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:09.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:09.431+0000 7f75cbe00640 1 -- 172.21.15.67:0/3021919954 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75c0041030 con 0x7f75d4104a50 2024-03-28T16:01:09.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:09.431+0000 7f75da000640 1 -- 172.21.15.67:0/3021919954 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75d419f7f0 con 0x7f75d4104a50 2024-03-28T16:01:09.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:09.431+0000 7f75cbe00640 1 -- 172.21.15.67:0/3021919954 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f75c002c9c0 con 0x7f75d4104a50 2024-03-28T16:01:09.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:09.431+0000 7f75cbe00640 1 -- 172.21.15.67:0/3021919954 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75c0039040 con 0x7f75d4104a50 2024-03-28T16:01:09.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:09.432+0000 7f75da000640 1 -- 172.21.15.67:0/3021919954 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75d419fd30 con 0x7f75d4104a50 2024-03-28T16:01:09.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:09.433+0000 7f75da000640 1 -- 172.21.15.67:0/3021919954 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75d40fd620 con 0x7f75d4104a50 2024-03-28T16:01:09.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:09.435+0000 7f75cbe00640 1 -- 172.21.15.67:0/3021919954 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f75c0044070 con 0x7f75d4104a50 2024-03-28T16:01:09.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:09.437+0000 7f75cbe00640 1 --2- 172.21.15.67:0/3021919954 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f75b80a2880 0x7f75b80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:09.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:09.437+0000 7f75cbe00640 1 -- 172.21.15.67:0/3021919954 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f75c00efff0 con 0x7f75d4104a50 2024-03-28T16:01:09.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:09.437+0000 7f75d3400640 1 --2- 172.21.15.67:0/3021919954 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f75b80a2880 0x7f75b80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:09.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:09.438+0000 7f75d3400640 1 --2- 172.21.15.67:0/3021919954 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f75b80a2880 0x7f75b80a4d40 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f75c4005fd0 tx=0x7f75c4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:09.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:09.441+0000 7f75cbe00640 1 -- 172.21.15.67:0/3021919954 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f75c00bd180 con 0x7f75d4104a50 2024-03-28T16:01:09.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:09 smithi150 ceph-mon[34825]: osdmap e89: 12 total, 12 up, 12 in 2024-03-28T16:01:09.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:09 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/2121311058' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-28T16:01:09.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:09 smithi150 ceph-mon[34825]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-28T16:01:09.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:09 smithi150 ceph-mon[34825]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-28T16:01:09.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:09 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/533781036' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:01:09.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:09 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/59309820' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:01:09.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:09 smithi067 ceph-mon[31707]: osdmap e89: 12 total, 12 up, 12 in 2024-03-28T16:01:09.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:09 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/2121311058' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-28T16:01:09.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:09 smithi067 ceph-mon[31707]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-28T16:01:09.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:09 smithi067 ceph-mon[31707]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-28T16:01:09.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:09 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/533781036' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:01:09.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:09 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/59309820' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:01:09.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:09 smithi183 ceph-mon[35580]: osdmap e89: 12 total, 12 up, 12 in 2024-03-28T16:01:09.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:09 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/2121311058' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-28T16:01:09.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:09 smithi183 ceph-mon[35580]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-28T16:01:09.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:09 smithi183 ceph-mon[35580]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-28T16:01:09.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:09 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/533781036' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:01:09.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:09 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/59309820' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:01:09.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:09.707+0000 7f75da000640 1 -- 172.21.15.67:0/3021919954 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f75d40622a0 con 0x7f75d4104a50 2024-03-28T16:01:10.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:10.272+0000 7f75cbe00640 1 -- 172.21.15.67:0/3021919954 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v16) v1 ==== 118+0+0 (secure 0 0 0) 0x7f75c00bca80 con 0x7f75d4104a50 2024-03-28T16:01:10.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:10.275+0000 7f75da000640 1 -- 172.21.15.67:0/3021919954 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f75b80a2880 msgr2=0x7f75b80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:10.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:10.275+0000 7f75da000640 1 --2- 172.21.15.67:0/3021919954 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f75b80a2880 0x7f75b80a4d40 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f75c4005fd0 tx=0x7f75c4009260 comp rx=0 tx=0).stop 2024-03-28T16:01:10.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:10.275+0000 7f75da000640 1 -- 172.21.15.67:0/3021919954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75d4104a50 msgr2=0x7f75d4197e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:10.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:10.275+0000 7f75da000640 1 --2- 172.21.15.67:0/3021919954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75d4104a50 0x7f75d4197e80 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f75c00057c0 tx=0x7f75c00042f0 comp rx=0 tx=0).stop 2024-03-28T16:01:10.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:10.275+0000 7f75da000640 1 -- 172.21.15.67:0/3021919954 shutdown_connections 2024-03-28T16:01:10.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:10.275+0000 7f75da000640 1 --2- 172.21.15.67:0/3021919954 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f75b80a2880 0x7f75b80a4d40 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:10.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:10.275+0000 7f75da000640 1 --2- 172.21.15.67:0/3021919954 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75d4107380 0x7f75d419eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:10.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:10.275+0000 7f75da000640 1 --2- 172.21.15.67:0/3021919954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75d4104a50 0x7f75d4197e80 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:10.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:10.275+0000 7f75da000640 1 --2- 172.21.15.67:0/3021919954 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f75d4102120 0x7f75d4197940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:10.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:10.275+0000 7f75da000640 1 -- 172.21.15.67:0/3021919954 >> 172.21.15.67:0/3021919954 conn(0x7f75d40fbb30 msgr2=0x7f75d4103fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:10.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:10.275+0000 7f75da000640 1 -- 172.21.15.67:0/3021919954 shutdown_connections 2024-03-28T16:01:10.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:10.275+0000 7f75da000640 1 -- 172.21.15.67:0/3021919954 wait complete. 2024-03-28T16:01:10.301 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 1, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-03-28T16:01:10.301 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-03-28T16:01:10.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:10.515+0000 7f7648400640 1 -- 172.21.15.67:0/3182667096 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7640103260 msgr2=0x7f7640109af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:10.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:10.515+0000 7f7648400640 1 --2- 172.21.15.67:0/3182667096 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7640103260 0x7f7640109af0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f7638009ca0 tx=0x7f763802c0f0 comp rx=0 tx=0).stop 2024-03-28T16:01:10.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:10.516+0000 7f7648400640 1 -- 172.21.15.67:0/3182667096 shutdown_connections 2024-03-28T16:01:10.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:10.516+0000 7f7648400640 1 --2- 172.21.15.67:0/3182667096 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7640103260 0x7f7640109af0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:10.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:10.516+0000 7f7648400640 1 --2- 172.21.15.67:0/3182667096 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76401028a0 0x7f7640102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:10.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:10.516+0000 7f7648400640 1 --2- 172.21.15.67:0/3182667096 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f76401016b0 0x7f7640101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:10.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:10.516+0000 7f7648400640 1 -- 172.21.15.67:0/3182667096 >> 172.21.15.67:0/3182667096 conn(0x7f76400fcc40 msgr2=0x7f76400ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:10.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:10.516+0000 7f7648400640 1 -- 172.21.15.67:0/3182667096 shutdown_connections 2024-03-28T16:01:10.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:10.516+0000 7f7648400640 1 -- 172.21.15.67:0/3182667096 wait complete. 2024-03-28T16:01:10.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:10.517+0000 7f7648400640 1 Processor -- start 2024-03-28T16:01:10.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:10.517+0000 7f7648400640 1 -- start start 2024-03-28T16:01:10.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:10.517+0000 7f7648400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f76401016b0 0x7f7640199aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:10.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:10.517+0000 7f7648400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76401028a0 0x7f7640199fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:10.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:10.517+0000 7f7648400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7640103260 0x7f76401a1050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:10.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:10.518+0000 7f7648400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f764010d1e0 con 0x7f76401028a0 2024-03-28T16:01:10.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:10.518+0000 7f7648400640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f764010d060 con 0x7f7640103260 2024-03-28T16:01:10.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:10.518+0000 7f7648400640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f764010d360 con 0x7f76401016b0 2024-03-28T16:01:10.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:10.518+0000 7f7645000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76401028a0 0x7f7640199fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:10.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:10.518+0000 7f7645000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76401028a0 0x7f7640199fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41170/0 (socket says 172.21.15.67:41170) 2024-03-28T16:01:10.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:10.518+0000 7f7645000640 1 -- 172.21.15.67:0/527840272 learned_addr learned my addr 172.21.15.67:0/527840272 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:01:10.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:10.518+0000 7f7645a00640 1 --2- 172.21.15.67:0/527840272 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f76401016b0 0x7f7640199aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:10.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:10.518+0000 7f7646400640 1 --2- 172.21.15.67:0/527840272 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7640103260 0x7f76401a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:10.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:10.518+0000 7f7645000640 1 -- 172.21.15.67:0/527840272 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f76401016b0 msgr2=0x7f7640199aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:10.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:10.518+0000 7f7645000640 1 --2- 172.21.15.67:0/527840272 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f76401016b0 0x7f7640199aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:10.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:10.518+0000 7f7645000640 1 -- 172.21.15.67:0/527840272 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7640103260 msgr2=0x7f76401a1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:10.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:10.518+0000 7f7645000640 1 --2- 172.21.15.67:0/527840272 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7640103260 0x7f76401a1050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:10.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:10.518+0000 7f7645000640 1 -- 172.21.15.67:0/527840272 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76401a16c0 con 0x7f76401028a0 2024-03-28T16:01:10.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:10.519+0000 7f7645000640 1 --2- 172.21.15.67:0/527840272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76401028a0 0x7f7640199fe0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f762c00d950 tx=0x7f762c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:10.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:10.519+0000 7f763e000640 1 -- 172.21.15.67:0/527840272 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f762c017070 con 0x7f76401028a0 2024-03-28T16:01:10.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:10.519+0000 7f763e000640 1 -- 172.21.15.67:0/527840272 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f762c009c30 con 0x7f76401028a0 2024-03-28T16:01:10.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:10.519+0000 7f7648400640 1 -- 172.21.15.67:0/527840272 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76401a19b0 con 0x7f76401028a0 2024-03-28T16:01:10.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:10.519+0000 7f763e000640 1 -- 172.21.15.67:0/527840272 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f762c00f620 con 0x7f76401028a0 2024-03-28T16:01:10.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:10.519+0000 7f7648400640 1 -- 172.21.15.67:0/527840272 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76401a1f50 con 0x7f76401028a0 2024-03-28T16:01:10.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:10.521+0000 7f7648400640 1 -- 172.21.15.67:0/527840272 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76401a2180 con 0x7f76401028a0 2024-03-28T16:01:10.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:10.522+0000 7f763e000640 1 -- 172.21.15.67:0/527840272 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f762c0109a0 con 0x7f76401028a0 2024-03-28T16:01:10.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:10.525+0000 7f763e000640 1 --2- 172.21.15.67:0/527840272 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f76280a2880 0x7f76280a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:10.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:10.525+0000 7f763e000640 1 -- 172.21.15.67:0/527840272 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f762c0ce4c0 con 0x7f76401028a0 2024-03-28T16:01:10.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:10.525+0000 7f7645a00640 1 --2- 172.21.15.67:0/527840272 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f76280a2880 0x7f76280a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:10.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:10.526+0000 7f7645a00640 1 --2- 172.21.15.67:0/527840272 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f76280a2880 0x7f76280a4d40 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f7634005fd0 tx=0x7f763400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:10.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:10.529+0000 7f763e000640 1 -- 172.21.15.67:0/527840272 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f762c009de0 con 0x7f76401028a0 2024-03-28T16:01:10.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:10 smithi150 ceph-mon[34825]: pgmap v364: 97 pgs: 21 creating+peering, 17 active+clean, 59 unknown; 577 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-03-28T16:01:10.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:10 smithi150 ceph-mon[34825]: osdmap e90: 12 total, 12 up, 12 in 2024-03-28T16:01:10.553 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:10 smithi150 ceph-mon[34825]: mds.? [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] up:active 2024-03-28T16:01:10.553 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:10 smithi150 ceph-mon[34825]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T16:01:10.553 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:10 smithi150 ceph-mon[34825]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-28T16:01:10.553 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:10 smithi150 ceph-mon[34825]: Cluster is now healthy 2024-03-28T16:01:10.553 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:10 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/3021919954' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-28T16:01:10.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:10 smithi067 ceph-mon[31707]: pgmap v364: 97 pgs: 21 creating+peering, 17 active+clean, 59 unknown; 577 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-03-28T16:01:10.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:10 smithi067 ceph-mon[31707]: osdmap e90: 12 total, 12 up, 12 in 2024-03-28T16:01:10.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:10 smithi067 ceph-mon[31707]: mds.? [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] up:active 2024-03-28T16:01:10.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:10 smithi067 ceph-mon[31707]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T16:01:10.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:10 smithi067 ceph-mon[31707]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-28T16:01:10.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:10 smithi067 ceph-mon[31707]: Cluster is now healthy 2024-03-28T16:01:10.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:10 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/3021919954' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-28T16:01:10.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:10 smithi183 ceph-mon[35580]: pgmap v364: 97 pgs: 21 creating+peering, 17 active+clean, 59 unknown; 577 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-03-28T16:01:10.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:10 smithi183 ceph-mon[35580]: osdmap e90: 12 total, 12 up, 12 in 2024-03-28T16:01:10.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:10 smithi183 ceph-mon[35580]: mds.? [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] up:active 2024-03-28T16:01:10.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:10 smithi183 ceph-mon[35580]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T16:01:10.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:10 smithi183 ceph-mon[35580]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-28T16:01:10.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:10 smithi183 ceph-mon[35580]: Cluster is now healthy 2024-03-28T16:01:10.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:10 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/3021919954' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-28T16:01:10.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:10.788+0000 7f7648400640 1 -- 172.21.15.67:0/527840272 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7f76401a2350 con 0x7f76401028a0 2024-03-28T16:01:11.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:11.484+0000 7f763e000640 1 -- 172.21.15.67:0/527840272 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v17) v1 ==== 121+0+0 (secure 0 0 0) 0x7f762c09afc0 con 0x7f76401028a0 2024-03-28T16:01:11.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:11.488+0000 7f7648400640 1 -- 172.21.15.67:0/527840272 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f76280a2880 msgr2=0x7f76280a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:11.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:11.488+0000 7f7648400640 1 --2- 172.21.15.67:0/527840272 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f76280a2880 0x7f76280a4d40 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f7634005fd0 tx=0x7f763400b390 comp rx=0 tx=0).stop 2024-03-28T16:01:11.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:11.488+0000 7f7648400640 1 -- 172.21.15.67:0/527840272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76401028a0 msgr2=0x7f7640199fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:11.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:11.488+0000 7f7648400640 1 --2- 172.21.15.67:0/527840272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76401028a0 0x7f7640199fe0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f762c00d950 tx=0x7f762c00de10 comp rx=0 tx=0).stop 2024-03-28T16:01:11.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:11.488+0000 7f7648400640 1 -- 172.21.15.67:0/527840272 shutdown_connections 2024-03-28T16:01:11.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:11.488+0000 7f7648400640 1 --2- 172.21.15.67:0/527840272 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f76280a2880 0x7f76280a4d40 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:11.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:11.488+0000 7f7648400640 1 --2- 172.21.15.67:0/527840272 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7640103260 0x7f76401a1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:11.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:11.488+0000 7f7648400640 1 --2- 172.21.15.67:0/527840272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76401028a0 0x7f7640199fe0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:11.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:11.488+0000 7f7648400640 1 --2- 172.21.15.67:0/527840272 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f76401016b0 0x7f7640199aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:11.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:11.488+0000 7f7648400640 1 -- 172.21.15.67:0/527840272 >> 172.21.15.67:0/527840272 conn(0x7f76400fcc40 msgr2=0x7f7640108070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:11.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:11.489+0000 7f7648400640 1 -- 172.21.15.67:0/527840272 shutdown_connections 2024-03-28T16:01:11.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:11.489+0000 7f7648400640 1 -- 172.21.15.67:0/527840272 wait complete. 2024-03-28T16:01:11.512 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-03-28T16:01:11.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:11 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/3021919954' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-28T16:01:11.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:11 smithi150 ceph-mon[34825]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T16:01:11.553 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:11 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:01:11.553 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:11 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:01:11.553 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:11 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:01:11.553 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:11 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:01:11.553 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:11 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:01:11.553 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:11 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:01:11.553 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:11 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:01:11.553 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:11 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T16:01:11.553 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:11 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:01:11.553 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:11 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/527840272' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-28T16:01:11.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:11 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/3021919954' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-28T16:01:11.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:11 smithi067 ceph-mon[31707]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T16:01:11.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:11 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:01:11.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:11 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:01:11.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:11 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:01:11.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:11 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:01:11.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:11 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:01:11.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:11 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:01:11.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:11 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:01:11.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:11 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T16:01:11.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:11 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:01:11.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:11 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/527840272' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-28T16:01:11.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:11 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/3021919954' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-28T16:01:11.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:11 smithi183 ceph-mon[35580]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T16:01:11.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:11 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:01:11.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:11 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:01:11.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:11 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:01:11.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:11 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:01:11.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:11 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:01:11.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:11 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:01:11.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:11 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:01:11.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:11 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T16:01:11.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:11 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:01:11.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:11 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/527840272' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-28T16:01:11.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:11.737+0000 7fafde200640 1 -- 172.21.15.67:0/3311618053 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fafd810db70 msgr2=0x7fafd810ff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:11.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:11.737+0000 7fafde200640 1 --2- 172.21.15.67:0/3311618053 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fafd810db70 0x7fafd810ff60 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fafd0009cd0 tx=0x7fafd002c0f0 comp rx=0 tx=0).stop 2024-03-28T16:01:11.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:11.737+0000 7fafde200640 1 -- 172.21.15.67:0/3311618053 shutdown_connections 2024-03-28T16:01:11.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:11.737+0000 7fafde200640 1 --2- 172.21.15.67:0/3311618053 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fafd810db70 0x7fafd810ff60 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:11.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:11.737+0000 7fafde200640 1 --2- 172.21.15.67:0/3311618053 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fafd8074100 0x7fafd8074580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:11.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:11.737+0000 7fafde200640 1 --2- 172.21.15.67:0/3311618053 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fafd80756e0 0x7fafd8073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:11.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:11.737+0000 7fafde200640 1 -- 172.21.15.67:0/3311618053 >> 172.21.15.67:0/3311618053 conn(0x7fafd80fcb10 msgr2=0x7fafd80fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:11.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:11.737+0000 7fafde200640 1 -- 172.21.15.67:0/3311618053 shutdown_connections 2024-03-28T16:01:11.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:11.737+0000 7fafde200640 1 -- 172.21.15.67:0/3311618053 wait complete. 2024-03-28T16:01:11.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:11.738+0000 7fafde200640 1 Processor -- start 2024-03-28T16:01:11.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:11.738+0000 7fafde200640 1 -- start start 2024-03-28T16:01:11.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:11.739+0000 7fafde200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fafd8074100 0x7fafd819dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:11.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:11.739+0000 7fafde200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fafd80756e0 0x7fafd819e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:11.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:11.739+0000 7fafde200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fafd810db70 0x7fafd81a5320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:11.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:11.739+0000 7fafde200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafd81136a0 con 0x7fafd80756e0 2024-03-28T16:01:11.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:11.739+0000 7fafde200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafd8113520 con 0x7fafd810db70 2024-03-28T16:01:11.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:11.739+0000 7fafde200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafd8113820 con 0x7fafd8074100 2024-03-28T16:01:11.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:11.739+0000 7fafcfe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fafd80756e0 0x7fafd819e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:11.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:11.739+0000 7fafcfe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fafd80756e0 0x7fafd819e2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41194/0 (socket says 172.21.15.67:41194) 2024-03-28T16:01:11.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:11.739+0000 7fafcfe00640 1 -- 172.21.15.67:0/2985693106 learned_addr learned my addr 172.21.15.67:0/2985693106 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:01:11.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:11.739+0000 7fafdd800640 1 --2- 172.21.15.67:0/2985693106 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fafd810db70 0x7fafd81a5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:11.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:11.739+0000 7fafdce00640 1 --2- 172.21.15.67:0/2985693106 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fafd8074100 0x7fafd819dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:11.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:11.740+0000 7fafcfe00640 1 -- 172.21.15.67:0/2985693106 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fafd8074100 msgr2=0x7fafd819dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:11.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:11.740+0000 7fafcfe00640 1 --2- 172.21.15.67:0/2985693106 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fafd8074100 0x7fafd819dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:11.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:11.740+0000 7fafcfe00640 1 -- 172.21.15.67:0/2985693106 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fafd810db70 msgr2=0x7fafd81a5320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:11.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:11.740+0000 7fafcfe00640 1 --2- 172.21.15.67:0/2985693106 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fafd810db70 0x7fafd81a5320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:11.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:11.740+0000 7fafcfe00640 1 -- 172.21.15.67:0/2985693106 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafd81a5990 con 0x7fafd80756e0 2024-03-28T16:01:11.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:11.740+0000 7fafdd800640 1 --2- 172.21.15.67:0/2985693106 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fafd810db70 0x7fafd81a5320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T16:01:11.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:11.740+0000 7fafcfe00640 1 --2- 172.21.15.67:0/2985693106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fafd80756e0 0x7fafd819e2b0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fafc800e9f0 tx=0x7fafc800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:11.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:11.740+0000 7fafd6000640 1 -- 172.21.15.67:0/2985693106 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc800cd80 con 0x7fafd80756e0 2024-03-28T16:01:11.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:11.740+0000 7fafd6000640 1 -- 172.21.15.67:0/2985693106 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fafc80108e0 con 0x7fafd80756e0 2024-03-28T16:01:11.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:11.740+0000 7fafd6000640 1 -- 172.21.15.67:0/2985693106 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc8018920 con 0x7fafd80756e0 2024-03-28T16:01:11.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:11.740+0000 7fafde200640 1 -- 172.21.15.67:0/2985693106 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafd81a5c80 con 0x7fafd80756e0 2024-03-28T16:01:11.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:11.741+0000 7fafde200640 1 -- 172.21.15.67:0/2985693106 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafd81a6180 con 0x7fafd80756e0 2024-03-28T16:01:11.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:11.742+0000 7fafde200640 1 -- 172.21.15.67:0/2985693106 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafd8103d40 con 0x7fafd80756e0 2024-03-28T16:01:11.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:11.744+0000 7fafd6000640 1 -- 172.21.15.67:0/2985693106 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fafc8010a90 con 0x7fafd80756e0 2024-03-28T16:01:11.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:11.746+0000 7fafd6000640 1 --2- 172.21.15.67:0/2985693106 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fafb80a2800 0x7fafb80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:11.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:11.746+0000 7fafd6000640 1 -- 172.21.15.67:0/2985693106 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fafc80cee00 con 0x7fafd80756e0 2024-03-28T16:01:11.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:11.746+0000 7fafdce00640 1 --2- 172.21.15.67:0/2985693106 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fafb80a2800 0x7fafb80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:11.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:11.747+0000 7fafdce00640 1 --2- 172.21.15.67:0/2985693106 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fafb80a2800 0x7fafb80a4cc0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fafc0005fd0 tx=0x7fafc0005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:11.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:11.750+0000 7fafd6000640 1 -- 172.21.15.67:0/2985693106 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fafc809bf50 con 0x7fafd80756e0 2024-03-28T16:01:11.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:11.995+0000 7fafde200640 1 -- 172.21.15.67:0/2985693106 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7fafd80622d0 con 0x7fafd80756e0 2024-03-28T16:01:12.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:12.499+0000 7fafd6000640 1 -- 172.21.15.67:0/2985693106 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v19) v1 ==== 115+0+0 (secure 0 0 0) 0x7fafc809b850 con 0x7fafd80756e0 2024-03-28T16:01:12.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:12.503+0000 7fafde200640 1 -- 172.21.15.67:0/2985693106 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fafb80a2800 msgr2=0x7fafb80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:12.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:12.503+0000 7fafde200640 1 --2- 172.21.15.67:0/2985693106 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fafb80a2800 0x7fafb80a4cc0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fafc0005fd0 tx=0x7fafc0005850 comp rx=0 tx=0).stop 2024-03-28T16:01:12.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:12.503+0000 7fafde200640 1 -- 172.21.15.67:0/2985693106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fafd80756e0 msgr2=0x7fafd819e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:12.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:12.503+0000 7fafde200640 1 --2- 172.21.15.67:0/2985693106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fafd80756e0 0x7fafd819e2b0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fafc800e9f0 tx=0x7fafc800eeb0 comp rx=0 tx=0).stop 2024-03-28T16:01:12.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:12.503+0000 7fafde200640 1 -- 172.21.15.67:0/2985693106 shutdown_connections 2024-03-28T16:01:12.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:12.503+0000 7fafde200640 1 --2- 172.21.15.67:0/2985693106 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fafb80a2800 0x7fafb80a4cc0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:12.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:12.503+0000 7fafde200640 1 --2- 172.21.15.67:0/2985693106 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fafd810db70 0x7fafd81a5320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:12.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:12.503+0000 7fafde200640 1 --2- 172.21.15.67:0/2985693106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fafd80756e0 0x7fafd819e2b0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:12.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:12.503+0000 7fafde200640 1 --2- 172.21.15.67:0/2985693106 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fafd8074100 0x7fafd819dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:12.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:12.503+0000 7fafde200640 1 -- 172.21.15.67:0/2985693106 >> 172.21.15.67:0/2985693106 conn(0x7fafd80fcb10 msgr2=0x7fafd8100f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:12.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:12.503+0000 7fafde200640 1 -- 172.21.15.67:0/2985693106 shutdown_connections 2024-03-28T16:01:12.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:12.503+0000 7fafde200640 1 -- 172.21.15.67:0/2985693106 wait complete. 2024-03-28T16:01:12.526 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--size 25000000000'} subvols for filesystem 'cephfs' 2024-03-28T16:01:12.527 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-03-28T16:01:12.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:12.744+0000 7fb272c00640 1 -- 172.21.15.67:0/3796936991 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb26c107ac0 msgr2=0x7fb26c107ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:12.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:12.744+0000 7fb272c00640 1 --2- 172.21.15.67:0/3796936991 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb26c107ac0 0x7fb26c107ee0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb260009900 tx=0x7fb26002bc30 comp rx=0 tx=0).stop 2024-03-28T16:01:12.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:12.744+0000 7fb272c00640 1 -- 172.21.15.67:0/3796936991 shutdown_connections 2024-03-28T16:01:12.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:12.744+0000 7fb272c00640 1 --2- 172.21.15.67:0/3796936991 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb26c109670 0x7fb26c10ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:12.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:12.744+0000 7fb272c00640 1 --2- 172.21.15.67:0/3796936991 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb26c108cb0 0x7fb26c109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:12.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:12.744+0000 7fb272c00640 1 --2- 172.21.15.67:0/3796936991 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb26c107ac0 0x7fb26c107ee0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:12.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:12.744+0000 7fb272c00640 1 -- 172.21.15.67:0/3796936991 >> 172.21.15.67:0/3796936991 conn(0x7fb26c076a60 msgr2=0x7fb26c078ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:12.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:12.744+0000 7fb272c00640 1 -- 172.21.15.67:0/3796936991 shutdown_connections 2024-03-28T16:01:12.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:12.744+0000 7fb272c00640 1 -- 172.21.15.67:0/3796936991 wait complete. 2024-03-28T16:01:12.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:12.745+0000 7fb272c00640 1 Processor -- start 2024-03-28T16:01:12.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:12.746+0000 7fb272c00640 1 -- start start 2024-03-28T16:01:12.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:12.746+0000 7fb272c00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb26c107ac0 0x7fb26c19dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:12.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:12.746+0000 7fb272c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb26c108cb0 0x7fb26c19e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:12.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:12.746+0000 7fb272c00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb26c109670 0x7fb26c1a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:12.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:12.746+0000 7fb272c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb26c113640 con 0x7fb26c108cb0 2024-03-28T16:01:12.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:12.746+0000 7fb272c00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb26c1134c0 con 0x7fb26c107ac0 2024-03-28T16:01:12.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:12.746+0000 7fb272c00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb26c1137c0 con 0x7fb26c109670 2024-03-28T16:01:12.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:12.746+0000 7fb270e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb26c108cb0 0x7fb26c19e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:12.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:12.746+0000 7fb270e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb26c108cb0 0x7fb26c19e1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41214/0 (socket says 172.21.15.67:41214) 2024-03-28T16:01:12.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:12.746+0000 7fb270e00640 1 -- 172.21.15.67:0/1774919981 learned_addr learned my addr 172.21.15.67:0/1774919981 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:01:12.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:12.747+0000 7fb271800640 1 --2- 172.21.15.67:0/1774919981 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb26c107ac0 0x7fb26c19dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:12.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:12.747+0000 7fb272200640 1 --2- 172.21.15.67:0/1774919981 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb26c109670 0x7fb26c1a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:12.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:12.747+0000 7fb270e00640 1 -- 172.21.15.67:0/1774919981 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb26c109670 msgr2=0x7fb26c1a5250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:12.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:12.747+0000 7fb270e00640 1 --2- 172.21.15.67:0/1774919981 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb26c109670 0x7fb26c1a5250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:12.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:12.747+0000 7fb270e00640 1 -- 172.21.15.67:0/1774919981 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb26c107ac0 msgr2=0x7fb26c19dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:12.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:12.747+0000 7fb270e00640 1 --2- 172.21.15.67:0/1774919981 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb26c107ac0 0x7fb26c19dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:12.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:12.747+0000 7fb270e00640 1 -- 172.21.15.67:0/1774919981 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb26c1a58f0 con 0x7fb26c108cb0 2024-03-28T16:01:12.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:12.747+0000 7fb271800640 1 --2- 172.21.15.67:0/1774919981 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb26c107ac0 0x7fb26c19dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T16:01:12.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:12.747+0000 7fb270e00640 1 --2- 172.21.15.67:0/1774919981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb26c108cb0 0x7fb26c19e1f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb25800e9f0 tx=0x7fb25800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:12.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:12.748+0000 7fb26a000640 1 -- 172.21.15.67:0/1774919981 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb25800cd80 con 0x7fb26c108cb0 2024-03-28T16:01:12.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:12.748+0000 7fb26a000640 1 -- 172.21.15.67:0/1774919981 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb2580108e0 con 0x7fb26c108cb0 2024-03-28T16:01:12.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:12.748+0000 7fb272c00640 1 -- 172.21.15.67:0/1774919981 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb26c1a5be0 con 0x7fb26c108cb0 2024-03-28T16:01:12.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:12.748+0000 7fb26a000640 1 -- 172.21.15.67:0/1774919981 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb258018920 con 0x7fb26c108cb0 2024-03-28T16:01:12.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:12.748+0000 7fb272c00640 1 -- 172.21.15.67:0/1774919981 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb26c1a6150 con 0x7fb26c108cb0 2024-03-28T16:01:12.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:12.749+0000 7fb272c00640 1 -- 172.21.15.67:0/1774919981 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb26c10ede0 con 0x7fb26c108cb0 2024-03-28T16:01:12.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:12.752+0000 7fb26a000640 1 -- 172.21.15.67:0/1774919981 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fb258010a90 con 0x7fb26c108cb0 2024-03-28T16:01:12.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:12.753+0000 7fb26a000640 1 --2- 172.21.15.67:0/1774919981 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb2540a2880 0x7fb2540a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:12.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:12.753+0000 7fb26a000640 1 -- 172.21.15.67:0/1774919981 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fb2580ced80 con 0x7fb26c108cb0 2024-03-28T16:01:12.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:12.753+0000 7fb271800640 1 --2- 172.21.15.67:0/1774919981 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb2540a2880 0x7fb2540a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:12.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:12.754+0000 7fb271800640 1 --2- 172.21.15.67:0/1774919981 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb2540a2880 0x7fb2540a4d40 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb26002c240 tx=0x7fb26002c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:12.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:12.757+0000 7fb26a000640 1 -- 172.21.15.67:0/1774919981 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb25809bed0 con 0x7fb26c108cb0 2024-03-28T16:01:12.802 INFO:journalctl@ceph.mds.b.smithi150.stdout:Mar 28 16:01:12 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-b[65534]: 2024-03-28T16:01:12.476+0000 7ff91a400700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T16:01:12.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:12 smithi150 ceph-mon[34825]: pgmap v366: 97 pgs: 21 creating+peering, 70 active+clean, 6 unknown; 578 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s wr, 6 op/s 2024-03-28T16:01:12.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:12 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/527840272' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-28T16:01:12.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:12 smithi150 ceph-mon[34825]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T16:01:12.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:12 smithi150 ceph-mon[34825]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-28T16:01:12.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:12 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/2985693106' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-28T16:01:12.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:12 smithi067 ceph-mon[31707]: pgmap v366: 97 pgs: 21 creating+peering, 70 active+clean, 6 unknown; 578 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s wr, 6 op/s 2024-03-28T16:01:12.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:12 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/527840272' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-28T16:01:12.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:12 smithi067 ceph-mon[31707]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T16:01:12.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:12 smithi067 ceph-mon[31707]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-28T16:01:12.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:12 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/2985693106' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-28T16:01:12.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:12 smithi183 ceph-mon[35580]: pgmap v366: 97 pgs: 21 creating+peering, 70 active+clean, 6 unknown; 578 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s wr, 6 op/s 2024-03-28T16:01:12.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:12 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/527840272' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-28T16:01:12.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:12 smithi183 ceph-mon[35580]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T16:01:12.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:12 smithi183 ceph-mon[35580]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-28T16:01:12.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:12 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/2985693106' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-28T16:01:12.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:12.988+0000 7fb272c00640 1 -- 172.21.15.67:0/1774919981 --> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7fb26c061ae0 con 0x7fb2540a2880 2024-03-28T16:01:13.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.036+0000 7fb26a000640 1 -- 172.21.15.67:0/1774919981 <== mgr.14152 v2:172.21.15.67:6800/208726654 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fb26c061ae0 con 0x7fb2540a2880 2024-03-28T16:01:13.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.040+0000 7fb272c00640 1 -- 172.21.15.67:0/1774919981 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb2540a2880 msgr2=0x7fb2540a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:13.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.040+0000 7fb272c00640 1 --2- 172.21.15.67:0/1774919981 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb2540a2880 0x7fb2540a4d40 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb26002c240 tx=0x7fb26002c7e0 comp rx=0 tx=0).stop 2024-03-28T16:01:13.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.040+0000 7fb272c00640 1 -- 172.21.15.67:0/1774919981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb26c108cb0 msgr2=0x7fb26c19e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:13.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.040+0000 7fb272c00640 1 --2- 172.21.15.67:0/1774919981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb26c108cb0 0x7fb26c19e1f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb25800e9f0 tx=0x7fb25800eeb0 comp rx=0 tx=0).stop 2024-03-28T16:01:13.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.040+0000 7fb272c00640 1 -- 172.21.15.67:0/1774919981 shutdown_connections 2024-03-28T16:01:13.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.040+0000 7fb272c00640 1 --2- 172.21.15.67:0/1774919981 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb2540a2880 0x7fb2540a4d40 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:13.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.040+0000 7fb272c00640 1 --2- 172.21.15.67:0/1774919981 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb26c109670 0x7fb26c1a5250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:13.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.040+0000 7fb272c00640 1 --2- 172.21.15.67:0/1774919981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb26c108cb0 0x7fb26c19e1f0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:13.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.040+0000 7fb272c00640 1 --2- 172.21.15.67:0/1774919981 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb26c107ac0 0x7fb26c19dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:13.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.040+0000 7fb272c00640 1 -- 172.21.15.67:0/1774919981 >> 172.21.15.67:0/1774919981 conn(0x7fb26c076a60 msgr2=0x7fb26c10bfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:13.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.040+0000 7fb272c00640 1 -- 172.21.15.67:0/1774919981 shutdown_connections 2024-03-28T16:01:13.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.040+0000 7fb272c00640 1 -- 172.21.15.67:0/1774919981 wait complete. 2024-03-28T16:01:13.063 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-03-28T16:01:13.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.287+0000 7fd0fdc00640 1 -- 172.21.15.67:0/2920608069 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0f81049b0 msgr2=0x7fd0f8104e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:13.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.287+0000 7fd0fdc00640 1 --2- 172.21.15.67:0/2920608069 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0f81049b0 0x7fd0f8104e30 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd0e0009900 tx=0x7fd0e002bc30 comp rx=0 tx=0).stop 2024-03-28T16:01:13.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.288+0000 7fd0fdc00640 1 -- 172.21.15.67:0/2920608069 shutdown_connections 2024-03-28T16:01:13.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.288+0000 7fd0fdc00640 1 --2- 172.21.15.67:0/2920608069 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd0f8105370 0x7fd0f810bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:13.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.288+0000 7fd0fdc00640 1 --2- 172.21.15.67:0/2920608069 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0f81049b0 0x7fd0f8104e30 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:13.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.288+0000 7fd0fdc00640 1 --2- 172.21.15.67:0/2920608069 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd0f81037c0 0x7fd0f8103be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:13.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.288+0000 7fd0fdc00640 1 -- 172.21.15.67:0/2920608069 >> 172.21.15.67:0/2920608069 conn(0x7fd0f80fed50 msgr2=0x7fd0f8101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:13.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.288+0000 7fd0fdc00640 1 -- 172.21.15.67:0/2920608069 shutdown_connections 2024-03-28T16:01:13.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.288+0000 7fd0fdc00640 1 -- 172.21.15.67:0/2920608069 wait complete. 2024-03-28T16:01:13.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.289+0000 7fd0fdc00640 1 Processor -- start 2024-03-28T16:01:13.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.289+0000 7fd0fdc00640 1 -- start start 2024-03-28T16:01:13.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.289+0000 7fd0fdc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0f81037c0 0x7fd0f81999d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:13.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.289+0000 7fd0fdc00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd0f81049b0 0x7fd0f8199f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:13.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.289+0000 7fd0fdc00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd0f8105370 0x7fd0f81a0f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:13.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.289+0000 7fd0fdc00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f810f2d0 con 0x7fd0f81037c0 2024-03-28T16:01:13.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.289+0000 7fd0fdc00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f810f150 con 0x7fd0f81049b0 2024-03-28T16:01:13.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.289+0000 7fd0fdc00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f810f450 con 0x7fd0f8105370 2024-03-28T16:01:13.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.289+0000 7fd0f7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0f81037c0 0x7fd0f81999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:13.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.289+0000 7fd0f7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0f81037c0 0x7fd0f81999d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41250/0 (socket says 172.21.15.67:41250) 2024-03-28T16:01:13.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.289+0000 7fd0f7e00640 1 -- 172.21.15.67:0/3683670762 learned_addr learned my addr 172.21.15.67:0/3683670762 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:01:13.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.290+0000 7fd0fd200640 1 --2- 172.21.15.67:0/3683670762 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd0f8105370 0x7fd0f81a0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:13.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.290+0000 7fd0efe00640 1 --2- 172.21.15.67:0/3683670762 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd0f81049b0 0x7fd0f8199f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:13.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.290+0000 7fd0f7e00640 1 -- 172.21.15.67:0/3683670762 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd0f8105370 msgr2=0x7fd0f81a0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:13.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.290+0000 7fd0f7e00640 1 --2- 172.21.15.67:0/3683670762 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd0f8105370 0x7fd0f81a0f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:13.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.290+0000 7fd0f7e00640 1 -- 172.21.15.67:0/3683670762 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd0f81049b0 msgr2=0x7fd0f8199f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:13.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.290+0000 7fd0f7e00640 1 --2- 172.21.15.67:0/3683670762 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd0f81049b0 0x7fd0f8199f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:13.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.290+0000 7fd0f7e00640 1 -- 172.21.15.67:0/3683670762 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0f81a15f0 con 0x7fd0f81037c0 2024-03-28T16:01:13.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.290+0000 7fd0fd200640 1 --2- 172.21.15.67:0/3683670762 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd0f8105370 0x7fd0f81a0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:01:13.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.290+0000 7fd0efe00640 1 --2- 172.21.15.67:0/3683670762 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd0f81049b0 0x7fd0f8199f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:01:13.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.290+0000 7fd0f7e00640 1 --2- 172.21.15.67:0/3683670762 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0f81037c0 0x7fd0f81999d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd0e800d950 tx=0x7fd0e800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:13.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.290+0000 7fd0f5600640 1 -- 172.21.15.67:0/3683670762 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0e8017020 con 0x7fd0f81037c0 2024-03-28T16:01:13.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.290+0000 7fd0fdc00640 1 -- 172.21.15.67:0/3683670762 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0f81a18e0 con 0x7fd0f81037c0 2024-03-28T16:01:13.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.290+0000 7fd0f5600640 1 -- 172.21.15.67:0/3683670762 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd0e8009c30 con 0x7fd0f81037c0 2024-03-28T16:01:13.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.290+0000 7fd0f5600640 1 -- 172.21.15.67:0/3683670762 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0e800f6b0 con 0x7fd0f81037c0 2024-03-28T16:01:13.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.291+0000 7fd0fdc00640 1 -- 172.21.15.67:0/3683670762 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0f81a1e30 con 0x7fd0f81037c0 2024-03-28T16:01:13.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.292+0000 7fd0fdc00640 1 -- 172.21.15.67:0/3683670762 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0f810ab50 con 0x7fd0f81037c0 2024-03-28T16:01:13.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.294+0000 7fd0f5600640 1 -- 172.21.15.67:0/3683670762 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fd0e801a020 con 0x7fd0f81037c0 2024-03-28T16:01:13.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.295+0000 7fd0f5600640 1 --2- 172.21.15.67:0/3683670762 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fd0dc0a2800 0x7fd0dc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:13.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.295+0000 7fd0f5600640 1 -- 172.21.15.67:0/3683670762 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fd0e80ced80 con 0x7fd0f81037c0 2024-03-28T16:01:13.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.295+0000 7fd0efe00640 1 --2- 172.21.15.67:0/3683670762 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fd0dc0a2800 0x7fd0dc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:13.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.296+0000 7fd0efe00640 1 --2- 172.21.15.67:0/3683670762 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fd0dc0a2800 0x7fd0dc0a4cc0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd0e00096c0 tx=0x7fd0e0000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:13.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.297+0000 7fd0f5600640 1 -- 172.21.15.67:0/3683670762 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd0e8021020 con 0x7fd0f81037c0 2024-03-28T16:01:13.319 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 16:01:13 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T16:01:13.000+0000 7f4f50e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T16:01:13.319 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 16:01:13 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T16:01:13.000+0000 7f4f50e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T16:01:13.320 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 16:01:13 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T16:01:13.000+0000 7f4f50e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T16:01:13.320 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 16:01:13 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T16:01:13.000+0000 7f4f50e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T16:01:13.320 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 16:01:13 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T16:01:13.000+0000 7f4f50e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T16:01:13.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.532+0000 7fd0fdc00640 1 -- 172.21.15.67:0/3683670762 --> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7fd0f8061ae0 con 0x7fd0dc0a2800 2024-03-28T16:01:13.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.542+0000 7fd0f5600640 1 -- 172.21.15.67:0/3683670762 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fd0e8097e40 con 0x7fd0f81037c0 2024-03-28T16:01:13.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.599+0000 7fd0f5600640 1 -- 172.21.15.67:0/3683670762 <== mgr.14152 v2:172.21.15.67:6800/208726654 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fd0f8061ae0 con 0x7fd0dc0a2800 2024-03-28T16:01:13.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.603+0000 7fd0fdc00640 1 -- 172.21.15.67:0/3683670762 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fd0dc0a2800 msgr2=0x7fd0dc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:13.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.603+0000 7fd0fdc00640 1 --2- 172.21.15.67:0/3683670762 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fd0dc0a2800 0x7fd0dc0a4cc0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd0e00096c0 tx=0x7fd0e0000bc0 comp rx=0 tx=0).stop 2024-03-28T16:01:13.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.603+0000 7fd0fdc00640 1 -- 172.21.15.67:0/3683670762 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0f81037c0 msgr2=0x7fd0f81999d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:13.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.603+0000 7fd0fdc00640 1 --2- 172.21.15.67:0/3683670762 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0f81037c0 0x7fd0f81999d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd0e800d950 tx=0x7fd0e800de10 comp rx=0 tx=0).stop 2024-03-28T16:01:13.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.603+0000 7fd0fdc00640 1 -- 172.21.15.67:0/3683670762 shutdown_connections 2024-03-28T16:01:13.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.603+0000 7fd0fdc00640 1 --2- 172.21.15.67:0/3683670762 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fd0dc0a2800 0x7fd0dc0a4cc0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:13.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.603+0000 7fd0fdc00640 1 --2- 172.21.15.67:0/3683670762 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd0f8105370 0x7fd0f81a0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:13.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.603+0000 7fd0fdc00640 1 --2- 172.21.15.67:0/3683670762 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd0f81049b0 0x7fd0f8199f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:13.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.603+0000 7fd0fdc00640 1 --2- 172.21.15.67:0/3683670762 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0f81037c0 0x7fd0f81999d0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:13.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.603+0000 7fd0fdc00640 1 -- 172.21.15.67:0/3683670762 >> 172.21.15.67:0/3683670762 conn(0x7fd0f80fed50 msgr2=0x7fd0f8107d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:13.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.603+0000 7fd0fdc00640 1 -- 172.21.15.67:0/3683670762 shutdown_connections 2024-03-28T16:01:13.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.604+0000 7fd0fdc00640 1 -- 172.21.15.67:0/3683670762 wait complete. 2024-03-28T16:01:13.624 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-03-28T16:01:13.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:13 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/2985693106' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-28T16:01:13.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:13 smithi150 ceph-mon[34825]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-28T16:01:13.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:13 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/2985693106' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-28T16:01:13.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:13 smithi067 ceph-mon[31707]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-28T16:01:13.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.846+0000 7f00b4a00640 1 -- 172.21.15.67:0/2942651615 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00b0104a50 msgr2=0x7f00b0106e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:13.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.846+0000 7f00b4a00640 1 --2- 172.21.15.67:0/2942651615 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00b0104a50 0x7f00b0106e40 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f0098009900 tx=0x7f009802bc30 comp rx=0 tx=0).stop 2024-03-28T16:01:13.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.846+0000 7f00b4a00640 1 -- 172.21.15.67:0/2942651615 shutdown_connections 2024-03-28T16:01:13.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.846+0000 7f00b4a00640 1 --2- 172.21.15.67:0/2942651615 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00b0107380 0x7f00b01097d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:13.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.846+0000 7f00b4a00640 1 --2- 172.21.15.67:0/2942651615 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00b0104a50 0x7f00b0106e40 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:13.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.846+0000 7f00b4a00640 1 --2- 172.21.15.67:0/2942651615 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f00b0102120 0x7f00b0104510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:13.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.846+0000 7f00b4a00640 1 -- 172.21.15.67:0/2942651615 >> 172.21.15.67:0/2942651615 conn(0x7f00b00fbb30 msgr2=0x7f00b00fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:13.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.846+0000 7f00b4a00640 1 -- 172.21.15.67:0/2942651615 shutdown_connections 2024-03-28T16:01:13.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.846+0000 7f00b4a00640 1 -- 172.21.15.67:0/2942651615 wait complete. 2024-03-28T16:01:13.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.847+0000 7f00b4a00640 1 Processor -- start 2024-03-28T16:01:13.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.847+0000 7f00b4a00640 1 -- start start 2024-03-28T16:01:13.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.848+0000 7f00b4a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00b0102120 0x7f00b0199b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:13.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.848+0000 7f00b4a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00b0104a50 0x7f00b019a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:13.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.848+0000 7f00b4a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f00b0107380 0x7f00b01a10e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:13.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.848+0000 7f00b4a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00b010d230 con 0x7f00b0104a50 2024-03-28T16:01:13.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.848+0000 7f00b4a00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00b010d0b0 con 0x7f00b0102120 2024-03-28T16:01:13.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.848+0000 7f00b4a00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00b010d3b0 con 0x7f00b0107380 2024-03-28T16:01:13.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.848+0000 7f00ad400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00b0104a50 0x7f00b019a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:13.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.848+0000 7f00ad400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00b0104a50 0x7f00b019a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41266/0 (socket says 172.21.15.67:41266) 2024-03-28T16:01:13.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.848+0000 7f00ad400640 1 -- 172.21.15.67:0/2991431641 learned_addr learned my addr 172.21.15.67:0/2991431641 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:01:13.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.848+0000 7f00ade00640 1 --2- 172.21.15.67:0/2991431641 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00b0102120 0x7f00b0199b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:13.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.848+0000 7f00ad400640 1 -- 172.21.15.67:0/2991431641 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f00b0107380 msgr2=0x7f00b01a10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:13.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.848+0000 7f00ad400640 1 --2- 172.21.15.67:0/2991431641 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f00b0107380 0x7f00b01a10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:13.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.848+0000 7f00ad400640 1 -- 172.21.15.67:0/2991431641 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00b0102120 msgr2=0x7f00b0199b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:13.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.848+0000 7f00ad400640 1 --2- 172.21.15.67:0/2991431641 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00b0102120 0x7f00b0199b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:13.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.848+0000 7f00ad400640 1 -- 172.21.15.67:0/2991431641 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00b01a1750 con 0x7f00b0104a50 2024-03-28T16:01:13.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.849+0000 7f00ade00640 1 --2- 172.21.15.67:0/2991431641 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00b0102120 0x7f00b0199b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:01:13.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.849+0000 7f00ad400640 1 --2- 172.21.15.67:0/2991431641 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00b0104a50 0x7f00b019a070 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f0098005ab0 tx=0x7f00980042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:13.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.849+0000 7f00a6a00640 1 -- 172.21.15.67:0/2991431641 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0098043020 con 0x7f00b0104a50 2024-03-28T16:01:13.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.849+0000 7f00a6a00640 1 -- 172.21.15.67:0/2991431641 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f00980054c0 con 0x7f00b0104a50 2024-03-28T16:01:13.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.849+0000 7f00b4a00640 1 -- 172.21.15.67:0/2991431641 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00b01a19e0 con 0x7f00b0104a50 2024-03-28T16:01:13.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.849+0000 7f00a6a00640 1 -- 172.21.15.67:0/2991431641 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0098003d10 con 0x7f00b0104a50 2024-03-28T16:01:13.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.849+0000 7f00b4a00640 1 -- 172.21.15.67:0/2991431641 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00b01a1f30 con 0x7f00b0104a50 2024-03-28T16:01:13.854 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:13 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/2985693106' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-28T16:01:13.854 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:13 smithi183 ceph-mon[35580]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-28T16:01:13.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.850+0000 7f00b4a00640 1 -- 172.21.15.67:0/2991431641 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00b0108ed0 con 0x7f00b0104a50 2024-03-28T16:01:13.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.852+0000 7f00a6a00640 1 -- 172.21.15.67:0/2991431641 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f0098046020 con 0x7f00b0104a50 2024-03-28T16:01:13.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.854+0000 7f00a6a00640 1 --2- 172.21.15.67:0/2991431641 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f00940a2890 0x7f00940a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:13.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.854+0000 7f00a6a00640 1 -- 172.21.15.67:0/2991431641 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f00980f0e80 con 0x7f00b0104a50 2024-03-28T16:01:13.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.854+0000 7f00ade00640 1 --2- 172.21.15.67:0/2991431641 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f00940a2890 0x7f00940a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:13.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.855+0000 7f00ade00640 1 --2- 172.21.15.67:0/2991431641 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f00940a2890 0x7f00940a4d50 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f00a0009b40 tx=0x7f00a0005e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:13.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:13.859+0000 7f00a6a00640 1 -- 172.21.15.67:0/2991431641 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f009804d020 con 0x7f00b0104a50 2024-03-28T16:01:14.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.091+0000 7f00b4a00640 1 -- 172.21.15.67:0/2991431641 --> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) v1 -- 0x7f00b0061b70 con 0x7f00940a2890 2024-03-28T16:01:14.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.105+0000 7f00a6a00640 1 -- 172.21.15.67:0/2991431641 <== mgr.14152 v2:172.21.15.67:6800/208726654 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+54 (secure 0 0 0) 0x7f00b0061b70 con 0x7f00940a2890 2024-03-28T16:01:14.105 INFO:teuthology.orchestra.run.smithi067.stdout:/volumes/qa/sv_0/7ad6a264-caac-444d-a6a0-0c4d0503695c 2024-03-28T16:01:14.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.107+0000 7f00b4a00640 1 -- 172.21.15.67:0/2991431641 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f00940a2890 msgr2=0x7f00940a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:14.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.107+0000 7f00b4a00640 1 --2- 172.21.15.67:0/2991431641 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f00940a2890 0x7f00940a4d50 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f00a0009b40 tx=0x7f00a0005e50 comp rx=0 tx=0).stop 2024-03-28T16:01:14.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.107+0000 7f00b4a00640 1 -- 172.21.15.67:0/2991431641 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00b0104a50 msgr2=0x7f00b019a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:14.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.107+0000 7f00b4a00640 1 --2- 172.21.15.67:0/2991431641 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00b0104a50 0x7f00b019a070 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f0098005ab0 tx=0x7f00980042f0 comp rx=0 tx=0).stop 2024-03-28T16:01:14.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.107+0000 7f00b4a00640 1 -- 172.21.15.67:0/2991431641 shutdown_connections 2024-03-28T16:01:14.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.107+0000 7f00b4a00640 1 --2- 172.21.15.67:0/2991431641 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f00940a2890 0x7f00940a4d50 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:14.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.107+0000 7f00b4a00640 1 --2- 172.21.15.67:0/2991431641 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f00b0107380 0x7f00b01a10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:14.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.107+0000 7f00b4a00640 1 --2- 172.21.15.67:0/2991431641 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00b0104a50 0x7f00b019a070 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:14.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.107+0000 7f00b4a00640 1 --2- 172.21.15.67:0/2991431641 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00b0102120 0x7f00b0199b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:14.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.107+0000 7f00b4a00640 1 -- 172.21.15.67:0/2991431641 >> 172.21.15.67:0/2991431641 conn(0x7f00b00fbb30 msgr2=0x7f00b0101850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:14.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.107+0000 7f00b4a00640 1 -- 172.21.15.67:0/2991431641 shutdown_connections 2024-03-28T16:01:14.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.107+0000 7f00b4a00640 1 -- 172.21.15.67:0/2991431641 wait complete. 2024-03-28T16:01:14.128 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-03-28T16:01:14.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.351+0000 7f0d37000640 1 -- 172.21.15.67:0/1780622150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d301037c0 msgr2=0x7f0d30103be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:14.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.351+0000 7f0d37000640 1 --2- 172.21.15.67:0/1780622150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d301037c0 0x7f0d30103be0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f0d24009900 tx=0x7f0d2402bc30 comp rx=0 tx=0).stop 2024-03-28T16:01:14.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.351+0000 7f0d37000640 1 -- 172.21.15.67:0/1780622150 shutdown_connections 2024-03-28T16:01:14.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.351+0000 7f0d37000640 1 --2- 172.21.15.67:0/1780622150 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0d30105370 0x7f0d3010bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:14.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.351+0000 7f0d37000640 1 --2- 172.21.15.67:0/1780622150 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d301049b0 0x7f0d30104e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:14.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.351+0000 7f0d37000640 1 --2- 172.21.15.67:0/1780622150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d301037c0 0x7f0d30103be0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:14.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.351+0000 7f0d37000640 1 -- 172.21.15.67:0/1780622150 >> 172.21.15.67:0/1780622150 conn(0x7f0d300fed50 msgr2=0x7f0d30101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:14.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.351+0000 7f0d37000640 1 -- 172.21.15.67:0/1780622150 shutdown_connections 2024-03-28T16:01:14.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.351+0000 7f0d37000640 1 -- 172.21.15.67:0/1780622150 wait complete. 2024-03-28T16:01:14.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.353+0000 7f0d37000640 1 Processor -- start 2024-03-28T16:01:14.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.353+0000 7f0d37000640 1 -- start start 2024-03-28T16:01:14.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.353+0000 7f0d37000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d301037c0 0x7f0d301999e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:14.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.353+0000 7f0d37000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d301049b0 0x7f0d30199f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:14.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.353+0000 7f0d37000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0d30105370 0x7f0d301a0f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:14.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.353+0000 7f0d37000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d3010f390 con 0x7f0d301049b0 2024-03-28T16:01:14.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.353+0000 7f0d37000640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d3010f210 con 0x7f0d301037c0 2024-03-28T16:01:14.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.353+0000 7f0d37000640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d3010f510 con 0x7f0d30105370 2024-03-28T16:01:14.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.353+0000 7f0d35200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d301049b0 0x7f0d30199f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:14.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.353+0000 7f0d35200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d301049b0 0x7f0d30199f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41286/0 (socket says 172.21.15.67:41286) 2024-03-28T16:01:14.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.353+0000 7f0d35200640 1 -- 172.21.15.67:0/2344078947 learned_addr learned my addr 172.21.15.67:0/2344078947 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:01:14.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.354+0000 7f0d35c00640 1 --2- 172.21.15.67:0/2344078947 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d301037c0 0x7f0d301999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:14.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.354+0000 7f0d36600640 1 --2- 172.21.15.67:0/2344078947 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0d30105370 0x7f0d301a0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:14.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.354+0000 7f0d35200640 1 -- 172.21.15.67:0/2344078947 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0d30105370 msgr2=0x7f0d301a0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:14.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.354+0000 7f0d35200640 1 --2- 172.21.15.67:0/2344078947 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0d30105370 0x7f0d301a0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:14.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.354+0000 7f0d35200640 1 -- 172.21.15.67:0/2344078947 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d301037c0 msgr2=0x7f0d301999e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:14.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.354+0000 7f0d35200640 1 --2- 172.21.15.67:0/2344078947 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d301037c0 0x7f0d301999e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:14.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.354+0000 7f0d35200640 1 -- 172.21.15.67:0/2344078947 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d301a1600 con 0x7f0d301049b0 2024-03-28T16:01:14.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.354+0000 7f0d36600640 1 --2- 172.21.15.67:0/2344078947 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0d30105370 0x7f0d301a0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:01:14.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.354+0000 7f0d35200640 1 --2- 172.21.15.67:0/2344078947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d301049b0 0x7f0d30199f20 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f0d1c00c920 tx=0x7f0d1c00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:14.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.354+0000 7f0d2e000640 1 -- 172.21.15.67:0/2344078947 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d1c007c70 con 0x7f0d301049b0 2024-03-28T16:01:14.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.354+0000 7f0d2e000640 1 -- 172.21.15.67:0/2344078947 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0d1c007e10 con 0x7f0d301049b0 2024-03-28T16:01:14.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.354+0000 7f0d2e000640 1 -- 172.21.15.67:0/2344078947 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d1c018820 con 0x7f0d301049b0 2024-03-28T16:01:14.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.354+0000 7f0d37000640 1 -- 172.21.15.67:0/2344078947 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d301a18f0 con 0x7f0d301049b0 2024-03-28T16:01:14.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.354+0000 7f0d37000640 1 -- 172.21.15.67:0/2344078947 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d301a1e70 con 0x7f0d301049b0 2024-03-28T16:01:14.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.356+0000 7f0d37000640 1 -- 172.21.15.67:0/2344078947 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d3010aae0 con 0x7f0d301049b0 2024-03-28T16:01:14.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.357+0000 7f0d2e000640 1 -- 172.21.15.67:0/2344078947 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f0d1c0189c0 con 0x7f0d301049b0 2024-03-28T16:01:14.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.359+0000 7f0d2e000640 1 --2- 172.21.15.67:0/2344078947 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f0d180a2910 0x7f0d180a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:14.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.359+0000 7f0d2e000640 1 -- 172.21.15.67:0/2344078947 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f0d1c0ceca0 con 0x7f0d301049b0 2024-03-28T16:01:14.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.359+0000 7f0d35c00640 1 --2- 172.21.15.67:0/2344078947 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f0d180a2910 0x7f0d180a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:14.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.360+0000 7f0d35c00640 1 --2- 172.21.15.67:0/2344078947 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f0d180a2910 0x7f0d180a4dd0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f0d240096c0 tx=0x7f0d24005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:14.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.362+0000 7f0d2e000640 1 -- 172.21.15.67:0/2344078947 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0d1c014030 con 0x7f0d301049b0 2024-03-28T16:01:14.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.597+0000 7f0d37000640 1 -- 172.21.15.67:0/2344078947 --> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7f0d30061ae0 con 0x7f0d180a2910 2024-03-28T16:01:14.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.668+0000 7f0d2e000640 1 -- 172.21.15.67:0/2344078947 <== mgr.14152 v2:172.21.15.67:6800/208726654 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f0d30061ae0 con 0x7f0d180a2910 2024-03-28T16:01:14.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.672+0000 7f0d37000640 1 -- 172.21.15.67:0/2344078947 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f0d180a2910 msgr2=0x7f0d180a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:14.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.672+0000 7f0d37000640 1 --2- 172.21.15.67:0/2344078947 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f0d180a2910 0x7f0d180a4dd0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f0d240096c0 tx=0x7f0d24005fd0 comp rx=0 tx=0).stop 2024-03-28T16:01:14.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.672+0000 7f0d37000640 1 -- 172.21.15.67:0/2344078947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d301049b0 msgr2=0x7f0d30199f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:14.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.672+0000 7f0d37000640 1 --2- 172.21.15.67:0/2344078947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d301049b0 0x7f0d30199f20 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f0d1c00c920 tx=0x7f0d1c00cde0 comp rx=0 tx=0).stop 2024-03-28T16:01:14.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.672+0000 7f0d37000640 1 -- 172.21.15.67:0/2344078947 shutdown_connections 2024-03-28T16:01:14.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.672+0000 7f0d37000640 1 --2- 172.21.15.67:0/2344078947 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f0d180a2910 0x7f0d180a4dd0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:14.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.672+0000 7f0d37000640 1 --2- 172.21.15.67:0/2344078947 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0d30105370 0x7f0d301a0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:14.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.672+0000 7f0d37000640 1 --2- 172.21.15.67:0/2344078947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d301049b0 0x7f0d30199f20 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:14.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.672+0000 7f0d37000640 1 --2- 172.21.15.67:0/2344078947 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d301037c0 0x7f0d301999e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:14.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.672+0000 7f0d37000640 1 -- 172.21.15.67:0/2344078947 >> 172.21.15.67:0/2344078947 conn(0x7f0d300fed50 msgr2=0x7f0d30107cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:14.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.673+0000 7f0d37000640 1 -- 172.21.15.67:0/2344078947 shutdown_connections 2024-03-28T16:01:14.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.673+0000 7f0d37000640 1 -- 172.21.15.67:0/2344078947 wait complete. 2024-03-28T16:01:14.693 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-03-28T16:01:14.772 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:14 smithi067 ceph-mon[31707]: from='client.14718 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T16:01:14.772 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:14 smithi067 ceph-mon[31707]: pgmap v367: 97 pgs: 6 creating+peering, 91 active+clean; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 2.3 KiB/s wr, 7 op/s 2024-03-28T16:01:14.773 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:14 smithi067 ceph-mon[31707]: from='client.14730 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T16:01:14.773 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:14 smithi067 ceph-mon[31707]: mgrmap e17: x(active, since 9m), standbys: z, y 2024-03-28T16:01:14.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:14 smithi150 ceph-mon[34825]: from='client.14718 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T16:01:14.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:14 smithi150 ceph-mon[34825]: pgmap v367: 97 pgs: 6 creating+peering, 91 active+clean; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 2.3 KiB/s wr, 7 op/s 2024-03-28T16:01:14.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:14 smithi150 ceph-mon[34825]: from='client.14730 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T16:01:14.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:14 smithi150 ceph-mon[34825]: mgrmap e17: x(active, since 9m), standbys: z, y 2024-03-28T16:01:14.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:14 smithi183 ceph-mon[35580]: from='client.14718 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T16:01:14.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:14 smithi183 ceph-mon[35580]: pgmap v367: 97 pgs: 6 creating+peering, 91 active+clean; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 2.3 KiB/s wr, 7 op/s 2024-03-28T16:01:14.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:14 smithi183 ceph-mon[35580]: from='client.14730 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T16:01:14.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:14 smithi183 ceph-mon[35580]: mgrmap e17: x(active, since 9m), standbys: z, y 2024-03-28T16:01:14.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.918+0000 7fbe9a000640 1 -- 172.21.15.67:0/2166576509 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe940ffdf0 msgr2=0x7fbe94102220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:14.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.918+0000 7fbe9a000640 1 --2- 172.21.15.67:0/2166576509 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe940ffdf0 0x7fbe94102220 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fbe88009900 tx=0x7fbe8802bc30 comp rx=0 tx=0).stop 2024-03-28T16:01:14.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.918+0000 7fbe9a000640 1 -- 172.21.15.67:0/2166576509 shutdown_connections 2024-03-28T16:01:14.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.918+0000 7fbe9a000640 1 --2- 172.21.15.67:0/2166576509 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe94105110 0x7fbe94105590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:14.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.918+0000 7fbe9a000640 1 --2- 172.21.15.67:0/2166576509 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbe94102760 0x7fbe94104bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:14.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.918+0000 7fbe9a000640 1 --2- 172.21.15.67:0/2166576509 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe940ffdf0 0x7fbe94102220 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:14.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.918+0000 7fbe9a000640 1 -- 172.21.15.67:0/2166576509 >> 172.21.15.67:0/2166576509 conn(0x7fbe940fba10 msgr2=0x7fbe940fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:14.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.918+0000 7fbe9a000640 1 -- 172.21.15.67:0/2166576509 shutdown_connections 2024-03-28T16:01:14.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.918+0000 7fbe9a000640 1 -- 172.21.15.67:0/2166576509 wait complete. 2024-03-28T16:01:14.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.919+0000 7fbe9a000640 1 Processor -- start 2024-03-28T16:01:14.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.919+0000 7fbe9a000640 1 -- start start 2024-03-28T16:01:14.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.919+0000 7fbe9a000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe940ffdf0 0x7fbe9419ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:14.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.919+0000 7fbe9a000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe94102760 0x7fbe9419e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:14.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.919+0000 7fbe9a000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbe94105110 0x7fbe941a53a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:14.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.919+0000 7fbe9a000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe941136f0 con 0x7fbe940ffdf0 2024-03-28T16:01:14.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.919+0000 7fbe9a000640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe94113570 con 0x7fbe94102760 2024-03-28T16:01:14.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.919+0000 7fbe9a000640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe94113870 con 0x7fbe94105110 2024-03-28T16:01:14.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.920+0000 7fbe98c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe940ffdf0 0x7fbe9419ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:14.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.920+0000 7fbe98c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe940ffdf0 0x7fbe9419ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41304/0 (socket says 172.21.15.67:41304) 2024-03-28T16:01:14.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.920+0000 7fbe98c00640 1 -- 172.21.15.67:0/4044335469 learned_addr learned my addr 172.21.15.67:0/4044335469 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:01:14.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.920+0000 7fbe93e00640 1 --2- 172.21.15.67:0/4044335469 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe94102760 0x7fbe9419e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:14.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.920+0000 7fbe98c00640 1 -- 172.21.15.67:0/4044335469 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbe94105110 msgr2=0x7fbe941a53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:14.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.920+0000 7fbe99600640 1 --2- 172.21.15.67:0/4044335469 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbe94105110 0x7fbe941a53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:14.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.920+0000 7fbe98c00640 1 --2- 172.21.15.67:0/4044335469 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbe94105110 0x7fbe941a53a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:14.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.920+0000 7fbe98c00640 1 -- 172.21.15.67:0/4044335469 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe94102760 msgr2=0x7fbe9419e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:14.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.920+0000 7fbe98c00640 1 --2- 172.21.15.67:0/4044335469 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe94102760 0x7fbe9419e330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:14.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.920+0000 7fbe98c00640 1 -- 172.21.15.67:0/4044335469 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe941a5a10 con 0x7fbe940ffdf0 2024-03-28T16:01:14.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.920+0000 7fbe98c00640 1 --2- 172.21.15.67:0/4044335469 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe940ffdf0 0x7fbe9419ddf0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fbe88000c00 tx=0x7fbe880042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:14.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.921+0000 7fbe91600640 1 -- 172.21.15.67:0/4044335469 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe88043020 con 0x7fbe940ffdf0 2024-03-28T16:01:14.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.921+0000 7fbe9a000640 1 -- 172.21.15.67:0/4044335469 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe941a5ca0 con 0x7fbe940ffdf0 2024-03-28T16:01:14.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.921+0000 7fbe91600640 1 -- 172.21.15.67:0/4044335469 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbe880054c0 con 0x7fbe940ffdf0 2024-03-28T16:01:14.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.921+0000 7fbe91600640 1 -- 172.21.15.67:0/4044335469 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe88003d40 con 0x7fbe940ffdf0 2024-03-28T16:01:14.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.921+0000 7fbe9a000640 1 -- 172.21.15.67:0/4044335469 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe941a61a0 con 0x7fbe940ffdf0 2024-03-28T16:01:14.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.922+0000 7fbe9a000640 1 -- 172.21.15.67:0/4044335469 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe940fd500 con 0x7fbe940ffdf0 2024-03-28T16:01:14.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.924+0000 7fbe91600640 1 -- 172.21.15.67:0/4044335469 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fbe88046020 con 0x7fbe940ffdf0 2024-03-28T16:01:14.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.926+0000 7fbe91600640 1 --2- 172.21.15.67:0/4044335469 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fbe7c0a2890 0x7fbe7c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:14.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.926+0000 7fbe91600640 1 -- 172.21.15.67:0/4044335469 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fbe880f0ed0 con 0x7fbe940ffdf0 2024-03-28T16:01:14.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.926+0000 7fbe93e00640 1 --2- 172.21.15.67:0/4044335469 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fbe7c0a2890 0x7fbe7c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:14.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.927+0000 7fbe93e00640 1 --2- 172.21.15.67:0/4044335469 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fbe7c0a2890 0x7fbe7c0a4d50 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fbe80005fd0 tx=0x7fbe80005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:14.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:14.930+0000 7fbe91600640 1 -- 172.21.15.67:0/4044335469 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fbe8804d020 con 0x7fbe940ffdf0 2024-03-28T16:01:15.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.164+0000 7fbe9a000640 1 -- 172.21.15.67:0/4044335469 --> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fbe94061b70 con 0x7fbe7c0a2890 2024-03-28T16:01:15.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.180+0000 7fbe91600640 1 -- 172.21.15.67:0/4044335469 <== mgr.14152 v2:172.21.15.67:6800/208726654 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+54 (secure 0 0 0) 0x7fbe94061b70 con 0x7fbe7c0a2890 2024-03-28T16:01:15.181 INFO:teuthology.orchestra.run.smithi067.stdout:/volumes/qa/sv_1/2e4ad0b5-8d11-4ad7-a446-ec6d8c02f368 2024-03-28T16:01:15.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.183+0000 7fbe9a000640 1 -- 172.21.15.67:0/4044335469 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fbe7c0a2890 msgr2=0x7fbe7c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:15.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.183+0000 7fbe9a000640 1 --2- 172.21.15.67:0/4044335469 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fbe7c0a2890 0x7fbe7c0a4d50 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fbe80005fd0 tx=0x7fbe80005850 comp rx=0 tx=0).stop 2024-03-28T16:01:15.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.183+0000 7fbe9a000640 1 -- 172.21.15.67:0/4044335469 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe940ffdf0 msgr2=0x7fbe9419ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:15.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.183+0000 7fbe9a000640 1 --2- 172.21.15.67:0/4044335469 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe940ffdf0 0x7fbe9419ddf0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fbe88000c00 tx=0x7fbe880042f0 comp rx=0 tx=0).stop 2024-03-28T16:01:15.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.183+0000 7fbe9a000640 1 -- 172.21.15.67:0/4044335469 shutdown_connections 2024-03-28T16:01:15.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.183+0000 7fbe9a000640 1 --2- 172.21.15.67:0/4044335469 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fbe7c0a2890 0x7fbe7c0a4d50 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:15.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.183+0000 7fbe9a000640 1 --2- 172.21.15.67:0/4044335469 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbe94105110 0x7fbe941a53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:15.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.183+0000 7fbe9a000640 1 --2- 172.21.15.67:0/4044335469 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe94102760 0x7fbe9419e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:15.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.183+0000 7fbe9a000640 1 --2- 172.21.15.67:0/4044335469 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe940ffdf0 0x7fbe9419ddf0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:15.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.183+0000 7fbe9a000640 1 -- 172.21.15.67:0/4044335469 >> 172.21.15.67:0/4044335469 conn(0x7fbe940fba10 msgr2=0x7fbe941031d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:15.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.183+0000 7fbe9a000640 1 -- 172.21.15.67:0/4044335469 shutdown_connections 2024-03-28T16:01:15.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.183+0000 7fbe9a000640 1 -- 172.21.15.67:0/4044335469 wait complete. 2024-03-28T16:01:15.203 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T16:01:15.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.426+0000 7f1e49200640 1 -- 172.21.15.67:0/3050038585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e44074100 msgr2=0x7f1e44074580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:15.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.426+0000 7f1e49200640 1 --2- 172.21.15.67:0/3050038585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e44074100 0x7f1e44074580 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f1e2c009ca0 tx=0x7f1e2c02c100 comp rx=0 tx=0).stop 2024-03-28T16:01:15.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.426+0000 7f1e49200640 1 -- 172.21.15.67:0/3050038585 shutdown_connections 2024-03-28T16:01:15.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.426+0000 7f1e49200640 1 --2- 172.21.15.67:0/3050038585 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1e441093d0 0x7f1e4410b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:15.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.426+0000 7f1e49200640 1 --2- 172.21.15.67:0/3050038585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e44074100 0x7f1e44074580 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:15.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.426+0000 7f1e49200640 1 --2- 172.21.15.67:0/3050038585 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e440756e0 0x7f1e44073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:15.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.426+0000 7f1e49200640 1 -- 172.21.15.67:0/3050038585 >> 172.21.15.67:0/3050038585 conn(0x7f1e440fca80 msgr2=0x7f1e440feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:15.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.426+0000 7f1e49200640 1 -- 172.21.15.67:0/3050038585 shutdown_connections 2024-03-28T16:01:15.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.426+0000 7f1e49200640 1 -- 172.21.15.67:0/3050038585 wait complete. 2024-03-28T16:01:15.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.427+0000 7f1e49200640 1 Processor -- start 2024-03-28T16:01:15.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.427+0000 7f1e49200640 1 -- start start 2024-03-28T16:01:15.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.428+0000 7f1e49200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e44074100 0x7f1e441998c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:15.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.428+0000 7f1e49200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e440756e0 0x7f1e44199e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:15.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.428+0000 7f1e49200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1e441093d0 0x7f1e441a0e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:15.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.428+0000 7f1e49200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e4410f200 con 0x7f1e440756e0 2024-03-28T16:01:15.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.428+0000 7f1e49200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e4410f080 con 0x7f1e44074100 2024-03-28T16:01:15.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.428+0000 7f1e49200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e4410f380 con 0x7f1e441093d0 2024-03-28T16:01:15.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.428+0000 7f1e42a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e440756e0 0x7f1e44199e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:15.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.428+0000 7f1e42a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e440756e0 0x7f1e44199e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41322/0 (socket says 172.21.15.67:41322) 2024-03-28T16:01:15.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.428+0000 7f1e42a00640 1 -- 172.21.15.67:0/986537402 learned_addr learned my addr 172.21.15.67:0/986537402 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:01:15.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.428+0000 7f1e43400640 1 --2- 172.21.15.67:0/986537402 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e44074100 0x7f1e441998c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:15.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.428+0000 7f1e43e00640 1 --2- 172.21.15.67:0/986537402 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1e441093d0 0x7f1e441a0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:15.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.428+0000 7f1e42a00640 1 -- 172.21.15.67:0/986537402 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1e441093d0 msgr2=0x7f1e441a0e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:15.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.428+0000 7f1e42a00640 1 --2- 172.21.15.67:0/986537402 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1e441093d0 0x7f1e441a0e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:15.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.428+0000 7f1e42a00640 1 -- 172.21.15.67:0/986537402 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e44074100 msgr2=0x7f1e441998c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:15.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.428+0000 7f1e42a00640 1 --2- 172.21.15.67:0/986537402 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e44074100 0x7f1e441998c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:15.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.428+0000 7f1e42a00640 1 -- 172.21.15.67:0/986537402 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e441a14e0 con 0x7f1e440756e0 2024-03-28T16:01:15.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.429+0000 7f1e42a00640 1 --2- 172.21.15.67:0/986537402 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e440756e0 0x7f1e44199e00 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f1e2c032380 tx=0x7f1e2c035ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:15.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.429+0000 7f1e3be00640 1 -- 172.21.15.67:0/986537402 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e2c039040 con 0x7f1e440756e0 2024-03-28T16:01:15.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.429+0000 7f1e3be00640 1 -- 172.21.15.67:0/986537402 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1e2c044070 con 0x7f1e440756e0 2024-03-28T16:01:15.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.429+0000 7f1e49200640 1 -- 172.21.15.67:0/986537402 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e441a1770 con 0x7f1e440756e0 2024-03-28T16:01:15.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.429+0000 7f1e3be00640 1 -- 172.21.15.67:0/986537402 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e2c004530 con 0x7f1e440756e0 2024-03-28T16:01:15.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.429+0000 7f1e49200640 1 -- 172.21.15.67:0/986537402 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e441a1cb0 con 0x7f1e440756e0 2024-03-28T16:01:15.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.431+0000 7f1e49200640 1 -- 172.21.15.67:0/986537402 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e44108100 con 0x7f1e440756e0 2024-03-28T16:01:15.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.432+0000 7f1e3be00640 1 -- 172.21.15.67:0/986537402 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f1e2c0327c0 con 0x7f1e440756e0 2024-03-28T16:01:15.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.434+0000 7f1e3be00640 1 --2- 172.21.15.67:0/986537402 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f1e240a2910 0x7f1e240a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:15.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.434+0000 7f1e3be00640 1 -- 172.21.15.67:0/986537402 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f1e2c0f0ff0 con 0x7f1e440756e0 2024-03-28T16:01:15.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.434+0000 7f1e43400640 1 --2- 172.21.15.67:0/986537402 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f1e240a2910 0x7f1e240a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:15.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.435+0000 7f1e43400640 1 --2- 172.21.15.67:0/986537402 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f1e240a2910 0x7f1e240a4dd0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f1e34005fd0 tx=0x7f1e34005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:15.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.438+0000 7f1e3be00640 1 -- 172.21.15.67:0/986537402 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1e2c04e020 con 0x7f1e440756e0 2024-03-28T16:01:15.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.684+0000 7f1e49200640 1 -- 172.21.15.67:0/986537402 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1e4402d2c0 con 0x7f1e440756e0 2024-03-28T16:01:15.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.686+0000 7f1e3be00640 1 -- 172.21.15.67:0/986537402 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11341 (secure 0 0 0) 0x7f1e2c0be1f0 con 0x7f1e440756e0 2024-03-28T16:01:15.687 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:01:15.687 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2008297659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2008297659},{"type":"v1","addr":"172.21.15.67:6835","nonce":2008297659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14532,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/320295745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":320295745},{"type":"v1","addr":"172.21.15.67:6837","nonce":320295745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6839/691821273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":691821273},{"type":"v1","addr":"172.21.15.67:6839","nonce":691821273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6841/3211254875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6840","nonce":3211254875},{"type":"v1","addr":"172.21.15.67:6841","nonce":3211254875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2129665423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2129665423},{"type":"v1","addr":"172.21.15.183:6833","nonce":2129665423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24496,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/1589784425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":1589784425},{"type":"v1","addr":"172.21.15.183:6835","nonce":1589784425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24508,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/2185863835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":2185863835},{"type":"v1","addr":"172.21.15.183:6837","nonce":2185863835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24520,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6839/1242814398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":1242814398},{"type":"v1","addr":"172.21.15.183:6839","nonce":1242814398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":34409,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1008335521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1008335521},{"type":"v1","addr":"172.21.15.150:6833","nonce":1008335521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":34445,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3269935695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3269935695},{"type":"v1","addr":"172.21.15.150:6835","nonce":3269935695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T16:01:07.696824+0000","modified":"2024-03-28T16:01:11.997773+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":34508},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34475":{"gid":34475,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/1112886277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":1112886277},{"type":"v1","addr":"172.21.15.150:6837","nonce":1112886277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_34508":{"gid":34508,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6839/1211011614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":1211011614},{"type":"v1","addr":"172.21.15.150:6839","nonce":1211011614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34508,"qdb_cluster":[34508]},"id":1}]} 2024-03-28T16:01:15.687 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 19 2024-03-28T16:01:15.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.689+0000 7f1e49200640 1 -- 172.21.15.67:0/986537402 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f1e240a2910 msgr2=0x7f1e240a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:15.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.689+0000 7f1e49200640 1 --2- 172.21.15.67:0/986537402 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f1e240a2910 0x7f1e240a4dd0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f1e34005fd0 tx=0x7f1e34005850 comp rx=0 tx=0).stop 2024-03-28T16:01:15.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.689+0000 7f1e49200640 1 -- 172.21.15.67:0/986537402 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e440756e0 msgr2=0x7f1e44199e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:15.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.689+0000 7f1e49200640 1 --2- 172.21.15.67:0/986537402 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e440756e0 0x7f1e44199e00 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f1e2c032380 tx=0x7f1e2c035ce0 comp rx=0 tx=0).stop 2024-03-28T16:01:15.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.689+0000 7f1e49200640 1 -- 172.21.15.67:0/986537402 shutdown_connections 2024-03-28T16:01:15.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.689+0000 7f1e49200640 1 --2- 172.21.15.67:0/986537402 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f1e240a2910 0x7f1e240a4dd0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:15.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.689+0000 7f1e49200640 1 --2- 172.21.15.67:0/986537402 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1e441093d0 0x7f1e441a0e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:15.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.689+0000 7f1e49200640 1 --2- 172.21.15.67:0/986537402 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e440756e0 0x7f1e44199e00 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:15.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.689+0000 7f1e49200640 1 --2- 172.21.15.67:0/986537402 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e44074100 0x7f1e441998c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:15.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.689+0000 7f1e49200640 1 -- 172.21.15.67:0/986537402 >> 172.21.15.67:0/986537402 conn(0x7f1e440fca80 msgr2=0x7f1e441052d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:15.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.690+0000 7f1e49200640 1 -- 172.21.15.67:0/986537402 shutdown_connections 2024-03-28T16:01:15.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.690+0000 7f1e49200640 1 -- 172.21.15.67:0/986537402 wait complete. 2024-03-28T16:01:15.710 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T16:01:15.790 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:15 smithi067 ceph-mon[31707]: from='client.14736 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T16:01:15.791 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:15 smithi067 ceph-mon[31707]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T16:01:15.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:15 smithi150 ceph-mon[34825]: from='client.14736 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T16:01:15.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:15 smithi150 ceph-mon[34825]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T16:01:15.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:15 smithi183 ceph-mon[35580]: from='client.14736 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T16:01:15.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:15 smithi183 ceph-mon[35580]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T16:01:15.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.929+0000 7f55ba600640 1 -- 172.21.15.67:0/42312124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55b41052a0 msgr2=0x7f55b410bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:15.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.929+0000 7f55ba600640 1 --2- 172.21.15.67:0/42312124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55b41052a0 0x7f55b410bb30 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f55ac009ca0 tx=0x7f55ac02c0f0 comp rx=0 tx=0).stop 2024-03-28T16:01:15.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.929+0000 7f55ba600640 1 -- 172.21.15.67:0/42312124 shutdown_connections 2024-03-28T16:01:15.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.929+0000 7f55ba600640 1 --2- 172.21.15.67:0/42312124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55b41052a0 0x7f55b410bb30 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:15.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.929+0000 7f55ba600640 1 --2- 172.21.15.67:0/42312124 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55b41048e0 0x7f55b4104d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:15.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.929+0000 7f55ba600640 1 --2- 172.21.15.67:0/42312124 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f55b41036f0 0x7f55b4103b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:15.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.929+0000 7f55ba600640 1 -- 172.21.15.67:0/42312124 >> 172.21.15.67:0/42312124 conn(0x7f55b40fec80 msgr2=0x7f55b41010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:15.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.929+0000 7f55ba600640 1 -- 172.21.15.67:0/42312124 shutdown_connections 2024-03-28T16:01:15.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.929+0000 7f55ba600640 1 -- 172.21.15.67:0/42312124 wait complete. 2024-03-28T16:01:15.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.930+0000 7f55ba600640 1 Processor -- start 2024-03-28T16:01:15.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.930+0000 7f55ba600640 1 -- start start 2024-03-28T16:01:15.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.930+0000 7f55ba600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55b41036f0 0x7f55b41998c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:15.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.930+0000 7f55ba600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f55b41048e0 0x7f55b4199e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:15.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.931+0000 7f55ba600640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55b41052a0 0x7f55b41a0e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:15.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.931+0000 7f55ba600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55b410f220 con 0x7f55b41036f0 2024-03-28T16:01:15.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.931+0000 7f55ba600640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55b410f0a0 con 0x7f55b41052a0 2024-03-28T16:01:15.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.931+0000 7f55ba600640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55b410f3a0 con 0x7f55b41048e0 2024-03-28T16:01:15.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.931+0000 7f55b9200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55b41036f0 0x7f55b41998c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:15.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.931+0000 7f55b9200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55b41036f0 0x7f55b41998c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41338/0 (socket says 172.21.15.67:41338) 2024-03-28T16:01:15.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.931+0000 7f55b9200640 1 -- 172.21.15.67:0/463679785 learned_addr learned my addr 172.21.15.67:0/463679785 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:01:15.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.931+0000 7f55b3e00640 1 --2- 172.21.15.67:0/463679785 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f55b41048e0 0x7f55b4199e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:15.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.931+0000 7f55b9c00640 1 --2- 172.21.15.67:0/463679785 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55b41052a0 0x7f55b41a0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:15.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.931+0000 7f55b9200640 1 -- 172.21.15.67:0/463679785 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f55b41048e0 msgr2=0x7f55b4199e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:15.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.931+0000 7f55b9200640 1 --2- 172.21.15.67:0/463679785 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f55b41048e0 0x7f55b4199e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:15.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.931+0000 7f55b9200640 1 -- 172.21.15.67:0/463679785 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55b41052a0 msgr2=0x7f55b41a0e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:15.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.931+0000 7f55b9200640 1 --2- 172.21.15.67:0/463679785 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55b41052a0 0x7f55b41a0e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:15.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.931+0000 7f55b9200640 1 -- 172.21.15.67:0/463679785 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55b41a14e0 con 0x7f55b41036f0 2024-03-28T16:01:15.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.932+0000 7f55b9200640 1 --2- 172.21.15.67:0/463679785 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55b41036f0 0x7f55b41998c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f55a400d950 tx=0x7f55a400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:15.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.932+0000 7f55b1600640 1 -- 172.21.15.67:0/463679785 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55a4017070 con 0x7f55b41036f0 2024-03-28T16:01:15.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.932+0000 7f55ba600640 1 -- 172.21.15.67:0/463679785 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55b41a17d0 con 0x7f55b41036f0 2024-03-28T16:01:15.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.932+0000 7f55b1600640 1 -- 172.21.15.67:0/463679785 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f55a4009c30 con 0x7f55b41036f0 2024-03-28T16:01:15.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.932+0000 7f55b1600640 1 -- 172.21.15.67:0/463679785 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55a400f620 con 0x7f55b41036f0 2024-03-28T16:01:15.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.932+0000 7f55ba600640 1 -- 172.21.15.67:0/463679785 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55b41a1d20 con 0x7f55b41036f0 2024-03-28T16:01:15.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.933+0000 7f55ba600640 1 -- 172.21.15.67:0/463679785 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55b410aa10 con 0x7f55b41036f0 2024-03-28T16:01:15.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.935+0000 7f55b1600640 1 -- 172.21.15.67:0/463679785 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f55a401a070 con 0x7f55b41036f0 2024-03-28T16:01:15.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.936+0000 7f55b1600640 1 --2- 172.21.15.67:0/463679785 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f55980a2910 0x7f55980a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:15.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.936+0000 7f55b1600640 1 -- 172.21.15.67:0/463679785 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f55a40ce5a0 con 0x7f55b41036f0 2024-03-28T16:01:15.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.937+0000 7f55b3e00640 1 --2- 172.21.15.67:0/463679785 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f55980a2910 0x7f55980a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:15.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.937+0000 7f55b3e00640 1 --2- 172.21.15.67:0/463679785 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f55980a2910 0x7f55980a4dd0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f55b419add0 tx=0x7f559c00a4c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:15.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:15.941+0000 7f55b1600640 1 -- 172.21.15.67:0/463679785 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f55a4009de0 con 0x7f55b41036f0 2024-03-28T16:01:16.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.143+0000 7f55ba600640 1 -- 172.21.15.67:0/463679785 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f55b40622d0 con 0x7f55b41036f0 2024-03-28T16:01:16.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.145+0000 7f55b1600640 1 -- 172.21.15.67:0/463679785 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+22221 (secure 0 0 0) 0x7f55a409b590 con 0x7f55b41036f0 2024-03-28T16:01:16.146 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:01:16.147 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":90,"fsid":"b5f8225a-ed1a-11ee-95d0-87774f69a715","created":"2024-03-28T15:50:21.471924+0000","modified":"2024-03-28T16:01:09.230050+0000","last_up_change":"2024-03-28T16:00:01.369123+0000","last_in_change":"2024-03-28T15:59:37.515483+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T15:54:55.588007+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T16:01:05.719801+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T16:01:07.206303+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7c1791b6-9c0e-4512-a327-cad166bb5581","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6803","nonce":1894681701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6805","nonce":1894681701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6809","nonce":1894681701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6807","nonce":1894681701}]},"public_addr":"172.21.15.67:6803/1894681701","cluster_addr":"172.21.15.67:6805/1894681701","heartbeat_back_addr":"172.21.15.67:6809/1894681701","heartbeat_front_addr":"172.21.15.67:6807/1894681701","state":["exists","up"]},{"osd":1,"uuid":"0cdda7f3-d00c-4436-bcc1-49fb8072b2e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6801","nonce":1734465762}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6803","nonce":1734465762}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6807","nonce":1734465762}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6805","nonce":1734465762}]},"public_addr":"172.21.15.150:6801/1734465762","cluster_addr":"172.21.15.150:6803/1734465762","heartbeat_back_addr":"172.21.15.150:6807/1734465762","heartbeat_front_addr":"172.21.15.150:6805/1734465762","state":["exists","up"]},{"osd":2,"uuid":"a55b9a02-c3fd-4061-95c0-53a24a17ade0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6801","nonce":3398095931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6803","nonce":3398095931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6807","nonce":3398095931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6805","nonce":3398095931}]},"public_addr":"172.21.15.183:6801/3398095931","cluster_addr":"172.21.15.183:6803/3398095931","heartbeat_back_addr":"172.21.15.183:6807/3398095931","heartbeat_front_addr":"172.21.15.183:6805/3398095931","state":["exists","up"]},{"osd":3,"uuid":"d34940c0-5ce9-4b8b-aa36-4872414a043f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6811","nonce":1933541875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6813","nonce":1933541875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6817","nonce":1933541875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6815","nonce":1933541875}]},"public_addr":"172.21.15.67:6811/1933541875","cluster_addr":"172.21.15.67:6813/1933541875","heartbeat_back_addr":"172.21.15.67:6817/1933541875","heartbeat_front_addr":"172.21.15.67:6815/1933541875","state":["exists","up"]},{"osd":4,"uuid":"5cc3e260-83a3-4a4d-aa6a-34a3b12b2b5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6809","nonce":3775269993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6811","nonce":3775269993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6815","nonce":3775269993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6813","nonce":3775269993}]},"public_addr":"172.21.15.150:6809/3775269993","cluster_addr":"172.21.15.150:6811/3775269993","heartbeat_back_addr":"172.21.15.150:6815/3775269993","heartbeat_front_addr":"172.21.15.150:6813/3775269993","state":["exists","up"]},{"osd":5,"uuid":"a2542b21-d6ac-413e-8b9b-4a6548b20840","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6809","nonce":164284137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6811","nonce":164284137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6815","nonce":164284137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6813","nonce":164284137}]},"public_addr":"172.21.15.183:6809/164284137","cluster_addr":"172.21.15.183:6811/164284137","heartbeat_back_addr":"172.21.15.183:6815/164284137","heartbeat_front_addr":"172.21.15.183:6813/164284137","state":["exists","up"]},{"osd":6,"uuid":"8905a2d3-0f1b-4853-a218-22da2635150a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6819","nonce":3565528013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6821","nonce":3565528013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6825","nonce":3565528013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6823","nonce":3565528013}]},"public_addr":"172.21.15.67:6819/3565528013","cluster_addr":"172.21.15.67:6821/3565528013","heartbeat_back_addr":"172.21.15.67:6825/3565528013","heartbeat_front_addr":"172.21.15.67:6823/3565528013","state":["exists","up"]},{"osd":7,"uuid":"b275055b-b451-4180-91b1-e6e61a828c48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6817","nonce":3735115340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6819","nonce":3735115340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6823","nonce":3735115340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6821","nonce":3735115340}]},"public_addr":"172.21.15.150:6817/3735115340","cluster_addr":"172.21.15.150:6819/3735115340","heartbeat_back_addr":"172.21.15.150:6823/3735115340","heartbeat_front_addr":"172.21.15.150:6821/3735115340","state":["exists","up"]},{"osd":8,"uuid":"b6abf7ad-eb34-4f60-a4eb-348a92d7195b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6817","nonce":1025400095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6819","nonce":1025400095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6823","nonce":1025400095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6821","nonce":1025400095}]},"public_addr":"172.21.15.183:6817/1025400095","cluster_addr":"172.21.15.183:6819/1025400095","heartbeat_back_addr":"172.21.15.183:6823/1025400095","heartbeat_front_addr":"172.21.15.183:6821/1025400095","state":["exists","up"]},{"osd":9,"uuid":"28a78e3a-0657-40aa-816f-16a6fadadcbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6827","nonce":76885409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6829","nonce":76885409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6833","nonce":76885409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6831","nonce":76885409}]},"public_addr":"172.21.15.67:6827/76885409","cluster_addr":"172.21.15.67:6829/76885409","heartbeat_back_addr":"172.21.15.67:6833/76885409","heartbeat_front_addr":"172.21.15.67:6831/76885409","state":["exists","up"]},{"osd":10,"uuid":"d287907e-424f-4b49-a06e-4613c6de0474","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6825","nonce":3489982080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6827","nonce":3489982080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6831","nonce":3489982080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6829","nonce":3489982080}]},"public_addr":"172.21.15.150:6825/3489982080","cluster_addr":"172.21.15.150:6827/3489982080","heartbeat_back_addr":"172.21.15.150:6831/3489982080","heartbeat_front_addr":"172.21.15.150:6829/3489982080","state":["exists","up"]},{"osd":11,"uuid":"a3c29c47-a228-4f42-a23c-fbdb0c093699","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":85,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6825","nonce":1892639849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6826","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6827","nonce":1892639849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6831","nonce":1892639849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6829","nonce":1892639849}]},"public_addr":"172.21.15.183:6825/1892639849","cluster_addr":"172.21.15.183:6827/1892639849","heartbeat_back_addr":"172.21.15.183:6831/1892639849","heartbeat_front_addr":"172.21.15.183:6829/1892639849","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:53:45.646211+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:54:18.956590+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:54:52.161646+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:55:27.502822+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:56:01.162932+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:56:34.674452+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:57:09.298679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:57:43.230406+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:58:17.003821+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:58:51.085863+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:59:25.665065+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T16:00:00.263457+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6801/1631001968":"2024-03-29T15:51:38.875334+0000","172.21.15.67:6800/1631001968":"2024-03-29T15:51:38.875334+0000","172.21.15.67:0/1533056310":"2024-03-29T15:51:38.875334+0000","172.21.15.67:6801/3580491071":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/2030412458":"2024-03-29T15:51:03.859740+0000","172.21.15.67:6800/3580491071":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/2905307533":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/3763759988":"2024-03-29T15:51:38.875334+0000","172.21.15.67:0/4089399880":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/4239108661":"2024-03-29T15:51:38.875334+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T16:01:16.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.148+0000 7f55ba600640 1 -- 172.21.15.67:0/463679785 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f55980a2910 msgr2=0x7f55980a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:16.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.148+0000 7f55ba600640 1 --2- 172.21.15.67:0/463679785 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f55980a2910 0x7f55980a4dd0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f55b419add0 tx=0x7f559c00a4c0 comp rx=0 tx=0).stop 2024-03-28T16:01:16.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.148+0000 7f55ba600640 1 -- 172.21.15.67:0/463679785 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55b41036f0 msgr2=0x7f55b41998c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:16.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.148+0000 7f55ba600640 1 --2- 172.21.15.67:0/463679785 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55b41036f0 0x7f55b41998c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f55a400d950 tx=0x7f55a400de10 comp rx=0 tx=0).stop 2024-03-28T16:01:16.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.148+0000 7f55ba600640 1 -- 172.21.15.67:0/463679785 shutdown_connections 2024-03-28T16:01:16.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.148+0000 7f55ba600640 1 --2- 172.21.15.67:0/463679785 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f55980a2910 0x7f55980a4dd0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:16.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.148+0000 7f55ba600640 1 --2- 172.21.15.67:0/463679785 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55b41052a0 0x7f55b41a0e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:16.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.148+0000 7f55ba600640 1 --2- 172.21.15.67:0/463679785 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f55b41048e0 0x7f55b4199e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:16.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.148+0000 7f55ba600640 1 --2- 172.21.15.67:0/463679785 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55b41036f0 0x7f55b41998c0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:16.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.148+0000 7f55ba600640 1 -- 172.21.15.67:0/463679785 >> 172.21.15.67:0/463679785 conn(0x7f55b40fec80 msgr2=0x7f55b4107be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:16.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.148+0000 7f55ba600640 1 -- 172.21.15.67:0/463679785 shutdown_connections 2024-03-28T16:01:16.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.149+0000 7f55ba600640 1 -- 172.21.15.67:0/463679785 wait complete. 2024-03-28T16:01:16.169 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-28T16:01:16.169 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T16:01:16.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.390+0000 7f749b200640 1 -- 172.21.15.67:0/3858361104 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f749c103090 msgr2=0x7f749c109920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:16.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.390+0000 7f749b200640 1 --2- 172.21.15.67:0/3858361104 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f749c103090 0x7f749c109920 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f7494009ca0 tx=0x7f749402c100 comp rx=0 tx=0).stop 2024-03-28T16:01:16.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.391+0000 7f749b200640 1 -- 172.21.15.67:0/3858361104 shutdown_connections 2024-03-28T16:01:16.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.391+0000 7f749b200640 1 --2- 172.21.15.67:0/3858361104 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f749c103090 0x7f749c109920 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:16.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.391+0000 7f749b200640 1 --2- 172.21.15.67:0/3858361104 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f749c1026d0 0x7f749c102b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:16.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.391+0000 7f749b200640 1 --2- 172.21.15.67:0/3858361104 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f749c1014e0 0x7f749c101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:16.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.391+0000 7f749b200640 1 -- 172.21.15.67:0/3858361104 >> 172.21.15.67:0/3858361104 conn(0x7f749c0fca30 msgr2=0x7f749c0fee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:16.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.391+0000 7f749b200640 1 -- 172.21.15.67:0/3858361104 shutdown_connections 2024-03-28T16:01:16.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.391+0000 7f749b200640 1 -- 172.21.15.67:0/3858361104 wait complete. 2024-03-28T16:01:16.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.392+0000 7f749b200640 1 Processor -- start 2024-03-28T16:01:16.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.392+0000 7f749b200640 1 -- start start 2024-03-28T16:01:16.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.392+0000 7f749b200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f749c1014e0 0x7f749c075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:16.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.392+0000 7f749b200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f749c1026d0 0x7f749c073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:16.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.392+0000 7f749b200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f749c103090 0x7f749c073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:16.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.392+0000 7f749b200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f749c10cff0 con 0x7f749c1014e0 2024-03-28T16:01:16.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.392+0000 7f749b200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f749c10ce70 con 0x7f749c1026d0 2024-03-28T16:01:16.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.392+0000 7f749b200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f749c10d170 con 0x7f749c103090 2024-03-28T16:01:16.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.393+0000 7f7499e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f749c1014e0 0x7f749c075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:16.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.393+0000 7f7499e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f749c1014e0 0x7f749c075450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41350/0 (socket says 172.21.15.67:41350) 2024-03-28T16:01:16.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.393+0000 7f7499e00640 1 -- 172.21.15.67:0/92382507 learned_addr learned my addr 172.21.15.67:0/92382507 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:01:16.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.393+0000 7f7499400640 1 --2- 172.21.15.67:0/92382507 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f749c1026d0 0x7f749c073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:16.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.393+0000 7f749a800640 1 --2- 172.21.15.67:0/92382507 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f749c103090 0x7f749c073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:16.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.393+0000 7f7499e00640 1 -- 172.21.15.67:0/92382507 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f749c103090 msgr2=0x7f749c073fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:16.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.393+0000 7f7499e00640 1 --2- 172.21.15.67:0/92382507 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f749c103090 0x7f749c073fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:16.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.393+0000 7f7499e00640 1 -- 172.21.15.67:0/92382507 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f749c1026d0 msgr2=0x7f749c073a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:16.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.393+0000 7f7499e00640 1 --2- 172.21.15.67:0/92382507 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f749c1026d0 0x7f749c073a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:16.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.393+0000 7f7499e00640 1 -- 172.21.15.67:0/92382507 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f749c074860 con 0x7f749c1014e0 2024-03-28T16:01:16.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.393+0000 7f7499e00640 1 --2- 172.21.15.67:0/92382507 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f749c1014e0 0x7f749c075450 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f748c00c970 tx=0x7f748c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:16.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.394+0000 7f7492a00640 1 -- 172.21.15.67:0/92382507 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f748c007c10 con 0x7f749c1014e0 2024-03-28T16:01:16.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.394+0000 7f7492a00640 1 -- 172.21.15.67:0/92382507 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f748c007db0 con 0x7f749c1014e0 2024-03-28T16:01:16.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.394+0000 7f7492a00640 1 -- 172.21.15.67:0/92382507 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f748c00f870 con 0x7f749c1014e0 2024-03-28T16:01:16.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.394+0000 7f749b200640 1 -- 172.21.15.67:0/92382507 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f749c1a1750 con 0x7f749c1014e0 2024-03-28T16:01:16.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.394+0000 7f749b200640 1 -- 172.21.15.67:0/92382507 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f749c1a1cf0 con 0x7f749c1014e0 2024-03-28T16:01:16.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.395+0000 7f749b200640 1 -- 172.21.15.67:0/92382507 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f749c108800 con 0x7f749c1014e0 2024-03-28T16:01:16.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.397+0000 7f7492a00640 1 -- 172.21.15.67:0/92382507 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f748c00fa10 con 0x7f749c1014e0 2024-03-28T16:01:16.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.398+0000 7f7492a00640 1 --2- 172.21.15.67:0/92382507 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f74800a2910 0x7f74800a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:16.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.399+0000 7f7492a00640 1 -- 172.21.15.67:0/92382507 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f748c0cee00 con 0x7f749c1014e0 2024-03-28T16:01:16.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.399+0000 7f7499400640 1 --2- 172.21.15.67:0/92382507 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f74800a2910 0x7f74800a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:16.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.399+0000 7f7499400640 1 --2- 172.21.15.67:0/92382507 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f74800a2910 0x7f74800a4dd0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f749c0751c0 tx=0x7f748800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:16.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.403+0000 7f7492a00640 1 -- 172.21.15.67:0/92382507 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f748c09be80 con 0x7f749c1014e0 2024-03-28T16:01:16.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.600+0000 7f749b200640 1 -- 172.21.15.67:0/92382507 --> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f749c061ae0 con 0x7f74800a2910 2024-03-28T16:01:16.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.632+0000 7f7492a00640 1 -- 172.21.15.67:0/92382507 <== mgr.14152 v2:172.21.15.67:6800/208726654 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+258733 (secure 0 0 0) 0x7f749c061ae0 con 0x7f74800a2910 2024-03-28T16:01:16.636 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:01:16.637 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-28T16:01:16.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.637+0000 7f749b200640 1 -- 172.21.15.67:0/92382507 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f74800a2910 msgr2=0x7f74800a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:16.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.637+0000 7f749b200640 1 --2- 172.21.15.67:0/92382507 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f74800a2910 0x7f74800a4dd0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f749c0751c0 tx=0x7f748800b390 comp rx=0 tx=0).stop 2024-03-28T16:01:16.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.637+0000 7f749b200640 1 -- 172.21.15.67:0/92382507 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f749c1014e0 msgr2=0x7f749c075450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:16.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.637+0000 7f749b200640 1 --2- 172.21.15.67:0/92382507 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f749c1014e0 0x7f749c075450 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f748c00c970 tx=0x7f748c00ce30 comp rx=0 tx=0).stop 2024-03-28T16:01:16.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.637+0000 7f749b200640 1 -- 172.21.15.67:0/92382507 shutdown_connections 2024-03-28T16:01:16.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.637+0000 7f749b200640 1 --2- 172.21.15.67:0/92382507 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f74800a2910 0x7f74800a4dd0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:16.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.637+0000 7f749b200640 1 --2- 172.21.15.67:0/92382507 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f749c103090 0x7f749c073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:16.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.637+0000 7f749b200640 1 --2- 172.21.15.67:0/92382507 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f749c1026d0 0x7f749c073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:16.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.637+0000 7f749b200640 1 --2- 172.21.15.67:0/92382507 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f749c1014e0 0x7f749c075450 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:16.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.637+0000 7f749b200640 1 -- 172.21.15.67:0/92382507 >> 172.21.15.67:0/92382507 conn(0x7f749c0fca30 msgr2=0x7f749c1059d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:16.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.637+0000 7f749b200640 1 -- 172.21.15.67:0/92382507 shutdown_connections 2024-03-28T16:01:16.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.637+0000 7f749b200640 1 -- 172.21.15.67:0/92382507 wait complete. 2024-03-28T16:01:16.657 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":368,"stamp":"2024-03-28T16:01:15.040422+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":612178,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":144,"num_read_kb":141,"num_write":234,"num_write_kb":2974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":184,"ondisk_log_size":184,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":268,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":383216,"kb_used_data":13208,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124558608,"statfs":{"total":1151940427776,"available":1151548014592,"internally_reserved":0,"allocated":13524992,"data_stored":7357759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":378713402},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21810,"num_objects":23,"num_object_clones":0,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":18,"num_read_kb":32,"num_write":53,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.827300"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:10.304055+0000","last_change":"2024-03-28T16:01:08.268196+0000","last_active":"2024-03-28T16:01:10.304055+0000","last_peered":"2024-03-28T16:01:10.304055+0000","last_clean":"2024-03-28T16:01:10.304055+0000","last_became_active":"2024-03-28T16:01:08.266956+0000","last_became_peered":"2024-03-28T16:01:08.266956+0000","last_unstale":"2024-03-28T16:01:10.304055+0000","last_undegraded":"2024-03-28T16:01:10.304055+0000","last_fullsized":"2024-03-28T16:01:10.304055+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:38:44.296129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.542544+0000","last_change":"2024-03-28T16:01:08.264985+0000","last_active":"2024-03-28T16:01:09.542544+0000","last_peered":"2024-03-28T16:01:09.542544+0000","last_clean":"2024-03-28T16:01:09.542544+0000","last_became_active":"2024-03-28T16:01:08.262484+0000","last_became_peered":"2024-03-28T16:01:08.262484+0000","last_unstale":"2024-03-28T16:01:09.542544+0000","last_undegraded":"2024-03-28T16:01:09.542544+0000","last_fullsized":"2024-03-28T16:01:09.542544+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:33:07.538908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.541985+0000","last_change":"2024-03-28T16:01:08.267423+0000","last_active":"2024-03-28T16:01:09.541985+0000","last_peered":"2024-03-28T16:01:09.541985+0000","last_clean":"2024-03-28T16:01:09.541985+0000","last_became_active":"2024-03-28T16:01:08.263665+0000","last_became_peered":"2024-03-28T16:01:08.263665+0000","last_unstale":"2024-03-28T16:01:09.541985+0000","last_undegraded":"2024-03-28T16:01:09.541985+0000","last_fullsized":"2024-03-28T16:01:09.541985+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:09:43.488003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.543679+0000","last_change":"2024-03-28T16:01:08.274978+0000","last_active":"2024-03-28T16:01:09.543679+0000","last_peered":"2024-03-28T16:01:09.543679+0000","last_clean":"2024-03-28T16:01:09.543679+0000","last_became_active":"2024-03-28T16:01:08.274553+0000","last_became_peered":"2024-03-28T16:01:08.274553+0000","last_unstale":"2024-03-28T16:01:09.543679+0000","last_undegraded":"2024-03-28T16:01:09.543679+0000","last_fullsized":"2024-03-28T16:01:09.543679+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:16:03.997334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.297381+0000","last_change":"2024-03-28T16:01:08.262155+0000","last_active":"2024-03-28T16:01:09.297381+0000","last_peered":"2024-03-28T16:01:09.297381+0000","last_clean":"2024-03-28T16:01:09.297381+0000","last_became_active":"2024-03-28T16:01:08.261814+0000","last_became_peered":"2024-03-28T16:01:08.261814+0000","last_unstale":"2024-03-28T16:01:09.297381+0000","last_undegraded":"2024-03-28T16:01:09.297381+0000","last_fullsized":"2024-03-28T16:01:09.297381+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:41:04.242203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.544137+0000","last_change":"2024-03-28T16:01:08.273342+0000","last_active":"2024-03-28T16:01:09.544137+0000","last_peered":"2024-03-28T16:01:09.544137+0000","last_clean":"2024-03-28T16:01:09.544137+0000","last_became_active":"2024-03-28T16:01:08.272606+0000","last_became_peered":"2024-03-28T16:01:08.272606+0000","last_unstale":"2024-03-28T16:01:09.544137+0000","last_undegraded":"2024-03-28T16:01:09.544137+0000","last_fullsized":"2024-03-28T16:01:09.544137+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:44:48.580214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.249900+0000","last_change":"2024-03-28T16:01:08.278060+0000","last_active":"2024-03-28T16:01:09.249900+0000","last_peered":"2024-03-28T16:01:09.249900+0000","last_clean":"2024-03-28T16:01:09.249900+0000","last_became_active":"2024-03-28T16:01:08.277047+0000","last_became_peered":"2024-03-28T16:01:08.277047+0000","last_unstale":"2024-03-28T16:01:09.249900+0000","last_undegraded":"2024-03-28T16:01:09.249900+0000","last_fullsized":"2024-03-28T16:01:09.249900+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:46:30.005708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.294527+0000","last_change":"2024-03-28T16:01:08.273199+0000","last_active":"2024-03-28T16:01:09.294527+0000","last_peered":"2024-03-28T16:01:09.294527+0000","last_clean":"2024-03-28T16:01:09.294527+0000","last_became_active":"2024-03-28T16:01:08.272822+0000","last_became_peered":"2024-03-28T16:01:08.272822+0000","last_unstale":"2024-03-28T16:01:09.294527+0000","last_undegraded":"2024-03-28T16:01:09.294527+0000","last_fullsized":"2024-03-28T16:01:09.294527+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:28:11.495665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.545290+0000","last_change":"2024-03-28T16:01:08.270262+0000","last_active":"2024-03-28T16:01:09.545290+0000","last_peered":"2024-03-28T16:01:09.545290+0000","last_clean":"2024-03-28T16:01:09.545290+0000","last_became_active":"2024-03-28T16:01:08.269959+0000","last_became_peered":"2024-03-28T16:01:08.269959+0000","last_unstale":"2024-03-28T16:01:09.545290+0000","last_undegraded":"2024-03-28T16:01:09.545290+0000","last_fullsized":"2024-03-28T16:01:09.545290+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:49:02.261893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.543775+0000","last_change":"2024-03-28T16:01:08.277064+0000","last_active":"2024-03-28T16:01:09.543775+0000","last_peered":"2024-03-28T16:01:09.543775+0000","last_clean":"2024-03-28T16:01:09.543775+0000","last_became_active":"2024-03-28T16:01:08.274699+0000","last_became_peered":"2024-03-28T16:01:08.274699+0000","last_unstale":"2024-03-28T16:01:09.543775+0000","last_undegraded":"2024-03-28T16:01:09.543775+0000","last_fullsized":"2024-03-28T16:01:09.543775+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:43:02.970898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.250349+0000","last_change":"2024-03-28T16:01:08.274943+0000","last_active":"2024-03-28T16:01:09.250349+0000","last_peered":"2024-03-28T16:01:09.250349+0000","last_clean":"2024-03-28T16:01:09.250349+0000","last_became_active":"2024-03-28T16:01:08.274267+0000","last_became_peered":"2024-03-28T16:01:08.274267+0000","last_unstale":"2024-03-28T16:01:09.250349+0000","last_undegraded":"2024-03-28T16:01:09.250349+0000","last_fullsized":"2024-03-28T16:01:09.250349+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:41:13.979918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.543491+0000","last_change":"2024-03-28T16:01:08.266817+0000","last_active":"2024-03-28T16:01:09.543491+0000","last_peered":"2024-03-28T16:01:09.543491+0000","last_clean":"2024-03-28T16:01:09.543491+0000","last_became_active":"2024-03-28T16:01:08.266470+0000","last_became_peered":"2024-03-28T16:01:08.266470+0000","last_unstale":"2024-03-28T16:01:09.543491+0000","last_undegraded":"2024-03-28T16:01:09.543491+0000","last_fullsized":"2024-03-28T16:01:09.543491+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:15:28.573153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.245403+0000","last_change":"2024-03-28T16:01:08.269896+0000","last_active":"2024-03-28T16:01:09.245403+0000","last_peered":"2024-03-28T16:01:09.245403+0000","last_clean":"2024-03-28T16:01:09.245403+0000","last_became_active":"2024-03-28T16:01:08.269500+0000","last_became_peered":"2024-03-28T16:01:08.269500+0000","last_unstale":"2024-03-28T16:01:09.245403+0000","last_undegraded":"2024-03-28T16:01:09.245403+0000","last_fullsized":"2024-03-28T16:01:09.245403+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:44:38.867590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.542051+0000","last_change":"2024-03-28T16:01:08.267072+0000","last_active":"2024-03-28T16:01:09.542051+0000","last_peered":"2024-03-28T16:01:09.542051+0000","last_clean":"2024-03-28T16:01:09.542051+0000","last_became_active":"2024-03-28T16:01:08.266688+0000","last_became_peered":"2024-03-28T16:01:08.266688+0000","last_unstale":"2024-03-28T16:01:09.542051+0000","last_undegraded":"2024-03-28T16:01:09.542051+0000","last_fullsized":"2024-03-28T16:01:09.542051+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:07:57.702952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.542001+0000","last_change":"2024-03-28T16:01:08.270912+0000","last_active":"2024-03-28T16:01:09.542001+0000","last_peered":"2024-03-28T16:01:09.542001+0000","last_clean":"2024-03-28T16:01:09.542001+0000","last_became_active":"2024-03-28T16:01:08.270567+0000","last_became_peered":"2024-03-28T16:01:08.270567+0000","last_unstale":"2024-03-28T16:01:09.542001+0000","last_undegraded":"2024-03-28T16:01:09.542001+0000","last_fullsized":"2024-03-28T16:01:09.542001+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:19:29.294502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.542451+0000","last_change":"2024-03-28T16:01:08.266811+0000","last_active":"2024-03-28T16:01:09.542451+0000","last_peered":"2024-03-28T16:01:09.542451+0000","last_clean":"2024-03-28T16:01:09.542451+0000","last_became_active":"2024-03-28T16:01:08.266517+0000","last_became_peered":"2024-03-28T16:01:08.266517+0000","last_unstale":"2024-03-28T16:01:09.542451+0000","last_undegraded":"2024-03-28T16:01:09.542451+0000","last_fullsized":"2024-03-28T16:01:09.542451+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:55:05.972683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:10.304157+0000","last_change":"2024-03-28T16:01:08.263864+0000","last_active":"2024-03-28T16:01:10.304157+0000","last_peered":"2024-03-28T16:01:10.304157+0000","last_clean":"2024-03-28T16:01:10.304157+0000","last_became_active":"2024-03-28T16:01:08.262956+0000","last_became_peered":"2024-03-28T16:01:08.262956+0000","last_unstale":"2024-03-28T16:01:10.304157+0000","last_undegraded":"2024-03-28T16:01:10.304157+0000","last_fullsized":"2024-03-28T16:01:10.304157+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:05:50.504237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.291828+0000","last_change":"2024-03-28T16:01:08.265151+0000","last_active":"2024-03-28T16:01:09.291828+0000","last_peered":"2024-03-28T16:01:09.291828+0000","last_clean":"2024-03-28T16:01:09.291828+0000","last_became_active":"2024-03-28T16:01:08.264861+0000","last_became_peered":"2024-03-28T16:01:08.264861+0000","last_unstale":"2024-03-28T16:01:09.291828+0000","last_undegraded":"2024-03-28T16:01:09.291828+0000","last_fullsized":"2024-03-28T16:01:09.291828+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:35:45.878983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.544112+0000","last_change":"2024-03-28T16:01:08.265993+0000","last_active":"2024-03-28T16:01:09.544112+0000","last_peered":"2024-03-28T16:01:09.544112+0000","last_clean":"2024-03-28T16:01:09.544112+0000","last_became_active":"2024-03-28T16:01:08.265488+0000","last_became_peered":"2024-03-28T16:01:08.265488+0000","last_unstale":"2024-03-28T16:01:09.544112+0000","last_undegraded":"2024-03-28T16:01:09.544112+0000","last_fullsized":"2024-03-28T16:01:09.544112+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:31:55.920705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'4","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.291637+0000","last_change":"2024-03-28T16:01:07.250361+0000","last_active":"2024-03-28T16:01:09.291637+0000","last_peered":"2024-03-28T16:01:09.291637+0000","last_clean":"2024-03-28T16:01:09.291637+0000","last_became_active":"2024-03-28T16:01:07.250004+0000","last_became_peered":"2024-03-28T16:01:07.250004+0000","last_unstale":"2024-03-28T16:01:09.291637+0000","last_undegraded":"2024-03-28T16:01:09.291637+0000","last_fullsized":"2024-03-28T16:01:09.291637+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:10:32.838716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.296283+0000","last_change":"2024-03-28T16:01:08.274436+0000","last_active":"2024-03-28T16:01:09.296283+0000","last_peered":"2024-03-28T16:01:09.296283+0000","last_clean":"2024-03-28T16:01:09.296283+0000","last_became_active":"2024-03-28T16:01:08.274124+0000","last_became_peered":"2024-03-28T16:01:08.274124+0000","last_unstale":"2024-03-28T16:01:09.296283+0000","last_undegraded":"2024-03-28T16:01:09.296283+0000","last_fullsized":"2024-03-28T16:01:09.296283+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:02:34.037686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"89'1","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.296211+0000","last_change":"2024-03-28T16:01:07.248565+0000","last_active":"2024-03-28T16:01:09.296211+0000","last_peered":"2024-03-28T16:01:09.296211+0000","last_clean":"2024-03-28T16:01:09.296211+0000","last_became_active":"2024-03-28T16:01:07.247366+0000","last_became_peered":"2024-03-28T16:01:07.247366+0000","last_unstale":"2024-03-28T16:01:09.296211+0000","last_undegraded":"2024-03-28T16:01:09.296211+0000","last_fullsized":"2024-03-28T16:01:09.296211+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:48:13.146398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.542626+0000","last_change":"2024-03-28T16:01:08.265639+0000","last_active":"2024-03-28T16:01:09.542626+0000","last_peered":"2024-03-28T16:01:09.542626+0000","last_clean":"2024-03-28T16:01:09.542626+0000","last_became_active":"2024-03-28T16:01:08.263488+0000","last_became_peered":"2024-03-28T16:01:08.263488+0000","last_unstale":"2024-03-28T16:01:09.542626+0000","last_undegraded":"2024-03-28T16:01:09.542626+0000","last_fullsized":"2024-03-28T16:01:09.542626+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:15:06.516016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"89'2","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.297382+0000","last_change":"2024-03-28T16:01:07.258327+0000","last_active":"2024-03-28T16:01:09.297382+0000","last_peered":"2024-03-28T16:01:09.297382+0000","last_clean":"2024-03-28T16:01:09.297382+0000","last_became_active":"2024-03-28T16:01:07.257881+0000","last_became_peered":"2024-03-28T16:01:07.257881+0000","last_unstale":"2024-03-28T16:01:09.297382+0000","last_undegraded":"2024-03-28T16:01:09.297382+0000","last_fullsized":"2024-03-28T16:01:09.297382+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:14:05.453004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.544031+0000","last_change":"2024-03-28T16:01:08.271534+0000","last_active":"2024-03-28T16:01:09.544031+0000","last_peered":"2024-03-28T16:01:09.544031+0000","last_clean":"2024-03-28T16:01:09.544031+0000","last_became_active":"2024-03-28T16:01:08.271104+0000","last_became_peered":"2024-03-28T16:01:08.271104+0000","last_unstale":"2024-03-28T16:01:09.544031+0000","last_undegraded":"2024-03-28T16:01:09.544031+0000","last_fullsized":"2024-03-28T16:01:09.544031+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:29:21.332346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"89'2","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.297533+0000","last_change":"2024-03-28T16:01:07.259175+0000","last_active":"2024-03-28T16:01:09.297533+0000","last_peered":"2024-03-28T16:01:09.297533+0000","last_clean":"2024-03-28T16:01:09.297533+0000","last_became_active":"2024-03-28T16:01:07.258151+0000","last_became_peered":"2024-03-28T16:01:07.258151+0000","last_unstale":"2024-03-28T16:01:09.297533+0000","last_undegraded":"2024-03-28T16:01:09.297533+0000","last_fullsized":"2024-03-28T16:01:09.297533+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:41:07.064800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.250343+0000","last_change":"2024-03-28T16:01:08.277406+0000","last_active":"2024-03-28T16:01:09.250343+0000","last_peered":"2024-03-28T16:01:09.250343+0000","last_clean":"2024-03-28T16:01:09.250343+0000","last_became_active":"2024-03-28T16:01:08.274653+0000","last_became_peered":"2024-03-28T16:01:08.274653+0000","last_unstale":"2024-03-28T16:01:09.250343+0000","last_undegraded":"2024-03-28T16:01:09.250343+0000","last_fullsized":"2024-03-28T16:01:09.250343+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:59:44.306616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"89'1","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.249280+0000","last_change":"2024-03-28T16:01:07.248538+0000","last_active":"2024-03-28T16:01:09.249280+0000","last_peered":"2024-03-28T16:01:09.249280+0000","last_clean":"2024-03-28T16:01:09.249280+0000","last_became_active":"2024-03-28T16:01:07.248116+0000","last_became_peered":"2024-03-28T16:01:07.248116+0000","last_unstale":"2024-03-28T16:01:09.249280+0000","last_undegraded":"2024-03-28T16:01:09.249280+0000","last_fullsized":"2024-03-28T16:01:09.249280+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:44:56.071286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.296884+0000","last_change":"2024-03-28T16:01:08.271059+0000","last_active":"2024-03-28T16:01:09.296884+0000","last_peered":"2024-03-28T16:01:09.296884+0000","last_clean":"2024-03-28T16:01:09.296884+0000","last_became_active":"2024-03-28T16:01:08.270682+0000","last_became_peered":"2024-03-28T16:01:08.270682+0000","last_unstale":"2024-03-28T16:01:09.296884+0000","last_undegraded":"2024-03-28T16:01:09.296884+0000","last_fullsized":"2024-03-28T16:01:09.296884+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:54:48.922125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"89'1","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.543879+0000","last_change":"2024-03-28T16:01:07.261157+0000","last_active":"2024-03-28T16:01:09.543879+0000","last_peered":"2024-03-28T16:01:09.543879+0000","last_clean":"2024-03-28T16:01:09.543879+0000","last_became_active":"2024-03-28T16:01:07.260751+0000","last_became_peered":"2024-03-28T16:01:07.260751+0000","last_unstale":"2024-03-28T16:01:09.543879+0000","last_undegraded":"2024-03-28T16:01:09.543879+0000","last_fullsized":"2024-03-28T16:01:09.543879+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:16:49.622168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.249617+0000","last_change":"2024-03-28T16:01:08.265773+0000","last_active":"2024-03-28T16:01:09.249617+0000","last_peered":"2024-03-28T16:01:09.249617+0000","last_clean":"2024-03-28T16:01:09.249617+0000","last_became_active":"2024-03-28T16:01:08.263724+0000","last_became_peered":"2024-03-28T16:01:08.263724+0000","last_unstale":"2024-03-28T16:01:09.249617+0000","last_undegraded":"2024-03-28T16:01:09.249617+0000","last_fullsized":"2024-03-28T16:01:09.249617+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:42:37.338946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.245061+0000","last_change":"2024-03-28T16:01:07.235361+0000","last_active":"2024-03-28T16:01:09.245061+0000","last_peered":"2024-03-28T16:01:09.245061+0000","last_clean":"2024-03-28T16:01:09.245061+0000","last_became_active":"2024-03-28T16:01:07.234969+0000","last_became_peered":"2024-03-28T16:01:07.234969+0000","last_unstale":"2024-03-28T16:01:09.245061+0000","last_undegraded":"2024-03-28T16:01:09.245061+0000","last_fullsized":"2024-03-28T16:01:09.245061+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:50:35.592147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.294620+0000","last_change":"2024-03-28T16:01:08.263878+0000","last_active":"2024-03-28T16:01:09.294620+0000","last_peered":"2024-03-28T16:01:09.294620+0000","last_clean":"2024-03-28T16:01:09.294620+0000","last_became_active":"2024-03-28T16:01:08.263549+0000","last_became_peered":"2024-03-28T16:01:08.263549+0000","last_unstale":"2024-03-28T16:01:09.294620+0000","last_undegraded":"2024-03-28T16:01:09.294620+0000","last_fullsized":"2024-03-28T16:01:09.294620+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:31:37.669215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.248824+0000","last_change":"2024-03-28T16:01:07.255701+0000","last_active":"2024-03-28T16:01:09.248824+0000","last_peered":"2024-03-28T16:01:09.248824+0000","last_clean":"2024-03-28T16:01:09.248824+0000","last_became_active":"2024-03-28T16:01:07.255247+0000","last_became_peered":"2024-03-28T16:01:07.255247+0000","last_unstale":"2024-03-28T16:01:09.248824+0000","last_undegraded":"2024-03-28T16:01:09.248824+0000","last_fullsized":"2024-03-28T16:01:09.248824+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:08:17.858395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.291383+0000","last_change":"2024-03-28T16:01:08.252949+0000","last_active":"2024-03-28T16:01:09.291383+0000","last_peered":"2024-03-28T16:01:09.291383+0000","last_clean":"2024-03-28T16:01:09.291383+0000","last_became_active":"2024-03-28T16:01:08.252471+0000","last_became_peered":"2024-03-28T16:01:08.252471+0000","last_unstale":"2024-03-28T16:01:09.291383+0000","last_undegraded":"2024-03-28T16:01:09.291383+0000","last_fullsized":"2024-03-28T16:01:09.291383+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:21:15.519020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.543974+0000","last_change":"2024-03-28T16:01:07.263973+0000","last_active":"2024-03-28T16:01:09.543974+0000","last_peered":"2024-03-28T16:01:09.543974+0000","last_clean":"2024-03-28T16:01:09.543974+0000","last_became_active":"2024-03-28T16:01:07.263560+0000","last_became_peered":"2024-03-28T16:01:07.263560+0000","last_unstale":"2024-03-28T16:01:09.543974+0000","last_undegraded":"2024-03-28T16:01:09.543974+0000","last_fullsized":"2024-03-28T16:01:09.543974+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:11.215312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.545319+0000","last_change":"2024-03-28T16:01:08.266122+0000","last_active":"2024-03-28T16:01:09.545319+0000","last_peered":"2024-03-28T16:01:09.545319+0000","last_clean":"2024-03-28T16:01:09.545319+0000","last_became_active":"2024-03-28T16:01:08.265727+0000","last_became_peered":"2024-03-28T16:01:08.265727+0000","last_unstale":"2024-03-28T16:01:09.545319+0000","last_undegraded":"2024-03-28T16:01:09.545319+0000","last_fullsized":"2024-03-28T16:01:09.545319+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:48:38.939777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.250386+0000","last_change":"2024-03-28T16:01:07.251814+0000","last_active":"2024-03-28T16:01:09.250386+0000","last_peered":"2024-03-28T16:01:09.250386+0000","last_clean":"2024-03-28T16:01:09.250386+0000","last_became_active":"2024-03-28T16:01:07.251311+0000","last_became_peered":"2024-03-28T16:01:07.251311+0000","last_unstale":"2024-03-28T16:01:09.250386+0000","last_undegraded":"2024-03-28T16:01:09.250386+0000","last_fullsized":"2024-03-28T16:01:09.250386+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:13:38.344070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.542555+0000","last_change":"2024-03-28T16:01:08.270879+0000","last_active":"2024-03-28T16:01:09.542555+0000","last_peered":"2024-03-28T16:01:09.542555+0000","last_clean":"2024-03-28T16:01:09.542555+0000","last_became_active":"2024-03-28T16:01:08.270513+0000","last_became_peered":"2024-03-28T16:01:08.270513+0000","last_unstale":"2024-03-28T16:01:09.542555+0000","last_undegraded":"2024-03-28T16:01:09.542555+0000","last_fullsized":"2024-03-28T16:01:09.542555+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:53:43.285797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"89'3","reported_seq":18,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.250641+0000","last_change":"2024-03-28T16:01:07.253595+0000","last_active":"2024-03-28T16:01:09.250641+0000","last_peered":"2024-03-28T16:01:09.250641+0000","last_clean":"2024-03-28T16:01:09.250641+0000","last_became_active":"2024-03-28T16:01:07.253137+0000","last_became_peered":"2024-03-28T16:01:07.253137+0000","last_unstale":"2024-03-28T16:01:09.250641+0000","last_undegraded":"2024-03-28T16:01:09.250641+0000","last_fullsized":"2024-03-28T16:01:09.250641+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:17:35.759791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.544273+0000","last_change":"2024-03-28T16:01:08.261185+0000","last_active":"2024-03-28T16:01:09.544273+0000","last_peered":"2024-03-28T16:01:09.544273+0000","last_clean":"2024-03-28T16:01:09.544273+0000","last_became_active":"2024-03-28T16:01:08.260885+0000","last_became_peered":"2024-03-28T16:01:08.260885+0000","last_unstale":"2024-03-28T16:01:09.544273+0000","last_undegraded":"2024-03-28T16:01:09.544273+0000","last_fullsized":"2024-03-28T16:01:09.544273+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:15:56.640586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"90'13","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:14.524435+0000","last_change":"2024-03-28T16:01:07.258655+0000","last_active":"2024-03-28T16:01:14.524435+0000","last_peered":"2024-03-28T16:01:14.524435+0000","last_clean":"2024-03-28T16:01:14.524435+0000","last_became_active":"2024-03-28T16:01:07.258230+0000","last_became_peered":"2024-03-28T16:01:07.258230+0000","last_unstale":"2024-03-28T16:01:14.524435+0000","last_undegraded":"2024-03-28T16:01:14.524435+0000","last_fullsized":"2024-03-28T16:01:14.524435+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:29:36.206061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20895,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":26,"num_write":13,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.297144+0000","last_change":"2024-03-28T16:01:08.265393+0000","last_active":"2024-03-28T16:01:09.297144+0000","last_peered":"2024-03-28T16:01:09.297144+0000","last_clean":"2024-03-28T16:01:09.297144+0000","last_became_active":"2024-03-28T16:01:08.264560+0000","last_became_peered":"2024-03-28T16:01:08.264560+0000","last_unstale":"2024-03-28T16:01:09.297144+0000","last_undegraded":"2024-03-28T16:01:09.297144+0000","last_fullsized":"2024-03-28T16:01:09.297144+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:43:52.181113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"89'2","reported_seq":18,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.543814+0000","last_change":"2024-03-28T16:01:07.251428+0000","last_active":"2024-03-28T16:01:09.543814+0000","last_peered":"2024-03-28T16:01:09.543814+0000","last_clean":"2024-03-28T16:01:09.543814+0000","last_became_active":"2024-03-28T16:01:07.251074+0000","last_became_peered":"2024-03-28T16:01:07.251074+0000","last_unstale":"2024-03-28T16:01:09.543814+0000","last_undegraded":"2024-03-28T16:01:09.543814+0000","last_fullsized":"2024-03-28T16:01:09.543814+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:18:41.359338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.245074+0000","last_change":"2024-03-28T16:01:08.268482+0000","last_active":"2024-03-28T16:01:09.245074+0000","last_peered":"2024-03-28T16:01:09.245074+0000","last_clean":"2024-03-28T16:01:09.245074+0000","last_became_active":"2024-03-28T16:01:08.268153+0000","last_became_peered":"2024-03-28T16:01:08.268153+0000","last_unstale":"2024-03-28T16:01:09.245074+0000","last_undegraded":"2024-03-28T16:01:09.245074+0000","last_fullsized":"2024-03-28T16:01:09.245074+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:21:21.874773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.248812+0000","last_change":"2024-03-28T16:01:07.243506+0000","last_active":"2024-03-28T16:01:09.248812+0000","last_peered":"2024-03-28T16:01:09.248812+0000","last_clean":"2024-03-28T16:01:09.248812+0000","last_became_active":"2024-03-28T16:01:07.242793+0000","last_became_peered":"2024-03-28T16:01:07.242793+0000","last_unstale":"2024-03-28T16:01:09.248812+0000","last_undegraded":"2024-03-28T16:01:09.248812+0000","last_fullsized":"2024-03-28T16:01:09.248812+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:31:56.497276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.250571+0000","last_change":"2024-03-28T16:01:08.278848+0000","last_active":"2024-03-28T16:01:09.250571+0000","last_peered":"2024-03-28T16:01:09.250571+0000","last_clean":"2024-03-28T16:01:09.250571+0000","last_became_active":"2024-03-28T16:01:08.278455+0000","last_became_peered":"2024-03-28T16:01:08.278455+0000","last_unstale":"2024-03-28T16:01:09.250571+0000","last_undegraded":"2024-03-28T16:01:09.250571+0000","last_fullsized":"2024-03-28T16:01:09.250571+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:41:17.302538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.542650+0000","last_change":"2024-03-28T16:01:07.255070+0000","last_active":"2024-03-28T16:01:09.542650+0000","last_peered":"2024-03-28T16:01:09.542650+0000","last_clean":"2024-03-28T16:01:09.542650+0000","last_became_active":"2024-03-28T16:01:07.253489+0000","last_became_peered":"2024-03-28T16:01:07.253489+0000","last_unstale":"2024-03-28T16:01:09.542650+0000","last_undegraded":"2024-03-28T16:01:09.542650+0000","last_fullsized":"2024-03-28T16:01:09.542650+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:31:57.501721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.249148+0000","last_change":"2024-03-28T16:01:08.273190+0000","last_active":"2024-03-28T16:01:09.249148+0000","last_peered":"2024-03-28T16:01:09.249148+0000","last_clean":"2024-03-28T16:01:09.249148+0000","last_became_active":"2024-03-28T16:01:08.272870+0000","last_became_peered":"2024-03-28T16:01:08.272870+0000","last_unstale":"2024-03-28T16:01:09.249148+0000","last_undegraded":"2024-03-28T16:01:09.249148+0000","last_fullsized":"2024-03-28T16:01:09.249148+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:26:50.874750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"89'8","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.296383+0000","last_change":"2024-03-28T16:01:07.257210+0000","last_active":"2024-03-28T16:01:09.296383+0000","last_peered":"2024-03-28T16:01:09.296383+0000","last_clean":"2024-03-28T16:01:09.296383+0000","last_became_active":"2024-03-28T16:01:07.255300+0000","last_became_peered":"2024-03-28T16:01:07.255300+0000","last_unstale":"2024-03-28T16:01:09.296383+0000","last_undegraded":"2024-03-28T16:01:09.296383+0000","last_fullsized":"2024-03-28T16:01:09.296383+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:38:37.761030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.249110+0000","last_change":"2024-03-28T16:01:08.269849+0000","last_active":"2024-03-28T16:01:09.249110+0000","last_peered":"2024-03-28T16:01:09.249110+0000","last_clean":"2024-03-28T16:01:09.249110+0000","last_became_active":"2024-03-28T16:01:08.269506+0000","last_became_peered":"2024-03-28T16:01:08.269506+0000","last_unstale":"2024-03-28T16:01:09.249110+0000","last_undegraded":"2024-03-28T16:01:09.249110+0000","last_fullsized":"2024-03-28T16:01:09.249110+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:04:01.708192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'1","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.697061+0000","last_change":"2024-03-28T16:01:07.254115+0000","last_active":"2024-03-28T16:01:09.697061+0000","last_peered":"2024-03-28T16:01:09.697061+0000","last_clean":"2024-03-28T16:01:09.697061+0000","last_became_active":"2024-03-28T16:01:07.253664+0000","last_became_peered":"2024-03-28T16:01:07.253664+0000","last_unstale":"2024-03-28T16:01:09.697061+0000","last_undegraded":"2024-03-28T16:01:09.697061+0000","last_fullsized":"2024-03-28T16:01:09.697061+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:19:32.386856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:10.303729+0000","last_change":"2024-03-28T16:01:08.274436+0000","last_active":"2024-03-28T16:01:10.303729+0000","last_peered":"2024-03-28T16:01:10.303729+0000","last_clean":"2024-03-28T16:01:10.303729+0000","last_became_active":"2024-03-28T16:01:08.274081+0000","last_became_peered":"2024-03-28T16:01:08.274081+0000","last_unstale":"2024-03-28T16:01:10.303729+0000","last_undegraded":"2024-03-28T16:01:10.303729+0000","last_fullsized":"2024-03-28T16:01:10.303729+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:57:00.531064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.291704+0000","last_change":"2024-03-28T16:01:07.247542+0000","last_active":"2024-03-28T16:01:09.291704+0000","last_peered":"2024-03-28T16:01:09.291704+0000","last_clean":"2024-03-28T16:01:09.291704+0000","last_became_active":"2024-03-28T16:01:07.247109+0000","last_became_peered":"2024-03-28T16:01:07.247109+0000","last_unstale":"2024-03-28T16:01:09.291704+0000","last_undegraded":"2024-03-28T16:01:09.291704+0000","last_fullsized":"2024-03-28T16:01:09.291704+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:14:19.387896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"90'1","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.557549+0000","last_change":"2024-03-28T16:01:08.264565+0000","last_active":"2024-03-28T16:01:09.557549+0000","last_peered":"2024-03-28T16:01:09.557549+0000","last_clean":"2024-03-28T16:01:09.557549+0000","last_became_active":"2024-03-28T16:01:08.264196+0000","last_became_peered":"2024-03-28T16:01:08.264196+0000","last_unstale":"2024-03-28T16:01:09.557549+0000","last_undegraded":"2024-03-28T16:01:09.557549+0000","last_fullsized":"2024-03-28T16:01:09.557549+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:43:24.750624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.249483+0000","last_change":"2024-03-28T16:01:07.248189+0000","last_active":"2024-03-28T16:01:09.249483+0000","last_peered":"2024-03-28T16:01:09.249483+0000","last_clean":"2024-03-28T16:01:09.249483+0000","last_became_active":"2024-03-28T16:01:07.247295+0000","last_became_peered":"2024-03-28T16:01:07.247295+0000","last_unstale":"2024-03-28T16:01:09.249483+0000","last_undegraded":"2024-03-28T16:01:09.249483+0000","last_fullsized":"2024-03-28T16:01:09.249483+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:46:51.875094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.542906+0000","last_change":"2024-03-28T16:01:08.269724+0000","last_active":"2024-03-28T16:01:09.542906+0000","last_peered":"2024-03-28T16:01:09.542906+0000","last_clean":"2024-03-28T16:01:09.542906+0000","last_became_active":"2024-03-28T16:01:08.269357+0000","last_became_peered":"2024-03-28T16:01:08.269357+0000","last_unstale":"2024-03-28T16:01:09.542906+0000","last_undegraded":"2024-03-28T16:01:09.542906+0000","last_fullsized":"2024-03-28T16:01:09.542906+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:20:12.622238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.250166+0000","last_change":"2024-03-28T16:01:08.266624+0000","last_active":"2024-03-28T16:01:09.250166+0000","last_peered":"2024-03-28T16:01:09.250166+0000","last_clean":"2024-03-28T16:01:09.250166+0000","last_became_active":"2024-03-28T16:01:08.265408+0000","last_became_peered":"2024-03-28T16:01:08.265408+0000","last_unstale":"2024-03-28T16:01:09.250166+0000","last_undegraded":"2024-03-28T16:01:09.250166+0000","last_fullsized":"2024-03-28T16:01:09.250166+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:37:08.831727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.244764+0000","last_change":"2024-03-28T16:01:07.241172+0000","last_active":"2024-03-28T16:01:09.244764+0000","last_peered":"2024-03-28T16:01:09.244764+0000","last_clean":"2024-03-28T16:01:09.244764+0000","last_became_active":"2024-03-28T16:01:07.240754+0000","last_became_peered":"2024-03-28T16:01:07.240754+0000","last_unstale":"2024-03-28T16:01:09.244764+0000","last_undegraded":"2024-03-28T16:01:09.244764+0000","last_fullsized":"2024-03-28T16:01:09.244764+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:52:45.805188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.295213+0000","last_change":"2024-03-28T16:01:08.278429+0000","last_active":"2024-03-28T16:01:09.295213+0000","last_peered":"2024-03-28T16:01:09.295213+0000","last_clean":"2024-03-28T16:01:09.295213+0000","last_became_active":"2024-03-28T16:01:08.275061+0000","last_became_peered":"2024-03-28T16:01:08.275061+0000","last_unstale":"2024-03-28T16:01:09.295213+0000","last_undegraded":"2024-03-28T16:01:09.295213+0000","last_fullsized":"2024-03-28T16:01:09.295213+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:35:42.015979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.544719+0000","last_change":"2024-03-28T16:01:07.252117+0000","last_active":"2024-03-28T16:01:09.544719+0000","last_peered":"2024-03-28T16:01:09.544719+0000","last_clean":"2024-03-28T16:01:09.544719+0000","last_became_active":"2024-03-28T16:01:07.249729+0000","last_became_peered":"2024-03-28T16:01:07.249729+0000","last_unstale":"2024-03-28T16:01:09.544719+0000","last_undegraded":"2024-03-28T16:01:09.544719+0000","last_fullsized":"2024-03-28T16:01:09.544719+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:33:32.719310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":342,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.248908+0000","last_change":"2024-03-28T15:59:32.593730+0000","last_active":"2024-03-28T16:01:09.248908+0000","last_peered":"2024-03-28T16:01:09.248908+0000","last_clean":"2024-03-28T16:01:09.248908+0000","last_became_active":"2024-03-28T15:59:32.592289+0000","last_became_peered":"2024-03-28T15:59:32.592289+0000","last_unstale":"2024-03-28T16:01:09.248908+0000","last_undegraded":"2024-03-28T16:01:09.248908+0000","last_fullsized":"2024-03-28T16:01:09.248908+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:54:55.638193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:54:55.638193+0000","last_clean_scrub_stamp":"2024-03-28T15:54:55.638193+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:20:37.087340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.244882+0000","last_change":"2024-03-28T16:01:08.265513+0000","last_active":"2024-03-28T16:01:09.244882+0000","last_peered":"2024-03-28T16:01:09.244882+0000","last_clean":"2024-03-28T16:01:09.244882+0000","last_became_active":"2024-03-28T16:01:08.264711+0000","last_became_peered":"2024-03-28T16:01:08.264711+0000","last_unstale":"2024-03-28T16:01:09.244882+0000","last_undegraded":"2024-03-28T16:01:09.244882+0000","last_fullsized":"2024-03-28T16:01:09.244882+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:10:01.936780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.249515+0000","last_change":"2024-03-28T16:01:07.252831+0000","last_active":"2024-03-28T16:01:09.249515+0000","last_peered":"2024-03-28T16:01:09.249515+0000","last_clean":"2024-03-28T16:01:09.249515+0000","last_became_active":"2024-03-28T16:01:07.252317+0000","last_became_peered":"2024-03-28T16:01:07.252317+0000","last_unstale":"2024-03-28T16:01:09.249515+0000","last_undegraded":"2024-03-28T16:01:09.249515+0000","last_fullsized":"2024-03-28T16:01:09.249515+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:55:37.461154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.296977+0000","last_change":"2024-03-28T16:01:07.262413+0000","last_active":"2024-03-28T16:01:09.296977+0000","last_peered":"2024-03-28T16:01:09.296977+0000","last_clean":"2024-03-28T16:01:09.296977+0000","last_became_active":"2024-03-28T16:01:07.261875+0000","last_became_peered":"2024-03-28T16:01:07.261875+0000","last_unstale":"2024-03-28T16:01:09.296977+0000","last_undegraded":"2024-03-28T16:01:09.296977+0000","last_fullsized":"2024-03-28T16:01:09.296977+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:25:49.124991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.245549+0000","last_change":"2024-03-28T16:01:08.259506+0000","last_active":"2024-03-28T16:01:09.245549+0000","last_peered":"2024-03-28T16:01:09.245549+0000","last_clean":"2024-03-28T16:01:09.245549+0000","last_became_active":"2024-03-28T16:01:08.258512+0000","last_became_peered":"2024-03-28T16:01:08.258512+0000","last_unstale":"2024-03-28T16:01:09.245549+0000","last_undegraded":"2024-03-28T16:01:09.245549+0000","last_fullsized":"2024-03-28T16:01:09.245549+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:20:17.017763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"90'1","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.291666+0000","last_change":"2024-03-28T16:01:07.254850+0000","last_active":"2024-03-28T16:01:09.291666+0000","last_peered":"2024-03-28T16:01:09.291666+0000","last_clean":"2024-03-28T16:01:09.291666+0000","last_became_active":"2024-03-28T16:01:07.252294+0000","last_became_peered":"2024-03-28T16:01:07.252294+0000","last_unstale":"2024-03-28T16:01:09.291666+0000","last_undegraded":"2024-03-28T16:01:09.291666+0000","last_fullsized":"2024-03-28T16:01:09.291666+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:54:26.700118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.296386+0000","last_change":"2024-03-28T16:01:08.267361+0000","last_active":"2024-03-28T16:01:09.296386+0000","last_peered":"2024-03-28T16:01:09.296386+0000","last_clean":"2024-03-28T16:01:09.296386+0000","last_became_active":"2024-03-28T16:01:08.267016+0000","last_became_peered":"2024-03-28T16:01:08.267016+0000","last_unstale":"2024-03-28T16:01:09.296386+0000","last_undegraded":"2024-03-28T16:01:09.296386+0000","last_fullsized":"2024-03-28T16:01:09.296386+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:16:07.202938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"89'2","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.542976+0000","last_change":"2024-03-28T16:01:07.245292+0000","last_active":"2024-03-28T16:01:09.542976+0000","last_peered":"2024-03-28T16:01:09.542976+0000","last_clean":"2024-03-28T16:01:09.542976+0000","last_became_active":"2024-03-28T16:01:07.244654+0000","last_became_peered":"2024-03-28T16:01:07.244654+0000","last_unstale":"2024-03-28T16:01:09.542976+0000","last_undegraded":"2024-03-28T16:01:09.542976+0000","last_fullsized":"2024-03-28T16:01:09.542976+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:03:39.027844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.542819+0000","last_change":"2024-03-28T16:01:08.274542+0000","last_active":"2024-03-28T16:01:09.542819+0000","last_peered":"2024-03-28T16:01:09.542819+0000","last_clean":"2024-03-28T16:01:09.542819+0000","last_became_active":"2024-03-28T16:01:08.274202+0000","last_became_peered":"2024-03-28T16:01:08.274202+0000","last_unstale":"2024-03-28T16:01:09.542819+0000","last_undegraded":"2024-03-28T16:01:09.542819+0000","last_fullsized":"2024-03-28T16:01:09.542819+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:09:18.199017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.542507+0000","last_change":"2024-03-28T16:01:07.253274+0000","last_active":"2024-03-28T16:01:09.542507+0000","last_peered":"2024-03-28T16:01:09.542507+0000","last_clean":"2024-03-28T16:01:09.542507+0000","last_became_active":"2024-03-28T16:01:07.250230+0000","last_became_peered":"2024-03-28T16:01:07.250230+0000","last_unstale":"2024-03-28T16:01:09.542507+0000","last_undegraded":"2024-03-28T16:01:09.542507+0000","last_fullsized":"2024-03-28T16:01:09.542507+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:20:31.459983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.296498+0000","last_change":"2024-03-28T16:01:08.266551+0000","last_active":"2024-03-28T16:01:09.296498+0000","last_peered":"2024-03-28T16:01:09.296498+0000","last_clean":"2024-03-28T16:01:09.296498+0000","last_became_active":"2024-03-28T16:01:08.266226+0000","last_became_peered":"2024-03-28T16:01:08.266226+0000","last_unstale":"2024-03-28T16:01:09.296498+0000","last_undegraded":"2024-03-28T16:01:09.296498+0000","last_fullsized":"2024-03-28T16:01:09.296498+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:51:16.520285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.294774+0000","last_change":"2024-03-28T16:01:07.252038+0000","last_active":"2024-03-28T16:01:09.294774+0000","last_peered":"2024-03-28T16:01:09.294774+0000","last_clean":"2024-03-28T16:01:09.294774+0000","last_became_active":"2024-03-28T16:01:07.251703+0000","last_became_peered":"2024-03-28T16:01:07.251703+0000","last_unstale":"2024-03-28T16:01:09.294774+0000","last_undegraded":"2024-03-28T16:01:09.294774+0000","last_fullsized":"2024-03-28T16:01:09.294774+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:35:42.518996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:10.303337+0000","last_change":"2024-03-28T16:01:08.260025+0000","last_active":"2024-03-28T16:01:10.303337+0000","last_peered":"2024-03-28T16:01:10.303337+0000","last_clean":"2024-03-28T16:01:10.303337+0000","last_became_active":"2024-03-28T16:01:08.259596+0000","last_became_peered":"2024-03-28T16:01:08.259596+0000","last_unstale":"2024-03-28T16:01:10.303337+0000","last_undegraded":"2024-03-28T16:01:10.303337+0000","last_fullsized":"2024-03-28T16:01:10.303337+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:45:29.228188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.543695+0000","last_change":"2024-03-28T16:01:07.259335+0000","last_active":"2024-03-28T16:01:09.543695+0000","last_peered":"2024-03-28T16:01:09.543695+0000","last_clean":"2024-03-28T16:01:09.543695+0000","last_became_active":"2024-03-28T16:01:07.258589+0000","last_became_peered":"2024-03-28T16:01:07.258589+0000","last_unstale":"2024-03-28T16:01:09.543695+0000","last_undegraded":"2024-03-28T16:01:09.543695+0000","last_fullsized":"2024-03-28T16:01:09.543695+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:06:41.742963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.295788+0000","last_change":"2024-03-28T16:01:08.266794+0000","last_active":"2024-03-28T16:01:09.295788+0000","last_peered":"2024-03-28T16:01:09.295788+0000","last_clean":"2024-03-28T16:01:09.295788+0000","last_became_active":"2024-03-28T16:01:08.266502+0000","last_became_peered":"2024-03-28T16:01:08.266502+0000","last_unstale":"2024-03-28T16:01:09.295788+0000","last_undegraded":"2024-03-28T16:01:09.295788+0000","last_fullsized":"2024-03-28T16:01:09.295788+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:17:12.671677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"89'2","reported_seq":18,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.697010+0000","last_change":"2024-03-28T16:01:07.259239+0000","last_active":"2024-03-28T16:01:09.697010+0000","last_peered":"2024-03-28T16:01:09.697010+0000","last_clean":"2024-03-28T16:01:09.697010+0000","last_became_active":"2024-03-28T16:01:07.258875+0000","last_became_peered":"2024-03-28T16:01:07.258875+0000","last_unstale":"2024-03-28T16:01:09.697010+0000","last_undegraded":"2024-03-28T16:01:09.697010+0000","last_fullsized":"2024-03-28T16:01:09.697010+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:35:12.729173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.250187+0000","last_change":"2024-03-28T16:01:08.278196+0000","last_active":"2024-03-28T16:01:09.250187+0000","last_peered":"2024-03-28T16:01:09.250187+0000","last_clean":"2024-03-28T16:01:09.250187+0000","last_became_active":"2024-03-28T16:01:08.276972+0000","last_became_peered":"2024-03-28T16:01:08.276972+0000","last_unstale":"2024-03-28T16:01:09.250187+0000","last_undegraded":"2024-03-28T16:01:09.250187+0000","last_fullsized":"2024-03-28T16:01:09.250187+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:28:25.231210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.544033+0000","last_change":"2024-03-28T16:01:07.246851+0000","last_active":"2024-03-28T16:01:09.544033+0000","last_peered":"2024-03-28T16:01:09.544033+0000","last_clean":"2024-03-28T16:01:09.544033+0000","last_became_active":"2024-03-28T16:01:07.246365+0000","last_became_peered":"2024-03-28T16:01:07.246365+0000","last_unstale":"2024-03-28T16:01:09.544033+0000","last_undegraded":"2024-03-28T16:01:09.544033+0000","last_fullsized":"2024-03-28T16:01:09.544033+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:16:13.593045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.250579+0000","last_change":"2024-03-28T16:01:08.279468+0000","last_active":"2024-03-28T16:01:09.250579+0000","last_peered":"2024-03-28T16:01:09.250579+0000","last_clean":"2024-03-28T16:01:09.250579+0000","last_became_active":"2024-03-28T16:01:08.274650+0000","last_became_peered":"2024-03-28T16:01:08.274650+0000","last_unstale":"2024-03-28T16:01:09.250579+0000","last_undegraded":"2024-03-28T16:01:09.250579+0000","last_fullsized":"2024-03-28T16:01:09.250579+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:36:32.776085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"89'1","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.250047+0000","last_change":"2024-03-28T16:01:07.258077+0000","last_active":"2024-03-28T16:01:09.250047+0000","last_peered":"2024-03-28T16:01:09.250047+0000","last_clean":"2024-03-28T16:01:09.250047+0000","last_became_active":"2024-03-28T16:01:07.256665+0000","last_became_peered":"2024-03-28T16:01:07.256665+0000","last_unstale":"2024-03-28T16:01:09.250047+0000","last_undegraded":"2024-03-28T16:01:09.250047+0000","last_fullsized":"2024-03-28T16:01:09.250047+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:01:18.852441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.250281+0000","last_change":"2024-03-28T16:01:08.278588+0000","last_active":"2024-03-28T16:01:09.250281+0000","last_peered":"2024-03-28T16:01:09.250281+0000","last_clean":"2024-03-28T16:01:09.250281+0000","last_became_active":"2024-03-28T16:01:08.277303+0000","last_became_peered":"2024-03-28T16:01:08.277303+0000","last_unstale":"2024-03-28T16:01:09.250281+0000","last_undegraded":"2024-03-28T16:01:09.250281+0000","last_fullsized":"2024-03-28T16:01:09.250281+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:01:19.927057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"89'2","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.245226+0000","last_change":"2024-03-28T16:01:07.249382+0000","last_active":"2024-03-28T16:01:09.245226+0000","last_peered":"2024-03-28T16:01:09.245226+0000","last_clean":"2024-03-28T16:01:09.245226+0000","last_became_active":"2024-03-28T16:01:07.249065+0000","last_became_peered":"2024-03-28T16:01:07.249065+0000","last_unstale":"2024-03-28T16:01:09.245226+0000","last_undegraded":"2024-03-28T16:01:09.245226+0000","last_fullsized":"2024-03-28T16:01:09.245226+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:07:22.851862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.251277+0000","last_change":"2024-03-28T16:01:08.274839+0000","last_active":"2024-03-28T16:01:09.251277+0000","last_peered":"2024-03-28T16:01:09.251277+0000","last_clean":"2024-03-28T16:01:09.251277+0000","last_became_active":"2024-03-28T16:01:08.274505+0000","last_became_peered":"2024-03-28T16:01:08.274505+0000","last_unstale":"2024-03-28T16:01:09.251277+0000","last_undegraded":"2024-03-28T16:01:09.251277+0000","last_fullsized":"2024-03-28T16:01:09.251277+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:57:37.635327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:10.303497+0000","last_change":"2024-03-28T16:01:08.268207+0000","last_active":"2024-03-28T16:01:10.303497+0000","last_peered":"2024-03-28T16:01:10.303497+0000","last_clean":"2024-03-28T16:01:10.303497+0000","last_became_active":"2024-03-28T16:01:08.267843+0000","last_became_peered":"2024-03-28T16:01:08.267843+0000","last_unstale":"2024-03-28T16:01:10.303497+0000","last_undegraded":"2024-03-28T16:01:10.303497+0000","last_fullsized":"2024-03-28T16:01:10.303497+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:39:05.675279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.543441+0000","last_change":"2024-03-28T16:01:08.272816+0000","last_active":"2024-03-28T16:01:09.543441+0000","last_peered":"2024-03-28T16:01:09.543441+0000","last_clean":"2024-03-28T16:01:09.543441+0000","last_became_active":"2024-03-28T16:01:08.270941+0000","last_became_peered":"2024-03-28T16:01:08.270941+0000","last_unstale":"2024-03-28T16:01:09.543441+0000","last_undegraded":"2024-03-28T16:01:09.543441+0000","last_fullsized":"2024-03-28T16:01:09.543441+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:19:00.481809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.543420+0000","last_change":"2024-03-28T16:01:08.258558+0000","last_active":"2024-03-28T16:01:09.543420+0000","last_peered":"2024-03-28T16:01:09.543420+0000","last_clean":"2024-03-28T16:01:09.543420+0000","last_became_active":"2024-03-28T16:01:08.258163+0000","last_became_peered":"2024-03-28T16:01:08.258163+0000","last_unstale":"2024-03-28T16:01:09.543420+0000","last_undegraded":"2024-03-28T16:01:09.543420+0000","last_fullsized":"2024-03-28T16:01:09.543420+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:53:48.358536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.542309+0000","last_change":"2024-03-28T16:01:08.268886+0000","last_active":"2024-03-28T16:01:09.542309+0000","last_peered":"2024-03-28T16:01:09.542309+0000","last_clean":"2024-03-28T16:01:09.542309+0000","last_became_active":"2024-03-28T16:01:08.268519+0000","last_became_peered":"2024-03-28T16:01:08.268519+0000","last_unstale":"2024-03-28T16:01:09.542309+0000","last_undegraded":"2024-03-28T16:01:09.542309+0000","last_fullsized":"2024-03-28T16:01:09.542309+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:49:05.266517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.245360+0000","last_change":"2024-03-28T16:01:08.265039+0000","last_active":"2024-03-28T16:01:09.245360+0000","last_peered":"2024-03-28T16:01:09.245360+0000","last_clean":"2024-03-28T16:01:09.245360+0000","last_became_active":"2024-03-28T16:01:08.264677+0000","last_became_peered":"2024-03-28T16:01:08.264677+0000","last_unstale":"2024-03-28T16:01:09.245360+0000","last_undegraded":"2024-03-28T16:01:09.245360+0000","last_fullsized":"2024-03-28T16:01:09.245360+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:17:27.667324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:10.303716+0000","last_change":"2024-03-28T16:01:08.271699+0000","last_active":"2024-03-28T16:01:10.303716+0000","last_peered":"2024-03-28T16:01:10.303716+0000","last_clean":"2024-03-28T16:01:10.303716+0000","last_became_active":"2024-03-28T16:01:08.270184+0000","last_became_peered":"2024-03-28T16:01:08.270184+0000","last_unstale":"2024-03-28T16:01:10.303716+0000","last_undegraded":"2024-03-28T16:01:10.303716+0000","last_fullsized":"2024-03-28T16:01:10.303716+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:17:15.699944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.544174+0000","last_change":"2024-03-28T16:01:08.262314+0000","last_active":"2024-03-28T16:01:09.544174+0000","last_peered":"2024-03-28T16:01:09.544174+0000","last_clean":"2024-03-28T16:01:09.544174+0000","last_became_active":"2024-03-28T16:01:08.261992+0000","last_became_peered":"2024-03-28T16:01:08.261992+0000","last_unstale":"2024-03-28T16:01:09.544174+0000","last_undegraded":"2024-03-28T16:01:09.544174+0000","last_fullsized":"2024-03-28T16:01:09.544174+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:40:52.048306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.296233+0000","last_change":"2024-03-28T16:01:08.259407+0000","last_active":"2024-03-28T16:01:09.296233+0000","last_peered":"2024-03-28T16:01:09.296233+0000","last_clean":"2024-03-28T16:01:09.296233+0000","last_became_active":"2024-03-28T16:01:08.259071+0000","last_became_peered":"2024-03-28T16:01:08.259071+0000","last_unstale":"2024-03-28T16:01:09.296233+0000","last_undegraded":"2024-03-28T16:01:09.296233+0000","last_fullsized":"2024-03-28T16:01:09.296233+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:30:17.819868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.249123+0000","last_change":"2024-03-28T16:01:08.271649+0000","last_active":"2024-03-28T16:01:09.249123+0000","last_peered":"2024-03-28T16:01:09.249123+0000","last_clean":"2024-03-28T16:01:09.249123+0000","last_became_active":"2024-03-28T16:01:08.271230+0000","last_became_peered":"2024-03-28T16:01:08.271230+0000","last_unstale":"2024-03-28T16:01:09.249123+0000","last_undegraded":"2024-03-28T16:01:09.249123+0000","last_fullsized":"2024-03-28T16:01:09.249123+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:50:14.404178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.294971+0000","last_change":"2024-03-28T16:01:08.260845+0000","last_active":"2024-03-28T16:01:09.294971+0000","last_peered":"2024-03-28T16:01:09.294971+0000","last_clean":"2024-03-28T16:01:09.294971+0000","last_became_active":"2024-03-28T16:01:08.259997+0000","last_became_peered":"2024-03-28T16:01:08.259997+0000","last_unstale":"2024-03-28T16:01:09.294971+0000","last_undegraded":"2024-03-28T16:01:09.294971+0000","last_fullsized":"2024-03-28T16:01:09.294971+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:08:55.080858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.543516+0000","last_change":"2024-03-28T16:01:08.268881+0000","last_active":"2024-03-28T16:01:09.543516+0000","last_peered":"2024-03-28T16:01:09.543516+0000","last_clean":"2024-03-28T16:01:09.543516+0000","last_became_active":"2024-03-28T16:01:08.268212+0000","last_became_peered":"2024-03-28T16:01:08.268212+0000","last_unstale":"2024-03-28T16:01:09.543516+0000","last_undegraded":"2024-03-28T16:01:09.543516+0000","last_fullsized":"2024-03-28T16:01:09.543516+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:46:22.447763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.245136+0000","last_change":"2024-03-28T16:01:08.265113+0000","last_active":"2024-03-28T16:01:09.245136+0000","last_peered":"2024-03-28T16:01:09.245136+0000","last_clean":"2024-03-28T16:01:09.245136+0000","last_became_active":"2024-03-28T16:01:08.264781+0000","last_became_peered":"2024-03-28T16:01:08.264781+0000","last_unstale":"2024-03-28T16:01:09.245136+0000","last_undegraded":"2024-03-28T16:01:09.245136+0000","last_fullsized":"2024-03-28T16:01:09.245136+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:19:14.720260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:10.303510+0000","last_change":"2024-03-28T16:01:08.274446+0000","last_active":"2024-03-28T16:01:10.303510+0000","last_peered":"2024-03-28T16:01:10.303510+0000","last_clean":"2024-03-28T16:01:10.303510+0000","last_became_active":"2024-03-28T16:01:08.274027+0000","last_became_peered":"2024-03-28T16:01:08.274027+0000","last_unstale":"2024-03-28T16:01:10.303510+0000","last_undegraded":"2024-03-28T16:01:10.303510+0000","last_fullsized":"2024-03-28T16:01:10.303510+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:33:37.395889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":21693,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":18,"num_read_kb":32,"num_write":52,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":45796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46,"ondisk_log_size":46,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":85,"seq":365072220176,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":966656,"data_stored":461709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":77,"seq":330712481816,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32744,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712408,"statfs":{"total":95995035648,"available":95961505792,"internally_reserved":0,"allocated":1605632,"data_stored":1073283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":71,"seq":304942678046,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":970752,"data_stored":462291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":63,"seq":270582939686,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":462313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":55,"seq":236223201324,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36868,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708284,"statfs":{"total":95995035648,"available":95957282816,"internally_reserved":0,"allocated":1564672,"data_stored":1052659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397555,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36288,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708864,"statfs":{"total":95995035648,"available":95957876736,"internally_reserved":0,"allocated":966656,"data_stored":461709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574932,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":978944,"data_stored":463321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738458,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36292,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708860,"statfs":{"total":95995035648,"available":95957872640,"internally_reserved":0,"allocated":970752,"data_stored":461799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411407,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":999424,"data_stored":482638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117063,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36296,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708856,"statfs":{"total":95995035648,"available":95957868544,"internally_reserved":0,"allocated":983040,"data_stored":462047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855424,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":970752,"data_stored":461731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659194,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36884,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708268,"statfs":{"total":95995035648,"available":95957266432,"internally_reserved":0,"allocated":1572864,"data_stored":1052259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":20929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":21089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T16:01:16.663 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T16:01:16.790 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:16 smithi067 ceph-mon[31707]: pgmap v368: 97 pgs: 97 active+clean; 598 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 4.6 KiB/s wr, 9 op/s 2024-03-28T16:01:16.790 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:16 smithi067 ceph-mon[31707]: from='client.14748 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T16:01:16.791 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:16 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/986537402' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:01:16.791 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:16 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/463679785' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:01:16.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:16 smithi150 ceph-mon[34825]: pgmap v368: 97 pgs: 97 active+clean; 598 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 4.6 KiB/s wr, 9 op/s 2024-03-28T16:01:16.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:16 smithi150 ceph-mon[34825]: from='client.14748 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T16:01:16.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:16 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/986537402' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:01:16.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:16 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/463679785' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:01:16.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:16 smithi183 ceph-mon[35580]: pgmap v368: 97 pgs: 97 active+clean; 598 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 4.6 KiB/s wr, 9 op/s 2024-03-28T16:01:16.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:16 smithi183 ceph-mon[35580]: from='client.14748 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T16:01:16.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:16 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/986537402' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:01:16.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:16 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/463679785' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:01:16.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.928+0000 7fd601000640 1 -- 172.21.15.67:0/620477982 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5fc108cf0 msgr2=0x7fd5fc109170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:16.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.928+0000 7fd601000640 1 --2- 172.21.15.67:0/620477982 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5fc108cf0 0x7fd5fc109170 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fd5e4009900 tx=0x7fd5e402bc30 comp rx=0 tx=0).stop 2024-03-28T16:01:16.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.928+0000 7fd601000640 1 -- 172.21.15.67:0/620477982 shutdown_connections 2024-03-28T16:01:16.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.928+0000 7fd601000640 1 --2- 172.21.15.67:0/620477982 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5fc1096b0 0x7fd5fc10ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:16.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.928+0000 7fd601000640 1 --2- 172.21.15.67:0/620477982 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5fc108cf0 0x7fd5fc109170 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:16.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.928+0000 7fd601000640 1 --2- 172.21.15.67:0/620477982 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd5fc107b00 0x7fd5fc107f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:16.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.928+0000 7fd601000640 1 -- 172.21.15.67:0/620477982 >> 172.21.15.67:0/620477982 conn(0x7fd5fc076ab0 msgr2=0x7fd5fc078ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:16.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.928+0000 7fd601000640 1 -- 172.21.15.67:0/620477982 shutdown_connections 2024-03-28T16:01:16.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.928+0000 7fd601000640 1 -- 172.21.15.67:0/620477982 wait complete. 2024-03-28T16:01:16.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.929+0000 7fd601000640 1 Processor -- start 2024-03-28T16:01:16.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.930+0000 7fd601000640 1 -- start start 2024-03-28T16:01:16.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.930+0000 7fd601000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd5fc107b00 0x7fd5fc19dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:16.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.930+0000 7fd601000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5fc108cf0 0x7fd5fc19e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:16.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.930+0000 7fd601000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5fc1096b0 0x7fd5fc1a5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:16.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.930+0000 7fd601000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5fc113660 con 0x7fd5fc1096b0 2024-03-28T16:01:16.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.930+0000 7fd601000640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5fc1134e0 con 0x7fd5fc108cf0 2024-03-28T16:01:16.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.930+0000 7fd601000640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5fc1137e0 con 0x7fd5fc107b00 2024-03-28T16:01:16.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.930+0000 7fd5fbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5fc1096b0 0x7fd5fc1a5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:16.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.930+0000 7fd5fbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5fc1096b0 0x7fd5fc1a5290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41376/0 (socket says 172.21.15.67:41376) 2024-03-28T16:01:16.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.930+0000 7fd5fbe00640 1 -- 172.21.15.67:0/4279555629 learned_addr learned my addr 172.21.15.67:0/4279555629 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:01:16.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.931+0000 7fd5f2a00640 1 --2- 172.21.15.67:0/4279555629 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5fc108cf0 0x7fd5fc19e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:16.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.931+0000 7fd5fb400640 1 --2- 172.21.15.67:0/4279555629 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd5fc107b00 0x7fd5fc19dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:16.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.931+0000 7fd5fbe00640 1 -- 172.21.15.67:0/4279555629 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd5fc107b00 msgr2=0x7fd5fc19dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:16.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.931+0000 7fd5fbe00640 1 --2- 172.21.15.67:0/4279555629 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd5fc107b00 0x7fd5fc19dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:16.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.931+0000 7fd5fbe00640 1 -- 172.21.15.67:0/4279555629 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5fc108cf0 msgr2=0x7fd5fc19e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:16.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.931+0000 7fd5fbe00640 1 --2- 172.21.15.67:0/4279555629 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5fc108cf0 0x7fd5fc19e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:16.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.931+0000 7fd5fbe00640 1 -- 172.21.15.67:0/4279555629 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5fc1a5930 con 0x7fd5fc1096b0 2024-03-28T16:01:16.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.931+0000 7fd5fbe00640 1 --2- 172.21.15.67:0/4279555629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5fc1096b0 0x7fd5fc1a5290 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fd5f400eaa0 tx=0x7fd5f400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:16.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.931+0000 7fd5f8c00640 1 -- 172.21.15.67:0/4279555629 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5f400cdb0 con 0x7fd5fc1096b0 2024-03-28T16:01:16.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.931+0000 7fd5f8c00640 1 -- 172.21.15.67:0/4279555629 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd5f4005020 con 0x7fd5fc1096b0 2024-03-28T16:01:16.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.931+0000 7fd601000640 1 -- 172.21.15.67:0/4279555629 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5fc1a5c20 con 0x7fd5fc1096b0 2024-03-28T16:01:16.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.931+0000 7fd5f8c00640 1 -- 172.21.15.67:0/4279555629 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5f40116d0 con 0x7fd5fc1096b0 2024-03-28T16:01:16.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.932+0000 7fd601000640 1 -- 172.21.15.67:0/4279555629 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5fc1a6170 con 0x7fd5fc1096b0 2024-03-28T16:01:16.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.933+0000 7fd601000640 1 -- 172.21.15.67:0/4279555629 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5fc10ee20 con 0x7fd5fc1096b0 2024-03-28T16:01:16.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.935+0000 7fd5f8c00640 1 -- 172.21.15.67:0/4279555629 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fd5f40051d0 con 0x7fd5fc1096b0 2024-03-28T16:01:16.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.936+0000 7fd5f8c00640 1 --2- 172.21.15.67:0/4279555629 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fd5e00a2910 0x7fd5e00a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:16.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.936+0000 7fd5f8c00640 1 -- 172.21.15.67:0/4279555629 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fd5f40cee20 con 0x7fd5fc1096b0 2024-03-28T16:01:16.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.936+0000 7fd5fb400640 1 --2- 172.21.15.67:0/4279555629 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fd5e00a2910 0x7fd5e00a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:16.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.937+0000 7fd5fb400640 1 --2- 172.21.15.67:0/4279555629 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fd5e00a2910 0x7fd5e00a4dd0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec005fd0 tx=0x7fd5ec005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:16.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:16.949+0000 7fd5f8c00640 1 -- 172.21.15.67:0/4279555629 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd5f4016030 con 0x7fd5fc1096b0 2024-03-28T16:01:17.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:17.147+0000 7fd601000640 1 -- 172.21.15.67:0/4279555629 --> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd5fc061ae0 con 0x7fd5e00a2910 2024-03-28T16:01:17.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:17.169+0000 7fd5f8c00640 1 -- 172.21.15.67:0/4279555629 <== mgr.14152 v2:172.21.15.67:6800/208726654 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+258747 (secure 0 0 0) 0x7fd5fc061ae0 con 0x7fd5e00a2910 2024-03-28T16:01:17.173 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:01:17.176 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-28T16:01:17.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:17.175+0000 7fd601000640 1 -- 172.21.15.67:0/4279555629 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fd5e00a2910 msgr2=0x7fd5e00a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:17.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:17.175+0000 7fd601000640 1 --2- 172.21.15.67:0/4279555629 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fd5e00a2910 0x7fd5e00a4dd0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec005fd0 tx=0x7fd5ec005850 comp rx=0 tx=0).stop 2024-03-28T16:01:17.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:17.175+0000 7fd601000640 1 -- 172.21.15.67:0/4279555629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5fc1096b0 msgr2=0x7fd5fc1a5290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:17.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:17.175+0000 7fd601000640 1 --2- 172.21.15.67:0/4279555629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5fc1096b0 0x7fd5fc1a5290 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fd5f400eaa0 tx=0x7fd5f400ef60 comp rx=0 tx=0).stop 2024-03-28T16:01:17.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:17.176+0000 7fd601000640 1 -- 172.21.15.67:0/4279555629 shutdown_connections 2024-03-28T16:01:17.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:17.176+0000 7fd601000640 1 --2- 172.21.15.67:0/4279555629 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fd5e00a2910 0x7fd5e00a4dd0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:17.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:17.176+0000 7fd601000640 1 --2- 172.21.15.67:0/4279555629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5fc1096b0 0x7fd5fc1a5290 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:17.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:17.176+0000 7fd601000640 1 --2- 172.21.15.67:0/4279555629 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5fc108cf0 0x7fd5fc19e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:17.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:17.176+0000 7fd601000640 1 --2- 172.21.15.67:0/4279555629 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd5fc107b00 0x7fd5fc19dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:17.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:17.176+0000 7fd601000640 1 -- 172.21.15.67:0/4279555629 >> 172.21.15.67:0/4279555629 conn(0x7fd5fc076ab0 msgr2=0x7fd5fc10bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:17.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:17.176+0000 7fd601000640 1 -- 172.21.15.67:0/4279555629 shutdown_connections 2024-03-28T16:01:17.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:17.176+0000 7fd601000640 1 -- 172.21.15.67:0/4279555629 wait complete. 2024-03-28T16:01:17.196 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":369,"stamp":"2024-03-28T16:01:17.041684+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":612295,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":144,"num_read_kb":141,"num_write":235,"num_write_kb":2975,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":185,"ondisk_log_size":185,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":383248,"kb_used_data":13248,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124558576,"statfs":{"total":1151940427776,"available":1151547981824,"internally_reserved":0,"allocated":13565952,"data_stored":7395892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":378713402},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21927,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":18,"num_read_kb":32,"num_write":54,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.811607"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:10.304055+0000","last_change":"2024-03-28T16:01:08.268196+0000","last_active":"2024-03-28T16:01:10.304055+0000","last_peered":"2024-03-28T16:01:10.304055+0000","last_clean":"2024-03-28T16:01:10.304055+0000","last_became_active":"2024-03-28T16:01:08.266956+0000","last_became_peered":"2024-03-28T16:01:08.266956+0000","last_unstale":"2024-03-28T16:01:10.304055+0000","last_undegraded":"2024-03-28T16:01:10.304055+0000","last_fullsized":"2024-03-28T16:01:10.304055+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:38:44.296129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.542544+0000","last_change":"2024-03-28T16:01:08.264985+0000","last_active":"2024-03-28T16:01:09.542544+0000","last_peered":"2024-03-28T16:01:09.542544+0000","last_clean":"2024-03-28T16:01:09.542544+0000","last_became_active":"2024-03-28T16:01:08.262484+0000","last_became_peered":"2024-03-28T16:01:08.262484+0000","last_unstale":"2024-03-28T16:01:09.542544+0000","last_undegraded":"2024-03-28T16:01:09.542544+0000","last_fullsized":"2024-03-28T16:01:09.542544+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:33:07.538908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.541985+0000","last_change":"2024-03-28T16:01:08.267423+0000","last_active":"2024-03-28T16:01:09.541985+0000","last_peered":"2024-03-28T16:01:09.541985+0000","last_clean":"2024-03-28T16:01:09.541985+0000","last_became_active":"2024-03-28T16:01:08.263665+0000","last_became_peered":"2024-03-28T16:01:08.263665+0000","last_unstale":"2024-03-28T16:01:09.541985+0000","last_undegraded":"2024-03-28T16:01:09.541985+0000","last_fullsized":"2024-03-28T16:01:09.541985+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:09:43.488003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.543679+0000","last_change":"2024-03-28T16:01:08.274978+0000","last_active":"2024-03-28T16:01:09.543679+0000","last_peered":"2024-03-28T16:01:09.543679+0000","last_clean":"2024-03-28T16:01:09.543679+0000","last_became_active":"2024-03-28T16:01:08.274553+0000","last_became_peered":"2024-03-28T16:01:08.274553+0000","last_unstale":"2024-03-28T16:01:09.543679+0000","last_undegraded":"2024-03-28T16:01:09.543679+0000","last_fullsized":"2024-03-28T16:01:09.543679+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:16:03.997334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.297381+0000","last_change":"2024-03-28T16:01:08.262155+0000","last_active":"2024-03-28T16:01:09.297381+0000","last_peered":"2024-03-28T16:01:09.297381+0000","last_clean":"2024-03-28T16:01:09.297381+0000","last_became_active":"2024-03-28T16:01:08.261814+0000","last_became_peered":"2024-03-28T16:01:08.261814+0000","last_unstale":"2024-03-28T16:01:09.297381+0000","last_undegraded":"2024-03-28T16:01:09.297381+0000","last_fullsized":"2024-03-28T16:01:09.297381+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:41:04.242203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.544137+0000","last_change":"2024-03-28T16:01:08.273342+0000","last_active":"2024-03-28T16:01:09.544137+0000","last_peered":"2024-03-28T16:01:09.544137+0000","last_clean":"2024-03-28T16:01:09.544137+0000","last_became_active":"2024-03-28T16:01:08.272606+0000","last_became_peered":"2024-03-28T16:01:08.272606+0000","last_unstale":"2024-03-28T16:01:09.544137+0000","last_undegraded":"2024-03-28T16:01:09.544137+0000","last_fullsized":"2024-03-28T16:01:09.544137+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:44:48.580214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.249900+0000","last_change":"2024-03-28T16:01:08.278060+0000","last_active":"2024-03-28T16:01:09.249900+0000","last_peered":"2024-03-28T16:01:09.249900+0000","last_clean":"2024-03-28T16:01:09.249900+0000","last_became_active":"2024-03-28T16:01:08.277047+0000","last_became_peered":"2024-03-28T16:01:08.277047+0000","last_unstale":"2024-03-28T16:01:09.249900+0000","last_undegraded":"2024-03-28T16:01:09.249900+0000","last_fullsized":"2024-03-28T16:01:09.249900+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:46:30.005708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.294527+0000","last_change":"2024-03-28T16:01:08.273199+0000","last_active":"2024-03-28T16:01:09.294527+0000","last_peered":"2024-03-28T16:01:09.294527+0000","last_clean":"2024-03-28T16:01:09.294527+0000","last_became_active":"2024-03-28T16:01:08.272822+0000","last_became_peered":"2024-03-28T16:01:08.272822+0000","last_unstale":"2024-03-28T16:01:09.294527+0000","last_undegraded":"2024-03-28T16:01:09.294527+0000","last_fullsized":"2024-03-28T16:01:09.294527+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:28:11.495665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.545290+0000","last_change":"2024-03-28T16:01:08.270262+0000","last_active":"2024-03-28T16:01:09.545290+0000","last_peered":"2024-03-28T16:01:09.545290+0000","last_clean":"2024-03-28T16:01:09.545290+0000","last_became_active":"2024-03-28T16:01:08.269959+0000","last_became_peered":"2024-03-28T16:01:08.269959+0000","last_unstale":"2024-03-28T16:01:09.545290+0000","last_undegraded":"2024-03-28T16:01:09.545290+0000","last_fullsized":"2024-03-28T16:01:09.545290+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:49:02.261893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.543775+0000","last_change":"2024-03-28T16:01:08.277064+0000","last_active":"2024-03-28T16:01:09.543775+0000","last_peered":"2024-03-28T16:01:09.543775+0000","last_clean":"2024-03-28T16:01:09.543775+0000","last_became_active":"2024-03-28T16:01:08.274699+0000","last_became_peered":"2024-03-28T16:01:08.274699+0000","last_unstale":"2024-03-28T16:01:09.543775+0000","last_undegraded":"2024-03-28T16:01:09.543775+0000","last_fullsized":"2024-03-28T16:01:09.543775+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:43:02.970898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.250349+0000","last_change":"2024-03-28T16:01:08.274943+0000","last_active":"2024-03-28T16:01:09.250349+0000","last_peered":"2024-03-28T16:01:09.250349+0000","last_clean":"2024-03-28T16:01:09.250349+0000","last_became_active":"2024-03-28T16:01:08.274267+0000","last_became_peered":"2024-03-28T16:01:08.274267+0000","last_unstale":"2024-03-28T16:01:09.250349+0000","last_undegraded":"2024-03-28T16:01:09.250349+0000","last_fullsized":"2024-03-28T16:01:09.250349+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:41:13.979918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.543491+0000","last_change":"2024-03-28T16:01:08.266817+0000","last_active":"2024-03-28T16:01:09.543491+0000","last_peered":"2024-03-28T16:01:09.543491+0000","last_clean":"2024-03-28T16:01:09.543491+0000","last_became_active":"2024-03-28T16:01:08.266470+0000","last_became_peered":"2024-03-28T16:01:08.266470+0000","last_unstale":"2024-03-28T16:01:09.543491+0000","last_undegraded":"2024-03-28T16:01:09.543491+0000","last_fullsized":"2024-03-28T16:01:09.543491+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:15:28.573153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.245403+0000","last_change":"2024-03-28T16:01:08.269896+0000","last_active":"2024-03-28T16:01:09.245403+0000","last_peered":"2024-03-28T16:01:09.245403+0000","last_clean":"2024-03-28T16:01:09.245403+0000","last_became_active":"2024-03-28T16:01:08.269500+0000","last_became_peered":"2024-03-28T16:01:08.269500+0000","last_unstale":"2024-03-28T16:01:09.245403+0000","last_undegraded":"2024-03-28T16:01:09.245403+0000","last_fullsized":"2024-03-28T16:01:09.245403+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:44:38.867590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.542051+0000","last_change":"2024-03-28T16:01:08.267072+0000","last_active":"2024-03-28T16:01:09.542051+0000","last_peered":"2024-03-28T16:01:09.542051+0000","last_clean":"2024-03-28T16:01:09.542051+0000","last_became_active":"2024-03-28T16:01:08.266688+0000","last_became_peered":"2024-03-28T16:01:08.266688+0000","last_unstale":"2024-03-28T16:01:09.542051+0000","last_undegraded":"2024-03-28T16:01:09.542051+0000","last_fullsized":"2024-03-28T16:01:09.542051+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:07:57.702952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.542001+0000","last_change":"2024-03-28T16:01:08.270912+0000","last_active":"2024-03-28T16:01:09.542001+0000","last_peered":"2024-03-28T16:01:09.542001+0000","last_clean":"2024-03-28T16:01:09.542001+0000","last_became_active":"2024-03-28T16:01:08.270567+0000","last_became_peered":"2024-03-28T16:01:08.270567+0000","last_unstale":"2024-03-28T16:01:09.542001+0000","last_undegraded":"2024-03-28T16:01:09.542001+0000","last_fullsized":"2024-03-28T16:01:09.542001+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:19:29.294502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.542451+0000","last_change":"2024-03-28T16:01:08.266811+0000","last_active":"2024-03-28T16:01:09.542451+0000","last_peered":"2024-03-28T16:01:09.542451+0000","last_clean":"2024-03-28T16:01:09.542451+0000","last_became_active":"2024-03-28T16:01:08.266517+0000","last_became_peered":"2024-03-28T16:01:08.266517+0000","last_unstale":"2024-03-28T16:01:09.542451+0000","last_undegraded":"2024-03-28T16:01:09.542451+0000","last_fullsized":"2024-03-28T16:01:09.542451+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:55:05.972683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:10.304157+0000","last_change":"2024-03-28T16:01:08.263864+0000","last_active":"2024-03-28T16:01:10.304157+0000","last_peered":"2024-03-28T16:01:10.304157+0000","last_clean":"2024-03-28T16:01:10.304157+0000","last_became_active":"2024-03-28T16:01:08.262956+0000","last_became_peered":"2024-03-28T16:01:08.262956+0000","last_unstale":"2024-03-28T16:01:10.304157+0000","last_undegraded":"2024-03-28T16:01:10.304157+0000","last_fullsized":"2024-03-28T16:01:10.304157+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:05:50.504237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.291828+0000","last_change":"2024-03-28T16:01:08.265151+0000","last_active":"2024-03-28T16:01:09.291828+0000","last_peered":"2024-03-28T16:01:09.291828+0000","last_clean":"2024-03-28T16:01:09.291828+0000","last_became_active":"2024-03-28T16:01:08.264861+0000","last_became_peered":"2024-03-28T16:01:08.264861+0000","last_unstale":"2024-03-28T16:01:09.291828+0000","last_undegraded":"2024-03-28T16:01:09.291828+0000","last_fullsized":"2024-03-28T16:01:09.291828+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:35:45.878983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.544112+0000","last_change":"2024-03-28T16:01:08.265993+0000","last_active":"2024-03-28T16:01:09.544112+0000","last_peered":"2024-03-28T16:01:09.544112+0000","last_clean":"2024-03-28T16:01:09.544112+0000","last_became_active":"2024-03-28T16:01:08.265488+0000","last_became_peered":"2024-03-28T16:01:08.265488+0000","last_unstale":"2024-03-28T16:01:09.544112+0000","last_undegraded":"2024-03-28T16:01:09.544112+0000","last_fullsized":"2024-03-28T16:01:09.544112+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:31:55.920705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'4","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.291637+0000","last_change":"2024-03-28T16:01:07.250361+0000","last_active":"2024-03-28T16:01:09.291637+0000","last_peered":"2024-03-28T16:01:09.291637+0000","last_clean":"2024-03-28T16:01:09.291637+0000","last_became_active":"2024-03-28T16:01:07.250004+0000","last_became_peered":"2024-03-28T16:01:07.250004+0000","last_unstale":"2024-03-28T16:01:09.291637+0000","last_undegraded":"2024-03-28T16:01:09.291637+0000","last_fullsized":"2024-03-28T16:01:09.291637+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:10:32.838716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.296283+0000","last_change":"2024-03-28T16:01:08.274436+0000","last_active":"2024-03-28T16:01:09.296283+0000","last_peered":"2024-03-28T16:01:09.296283+0000","last_clean":"2024-03-28T16:01:09.296283+0000","last_became_active":"2024-03-28T16:01:08.274124+0000","last_became_peered":"2024-03-28T16:01:08.274124+0000","last_unstale":"2024-03-28T16:01:09.296283+0000","last_undegraded":"2024-03-28T16:01:09.296283+0000","last_fullsized":"2024-03-28T16:01:09.296283+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:02:34.037686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"89'1","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.296211+0000","last_change":"2024-03-28T16:01:07.248565+0000","last_active":"2024-03-28T16:01:09.296211+0000","last_peered":"2024-03-28T16:01:09.296211+0000","last_clean":"2024-03-28T16:01:09.296211+0000","last_became_active":"2024-03-28T16:01:07.247366+0000","last_became_peered":"2024-03-28T16:01:07.247366+0000","last_unstale":"2024-03-28T16:01:09.296211+0000","last_undegraded":"2024-03-28T16:01:09.296211+0000","last_fullsized":"2024-03-28T16:01:09.296211+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:48:13.146398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.542626+0000","last_change":"2024-03-28T16:01:08.265639+0000","last_active":"2024-03-28T16:01:09.542626+0000","last_peered":"2024-03-28T16:01:09.542626+0000","last_clean":"2024-03-28T16:01:09.542626+0000","last_became_active":"2024-03-28T16:01:08.263488+0000","last_became_peered":"2024-03-28T16:01:08.263488+0000","last_unstale":"2024-03-28T16:01:09.542626+0000","last_undegraded":"2024-03-28T16:01:09.542626+0000","last_fullsized":"2024-03-28T16:01:09.542626+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:15:06.516016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"89'2","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.297382+0000","last_change":"2024-03-28T16:01:07.258327+0000","last_active":"2024-03-28T16:01:09.297382+0000","last_peered":"2024-03-28T16:01:09.297382+0000","last_clean":"2024-03-28T16:01:09.297382+0000","last_became_active":"2024-03-28T16:01:07.257881+0000","last_became_peered":"2024-03-28T16:01:07.257881+0000","last_unstale":"2024-03-28T16:01:09.297382+0000","last_undegraded":"2024-03-28T16:01:09.297382+0000","last_fullsized":"2024-03-28T16:01:09.297382+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:14:05.453004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.544031+0000","last_change":"2024-03-28T16:01:08.271534+0000","last_active":"2024-03-28T16:01:09.544031+0000","last_peered":"2024-03-28T16:01:09.544031+0000","last_clean":"2024-03-28T16:01:09.544031+0000","last_became_active":"2024-03-28T16:01:08.271104+0000","last_became_peered":"2024-03-28T16:01:08.271104+0000","last_unstale":"2024-03-28T16:01:09.544031+0000","last_undegraded":"2024-03-28T16:01:09.544031+0000","last_fullsized":"2024-03-28T16:01:09.544031+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:29:21.332346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"89'2","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.297533+0000","last_change":"2024-03-28T16:01:07.259175+0000","last_active":"2024-03-28T16:01:09.297533+0000","last_peered":"2024-03-28T16:01:09.297533+0000","last_clean":"2024-03-28T16:01:09.297533+0000","last_became_active":"2024-03-28T16:01:07.258151+0000","last_became_peered":"2024-03-28T16:01:07.258151+0000","last_unstale":"2024-03-28T16:01:09.297533+0000","last_undegraded":"2024-03-28T16:01:09.297533+0000","last_fullsized":"2024-03-28T16:01:09.297533+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:41:07.064800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.250343+0000","last_change":"2024-03-28T16:01:08.277406+0000","last_active":"2024-03-28T16:01:09.250343+0000","last_peered":"2024-03-28T16:01:09.250343+0000","last_clean":"2024-03-28T16:01:09.250343+0000","last_became_active":"2024-03-28T16:01:08.274653+0000","last_became_peered":"2024-03-28T16:01:08.274653+0000","last_unstale":"2024-03-28T16:01:09.250343+0000","last_undegraded":"2024-03-28T16:01:09.250343+0000","last_fullsized":"2024-03-28T16:01:09.250343+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:59:44.306616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"89'1","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.249280+0000","last_change":"2024-03-28T16:01:07.248538+0000","last_active":"2024-03-28T16:01:09.249280+0000","last_peered":"2024-03-28T16:01:09.249280+0000","last_clean":"2024-03-28T16:01:09.249280+0000","last_became_active":"2024-03-28T16:01:07.248116+0000","last_became_peered":"2024-03-28T16:01:07.248116+0000","last_unstale":"2024-03-28T16:01:09.249280+0000","last_undegraded":"2024-03-28T16:01:09.249280+0000","last_fullsized":"2024-03-28T16:01:09.249280+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:44:56.071286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.296884+0000","last_change":"2024-03-28T16:01:08.271059+0000","last_active":"2024-03-28T16:01:09.296884+0000","last_peered":"2024-03-28T16:01:09.296884+0000","last_clean":"2024-03-28T16:01:09.296884+0000","last_became_active":"2024-03-28T16:01:08.270682+0000","last_became_peered":"2024-03-28T16:01:08.270682+0000","last_unstale":"2024-03-28T16:01:09.296884+0000","last_undegraded":"2024-03-28T16:01:09.296884+0000","last_fullsized":"2024-03-28T16:01:09.296884+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:54:48.922125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"89'1","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.543879+0000","last_change":"2024-03-28T16:01:07.261157+0000","last_active":"2024-03-28T16:01:09.543879+0000","last_peered":"2024-03-28T16:01:09.543879+0000","last_clean":"2024-03-28T16:01:09.543879+0000","last_became_active":"2024-03-28T16:01:07.260751+0000","last_became_peered":"2024-03-28T16:01:07.260751+0000","last_unstale":"2024-03-28T16:01:09.543879+0000","last_undegraded":"2024-03-28T16:01:09.543879+0000","last_fullsized":"2024-03-28T16:01:09.543879+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:16:49.622168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.249617+0000","last_change":"2024-03-28T16:01:08.265773+0000","last_active":"2024-03-28T16:01:09.249617+0000","last_peered":"2024-03-28T16:01:09.249617+0000","last_clean":"2024-03-28T16:01:09.249617+0000","last_became_active":"2024-03-28T16:01:08.263724+0000","last_became_peered":"2024-03-28T16:01:08.263724+0000","last_unstale":"2024-03-28T16:01:09.249617+0000","last_undegraded":"2024-03-28T16:01:09.249617+0000","last_fullsized":"2024-03-28T16:01:09.249617+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:42:37.338946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.245061+0000","last_change":"2024-03-28T16:01:07.235361+0000","last_active":"2024-03-28T16:01:09.245061+0000","last_peered":"2024-03-28T16:01:09.245061+0000","last_clean":"2024-03-28T16:01:09.245061+0000","last_became_active":"2024-03-28T16:01:07.234969+0000","last_became_peered":"2024-03-28T16:01:07.234969+0000","last_unstale":"2024-03-28T16:01:09.245061+0000","last_undegraded":"2024-03-28T16:01:09.245061+0000","last_fullsized":"2024-03-28T16:01:09.245061+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:50:35.592147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.294620+0000","last_change":"2024-03-28T16:01:08.263878+0000","last_active":"2024-03-28T16:01:09.294620+0000","last_peered":"2024-03-28T16:01:09.294620+0000","last_clean":"2024-03-28T16:01:09.294620+0000","last_became_active":"2024-03-28T16:01:08.263549+0000","last_became_peered":"2024-03-28T16:01:08.263549+0000","last_unstale":"2024-03-28T16:01:09.294620+0000","last_undegraded":"2024-03-28T16:01:09.294620+0000","last_fullsized":"2024-03-28T16:01:09.294620+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:31:37.669215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.248824+0000","last_change":"2024-03-28T16:01:07.255701+0000","last_active":"2024-03-28T16:01:09.248824+0000","last_peered":"2024-03-28T16:01:09.248824+0000","last_clean":"2024-03-28T16:01:09.248824+0000","last_became_active":"2024-03-28T16:01:07.255247+0000","last_became_peered":"2024-03-28T16:01:07.255247+0000","last_unstale":"2024-03-28T16:01:09.248824+0000","last_undegraded":"2024-03-28T16:01:09.248824+0000","last_fullsized":"2024-03-28T16:01:09.248824+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:08:17.858395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.291383+0000","last_change":"2024-03-28T16:01:08.252949+0000","last_active":"2024-03-28T16:01:09.291383+0000","last_peered":"2024-03-28T16:01:09.291383+0000","last_clean":"2024-03-28T16:01:09.291383+0000","last_became_active":"2024-03-28T16:01:08.252471+0000","last_became_peered":"2024-03-28T16:01:08.252471+0000","last_unstale":"2024-03-28T16:01:09.291383+0000","last_undegraded":"2024-03-28T16:01:09.291383+0000","last_fullsized":"2024-03-28T16:01:09.291383+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:21:15.519020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.543974+0000","last_change":"2024-03-28T16:01:07.263973+0000","last_active":"2024-03-28T16:01:09.543974+0000","last_peered":"2024-03-28T16:01:09.543974+0000","last_clean":"2024-03-28T16:01:09.543974+0000","last_became_active":"2024-03-28T16:01:07.263560+0000","last_became_peered":"2024-03-28T16:01:07.263560+0000","last_unstale":"2024-03-28T16:01:09.543974+0000","last_undegraded":"2024-03-28T16:01:09.543974+0000","last_fullsized":"2024-03-28T16:01:09.543974+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:11.215312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.545319+0000","last_change":"2024-03-28T16:01:08.266122+0000","last_active":"2024-03-28T16:01:09.545319+0000","last_peered":"2024-03-28T16:01:09.545319+0000","last_clean":"2024-03-28T16:01:09.545319+0000","last_became_active":"2024-03-28T16:01:08.265727+0000","last_became_peered":"2024-03-28T16:01:08.265727+0000","last_unstale":"2024-03-28T16:01:09.545319+0000","last_undegraded":"2024-03-28T16:01:09.545319+0000","last_fullsized":"2024-03-28T16:01:09.545319+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:48:38.939777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.250386+0000","last_change":"2024-03-28T16:01:07.251814+0000","last_active":"2024-03-28T16:01:09.250386+0000","last_peered":"2024-03-28T16:01:09.250386+0000","last_clean":"2024-03-28T16:01:09.250386+0000","last_became_active":"2024-03-28T16:01:07.251311+0000","last_became_peered":"2024-03-28T16:01:07.251311+0000","last_unstale":"2024-03-28T16:01:09.250386+0000","last_undegraded":"2024-03-28T16:01:09.250386+0000","last_fullsized":"2024-03-28T16:01:09.250386+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:13:38.344070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.542555+0000","last_change":"2024-03-28T16:01:08.270879+0000","last_active":"2024-03-28T16:01:09.542555+0000","last_peered":"2024-03-28T16:01:09.542555+0000","last_clean":"2024-03-28T16:01:09.542555+0000","last_became_active":"2024-03-28T16:01:08.270513+0000","last_became_peered":"2024-03-28T16:01:08.270513+0000","last_unstale":"2024-03-28T16:01:09.542555+0000","last_undegraded":"2024-03-28T16:01:09.542555+0000","last_fullsized":"2024-03-28T16:01:09.542555+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:53:43.285797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"89'3","reported_seq":18,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.250641+0000","last_change":"2024-03-28T16:01:07.253595+0000","last_active":"2024-03-28T16:01:09.250641+0000","last_peered":"2024-03-28T16:01:09.250641+0000","last_clean":"2024-03-28T16:01:09.250641+0000","last_became_active":"2024-03-28T16:01:07.253137+0000","last_became_peered":"2024-03-28T16:01:07.253137+0000","last_unstale":"2024-03-28T16:01:09.250641+0000","last_undegraded":"2024-03-28T16:01:09.250641+0000","last_fullsized":"2024-03-28T16:01:09.250641+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:17:35.759791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.544273+0000","last_change":"2024-03-28T16:01:08.261185+0000","last_active":"2024-03-28T16:01:09.544273+0000","last_peered":"2024-03-28T16:01:09.544273+0000","last_clean":"2024-03-28T16:01:09.544273+0000","last_became_active":"2024-03-28T16:01:08.260885+0000","last_became_peered":"2024-03-28T16:01:08.260885+0000","last_unstale":"2024-03-28T16:01:09.544273+0000","last_undegraded":"2024-03-28T16:01:09.544273+0000","last_fullsized":"2024-03-28T16:01:09.544273+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:15:56.640586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"90'13","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:14.524435+0000","last_change":"2024-03-28T16:01:07.258655+0000","last_active":"2024-03-28T16:01:14.524435+0000","last_peered":"2024-03-28T16:01:14.524435+0000","last_clean":"2024-03-28T16:01:14.524435+0000","last_became_active":"2024-03-28T16:01:07.258230+0000","last_became_peered":"2024-03-28T16:01:07.258230+0000","last_unstale":"2024-03-28T16:01:14.524435+0000","last_undegraded":"2024-03-28T16:01:14.524435+0000","last_fullsized":"2024-03-28T16:01:14.524435+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:29:36.206061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20895,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":26,"num_write":13,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.297144+0000","last_change":"2024-03-28T16:01:08.265393+0000","last_active":"2024-03-28T16:01:09.297144+0000","last_peered":"2024-03-28T16:01:09.297144+0000","last_clean":"2024-03-28T16:01:09.297144+0000","last_became_active":"2024-03-28T16:01:08.264560+0000","last_became_peered":"2024-03-28T16:01:08.264560+0000","last_unstale":"2024-03-28T16:01:09.297144+0000","last_undegraded":"2024-03-28T16:01:09.297144+0000","last_fullsized":"2024-03-28T16:01:09.297144+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:43:52.181113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"89'2","reported_seq":18,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.543814+0000","last_change":"2024-03-28T16:01:07.251428+0000","last_active":"2024-03-28T16:01:09.543814+0000","last_peered":"2024-03-28T16:01:09.543814+0000","last_clean":"2024-03-28T16:01:09.543814+0000","last_became_active":"2024-03-28T16:01:07.251074+0000","last_became_peered":"2024-03-28T16:01:07.251074+0000","last_unstale":"2024-03-28T16:01:09.543814+0000","last_undegraded":"2024-03-28T16:01:09.543814+0000","last_fullsized":"2024-03-28T16:01:09.543814+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:18:41.359338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.245074+0000","last_change":"2024-03-28T16:01:08.268482+0000","last_active":"2024-03-28T16:01:09.245074+0000","last_peered":"2024-03-28T16:01:09.245074+0000","last_clean":"2024-03-28T16:01:09.245074+0000","last_became_active":"2024-03-28T16:01:08.268153+0000","last_became_peered":"2024-03-28T16:01:08.268153+0000","last_unstale":"2024-03-28T16:01:09.245074+0000","last_undegraded":"2024-03-28T16:01:09.245074+0000","last_fullsized":"2024-03-28T16:01:09.245074+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:21:21.874773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.248812+0000","last_change":"2024-03-28T16:01:07.243506+0000","last_active":"2024-03-28T16:01:09.248812+0000","last_peered":"2024-03-28T16:01:09.248812+0000","last_clean":"2024-03-28T16:01:09.248812+0000","last_became_active":"2024-03-28T16:01:07.242793+0000","last_became_peered":"2024-03-28T16:01:07.242793+0000","last_unstale":"2024-03-28T16:01:09.248812+0000","last_undegraded":"2024-03-28T16:01:09.248812+0000","last_fullsized":"2024-03-28T16:01:09.248812+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:31:56.497276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.250571+0000","last_change":"2024-03-28T16:01:08.278848+0000","last_active":"2024-03-28T16:01:09.250571+0000","last_peered":"2024-03-28T16:01:09.250571+0000","last_clean":"2024-03-28T16:01:09.250571+0000","last_became_active":"2024-03-28T16:01:08.278455+0000","last_became_peered":"2024-03-28T16:01:08.278455+0000","last_unstale":"2024-03-28T16:01:09.250571+0000","last_undegraded":"2024-03-28T16:01:09.250571+0000","last_fullsized":"2024-03-28T16:01:09.250571+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:41:17.302538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.542650+0000","last_change":"2024-03-28T16:01:07.255070+0000","last_active":"2024-03-28T16:01:09.542650+0000","last_peered":"2024-03-28T16:01:09.542650+0000","last_clean":"2024-03-28T16:01:09.542650+0000","last_became_active":"2024-03-28T16:01:07.253489+0000","last_became_peered":"2024-03-28T16:01:07.253489+0000","last_unstale":"2024-03-28T16:01:09.542650+0000","last_undegraded":"2024-03-28T16:01:09.542650+0000","last_fullsized":"2024-03-28T16:01:09.542650+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:31:57.501721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.249148+0000","last_change":"2024-03-28T16:01:08.273190+0000","last_active":"2024-03-28T16:01:09.249148+0000","last_peered":"2024-03-28T16:01:09.249148+0000","last_clean":"2024-03-28T16:01:09.249148+0000","last_became_active":"2024-03-28T16:01:08.272870+0000","last_became_peered":"2024-03-28T16:01:08.272870+0000","last_unstale":"2024-03-28T16:01:09.249148+0000","last_undegraded":"2024-03-28T16:01:09.249148+0000","last_fullsized":"2024-03-28T16:01:09.249148+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:26:50.874750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"89'8","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.296383+0000","last_change":"2024-03-28T16:01:07.257210+0000","last_active":"2024-03-28T16:01:09.296383+0000","last_peered":"2024-03-28T16:01:09.296383+0000","last_clean":"2024-03-28T16:01:09.296383+0000","last_became_active":"2024-03-28T16:01:07.255300+0000","last_became_peered":"2024-03-28T16:01:07.255300+0000","last_unstale":"2024-03-28T16:01:09.296383+0000","last_undegraded":"2024-03-28T16:01:09.296383+0000","last_fullsized":"2024-03-28T16:01:09.296383+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:38:37.761030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.249110+0000","last_change":"2024-03-28T16:01:08.269849+0000","last_active":"2024-03-28T16:01:09.249110+0000","last_peered":"2024-03-28T16:01:09.249110+0000","last_clean":"2024-03-28T16:01:09.249110+0000","last_became_active":"2024-03-28T16:01:08.269506+0000","last_became_peered":"2024-03-28T16:01:08.269506+0000","last_unstale":"2024-03-28T16:01:09.249110+0000","last_undegraded":"2024-03-28T16:01:09.249110+0000","last_fullsized":"2024-03-28T16:01:09.249110+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:04:01.708192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'1","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.697061+0000","last_change":"2024-03-28T16:01:07.254115+0000","last_active":"2024-03-28T16:01:09.697061+0000","last_peered":"2024-03-28T16:01:09.697061+0000","last_clean":"2024-03-28T16:01:09.697061+0000","last_became_active":"2024-03-28T16:01:07.253664+0000","last_became_peered":"2024-03-28T16:01:07.253664+0000","last_unstale":"2024-03-28T16:01:09.697061+0000","last_undegraded":"2024-03-28T16:01:09.697061+0000","last_fullsized":"2024-03-28T16:01:09.697061+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:19:32.386856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:10.303729+0000","last_change":"2024-03-28T16:01:08.274436+0000","last_active":"2024-03-28T16:01:10.303729+0000","last_peered":"2024-03-28T16:01:10.303729+0000","last_clean":"2024-03-28T16:01:10.303729+0000","last_became_active":"2024-03-28T16:01:08.274081+0000","last_became_peered":"2024-03-28T16:01:08.274081+0000","last_unstale":"2024-03-28T16:01:10.303729+0000","last_undegraded":"2024-03-28T16:01:10.303729+0000","last_fullsized":"2024-03-28T16:01:10.303729+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:57:00.531064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.291704+0000","last_change":"2024-03-28T16:01:07.247542+0000","last_active":"2024-03-28T16:01:09.291704+0000","last_peered":"2024-03-28T16:01:09.291704+0000","last_clean":"2024-03-28T16:01:09.291704+0000","last_became_active":"2024-03-28T16:01:07.247109+0000","last_became_peered":"2024-03-28T16:01:07.247109+0000","last_unstale":"2024-03-28T16:01:09.291704+0000","last_undegraded":"2024-03-28T16:01:09.291704+0000","last_fullsized":"2024-03-28T16:01:09.291704+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:14:19.387896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"90'1","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.557549+0000","last_change":"2024-03-28T16:01:08.264565+0000","last_active":"2024-03-28T16:01:09.557549+0000","last_peered":"2024-03-28T16:01:09.557549+0000","last_clean":"2024-03-28T16:01:09.557549+0000","last_became_active":"2024-03-28T16:01:08.264196+0000","last_became_peered":"2024-03-28T16:01:08.264196+0000","last_unstale":"2024-03-28T16:01:09.557549+0000","last_undegraded":"2024-03-28T16:01:09.557549+0000","last_fullsized":"2024-03-28T16:01:09.557549+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:43:24.750624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.249483+0000","last_change":"2024-03-28T16:01:07.248189+0000","last_active":"2024-03-28T16:01:09.249483+0000","last_peered":"2024-03-28T16:01:09.249483+0000","last_clean":"2024-03-28T16:01:09.249483+0000","last_became_active":"2024-03-28T16:01:07.247295+0000","last_became_peered":"2024-03-28T16:01:07.247295+0000","last_unstale":"2024-03-28T16:01:09.249483+0000","last_undegraded":"2024-03-28T16:01:09.249483+0000","last_fullsized":"2024-03-28T16:01:09.249483+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:46:51.875094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.542906+0000","last_change":"2024-03-28T16:01:08.269724+0000","last_active":"2024-03-28T16:01:09.542906+0000","last_peered":"2024-03-28T16:01:09.542906+0000","last_clean":"2024-03-28T16:01:09.542906+0000","last_became_active":"2024-03-28T16:01:08.269357+0000","last_became_peered":"2024-03-28T16:01:08.269357+0000","last_unstale":"2024-03-28T16:01:09.542906+0000","last_undegraded":"2024-03-28T16:01:09.542906+0000","last_fullsized":"2024-03-28T16:01:09.542906+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:20:12.622238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.250166+0000","last_change":"2024-03-28T16:01:08.266624+0000","last_active":"2024-03-28T16:01:09.250166+0000","last_peered":"2024-03-28T16:01:09.250166+0000","last_clean":"2024-03-28T16:01:09.250166+0000","last_became_active":"2024-03-28T16:01:08.265408+0000","last_became_peered":"2024-03-28T16:01:08.265408+0000","last_unstale":"2024-03-28T16:01:09.250166+0000","last_undegraded":"2024-03-28T16:01:09.250166+0000","last_fullsized":"2024-03-28T16:01:09.250166+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:37:08.831727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.244764+0000","last_change":"2024-03-28T16:01:07.241172+0000","last_active":"2024-03-28T16:01:09.244764+0000","last_peered":"2024-03-28T16:01:09.244764+0000","last_clean":"2024-03-28T16:01:09.244764+0000","last_became_active":"2024-03-28T16:01:07.240754+0000","last_became_peered":"2024-03-28T16:01:07.240754+0000","last_unstale":"2024-03-28T16:01:09.244764+0000","last_undegraded":"2024-03-28T16:01:09.244764+0000","last_fullsized":"2024-03-28T16:01:09.244764+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:52:45.805188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.295213+0000","last_change":"2024-03-28T16:01:08.278429+0000","last_active":"2024-03-28T16:01:09.295213+0000","last_peered":"2024-03-28T16:01:09.295213+0000","last_clean":"2024-03-28T16:01:09.295213+0000","last_became_active":"2024-03-28T16:01:08.275061+0000","last_became_peered":"2024-03-28T16:01:08.275061+0000","last_unstale":"2024-03-28T16:01:09.295213+0000","last_undegraded":"2024-03-28T16:01:09.295213+0000","last_fullsized":"2024-03-28T16:01:09.295213+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:35:42.015979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.544719+0000","last_change":"2024-03-28T16:01:07.252117+0000","last_active":"2024-03-28T16:01:09.544719+0000","last_peered":"2024-03-28T16:01:09.544719+0000","last_clean":"2024-03-28T16:01:09.544719+0000","last_became_active":"2024-03-28T16:01:07.249729+0000","last_became_peered":"2024-03-28T16:01:07.249729+0000","last_unstale":"2024-03-28T16:01:09.544719+0000","last_undegraded":"2024-03-28T16:01:09.544719+0000","last_fullsized":"2024-03-28T16:01:09.544719+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:33:32.719310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":342,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.248908+0000","last_change":"2024-03-28T15:59:32.593730+0000","last_active":"2024-03-28T16:01:09.248908+0000","last_peered":"2024-03-28T16:01:09.248908+0000","last_clean":"2024-03-28T16:01:09.248908+0000","last_became_active":"2024-03-28T15:59:32.592289+0000","last_became_peered":"2024-03-28T15:59:32.592289+0000","last_unstale":"2024-03-28T16:01:09.248908+0000","last_undegraded":"2024-03-28T16:01:09.248908+0000","last_fullsized":"2024-03-28T16:01:09.248908+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:54:55.638193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:54:55.638193+0000","last_clean_scrub_stamp":"2024-03-28T15:54:55.638193+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:20:37.087340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.244882+0000","last_change":"2024-03-28T16:01:08.265513+0000","last_active":"2024-03-28T16:01:09.244882+0000","last_peered":"2024-03-28T16:01:09.244882+0000","last_clean":"2024-03-28T16:01:09.244882+0000","last_became_active":"2024-03-28T16:01:08.264711+0000","last_became_peered":"2024-03-28T16:01:08.264711+0000","last_unstale":"2024-03-28T16:01:09.244882+0000","last_undegraded":"2024-03-28T16:01:09.244882+0000","last_fullsized":"2024-03-28T16:01:09.244882+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:10:01.936780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.249515+0000","last_change":"2024-03-28T16:01:07.252831+0000","last_active":"2024-03-28T16:01:09.249515+0000","last_peered":"2024-03-28T16:01:09.249515+0000","last_clean":"2024-03-28T16:01:09.249515+0000","last_became_active":"2024-03-28T16:01:07.252317+0000","last_became_peered":"2024-03-28T16:01:07.252317+0000","last_unstale":"2024-03-28T16:01:09.249515+0000","last_undegraded":"2024-03-28T16:01:09.249515+0000","last_fullsized":"2024-03-28T16:01:09.249515+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:55:37.461154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.296977+0000","last_change":"2024-03-28T16:01:07.262413+0000","last_active":"2024-03-28T16:01:09.296977+0000","last_peered":"2024-03-28T16:01:09.296977+0000","last_clean":"2024-03-28T16:01:09.296977+0000","last_became_active":"2024-03-28T16:01:07.261875+0000","last_became_peered":"2024-03-28T16:01:07.261875+0000","last_unstale":"2024-03-28T16:01:09.296977+0000","last_undegraded":"2024-03-28T16:01:09.296977+0000","last_fullsized":"2024-03-28T16:01:09.296977+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:25:49.124991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.245549+0000","last_change":"2024-03-28T16:01:08.259506+0000","last_active":"2024-03-28T16:01:09.245549+0000","last_peered":"2024-03-28T16:01:09.245549+0000","last_clean":"2024-03-28T16:01:09.245549+0000","last_became_active":"2024-03-28T16:01:08.258512+0000","last_became_peered":"2024-03-28T16:01:08.258512+0000","last_unstale":"2024-03-28T16:01:09.245549+0000","last_undegraded":"2024-03-28T16:01:09.245549+0000","last_fullsized":"2024-03-28T16:01:09.245549+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:20:17.017763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"90'1","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.291666+0000","last_change":"2024-03-28T16:01:07.254850+0000","last_active":"2024-03-28T16:01:09.291666+0000","last_peered":"2024-03-28T16:01:09.291666+0000","last_clean":"2024-03-28T16:01:09.291666+0000","last_became_active":"2024-03-28T16:01:07.252294+0000","last_became_peered":"2024-03-28T16:01:07.252294+0000","last_unstale":"2024-03-28T16:01:09.291666+0000","last_undegraded":"2024-03-28T16:01:09.291666+0000","last_fullsized":"2024-03-28T16:01:09.291666+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:54:26.700118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.296386+0000","last_change":"2024-03-28T16:01:08.267361+0000","last_active":"2024-03-28T16:01:09.296386+0000","last_peered":"2024-03-28T16:01:09.296386+0000","last_clean":"2024-03-28T16:01:09.296386+0000","last_became_active":"2024-03-28T16:01:08.267016+0000","last_became_peered":"2024-03-28T16:01:08.267016+0000","last_unstale":"2024-03-28T16:01:09.296386+0000","last_undegraded":"2024-03-28T16:01:09.296386+0000","last_fullsized":"2024-03-28T16:01:09.296386+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:16:07.202938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"89'2","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.542976+0000","last_change":"2024-03-28T16:01:07.245292+0000","last_active":"2024-03-28T16:01:09.542976+0000","last_peered":"2024-03-28T16:01:09.542976+0000","last_clean":"2024-03-28T16:01:09.542976+0000","last_became_active":"2024-03-28T16:01:07.244654+0000","last_became_peered":"2024-03-28T16:01:07.244654+0000","last_unstale":"2024-03-28T16:01:09.542976+0000","last_undegraded":"2024-03-28T16:01:09.542976+0000","last_fullsized":"2024-03-28T16:01:09.542976+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:03:39.027844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.542819+0000","last_change":"2024-03-28T16:01:08.274542+0000","last_active":"2024-03-28T16:01:09.542819+0000","last_peered":"2024-03-28T16:01:09.542819+0000","last_clean":"2024-03-28T16:01:09.542819+0000","last_became_active":"2024-03-28T16:01:08.274202+0000","last_became_peered":"2024-03-28T16:01:08.274202+0000","last_unstale":"2024-03-28T16:01:09.542819+0000","last_undegraded":"2024-03-28T16:01:09.542819+0000","last_fullsized":"2024-03-28T16:01:09.542819+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:09:18.199017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.542507+0000","last_change":"2024-03-28T16:01:07.253274+0000","last_active":"2024-03-28T16:01:09.542507+0000","last_peered":"2024-03-28T16:01:09.542507+0000","last_clean":"2024-03-28T16:01:09.542507+0000","last_became_active":"2024-03-28T16:01:07.250230+0000","last_became_peered":"2024-03-28T16:01:07.250230+0000","last_unstale":"2024-03-28T16:01:09.542507+0000","last_undegraded":"2024-03-28T16:01:09.542507+0000","last_fullsized":"2024-03-28T16:01:09.542507+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:20:31.459983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.296498+0000","last_change":"2024-03-28T16:01:08.266551+0000","last_active":"2024-03-28T16:01:09.296498+0000","last_peered":"2024-03-28T16:01:09.296498+0000","last_clean":"2024-03-28T16:01:09.296498+0000","last_became_active":"2024-03-28T16:01:08.266226+0000","last_became_peered":"2024-03-28T16:01:08.266226+0000","last_unstale":"2024-03-28T16:01:09.296498+0000","last_undegraded":"2024-03-28T16:01:09.296498+0000","last_fullsized":"2024-03-28T16:01:09.296498+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:51:16.520285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.294774+0000","last_change":"2024-03-28T16:01:07.252038+0000","last_active":"2024-03-28T16:01:09.294774+0000","last_peered":"2024-03-28T16:01:09.294774+0000","last_clean":"2024-03-28T16:01:09.294774+0000","last_became_active":"2024-03-28T16:01:07.251703+0000","last_became_peered":"2024-03-28T16:01:07.251703+0000","last_unstale":"2024-03-28T16:01:09.294774+0000","last_undegraded":"2024-03-28T16:01:09.294774+0000","last_fullsized":"2024-03-28T16:01:09.294774+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:35:42.518996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:10.303337+0000","last_change":"2024-03-28T16:01:08.260025+0000","last_active":"2024-03-28T16:01:10.303337+0000","last_peered":"2024-03-28T16:01:10.303337+0000","last_clean":"2024-03-28T16:01:10.303337+0000","last_became_active":"2024-03-28T16:01:08.259596+0000","last_became_peered":"2024-03-28T16:01:08.259596+0000","last_unstale":"2024-03-28T16:01:10.303337+0000","last_undegraded":"2024-03-28T16:01:10.303337+0000","last_fullsized":"2024-03-28T16:01:10.303337+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:45:29.228188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.543695+0000","last_change":"2024-03-28T16:01:07.259335+0000","last_active":"2024-03-28T16:01:09.543695+0000","last_peered":"2024-03-28T16:01:09.543695+0000","last_clean":"2024-03-28T16:01:09.543695+0000","last_became_active":"2024-03-28T16:01:07.258589+0000","last_became_peered":"2024-03-28T16:01:07.258589+0000","last_unstale":"2024-03-28T16:01:09.543695+0000","last_undegraded":"2024-03-28T16:01:09.543695+0000","last_fullsized":"2024-03-28T16:01:09.543695+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:06:41.742963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.295788+0000","last_change":"2024-03-28T16:01:08.266794+0000","last_active":"2024-03-28T16:01:09.295788+0000","last_peered":"2024-03-28T16:01:09.295788+0000","last_clean":"2024-03-28T16:01:09.295788+0000","last_became_active":"2024-03-28T16:01:08.266502+0000","last_became_peered":"2024-03-28T16:01:08.266502+0000","last_unstale":"2024-03-28T16:01:09.295788+0000","last_undegraded":"2024-03-28T16:01:09.295788+0000","last_fullsized":"2024-03-28T16:01:09.295788+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:17:12.671677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"89'2","reported_seq":18,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.697010+0000","last_change":"2024-03-28T16:01:07.259239+0000","last_active":"2024-03-28T16:01:09.697010+0000","last_peered":"2024-03-28T16:01:09.697010+0000","last_clean":"2024-03-28T16:01:09.697010+0000","last_became_active":"2024-03-28T16:01:07.258875+0000","last_became_peered":"2024-03-28T16:01:07.258875+0000","last_unstale":"2024-03-28T16:01:09.697010+0000","last_undegraded":"2024-03-28T16:01:09.697010+0000","last_fullsized":"2024-03-28T16:01:09.697010+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:35:12.729173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.250187+0000","last_change":"2024-03-28T16:01:08.278196+0000","last_active":"2024-03-28T16:01:09.250187+0000","last_peered":"2024-03-28T16:01:09.250187+0000","last_clean":"2024-03-28T16:01:09.250187+0000","last_became_active":"2024-03-28T16:01:08.276972+0000","last_became_peered":"2024-03-28T16:01:08.276972+0000","last_unstale":"2024-03-28T16:01:09.250187+0000","last_undegraded":"2024-03-28T16:01:09.250187+0000","last_fullsized":"2024-03-28T16:01:09.250187+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:28:25.231210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.544033+0000","last_change":"2024-03-28T16:01:07.246851+0000","last_active":"2024-03-28T16:01:09.544033+0000","last_peered":"2024-03-28T16:01:09.544033+0000","last_clean":"2024-03-28T16:01:09.544033+0000","last_became_active":"2024-03-28T16:01:07.246365+0000","last_became_peered":"2024-03-28T16:01:07.246365+0000","last_unstale":"2024-03-28T16:01:09.544033+0000","last_undegraded":"2024-03-28T16:01:09.544033+0000","last_fullsized":"2024-03-28T16:01:09.544033+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:16:13.593045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.250579+0000","last_change":"2024-03-28T16:01:08.279468+0000","last_active":"2024-03-28T16:01:09.250579+0000","last_peered":"2024-03-28T16:01:09.250579+0000","last_clean":"2024-03-28T16:01:09.250579+0000","last_became_active":"2024-03-28T16:01:08.274650+0000","last_became_peered":"2024-03-28T16:01:08.274650+0000","last_unstale":"2024-03-28T16:01:09.250579+0000","last_undegraded":"2024-03-28T16:01:09.250579+0000","last_fullsized":"2024-03-28T16:01:09.250579+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:36:32.776085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"89'1","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.250047+0000","last_change":"2024-03-28T16:01:07.258077+0000","last_active":"2024-03-28T16:01:09.250047+0000","last_peered":"2024-03-28T16:01:09.250047+0000","last_clean":"2024-03-28T16:01:09.250047+0000","last_became_active":"2024-03-28T16:01:07.256665+0000","last_became_peered":"2024-03-28T16:01:07.256665+0000","last_unstale":"2024-03-28T16:01:09.250047+0000","last_undegraded":"2024-03-28T16:01:09.250047+0000","last_fullsized":"2024-03-28T16:01:09.250047+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:01:18.852441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"90'1","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:10.995866+0000","last_change":"2024-03-28T16:01:08.278588+0000","last_active":"2024-03-28T16:01:10.995866+0000","last_peered":"2024-03-28T16:01:10.995866+0000","last_clean":"2024-03-28T16:01:10.995866+0000","last_became_active":"2024-03-28T16:01:08.277303+0000","last_became_peered":"2024-03-28T16:01:08.277303+0000","last_unstale":"2024-03-28T16:01:10.995866+0000","last_undegraded":"2024-03-28T16:01:10.995866+0000","last_fullsized":"2024-03-28T16:01:10.995866+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:01:19.927057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"89'2","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.245226+0000","last_change":"2024-03-28T16:01:07.249382+0000","last_active":"2024-03-28T16:01:09.245226+0000","last_peered":"2024-03-28T16:01:09.245226+0000","last_clean":"2024-03-28T16:01:09.245226+0000","last_became_active":"2024-03-28T16:01:07.249065+0000","last_became_peered":"2024-03-28T16:01:07.249065+0000","last_unstale":"2024-03-28T16:01:09.245226+0000","last_undegraded":"2024-03-28T16:01:09.245226+0000","last_fullsized":"2024-03-28T16:01:09.245226+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:07:22.851862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.251277+0000","last_change":"2024-03-28T16:01:08.274839+0000","last_active":"2024-03-28T16:01:09.251277+0000","last_peered":"2024-03-28T16:01:09.251277+0000","last_clean":"2024-03-28T16:01:09.251277+0000","last_became_active":"2024-03-28T16:01:08.274505+0000","last_became_peered":"2024-03-28T16:01:08.274505+0000","last_unstale":"2024-03-28T16:01:09.251277+0000","last_undegraded":"2024-03-28T16:01:09.251277+0000","last_fullsized":"2024-03-28T16:01:09.251277+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:57:37.635327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:10.303497+0000","last_change":"2024-03-28T16:01:08.268207+0000","last_active":"2024-03-28T16:01:10.303497+0000","last_peered":"2024-03-28T16:01:10.303497+0000","last_clean":"2024-03-28T16:01:10.303497+0000","last_became_active":"2024-03-28T16:01:08.267843+0000","last_became_peered":"2024-03-28T16:01:08.267843+0000","last_unstale":"2024-03-28T16:01:10.303497+0000","last_undegraded":"2024-03-28T16:01:10.303497+0000","last_fullsized":"2024-03-28T16:01:10.303497+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:39:05.675279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.543441+0000","last_change":"2024-03-28T16:01:08.272816+0000","last_active":"2024-03-28T16:01:09.543441+0000","last_peered":"2024-03-28T16:01:09.543441+0000","last_clean":"2024-03-28T16:01:09.543441+0000","last_became_active":"2024-03-28T16:01:08.270941+0000","last_became_peered":"2024-03-28T16:01:08.270941+0000","last_unstale":"2024-03-28T16:01:09.543441+0000","last_undegraded":"2024-03-28T16:01:09.543441+0000","last_fullsized":"2024-03-28T16:01:09.543441+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:19:00.481809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.543420+0000","last_change":"2024-03-28T16:01:08.258558+0000","last_active":"2024-03-28T16:01:09.543420+0000","last_peered":"2024-03-28T16:01:09.543420+0000","last_clean":"2024-03-28T16:01:09.543420+0000","last_became_active":"2024-03-28T16:01:08.258163+0000","last_became_peered":"2024-03-28T16:01:08.258163+0000","last_unstale":"2024-03-28T16:01:09.543420+0000","last_undegraded":"2024-03-28T16:01:09.543420+0000","last_fullsized":"2024-03-28T16:01:09.543420+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:53:48.358536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.542309+0000","last_change":"2024-03-28T16:01:08.268886+0000","last_active":"2024-03-28T16:01:09.542309+0000","last_peered":"2024-03-28T16:01:09.542309+0000","last_clean":"2024-03-28T16:01:09.542309+0000","last_became_active":"2024-03-28T16:01:08.268519+0000","last_became_peered":"2024-03-28T16:01:08.268519+0000","last_unstale":"2024-03-28T16:01:09.542309+0000","last_undegraded":"2024-03-28T16:01:09.542309+0000","last_fullsized":"2024-03-28T16:01:09.542309+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:49:05.266517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.245360+0000","last_change":"2024-03-28T16:01:08.265039+0000","last_active":"2024-03-28T16:01:09.245360+0000","last_peered":"2024-03-28T16:01:09.245360+0000","last_clean":"2024-03-28T16:01:09.245360+0000","last_became_active":"2024-03-28T16:01:08.264677+0000","last_became_peered":"2024-03-28T16:01:08.264677+0000","last_unstale":"2024-03-28T16:01:09.245360+0000","last_undegraded":"2024-03-28T16:01:09.245360+0000","last_fullsized":"2024-03-28T16:01:09.245360+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:17:27.667324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:10.303716+0000","last_change":"2024-03-28T16:01:08.271699+0000","last_active":"2024-03-28T16:01:10.303716+0000","last_peered":"2024-03-28T16:01:10.303716+0000","last_clean":"2024-03-28T16:01:10.303716+0000","last_became_active":"2024-03-28T16:01:08.270184+0000","last_became_peered":"2024-03-28T16:01:08.270184+0000","last_unstale":"2024-03-28T16:01:10.303716+0000","last_undegraded":"2024-03-28T16:01:10.303716+0000","last_fullsized":"2024-03-28T16:01:10.303716+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:17:15.699944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.544174+0000","last_change":"2024-03-28T16:01:08.262314+0000","last_active":"2024-03-28T16:01:09.544174+0000","last_peered":"2024-03-28T16:01:09.544174+0000","last_clean":"2024-03-28T16:01:09.544174+0000","last_became_active":"2024-03-28T16:01:08.261992+0000","last_became_peered":"2024-03-28T16:01:08.261992+0000","last_unstale":"2024-03-28T16:01:09.544174+0000","last_undegraded":"2024-03-28T16:01:09.544174+0000","last_fullsized":"2024-03-28T16:01:09.544174+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:40:52.048306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.296233+0000","last_change":"2024-03-28T16:01:08.259407+0000","last_active":"2024-03-28T16:01:09.296233+0000","last_peered":"2024-03-28T16:01:09.296233+0000","last_clean":"2024-03-28T16:01:09.296233+0000","last_became_active":"2024-03-28T16:01:08.259071+0000","last_became_peered":"2024-03-28T16:01:08.259071+0000","last_unstale":"2024-03-28T16:01:09.296233+0000","last_undegraded":"2024-03-28T16:01:09.296233+0000","last_fullsized":"2024-03-28T16:01:09.296233+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:30:17.819868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.249123+0000","last_change":"2024-03-28T16:01:08.271649+0000","last_active":"2024-03-28T16:01:09.249123+0000","last_peered":"2024-03-28T16:01:09.249123+0000","last_clean":"2024-03-28T16:01:09.249123+0000","last_became_active":"2024-03-28T16:01:08.271230+0000","last_became_peered":"2024-03-28T16:01:08.271230+0000","last_unstale":"2024-03-28T16:01:09.249123+0000","last_undegraded":"2024-03-28T16:01:09.249123+0000","last_fullsized":"2024-03-28T16:01:09.249123+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:50:14.404178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.294971+0000","last_change":"2024-03-28T16:01:08.260845+0000","last_active":"2024-03-28T16:01:09.294971+0000","last_peered":"2024-03-28T16:01:09.294971+0000","last_clean":"2024-03-28T16:01:09.294971+0000","last_became_active":"2024-03-28T16:01:08.259997+0000","last_became_peered":"2024-03-28T16:01:08.259997+0000","last_unstale":"2024-03-28T16:01:09.294971+0000","last_undegraded":"2024-03-28T16:01:09.294971+0000","last_fullsized":"2024-03-28T16:01:09.294971+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:08:55.080858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.543516+0000","last_change":"2024-03-28T16:01:08.268881+0000","last_active":"2024-03-28T16:01:09.543516+0000","last_peered":"2024-03-28T16:01:09.543516+0000","last_clean":"2024-03-28T16:01:09.543516+0000","last_became_active":"2024-03-28T16:01:08.268212+0000","last_became_peered":"2024-03-28T16:01:08.268212+0000","last_unstale":"2024-03-28T16:01:09.543516+0000","last_undegraded":"2024-03-28T16:01:09.543516+0000","last_fullsized":"2024-03-28T16:01:09.543516+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:46:22.447763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:09.245136+0000","last_change":"2024-03-28T16:01:08.265113+0000","last_active":"2024-03-28T16:01:09.245136+0000","last_peered":"2024-03-28T16:01:09.245136+0000","last_clean":"2024-03-28T16:01:09.245136+0000","last_became_active":"2024-03-28T16:01:08.264781+0000","last_became_peered":"2024-03-28T16:01:08.264781+0000","last_unstale":"2024-03-28T16:01:09.245136+0000","last_undegraded":"2024-03-28T16:01:09.245136+0000","last_fullsized":"2024-03-28T16:01:09.245136+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:19:14.720260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T16:01:10.303510+0000","last_change":"2024-03-28T16:01:08.274446+0000","last_active":"2024-03-28T16:01:10.303510+0000","last_peered":"2024-03-28T16:01:10.303510+0000","last_clean":"2024-03-28T16:01:10.303510+0000","last_became_active":"2024-03-28T16:01:08.274027+0000","last_became_peered":"2024-03-28T16:01:08.274027+0000","last_unstale":"2024-03-28T16:01:10.303510+0000","last_undegraded":"2024-03-28T16:01:10.303510+0000","last_fullsized":"2024-03-28T16:01:10.303510+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:33:37.395889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":21693,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":18,"num_read_kb":32,"num_write":52,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":83812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46,"ondisk_log_size":46,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":85,"seq":365072220176,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":966656,"data_stored":461709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":77,"seq":330712481816,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32744,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712408,"statfs":{"total":95995035648,"available":95961505792,"internally_reserved":0,"allocated":1605632,"data_stored":1073283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":71,"seq":304942678046,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":970752,"data_stored":462291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":63,"seq":270582939686,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":462313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":55,"seq":236223201325,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36868,"kb_used_data":1532,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708284,"statfs":{"total":95995035648,"available":95957282816,"internally_reserved":0,"allocated":1568768,"data_stored":1052776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397555,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36288,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708864,"statfs":{"total":95995035648,"available":95957876736,"internally_reserved":0,"allocated":966656,"data_stored":461709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574933,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":501337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738458,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36292,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708860,"statfs":{"total":95995035648,"available":95957872640,"internally_reserved":0,"allocated":970752,"data_stored":461799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411407,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":999424,"data_stored":482638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117063,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36296,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708856,"statfs":{"total":95995035648,"available":95957868544,"internally_reserved":0,"allocated":983040,"data_stored":462047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855424,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":970752,"data_stored":461731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659194,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36884,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708268,"statfs":{"total":95995035648,"available":95957266432,"internally_reserved":0,"allocated":1572864,"data_stored":1052259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":39628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":20929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":21089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T16:01:17.202 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-28T16:01:17.206 INFO:tasks.cephadm:Setting up client nodes... 2024-03-28T16:01:17.206 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-28T16:01:17.701 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T16:01:17.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:17 smithi150 ceph-mon[34825]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T16:01:17.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:17 smithi067 ceph-mon[31707]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T16:01:17.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:17 smithi183 ceph-mon[35580]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T16:01:18.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:18 smithi067 ceph-mon[31707]: pgmap v369: 97 pgs: 97 active+clean; 598 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 4.2 KiB/s wr, 8 op/s 2024-03-28T16:01:18.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:18 smithi067 ceph-mon[31707]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T16:01:18.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:18 smithi183 ceph-mon[35580]: pgmap v369: 97 pgs: 97 active+clean; 598 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 4.2 KiB/s wr, 8 op/s 2024-03-28T16:01:18.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:18 smithi183 ceph-mon[35580]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T16:01:19.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:18 smithi150 ceph-mon[34825]: pgmap v369: 97 pgs: 97 active+clean; 598 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 4.2 KiB/s wr, 8 op/s 2024-03-28T16:01:19.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:18 smithi150 ceph-mon[34825]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T16:01:19.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:19.823+0000 7f5caac00700 1 -- 172.21.15.67:0/2987374992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ca4101de0 msgr2=0x7f5ca4102260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:19.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:19.823+0000 7f5caac00700 1 --2- 172.21.15.67:0/2987374992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ca4101de0 0x7f5ca4102260 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f5c9c00b0a0 tx=0x7f5c9c00b3b0 comp rx=0 tx=0).stop 2024-03-28T16:01:19.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:19.823+0000 7f5caac00700 1 -- 172.21.15.67:0/2987374992 shutdown_connections 2024-03-28T16:01:19.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:19.823+0000 7f5caac00700 1 --2- 172.21.15.67:0/2987374992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ca4101de0 0x7f5ca4102260 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:19.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:19.823+0000 7f5caac00700 1 --2- 172.21.15.67:0/2987374992 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ca4100ac0 0x7f5ca4100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:19.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:19.823+0000 7f5caac00700 1 --2- 172.21.15.67:0/2987374992 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5ca4101420 0x7f5ca41018a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:19.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:19.823+0000 7f5caac00700 1 -- 172.21.15.67:0/2987374992 >> 172.21.15.67:0/2987374992 conn(0x7f5ca40fc680 msgr2=0x7f5ca40feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:19.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:19.823+0000 7f5caac00700 1 -- 172.21.15.67:0/2987374992 shutdown_connections 2024-03-28T16:01:19.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:19.823+0000 7f5caac00700 1 -- 172.21.15.67:0/2987374992 wait complete. 2024-03-28T16:01:19.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:19.824+0000 7f5caac00700 1 Processor -- start 2024-03-28T16:01:19.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:19.825+0000 7f5caac00700 1 -- start start 2024-03-28T16:01:19.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:19.825+0000 7f5caac00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5ca4100ac0 0x7f5ca419a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:19.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:19.825+0000 7f5caac00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ca4101420 0x7f5ca419a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:19.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:19.825+0000 7f5caac00700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ca4101de0 0x7f5ca41a19c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:19.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:19.825+0000 7f5caac00700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca410feb0 con 0x7f5ca4101420 2024-03-28T16:01:19.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:19.825+0000 7f5caac00700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca410fd30 con 0x7f5ca4101de0 2024-03-28T16:01:19.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:19.825+0000 7f5caac00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca4110030 con 0x7f5ca4100ac0 2024-03-28T16:01:19.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:19.825+0000 7f5ca3400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ca4101420 0x7f5ca419a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:19.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:19.825+0000 7f5ca3400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ca4101420 0x7f5ca419a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40962/0 (socket says 172.21.15.67:40962) 2024-03-28T16:01:19.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:19.825+0000 7f5ca3400700 1 -- 172.21.15.67:0/125214524 learned_addr learned my addr 172.21.15.67:0/125214524 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:01:19.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:19.826+0000 7f5ca3e00700 1 --2- 172.21.15.67:0/125214524 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5ca4100ac0 0x7f5ca419a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:19.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:19.826+0000 7f5ca8c00700 1 --2- 172.21.15.67:0/125214524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ca4101de0 0x7f5ca41a19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:19.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:19.826+0000 7f5ca3400700 1 -- 172.21.15.67:0/125214524 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5ca4100ac0 msgr2=0x7f5ca419a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:19.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:19.826+0000 7f5ca3400700 1 --2- 172.21.15.67:0/125214524 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5ca4100ac0 0x7f5ca419a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:19.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:19.826+0000 7f5ca3400700 1 -- 172.21.15.67:0/125214524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ca4101de0 msgr2=0x7f5ca41a19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:19.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:19.826+0000 7f5ca3400700 1 --2- 172.21.15.67:0/125214524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ca4101de0 0x7f5ca41a19c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:19.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:19.826+0000 7f5ca3400700 1 -- 172.21.15.67:0/125214524 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ca41a2060 con 0x7f5ca4101420 2024-03-28T16:01:19.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:19.826+0000 7f5ca3400700 1 --2- 172.21.15.67:0/125214524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ca4101420 0x7f5ca419a940 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f5c9000d9b0 tx=0x7f5c9000dd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:19.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:19.827+0000 7f5ca0c00700 1 -- 172.21.15.67:0/125214524 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c90017070 con 0x7f5ca4101420 2024-03-28T16:01:19.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:19.827+0000 7f5ca0c00700 1 -- 172.21.15.67:0/125214524 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5c90004830 con 0x7f5ca4101420 2024-03-28T16:01:19.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:19.827+0000 7f5caac00700 1 -- 172.21.15.67:0/125214524 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ca41a2350 con 0x7f5ca4101420 2024-03-28T16:01:19.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:19.827+0000 7f5ca0c00700 1 -- 172.21.15.67:0/125214524 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c900053a0 con 0x7f5ca4101420 2024-03-28T16:01:19.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:19.827+0000 7f5caac00700 1 -- 172.21.15.67:0/125214524 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ca41a2840 con 0x7f5ca4101420 2024-03-28T16:01:19.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:19.828+0000 7f5caac00700 1 -- 172.21.15.67:0/125214524 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ca40506a0 con 0x7f5ca4101420 2024-03-28T16:01:19.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:19.830+0000 7f5ca0c00700 1 -- 172.21.15.67:0/125214524 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f5c900049e0 con 0x7f5ca4101420 2024-03-28T16:01:19.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:19.831+0000 7f5ca0c00700 1 --2- 172.21.15.67:0/125214524 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f5c880a29d0 0x7f5c880a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:19.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:19.831+0000 7f5ca0c00700 1 -- 172.21.15.67:0/125214524 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f5c900cea70 con 0x7f5ca4101420 2024-03-28T16:01:19.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:19.832+0000 7f5ca3e00700 1 --2- 172.21.15.67:0/125214524 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f5c880a29d0 0x7f5c880a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:19.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:19.832+0000 7f5ca3e00700 1 --2- 172.21.15.67:0/125214524 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f5c880a29d0 0x7f5c880a4e90 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f5c94005fd0 tx=0x7f5c9400a560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:19.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:19.836+0000 7f5ca0c00700 1 -- 172.21.15.67:0/125214524 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5c90020020 con 0x7f5ca4101420 2024-03-28T16:01:20.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:20.162+0000 7f5caac00700 1 -- 172.21.15.67:0/125214524 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f5ca402dad0 con 0x7f5ca4101420 2024-03-28T16:01:20.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:20.167+0000 7f5ca0c00700 1 -- 172.21.15.67:0/125214524 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v33) v1 ==== 170+0+59 (secure 0 0 0) 0x7f5ca402dad0 con 0x7f5ca4101420 2024-03-28T16:01:20.168 INFO:teuthology.orchestra.run.smithi067.stdout:[client.0] 2024-03-28T16:01:20.168 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQBQlAVmD5y/CRAAOVxuD8Xoit4We0NYGp+7AA== 2024-03-28T16:01:20.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:20.170+0000 7f5caac00700 1 -- 172.21.15.67:0/125214524 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f5c880a29d0 msgr2=0x7f5c880a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:20.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:20.170+0000 7f5caac00700 1 --2- 172.21.15.67:0/125214524 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f5c880a29d0 0x7f5c880a4e90 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f5c94005fd0 tx=0x7f5c9400a560 comp rx=0 tx=0).stop 2024-03-28T16:01:20.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:20.170+0000 7f5caac00700 1 -- 172.21.15.67:0/125214524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ca4101420 msgr2=0x7f5ca419a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:20.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:20.170+0000 7f5caac00700 1 --2- 172.21.15.67:0/125214524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ca4101420 0x7f5ca419a940 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f5c9000d9b0 tx=0x7f5c9000dd70 comp rx=0 tx=0).stop 2024-03-28T16:01:20.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:20.170+0000 7f5caac00700 1 -- 172.21.15.67:0/125214524 shutdown_connections 2024-03-28T16:01:20.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:20.170+0000 7f5caac00700 1 --2- 172.21.15.67:0/125214524 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f5c880a29d0 0x7f5c880a4e90 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:20.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:20.170+0000 7f5caac00700 1 --2- 172.21.15.67:0/125214524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ca4101de0 0x7f5ca41a19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:20.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:20.170+0000 7f5caac00700 1 --2- 172.21.15.67:0/125214524 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5ca4100ac0 0x7f5ca419a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:20.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:20.170+0000 7f5caac00700 1 --2- 172.21.15.67:0/125214524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ca4101420 0x7f5ca419a940 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:20.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:20.170+0000 7f5caac00700 1 -- 172.21.15.67:0/125214524 >> 172.21.15.67:0/125214524 conn(0x7f5ca40fc680 msgr2=0x7f5ca40fe180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:20.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:20.170+0000 7f5caac00700 1 -- 172.21.15.67:0/125214524 shutdown_connections 2024-03-28T16:01:20.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:20.171+0000 7f5caac00700 1 -- 172.21.15.67:0/125214524 wait complete. 2024-03-28T16:01:20.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:20 smithi067 ceph-mon[31707]: pgmap v370: 97 pgs: 97 active+clean; 598 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 4.1 KiB/s wr, 7 op/s 2024-03-28T16:01:20.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:20 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/125214524' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-28T16:01:20.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:20 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/125214524' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-28T16:01:20.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:20 smithi183 ceph-mon[35580]: pgmap v370: 97 pgs: 97 active+clean; 598 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 4.1 KiB/s wr, 7 op/s 2024-03-28T16:01:20.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:20 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/125214524' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-28T16:01:20.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:20 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/125214524' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-28T16:01:20.930 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-28T16:01:20.930 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-28T16:01:20.930 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-28T16:01:21.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:20 smithi150 ceph-mon[34825]: pgmap v370: 97 pgs: 97 active+clean; 598 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 4.1 KiB/s wr, 7 op/s 2024-03-28T16:01:21.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:20 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/125214524' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-28T16:01:21.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:20 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/125214524' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-28T16:01:21.100 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-28T16:01:21.100 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-28T16:01:21.101 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph mgr dump --format=json 2024-03-28T16:01:21.620 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T16:01:22.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:22 smithi067 ceph-mon[31707]: pgmap v371: 97 pgs: 97 active+clean; 621 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 5.5 KiB/s wr, 7 op/s 2024-03-28T16:01:22.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:22 smithi183 ceph-mon[35580]: pgmap v371: 97 pgs: 97 active+clean; 621 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 5.5 KiB/s wr, 7 op/s 2024-03-28T16:01:23.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:22 smithi150 ceph-mon[34825]: pgmap v371: 97 pgs: 97 active+clean; 621 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 5.5 KiB/s wr, 7 op/s 2024-03-28T16:01:23.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:23.729+0000 7f0c47a00700 1 -- 172.21.15.67:0/2208410790 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c401087a0 msgr2=0x7f0c40108c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:23.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:23.729+0000 7f0c47a00700 1 --2- 172.21.15.67:0/2208410790 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c401087a0 0x7f0c40108c20 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c009990 tx=0x7f0c2c009ca0 comp rx=0 tx=0).stop 2024-03-28T16:01:23.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:23.729+0000 7f0c47a00700 1 -- 172.21.15.67:0/2208410790 shutdown_connections 2024-03-28T16:01:23.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:23.729+0000 7f0c47a00700 1 --2- 172.21.15.67:0/2208410790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0c40109160 0x7f0c40115670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:23.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:23.729+0000 7f0c47a00700 1 --2- 172.21.15.67:0/2208410790 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0c4010e7a0 0x7f0c4010eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:23.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:23.729+0000 7f0c47a00700 1 --2- 172.21.15.67:0/2208410790 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c401087a0 0x7f0c40108c20 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:23.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:23.729+0000 7f0c47a00700 1 -- 172.21.15.67:0/2208410790 >> 172.21.15.67:0/2208410790 conn(0x7f0c40075960 msgr2=0x7f0c40077d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:23.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:23.729+0000 7f0c47a00700 1 -- 172.21.15.67:0/2208410790 shutdown_connections 2024-03-28T16:01:23.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:23.729+0000 7f0c47a00700 1 -- 172.21.15.67:0/2208410790 wait complete. 2024-03-28T16:01:23.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:23.730+0000 7f0c47a00700 1 Processor -- start 2024-03-28T16:01:23.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:23.731+0000 7f0c47a00700 1 -- start start 2024-03-28T16:01:23.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:23.731+0000 7f0c47a00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0c401087a0 0x7f0c4019edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:23.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:23.731+0000 7f0c47a00700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0c40109160 0x7f0c4019f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:23.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:23.731+0000 7f0c47a00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c4010e7a0 0x7f0c401a36c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:23.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:23.731+0000 7f0c47a00700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c401189a0 con 0x7f0c4010e7a0 2024-03-28T16:01:23.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:23.731+0000 7f0c47a00700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c40118820 con 0x7f0c40109160 2024-03-28T16:01:23.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:23.731+0000 7f0c47a00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c40118b20 con 0x7f0c401087a0 2024-03-28T16:01:23.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:23.731+0000 7f0c45a00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c4010e7a0 0x7f0c401a36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:23.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:23.731+0000 7f0c45a00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c4010e7a0 0x7f0c401a36c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40984/0 (socket says 172.21.15.67:40984) 2024-03-28T16:01:23.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:23.731+0000 7f0c45a00700 1 -- 172.21.15.67:0/4259117064 learned_addr learned my addr 172.21.15.67:0/4259117064 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:01:23.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:23.732+0000 7f0c45000700 1 --2- 172.21.15.67:0/4259117064 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0c401087a0 0x7f0c4019edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:23.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:23.732+0000 7f0c3fe00700 1 --2- 172.21.15.67:0/4259117064 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0c40109160 0x7f0c4019f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:23.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:23.732+0000 7f0c45a00700 1 -- 172.21.15.67:0/4259117064 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0c401087a0 msgr2=0x7f0c4019edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:23.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:23.732+0000 7f0c45a00700 1 --2- 172.21.15.67:0/4259117064 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0c401087a0 0x7f0c4019edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:23.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:23.732+0000 7f0c45a00700 1 -- 172.21.15.67:0/4259117064 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0c40109160 msgr2=0x7f0c4019f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:23.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:23.732+0000 7f0c45a00700 1 --2- 172.21.15.67:0/4259117064 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0c40109160 0x7f0c4019f330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:23.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:23.732+0000 7f0c45a00700 1 -- 172.21.15.67:0/4259117064 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c401a3e00 con 0x7f0c4010e7a0 2024-03-28T16:01:23.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:23.732+0000 7f0c45a00700 1 --2- 172.21.15.67:0/4259117064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c4010e7a0 0x7f0c401a36c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f0c38007ee0 tx=0x7f0c3800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:23.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:23.733+0000 7f0c3d600700 1 -- 172.21.15.67:0/4259117064 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c3800dbd0 con 0x7f0c4010e7a0 2024-03-28T16:01:23.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:23.733+0000 7f0c3d600700 1 -- 172.21.15.67:0/4259117064 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0c3800dd70 con 0x7f0c4010e7a0 2024-03-28T16:01:23.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:23.733+0000 7f0c47a00700 1 -- 172.21.15.67:0/4259117064 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c401a40f0 con 0x7f0c4010e7a0 2024-03-28T16:01:23.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:23.733+0000 7f0c3d600700 1 -- 172.21.15.67:0/4259117064 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c38005a20 con 0x7f0c4010e7a0 2024-03-28T16:01:23.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:23.733+0000 7f0c47a00700 1 -- 172.21.15.67:0/4259117064 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c401ab970 con 0x7f0c4010e7a0 2024-03-28T16:01:23.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:23.734+0000 7f0c47a00700 1 -- 172.21.15.67:0/4259117064 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c40068e50 con 0x7f0c4010e7a0 2024-03-28T16:01:23.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:23.736+0000 7f0c3d600700 1 -- 172.21.15.67:0/4259117064 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f0c3801a020 con 0x7f0c4010e7a0 2024-03-28T16:01:23.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:23.737+0000 7f0c3d600700 1 --2- 172.21.15.67:0/4259117064 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f0c280a29a0 0x7f0c280a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:23.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:23.738+0000 7f0c3d600700 1 -- 172.21.15.67:0/4259117064 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f0c380ce310 con 0x7f0c4010e7a0 2024-03-28T16:01:23.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:23.738+0000 7f0c45000700 1 --2- 172.21.15.67:0/4259117064 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f0c280a29a0 0x7f0c280a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:23.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:23.738+0000 7f0c45000700 1 --2- 172.21.15.67:0/4259117064 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f0c280a29a0 0x7f0c280a4e60 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f0c3400bbb0 tx=0x7f0c3400b590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:23.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:23.742+0000 7f0c3d600700 1 -- 172.21.15.67:0/4259117064 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0c38010070 con 0x7f0c4010e7a0 2024-03-28T16:01:24.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:24.034+0000 7f0c47a00700 1 -- 172.21.15.67:0/4259117064 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f0c4002da30 con 0x7f0c4010e7a0 2024-03-28T16:01:24.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:24.048+0000 7f0c3d600700 1 -- 172.21.15.67:0/4259117064 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) v1 ==== 74+0+262656 (secure 0 0 0) 0x7f0c3809b300 con 0x7f0c4010e7a0 2024-03-28T16:01:24.051 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:01:24.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:24.053+0000 7f0c47a00700 1 -- 172.21.15.67:0/4259117064 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f0c280a29a0 msgr2=0x7f0c280a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:24.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:24.053+0000 7f0c47a00700 1 --2- 172.21.15.67:0/4259117064 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f0c280a29a0 0x7f0c280a4e60 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f0c3400bbb0 tx=0x7f0c3400b590 comp rx=0 tx=0).stop 2024-03-28T16:01:24.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:24.053+0000 7f0c47a00700 1 -- 172.21.15.67:0/4259117064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c4010e7a0 msgr2=0x7f0c401a36c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:24.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:24.053+0000 7f0c47a00700 1 --2- 172.21.15.67:0/4259117064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c4010e7a0 0x7f0c401a36c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f0c38007ee0 tx=0x7f0c3800d3b0 comp rx=0 tx=0).stop 2024-03-28T16:01:24.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:24.054+0000 7f0c47a00700 1 -- 172.21.15.67:0/4259117064 shutdown_connections 2024-03-28T16:01:24.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:24.054+0000 7f0c47a00700 1 --2- 172.21.15.67:0/4259117064 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f0c280a29a0 0x7f0c280a4e60 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:24.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:24.054+0000 7f0c47a00700 1 --2- 172.21.15.67:0/4259117064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c4010e7a0 0x7f0c401a36c0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:24.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:24.054+0000 7f0c47a00700 1 --2- 172.21.15.67:0/4259117064 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0c401087a0 0x7f0c4019edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:24.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:24.054+0000 7f0c47a00700 1 --2- 172.21.15.67:0/4259117064 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0c40109160 0x7f0c4019f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:24.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:24.054+0000 7f0c47a00700 1 -- 172.21.15.67:0/4259117064 >> 172.21.15.67:0/4259117064 conn(0x7f0c40075960 msgr2=0x7f0c40076040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:24.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:24.054+0000 7f0c47a00700 1 -- 172.21.15.67:0/4259117064 shutdown_connections 2024-03-28T16:01:24.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:24.054+0000 7f0c47a00700 1 -- 172.21.15.67:0/4259117064 wait complete. 2024-03-28T16:01:24.801 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":208726654},{"type":"v1","addr":"172.21.15.67:6801","nonce":208726654}]},"active_addr":"172.21.15.67:6801/208726654","active_change":"2024-03-28T15:51:38.875557+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":34103,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":34109,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.67:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.67:0","nonce":457469350}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.67:0","nonce":1633803673}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.67:0","nonce":3523948111}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.67:0","nonce":2423556346}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.67:0","nonce":514673831}]}]} 2024-03-28T16:01:24.807 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-28T16:01:24.807 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-28T16:01:24.807 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-28T16:01:24.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:24 smithi183 ceph-mon[35580]: pgmap v372: 97 pgs: 97 active+clean; 621 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 4.5 KiB/s wr, 5 op/s 2024-03-28T16:01:24.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:24 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/4259117064' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-28T16:01:24.891 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:24 smithi067 ceph-mon[31707]: pgmap v372: 97 pgs: 97 active+clean; 621 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 4.5 KiB/s wr, 5 op/s 2024-03-28T16:01:24.891 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:24 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/4259117064' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-28T16:01:25.055 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:24 smithi150 ceph-mon[34825]: pgmap v372: 97 pgs: 97 active+clean; 621 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 4.5 KiB/s wr, 5 op/s 2024-03-28T16:01:25.055 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:24 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/4259117064' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-28T16:01:25.246 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T16:01:26.806 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:26 smithi067 ceph-mon[31707]: pgmap v373: 97 pgs: 97 active+clean; 636 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 5.8 KiB/s wr, 6 op/s 2024-03-28T16:01:26.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:26 smithi183 ceph-mon[35580]: pgmap v373: 97 pgs: 97 active+clean; 636 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 5.8 KiB/s wr, 6 op/s 2024-03-28T16:01:27.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:26 smithi150 ceph-mon[34825]: pgmap v373: 97 pgs: 97 active+clean; 636 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 5.8 KiB/s wr, 6 op/s 2024-03-28T16:01:27.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:27.277+0000 7fe35a000700 1 -- 172.21.15.67:0/2862332573 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3540743c0 msgr2=0x7fe3540747a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:27.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:27.277+0000 7fe35a000700 1 --2- 172.21.15.67:0/2862332573 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3540743c0 0x7fe3540747a0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fe34400b0a0 tx=0x7fe34400b3b0 comp rx=0 tx=0).stop 2024-03-28T16:01:27.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:27.277+0000 7fe35a000700 1 -- 172.21.15.67:0/2862332573 shutdown_connections 2024-03-28T16:01:27.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:27.277+0000 7fe35a000700 1 --2- 172.21.15.67:0/2862332573 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe35410ebc0 0x7fe354110fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:27.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:27.277+0000 7fe35a000700 1 --2- 172.21.15.67:0/2862332573 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3540743c0 0x7fe3540747a0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:27.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:27.277+0000 7fe35a000700 1 --2- 172.21.15.67:0/2862332573 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe354074ce0 0x7fe35410e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:27.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:27.277+0000 7fe35a000700 1 -- 172.21.15.67:0/2862332573 >> 172.21.15.67:0/2862332573 conn(0x7fe3540792e0 msgr2=0x7fe3540796f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:27.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:27.277+0000 7fe35a000700 1 -- 172.21.15.67:0/2862332573 shutdown_connections 2024-03-28T16:01:27.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:27.277+0000 7fe35a000700 1 -- 172.21.15.67:0/2862332573 wait complete. 2024-03-28T16:01:27.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:27.278+0000 7fe35a000700 1 Processor -- start 2024-03-28T16:01:27.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:27.279+0000 7fe35a000700 1 -- start start 2024-03-28T16:01:27.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:27.279+0000 7fe35a000700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3540743c0 0x7fe3541a0760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:27.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:27.279+0000 7fe35a000700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe354074ce0 0x7fe3541a0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:27.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:27.279+0000 7fe35a000700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe35410ebc0 0x7fe35419a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:27.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:27.279+0000 7fe35a000700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe354114640 con 0x7fe35410ebc0 2024-03-28T16:01:27.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:27.279+0000 7fe35a000700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3541144c0 con 0x7fe354074ce0 2024-03-28T16:01:27.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:27.279+0000 7fe35a000700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3541147c0 con 0x7fe3540743c0 2024-03-28T16:01:27.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:27.279+0000 7fe353e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe35410ebc0 0x7fe35419a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:27.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:27.279+0000 7fe353e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe35410ebc0 0x7fe35419a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41002/0 (socket says 172.21.15.67:41002) 2024-03-28T16:01:27.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:27.279+0000 7fe353e00700 1 -- 172.21.15.67:0/3584876819 learned_addr learned my addr 172.21.15.67:0/3584876819 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:01:27.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:27.280+0000 7fe353400700 1 --2- 172.21.15.67:0/3584876819 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3540743c0 0x7fe3541a0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:27.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:27.280+0000 7fe34aa00700 1 --2- 172.21.15.67:0/3584876819 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe354074ce0 0x7fe3541a0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:27.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:27.280+0000 7fe353e00700 1 -- 172.21.15.67:0/3584876819 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3540743c0 msgr2=0x7fe3541a0760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:27.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:27.280+0000 7fe353e00700 1 --2- 172.21.15.67:0/3584876819 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3540743c0 0x7fe3541a0760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:27.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:27.280+0000 7fe353e00700 1 -- 172.21.15.67:0/3584876819 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe354074ce0 msgr2=0x7fe3541a0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:27.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:27.280+0000 7fe353e00700 1 --2- 172.21.15.67:0/3584876819 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe354074ce0 0x7fe3541a0ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:27.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:27.280+0000 7fe353e00700 1 -- 172.21.15.67:0/3584876819 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe35419b080 con 0x7fe35410ebc0 2024-03-28T16:01:27.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:27.280+0000 7fe353400700 1 --2- 172.21.15.67:0/3584876819 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3540743c0 0x7fe3541a0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:01:27.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:27.280+0000 7fe34aa00700 1 --2- 172.21.15.67:0/3584876819 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe354074ce0 0x7fe3541a0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:01:27.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:27.280+0000 7fe353e00700 1 --2- 172.21.15.67:0/3584876819 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe35410ebc0 0x7fe35419a920 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fe34c00d6a0 tx=0x7fe34c00da60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:27.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:27.280+0000 7fe350c00700 1 -- 172.21.15.67:0/3584876819 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe34c017070 con 0x7fe35410ebc0 2024-03-28T16:01:27.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:27.281+0000 7fe350c00700 1 -- 172.21.15.67:0/3584876819 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe34c005020 con 0x7fe35410ebc0 2024-03-28T16:01:27.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:27.281+0000 7fe350c00700 1 -- 172.21.15.67:0/3584876819 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe34c00fab0 con 0x7fe35410ebc0 2024-03-28T16:01:27.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:27.281+0000 7fe35a000700 1 -- 172.21.15.67:0/3584876819 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe35419b370 con 0x7fe35410ebc0 2024-03-28T16:01:27.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:27.281+0000 7fe35a000700 1 -- 172.21.15.67:0/3584876819 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3541a7630 con 0x7fe35410ebc0 2024-03-28T16:01:27.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:27.282+0000 7fe35a000700 1 -- 172.21.15.67:0/3584876819 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe354068e50 con 0x7fe35410ebc0 2024-03-28T16:01:27.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:27.284+0000 7fe350c00700 1 -- 172.21.15.67:0/3584876819 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fe34c0051d0 con 0x7fe35410ebc0 2024-03-28T16:01:27.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:27.285+0000 7fe350c00700 1 --2- 172.21.15.67:0/3584876819 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fe3380a29a0 0x7fe3380a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:27.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:27.286+0000 7fe350c00700 1 -- 172.21.15.67:0/3584876819 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fe34c0ce140 con 0x7fe35410ebc0 2024-03-28T16:01:27.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:27.286+0000 7fe353400700 1 --2- 172.21.15.67:0/3584876819 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fe3380a29a0 0x7fe3380a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:27.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:27.286+0000 7fe353400700 1 --2- 172.21.15.67:0/3584876819 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fe3380a29a0 0x7fe3380a4e60 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe34400b830 tx=0x7fe344006210 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:27.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:27.290+0000 7fe350c00700 1 -- 172.21.15.67:0/3584876819 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe34c09b230 con 0x7fe35410ebc0 2024-03-28T16:01:27.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:27.519+0000 7fe35a000700 1 -- 172.21.15.67:0/3584876819 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe3541a78c0 con 0x7fe35410ebc0 2024-03-28T16:01:27.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:27.525+0000 7fe350c00700 1 -- 172.21.15.67:0/3584876819 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+22221 (secure 0 0 0) 0x7fe34c09ab30 con 0x7fe35410ebc0 2024-03-28T16:01:27.526 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:01:27.527 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":90,"fsid":"b5f8225a-ed1a-11ee-95d0-87774f69a715","created":"2024-03-28T15:50:21.471924+0000","modified":"2024-03-28T16:01:09.230050+0000","last_up_change":"2024-03-28T16:00:01.369123+0000","last_in_change":"2024-03-28T15:59:37.515483+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T15:54:55.588007+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T16:01:05.719801+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T16:01:07.206303+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7c1791b6-9c0e-4512-a327-cad166bb5581","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6803","nonce":1894681701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6805","nonce":1894681701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6809","nonce":1894681701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6807","nonce":1894681701}]},"public_addr":"172.21.15.67:6803/1894681701","cluster_addr":"172.21.15.67:6805/1894681701","heartbeat_back_addr":"172.21.15.67:6809/1894681701","heartbeat_front_addr":"172.21.15.67:6807/1894681701","state":["exists","up"]},{"osd":1,"uuid":"0cdda7f3-d00c-4436-bcc1-49fb8072b2e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6801","nonce":1734465762}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6803","nonce":1734465762}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6807","nonce":1734465762}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6805","nonce":1734465762}]},"public_addr":"172.21.15.150:6801/1734465762","cluster_addr":"172.21.15.150:6803/1734465762","heartbeat_back_addr":"172.21.15.150:6807/1734465762","heartbeat_front_addr":"172.21.15.150:6805/1734465762","state":["exists","up"]},{"osd":2,"uuid":"a55b9a02-c3fd-4061-95c0-53a24a17ade0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6801","nonce":3398095931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6803","nonce":3398095931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6807","nonce":3398095931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6805","nonce":3398095931}]},"public_addr":"172.21.15.183:6801/3398095931","cluster_addr":"172.21.15.183:6803/3398095931","heartbeat_back_addr":"172.21.15.183:6807/3398095931","heartbeat_front_addr":"172.21.15.183:6805/3398095931","state":["exists","up"]},{"osd":3,"uuid":"d34940c0-5ce9-4b8b-aa36-4872414a043f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6811","nonce":1933541875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6813","nonce":1933541875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6817","nonce":1933541875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6815","nonce":1933541875}]},"public_addr":"172.21.15.67:6811/1933541875","cluster_addr":"172.21.15.67:6813/1933541875","heartbeat_back_addr":"172.21.15.67:6817/1933541875","heartbeat_front_addr":"172.21.15.67:6815/1933541875","state":["exists","up"]},{"osd":4,"uuid":"5cc3e260-83a3-4a4d-aa6a-34a3b12b2b5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6809","nonce":3775269993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6811","nonce":3775269993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6815","nonce":3775269993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6813","nonce":3775269993}]},"public_addr":"172.21.15.150:6809/3775269993","cluster_addr":"172.21.15.150:6811/3775269993","heartbeat_back_addr":"172.21.15.150:6815/3775269993","heartbeat_front_addr":"172.21.15.150:6813/3775269993","state":["exists","up"]},{"osd":5,"uuid":"a2542b21-d6ac-413e-8b9b-4a6548b20840","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6809","nonce":164284137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6811","nonce":164284137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6815","nonce":164284137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6813","nonce":164284137}]},"public_addr":"172.21.15.183:6809/164284137","cluster_addr":"172.21.15.183:6811/164284137","heartbeat_back_addr":"172.21.15.183:6815/164284137","heartbeat_front_addr":"172.21.15.183:6813/164284137","state":["exists","up"]},{"osd":6,"uuid":"8905a2d3-0f1b-4853-a218-22da2635150a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6819","nonce":3565528013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6821","nonce":3565528013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6825","nonce":3565528013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6823","nonce":3565528013}]},"public_addr":"172.21.15.67:6819/3565528013","cluster_addr":"172.21.15.67:6821/3565528013","heartbeat_back_addr":"172.21.15.67:6825/3565528013","heartbeat_front_addr":"172.21.15.67:6823/3565528013","state":["exists","up"]},{"osd":7,"uuid":"b275055b-b451-4180-91b1-e6e61a828c48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6817","nonce":3735115340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6819","nonce":3735115340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6823","nonce":3735115340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6821","nonce":3735115340}]},"public_addr":"172.21.15.150:6817/3735115340","cluster_addr":"172.21.15.150:6819/3735115340","heartbeat_back_addr":"172.21.15.150:6823/3735115340","heartbeat_front_addr":"172.21.15.150:6821/3735115340","state":["exists","up"]},{"osd":8,"uuid":"b6abf7ad-eb34-4f60-a4eb-348a92d7195b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6817","nonce":1025400095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6819","nonce":1025400095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6823","nonce":1025400095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6821","nonce":1025400095}]},"public_addr":"172.21.15.183:6817/1025400095","cluster_addr":"172.21.15.183:6819/1025400095","heartbeat_back_addr":"172.21.15.183:6823/1025400095","heartbeat_front_addr":"172.21.15.183:6821/1025400095","state":["exists","up"]},{"osd":9,"uuid":"28a78e3a-0657-40aa-816f-16a6fadadcbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6827","nonce":76885409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6829","nonce":76885409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6833","nonce":76885409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6831","nonce":76885409}]},"public_addr":"172.21.15.67:6827/76885409","cluster_addr":"172.21.15.67:6829/76885409","heartbeat_back_addr":"172.21.15.67:6833/76885409","heartbeat_front_addr":"172.21.15.67:6831/76885409","state":["exists","up"]},{"osd":10,"uuid":"d287907e-424f-4b49-a06e-4613c6de0474","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6825","nonce":3489982080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6827","nonce":3489982080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6831","nonce":3489982080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6829","nonce":3489982080}]},"public_addr":"172.21.15.150:6825/3489982080","cluster_addr":"172.21.15.150:6827/3489982080","heartbeat_back_addr":"172.21.15.150:6831/3489982080","heartbeat_front_addr":"172.21.15.150:6829/3489982080","state":["exists","up"]},{"osd":11,"uuid":"a3c29c47-a228-4f42-a23c-fbdb0c093699","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":85,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6825","nonce":1892639849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6826","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6827","nonce":1892639849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6831","nonce":1892639849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6829","nonce":1892639849}]},"public_addr":"172.21.15.183:6825/1892639849","cluster_addr":"172.21.15.183:6827/1892639849","heartbeat_back_addr":"172.21.15.183:6831/1892639849","heartbeat_front_addr":"172.21.15.183:6829/1892639849","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:53:45.646211+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:54:18.956590+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:54:52.161646+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:55:27.502822+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:56:01.162932+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:56:34.674452+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:57:09.298679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:57:43.230406+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:58:17.003821+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:58:51.085863+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:59:25.665065+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T16:00:00.263457+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6801/1631001968":"2024-03-29T15:51:38.875334+0000","172.21.15.67:6800/1631001968":"2024-03-29T15:51:38.875334+0000","172.21.15.67:0/1533056310":"2024-03-29T15:51:38.875334+0000","172.21.15.67:6801/3580491071":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/2030412458":"2024-03-29T15:51:03.859740+0000","172.21.15.67:6800/3580491071":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/2905307533":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/3763759988":"2024-03-29T15:51:38.875334+0000","172.21.15.67:0/4089399880":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/4239108661":"2024-03-29T15:51:38.875334+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T16:01:27.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:27.529+0000 7fe35a000700 1 -- 172.21.15.67:0/3584876819 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fe3380a29a0 msgr2=0x7fe3380a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:27.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:27.529+0000 7fe35a000700 1 --2- 172.21.15.67:0/3584876819 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fe3380a29a0 0x7fe3380a4e60 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe34400b830 tx=0x7fe344006210 comp rx=0 tx=0).stop 2024-03-28T16:01:27.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:27.529+0000 7fe35a000700 1 -- 172.21.15.67:0/3584876819 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe35410ebc0 msgr2=0x7fe35419a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:27.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:27.529+0000 7fe35a000700 1 --2- 172.21.15.67:0/3584876819 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe35410ebc0 0x7fe35419a920 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fe34c00d6a0 tx=0x7fe34c00da60 comp rx=0 tx=0).stop 2024-03-28T16:01:27.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:27.529+0000 7fe35a000700 1 -- 172.21.15.67:0/3584876819 shutdown_connections 2024-03-28T16:01:27.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:27.529+0000 7fe35a000700 1 --2- 172.21.15.67:0/3584876819 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fe3380a29a0 0x7fe3380a4e60 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:27.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:27.529+0000 7fe35a000700 1 --2- 172.21.15.67:0/3584876819 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe35410ebc0 0x7fe35419a920 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:27.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:27.529+0000 7fe35a000700 1 --2- 172.21.15.67:0/3584876819 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3540743c0 0x7fe3541a0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:27.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:27.529+0000 7fe35a000700 1 --2- 172.21.15.67:0/3584876819 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe354074ce0 0x7fe3541a0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:27.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:27.529+0000 7fe35a000700 1 -- 172.21.15.67:0/3584876819 >> 172.21.15.67:0/3584876819 conn(0x7fe3540792e0 msgr2=0x7fe3540ff3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:27.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:27.529+0000 7fe35a000700 1 -- 172.21.15.67:0/3584876819 shutdown_connections 2024-03-28T16:01:27.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:27.529+0000 7fe35a000700 1 -- 172.21.15.67:0/3584876819 wait complete. 2024-03-28T16:01:27.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:27 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/3584876819' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:01:27.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:27 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/3584876819' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:01:28.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:27 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/3584876819' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:01:28.287 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-28T16:01:28.287 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-28T16:01:28.758 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T16:01:28.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:28 smithi183 ceph-mon[35580]: pgmap v374: 97 pgs: 97 active+clean; 636 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 4.0 KiB/s wr, 3 op/s 2024-03-28T16:01:28.906 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:28 smithi067 ceph-mon[31707]: pgmap v374: 97 pgs: 97 active+clean; 636 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 4.0 KiB/s wr, 3 op/s 2024-03-28T16:01:29.055 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:28 smithi150 ceph-mon[34825]: pgmap v374: 97 pgs: 97 active+clean; 636 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 4.0 KiB/s wr, 3 op/s 2024-03-28T16:01:30.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:30.798+0000 7fd51a400700 1 -- 172.21.15.67:0/2918250696 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd51406b230 msgr2=0x7fd51410f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:30.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:30.798+0000 7fd51a400700 1 --2- 172.21.15.67:0/2918250696 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd51406b230 0x7fd51410f050 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fd50c00b0d0 tx=0x7fd50c00b3e0 comp rx=0 tx=0).stop 2024-03-28T16:01:30.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:30.798+0000 7fd51a400700 1 -- 172.21.15.67:0/2918250696 shutdown_connections 2024-03-28T16:01:30.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:30.798+0000 7fd51a400700 1 --2- 172.21.15.67:0/2918250696 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd51406b230 0x7fd51410f050 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:30.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:30.798+0000 7fd51a400700 1 --2- 172.21.15.67:0/2918250696 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd514108210 0x7fd5141085f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:30.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:30.798+0000 7fd51a400700 1 --2- 172.21.15.67:0/2918250696 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd51406a890 0x7fd51406acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:30.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:30.798+0000 7fd51a400700 1 -- 172.21.15.67:0/2918250696 >> 172.21.15.67:0/2918250696 conn(0x7fd5140770b0 msgr2=0x7fd5140774c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:30.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:30.798+0000 7fd51a400700 1 -- 172.21.15.67:0/2918250696 shutdown_connections 2024-03-28T16:01:30.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:30.798+0000 7fd51a400700 1 -- 172.21.15.67:0/2918250696 wait complete. 2024-03-28T16:01:30.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:30.800+0000 7fd51a400700 1 Processor -- start 2024-03-28T16:01:30.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:30.800+0000 7fd51a400700 1 -- start start 2024-03-28T16:01:30.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:30.800+0000 7fd51a400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd51406a890 0x7fd51419e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:30.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:30.800+0000 7fd51a400700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd51406b230 0x7fd51419e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:30.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:30.800+0000 7fd51a400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd514108210 0x7fd5141985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:30.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:30.800+0000 7fd51a400700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd514112340 con 0x7fd514108210 2024-03-28T16:01:30.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:30.800+0000 7fd51a400700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5141121c0 con 0x7fd51406b230 2024-03-28T16:01:30.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:30.800+0000 7fd51a400700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5141124c0 con 0x7fd51406a890 2024-03-28T16:01:30.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:30.801+0000 7fd513e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd514108210 0x7fd5141985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:30.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:30.801+0000 7fd513e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd514108210 0x7fd5141985a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40904/0 (socket says 172.21.15.67:40904) 2024-03-28T16:01:30.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:30.801+0000 7fd513e00700 1 -- 172.21.15.67:0/2330667052 learned_addr learned my addr 172.21.15.67:0/2330667052 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:01:30.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:30.801+0000 7fd513400700 1 --2- 172.21.15.67:0/2330667052 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd51406a890 0x7fd51419e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:30.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:30.801+0000 7fd512a00700 1 --2- 172.21.15.67:0/2330667052 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd51406b230 0x7fd51419e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:30.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:30.801+0000 7fd513e00700 1 -- 172.21.15.67:0/2330667052 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd51406a890 msgr2=0x7fd51419e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:30.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:30.801+0000 7fd513e00700 1 --2- 172.21.15.67:0/2330667052 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd51406a890 0x7fd51419e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:30.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:30.801+0000 7fd513e00700 1 -- 172.21.15.67:0/2330667052 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd51406b230 msgr2=0x7fd51419e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:30.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:30.801+0000 7fd513e00700 1 --2- 172.21.15.67:0/2330667052 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd51406b230 0x7fd51419e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:30.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:30.801+0000 7fd513e00700 1 -- 172.21.15.67:0/2330667052 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd514198d00 con 0x7fd514108210 2024-03-28T16:01:30.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:30.801+0000 7fd513400700 1 --2- 172.21.15.67:0/2330667052 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd51406a890 0x7fd51419e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:01:30.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:30.801+0000 7fd512a00700 1 --2- 172.21.15.67:0/2330667052 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd51406b230 0x7fd51419e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:01:30.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:30.802+0000 7fd513e00700 1 --2- 172.21.15.67:0/2330667052 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd514108210 0x7fd5141985a0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fd50c0095c0 tx=0x7fd50c007ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:30.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:30.802+0000 7fd50be00700 1 -- 172.21.15.67:0/2330667052 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd50c01f070 con 0x7fd514108210 2024-03-28T16:01:30.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:30.802+0000 7fd51a400700 1 -- 172.21.15.67:0/2330667052 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd514198f90 con 0x7fd514108210 2024-03-28T16:01:30.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:30.802+0000 7fd50be00700 1 -- 172.21.15.67:0/2330667052 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd50c003c00 con 0x7fd514108210 2024-03-28T16:01:30.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:30.802+0000 7fd50be00700 1 -- 172.21.15.67:0/2330667052 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd50c01b6a0 con 0x7fd514108210 2024-03-28T16:01:30.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:30.802+0000 7fd51a400700 1 -- 172.21.15.67:0/2330667052 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5141a52b0 con 0x7fd514108210 2024-03-28T16:01:30.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:30.803+0000 7fd51a400700 1 -- 172.21.15.67:0/2330667052 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd514068e50 con 0x7fd514108210 2024-03-28T16:01:30.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:30.805+0000 7fd50be00700 1 -- 172.21.15.67:0/2330667052 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fd50c022070 con 0x7fd514108210 2024-03-28T16:01:30.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:30.807+0000 7fd50be00700 1 --2- 172.21.15.67:0/2330667052 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fd4f80a29a0 0x7fd4f80a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:30.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:30.807+0000 7fd50be00700 1 -- 172.21.15.67:0/2330667052 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fd50c00f020 con 0x7fd514108210 2024-03-28T16:01:30.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:30.807+0000 7fd513400700 1 --2- 172.21.15.67:0/2330667052 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fd4f80a29a0 0x7fd4f80a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:30.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:30.808+0000 7fd513400700 1 --2- 172.21.15.67:0/2330667052 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fd4f80a29a0 0x7fd4f80a4e60 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd504009b50 tx=0x7fd504009550 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:30.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:30.812+0000 7fd50be00700 1 -- 172.21.15.67:0/2330667052 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd50c029020 con 0x7fd514108210 2024-03-28T16:01:30.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:30 smithi183 ceph-mon[35580]: pgmap v375: 97 pgs: 97 active+clean; 636 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 3.9 KiB/s wr, 3 op/s 2024-03-28T16:01:31.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:31.038+0000 7fd51a400700 1 -- 172.21.15.67:0/2330667052 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd514199ad0 con 0x7fd514108210 2024-03-28T16:01:31.039 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:30 smithi067 ceph-mon[31707]: pgmap v375: 97 pgs: 97 active+clean; 636 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 3.9 KiB/s wr, 3 op/s 2024-03-28T16:01:31.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:31.040+0000 7fd50be00700 1 -- 172.21.15.67:0/2330667052 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+22221 (secure 0 0 0) 0x7fd50c09cea0 con 0x7fd514108210 2024-03-28T16:01:31.041 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:01:31.042 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":90,"fsid":"b5f8225a-ed1a-11ee-95d0-87774f69a715","created":"2024-03-28T15:50:21.471924+0000","modified":"2024-03-28T16:01:09.230050+0000","last_up_change":"2024-03-28T16:00:01.369123+0000","last_in_change":"2024-03-28T15:59:37.515483+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T15:54:55.588007+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T16:01:05.719801+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T16:01:07.206303+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7c1791b6-9c0e-4512-a327-cad166bb5581","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6803","nonce":1894681701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6805","nonce":1894681701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6809","nonce":1894681701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6807","nonce":1894681701}]},"public_addr":"172.21.15.67:6803/1894681701","cluster_addr":"172.21.15.67:6805/1894681701","heartbeat_back_addr":"172.21.15.67:6809/1894681701","heartbeat_front_addr":"172.21.15.67:6807/1894681701","state":["exists","up"]},{"osd":1,"uuid":"0cdda7f3-d00c-4436-bcc1-49fb8072b2e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6801","nonce":1734465762}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6803","nonce":1734465762}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6807","nonce":1734465762}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6805","nonce":1734465762}]},"public_addr":"172.21.15.150:6801/1734465762","cluster_addr":"172.21.15.150:6803/1734465762","heartbeat_back_addr":"172.21.15.150:6807/1734465762","heartbeat_front_addr":"172.21.15.150:6805/1734465762","state":["exists","up"]},{"osd":2,"uuid":"a55b9a02-c3fd-4061-95c0-53a24a17ade0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6801","nonce":3398095931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6803","nonce":3398095931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6807","nonce":3398095931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6805","nonce":3398095931}]},"public_addr":"172.21.15.183:6801/3398095931","cluster_addr":"172.21.15.183:6803/3398095931","heartbeat_back_addr":"172.21.15.183:6807/3398095931","heartbeat_front_addr":"172.21.15.183:6805/3398095931","state":["exists","up"]},{"osd":3,"uuid":"d34940c0-5ce9-4b8b-aa36-4872414a043f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6811","nonce":1933541875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6813","nonce":1933541875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6817","nonce":1933541875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6815","nonce":1933541875}]},"public_addr":"172.21.15.67:6811/1933541875","cluster_addr":"172.21.15.67:6813/1933541875","heartbeat_back_addr":"172.21.15.67:6817/1933541875","heartbeat_front_addr":"172.21.15.67:6815/1933541875","state":["exists","up"]},{"osd":4,"uuid":"5cc3e260-83a3-4a4d-aa6a-34a3b12b2b5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6809","nonce":3775269993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6811","nonce":3775269993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6815","nonce":3775269993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6813","nonce":3775269993}]},"public_addr":"172.21.15.150:6809/3775269993","cluster_addr":"172.21.15.150:6811/3775269993","heartbeat_back_addr":"172.21.15.150:6815/3775269993","heartbeat_front_addr":"172.21.15.150:6813/3775269993","state":["exists","up"]},{"osd":5,"uuid":"a2542b21-d6ac-413e-8b9b-4a6548b20840","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6809","nonce":164284137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6811","nonce":164284137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6815","nonce":164284137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6813","nonce":164284137}]},"public_addr":"172.21.15.183:6809/164284137","cluster_addr":"172.21.15.183:6811/164284137","heartbeat_back_addr":"172.21.15.183:6815/164284137","heartbeat_front_addr":"172.21.15.183:6813/164284137","state":["exists","up"]},{"osd":6,"uuid":"8905a2d3-0f1b-4853-a218-22da2635150a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6819","nonce":3565528013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6821","nonce":3565528013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6825","nonce":3565528013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6823","nonce":3565528013}]},"public_addr":"172.21.15.67:6819/3565528013","cluster_addr":"172.21.15.67:6821/3565528013","heartbeat_back_addr":"172.21.15.67:6825/3565528013","heartbeat_front_addr":"172.21.15.67:6823/3565528013","state":["exists","up"]},{"osd":7,"uuid":"b275055b-b451-4180-91b1-e6e61a828c48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6817","nonce":3735115340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6819","nonce":3735115340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6823","nonce":3735115340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6821","nonce":3735115340}]},"public_addr":"172.21.15.150:6817/3735115340","cluster_addr":"172.21.15.150:6819/3735115340","heartbeat_back_addr":"172.21.15.150:6823/3735115340","heartbeat_front_addr":"172.21.15.150:6821/3735115340","state":["exists","up"]},{"osd":8,"uuid":"b6abf7ad-eb34-4f60-a4eb-348a92d7195b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6817","nonce":1025400095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6819","nonce":1025400095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6823","nonce":1025400095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6821","nonce":1025400095}]},"public_addr":"172.21.15.183:6817/1025400095","cluster_addr":"172.21.15.183:6819/1025400095","heartbeat_back_addr":"172.21.15.183:6823/1025400095","heartbeat_front_addr":"172.21.15.183:6821/1025400095","state":["exists","up"]},{"osd":9,"uuid":"28a78e3a-0657-40aa-816f-16a6fadadcbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6827","nonce":76885409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6829","nonce":76885409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6833","nonce":76885409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6831","nonce":76885409}]},"public_addr":"172.21.15.67:6827/76885409","cluster_addr":"172.21.15.67:6829/76885409","heartbeat_back_addr":"172.21.15.67:6833/76885409","heartbeat_front_addr":"172.21.15.67:6831/76885409","state":["exists","up"]},{"osd":10,"uuid":"d287907e-424f-4b49-a06e-4613c6de0474","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6825","nonce":3489982080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6827","nonce":3489982080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6831","nonce":3489982080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6829","nonce":3489982080}]},"public_addr":"172.21.15.150:6825/3489982080","cluster_addr":"172.21.15.150:6827/3489982080","heartbeat_back_addr":"172.21.15.150:6831/3489982080","heartbeat_front_addr":"172.21.15.150:6829/3489982080","state":["exists","up"]},{"osd":11,"uuid":"a3c29c47-a228-4f42-a23c-fbdb0c093699","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":85,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6825","nonce":1892639849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6826","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6827","nonce":1892639849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6831","nonce":1892639849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6829","nonce":1892639849}]},"public_addr":"172.21.15.183:6825/1892639849","cluster_addr":"172.21.15.183:6827/1892639849","heartbeat_back_addr":"172.21.15.183:6831/1892639849","heartbeat_front_addr":"172.21.15.183:6829/1892639849","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:53:45.646211+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:54:18.956590+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:54:52.161646+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:55:27.502822+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:56:01.162932+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:56:34.674452+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:57:09.298679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:57:43.230406+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:58:17.003821+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:58:51.085863+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:59:25.665065+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T16:00:00.263457+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6801/1631001968":"2024-03-29T15:51:38.875334+0000","172.21.15.67:6800/1631001968":"2024-03-29T15:51:38.875334+0000","172.21.15.67:0/1533056310":"2024-03-29T15:51:38.875334+0000","172.21.15.67:6801/3580491071":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/2030412458":"2024-03-29T15:51:03.859740+0000","172.21.15.67:6800/3580491071":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/2905307533":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/3763759988":"2024-03-29T15:51:38.875334+0000","172.21.15.67:0/4089399880":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/4239108661":"2024-03-29T15:51:38.875334+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T16:01:31.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:31.043+0000 7fd51a400700 1 -- 172.21.15.67:0/2330667052 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fd4f80a29a0 msgr2=0x7fd4f80a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:31.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:31.043+0000 7fd51a400700 1 --2- 172.21.15.67:0/2330667052 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fd4f80a29a0 0x7fd4f80a4e60 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd504009b50 tx=0x7fd504009550 comp rx=0 tx=0).stop 2024-03-28T16:01:31.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:31.043+0000 7fd51a400700 1 -- 172.21.15.67:0/2330667052 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd514108210 msgr2=0x7fd5141985a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:31.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:31.043+0000 7fd51a400700 1 --2- 172.21.15.67:0/2330667052 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd514108210 0x7fd5141985a0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fd50c0095c0 tx=0x7fd50c007ad0 comp rx=0 tx=0).stop 2024-03-28T16:01:31.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:31.044+0000 7fd51a400700 1 -- 172.21.15.67:0/2330667052 shutdown_connections 2024-03-28T16:01:31.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:31.044+0000 7fd51a400700 1 --2- 172.21.15.67:0/2330667052 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fd4f80a29a0 0x7fd4f80a4e60 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:31.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:31.044+0000 7fd51a400700 1 --2- 172.21.15.67:0/2330667052 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd514108210 0x7fd5141985a0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:31.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:31.044+0000 7fd51a400700 1 --2- 172.21.15.67:0/2330667052 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd51406a890 0x7fd51419e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:31.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:31.044+0000 7fd51a400700 1 --2- 172.21.15.67:0/2330667052 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd51406b230 0x7fd51419e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:31.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:31.044+0000 7fd51a400700 1 -- 172.21.15.67:0/2330667052 >> 172.21.15.67:0/2330667052 conn(0x7fd5140770b0 msgr2=0x7fd5140ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:31.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:31.044+0000 7fd51a400700 1 -- 172.21.15.67:0/2330667052 shutdown_connections 2024-03-28T16:01:31.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:31.044+0000 7fd51a400700 1 -- 172.21.15.67:0/2330667052 wait complete. 2024-03-28T16:01:31.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:30 smithi150 ceph-mon[34825]: pgmap v375: 97 pgs: 97 active+clean; 636 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 3.9 KiB/s wr, 3 op/s 2024-03-28T16:01:31.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:31 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/2330667052' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:01:31.762 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-28T16:01:31.763 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-28T16:01:31.763 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-28T16:01:31.764 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-28T16:01:31.764 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-28T16:01:31.766 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-28T16:01:31.767 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-28T16:01:31.767 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-28T16:01:31.768 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph tell osd.8 flush_pg_stats 2024-03-28T16:01:31.769 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph tell osd.9 flush_pg_stats 2024-03-28T16:01:31.769 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph tell osd.10 flush_pg_stats 2024-03-28T16:01:31.769 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph tell osd.11 flush_pg_stats 2024-03-28T16:01:31.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:31 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/2330667052' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:01:31.954 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:31 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/2330667052' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:01:32.970 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T16:01:32.979 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T16:01:32.986 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T16:01:32.988 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T16:01:32.991 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T16:01:32.996 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T16:01:32.999 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T16:01:33.001 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T16:01:33.010 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T16:01:33.010 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T16:01:33.018 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T16:01:33.024 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T16:01:33.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:32 smithi150 ceph-mon[34825]: pgmap v376: 97 pgs: 97 active+clean; 636 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 3.6 KiB/s wr, 3 op/s 2024-03-28T16:01:33.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:32 smithi067 ceph-mon[31707]: pgmap v376: 97 pgs: 97 active+clean; 636 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 3.6 KiB/s wr, 3 op/s 2024-03-28T16:01:33.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:32 smithi183 ceph-mon[35580]: pgmap v376: 97 pgs: 97 active+clean; 636 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 3.6 KiB/s wr, 3 op/s 2024-03-28T16:01:34.726 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:34 smithi067 ceph-mon[31707]: pgmap v377: 97 pgs: 97 active+clean; 636 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 1.6 KiB/s wr, 2 op/s 2024-03-28T16:01:35.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:34 smithi150 ceph-mon[34825]: pgmap v377: 97 pgs: 97 active+clean; 636 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 1.6 KiB/s wr, 2 op/s 2024-03-28T16:01:35.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:34 smithi183 ceph-mon[35580]: pgmap v377: 97 pgs: 97 active+clean; 636 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 1.6 KiB/s wr, 2 op/s 2024-03-28T16:01:37.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:36 smithi150 ceph-mon[34825]: pgmap v378: 97 pgs: 97 active+clean; 638 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.8 KiB/s wr, 3 op/s 2024-03-28T16:01:37.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:36 smithi067 ceph-mon[31707]: pgmap v378: 97 pgs: 97 active+clean; 638 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.8 KiB/s wr, 3 op/s 2024-03-28T16:01:37.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:36 smithi183 ceph-mon[35580]: pgmap v378: 97 pgs: 97 active+clean; 638 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.8 KiB/s wr, 3 op/s 2024-03-28T16:01:38.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:38 smithi067 ceph-mon[31707]: pgmap v379: 97 pgs: 97 active+clean; 638 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 1 op/s 2024-03-28T16:01:39.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:38 smithi150 ceph-mon[34825]: pgmap v379: 97 pgs: 97 active+clean; 638 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 1 op/s 2024-03-28T16:01:39.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:38 smithi183 ceph-mon[35580]: pgmap v379: 97 pgs: 97 active+clean; 638 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 1 op/s 2024-03-28T16:01:39.922 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:39 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]: dispatch 2024-03-28T16:01:39.922 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:39 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4, 8, 11]}]: dispatch 2024-03-28T16:01:39.922 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:39 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 1]}]: dispatch 2024-03-28T16:01:39.922 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:39 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.7", "id": [3, 9]}]: dispatch 2024-03-28T16:01:39.923 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:39 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]}]: dispatch 2024-03-28T16:01:39.923 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:39 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]: dispatch 2024-03-28T16:01:39.923 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:39 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]: dispatch 2024-03-28T16:01:39.923 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:39 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]: dispatch 2024-03-28T16:01:40.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:39 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]: dispatch 2024-03-28T16:01:40.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:39 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4, 8, 11]}]: dispatch 2024-03-28T16:01:40.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:39 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 1]}]: dispatch 2024-03-28T16:01:40.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:39 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.7", "id": [3, 9]}]: dispatch 2024-03-28T16:01:40.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:39 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]}]: dispatch 2024-03-28T16:01:40.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:39 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]: dispatch 2024-03-28T16:01:40.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:39 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]: dispatch 2024-03-28T16:01:40.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:39 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]: dispatch 2024-03-28T16:01:40.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:39 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]: dispatch 2024-03-28T16:01:40.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:39 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4, 8, 11]}]: dispatch 2024-03-28T16:01:40.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:39 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 1]}]: dispatch 2024-03-28T16:01:40.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:39 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.7", "id": [3, 9]}]: dispatch 2024-03-28T16:01:40.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:39 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]}]: dispatch 2024-03-28T16:01:40.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:39 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]: dispatch 2024-03-28T16:01:40.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:39 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]: dispatch 2024-03-28T16:01:40.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:39 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]: dispatch 2024-03-28T16:01:40.752 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:40 smithi067 ceph-mon[31707]: pgmap v380: 97 pgs: 97 active+clean; 638 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 1 op/s 2024-03-28T16:01:40.752 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:40 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]': finished 2024-03-28T16:01:40.752 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:40 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4, 8, 11]}]': finished 2024-03-28T16:01:40.752 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:40 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 1]}]': finished 2024-03-28T16:01:40.752 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:40 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.7", "id": [3, 9]}]': finished 2024-03-28T16:01:40.752 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:40 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]}]': finished 2024-03-28T16:01:40.752 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:40 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]': finished 2024-03-28T16:01:40.752 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:40 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]': finished 2024-03-28T16:01:40.752 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:40 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]': finished 2024-03-28T16:01:40.752 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:40 smithi067 ceph-mon[31707]: osdmap e91: 12 total, 12 up, 12 in 2024-03-28T16:01:41.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:40 smithi150 ceph-mon[34825]: pgmap v380: 97 pgs: 97 active+clean; 638 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 1 op/s 2024-03-28T16:01:41.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:40 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]': finished 2024-03-28T16:01:41.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:40 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4, 8, 11]}]': finished 2024-03-28T16:01:41.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:40 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 1]}]': finished 2024-03-28T16:01:41.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:40 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.7", "id": [3, 9]}]': finished 2024-03-28T16:01:41.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:40 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]}]': finished 2024-03-28T16:01:41.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:40 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]': finished 2024-03-28T16:01:41.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:40 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]': finished 2024-03-28T16:01:41.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:40 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]': finished 2024-03-28T16:01:41.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:40 smithi150 ceph-mon[34825]: osdmap e91: 12 total, 12 up, 12 in 2024-03-28T16:01:41.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:40 smithi183 ceph-mon[35580]: pgmap v380: 97 pgs: 97 active+clean; 638 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 1 op/s 2024-03-28T16:01:41.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:40 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]': finished 2024-03-28T16:01:41.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:40 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4, 8, 11]}]': finished 2024-03-28T16:01:41.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:40 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 1]}]': finished 2024-03-28T16:01:41.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:40 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.7", "id": [3, 9]}]': finished 2024-03-28T16:01:41.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:40 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]}]': finished 2024-03-28T16:01:41.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:40 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]': finished 2024-03-28T16:01:41.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:40 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]': finished 2024-03-28T16:01:41.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:40 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]': finished 2024-03-28T16:01:41.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:40 smithi183 ceph-mon[35580]: osdmap e91: 12 total, 12 up, 12 in 2024-03-28T16:01:41.896 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:41 smithi067 ceph-mon[31707]: osdmap e92: 12 total, 12 up, 12 in 2024-03-28T16:01:42.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:41 smithi150 ceph-mon[34825]: osdmap e92: 12 total, 12 up, 12 in 2024-03-28T16:01:42.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:41 smithi183 ceph-mon[35580]: osdmap e92: 12 total, 12 up, 12 in 2024-03-28T16:01:42.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:42.881+0000 7f119ac00700 1 -- 172.21.15.67:0/814340587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11941023d0 msgr2=0x7f11941027b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:42.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:42.881+0000 7f119ac00700 1 --2- 172.21.15.67:0/814340587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11941023d0 0x7f11941027b0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f11840099c0 tx=0x7f1184009cd0 comp rx=0 tx=0).stop 2024-03-28T16:01:42.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:42.881+0000 7f119ac00700 1 -- 172.21.15.67:0/814340587 shutdown_connections 2024-03-28T16:01:42.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:42.881+0000 7f119ac00700 1 --2- 172.21.15.67:0/814340587 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f11941074b0 0x7f119410f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:42.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:42.881+0000 7f119ac00700 1 --2- 172.21.15.67:0/814340587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11941023d0 0x7f11941027b0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:42.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:42.881+0000 7f119ac00700 1 --2- 172.21.15.67:0/814340587 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1194102d80 0x7f1194106dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:42.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:42.881+0000 7f119ac00700 1 -- 172.21.15.67:0/814340587 >> 172.21.15.67:0/814340587 conn(0x7f11940fdc40 msgr2=0x7f1194100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:42.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:42.881+0000 7f119ac00700 1 -- 172.21.15.67:0/814340587 shutdown_connections 2024-03-28T16:01:42.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:42.882+0000 7f119ac00700 1 -- 172.21.15.67:0/814340587 wait complete. 2024-03-28T16:01:42.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:42.883+0000 7f119ac00700 1 Processor -- start 2024-03-28T16:01:42.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:42.883+0000 7f119ac00700 1 -- start start 2024-03-28T16:01:42.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:42.883+0000 7f119ac00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11941023d0 0x7f119419e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:42.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:42.883+0000 7f119ac00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1194102d80 0x7f119419eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:42.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:42.883+0000 7f119ac00700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11941074b0 0x7f1194198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:42.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:42.883+0000 7f119ac00700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1194112390 con 0x7f11941023d0 2024-03-28T16:01:42.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:42.883+0000 7f119ac00700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1194112210 con 0x7f11941074b0 2024-03-28T16:01:42.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:42.883+0000 7f119ac00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1194112510 con 0x7f1194102d80 2024-03-28T16:01:42.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:42.884+0000 7f1193e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11941023d0 0x7f119419e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:42.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:42.884+0000 7f1193e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11941023d0 0x7f119419e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52182/0 (socket says 172.21.15.67:52182) 2024-03-28T16:01:42.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:42.884+0000 7f1193e00700 1 -- 172.21.15.67:0/3948365317 learned_addr learned my addr 172.21.15.67:0/3948365317 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:01:42.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:42.884+0000 7f1193400700 1 --2- 172.21.15.67:0/3948365317 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1194102d80 0x7f119419eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:42.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:42.884+0000 7f1198c00700 1 --2- 172.21.15.67:0/3948365317 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11941074b0 0x7f1194198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:42.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:42.884+0000 7f1193e00700 1 -- 172.21.15.67:0/3948365317 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1194102d80 msgr2=0x7f119419eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:42.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:42.884+0000 7f1193e00700 1 --2- 172.21.15.67:0/3948365317 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1194102d80 0x7f119419eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:42.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:42.884+0000 7f1193e00700 1 -- 172.21.15.67:0/3948365317 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11941074b0 msgr2=0x7f1194198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:42.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:42.884+0000 7f1193e00700 1 --2- 172.21.15.67:0/3948365317 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11941074b0 0x7f1194198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:42.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:42.884+0000 7f1193e00700 1 -- 172.21.15.67:0/3948365317 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1194199010 con 0x7f11941023d0 2024-03-28T16:01:42.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:42.885+0000 7f1193e00700 1 --2- 172.21.15.67:0/3948365317 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11941023d0 0x7f119419e610 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f1184006010 tx=0x7f1184003680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:42.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:42.885+0000 7f1190c00700 1 -- 172.21.15.67:0/3948365317 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1184027070 con 0x7f11941023d0 2024-03-28T16:01:42.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:42.885+0000 7f1190c00700 1 -- 172.21.15.67:0/3948365317 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1184004050 con 0x7f11941023d0 2024-03-28T16:01:42.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:42.885+0000 7f119ac00700 1 -- 172.21.15.67:0/3948365317 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11941992a0 con 0x7f11941023d0 2024-03-28T16:01:42.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:42.885+0000 7f1190c00700 1 -- 172.21.15.67:0/3948365317 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1184022e10 con 0x7f11941023d0 2024-03-28T16:01:42.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:42.885+0000 7f119ac00700 1 -- 172.21.15.67:0/3948365317 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11941a53c0 con 0x7f11941023d0 2024-03-28T16:01:42.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:42.886+0000 7f119ac00700 1 -- 172.21.15.67:0/3948365317 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f11940628e0 con 0x7f11941023d0 2024-03-28T16:01:42.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:42.888+0000 7f1190c00700 1 -- 172.21.15.67:0/3948365317 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f118402a070 con 0x7f11941023d0 2024-03-28T16:01:42.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:42.889+0000 7f1190c00700 1 --2- 172.21.15.67:0/3948365317 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f11780a2920 0x7f11780a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:42.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:42.890+0000 7f1190c00700 1 -- 172.21.15.67:0/3948365317 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f11840d05e0 con 0x7f11941023d0 2024-03-28T16:01:42.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:42.890+0000 7f1193400700 1 --2- 172.21.15.67:0/3948365317 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f11780a2920 0x7f11780a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:42.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:42.890+0000 7f1190c00700 1 --2- 172.21.15.67:0/3948365317 >> [v2:172.21.15.183:6824/1892639849,v1:172.21.15.183:6825/1892639849] conn(0x7f11780ac460 0x7f11780ae900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:42.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:42.890+0000 7f1190c00700 1 -- 172.21.15.67:0/3948365317 --> [v2:172.21.15.183:6824/1892639849,v1:172.21.15.183:6825/1892639849] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1184029e50 con 0x7f11780ac460 2024-03-28T16:01:42.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:42.890+0000 7f1190c00700 1 -- 172.21.15.67:0/3948365317 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) v2 ==== 24+0+0 (secure 0 0 0) 0x7f11840d0990 con 0x7f11941023d0 2024-03-28T16:01:42.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:42.891+0000 7f1198c00700 1 --2- 172.21.15.67:0/3948365317 >> [v2:172.21.15.183:6824/1892639849,v1:172.21.15.183:6825/1892639849] conn(0x7f11780ac460 0x7f11780ae900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:42.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:42.891+0000 7f1193400700 1 --2- 172.21.15.67:0/3948365317 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f11780a2920 0x7f11780a4de0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f1180005fd0 tx=0x7f118000a560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:42.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:42.892+0000 7f1198c00700 1 --2- 172.21.15.67:0/3948365317 >> [v2:172.21.15.183:6824/1892639849,v1:172.21.15.183:6825/1892639849] conn(0x7f11780ac460 0x7f11780ae900 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:42.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:42.894+0000 7f1190c00700 1 -- 172.21.15.67:0/3948365317 <== osd.11 v2:172.21.15.183:6824/1892639849 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f1184029e50 con 0x7f11780ac460 2024-03-28T16:01:42.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:42.913+0000 7f119ac00700 1 -- 172.21.15.67:0/3948365317 --> [v2:172.21.15.183:6824/1892639849,v1:172.21.15.183:6825/1892639849] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1194062ce0 con 0x7f11780ac460 2024-03-28T16:01:42.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:42.914+0000 7f1190c00700 1 -- 172.21.15.67:0/3948365317 <== osd.11 v2:172.21.15.183:6824/1892639849 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f1194062ce0 con 0x7f11780ac460 2024-03-28T16:01:42.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:42.914+0000 7f119ac00700 1 -- 172.21.15.67:0/3948365317 >> [v2:172.21.15.183:6824/1892639849,v1:172.21.15.183:6825/1892639849] conn(0x7f11780ac460 msgr2=0x7f11780ae900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:42.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:42.914+0000 7f119ac00700 1 --2- 172.21.15.67:0/3948365317 >> [v2:172.21.15.183:6824/1892639849,v1:172.21.15.183:6825/1892639849] conn(0x7f11780ac460 0x7f11780ae900 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:42.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:42.914+0000 7f119ac00700 1 -- 172.21.15.67:0/3948365317 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f11780a2920 msgr2=0x7f11780a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:42.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:42.914+0000 7f119ac00700 1 --2- 172.21.15.67:0/3948365317 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f11780a2920 0x7f11780a4de0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f1180005fd0 tx=0x7f118000a560 comp rx=0 tx=0).stop 2024-03-28T16:01:42.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:42.914+0000 7f119ac00700 1 -- 172.21.15.67:0/3948365317 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11941023d0 msgr2=0x7f119419e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:42.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:42.914+0000 7f119ac00700 1 --2- 172.21.15.67:0/3948365317 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11941023d0 0x7f119419e610 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f1184006010 tx=0x7f1184003680 comp rx=0 tx=0).stop 2024-03-28T16:01:42.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:42.915+0000 7f1198c00700 1 -- 172.21.15.67:0/3948365317 reap_dead start 2024-03-28T16:01:42.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:42.915+0000 7f119ac00700 1 -- 172.21.15.67:0/3948365317 shutdown_connections 2024-03-28T16:01:42.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:42.915+0000 7f119ac00700 1 -- 172.21.15.67:0/3948365317 >> 172.21.15.67:0/3948365317 conn(0x7f11940fdc40 msgr2=0x7f11940fefa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:42.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:42.915+0000 7f119ac00700 1 -- 172.21.15.67:0/3948365317 shutdown_connections 2024-03-28T16:01:42.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:42.915+0000 7f119ac00700 1 -- 172.21.15.67:0/3948365317 wait complete. 2024-03-28T16:01:42.932 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:42 smithi067 ceph-mon[31707]: pgmap v383: 97 pgs: 1 peering, 96 active+clean; 638 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 383 B/s wr, 2 op/s 2024-03-28T16:01:43.007 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:42 smithi150 ceph-mon[34825]: pgmap v383: 97 pgs: 1 peering, 96 active+clean; 638 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 383 B/s wr, 2 op/s 2024-03-28T16:01:43.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:42 smithi183 ceph-mon[35580]: pgmap v383: 97 pgs: 1 peering, 96 active+clean; 638 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 383 B/s wr, 2 op/s 2024-03-28T16:01:44.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:43.816+0000 7f2ca3200700 1 -- 172.21.15.67:0/3214650807 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c9c109160 msgr2=0x7f2c9c115670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:44.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:43.816+0000 7f2ca3200700 1 --2- 172.21.15.67:0/3214650807 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c9c109160 0x7f2c9c115670 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f2c9400b0a0 tx=0x7f2c9400b3b0 comp rx=0 tx=0).stop 2024-03-28T16:01:44.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:43.816+0000 7f2ca3200700 1 -- 172.21.15.67:0/3214650807 shutdown_connections 2024-03-28T16:01:44.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:43.816+0000 7f2ca3200700 1 --2- 172.21.15.67:0/3214650807 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c9c109160 0x7f2c9c115670 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:44.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:43.816+0000 7f2ca3200700 1 --2- 172.21.15.67:0/3214650807 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2c9c10e7a0 0x7f2c9c10eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:44.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:43.816+0000 7f2ca3200700 1 --2- 172.21.15.67:0/3214650807 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c9c1087a0 0x7f2c9c108c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:44.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:43.816+0000 7f2ca3200700 1 -- 172.21.15.67:0/3214650807 >> 172.21.15.67:0/3214650807 conn(0x7f2c9c075960 msgr2=0x7f2c9c077d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:44.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:43.816+0000 7f2ca3200700 1 -- 172.21.15.67:0/3214650807 shutdown_connections 2024-03-28T16:01:44.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:43.817+0000 7f2ca3200700 1 -- 172.21.15.67:0/3214650807 wait complete. 2024-03-28T16:01:44.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:43.818+0000 7f2ca3200700 1 Processor -- start 2024-03-28T16:01:44.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:43.818+0000 7f2ca3200700 1 -- start start 2024-03-28T16:01:44.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:43.818+0000 7f2ca3200700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c9c1087a0 0x7f2c9c19edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:44.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:43.818+0000 7f2ca3200700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c9c109160 0x7f2c9c19f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:44.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:43.818+0000 7f2ca3200700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2c9c10e7a0 0x7f2c9c1a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:44.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:43.818+0000 7f2ca3200700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c9c118980 con 0x7f2c9c1087a0 2024-03-28T16:01:44.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:43.818+0000 7f2ca3200700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c9c118800 con 0x7f2c9c109160 2024-03-28T16:01:44.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:43.818+0000 7f2ca3200700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c9c118b00 con 0x7f2c9c10e7a0 2024-03-28T16:01:44.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:43.818+0000 7f2c9be00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c9c1087a0 0x7f2c9c19edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:44.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:43.818+0000 7f2c9be00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c9c1087a0 0x7f2c9c19edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52196/0 (socket says 172.21.15.67:52196) 2024-03-28T16:01:44.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:43.818+0000 7f2c9be00700 1 -- 172.21.15.67:0/3066719345 learned_addr learned my addr 172.21.15.67:0/3066719345 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:01:44.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:43.819+0000 7f2c9b400700 1 --2- 172.21.15.67:0/3066719345 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c9c109160 0x7f2c9c19f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:44.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:43.819+0000 7f2ca1200700 1 --2- 172.21.15.67:0/3066719345 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2c9c10e7a0 0x7f2c9c1a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:44.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:43.819+0000 7f2c9be00700 1 -- 172.21.15.67:0/3066719345 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2c9c10e7a0 msgr2=0x7f2c9c1a3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:44.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:43.819+0000 7f2c9be00700 1 --2- 172.21.15.67:0/3066719345 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2c9c10e7a0 0x7f2c9c1a3680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:44.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:43.819+0000 7f2c9be00700 1 -- 172.21.15.67:0/3066719345 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c9c109160 msgr2=0x7f2c9c19f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:44.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:43.819+0000 7f2c9be00700 1 --2- 172.21.15.67:0/3066719345 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c9c109160 0x7f2c9c19f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:44.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:43.819+0000 7f2c9be00700 1 -- 172.21.15.67:0/3066719345 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c9c1a3dc0 con 0x7f2c9c1087a0 2024-03-28T16:01:44.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:43.819+0000 7f2c9be00700 1 --2- 172.21.15.67:0/3066719345 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c9c1087a0 0x7f2c9c19edb0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c00d950 tx=0x7f2c8c00dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:44.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:43.819+0000 7f2c98c00700 1 -- 172.21.15.67:0/3066719345 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c8c016070 con 0x7f2c9c1087a0 2024-03-28T16:01:44.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:43.819+0000 7f2c98c00700 1 -- 172.21.15.67:0/3066719345 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f2c8c010460 con 0x7f2c9c1087a0 2024-03-28T16:01:44.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:43.819+0000 7f2ca3200700 1 -- 172.21.15.67:0/3066719345 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c9c1a40b0 con 0x7f2c9c1087a0 2024-03-28T16:01:44.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:43.820+0000 7f2c98c00700 1 -- 172.21.15.67:0/3066719345 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c8c00f620 con 0x7f2c9c1087a0 2024-03-28T16:01:44.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:43.820+0000 7f2ca3200700 1 -- 172.21.15.67:0/3066719345 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c9c1ab930 con 0x7f2c9c1087a0 2024-03-28T16:01:44.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:43.820+0000 7f2ca3200700 1 -- 172.21.15.67:0/3066719345 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2c9c062920 con 0x7f2c9c1087a0 2024-03-28T16:01:44.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:43.823+0000 7f2c98c00700 1 -- 172.21.15.67:0/3066719345 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f2c8c019070 con 0x7f2c9c1087a0 2024-03-28T16:01:44.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:43.824+0000 7f2c98c00700 1 --2- 172.21.15.67:0/3066719345 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f2c800a29a0 0x7f2c800a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:44.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:43.824+0000 7f2c98c00700 1 -- 172.21.15.67:0/3066719345 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f2c8c0cecf0 con 0x7f2c9c1087a0 2024-03-28T16:01:44.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:43.824+0000 7f2c9b400700 1 --2- 172.21.15.67:0/3066719345 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f2c800a29a0 0x7f2c800a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:44.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:43.825+0000 7f2c98c00700 1 --2- 172.21.15.67:0/3066719345 >> [v2:172.21.15.183:6816/1025400095,v1:172.21.15.183:6817/1025400095] conn(0x7f2c800ac4e0 0x7f2c800ae980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:44.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:43.825+0000 7f2c98c00700 1 -- 172.21.15.67:0/3066719345 --> [v2:172.21.15.183:6816/1025400095,v1:172.21.15.183:6817/1025400095] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2c8c00ae50 con 0x7f2c800ac4e0 2024-03-28T16:01:44.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:43.825+0000 7f2c98c00700 1 -- 172.21.15.67:0/3066719345 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2c8c0cf0a0 con 0x7f2c9c1087a0 2024-03-28T16:01:44.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:43.825+0000 7f2c9b400700 1 --2- 172.21.15.67:0/3066719345 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f2c800a29a0 0x7f2c800a4e60 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f2c88005fd0 tx=0x7f2c88005f60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:44.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:43.825+0000 7f2ca1200700 1 --2- 172.21.15.67:0/3066719345 >> [v2:172.21.15.183:6816/1025400095,v1:172.21.15.183:6817/1025400095] conn(0x7f2c800ac4e0 0x7f2c800ae980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:44.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:43.826+0000 7f2ca1200700 1 --2- 172.21.15.67:0/3066719345 >> [v2:172.21.15.183:6816/1025400095,v1:172.21.15.183:6817/1025400095] conn(0x7f2c800ac4e0 0x7f2c800ae980 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:44.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:43.829+0000 7f2c98c00700 1 -- 172.21.15.67:0/3066719345 <== osd.8 v2:172.21.15.183:6816/1025400095 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f2c8c00ae50 con 0x7f2c800ac4e0 2024-03-28T16:01:44.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:43.847+0000 7f2ca3200700 1 -- 172.21.15.67:0/3066719345 --> [v2:172.21.15.183:6816/1025400095,v1:172.21.15.183:6817/1025400095] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2c9c062d20 con 0x7f2c800ac4e0 2024-03-28T16:01:44.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:43.849+0000 7f2c98c00700 1 -- 172.21.15.67:0/3066719345 <== osd.8 v2:172.21.15.183:6816/1025400095 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f2c9c062d20 con 0x7f2c800ac4e0 2024-03-28T16:01:44.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:43.849+0000 7f2ca3200700 1 -- 172.21.15.67:0/3066719345 >> [v2:172.21.15.183:6816/1025400095,v1:172.21.15.183:6817/1025400095] conn(0x7f2c800ac4e0 msgr2=0x7f2c800ae980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:44.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:43.849+0000 7f2ca3200700 1 --2- 172.21.15.67:0/3066719345 >> [v2:172.21.15.183:6816/1025400095,v1:172.21.15.183:6817/1025400095] conn(0x7f2c800ac4e0 0x7f2c800ae980 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:44.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:43.849+0000 7f2ca3200700 1 -- 172.21.15.67:0/3066719345 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f2c800a29a0 msgr2=0x7f2c800a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:44.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:43.849+0000 7f2ca3200700 1 --2- 172.21.15.67:0/3066719345 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f2c800a29a0 0x7f2c800a4e60 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f2c88005fd0 tx=0x7f2c88005f60 comp rx=0 tx=0).stop 2024-03-28T16:01:44.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:43.849+0000 7f2ca3200700 1 -- 172.21.15.67:0/3066719345 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c9c1087a0 msgr2=0x7f2c9c19edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:44.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:43.849+0000 7f2ca3200700 1 --2- 172.21.15.67:0/3066719345 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c9c1087a0 0x7f2c9c19edb0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c00d950 tx=0x7f2c8c00dd10 comp rx=0 tx=0).stop 2024-03-28T16:01:44.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:43.849+0000 7f2ca1200700 1 -- 172.21.15.67:0/3066719345 reap_dead start 2024-03-28T16:01:44.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:43.849+0000 7f2ca3200700 1 -- 172.21.15.67:0/3066719345 shutdown_connections 2024-03-28T16:01:44.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:43.849+0000 7f2ca3200700 1 -- 172.21.15.67:0/3066719345 >> 172.21.15.67:0/3066719345 conn(0x7f2c9c075960 msgr2=0x7f2c9c075ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:44.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:43.850+0000 7f2ca3200700 1 -- 172.21.15.67:0/3066719345 shutdown_connections 2024-03-28T16:01:44.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:43.850+0000 7f2ca3200700 1 -- 172.21.15.67:0/3066719345 wait complete. 2024-03-28T16:01:44.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:44 smithi067 ceph-mon[31707]: pgmap v384: 97 pgs: 1 peering, 96 active+clean; 638 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 1 op/s; 2 B/s, 0 objects/s recovering 2024-03-28T16:01:45.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:44 smithi150 ceph-mon[34825]: pgmap v384: 97 pgs: 1 peering, 96 active+clean; 638 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 1 op/s; 2 B/s, 0 objects/s recovering 2024-03-28T16:01:45.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:44 smithi183 ceph-mon[35580]: pgmap v384: 97 pgs: 1 peering, 96 active+clean; 638 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 1 op/s; 2 B/s, 0 objects/s recovering 2024-03-28T16:01:45.852 INFO:teuthology.orchestra.run.smithi067.stdout:365072220183 2024-03-28T16:01:45.853 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.11 2024-03-28T16:01:46.942 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:46 smithi067 ceph-mon[31707]: pgmap v385: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s; 2 B/s, 0 objects/s recovering 2024-03-28T16:01:47.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:46 smithi150 ceph-mon[34825]: pgmap v385: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s; 2 B/s, 0 objects/s recovering 2024-03-28T16:01:47.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:46 smithi183 ceph-mon[35580]: pgmap v385: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s; 2 B/s, 0 objects/s recovering 2024-03-28T16:01:47.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:46.980+0000 7f35a0200700 1 -- 172.21.15.67:0/3401769086 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35981081c0 msgr2=0x7f35981085a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:47.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:46.980+0000 7f35a0200700 1 --2- 172.21.15.67:0/3401769086 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35981081c0 0x7f35981085a0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f358c0099c0 tx=0x7f358c009cd0 comp rx=0 tx=0).stop 2024-03-28T16:01:47.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:46.980+0000 7f35a0200700 1 -- 172.21.15.67:0/3401769086 shutdown_connections 2024-03-28T16:01:47.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:46.980+0000 7f35a0200700 1 --2- 172.21.15.67:0/3401769086 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3598102ba0 0x7f359810efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:47.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:46.980+0000 7f35a0200700 1 --2- 172.21.15.67:0/3401769086 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35981081c0 0x7f35981085a0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:47.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:46.980+0000 7f35a0200700 1 --2- 172.21.15.67:0/3401769086 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f35981021e0 0x7f3598102660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:47.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:46.980+0000 7f35a0200700 1 -- 172.21.15.67:0/3401769086 >> 172.21.15.67:0/3401769086 conn(0x7f35980792c0 msgr2=0x7f35980796d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:47.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:46.980+0000 7f35a0200700 1 -- 172.21.15.67:0/3401769086 shutdown_connections 2024-03-28T16:01:47.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:46.981+0000 7f35a0200700 1 -- 172.21.15.67:0/3401769086 wait complete. 2024-03-28T16:01:47.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:46.981+0000 7f35a0200700 1 Processor -- start 2024-03-28T16:01:47.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:46.982+0000 7f35a0200700 1 -- start start 2024-03-28T16:01:47.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:46.982+0000 7f35a0200700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35981021e0 0x7f35981a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:47.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:46.982+0000 7f35a0200700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3598102ba0 0x7f35981a0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:47.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:46.982+0000 7f35a0200700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f35981081c0 0x7f359819a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:47.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:46.982+0000 7f35a0200700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3598112340 con 0x7f35981021e0 2024-03-28T16:01:47.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:46.982+0000 7f35a0200700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35981121c0 con 0x7f3598102ba0 2024-03-28T16:01:47.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:46.982+0000 7f35a0200700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35981124c0 con 0x7f35981081c0 2024-03-28T16:01:47.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:46.982+0000 7f359d800700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35981021e0 0x7f35981a06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:47.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:46.982+0000 7f359d800700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35981021e0 0x7f35981a06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52212/0 (socket says 172.21.15.67:52212) 2024-03-28T16:01:47.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:46.982+0000 7f359d800700 1 -- 172.21.15.67:0/4184318527 learned_addr learned my addr 172.21.15.67:0/4184318527 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:01:47.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:46.983+0000 7f359ce00700 1 --2- 172.21.15.67:0/4184318527 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3598102ba0 0x7f35981a0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:47.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:46.983+0000 7f359d800700 1 -- 172.21.15.67:0/4184318527 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f35981081c0 msgr2=0x7f359819a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:47.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:46.983+0000 7f359d800700 1 --2- 172.21.15.67:0/4184318527 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f35981081c0 0x7f359819a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:47.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:46.983+0000 7f359d800700 1 -- 172.21.15.67:0/4184318527 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3598102ba0 msgr2=0x7f35981a0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:47.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:46.983+0000 7f359d800700 1 --2- 172.21.15.67:0/4184318527 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3598102ba0 0x7f35981a0bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:47.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:46.983+0000 7f359d800700 1 -- 172.21.15.67:0/4184318527 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f359819b010 con 0x7f35981021e0 2024-03-28T16:01:47.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:46.983+0000 7f359d800700 1 --2- 172.21.15.67:0/4184318527 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35981021e0 0x7f35981a06b0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f358c005b40 tx=0x7f358c005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:47.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:46.983+0000 7f3596000700 1 -- 172.21.15.67:0/4184318527 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f358c027070 con 0x7f35981021e0 2024-03-28T16:01:47.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:46.983+0000 7f35a0200700 1 -- 172.21.15.67:0/4184318527 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f359819b2a0 con 0x7f35981021e0 2024-03-28T16:01:47.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:46.983+0000 7f3596000700 1 -- 172.21.15.67:0/4184318527 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f358c003fd0 con 0x7f35981021e0 2024-03-28T16:01:47.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:46.984+0000 7f3596000700 1 -- 172.21.15.67:0/4184318527 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f358c022c60 con 0x7f35981021e0 2024-03-28T16:01:47.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:46.984+0000 7f35a0200700 1 -- 172.21.15.67:0/4184318527 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35981a7510 con 0x7f35981021e0 2024-03-28T16:01:47.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:46.985+0000 7f35a0200700 1 -- 172.21.15.67:0/4184318527 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f359810ed80 con 0x7f35981021e0 2024-03-28T16:01:47.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:46.987+0000 7f3596000700 1 -- 172.21.15.67:0/4184318527 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f358c02a070 con 0x7f35981021e0 2024-03-28T16:01:47.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:46.988+0000 7f3596000700 1 --2- 172.21.15.67:0/4184318527 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f35800a29a0 0x7f35800a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:47.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:46.989+0000 7f3596000700 1 -- 172.21.15.67:0/4184318527 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f358c022e00 con 0x7f35981021e0 2024-03-28T16:01:47.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:46.989+0000 7f359ce00700 1 --2- 172.21.15.67:0/4184318527 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f35800a29a0 0x7f35800a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:47.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:46.989+0000 7f3596000700 1 --2- 172.21.15.67:0/4184318527 >> [v2:172.21.15.150:6800/1734465762,v1:172.21.15.150:6801/1734465762] conn(0x7f35800ac4e0 0x7f35800ae980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:47.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:46.989+0000 7f3596000700 1 -- 172.21.15.67:0/4184318527 --> [v2:172.21.15.150:6800/1734465762,v1:172.21.15.150:6801/1734465762] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f358c0095d0 con 0x7f35800ac4e0 2024-03-28T16:01:47.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:46.989+0000 7f3596000700 1 -- 172.21.15.67:0/4184318527 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) v2 ==== 24+0+0 (secure 0 0 0) 0x7f358c0d0650 con 0x7f35981021e0 2024-03-28T16:01:47.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:46.989+0000 7f359ce00700 1 --2- 172.21.15.67:0/4184318527 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f35800a29a0 0x7f35800a4e60 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f3584005950 tx=0x7f35840058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:47.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:46.990+0000 7f359e200700 1 --2- 172.21.15.67:0/4184318527 >> [v2:172.21.15.150:6800/1734465762,v1:172.21.15.150:6801/1734465762] conn(0x7f35800ac4e0 0x7f35800ae980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:47.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:46.991+0000 7f359e200700 1 --2- 172.21.15.67:0/4184318527 >> [v2:172.21.15.150:6800/1734465762,v1:172.21.15.150:6801/1734465762] conn(0x7f35800ac4e0 0x7f35800ae980 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:47.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:46.993+0000 7f3596000700 1 -- 172.21.15.67:0/4184318527 <== osd.1 v2:172.21.15.150:6800/1734465762 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f358c0095d0 con 0x7f35800ac4e0 2024-03-28T16:01:47.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.011+0000 7f35a0200700 1 -- 172.21.15.67:0/4184318527 --> [v2:172.21.15.150:6800/1734465762,v1:172.21.15.150:6801/1734465762] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3598103870 con 0x7f35800ac4e0 2024-03-28T16:01:47.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.012+0000 7f3596000700 1 -- 172.21.15.67:0/4184318527 <== osd.1 v2:172.21.15.150:6800/1734465762 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f3598103870 con 0x7f35800ac4e0 2024-03-28T16:01:47.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.013+0000 7f35a0200700 1 -- 172.21.15.67:0/4184318527 >> [v2:172.21.15.150:6800/1734465762,v1:172.21.15.150:6801/1734465762] conn(0x7f35800ac4e0 msgr2=0x7f35800ae980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:47.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.013+0000 7f35a0200700 1 --2- 172.21.15.67:0/4184318527 >> [v2:172.21.15.150:6800/1734465762,v1:172.21.15.150:6801/1734465762] conn(0x7f35800ac4e0 0x7f35800ae980 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:47.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.013+0000 7f35a0200700 1 -- 172.21.15.67:0/4184318527 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f35800a29a0 msgr2=0x7f35800a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:47.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.013+0000 7f35a0200700 1 --2- 172.21.15.67:0/4184318527 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f35800a29a0 0x7f35800a4e60 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f3584005950 tx=0x7f35840058e0 comp rx=0 tx=0).stop 2024-03-28T16:01:47.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.013+0000 7f35a0200700 1 -- 172.21.15.67:0/4184318527 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35981021e0 msgr2=0x7f35981a06b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:47.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.013+0000 7f35a0200700 1 --2- 172.21.15.67:0/4184318527 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35981021e0 0x7f35981a06b0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f358c005b40 tx=0x7f358c005e80 comp rx=0 tx=0).stop 2024-03-28T16:01:47.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.013+0000 7f359e200700 1 -- 172.21.15.67:0/4184318527 reap_dead start 2024-03-28T16:01:47.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.013+0000 7f35a0200700 1 -- 172.21.15.67:0/4184318527 shutdown_connections 2024-03-28T16:01:47.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.013+0000 7f35a0200700 1 -- 172.21.15.67:0/4184318527 >> 172.21.15.67:0/4184318527 conn(0x7f35980792c0 msgr2=0x7f359810d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:47.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.013+0000 7f35a0200700 1 -- 172.21.15.67:0/4184318527 shutdown_connections 2024-03-28T16:01:47.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.013+0000 7f35a0200700 1 -- 172.21.15.67:0/4184318527 wait complete. 2024-03-28T16:01:47.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.195+0000 7f187e000700 1 -- 172.21.15.67:0/3133052726 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1878101310 msgr2=0x7f1878110f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:47.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.195+0000 7f187e000700 1 --2- 172.21.15.67:0/3133052726 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1878101310 0x7f1878110f10 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f187000b0a0 tx=0x7f187000b3b0 comp rx=0 tx=0).stop 2024-03-28T16:01:47.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.195+0000 7f187e000700 1 -- 172.21.15.67:0/3133052726 shutdown_connections 2024-03-28T16:01:47.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.195+0000 7f187e000700 1 --2- 172.21.15.67:0/3133052726 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1878101310 0x7f1878110f10 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:47.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.195+0000 7f187e000700 1 --2- 172.21.15.67:0/3133052726 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f18781030d0 0x7f18781034b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:47.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.195+0000 7f187e000700 1 --2- 172.21.15.67:0/3133052726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1878100780 0x7f1878100be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:47.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.195+0000 7f187e000700 1 -- 172.21.15.67:0/3133052726 >> 172.21.15.67:0/3133052726 conn(0x7f1878077690 msgr2=0x7f18780feab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:47.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.196+0000 7f187e000700 1 -- 172.21.15.67:0/3133052726 shutdown_connections 2024-03-28T16:01:47.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.196+0000 7f187e000700 1 -- 172.21.15.67:0/3133052726 wait complete. 2024-03-28T16:01:47.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.197+0000 7f187e000700 1 Processor -- start 2024-03-28T16:01:47.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.197+0000 7f187e000700 1 -- start start 2024-03-28T16:01:47.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.197+0000 7f187e000700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1878100780 0x7f18781a06f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:47.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.197+0000 7f187e000700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1878101310 0x7f18781a0c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:47.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.197+0000 7f187e000700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18781030d0 0x7f187819a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:47.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.197+0000 7f187e000700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18781145b0 con 0x7f18781030d0 2024-03-28T16:01:47.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.197+0000 7f187e000700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1878114430 con 0x7f1878100780 2024-03-28T16:01:47.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.197+0000 7f187e000700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1878114730 con 0x7f1878101310 2024-03-28T16:01:47.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.198+0000 7f1877e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18781030d0 0x7f187819a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:47.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.198+0000 7f1877e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18781030d0 0x7f187819a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52228/0 (socket says 172.21.15.67:52228) 2024-03-28T16:01:47.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.198+0000 7f1877e00700 1 -- 172.21.15.67:0/2515988867 learned_addr learned my addr 172.21.15.67:0/2515988867 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:01:47.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.198+0000 7f1877400700 1 --2- 172.21.15.67:0/2515988867 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1878100780 0x7f18781a06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:47.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.198+0000 7f1876a00700 1 --2- 172.21.15.67:0/2515988867 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1878101310 0x7f18781a0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:47.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.198+0000 7f1877e00700 1 -- 172.21.15.67:0/2515988867 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1878101310 msgr2=0x7f18781a0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:47.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.198+0000 7f1877e00700 1 --2- 172.21.15.67:0/2515988867 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1878101310 0x7f18781a0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:47.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.198+0000 7f1877e00700 1 -- 172.21.15.67:0/2515988867 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1878100780 msgr2=0x7f18781a06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:47.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.198+0000 7f1877e00700 1 --2- 172.21.15.67:0/2515988867 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1878100780 0x7f18781a06f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:47.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.198+0000 7f1877e00700 1 -- 172.21.15.67:0/2515988867 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f187819b010 con 0x7f18781030d0 2024-03-28T16:01:47.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.199+0000 7f1877e00700 1 --2- 172.21.15.67:0/2515988867 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18781030d0 0x7f187819a8b0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f187000b070 tx=0x7f1870004d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:47.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.199+0000 7f186fe00700 1 -- 172.21.15.67:0/2515988867 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1870021070 con 0x7f18781030d0 2024-03-28T16:01:47.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.199+0000 7f186fe00700 1 -- 172.21.15.67:0/2515988867 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1870007c00 con 0x7f18781030d0 2024-03-28T16:01:47.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.199+0000 7f187e000700 1 -- 172.21.15.67:0/2515988867 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f187819b2a0 con 0x7f18781030d0 2024-03-28T16:01:47.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.199+0000 7f186fe00700 1 -- 172.21.15.67:0/2515988867 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18700052a0 con 0x7f18781030d0 2024-03-28T16:01:47.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.199+0000 7f187e000700 1 -- 172.21.15.67:0/2515988867 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18781a75c0 con 0x7f18781030d0 2024-03-28T16:01:47.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.200+0000 7f187e000700 1 -- 172.21.15.67:0/2515988867 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f18780628a0 con 0x7f18781030d0 2024-03-28T16:01:47.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.202+0000 7f186fe00700 1 -- 172.21.15.67:0/2515988867 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f1870024070 con 0x7f18781030d0 2024-03-28T16:01:47.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.204+0000 7f186fe00700 1 --2- 172.21.15.67:0/2515988867 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f185c0a2920 0x7f185c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:47.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.204+0000 7f186fe00700 1 -- 172.21.15.67:0/2515988867 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f187000f020 con 0x7f18781030d0 2024-03-28T16:01:47.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.204+0000 7f1877400700 1 --2- 172.21.15.67:0/2515988867 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f185c0a2920 0x7f185c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:47.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.204+0000 7f186fe00700 1 --2- 172.21.15.67:0/2515988867 >> [v2:172.21.15.67:6810/1933541875,v1:172.21.15.67:6811/1933541875] conn(0x7f185c0ac460 0x7f185c0ae900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:47.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.204+0000 7f186fe00700 1 -- 172.21.15.67:0/2515988867 --> [v2:172.21.15.67:6810/1933541875,v1:172.21.15.67:6811/1933541875] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1870027e50 con 0x7f185c0ac460 2024-03-28T16:01:47.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.204+0000 7f186fe00700 1 -- 172.21.15.67:0/2515988867 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) v2 ==== 24+0+0 (secure 0 0 0) 0x7f18700d08c0 con 0x7f18781030d0 2024-03-28T16:01:47.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.205+0000 7f1876a00700 1 --2- 172.21.15.67:0/2515988867 >> [v2:172.21.15.67:6810/1933541875,v1:172.21.15.67:6811/1933541875] conn(0x7f185c0ac460 0x7f185c0ae900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:47.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.205+0000 7f1877400700 1 --2- 172.21.15.67:0/2515988867 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f185c0a2920 0x7f185c0a4de0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f1868005e10 tx=0x7f1868005d40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:47.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.205+0000 7f1876a00700 1 --2- 172.21.15.67:0/2515988867 >> [v2:172.21.15.67:6810/1933541875,v1:172.21.15.67:6811/1933541875] conn(0x7f185c0ac460 0x7f185c0ae900 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:47.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.206+0000 7f186fe00700 1 -- 172.21.15.67:0/2515988867 <== osd.3 v2:172.21.15.67:6810/1933541875 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f1870027e50 con 0x7f185c0ac460 2024-03-28T16:01:47.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.226+0000 7f187e000700 1 -- 172.21.15.67:0/2515988867 --> [v2:172.21.15.67:6810/1933541875,v1:172.21.15.67:6811/1933541875] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1878062d20 con 0x7f185c0ac460 2024-03-28T16:01:47.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.226+0000 7f186fe00700 1 -- 172.21.15.67:0/2515988867 <== osd.3 v2:172.21.15.67:6810/1933541875 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f1878062d20 con 0x7f185c0ac460 2024-03-28T16:01:47.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.227+0000 7f187e000700 1 -- 172.21.15.67:0/2515988867 >> [v2:172.21.15.67:6810/1933541875,v1:172.21.15.67:6811/1933541875] conn(0x7f185c0ac460 msgr2=0x7f185c0ae900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:47.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.227+0000 7f187e000700 1 --2- 172.21.15.67:0/2515988867 >> [v2:172.21.15.67:6810/1933541875,v1:172.21.15.67:6811/1933541875] conn(0x7f185c0ac460 0x7f185c0ae900 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:47.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.227+0000 7f187e000700 1 -- 172.21.15.67:0/2515988867 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f185c0a2920 msgr2=0x7f185c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:47.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.227+0000 7f187e000700 1 --2- 172.21.15.67:0/2515988867 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f185c0a2920 0x7f185c0a4de0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f1868005e10 tx=0x7f1868005d40 comp rx=0 tx=0).stop 2024-03-28T16:01:47.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.227+0000 7f187e000700 1 -- 172.21.15.67:0/2515988867 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18781030d0 msgr2=0x7f187819a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:47.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.227+0000 7f187e000700 1 --2- 172.21.15.67:0/2515988867 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18781030d0 0x7f187819a8b0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f187000b070 tx=0x7f1870004d90 comp rx=0 tx=0).stop 2024-03-28T16:01:47.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.227+0000 7f1877e00700 1 -- 172.21.15.67:0/2515988867 reap_dead start 2024-03-28T16:01:47.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.227+0000 7f187e000700 1 -- 172.21.15.67:0/2515988867 shutdown_connections 2024-03-28T16:01:47.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.227+0000 7f187e000700 1 -- 172.21.15.67:0/2515988867 >> 172.21.15.67:0/2515988867 conn(0x7f1878077690 msgr2=0x7f18780fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:47.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.228+0000 7f187e000700 1 -- 172.21.15.67:0/2515988867 shutdown_connections 2024-03-28T16:01:47.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:47.228+0000 7f187e000700 1 -- 172.21.15.67:0/2515988867 wait complete. 2024-03-28T16:01:49.003 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:48 smithi067 ceph-mon[31707]: pgmap v386: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s; 2 B/s, 0 objects/s recovering 2024-03-28T16:01:49.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:48 smithi150 ceph-mon[34825]: pgmap v386: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s; 2 B/s, 0 objects/s recovering 2024-03-28T16:01:49.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:48 smithi183 ceph-mon[35580]: pgmap v386: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s; 2 B/s, 0 objects/s recovering 2024-03-28T16:01:49.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:49.388+0000 7f70cc000700 1 -- 172.21.15.67:0/877840234 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70c41045e0 msgr2=0x7f70c41049c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:49.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:49.388+0000 7f70cc000700 1 --2- 172.21.15.67:0/877840234 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70c41045e0 0x7f70c41049c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f70b80099c0 tx=0x7f70b8009cd0 comp rx=0 tx=0).stop 2024-03-28T16:01:49.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:49.388+0000 7f70cc000700 1 -- 172.21.15.67:0/877840234 shutdown_connections 2024-03-28T16:01:49.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:49.388+0000 7f70cc000700 1 --2- 172.21.15.67:0/877840234 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f70c41096c0 0x7f70c4111290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:49.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:49.388+0000 7f70cc000700 1 --2- 172.21.15.67:0/877840234 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70c41045e0 0x7f70c41049c0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:49.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:49.388+0000 7f70cc000700 1 --2- 172.21.15.67:0/877840234 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f70c4104f90 0x7f70c4108fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:49.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:49.388+0000 7f70cc000700 1 -- 172.21.15.67:0/877840234 >> 172.21.15.67:0/877840234 conn(0x7f70c40ffe70 msgr2=0x7f70c4102290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:49.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:49.388+0000 7f70cc000700 1 -- 172.21.15.67:0/877840234 shutdown_connections 2024-03-28T16:01:49.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:49.388+0000 7f70cc000700 1 -- 172.21.15.67:0/877840234 wait complete. 2024-03-28T16:01:49.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:49.390+0000 7f70cc000700 1 Processor -- start 2024-03-28T16:01:49.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:49.390+0000 7f70cc000700 1 -- start start 2024-03-28T16:01:49.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:49.390+0000 7f70cc000700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f70c41045e0 0x7f70c40766a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:49.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:49.390+0000 7f70cc000700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70c4104f90 0x7f70c4076be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:49.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:49.391+0000 7f70cc000700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f70c41096c0 0x7f70c4077120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:49.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:49.391+0000 7f70cc000700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70c4114620 con 0x7f70c4104f90 2024-03-28T16:01:49.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:49.391+0000 7f70cc000700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70c41144a0 con 0x7f70c41096c0 2024-03-28T16:01:49.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:49.391+0000 7f70cc000700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70c41147a0 con 0x7f70c41045e0 2024-03-28T16:01:49.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:49.391+0000 7f70c8c00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70c4104f90 0x7f70c4076be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:49.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:49.391+0000 7f70c8c00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70c4104f90 0x7f70c4076be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50620/0 (socket says 172.21.15.67:50620) 2024-03-28T16:01:49.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:49.391+0000 7f70c8c00700 1 -- 172.21.15.67:0/1489486070 learned_addr learned my addr 172.21.15.67:0/1489486070 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:01:49.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:49.391+0000 7f70c9600700 1 --2- 172.21.15.67:0/1489486070 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f70c41045e0 0x7f70c40766a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:49.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:49.391+0000 7f70ca000700 1 --2- 172.21.15.67:0/1489486070 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f70c41096c0 0x7f70c4077120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:49.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:49.392+0000 7f70c8c00700 1 -- 172.21.15.67:0/1489486070 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f70c41045e0 msgr2=0x7f70c40766a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:49.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:49.392+0000 7f70c8c00700 1 --2- 172.21.15.67:0/1489486070 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f70c41045e0 0x7f70c40766a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:49.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:49.392+0000 7f70c8c00700 1 -- 172.21.15.67:0/1489486070 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f70c41096c0 msgr2=0x7f70c4077120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:49.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:49.392+0000 7f70c8c00700 1 --2- 172.21.15.67:0/1489486070 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f70c41096c0 0x7f70c4077120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:49.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:49.392+0000 7f70c8c00700 1 -- 172.21.15.67:0/1489486070 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70c41ab1e0 con 0x7f70c4104f90 2024-03-28T16:01:49.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:49.392+0000 7f70ca000700 1 --2- 172.21.15.67:0/1489486070 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f70c41096c0 0x7f70c4077120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T16:01:49.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:49.392+0000 7f70c8c00700 1 --2- 172.21.15.67:0/1489486070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70c4104f90 0x7f70c4076be0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f70b000b500 tx=0x7f70b000b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:49.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:49.392+0000 7f70c2000700 1 -- 172.21.15.67:0/1489486070 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70b0015020 con 0x7f70c4104f90 2024-03-28T16:01:49.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:49.392+0000 7f70c2000700 1 -- 172.21.15.67:0/1489486070 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f70b0011ce0 con 0x7f70c4104f90 2024-03-28T16:01:49.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:49.392+0000 7f70cc000700 1 -- 172.21.15.67:0/1489486070 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70c41ab4d0 con 0x7f70c4104f90 2024-03-28T16:01:49.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:49.392+0000 7f70c2000700 1 -- 172.21.15.67:0/1489486070 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70b0010d70 con 0x7f70c4104f90 2024-03-28T16:01:49.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:49.392+0000 7f70cc000700 1 -- 172.21.15.67:0/1489486070 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70c41ab9c0 con 0x7f70c4104f90 2024-03-28T16:01:49.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:49.393+0000 7f70cc000700 1 -- 172.21.15.67:0/1489486070 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f70c4062920 con 0x7f70c4104f90 2024-03-28T16:01:49.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:49.395+0000 7f70c2000700 1 -- 172.21.15.67:0/1489486070 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f70b0018020 con 0x7f70c4104f90 2024-03-28T16:01:49.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:49.397+0000 7f70c2000700 1 --2- 172.21.15.67:0/1489486070 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f70ac0a2920 0x7f70ac0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:49.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:49.397+0000 7f70c2000700 1 -- 172.21.15.67:0/1489486070 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f70b00ce970 con 0x7f70c4104f90 2024-03-28T16:01:49.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:49.397+0000 7f70c9600700 1 --2- 172.21.15.67:0/1489486070 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f70ac0a2920 0x7f70ac0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:49.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:49.397+0000 7f70c2000700 1 --2- 172.21.15.67:0/1489486070 >> [v2:172.21.15.150:6824/3489982080,v1:172.21.15.150:6825/3489982080] conn(0x7f70ac0ac460 0x7f70ac0ae900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:49.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:49.397+0000 7f70c2000700 1 -- 172.21.15.67:0/1489486070 --> [v2:172.21.15.150:6824/3489982080,v1:172.21.15.150:6825/3489982080] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f70b0013ec0 con 0x7f70ac0ac460 2024-03-28T16:01:49.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:49.397+0000 7f70c2000700 1 -- 172.21.15.67:0/1489486070 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) v2 ==== 24+0+0 (secure 0 0 0) 0x7f70b00ced20 con 0x7f70c4104f90 2024-03-28T16:01:49.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:49.398+0000 7f70c9600700 1 --2- 172.21.15.67:0/1489486070 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f70ac0a2920 0x7f70ac0a4de0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f70b8009f50 tx=0x7f70b8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:49.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:49.398+0000 7f70ca000700 1 --2- 172.21.15.67:0/1489486070 >> [v2:172.21.15.150:6824/3489982080,v1:172.21.15.150:6825/3489982080] conn(0x7f70ac0ac460 0x7f70ac0ae900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:49.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:49.399+0000 7f70ca000700 1 --2- 172.21.15.67:0/1489486070 >> [v2:172.21.15.150:6824/3489982080,v1:172.21.15.150:6825/3489982080] conn(0x7f70ac0ac460 0x7f70ac0ae900 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:49.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:49.401+0000 7f70c2000700 1 -- 172.21.15.67:0/1489486070 <== osd.10 v2:172.21.15.150:6824/3489982080 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f70b0013ec0 con 0x7f70ac0ac460 2024-03-28T16:01:49.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:49.419+0000 7f70cc000700 1 -- 172.21.15.67:0/1489486070 --> [v2:172.21.15.150:6824/3489982080,v1:172.21.15.150:6825/3489982080] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f70c4062da0 con 0x7f70ac0ac460 2024-03-28T16:01:49.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:49.421+0000 7f70c2000700 1 -- 172.21.15.67:0/1489486070 <== osd.10 v2:172.21.15.150:6824/3489982080 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f70c4062da0 con 0x7f70ac0ac460 2024-03-28T16:01:49.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:49.421+0000 7f70cc000700 1 -- 172.21.15.67:0/1489486070 >> [v2:172.21.15.150:6824/3489982080,v1:172.21.15.150:6825/3489982080] conn(0x7f70ac0ac460 msgr2=0x7f70ac0ae900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:49.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:49.421+0000 7f70cc000700 1 --2- 172.21.15.67:0/1489486070 >> [v2:172.21.15.150:6824/3489982080,v1:172.21.15.150:6825/3489982080] conn(0x7f70ac0ac460 0x7f70ac0ae900 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:49.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:49.421+0000 7f70cc000700 1 -- 172.21.15.67:0/1489486070 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f70ac0a2920 msgr2=0x7f70ac0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:49.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:49.421+0000 7f70cc000700 1 --2- 172.21.15.67:0/1489486070 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f70ac0a2920 0x7f70ac0a4de0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f70b8009f50 tx=0x7f70b8005fb0 comp rx=0 tx=0).stop 2024-03-28T16:01:49.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:49.421+0000 7f70cc000700 1 -- 172.21.15.67:0/1489486070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70c4104f90 msgr2=0x7f70c4076be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:49.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:49.421+0000 7f70cc000700 1 --2- 172.21.15.67:0/1489486070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70c4104f90 0x7f70c4076be0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f70b000b500 tx=0x7f70b000b810 comp rx=0 tx=0).stop 2024-03-28T16:01:49.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:49.421+0000 7f70ca000700 1 -- 172.21.15.67:0/1489486070 reap_dead start 2024-03-28T16:01:49.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:49.422+0000 7f70cc000700 1 -- 172.21.15.67:0/1489486070 shutdown_connections 2024-03-28T16:01:49.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:49.422+0000 7f70cc000700 1 -- 172.21.15.67:0/1489486070 >> 172.21.15.67:0/1489486070 conn(0x7f70c40ffe70 msgr2=0x7f70c41021d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:49.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:49.422+0000 7f70cc000700 1 -- 172.21.15.67:0/1489486070 shutdown_connections 2024-03-28T16:01:49.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:49.422+0000 7f70cc000700 1 -- 172.21.15.67:0/1489486070 wait complete. 2024-03-28T16:01:50.039 INFO:teuthology.orchestra.run.smithi067.stdout:115964117070 2024-03-28T16:01:50.039 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-28T16:01:50.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:50.679+0000 7f2281e00700 1 -- 172.21.15.67:0/3828137623 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f227c101d00 msgr2=0x7f227c105cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:50.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:50.679+0000 7f2281e00700 1 --2- 172.21.15.67:0/3828137623 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f227c101d00 0x7f227c105cc0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f22680099c0 tx=0x7f2268009cd0 comp rx=0 tx=0).stop 2024-03-28T16:01:50.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:50.679+0000 7f2281e00700 1 -- 172.21.15.67:0/3828137623 shutdown_connections 2024-03-28T16:01:50.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:50.679+0000 7f2281e00700 1 --2- 172.21.15.67:0/3828137623 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f227c1063a0 0x7f227c10df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:50.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:50.679+0000 7f2281e00700 1 --2- 172.21.15.67:0/3828137623 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f227c101350 0x7f227c101730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:50.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:50.679+0000 7f2281e00700 1 --2- 172.21.15.67:0/3828137623 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f227c101d00 0x7f227c105cc0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:50.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:50.679+0000 7f2281e00700 1 -- 172.21.15.67:0/3828137623 >> 172.21.15.67:0/3828137623 conn(0x7f227c0770e0 msgr2=0x7f227c0774f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:50.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:50.679+0000 7f2281e00700 1 -- 172.21.15.67:0/3828137623 shutdown_connections 2024-03-28T16:01:50.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:50.679+0000 7f2281e00700 1 -- 172.21.15.67:0/3828137623 wait complete. 2024-03-28T16:01:50.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:50.680+0000 7f2281e00700 1 Processor -- start 2024-03-28T16:01:50.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:50.680+0000 7f2281e00700 1 -- start start 2024-03-28T16:01:50.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:50.681+0000 7f2281e00700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f227c101350 0x7f227c1a06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:50.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:50.681+0000 7f2281e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f227c101d00 0x7f227c1a0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:50.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:50.681+0000 7f2281e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f227c1063a0 0x7f227c19a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:50.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:50.681+0000 7f2281e00700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f227c111280 con 0x7f227c101d00 2024-03-28T16:01:50.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:50.681+0000 7f2281e00700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f227c111100 con 0x7f227c101350 2024-03-28T16:01:50.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:50.681+0000 7f2281e00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f227c111400 con 0x7f227c1063a0 2024-03-28T16:01:50.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:50.681+0000 7f227a000700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f227c101d00 0x7f227c1a0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:50.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:50.681+0000 7f227a000700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f227c101d00 0x7f227c1a0c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50642/0 (socket says 172.21.15.67:50642) 2024-03-28T16:01:50.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:50.681+0000 7f227a000700 1 -- 172.21.15.67:0/1647179394 learned_addr learned my addr 172.21.15.67:0/1647179394 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:01:50.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:50.681+0000 7f227aa00700 1 --2- 172.21.15.67:0/1647179394 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f227c101350 0x7f227c1a06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:50.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:50.681+0000 7f227b400700 1 --2- 172.21.15.67:0/1647179394 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f227c1063a0 0x7f227c19a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:50.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:50.681+0000 7f227a000700 1 -- 172.21.15.67:0/1647179394 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f227c1063a0 msgr2=0x7f227c19a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:50.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:50.681+0000 7f227a000700 1 --2- 172.21.15.67:0/1647179394 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f227c1063a0 0x7f227c19a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:50.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:50.682+0000 7f227a000700 1 -- 172.21.15.67:0/1647179394 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f227c101350 msgr2=0x7f227c1a06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:50.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:50.682+0000 7f227a000700 1 --2- 172.21.15.67:0/1647179394 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f227c101350 0x7f227c1a06d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:50.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:50.682+0000 7f227a000700 1 -- 172.21.15.67:0/1647179394 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f227c19b040 con 0x7f227c101d00 2024-03-28T16:01:50.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:50.682+0000 7f227b400700 1 --2- 172.21.15.67:0/1647179394 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f227c1063a0 0x7f227c19a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:01:50.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:50.682+0000 7f227a000700 1 --2- 172.21.15.67:0/1647179394 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f227c101d00 0x7f227c1a0c10 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f2268009fd0 tx=0x7f2268005f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:50.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:50.682+0000 7f2280a00700 1 -- 172.21.15.67:0/1647179394 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2268004830 con 0x7f227c101d00 2024-03-28T16:01:50.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:50.682+0000 7f2280a00700 1 -- 172.21.15.67:0/1647179394 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f2268003ba0 con 0x7f227c101d00 2024-03-28T16:01:50.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:50.682+0000 7f2281e00700 1 -- 172.21.15.67:0/1647179394 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f227c19b2d0 con 0x7f227c101d00 2024-03-28T16:01:50.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:50.682+0000 7f2280a00700 1 -- 172.21.15.67:0/1647179394 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2268005380 con 0x7f227c101d00 2024-03-28T16:01:50.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:50.682+0000 7f2281e00700 1 -- 172.21.15.67:0/1647179394 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f227c1a75a0 con 0x7f227c101d00 2024-03-28T16:01:50.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:50.683+0000 7f2281e00700 1 -- 172.21.15.67:0/1647179394 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f227c10d690 con 0x7f227c101d00 2024-03-28T16:01:50.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:50.686+0000 7f2280a00700 1 -- 172.21.15.67:0/1647179394 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f2268003d50 con 0x7f227c101d00 2024-03-28T16:01:50.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:50.687+0000 7f2280a00700 1 --2- 172.21.15.67:0/1647179394 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f22600a29a0 0x7f22600a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:50.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:50.687+0000 7f2280a00700 1 -- 172.21.15.67:0/1647179394 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f22680d0450 con 0x7f227c101d00 2024-03-28T16:01:50.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:50.687+0000 7f227aa00700 1 --2- 172.21.15.67:0/1647179394 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f22600a29a0 0x7f22600a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:50.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:50.688+0000 7f2280a00700 1 --2- 172.21.15.67:0/1647179394 >> [v2:172.21.15.67:6802/1894681701,v1:172.21.15.67:6803/1894681701] conn(0x7f22600ac4e0 0x7f22600ae980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:50.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:50.688+0000 7f2280a00700 1 -- 172.21.15.67:0/1647179394 --> [v2:172.21.15.67:6802/1894681701,v1:172.21.15.67:6803/1894681701] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f22600af050 con 0x7f22600ac4e0 2024-03-28T16:01:50.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:50.688+0000 7f2280a00700 1 -- 172.21.15.67:0/1647179394 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) v2 ==== 24+0+0 (secure 0 0 0) 0x7f22680d0800 con 0x7f227c101d00 2024-03-28T16:01:50.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:50.688+0000 7f227b400700 1 --2- 172.21.15.67:0/1647179394 >> [v2:172.21.15.67:6802/1894681701,v1:172.21.15.67:6803/1894681701] conn(0x7f22600ac4e0 0x7f22600ae980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:50.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:50.688+0000 7f227aa00700 1 --2- 172.21.15.67:0/1647179394 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f22600a29a0 0x7f22600a4e60 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f226c005950 tx=0x7f226c005eb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:50.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:50.689+0000 7f227b400700 1 --2- 172.21.15.67:0/1647179394 >> [v2:172.21.15.67:6802/1894681701,v1:172.21.15.67:6803/1894681701] conn(0x7f22600ac4e0 0x7f22600ae980 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:50.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:50.690+0000 7f2280a00700 1 -- 172.21.15.67:0/1647179394 <== osd.0 v2:172.21.15.67:6802/1894681701 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f22600af050 con 0x7f22600ac4e0 2024-03-28T16:01:50.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:50.708+0000 7f2281e00700 1 -- 172.21.15.67:0/1647179394 --> [v2:172.21.15.67:6802/1894681701,v1:172.21.15.67:6803/1894681701] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f227c06a0a0 con 0x7f22600ac4e0 2024-03-28T16:01:50.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:50.709+0000 7f2280a00700 1 -- 172.21.15.67:0/1647179394 <== osd.0 v2:172.21.15.67:6802/1894681701 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f227c06a0a0 con 0x7f22600ac4e0 2024-03-28T16:01:50.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:50.709+0000 7f2281e00700 1 -- 172.21.15.67:0/1647179394 >> [v2:172.21.15.67:6802/1894681701,v1:172.21.15.67:6803/1894681701] conn(0x7f22600ac4e0 msgr2=0x7f22600ae980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:50.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:50.709+0000 7f2281e00700 1 --2- 172.21.15.67:0/1647179394 >> [v2:172.21.15.67:6802/1894681701,v1:172.21.15.67:6803/1894681701] conn(0x7f22600ac4e0 0x7f22600ae980 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:50.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:50.709+0000 7f2281e00700 1 -- 172.21.15.67:0/1647179394 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f22600a29a0 msgr2=0x7f22600a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:50.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:50.709+0000 7f2281e00700 1 --2- 172.21.15.67:0/1647179394 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f22600a29a0 0x7f22600a4e60 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f226c005950 tx=0x7f226c005eb0 comp rx=0 tx=0).stop 2024-03-28T16:01:50.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:50.710+0000 7f2281e00700 1 -- 172.21.15.67:0/1647179394 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f227c101d00 msgr2=0x7f227c1a0c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:50.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:50.710+0000 7f2281e00700 1 --2- 172.21.15.67:0/1647179394 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f227c101d00 0x7f227c1a0c10 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f2268009fd0 tx=0x7f2268005f10 comp rx=0 tx=0).stop 2024-03-28T16:01:50.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:50.710+0000 7f227b400700 1 -- 172.21.15.67:0/1647179394 reap_dead start 2024-03-28T16:01:50.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:50.710+0000 7f2281e00700 1 -- 172.21.15.67:0/1647179394 shutdown_connections 2024-03-28T16:01:50.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:50.710+0000 7f2281e00700 1 -- 172.21.15.67:0/1647179394 >> 172.21.15.67:0/1647179394 conn(0x7f227c0770e0 msgr2=0x7f227c06aa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:50.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:50.710+0000 7f2281e00700 1 -- 172.21.15.67:0/1647179394 shutdown_connections 2024-03-28T16:01:50.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:50.710+0000 7f2281e00700 1 -- 172.21.15.67:0/1647179394 wait complete. 2024-03-28T16:01:51.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:50 smithi150 ceph-mon[34825]: pgmap v387: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 435 B/s rd, 0 op/s; 2 B/s, 0 objects/s recovering 2024-03-28T16:01:51.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:50 smithi067 ceph-mon[31707]: pgmap v387: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 435 B/s rd, 0 op/s; 2 B/s, 0 objects/s recovering 2024-03-28T16:01:51.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:50 smithi183 ceph-mon[35580]: pgmap v387: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 435 B/s rd, 0 op/s; 2 B/s, 0 objects/s recovering 2024-03-28T16:01:52.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:52.074+0000 7f7ae3000700 1 -- 172.21.15.67:0/1404374180 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7adc100c70 msgr2=0x7f7adc10f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:52.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:52.074+0000 7f7ae3000700 1 --2- 172.21.15.67:0/1404374180 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7adc100c70 0x7f7adc10f020 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f7ad400b0a0 tx=0x7f7ad400b3b0 comp rx=0 tx=0).stop 2024-03-28T16:01:52.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:52.074+0000 7f7ae3000700 1 -- 172.21.15.67:0/1404374180 shutdown_connections 2024-03-28T16:01:52.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:52.074+0000 7f7ae3000700 1 --2- 172.21.15.67:0/1404374180 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7adc100c70 0x7f7adc10f020 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:52.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:52.074+0000 7f7ae3000700 1 --2- 172.21.15.67:0/1404374180 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7adc1082d0 0x7f7adc1086b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:52.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:52.074+0000 7f7ae3000700 1 --2- 172.21.15.67:0/1404374180 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7adc1002b0 0x7f7adc100730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:52.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:52.074+0000 7f7ae3000700 1 -- 172.21.15.67:0/1404374180 >> 172.21.15.67:0/1404374180 conn(0x7f7adc077260 msgr2=0x7f7adc0fe5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:52.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:52.074+0000 7f7ae3000700 1 -- 172.21.15.67:0/1404374180 shutdown_connections 2024-03-28T16:01:52.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:52.074+0000 7f7ae3000700 1 -- 172.21.15.67:0/1404374180 wait complete. 2024-03-28T16:01:52.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:52.075+0000 7f7ae3000700 1 Processor -- start 2024-03-28T16:01:52.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:52.076+0000 7f7ae3000700 1 -- start start 2024-03-28T16:01:52.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:52.076+0000 7f7ae3000700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7adc1002b0 0x7f7adc1a0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:52.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:52.076+0000 7f7ae3000700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7adc100c70 0x7f7adc1a0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:52.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:52.076+0000 7f7ae3000700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7adc1082d0 0x7f7adc19a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:52.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:52.076+0000 7f7ae3000700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7adc112340 con 0x7f7adc100c70 2024-03-28T16:01:52.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:52.076+0000 7f7ae3000700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7adc1121c0 con 0x7f7adc1082d0 2024-03-28T16:01:52.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:52.076+0000 7f7ae3000700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7adc1124c0 con 0x7f7adc1002b0 2024-03-28T16:01:52.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:52.076+0000 7f7adb400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7adc100c70 0x7f7adc1a0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:52.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:52.076+0000 7f7adb400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7adc100c70 0x7f7adc1a0bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50660/0 (socket says 172.21.15.67:50660) 2024-03-28T16:01:52.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:52.076+0000 7f7adb400700 1 -- 172.21.15.67:0/440733157 learned_addr learned my addr 172.21.15.67:0/440733157 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:01:52.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:52.077+0000 7f7adbe00700 1 --2- 172.21.15.67:0/440733157 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7adc1002b0 0x7f7adc1a0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:52.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:52.077+0000 7f7ae1000700 1 --2- 172.21.15.67:0/440733157 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7adc1082d0 0x7f7adc19a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:52.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:52.077+0000 7f7adb400700 1 -- 172.21.15.67:0/440733157 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7adc1002b0 msgr2=0x7f7adc1a0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:52.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:52.077+0000 7f7adb400700 1 --2- 172.21.15.67:0/440733157 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7adc1002b0 0x7f7adc1a0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:52.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:52.077+0000 7f7adb400700 1 -- 172.21.15.67:0/440733157 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7adc1082d0 msgr2=0x7f7adc19a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:52.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:52.077+0000 7f7adb400700 1 --2- 172.21.15.67:0/440733157 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7adc1082d0 0x7f7adc19a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:52.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:52.077+0000 7f7adb400700 1 -- 172.21.15.67:0/440733157 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7adc19afe0 con 0x7f7adc100c70 2024-03-28T16:01:52.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:52.077+0000 7f7ae1000700 1 --2- 172.21.15.67:0/440733157 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7adc1082d0 0x7f7adc19a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:01:52.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:52.077+0000 7f7adb400700 1 --2- 172.21.15.67:0/440733157 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7adc100c70 0x7f7adc1a0bc0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f7ac800e9c0 tx=0x7f7ac800ed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:52.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:52.077+0000 7f7ad8c00700 1 -- 172.21.15.67:0/440733157 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ac800cbd0 con 0x7f7adc100c70 2024-03-28T16:01:52.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:52.077+0000 7f7ad8c00700 1 -- 172.21.15.67:0/440733157 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7ac800cd70 con 0x7f7adc100c70 2024-03-28T16:01:52.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:52.077+0000 7f7ad8c00700 1 -- 172.21.15.67:0/440733157 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ac8010430 con 0x7f7adc100c70 2024-03-28T16:01:52.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:52.078+0000 7f7ae3000700 1 -- 172.21.15.67:0/440733157 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7adc19b2d0 con 0x7f7adc100c70 2024-03-28T16:01:52.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:52.078+0000 7f7ae3000700 1 -- 172.21.15.67:0/440733157 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7adc1a74e0 con 0x7f7adc100c70 2024-03-28T16:01:52.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:52.079+0000 7f7ae3000700 1 -- 172.21.15.67:0/440733157 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7adc0ff040 con 0x7f7adc100c70 2024-03-28T16:01:52.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:52.081+0000 7f7ad8c00700 1 -- 172.21.15.67:0/440733157 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f7ac8004750 con 0x7f7adc100c70 2024-03-28T16:01:52.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:52.082+0000 7f7ad8c00700 1 --2- 172.21.15.67:0/440733157 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7ac00a29a0 0x7f7ac00a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:52.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:52.082+0000 7f7ad8c00700 1 -- 172.21.15.67:0/440733157 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f7ac80cf450 con 0x7f7adc100c70 2024-03-28T16:01:52.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:52.083+0000 7f7adbe00700 1 --2- 172.21.15.67:0/440733157 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7ac00a29a0 0x7f7ac00a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:52.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:52.083+0000 7f7ad8c00700 1 --2- 172.21.15.67:0/440733157 >> [v2:172.21.15.183:6808/164284137,v1:172.21.15.183:6809/164284137] conn(0x7f7ac00ac4e0 0x7f7ac00ae980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:52.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:52.083+0000 7f7ad8c00700 1 -- 172.21.15.67:0/440733157 --> [v2:172.21.15.183:6808/164284137,v1:172.21.15.183:6809/164284137] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7ac8004500 con 0x7f7ac00ac4e0 2024-03-28T16:01:52.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:52.083+0000 7f7ad8c00700 1 -- 172.21.15.67:0/440733157 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7ac80cf800 con 0x7f7adc100c70 2024-03-28T16:01:52.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:52.083+0000 7f7adbe00700 1 --2- 172.21.15.67:0/440733157 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7ac00a29a0 0x7f7ac00a4e60 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f7acc005950 tx=0x7f7acc0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:52.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:52.083+0000 7f7ae1000700 1 --2- 172.21.15.67:0/440733157 >> [v2:172.21.15.183:6808/164284137,v1:172.21.15.183:6809/164284137] conn(0x7f7ac00ac4e0 0x7f7ac00ae980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:52.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:52.084+0000 7f7ae1000700 1 --2- 172.21.15.67:0/440733157 >> [v2:172.21.15.183:6808/164284137,v1:172.21.15.183:6809/164284137] conn(0x7f7ac00ac4e0 0x7f7ac00ae980 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:52.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:52.086+0000 7f7ad8c00700 1 -- 172.21.15.67:0/440733157 <== osd.5 v2:172.21.15.183:6808/164284137 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f7ac8004500 con 0x7f7ac00ac4e0 2024-03-28T16:01:52.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:52.105+0000 7f7ae3000700 1 -- 172.21.15.67:0/440733157 --> [v2:172.21.15.183:6808/164284137,v1:172.21.15.183:6809/164284137] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7adc101680 con 0x7f7ac00ac4e0 2024-03-28T16:01:52.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:52.106+0000 7f7ad8c00700 1 -- 172.21.15.67:0/440733157 <== osd.5 v2:172.21.15.183:6808/164284137 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f7adc101680 con 0x7f7ac00ac4e0 2024-03-28T16:01:52.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:52.106+0000 7f7ae3000700 1 -- 172.21.15.67:0/440733157 >> [v2:172.21.15.183:6808/164284137,v1:172.21.15.183:6809/164284137] conn(0x7f7ac00ac4e0 msgr2=0x7f7ac00ae980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:52.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:52.106+0000 7f7ae3000700 1 --2- 172.21.15.67:0/440733157 >> [v2:172.21.15.183:6808/164284137,v1:172.21.15.183:6809/164284137] conn(0x7f7ac00ac4e0 0x7f7ac00ae980 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:52.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:52.107+0000 7f7ae3000700 1 -- 172.21.15.67:0/440733157 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7ac00a29a0 msgr2=0x7f7ac00a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:52.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:52.107+0000 7f7ae3000700 1 --2- 172.21.15.67:0/440733157 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7ac00a29a0 0x7f7ac00a4e60 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f7acc005950 tx=0x7f7acc0058e0 comp rx=0 tx=0).stop 2024-03-28T16:01:52.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:52.107+0000 7f7ae3000700 1 -- 172.21.15.67:0/440733157 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7adc100c70 msgr2=0x7f7adc1a0bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:52.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:52.107+0000 7f7ae3000700 1 --2- 172.21.15.67:0/440733157 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7adc100c70 0x7f7adc1a0bc0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f7ac800e9c0 tx=0x7f7ac800ed80 comp rx=0 tx=0).stop 2024-03-28T16:01:52.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:52.107+0000 7f7ae1000700 1 -- 172.21.15.67:0/440733157 reap_dead start 2024-03-28T16:01:52.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:52.107+0000 7f7ae3000700 1 -- 172.21.15.67:0/440733157 shutdown_connections 2024-03-28T16:01:52.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:52.107+0000 7f7ae3000700 1 -- 172.21.15.67:0/440733157 >> 172.21.15.67:0/440733157 conn(0x7f7adc077260 msgr2=0x7f7adc0fe620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:52.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:52.107+0000 7f7ae3000700 1 -- 172.21.15.67:0/440733157 shutdown_connections 2024-03-28T16:01:52.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:52.107+0000 7f7ae3000700 1 -- 172.21.15.67:0/440733157 wait complete. 2024-03-28T16:01:52.961 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:52 smithi067 ceph-mon[31707]: pgmap v388: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 885 B/s rd, 1 op/s; 2 B/s, 0 objects/s recovering 2024-03-28T16:01:53.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:52 smithi150 ceph-mon[34825]: pgmap v388: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 885 B/s rd, 1 op/s; 2 B/s, 0 objects/s recovering 2024-03-28T16:01:53.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:52 smithi183 ceph-mon[35580]: pgmap v388: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 885 B/s rd, 1 op/s; 2 B/s, 0 objects/s recovering 2024-03-28T16:01:53.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.628+0000 7f2a6b200700 1 -- 172.21.15.67:0/3911385408 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a6410c660 msgr2=0x7f2a6410eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:53.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.628+0000 7f2a6b200700 1 --2- 172.21.15.67:0/3911385408 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a6410c660 0x7f2a6410eb20 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f2a6000b0a0 tx=0x7f2a6000b3b0 comp rx=0 tx=0).stop 2024-03-28T16:01:53.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.628+0000 7f2a6b200700 1 -- 172.21.15.67:0/3911385408 shutdown_connections 2024-03-28T16:01:53.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.628+0000 7f2a6b200700 1 --2- 172.21.15.67:0/3911385408 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a6410c660 0x7f2a6410eb20 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:53.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.628+0000 7f2a6b200700 1 --2- 172.21.15.67:0/3911385408 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2a64103230 0x7f2a64103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:53.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.628+0000 7f2a6b200700 1 --2- 172.21.15.67:0/3911385408 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2a64103b50 0x7f2a6410c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:53.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.628+0000 7f2a6b200700 1 -- 172.21.15.67:0/3911385408 >> 172.21.15.67:0/3911385408 conn(0x7f2a640fca60 msgr2=0x7f2a640fee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:53.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.628+0000 7f2a6b200700 1 -- 172.21.15.67:0/3911385408 shutdown_connections 2024-03-28T16:01:53.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.628+0000 7f2a6b200700 1 -- 172.21.15.67:0/3911385408 wait complete. 2024-03-28T16:01:53.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.629+0000 7f2a6b200700 1 Processor -- start 2024-03-28T16:01:53.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.630+0000 7f2a6b200700 1 -- start start 2024-03-28T16:01:53.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.630+0000 7f2a6b200700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2a64103230 0x7f2a64100e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:53.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.630+0000 7f2a6b200700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a64103b50 0x7f2a641013d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:53.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.630+0000 7f2a6b200700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2a6410c660 0x7f2a64101910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:53.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.630+0000 7f2a6b200700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a641122d0 con 0x7f2a64103b50 2024-03-28T16:01:53.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.630+0000 7f2a6b200700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a64112150 con 0x7f2a64103230 2024-03-28T16:01:53.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.630+0000 7f2a6b200700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a64112450 con 0x7f2a6410c660 2024-03-28T16:01:53.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.630+0000 7f2a69400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a64103b50 0x7f2a641013d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:53.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.630+0000 7f2a69400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a64103b50 0x7f2a641013d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50700/0 (socket says 172.21.15.67:50700) 2024-03-28T16:01:53.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.630+0000 7f2a69400700 1 -- 172.21.15.67:0/2472943905 learned_addr learned my addr 172.21.15.67:0/2472943905 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:01:53.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.630+0000 7f2a69e00700 1 --2- 172.21.15.67:0/2472943905 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2a64103230 0x7f2a64100e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:53.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.631+0000 7f2a6a800700 1 --2- 172.21.15.67:0/2472943905 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2a6410c660 0x7f2a64101910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:53.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.631+0000 7f2a69400700 1 -- 172.21.15.67:0/2472943905 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2a6410c660 msgr2=0x7f2a64101910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:53.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.631+0000 7f2a69400700 1 --2- 172.21.15.67:0/2472943905 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2a6410c660 0x7f2a64101910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:53.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.631+0000 7f2a69400700 1 -- 172.21.15.67:0/2472943905 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2a64103230 msgr2=0x7f2a64100e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:53.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.631+0000 7f2a69400700 1 --2- 172.21.15.67:0/2472943905 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2a64103230 0x7f2a64100e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:53.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.631+0000 7f2a69400700 1 -- 172.21.15.67:0/2472943905 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a641a6ca0 con 0x7f2a64103b50 2024-03-28T16:01:53.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.631+0000 7f2a69400700 1 --2- 172.21.15.67:0/2472943905 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a64103b50 0x7f2a641013d0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f2a5400d720 tx=0x7f2a5400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:53.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.631+0000 7f2a5ea00700 1 -- 172.21.15.67:0/2472943905 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a54017070 con 0x7f2a64103b50 2024-03-28T16:01:53.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.631+0000 7f2a5ea00700 1 -- 172.21.15.67:0/2472943905 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f2a54005020 con 0x7f2a64103b50 2024-03-28T16:01:53.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.631+0000 7f2a6b200700 1 -- 172.21.15.67:0/2472943905 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a641a6f90 con 0x7f2a64103b50 2024-03-28T16:01:53.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.631+0000 7f2a5ea00700 1 -- 172.21.15.67:0/2472943905 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a5400fab0 con 0x7f2a64103b50 2024-03-28T16:01:53.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.632+0000 7f2a6b200700 1 -- 172.21.15.67:0/2472943905 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a641a73f0 con 0x7f2a64103b50 2024-03-28T16:01:53.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.632+0000 7f2a6b200700 1 -- 172.21.15.67:0/2472943905 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2a640628a0 con 0x7f2a64103b50 2024-03-28T16:01:53.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.636+0000 7f2a5ea00700 1 -- 172.21.15.67:0/2472943905 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f2a540051d0 con 0x7f2a64103b50 2024-03-28T16:01:53.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.637+0000 7f2a5ea00700 1 --2- 172.21.15.67:0/2472943905 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f2a4c0a2920 0x7f2a4c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:53.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.637+0000 7f2a5ea00700 1 -- 172.21.15.67:0/2472943905 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f2a540ce780 con 0x7f2a64103b50 2024-03-28T16:01:53.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.637+0000 7f2a69e00700 1 --2- 172.21.15.67:0/2472943905 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f2a4c0a2920 0x7f2a4c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:53.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.637+0000 7f2a5ea00700 1 --2- 172.21.15.67:0/2472943905 >> [v2:172.21.15.150:6808/3775269993,v1:172.21.15.150:6809/3775269993] conn(0x7f2a4c0ac460 0x7f2a4c0ae900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:53.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.637+0000 7f2a5ea00700 1 -- 172.21.15.67:0/2472943905 --> [v2:172.21.15.150:6808/3775269993,v1:172.21.15.150:6809/3775269993] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2a54014ec0 con 0x7f2a4c0ac460 2024-03-28T16:01:53.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.637+0000 7f2a5ea00700 1 -- 172.21.15.67:0/2472943905 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2a540ceb30 con 0x7f2a64103b50 2024-03-28T16:01:53.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.638+0000 7f2a69e00700 1 --2- 172.21.15.67:0/2472943905 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f2a4c0a2920 0x7f2a4c0a4de0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f2a58005950 tx=0x7f2a580058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:53.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.638+0000 7f2a6a800700 1 --2- 172.21.15.67:0/2472943905 >> [v2:172.21.15.150:6808/3775269993,v1:172.21.15.150:6809/3775269993] conn(0x7f2a4c0ac460 0x7f2a4c0ae900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:53.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.639+0000 7f2a6a800700 1 --2- 172.21.15.67:0/2472943905 >> [v2:172.21.15.150:6808/3775269993,v1:172.21.15.150:6809/3775269993] conn(0x7f2a4c0ac460 0x7f2a4c0ae900 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:53.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.641+0000 7f2a5ea00700 1 -- 172.21.15.67:0/2472943905 <== osd.4 v2:172.21.15.150:6808/3775269993 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f2a54014ec0 con 0x7f2a4c0ac460 2024-03-28T16:01:53.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.659+0000 7f2a6b200700 1 -- 172.21.15.67:0/2472943905 --> [v2:172.21.15.150:6808/3775269993,v1:172.21.15.150:6809/3775269993] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2a64062d20 con 0x7f2a4c0ac460 2024-03-28T16:01:53.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.660+0000 7f2a5ea00700 1 -- 172.21.15.67:0/2472943905 <== osd.4 v2:172.21.15.150:6808/3775269993 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f2a64062d20 con 0x7f2a4c0ac460 2024-03-28T16:01:53.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.660+0000 7f2a6b200700 1 -- 172.21.15.67:0/2472943905 >> [v2:172.21.15.150:6808/3775269993,v1:172.21.15.150:6809/3775269993] conn(0x7f2a4c0ac460 msgr2=0x7f2a4c0ae900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:53.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.660+0000 7f2a6b200700 1 --2- 172.21.15.67:0/2472943905 >> [v2:172.21.15.150:6808/3775269993,v1:172.21.15.150:6809/3775269993] conn(0x7f2a4c0ac460 0x7f2a4c0ae900 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:53.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.660+0000 7f2a6b200700 1 -- 172.21.15.67:0/2472943905 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f2a4c0a2920 msgr2=0x7f2a4c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:53.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.660+0000 7f2a6b200700 1 --2- 172.21.15.67:0/2472943905 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f2a4c0a2920 0x7f2a4c0a4de0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f2a58005950 tx=0x7f2a580058e0 comp rx=0 tx=0).stop 2024-03-28T16:01:53.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.661+0000 7f2a6b200700 1 -- 172.21.15.67:0/2472943905 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a64103b50 msgr2=0x7f2a641013d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:53.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.661+0000 7f2a6b200700 1 --2- 172.21.15.67:0/2472943905 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a64103b50 0x7f2a641013d0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f2a5400d720 tx=0x7f2a5400dae0 comp rx=0 tx=0).stop 2024-03-28T16:01:53.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.661+0000 7f2a6a800700 1 -- 172.21.15.67:0/2472943905 reap_dead start 2024-03-28T16:01:53.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.661+0000 7f2a6b200700 1 -- 172.21.15.67:0/2472943905 shutdown_connections 2024-03-28T16:01:53.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.661+0000 7f2a6b200700 1 -- 172.21.15.67:0/2472943905 >> 172.21.15.67:0/2472943905 conn(0x7f2a640fca60 msgr2=0x7f2a640fd770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:53.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.661+0000 7f2a6b200700 1 -- 172.21.15.67:0/2472943905 shutdown_connections 2024-03-28T16:01:53.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.661+0000 7f2a6b200700 1 -- 172.21.15.67:0/2472943905 wait complete. 2024-03-28T16:01:54.542 INFO:teuthology.orchestra.run.smithi067.stdout:55834574940 2024-03-28T16:01:54.543 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-28T16:01:54.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.208+0000 7f091ae00700 1 -- 172.21.15.67:0/3848223939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09141086c0 msgr2=0x7f0914108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:54.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.208+0000 7f091ae00700 1 --2- 172.21.15.67:0/3848223939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09141086c0 0x7f0914108b40 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f0900009990 tx=0x7f0900009ca0 comp rx=0 tx=0).stop 2024-03-28T16:01:54.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.208+0000 7f091ae00700 1 -- 172.21.15.67:0/3848223939 shutdown_connections 2024-03-28T16:01:54.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.208+0000 7f091ae00700 1 --2- 172.21.15.67:0/3848223939 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0914109080 0x7f0914115540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:54.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.208+0000 7f091ae00700 1 --2- 172.21.15.67:0/3848223939 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f091410e670 0x7f091410ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:54.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.208+0000 7f091ae00700 1 --2- 172.21.15.67:0/3848223939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09141086c0 0x7f0914108b40 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:54.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.208+0000 7f091ae00700 1 -- 172.21.15.67:0/3848223939 >> 172.21.15.67:0/3848223939 conn(0x7f0914077e30 msgr2=0x7f091407a250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:54.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.208+0000 7f091ae00700 1 -- 172.21.15.67:0/3848223939 shutdown_connections 2024-03-28T16:01:54.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.208+0000 7f091ae00700 1 -- 172.21.15.67:0/3848223939 wait complete. 2024-03-28T16:01:54.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.209+0000 7f091ae00700 1 Processor -- start 2024-03-28T16:01:54.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.209+0000 7f091ae00700 1 -- start start 2024-03-28T16:01:54.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.210+0000 7f091ae00700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09141086c0 0x7f09141a4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:54.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.210+0000 7f091ae00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0914109080 0x7f09141a4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:54.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.210+0000 7f091ae00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f091410e670 0x7f091419eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:54.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.210+0000 7f091ae00700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09141188b0 con 0x7f091410e670 2024-03-28T16:01:54.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.210+0000 7f091ae00700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0914118730 con 0x7f09141086c0 2024-03-28T16:01:54.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.210+0000 7f091ae00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0914118a30 con 0x7f0914109080 2024-03-28T16:01:54.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.210+0000 7f091a400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f091410e670 0x7f091419eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:54.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.210+0000 7f091a400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f091410e670 0x7f091419eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50678/0 (socket says 172.21.15.67:50678) 2024-03-28T16:01:54.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.211+0000 7f091a400700 1 -- 172.21.15.67:0/610558826 learned_addr learned my addr 172.21.15.67:0/610558826 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:01:54.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.211+0000 7f0919a00700 1 --2- 172.21.15.67:0/610558826 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09141086c0 0x7f09141a4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:54.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.211+0000 7f0919000700 1 --2- 172.21.15.67:0/610558826 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0914109080 0x7f09141a4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:54.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.211+0000 7f091a400700 1 -- 172.21.15.67:0/610558826 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0914109080 msgr2=0x7f09141a4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:54.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.211+0000 7f091a400700 1 --2- 172.21.15.67:0/610558826 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0914109080 0x7f09141a4fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:54.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.211+0000 7f091a400700 1 -- 172.21.15.67:0/610558826 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09141086c0 msgr2=0x7f09141a4a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:54.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.211+0000 7f091a400700 1 --2- 172.21.15.67:0/610558826 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09141086c0 0x7f09141a4a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:54.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.211+0000 7f091a400700 1 -- 172.21.15.67:0/610558826 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f091419f410 con 0x7f091410e670 2024-03-28T16:01:54.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.212+0000 7f091a400700 1 --2- 172.21.15.67:0/610558826 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f091410e670 0x7f091419eb30 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f090c007d70 tx=0x7f090c00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:54.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.212+0000 7f0912000700 1 -- 172.21.15.67:0/610558826 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f090c016020 con 0x7f091410e670 2024-03-28T16:01:54.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.212+0000 7f0912000700 1 -- 172.21.15.67:0/610558826 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f090c00ac90 con 0x7f091410e670 2024-03-28T16:01:54.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.212+0000 7f0912000700 1 -- 172.21.15.67:0/610558826 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f090c0115e0 con 0x7f091410e670 2024-03-28T16:01:54.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.212+0000 7f091ae00700 1 -- 172.21.15.67:0/610558826 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f091419f700 con 0x7f091410e670 2024-03-28T16:01:54.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.212+0000 7f091ae00700 1 -- 172.21.15.67:0/610558826 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09141ab9e0 con 0x7f091410e670 2024-03-28T16:01:54.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.213+0000 7f091ae00700 1 -- 172.21.15.67:0/610558826 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0914062920 con 0x7f091410e670 2024-03-28T16:01:54.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.215+0000 7f0912000700 1 -- 172.21.15.67:0/610558826 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f090c019020 con 0x7f091410e670 2024-03-28T16:01:54.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.217+0000 7f0912000700 1 --2- 172.21.15.67:0/610558826 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f08fc0a29a0 0x7f08fc0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:54.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.217+0000 7f0912000700 1 -- 172.21.15.67:0/610558826 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f090c0cede0 con 0x7f091410e670 2024-03-28T16:01:54.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.217+0000 7f0919a00700 1 --2- 172.21.15.67:0/610558826 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f08fc0a29a0 0x7f08fc0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:54.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.217+0000 7f0912000700 1 --2- 172.21.15.67:0/610558826 >> [v2:172.21.15.67:6818/3565528013,v1:172.21.15.67:6819/3565528013] conn(0x7f08fc0ac4e0 0x7f08fc0ae980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:54.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.217+0000 7f0912000700 1 -- 172.21.15.67:0/610558826 --> [v2:172.21.15.67:6818/3565528013,v1:172.21.15.67:6819/3565528013] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f08fc0af050 con 0x7f08fc0ac4e0 2024-03-28T16:01:54.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.217+0000 7f0912000700 1 -- 172.21.15.67:0/610558826 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) v2 ==== 24+0+0 (secure 0 0 0) 0x7f090c0cf190 con 0x7f091410e670 2024-03-28T16:01:54.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.218+0000 7f0919000700 1 --2- 172.21.15.67:0/610558826 >> [v2:172.21.15.67:6818/3565528013,v1:172.21.15.67:6819/3565528013] conn(0x7f08fc0ac4e0 0x7f08fc0ae980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:54.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.218+0000 7f0919a00700 1 --2- 172.21.15.67:0/610558826 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f08fc0a29a0 0x7f08fc0a4e60 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f0908009a10 tx=0x7f0908009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:54.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.218+0000 7f0919000700 1 --2- 172.21.15.67:0/610558826 >> [v2:172.21.15.67:6818/3565528013,v1:172.21.15.67:6819/3565528013] conn(0x7f08fc0ac4e0 0x7f08fc0ae980 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:54.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.220+0000 7f0912000700 1 -- 172.21.15.67:0/610558826 <== osd.6 v2:172.21.15.67:6818/3565528013 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f08fc0af050 con 0x7f08fc0ac4e0 2024-03-28T16:01:54.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.238+0000 7f091ae00700 1 -- 172.21.15.67:0/610558826 --> [v2:172.21.15.67:6818/3565528013,v1:172.21.15.67:6819/3565528013] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0914062da0 con 0x7f08fc0ac4e0 2024-03-28T16:01:54.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.239+0000 7f0912000700 1 -- 172.21.15.67:0/610558826 <== osd.6 v2:172.21.15.67:6818/3565528013 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f0914062da0 con 0x7f08fc0ac4e0 2024-03-28T16:01:54.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.239+0000 7f091ae00700 1 -- 172.21.15.67:0/610558826 >> [v2:172.21.15.67:6818/3565528013,v1:172.21.15.67:6819/3565528013] conn(0x7f08fc0ac4e0 msgr2=0x7f08fc0ae980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:54.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.239+0000 7f091ae00700 1 --2- 172.21.15.67:0/610558826 >> [v2:172.21.15.67:6818/3565528013,v1:172.21.15.67:6819/3565528013] conn(0x7f08fc0ac4e0 0x7f08fc0ae980 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:54.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.239+0000 7f091ae00700 1 -- 172.21.15.67:0/610558826 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f08fc0a29a0 msgr2=0x7f08fc0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:54.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.239+0000 7f091ae00700 1 --2- 172.21.15.67:0/610558826 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f08fc0a29a0 0x7f08fc0a4e60 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f0908009a10 tx=0x7f0908009190 comp rx=0 tx=0).stop 2024-03-28T16:01:54.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.239+0000 7f091ae00700 1 -- 172.21.15.67:0/610558826 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f091410e670 msgr2=0x7f091419eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:54.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.239+0000 7f091ae00700 1 --2- 172.21.15.67:0/610558826 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f091410e670 0x7f091419eb30 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f090c007d70 tx=0x7f090c00a340 comp rx=0 tx=0).stop 2024-03-28T16:01:54.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.239+0000 7f091a400700 1 -- 172.21.15.67:0/610558826 reap_dead start 2024-03-28T16:01:54.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.240+0000 7f091ae00700 1 -- 172.21.15.67:0/610558826 shutdown_connections 2024-03-28T16:01:54.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.240+0000 7f091ae00700 1 -- 172.21.15.67:0/610558826 >> 172.21.15.67:0/610558826 conn(0x7f0914077e30 msgr2=0x7f0914078420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:54.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.240+0000 7f091ae00700 1 -- 172.21.15.67:0/610558826 shutdown_connections 2024-03-28T16:01:54.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:53.240+0000 7f091ae00700 1 -- 172.21.15.67:0/610558826 wait complete. 2024-03-28T16:01:54.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:54 smithi067 ceph-mon[31707]: pgmap v389: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 1 op/s; 1 B/s, 0 objects/s recovering 2024-03-28T16:01:55.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:54 smithi150 ceph-mon[34825]: pgmap v389: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 1 op/s; 1 B/s, 0 objects/s recovering 2024-03-28T16:01:55.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:54 smithi183 ceph-mon[35580]: pgmap v389: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 1 op/s; 1 B/s, 0 objects/s recovering 2024-03-28T16:01:55.588 INFO:teuthology.orchestra.run.smithi067.stdout:330712481823 2024-03-28T16:01:55.589 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.10 2024-03-28T16:01:55.720 INFO:teuthology.orchestra.run.smithi067.stdout:176093659202 2024-03-28T16:01:55.721 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-28T16:01:56.050 INFO:teuthology.orchestra.run.smithi067.stdout:210453397564 2024-03-28T16:01:56.050 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-28T16:01:56.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:56.261+0000 7efd9c000700 1 -- 172.21.15.67:0/584472132 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd94069e40 msgr2=0x7efd9406a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:56.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:56.261+0000 7efd9c000700 1 --2- 172.21.15.67:0/584472132 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd94069e40 0x7efd9406a2a0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7efd800099c0 tx=0x7efd80009cd0 comp rx=0 tx=0).stop 2024-03-28T16:01:56.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:56.261+0000 7efd9c000700 1 -- 172.21.15.67:0/584472132 shutdown_connections 2024-03-28T16:01:56.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:56.261+0000 7efd9c000700 1 --2- 172.21.15.67:0/584472132 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efd9406a7e0 0x7efd9410f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:56.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:56.261+0000 7efd9c000700 1 --2- 172.21.15.67:0/584472132 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd941081e0 0x7efd941085c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:56.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:56.261+0000 7efd9c000700 1 --2- 172.21.15.67:0/584472132 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd94069e40 0x7efd9406a2a0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:56.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:56.261+0000 7efd9c000700 1 -- 172.21.15.67:0/584472132 >> 172.21.15.67:0/584472132 conn(0x7efd94077fc0 msgr2=0x7efd940783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:56.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:56.261+0000 7efd9c000700 1 -- 172.21.15.67:0/584472132 shutdown_connections 2024-03-28T16:01:56.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:56.261+0000 7efd9c000700 1 -- 172.21.15.67:0/584472132 wait complete. 2024-03-28T16:01:56.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:56.263+0000 7efd9c000700 1 Processor -- start 2024-03-28T16:01:56.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:56.263+0000 7efd9c000700 1 -- start start 2024-03-28T16:01:56.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:56.264+0000 7efd9c000700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd94069e40 0x7efd941a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:56.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:56.264+0000 7efd9c000700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd9406a7e0 0x7efd941a0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:56.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:56.264+0000 7efd9c000700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efd941081e0 0x7efd9419a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:56.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:56.264+0000 7efd9c000700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd94112310 con 0x7efd9406a7e0 2024-03-28T16:01:56.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:56.264+0000 7efd9c000700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd94112190 con 0x7efd94069e40 2024-03-28T16:01:56.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:56.264+0000 7efd9c000700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd94112490 con 0x7efd941081e0 2024-03-28T16:01:56.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:56.264+0000 7efd98c00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd9406a7e0 0x7efd941a0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:56.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:56.264+0000 7efd98c00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd9406a7e0 0x7efd941a0bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50718/0 (socket says 172.21.15.67:50718) 2024-03-28T16:01:56.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:56.264+0000 7efd98c00700 1 -- 172.21.15.67:0/3546138437 learned_addr learned my addr 172.21.15.67:0/3546138437 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:01:56.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:56.265+0000 7efd99600700 1 --2- 172.21.15.67:0/3546138437 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd94069e40 0x7efd941a06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:56.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:56.265+0000 7efd9a000700 1 --2- 172.21.15.67:0/3546138437 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efd941081e0 0x7efd9419a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:56.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:56.265+0000 7efd98c00700 1 -- 172.21.15.67:0/3546138437 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efd941081e0 msgr2=0x7efd9419a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:56.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:56.265+0000 7efd98c00700 1 --2- 172.21.15.67:0/3546138437 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efd941081e0 0x7efd9419a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:56.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:56.265+0000 7efd98c00700 1 -- 172.21.15.67:0/3546138437 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd94069e40 msgr2=0x7efd941a06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:56.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:56.265+0000 7efd98c00700 1 --2- 172.21.15.67:0/3546138437 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd94069e40 0x7efd941a06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:56.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:56.265+0000 7efd98c00700 1 -- 172.21.15.67:0/3546138437 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd9419b010 con 0x7efd9406a7e0 2024-03-28T16:01:56.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:56.265+0000 7efd9a000700 1 --2- 172.21.15.67:0/3546138437 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efd941081e0 0x7efd9419a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:01:56.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:56.266+0000 7efd98c00700 1 --2- 172.21.15.67:0/3546138437 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd9406a7e0 0x7efd941a0bf0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7efd80006010 tx=0x7efd80009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:56.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:56.266+0000 7efd92000700 1 -- 172.21.15.67:0/3546138437 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd80027070 con 0x7efd9406a7e0 2024-03-28T16:01:56.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:56.267+0000 7efd92000700 1 -- 172.21.15.67:0/3546138437 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7efd80003fd0 con 0x7efd9406a7e0 2024-03-28T16:01:56.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:56.267+0000 7efd92000700 1 -- 172.21.15.67:0/3546138437 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd80022d70 con 0x7efd9406a7e0 2024-03-28T16:01:56.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:56.267+0000 7efd9c000700 1 -- 172.21.15.67:0/3546138437 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd9419b2a0 con 0x7efd9406a7e0 2024-03-28T16:01:56.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:56.267+0000 7efd9c000700 1 -- 172.21.15.67:0/3546138437 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd94108f60 con 0x7efd9406a7e0 2024-03-28T16:01:56.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:56.268+0000 7efd9c000700 1 -- 172.21.15.67:0/3546138437 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7efd9410ddd0 con 0x7efd9406a7e0 2024-03-28T16:01:56.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:56.271+0000 7efd92000700 1 -- 172.21.15.67:0/3546138437 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7efd8002a070 con 0x7efd9406a7e0 2024-03-28T16:01:56.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:56.273+0000 7efd92000700 1 --2- 172.21.15.67:0/3546138437 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7efd7c0ab1b0 0x7efd7c0ad670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:56.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:56.273+0000 7efd92000700 1 -- 172.21.15.67:0/3546138437 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7efd800cfd30 con 0x7efd9406a7e0 2024-03-28T16:01:56.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:56.273+0000 7efd99600700 1 --2- 172.21.15.67:0/3546138437 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7efd7c0ab1b0 0x7efd7c0ad670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:56.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:56.273+0000 7efd92000700 1 --2- 172.21.15.67:0/3546138437 >> [v2:172.21.15.183:6800/3398095931,v1:172.21.15.183:6801/3398095931] conn(0x7efd7c0b4cf0 0x7efd7c0b7190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:56.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:56.273+0000 7efd92000700 1 -- 172.21.15.67:0/3546138437 --> [v2:172.21.15.183:6800/3398095931,v1:172.21.15.183:6801/3398095931] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efd7c0b7860 con 0x7efd7c0b4cf0 2024-03-28T16:01:56.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:56.273+0000 7efd92000700 1 -- 172.21.15.67:0/3546138437 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) v2 ==== 24+0+0 (secure 0 0 0) 0x7efd800d00e0 con 0x7efd9406a7e0 2024-03-28T16:01:56.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:56.274+0000 7efd9a000700 1 --2- 172.21.15.67:0/3546138437 >> [v2:172.21.15.183:6800/3398095931,v1:172.21.15.183:6801/3398095931] conn(0x7efd7c0b4cf0 0x7efd7c0b7190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:56.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:56.274+0000 7efd99600700 1 --2- 172.21.15.67:0/3546138437 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7efd7c0ab1b0 0x7efd7c0ad670 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7efd88005950 tx=0x7efd880058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:56.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:56.276+0000 7efd9a000700 1 --2- 172.21.15.67:0/3546138437 >> [v2:172.21.15.183:6800/3398095931,v1:172.21.15.183:6801/3398095931] conn(0x7efd7c0b4cf0 0x7efd7c0b7190 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:56.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:56.279+0000 7efd92000700 1 -- 172.21.15.67:0/3546138437 <== osd.2 v2:172.21.15.183:6800/3398095931 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7efd7c0b7860 con 0x7efd7c0b4cf0 2024-03-28T16:01:56.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:56.297+0000 7efd9c000700 1 -- 172.21.15.67:0/3546138437 --> [v2:172.21.15.183:6800/3398095931,v1:172.21.15.183:6801/3398095931] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7efd94103890 con 0x7efd7c0b4cf0 2024-03-28T16:01:56.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:56.298+0000 7efd92000700 1 -- 172.21.15.67:0/3546138437 <== osd.2 v2:172.21.15.183:6800/3398095931 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7efd94103890 con 0x7efd7c0b4cf0 2024-03-28T16:01:56.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:56.299+0000 7efd9c000700 1 -- 172.21.15.67:0/3546138437 >> [v2:172.21.15.183:6800/3398095931,v1:172.21.15.183:6801/3398095931] conn(0x7efd7c0b4cf0 msgr2=0x7efd7c0b7190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:56.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:56.299+0000 7efd9c000700 1 --2- 172.21.15.67:0/3546138437 >> [v2:172.21.15.183:6800/3398095931,v1:172.21.15.183:6801/3398095931] conn(0x7efd7c0b4cf0 0x7efd7c0b7190 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:56.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:56.299+0000 7efd9c000700 1 -- 172.21.15.67:0/3546138437 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7efd7c0ab1b0 msgr2=0x7efd7c0ad670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:56.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:56.299+0000 7efd9c000700 1 --2- 172.21.15.67:0/3546138437 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7efd7c0ab1b0 0x7efd7c0ad670 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7efd88005950 tx=0x7efd880058e0 comp rx=0 tx=0).stop 2024-03-28T16:01:56.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:56.299+0000 7efd9c000700 1 -- 172.21.15.67:0/3546138437 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd9406a7e0 msgr2=0x7efd941a0bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:56.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:56.299+0000 7efd9c000700 1 --2- 172.21.15.67:0/3546138437 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd9406a7e0 0x7efd941a0bf0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7efd80006010 tx=0x7efd80009f80 comp rx=0 tx=0).stop 2024-03-28T16:01:56.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:56.299+0000 7efd9a000700 1 -- 172.21.15.67:0/3546138437 reap_dead start 2024-03-28T16:01:56.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:56.299+0000 7efd9c000700 1 -- 172.21.15.67:0/3546138437 shutdown_connections 2024-03-28T16:01:56.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:56.299+0000 7efd9c000700 1 -- 172.21.15.67:0/3546138437 >> 172.21.15.67:0/3546138437 conn(0x7efd94077fc0 msgr2=0x7efd9410d370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:56.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:56.299+0000 7efd9c000700 1 -- 172.21.15.67:0/3546138437 shutdown_connections 2024-03-28T16:01:56.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:56.299+0000 7efd9c000700 1 -- 172.21.15.67:0/3546138437 wait complete. 2024-03-28T16:01:56.524 INFO:teuthology.orchestra.run.smithi067.stdout:270582939692 2024-03-28T16:01:56.524 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.8 2024-03-28T16:01:57.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:56 smithi150 ceph-mon[34825]: pgmap v390: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s; 0 B/s, 0 objects/s recovering 2024-03-28T16:01:57.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:56 smithi067 ceph-mon[31707]: pgmap v390: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s; 0 B/s, 0 objects/s recovering 2024-03-28T16:01:57.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:56 smithi183 ceph-mon[35580]: pgmap v390: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s; 0 B/s, 0 objects/s recovering 2024-03-28T16:01:57.489 INFO:teuthology.orchestra.run.smithi067.stdout:150323855432 2024-03-28T16:01:57.489 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-28T16:01:58.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:58.060+0000 7f9331200700 1 -- 172.21.15.67:0/103612221 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f932c0782f0 msgr2=0x7f932c0786d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:58.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:58.060+0000 7f9331200700 1 --2- 172.21.15.67:0/103612221 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f932c0782f0 0x7f932c0786d0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f931c009990 tx=0x7f931c009ca0 comp rx=0 tx=0).stop 2024-03-28T16:01:58.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:58.060+0000 7f9331200700 1 -- 172.21.15.67:0/103612221 shutdown_connections 2024-03-28T16:01:58.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:58.060+0000 7f9331200700 1 --2- 172.21.15.67:0/103612221 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f932c113000 0x7f932c1153f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:58.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:58.060+0000 7f9331200700 1 --2- 172.21.15.67:0/103612221 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f932c0782f0 0x7f932c0786d0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:58.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:58.060+0000 7f9331200700 1 --2- 172.21.15.67:0/103612221 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f932c078ca0 0x7f932c112ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:58.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:58.060+0000 7f9331200700 1 -- 172.21.15.67:0/103612221 >> 172.21.15.67:0/103612221 conn(0x7f932c0ffe30 msgr2=0x7f932c102250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:58.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:58.060+0000 7f9331200700 1 -- 172.21.15.67:0/103612221 shutdown_connections 2024-03-28T16:01:58.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:58.060+0000 7f9331200700 1 -- 172.21.15.67:0/103612221 wait complete. 2024-03-28T16:01:58.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:58.061+0000 7f9331200700 1 Processor -- start 2024-03-28T16:01:58.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:58.061+0000 7f9331200700 1 -- start start 2024-03-28T16:01:58.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:58.061+0000 7f9331200700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f932c0782f0 0x7f932c19f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:58.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:58.062+0000 7f9331200700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f932c078ca0 0x7f932c19f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:58.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:58.062+0000 7f9331200700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f932c113000 0x7f932c1a38d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:58.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:58.062+0000 7f9331200700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f932c118af0 con 0x7f932c113000 2024-03-28T16:01:58.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:58.062+0000 7f9331200700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f932c118970 con 0x7f932c078ca0 2024-03-28T16:01:58.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:58.062+0000 7f9331200700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f932c118c70 con 0x7f932c0782f0 2024-03-28T16:01:58.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:58.062+0000 7f932a800700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f932c113000 0x7f932c1a38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:58.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:58.062+0000 7f932a800700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f932c113000 0x7f932c1a38d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50734/0 (socket says 172.21.15.67:50734) 2024-03-28T16:01:58.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:58.062+0000 7f932a800700 1 -- 172.21.15.67:0/1833996944 learned_addr learned my addr 172.21.15.67:0/1833996944 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:01:58.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:58.062+0000 7f9329e00700 1 --2- 172.21.15.67:0/1833996944 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f932c0782f0 0x7f932c19f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:58.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:58.062+0000 7f9329400700 1 --2- 172.21.15.67:0/1833996944 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f932c078ca0 0x7f932c19f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:58.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:58.063+0000 7f932a800700 1 -- 172.21.15.67:0/1833996944 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f932c0782f0 msgr2=0x7f932c19f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:58.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:58.063+0000 7f932a800700 1 --2- 172.21.15.67:0/1833996944 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f932c0782f0 0x7f932c19f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:58.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:58.063+0000 7f932a800700 1 -- 172.21.15.67:0/1833996944 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f932c078ca0 msgr2=0x7f932c19f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:58.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:58.063+0000 7f932a800700 1 --2- 172.21.15.67:0/1833996944 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f932c078ca0 0x7f932c19f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:58.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:58.063+0000 7f932a800700 1 -- 172.21.15.67:0/1833996944 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f932c1a4010 con 0x7f932c113000 2024-03-28T16:01:58.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:58.063+0000 7f9329400700 1 --2- 172.21.15.67:0/1833996944 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f932c078ca0 0x7f932c19f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T16:01:58.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:58.063+0000 7f932a800700 1 --2- 172.21.15.67:0/1833996944 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f932c113000 0x7f932c1a38d0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f932400b970 tx=0x7f932400bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:58.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:58.063+0000 7f9322a00700 1 -- 172.21.15.67:0/1833996944 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9324019070 con 0x7f932c113000 2024-03-28T16:01:58.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:58.063+0000 7f9322a00700 1 -- 172.21.15.67:0/1833996944 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f932400cde0 con 0x7f932c113000 2024-03-28T16:01:58.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:58.063+0000 7f9322a00700 1 -- 172.21.15.67:0/1833996944 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9324007370 con 0x7f932c113000 2024-03-28T16:01:58.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:58.063+0000 7f9331200700 1 -- 172.21.15.67:0/1833996944 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f932c1a4300 con 0x7f932c113000 2024-03-28T16:01:58.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:58.064+0000 7f9331200700 1 -- 172.21.15.67:0/1833996944 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f932c1abb80 con 0x7f932c113000 2024-03-28T16:01:58.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:58.065+0000 7f9331200700 1 -- 172.21.15.67:0/1833996944 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f932c062920 con 0x7f932c113000 2024-03-28T16:01:58.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:58.067+0000 7f9322a00700 1 -- 172.21.15.67:0/1833996944 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f932401c070 con 0x7f932c113000 2024-03-28T16:01:58.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:58.068+0000 7f9322a00700 1 --2- 172.21.15.67:0/1833996944 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f93100a29a0 0x7f93100a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:58.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:58.068+0000 7f9322a00700 1 -- 172.21.15.67:0/1833996944 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f93240ce9e0 con 0x7f932c113000 2024-03-28T16:01:58.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:58.068+0000 7f9329e00700 1 --2- 172.21.15.67:0/1833996944 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f93100a29a0 0x7f93100a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:58.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:58.069+0000 7f9322a00700 1 --2- 172.21.15.67:0/1833996944 >> [v2:172.21.15.150:6816/3735115340,v1:172.21.15.150:6817/3735115340] conn(0x7f93100ac4e0 0x7f93100ae980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:58.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:58.069+0000 7f9322a00700 1 -- 172.21.15.67:0/1833996944 --> [v2:172.21.15.150:6816/3735115340,v1:172.21.15.150:6817/3735115340] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9324025ea0 con 0x7f93100ac4e0 2024-03-28T16:01:58.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:58.069+0000 7f9322a00700 1 -- 172.21.15.67:0/1833996944 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) v2 ==== 24+0+0 (secure 0 0 0) 0x7f93240ced90 con 0x7f932c113000 2024-03-28T16:01:58.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:58.069+0000 7f9329e00700 1 --2- 172.21.15.67:0/1833996944 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f93100a29a0 0x7f93100a4e60 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f931c0098e0 tx=0x7f931c005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:58.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:58.069+0000 7f9329400700 1 --2- 172.21.15.67:0/1833996944 >> [v2:172.21.15.150:6816/3735115340,v1:172.21.15.150:6817/3735115340] conn(0x7f93100ac4e0 0x7f93100ae980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:58.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:58.070+0000 7f9329400700 1 --2- 172.21.15.67:0/1833996944 >> [v2:172.21.15.150:6816/3735115340,v1:172.21.15.150:6817/3735115340] conn(0x7f93100ac4e0 0x7f93100ae980 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:58.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:58.073+0000 7f9322a00700 1 -- 172.21.15.67:0/1833996944 <== osd.7 v2:172.21.15.150:6816/3735115340 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9324025ea0 con 0x7f93100ac4e0 2024-03-28T16:01:58.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:58.091+0000 7f9331200700 1 -- 172.21.15.67:0/1833996944 --> [v2:172.21.15.150:6816/3735115340,v1:172.21.15.150:6817/3735115340] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f932c062da0 con 0x7f93100ac4e0 2024-03-28T16:01:58.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:58.092+0000 7f9322a00700 1 -- 172.21.15.67:0/1833996944 <== osd.7 v2:172.21.15.150:6816/3735115340 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f932c062da0 con 0x7f93100ac4e0 2024-03-28T16:01:58.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:58.093+0000 7f9331200700 1 -- 172.21.15.67:0/1833996944 >> [v2:172.21.15.150:6816/3735115340,v1:172.21.15.150:6817/3735115340] conn(0x7f93100ac4e0 msgr2=0x7f93100ae980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:58.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:58.093+0000 7f9331200700 1 --2- 172.21.15.67:0/1833996944 >> [v2:172.21.15.150:6816/3735115340,v1:172.21.15.150:6817/3735115340] conn(0x7f93100ac4e0 0x7f93100ae980 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:58.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:58.093+0000 7f9331200700 1 -- 172.21.15.67:0/1833996944 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f93100a29a0 msgr2=0x7f93100a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:58.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:58.093+0000 7f9331200700 1 --2- 172.21.15.67:0/1833996944 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f93100a29a0 0x7f93100a4e60 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f931c0098e0 tx=0x7f931c005960 comp rx=0 tx=0).stop 2024-03-28T16:01:58.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:58.093+0000 7f9331200700 1 -- 172.21.15.67:0/1833996944 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f932c113000 msgr2=0x7f932c1a38d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:58.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:58.093+0000 7f9331200700 1 --2- 172.21.15.67:0/1833996944 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f932c113000 0x7f932c1a38d0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f932400b970 tx=0x7f932400bc80 comp rx=0 tx=0).stop 2024-03-28T16:01:58.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:58.093+0000 7f932a800700 1 -- 172.21.15.67:0/1833996944 reap_dead start 2024-03-28T16:01:58.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:58.093+0000 7f9331200700 1 -- 172.21.15.67:0/1833996944 shutdown_connections 2024-03-28T16:01:58.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:58.093+0000 7f9331200700 1 -- 172.21.15.67:0/1833996944 >> 172.21.15.67:0/1833996944 conn(0x7f932c0ffe30 msgr2=0x7f932c102220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:58.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:58.093+0000 7f9331200700 1 -- 172.21.15.67:0/1833996944 shutdown_connections 2024-03-28T16:01:58.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:58.093+0000 7f9331200700 1 -- 172.21.15.67:0/1833996944 wait complete. 2024-03-28T16:01:58.298 INFO:teuthology.orchestra.run.smithi067.stdout:77309411416 2024-03-28T16:01:58.299 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-28T16:01:58.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:01:58 smithi067 ceph-mon[31707]: pgmap v391: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:01:58.924 INFO:teuthology.orchestra.run.smithi067.stdout:34359738466 2024-03-28T16:01:58.924 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-28T16:01:59.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:01:58 smithi150 ceph-mon[34825]: pgmap v391: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:01:59.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:01:58 smithi183 ceph-mon[35580]: pgmap v391: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:01:59.800 INFO:teuthology.orchestra.run.smithi067.stdout:236223201334 2024-03-28T16:01:59.801 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-28T16:01:59.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:59.529+0000 7fac5a800700 1 -- 172.21.15.67:0/2222629859 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac54104f90 msgr2=0x7fac54108fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:59.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:59.529+0000 7fac5a800700 1 --2- 172.21.15.67:0/2222629859 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac54104f90 0x7fac54108fe0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fac400099c0 tx=0x7fac40009cd0 comp rx=0 tx=0).stop 2024-03-28T16:01:59.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:59.529+0000 7fac5a800700 1 -- 172.21.15.67:0/2222629859 shutdown_connections 2024-03-28T16:01:59.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:59.529+0000 7fac5a800700 1 --2- 172.21.15.67:0/2222629859 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac541096c0 0x7fac54111290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:59.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:59.529+0000 7fac5a800700 1 --2- 172.21.15.67:0/2222629859 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fac541045e0 0x7fac541049c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:59.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:59.529+0000 7fac5a800700 1 --2- 172.21.15.67:0/2222629859 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac54104f90 0x7fac54108fe0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:59.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:59.529+0000 7fac5a800700 1 -- 172.21.15.67:0/2222629859 >> 172.21.15.67:0/2222629859 conn(0x7fac540ffe70 msgr2=0x7fac54102290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:59.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:59.530+0000 7fac5a800700 1 -- 172.21.15.67:0/2222629859 shutdown_connections 2024-03-28T16:01:59.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:59.530+0000 7fac5a800700 1 -- 172.21.15.67:0/2222629859 wait complete. 2024-03-28T16:01:59.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:59.531+0000 7fac5a800700 1 Processor -- start 2024-03-28T16:01:59.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:59.531+0000 7fac5a800700 1 -- start start 2024-03-28T16:01:59.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:59.531+0000 7fac5a800700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac541045e0 0x7fac541a06f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:59.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:59.531+0000 7fac5a800700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fac54104f90 0x7fac541a0c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:59.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:59.531+0000 7fac5a800700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac541096c0 0x7fac5419a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:59.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:59.531+0000 7fac5a800700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac541145a0 con 0x7fac541045e0 2024-03-28T16:01:59.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:59.531+0000 7fac5a800700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac54114420 con 0x7fac541096c0 2024-03-28T16:01:59.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:59.531+0000 7fac5a800700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac54114720 con 0x7fac54104f90 2024-03-28T16:01:59.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:59.532+0000 7fac53400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac541045e0 0x7fac541a06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:59.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:59.532+0000 7fac53400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac541045e0 0x7fac541a06f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34848/0 (socket says 172.21.15.67:34848) 2024-03-28T16:01:59.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:59.532+0000 7fac53400700 1 -- 172.21.15.67:0/2569172857 learned_addr learned my addr 172.21.15.67:0/2569172857 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:01:59.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:59.532+0000 7fac52a00700 1 --2- 172.21.15.67:0/2569172857 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fac54104f90 0x7fac541a0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:59.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:59.532+0000 7fac53e00700 1 --2- 172.21.15.67:0/2569172857 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac541096c0 0x7fac5419a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:59.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:59.532+0000 7fac53400700 1 -- 172.21.15.67:0/2569172857 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fac54104f90 msgr2=0x7fac541a0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:59.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:59.532+0000 7fac53400700 1 --2- 172.21.15.67:0/2569172857 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fac54104f90 0x7fac541a0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:59.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:59.532+0000 7fac53400700 1 -- 172.21.15.67:0/2569172857 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac541096c0 msgr2=0x7fac5419a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:59.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:59.532+0000 7fac53400700 1 --2- 172.21.15.67:0/2569172857 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac541096c0 0x7fac5419a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:59.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:59.532+0000 7fac53400700 1 -- 172.21.15.67:0/2569172857 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac5419b060 con 0x7fac541045e0 2024-03-28T16:01:59.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:59.532+0000 7fac53400700 1 --2- 172.21.15.67:0/2569172857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac541045e0 0x7fac541a06f0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fac4400d980 tx=0x7fac4400dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:59.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:59.533+0000 7fac4be00700 1 -- 172.21.15.67:0/2569172857 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac44017070 con 0x7fac541045e0 2024-03-28T16:01:59.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:59.533+0000 7fac4be00700 1 -- 172.21.15.67:0/2569172857 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fac44004830 con 0x7fac541045e0 2024-03-28T16:01:59.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:59.533+0000 7fac4be00700 1 -- 172.21.15.67:0/2569172857 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac44005440 con 0x7fac541045e0 2024-03-28T16:01:59.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:59.533+0000 7fac5a800700 1 -- 172.21.15.67:0/2569172857 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac5419b350 con 0x7fac541045e0 2024-03-28T16:01:59.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:59.533+0000 7fac5a800700 1 -- 172.21.15.67:0/2569172857 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac541a75c0 con 0x7fac541045e0 2024-03-28T16:01:59.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:59.534+0000 7fac5a800700 1 -- 172.21.15.67:0/2569172857 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fac54062920 con 0x7fac541045e0 2024-03-28T16:01:59.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:59.536+0000 7fac4be00700 1 -- 172.21.15.67:0/2569172857 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fac440055e0 con 0x7fac541045e0 2024-03-28T16:01:59.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:59.537+0000 7fac4be00700 1 --2- 172.21.15.67:0/2569172857 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fac380a29a0 0x7fac380a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:59.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:59.537+0000 7fac4be00700 1 -- 172.21.15.67:0/2569172857 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fac440cf050 con 0x7fac541045e0 2024-03-28T16:01:59.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:59.538+0000 7fac52a00700 1 --2- 172.21.15.67:0/2569172857 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fac380a29a0 0x7fac380a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:59.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:59.538+0000 7fac4be00700 1 --2- 172.21.15.67:0/2569172857 >> [v2:172.21.15.67:6826/76885409,v1:172.21.15.67:6827/76885409] conn(0x7fac380ac4e0 0x7fac380ae980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:01:59.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:59.538+0000 7fac4be00700 1 -- 172.21.15.67:0/2569172857 --> [v2:172.21.15.67:6826/76885409,v1:172.21.15.67:6827/76885409] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fac380af050 con 0x7fac380ac4e0 2024-03-28T16:01:59.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:59.538+0000 7fac4be00700 1 -- 172.21.15.67:0/2569172857 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) v2 ==== 24+0+0 (secure 0 0 0) 0x7fac440cf4a0 con 0x7fac541045e0 2024-03-28T16:01:59.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:59.538+0000 7fac53e00700 1 --2- 172.21.15.67:0/2569172857 >> [v2:172.21.15.67:6826/76885409,v1:172.21.15.67:6827/76885409] conn(0x7fac380ac4e0 0x7fac380ae980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:01:59.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:59.539+0000 7fac52a00700 1 --2- 172.21.15.67:0/2569172857 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fac380a29a0 0x7fac380a4e60 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fac40000c00 tx=0x7fac40005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:59.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:59.539+0000 7fac53e00700 1 --2- 172.21.15.67:0/2569172857 >> [v2:172.21.15.67:6826/76885409,v1:172.21.15.67:6827/76885409] conn(0x7fac380ac4e0 0x7fac380ae980 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:01:59.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:59.540+0000 7fac4be00700 1 -- 172.21.15.67:0/2569172857 <== osd.9 v2:172.21.15.67:6826/76885409 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fac380af050 con 0x7fac380ac4e0 2024-03-28T16:01:59.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:59.559+0000 7fac5a800700 1 -- 172.21.15.67:0/2569172857 --> [v2:172.21.15.67:6826/76885409,v1:172.21.15.67:6827/76885409] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fac54062da0 con 0x7fac380ac4e0 2024-03-28T16:01:59.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:59.560+0000 7fac4be00700 1 -- 172.21.15.67:0/2569172857 <== osd.9 v2:172.21.15.67:6826/76885409 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fac54062da0 con 0x7fac380ac4e0 2024-03-28T16:01:59.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:59.561+0000 7fac5a800700 1 -- 172.21.15.67:0/2569172857 >> [v2:172.21.15.67:6826/76885409,v1:172.21.15.67:6827/76885409] conn(0x7fac380ac4e0 msgr2=0x7fac380ae980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:59.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:59.561+0000 7fac5a800700 1 --2- 172.21.15.67:0/2569172857 >> [v2:172.21.15.67:6826/76885409,v1:172.21.15.67:6827/76885409] conn(0x7fac380ac4e0 0x7fac380ae980 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:01:59.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:59.561+0000 7fac5a800700 1 -- 172.21.15.67:0/2569172857 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fac380a29a0 msgr2=0x7fac380a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:59.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:59.561+0000 7fac5a800700 1 --2- 172.21.15.67:0/2569172857 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fac380a29a0 0x7fac380a4e60 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fac40000c00 tx=0x7fac40005fb0 comp rx=0 tx=0).stop 2024-03-28T16:01:59.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:59.561+0000 7fac5a800700 1 -- 172.21.15.67:0/2569172857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac541045e0 msgr2=0x7fac541a06f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:01:59.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:59.561+0000 7fac5a800700 1 --2- 172.21.15.67:0/2569172857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac541045e0 0x7fac541a06f0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fac4400d980 tx=0x7fac4400dd40 comp rx=0 tx=0).stop 2024-03-28T16:01:59.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:59.561+0000 7fac53e00700 1 -- 172.21.15.67:0/2569172857 reap_dead start 2024-03-28T16:01:59.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:59.561+0000 7fac5a800700 1 -- 172.21.15.67:0/2569172857 shutdown_connections 2024-03-28T16:01:59.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:59.561+0000 7fac5a800700 1 -- 172.21.15.67:0/2569172857 >> 172.21.15.67:0/2569172857 conn(0x7fac540ffe70 msgr2=0x7fac54102120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:01:59.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:59.561+0000 7fac5a800700 1 -- 172.21.15.67:0/2569172857 shutdown_connections 2024-03-28T16:01:59.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:01:59.561+0000 7fac5a800700 1 -- 172.21.15.67:0/2569172857 wait complete. 2024-03-28T16:02:00.688 INFO:teuthology.orchestra.run.smithi067.stdout:304942678056 2024-03-28T16:02:00.688 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.9 2024-03-28T16:02:00.747 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T16:02:00.747 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T16:02:00.749 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T16:02:00.760 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T16:02:00.767 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T16:02:00.850 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T16:02:00.892 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T16:02:00.965 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:00 smithi067 ceph-mon[31707]: pgmap v392: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:02:01.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:00 smithi150 ceph-mon[34825]: pgmap v392: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:02:01.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:00 smithi183 ceph-mon[35580]: pgmap v392: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:02:01.566 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T16:02:01.566 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T16:02:01.567 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T16:02:01.570 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T16:02:02.797 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:02 smithi067 ceph-mon[31707]: pgmap v393: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:02:03.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:02 smithi150 ceph-mon[34825]: pgmap v393: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:02:03.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:02 smithi183 ceph-mon[35580]: pgmap v393: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:02:05.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:04 smithi150 ceph-mon[34825]: pgmap v394: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:02:05.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:04 smithi067 ceph-mon[31707]: pgmap v394: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:02:05.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:04 smithi183 ceph-mon[35580]: pgmap v394: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:02:06.962 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:06 smithi067 ceph-mon[31707]: pgmap v395: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:02:07.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:06 smithi150 ceph-mon[34825]: pgmap v395: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:02:07.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:06 smithi183 ceph-mon[35580]: pgmap v395: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:02:08.928 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:08 smithi067 ceph-mon[31707]: pgmap v396: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:02:09.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:08 smithi150 ceph-mon[34825]: pgmap v396: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:02:09.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:08 smithi183 ceph-mon[35580]: pgmap v396: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:02:09.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:09 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/954662283' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-28T16:02:10.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:09 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/954662283' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-28T16:02:10.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:09 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/954662283' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-28T16:02:10.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:08.689+0000 7f7a0ec00700 1 -- 172.21.15.67:0/3791608838 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a0810ea50 msgr2=0x7f7a08110e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:10.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:08.689+0000 7f7a0ec00700 1 --2- 172.21.15.67:0/3791608838 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a0810ea50 0x7f7a08110e60 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f7a0000b0a0 tx=0x7f7a0000b3b0 comp rx=0 tx=0).stop 2024-03-28T16:02:10.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:08.689+0000 7f7a0ec00700 1 -- 172.21.15.67:0/3791608838 shutdown_connections 2024-03-28T16:02:10.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:08.689+0000 7f7a0ec00700 1 --2- 172.21.15.67:0/3791608838 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a0810ea50 0x7f7a08110e60 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:10.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:08.689+0000 7f7a0ec00700 1 --2- 172.21.15.67:0/3791608838 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7a08074560 0x7f7a08074940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:10.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:08.689+0000 7f7a0ec00700 1 --2- 172.21.15.67:0/3791608838 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a08074e80 0x7f7a0810e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:10.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:08.689+0000 7f7a0ec00700 1 -- 172.21.15.67:0/3791608838 >> 172.21.15.67:0/3791608838 conn(0x7f7a080fdc60 msgr2=0x7f7a08100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:02:10.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:08.689+0000 7f7a0ec00700 1 -- 172.21.15.67:0/3791608838 shutdown_connections 2024-03-28T16:02:10.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:08.689+0000 7f7a0ec00700 1 -- 172.21.15.67:0/3791608838 wait complete. 2024-03-28T16:02:10.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:08.690+0000 7f7a0ec00700 1 Processor -- start 2024-03-28T16:02:10.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:08.690+0000 7f7a0ec00700 1 -- start start 2024-03-28T16:02:10.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:08.691+0000 7f7a0ec00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a08074560 0x7f7a081a0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:10.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:08.691+0000 7f7a0ec00700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a08074e80 0x7f7a081a0c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:10.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:08.691+0000 7f7a0ec00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7a0810ea50 0x7f7a0819a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:10.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:08.691+0000 7f7a0ec00700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a08114640 con 0x7f7a08074560 2024-03-28T16:02:10.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:08.691+0000 7f7a0ec00700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a081144c0 con 0x7f7a08074e80 2024-03-28T16:02:10.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:08.691+0000 7f7a0ec00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a081147c0 con 0x7f7a0810ea50 2024-03-28T16:02:10.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:08.691+0000 7f7a07e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a08074560 0x7f7a081a0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:10.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:08.691+0000 7f7a07e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a08074560 0x7f7a081a0750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34876/0 (socket says 172.21.15.67:34876) 2024-03-28T16:02:10.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:08.691+0000 7f7a07e00700 1 -- 172.21.15.67:0/954662283 learned_addr learned my addr 172.21.15.67:0/954662283 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:02:10.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:08.691+0000 7f7a07400700 1 --2- 172.21.15.67:0/954662283 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a08074e80 0x7f7a081a0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:10.860 INFO:teuthology.orchestra.run.smithi067.stdout:34359738469 2024-03-28T16:02:10.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:08.691+0000 7f7a07e00700 1 -- 172.21.15.67:0/954662283 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7a0810ea50 msgr2=0x7f7a0819a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:10.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:08.691+0000 7f7a07e00700 1 --2- 172.21.15.67:0/954662283 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7a0810ea50 0x7f7a0819a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:10.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:08.691+0000 7f7a07e00700 1 -- 172.21.15.67:0/954662283 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a08074e80 msgr2=0x7f7a081a0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:10.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:08.691+0000 7f7a07e00700 1 --2- 172.21.15.67:0/954662283 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a08074e80 0x7f7a081a0c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:10.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:08.691+0000 7f7a07e00700 1 -- 172.21.15.67:0/954662283 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a0819b0c0 con 0x7f7a08074560 2024-03-28T16:02:10.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:08.691+0000 7f7a07400700 1 --2- 172.21.15.67:0/954662283 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a08074e80 0x7f7a081a0c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:02:10.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:08.692+0000 7f7a07e00700 1 --2- 172.21.15.67:0/954662283 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a08074560 0x7f7a081a0750 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f79f800b530 tx=0x7f79f800b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:02:10.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:08.692+0000 7f7a04c00700 1 -- 172.21.15.67:0/954662283 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f8015020 con 0x7f7a08074560 2024-03-28T16:02:10.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:08.692+0000 7f7a04c00700 1 -- 172.21.15.67:0/954662283 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f79f8011ce0 con 0x7f7a08074560 2024-03-28T16:02:10.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:08.692+0000 7f7a0ec00700 1 -- 172.21.15.67:0/954662283 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a0819b3b0 con 0x7f7a08074560 2024-03-28T16:02:10.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:08.692+0000 7f7a04c00700 1 -- 172.21.15.67:0/954662283 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f8010d60 con 0x7f7a08074560 2024-03-28T16:02:10.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:08.692+0000 7f7a0ec00700 1 -- 172.21.15.67:0/954662283 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a081a7620 con 0x7f7a08074560 2024-03-28T16:02:10.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:08.693+0000 7f7a0ec00700 1 -- 172.21.15.67:0/954662283 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a08068e50 con 0x7f7a08074560 2024-03-28T16:02:10.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:08.695+0000 7f7a04c00700 1 -- 172.21.15.67:0/954662283 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f79f8018020 con 0x7f7a08074560 2024-03-28T16:02:10.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:08.697+0000 7f7a04c00700 1 --2- 172.21.15.67:0/954662283 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f79ec0a2920 0x7f79ec0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:10.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:08.697+0000 7f7a04c00700 1 -- 172.21.15.67:0/954662283 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f79f80ce400 con 0x7f7a08074560 2024-03-28T16:02:10.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:08.697+0000 7f7a07400700 1 --2- 172.21.15.67:0/954662283 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f79ec0a2920 0x7f79ec0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:10.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:08.698+0000 7f7a07400700 1 --2- 172.21.15.67:0/954662283 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f79ec0a2920 0x7f79ec0a4de0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f79f4005fd0 tx=0x7f79f4005dc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:02:10.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:08.701+0000 7f7a04c00700 1 -- 172.21.15.67:0/954662283 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f79f801f020 con 0x7f7a08074560 2024-03-28T16:02:10.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:08.927+0000 7f7a0ec00700 1 -- 172.21.15.67:0/954662283 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f7a0802da30 con 0x7f7a08074560 2024-03-28T16:02:10.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:08.928+0000 7f7a04c00700 1 -- 172.21.15.67:0/954662283 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f79f809b400 con 0x7f7a08074560 2024-03-28T16:02:10.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:08.931+0000 7f7a0ec00700 1 -- 172.21.15.67:0/954662283 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f79ec0a2920 msgr2=0x7f79ec0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:10.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:08.931+0000 7f7a0ec00700 1 --2- 172.21.15.67:0/954662283 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f79ec0a2920 0x7f79ec0a4de0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f79f4005fd0 tx=0x7f79f4005dc0 comp rx=0 tx=0).stop 2024-03-28T16:02:10.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:08.931+0000 7f7a0ec00700 1 -- 172.21.15.67:0/954662283 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a08074560 msgr2=0x7f7a081a0750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:10.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:08.931+0000 7f7a0ec00700 1 --2- 172.21.15.67:0/954662283 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a08074560 0x7f7a081a0750 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f79f800b530 tx=0x7f79f800b8f0 comp rx=0 tx=0).stop 2024-03-28T16:02:10.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:08.931+0000 7f7a0ec00700 1 -- 172.21.15.67:0/954662283 shutdown_connections 2024-03-28T16:02:10.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:08.931+0000 7f7a0ec00700 1 --2- 172.21.15.67:0/954662283 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f79ec0a2920 0x7f79ec0a4de0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:10.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:08.931+0000 7f7a0ec00700 1 --2- 172.21.15.67:0/954662283 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7a0810ea50 0x7f7a0819a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:10.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:08.931+0000 7f7a0ec00700 1 --2- 172.21.15.67:0/954662283 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a08074560 0x7f7a081a0750 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:10.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:08.931+0000 7f7a0ec00700 1 --2- 172.21.15.67:0/954662283 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a08074e80 0x7f7a081a0c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:10.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:08.931+0000 7f7a0ec00700 1 -- 172.21.15.67:0/954662283 >> 172.21.15.67:0/954662283 conn(0x7f7a080fdc60 msgr2=0x7f7a080fe970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:02:10.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:08.932+0000 7f7a0ec00700 1 -- 172.21.15.67:0/954662283 shutdown_connections 2024-03-28T16:02:10.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:08.932+0000 7f7a0ec00700 1 -- 172.21.15.67:0/954662283 wait complete. 2024-03-28T16:02:11.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:10 smithi150 ceph-mon[34825]: pgmap v397: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:02:11.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:10 smithi067 ceph-mon[31707]: pgmap v397: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:02:11.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:10 smithi183 ceph-mon[35580]: pgmap v397: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:02:11.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:11 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T16:02:11.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:11 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:02:11.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:11 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:02:11.828 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738466 got 34359738469 for osd.0 2024-03-28T16:02:11.828 DEBUG:teuthology.parallel:result is None 2024-03-28T16:02:12.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:11 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T16:02:12.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:11 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:02:12.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:11 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:02:12.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:11 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T16:02:12.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:11 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:02:12.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:11 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:02:13.147 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:12 smithi150 ceph-mon[34825]: pgmap v398: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:02:13.148 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:12 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:02:13.148 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:12 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:02:13.213 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:12 smithi067 ceph-mon[31707]: pgmap v398: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:02:13.213 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:12 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:02:13.213 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:12 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:02:13.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:12 smithi183 ceph-mon[35580]: pgmap v398: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:02:13.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:12 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:02:13.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:12 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:02:14.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.448+0000 7f5c5e000700 1 -- 172.21.15.67:0/112095127 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c58107200 msgr2=0x7f5c581075e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:14.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.448+0000 7f5c5e000700 1 --2- 172.21.15.67:0/112095127 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c58107200 0x7f5c581075e0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f5c48009990 tx=0x7f5c48009ca0 comp rx=0 tx=0).stop 2024-03-28T16:02:14.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.449+0000 7f5c5e000700 1 -- 172.21.15.67:0/112095127 shutdown_connections 2024-03-28T16:02:14.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.449+0000 7f5c5e000700 1 --2- 172.21.15.67:0/112095127 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5c58101140 0x7f5c5810f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:14.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.449+0000 7f5c5e000700 1 --2- 172.21.15.67:0/112095127 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c58107200 0x7f5c581075e0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:14.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.449+0000 7f5c5e000700 1 --2- 172.21.15.67:0/112095127 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c58100780 0x7f5c58100c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:14.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.449+0000 7f5c5e000700 1 -- 172.21.15.67:0/112095127 >> 172.21.15.67:0/112095127 conn(0x7f5c58077690 msgr2=0x7f5c580feab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:02:14.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.449+0000 7f5c5e000700 1 -- 172.21.15.67:0/112095127 shutdown_connections 2024-03-28T16:02:14.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.449+0000 7f5c5e000700 1 -- 172.21.15.67:0/112095127 wait complete. 2024-03-28T16:02:14.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.450+0000 7f5c5e000700 1 Processor -- start 2024-03-28T16:02:14.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.450+0000 7f5c5e000700 1 -- start start 2024-03-28T16:02:14.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.450+0000 7f5c5e000700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c58100780 0x7f5c581a0800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:14.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.450+0000 7f5c5e000700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5c58101140 0x7f5c581a0d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:14.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.450+0000 7f5c5e000700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c58107200 0x7f5c5819a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:14.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.450+0000 7f5c5e000700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c581124e0 con 0x7f5c58107200 2024-03-28T16:02:14.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.450+0000 7f5c5e000700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c58112360 con 0x7f5c58100780 2024-03-28T16:02:14.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.450+0000 7f5c5e000700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c58112660 con 0x7f5c58101140 2024-03-28T16:02:14.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.451+0000 7f5c57e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c58107200 0x7f5c5819a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:14.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.451+0000 7f5c57e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c58107200 0x7f5c5819a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56682/0 (socket says 172.21.15.67:56682) 2024-03-28T16:02:14.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.451+0000 7f5c57e00700 1 -- 172.21.15.67:0/2093482303 learned_addr learned my addr 172.21.15.67:0/2093482303 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:02:14.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.451+0000 7f5c57400700 1 --2- 172.21.15.67:0/2093482303 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c58100780 0x7f5c581a0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:14.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.451+0000 7f5c56a00700 1 --2- 172.21.15.67:0/2093482303 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5c58101140 0x7f5c581a0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:14.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.451+0000 7f5c57400700 1 -- 172.21.15.67:0/2093482303 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5c58101140 msgr2=0x7f5c581a0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:14.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.451+0000 7f5c57400700 1 --2- 172.21.15.67:0/2093482303 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5c58101140 0x7f5c581a0d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:14.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.451+0000 7f5c57400700 1 -- 172.21.15.67:0/2093482303 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c58107200 msgr2=0x7f5c5819a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:14.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.451+0000 7f5c57400700 1 --2- 172.21.15.67:0/2093482303 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c58107200 0x7f5c5819a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:14.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.451+0000 7f5c57400700 1 -- 172.21.15.67:0/2093482303 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c5819b120 con 0x7f5c58100780 2024-03-28T16:02:14.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.451+0000 7f5c57e00700 1 --2- 172.21.15.67:0/2093482303 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c58107200 0x7f5c5819a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T16:02:14.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.452+0000 7f5c57400700 1 --2- 172.21.15.67:0/2093482303 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c58100780 0x7f5c581a0800 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5c480098e0 tx=0x7f5c480057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:02:14.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.452+0000 7f5c4fe00700 1 -- 172.21.15.67:0/2093482303 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c48027070 con 0x7f5c58100780 2024-03-28T16:02:14.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.452+0000 7f5c5e000700 1 -- 172.21.15.67:0/2093482303 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c5819b3b0 con 0x7f5c58100780 2024-03-28T16:02:14.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.452+0000 7f5c5e000700 1 -- 172.21.15.67:0/2093482303 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c581a76d0 con 0x7f5c58100780 2024-03-28T16:02:14.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.452+0000 7f5c4fe00700 1 -- 172.21.15.67:0/2093482303 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5c48022470 con 0x7f5c58100780 2024-03-28T16:02:14.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.452+0000 7f5c4fe00700 1 -- 172.21.15.67:0/2093482303 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c4800b610 con 0x7f5c58100780 2024-03-28T16:02:14.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.453+0000 7f5c5e000700 1 -- 172.21.15.67:0/2093482303 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c58103aa0 con 0x7f5c58100780 2024-03-28T16:02:14.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.455+0000 7f5c4fe00700 1 -- 172.21.15.67:0/2093482303 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f5c4802a070 con 0x7f5c58100780 2024-03-28T16:02:14.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.457+0000 7f5c4fe00700 1 --2- 172.21.15.67:0/2093482303 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f5c3c0a29a0 0x7f5c3c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:14.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.457+0000 7f5c4fe00700 1 -- 172.21.15.67:0/2093482303 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f5c480cf7a0 con 0x7f5c58100780 2024-03-28T16:02:14.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.457+0000 7f5c56a00700 1 --2- 172.21.15.67:0/2093482303 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f5c3c0a29a0 0x7f5c3c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:14.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.458+0000 7f5c56a00700 1 --2- 172.21.15.67:0/2093482303 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f5c3c0a29a0 0x7f5c3c0a4e60 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f5c581038f0 tx=0x7f5c44009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:02:14.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.468+0000 7f5c4fe00700 1 -- 172.21.15.67:0/2093482303 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5c48013070 con 0x7f5c58100780 2024-03-28T16:02:14.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.715+0000 7f5c5e000700 1 -- 172.21.15.67:0/2093482303 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) v1 -- 0x7f5c5819bef0 con 0x7f5c58100780 2024-03-28T16:02:14.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.716+0000 7f5c4fe00700 1 -- 172.21.15.67:0/2093482303 <== mon.2 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7f5c4809c7b0 con 0x7f5c58100780 2024-03-28T16:02:14.717 INFO:teuthology.orchestra.run.smithi067.stdout:330712481828 2024-03-28T16:02:14.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.720+0000 7f5c5e000700 1 -- 172.21.15.67:0/2093482303 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f5c3c0a29a0 msgr2=0x7f5c3c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:14.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.720+0000 7f5c5e000700 1 --2- 172.21.15.67:0/2093482303 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f5c3c0a29a0 0x7f5c3c0a4e60 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f5c581038f0 tx=0x7f5c44009260 comp rx=0 tx=0).stop 2024-03-28T16:02:14.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.721+0000 7f5c5e000700 1 -- 172.21.15.67:0/2093482303 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c58100780 msgr2=0x7f5c581a0800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:14.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.721+0000 7f5c5e000700 1 --2- 172.21.15.67:0/2093482303 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c58100780 0x7f5c581a0800 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5c480098e0 tx=0x7f5c480057c0 comp rx=0 tx=0).stop 2024-03-28T16:02:14.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.721+0000 7f5c5e000700 1 -- 172.21.15.67:0/2093482303 shutdown_connections 2024-03-28T16:02:14.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.721+0000 7f5c5e000700 1 --2- 172.21.15.67:0/2093482303 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f5c3c0a29a0 0x7f5c3c0a4e60 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:14.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.721+0000 7f5c5e000700 1 --2- 172.21.15.67:0/2093482303 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c58107200 0x7f5c5819a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:14.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.721+0000 7f5c5e000700 1 --2- 172.21.15.67:0/2093482303 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c58100780 0x7f5c581a0800 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:14.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.721+0000 7f5c5e000700 1 --2- 172.21.15.67:0/2093482303 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5c58101140 0x7f5c581a0d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:14.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.721+0000 7f5c5e000700 1 -- 172.21.15.67:0/2093482303 >> 172.21.15.67:0/2093482303 conn(0x7f5c58077690 msgr2=0x7f5c58105530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:02:14.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.721+0000 7f5c5e000700 1 -- 172.21.15.67:0/2093482303 shutdown_connections 2024-03-28T16:02:14.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.721+0000 7f5c5e000700 1 -- 172.21.15.67:0/2093482303 wait complete. 2024-03-28T16:02:15.036 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:14 smithi067 ceph-mon[31707]: pgmap v399: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:02:15.036 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:14 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:02:15.036 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:14 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:02:15.036 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:14 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:02:15.036 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:14 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:02:15.036 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:14 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/2093482303' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-28T16:02:15.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:14 smithi150 ceph-mon[34825]: pgmap v399: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:02:15.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:14 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:02:15.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:14 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:02:15.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:14 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:02:15.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:14 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:02:15.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:14 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/2093482303' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-28T16:02:15.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:14 smithi183 ceph-mon[35580]: pgmap v399: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:02:15.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:14 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:02:15.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:14 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:02:15.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:14 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:02:15.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:14 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:02:15.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:14 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/2093482303' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-28T16:02:15.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:15.334+0000 7f933b800700 1 -- 172.21.15.67:0/3373979687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9334102410 msgr2=0x7f93341027f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:15.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:15.334+0000 7f933b800700 1 --2- 172.21.15.67:0/3373979687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9334102410 0x7f93341027f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f93280099c0 tx=0x7f9328009cd0 comp rx=0 tx=0).stop 2024-03-28T16:02:15.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:15.334+0000 7f933b800700 1 -- 172.21.15.67:0/3373979687 shutdown_connections 2024-03-28T16:02:15.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:15.334+0000 7f933b800700 1 --2- 172.21.15.67:0/3373979687 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f93341074f0 0x7f933410f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:15.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:15.334+0000 7f933b800700 1 --2- 172.21.15.67:0/3373979687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9334102410 0x7f93341027f0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:15.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:15.334+0000 7f933b800700 1 --2- 172.21.15.67:0/3373979687 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9334102dc0 0x7f9334106e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:15.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:15.334+0000 7f933b800700 1 -- 172.21.15.67:0/3373979687 >> 172.21.15.67:0/3373979687 conn(0x7f93340fdc60 msgr2=0x7f9334100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:02:15.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:15.334+0000 7f933b800700 1 -- 172.21.15.67:0/3373979687 shutdown_connections 2024-03-28T16:02:15.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:15.334+0000 7f933b800700 1 -- 172.21.15.67:0/3373979687 wait complete. 2024-03-28T16:02:15.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:15.335+0000 7f933b800700 1 Processor -- start 2024-03-28T16:02:15.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:15.335+0000 7f933b800700 1 -- start start 2024-03-28T16:02:15.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:15.336+0000 7f933b800700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9334102410 0x7f93341a0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:15.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:15.336+0000 7f933b800700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9334102dc0 0x7f93341a0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:15.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:15.336+0000 7f933b800700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f93341074f0 0x7f933419a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:15.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:15.336+0000 7f933b800700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9334112450 con 0x7f9334102dc0 2024-03-28T16:02:15.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:15.336+0000 7f933b800700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93341122d0 con 0x7f9334102410 2024-03-28T16:02:15.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:15.336+0000 7f933b800700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93341125d0 con 0x7f93341074f0 2024-03-28T16:02:15.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:15.336+0000 7f9333e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9334102dc0 0x7f93341a0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:15.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:15.336+0000 7f9333e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9334102dc0 0x7f93341a0cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56712/0 (socket says 172.21.15.67:56712) 2024-03-28T16:02:15.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:15.336+0000 7f9333e00700 1 -- 172.21.15.67:0/972189637 learned_addr learned my addr 172.21.15.67:0/972189637 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:02:15.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:15.336+0000 7f9338e00700 1 --2- 172.21.15.67:0/972189637 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9334102410 0x7f93341a0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:15.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:15.336+0000 7f9339800700 1 --2- 172.21.15.67:0/972189637 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f93341074f0 0x7f933419a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:15.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:15.336+0000 7f9333e00700 1 -- 172.21.15.67:0/972189637 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f93341074f0 msgr2=0x7f933419a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:15.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:15.336+0000 7f9333e00700 1 --2- 172.21.15.67:0/972189637 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f93341074f0 0x7f933419a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:15.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:15.336+0000 7f9333e00700 1 -- 172.21.15.67:0/972189637 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9334102410 msgr2=0x7f93341a0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:15.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:15.336+0000 7f9333e00700 1 --2- 172.21.15.67:0/972189637 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9334102410 0x7f93341a0780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:15.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:15.336+0000 7f9333e00700 1 -- 172.21.15.67:0/972189637 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f933419b0f0 con 0x7f9334102dc0 2024-03-28T16:02:15.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:15.336+0000 7f9338e00700 1 --2- 172.21.15.67:0/972189637 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9334102410 0x7f93341a0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T16:02:15.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:15.337+0000 7f9333e00700 1 --2- 172.21.15.67:0/972189637 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9334102dc0 0x7f93341a0cc0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f932000d980 tx=0x7f932000dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:02:15.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:15.337+0000 7f9331600700 1 -- 172.21.15.67:0/972189637 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9320017070 con 0x7f9334102dc0 2024-03-28T16:02:15.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:15.337+0000 7f9331600700 1 -- 172.21.15.67:0/972189637 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9320010460 con 0x7f9334102dc0 2024-03-28T16:02:15.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:15.337+0000 7f933b800700 1 -- 172.21.15.67:0/972189637 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f933419b3e0 con 0x7f9334102dc0 2024-03-28T16:02:15.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:15.337+0000 7f9331600700 1 -- 172.21.15.67:0/972189637 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f932000de60 con 0x7f9334102dc0 2024-03-28T16:02:15.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:15.337+0000 7f933b800700 1 -- 172.21.15.67:0/972189637 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93341a7650 con 0x7f9334102dc0 2024-03-28T16:02:15.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:15.338+0000 7f933b800700 1 -- 172.21.15.67:0/972189637 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9334068e50 con 0x7f9334102dc0 2024-03-28T16:02:15.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:15.342+0000 7f9331600700 1 -- 172.21.15.67:0/972189637 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f9320010610 con 0x7f9334102dc0 2024-03-28T16:02:15.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:15.343+0000 7f9331600700 1 --2- 172.21.15.67:0/972189637 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f931c0a2920 0x7f931c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:15.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:15.344+0000 7f9331600700 1 -- 172.21.15.67:0/972189637 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f93200ce7c0 con 0x7f9334102dc0 2024-03-28T16:02:15.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:15.344+0000 7f9338e00700 1 --2- 172.21.15.67:0/972189637 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f931c0a2920 0x7f931c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:15.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:15.344+0000 7f9338e00700 1 --2- 172.21.15.67:0/972189637 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f931c0a2920 0x7f931c0a4de0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f9328000c00 tx=0x7f9328005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:02:15.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:15.346+0000 7f9331600700 1 -- 172.21.15.67:0/972189637 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f932009b6b0 con 0x7f9334102dc0 2024-03-28T16:02:15.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:15.588+0000 7f933b800700 1 -- 172.21.15.67:0/972189637 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f93341a7880 con 0x7f9334102dc0 2024-03-28T16:02:15.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:15.589+0000 7f9331600700 1 -- 172.21.15.67:0/972189637 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f932009afb0 con 0x7f9334102dc0 2024-03-28T16:02:15.590 INFO:teuthology.orchestra.run.smithi067.stdout:115964117075 2024-03-28T16:02:15.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.825+0000 7f9409000700 1 -- 172.21.15.67:0/953870697 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94041002b0 msgr2=0x7f9404100730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:15.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.825+0000 7f9409000700 1 --2- 172.21.15.67:0/953870697 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94041002b0 0x7f9404100730 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f93f0009990 tx=0x7f93f0009ca0 comp rx=0 tx=0).stop 2024-03-28T16:02:15.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.825+0000 7f9409000700 1 -- 172.21.15.67:0/953870697 shutdown_connections 2024-03-28T16:02:15.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.825+0000 7f9409000700 1 --2- 172.21.15.67:0/953870697 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9404100c70 0x7f940410f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:15.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.825+0000 7f9409000700 1 --2- 172.21.15.67:0/953870697 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94041082d0 0x7f94041086b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:15.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.825+0000 7f9409000700 1 --2- 172.21.15.67:0/953870697 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94041002b0 0x7f9404100730 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:15.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.826+0000 7f9409000700 1 -- 172.21.15.67:0/953870697 >> 172.21.15.67:0/953870697 conn(0x7f9404077260 msgr2=0x7f94040fe5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:02:15.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.826+0000 7f9409000700 1 -- 172.21.15.67:0/953870697 shutdown_connections 2024-03-28T16:02:15.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.826+0000 7f9409000700 1 -- 172.21.15.67:0/953870697 wait complete. 2024-03-28T16:02:15.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.827+0000 7f9409000700 1 Processor -- start 2024-03-28T16:02:15.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.827+0000 7f9409000700 1 -- start start 2024-03-28T16:02:15.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.827+0000 7f9409000700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f94041002b0 0x7f9404101d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:15.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.827+0000 7f9409000700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9404100c70 0x7f9404102260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:15.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.827+0000 7f9409000700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94041082d0 0x7f94041027a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:15.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.827+0000 7f9409000700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9404112340 con 0x7f94041082d0 2024-03-28T16:02:15.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.827+0000 7f9409000700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94041121c0 con 0x7f9404100c70 2024-03-28T16:02:15.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.827+0000 7f9409000700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94041124c0 con 0x7f94041002b0 2024-03-28T16:02:15.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.828+0000 7f9402800700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94041082d0 0x7f94041027a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:15.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.828+0000 7f9402800700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94041082d0 0x7f94041027a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56692/0 (socket says 172.21.15.67:56692) 2024-03-28T16:02:15.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.828+0000 7f9402800700 1 -- 172.21.15.67:0/1108566983 learned_addr learned my addr 172.21.15.67:0/1108566983 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:02:15.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.828+0000 7f9401400700 1 --2- 172.21.15.67:0/1108566983 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9404100c70 0x7f9404102260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:15.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.828+0000 7f9401e00700 1 --2- 172.21.15.67:0/1108566983 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f94041002b0 0x7f9404101d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:15.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:15.592+0000 7f933b800700 1 -- 172.21.15.67:0/972189637 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f931c0a2920 msgr2=0x7f931c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:15.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:15.592+0000 7f933b800700 1 --2- 172.21.15.67:0/972189637 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f931c0a2920 0x7f931c0a4de0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f9328000c00 tx=0x7f9328005fd0 comp rx=0 tx=0).stop 2024-03-28T16:02:15.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:15.592+0000 7f933b800700 1 -- 172.21.15.67:0/972189637 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9334102dc0 msgr2=0x7f93341a0cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:15.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:15.592+0000 7f933b800700 1 --2- 172.21.15.67:0/972189637 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9334102dc0 0x7f93341a0cc0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f932000d980 tx=0x7f932000dd40 comp rx=0 tx=0).stop 2024-03-28T16:02:15.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.828+0000 7f9402800700 1 -- 172.21.15.67:0/1108566983 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f94041002b0 msgr2=0x7f9404101d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:15.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.828+0000 7f9402800700 1 --2- 172.21.15.67:0/1108566983 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f94041002b0 0x7f9404101d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:15.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.828+0000 7f9402800700 1 -- 172.21.15.67:0/1108566983 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9404100c70 msgr2=0x7f9404102260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:15.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.828+0000 7f9402800700 1 --2- 172.21.15.67:0/1108566983 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9404100c70 0x7f9404102260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:15.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.828+0000 7f9402800700 1 -- 172.21.15.67:0/1108566983 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94041a6d40 con 0x7f94041082d0 2024-03-28T16:02:15.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.828+0000 7f9402800700 1 --2- 172.21.15.67:0/1108566983 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94041082d0 0x7f94041027a0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f93fc007ee0 tx=0x7f93fc00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:02:15.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.829+0000 7f93faa00700 1 -- 172.21.15.67:0/1108566983 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93fc00dbd0 con 0x7f94041082d0 2024-03-28T16:02:15.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.829+0000 7f93faa00700 1 -- 172.21.15.67:0/1108566983 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f93fc00dd70 con 0x7f94041082d0 2024-03-28T16:02:15.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.829+0000 7f9409000700 1 -- 172.21.15.67:0/1108566983 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94041a7030 con 0x7f94041082d0 2024-03-28T16:02:15.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.829+0000 7f93faa00700 1 -- 172.21.15.67:0/1108566983 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93fc005990 con 0x7f94041082d0 2024-03-28T16:02:15.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.829+0000 7f9409000700 1 -- 172.21.15.67:0/1108566983 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94041a7590 con 0x7f94041082d0 2024-03-28T16:02:15.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.830+0000 7f9409000700 1 -- 172.21.15.67:0/1108566983 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9404068e50 con 0x7f94041082d0 2024-03-28T16:02:15.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.832+0000 7f93faa00700 1 -- 172.21.15.67:0/1108566983 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f93fc01a020 con 0x7f94041082d0 2024-03-28T16:02:15.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.833+0000 7f93faa00700 1 --2- 172.21.15.67:0/1108566983 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f93e80a29a0 0x7f93e80a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:15.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.834+0000 7f93faa00700 1 -- 172.21.15.67:0/1108566983 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f93fc0ce2c0 con 0x7f94041082d0 2024-03-28T16:02:15.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.834+0000 7f9401e00700 1 --2- 172.21.15.67:0/1108566983 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f93e80a29a0 0x7f93e80a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:15.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.834+0000 7f9401e00700 1 --2- 172.21.15.67:0/1108566983 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f93e80a29a0 0x7f93e80a4e60 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f93f400dad0 tx=0x7f93f400d4b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:02:15.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:14.838+0000 7f93faa00700 1 -- 172.21.15.67:0/1108566983 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f93fc010070 con 0x7f94041082d0 2024-03-28T16:02:15.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:15.073+0000 7f9409000700 1 -- 172.21.15.67:0/1108566983 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f940402dc90 con 0x7f94041082d0 2024-03-28T16:02:15.596 INFO:teuthology.orchestra.run.smithi067.stdout:236223201337 2024-03-28T16:02:15.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:15.074+0000 7f93faa00700 1 -- 172.21.15.67:0/1108566983 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f93fc09b1b0 con 0x7f94041082d0 2024-03-28T16:02:15.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:15.076+0000 7f9409000700 1 -- 172.21.15.67:0/1108566983 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f93e80a29a0 msgr2=0x7f93e80a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:15.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:15.076+0000 7f9409000700 1 --2- 172.21.15.67:0/1108566983 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f93e80a29a0 0x7f93e80a4e60 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f93f400dad0 tx=0x7f93f400d4b0 comp rx=0 tx=0).stop 2024-03-28T16:02:15.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:15.076+0000 7f9409000700 1 -- 172.21.15.67:0/1108566983 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94041082d0 msgr2=0x7f94041027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:15.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:15.076+0000 7f9409000700 1 --2- 172.21.15.67:0/1108566983 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94041082d0 0x7f94041027a0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f93fc007ee0 tx=0x7f93fc00d3b0 comp rx=0 tx=0).stop 2024-03-28T16:02:15.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:15.077+0000 7f9409000700 1 -- 172.21.15.67:0/1108566983 shutdown_connections 2024-03-28T16:02:15.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:15.077+0000 7f9409000700 1 --2- 172.21.15.67:0/1108566983 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f93e80a29a0 0x7f93e80a4e60 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:15.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:15.077+0000 7f9409000700 1 --2- 172.21.15.67:0/1108566983 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94041082d0 0x7f94041027a0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:15.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:15.077+0000 7f9409000700 1 --2- 172.21.15.67:0/1108566983 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f94041002b0 0x7f9404101d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:15.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:15.077+0000 7f9409000700 1 --2- 172.21.15.67:0/1108566983 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9404100c70 0x7f9404102260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:15.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:15.077+0000 7f9409000700 1 -- 172.21.15.67:0/1108566983 >> 172.21.15.67:0/1108566983 conn(0x7f9404077260 msgr2=0x7f94040fe980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:02:15.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:15.077+0000 7f9409000700 1 -- 172.21.15.67:0/1108566983 shutdown_connections 2024-03-28T16:02:15.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:15.077+0000 7f9409000700 1 -- 172.21.15.67:0/1108566983 wait complete. 2024-03-28T16:02:15.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:15.593+0000 7f933b800700 1 -- 172.21.15.67:0/972189637 shutdown_connections 2024-03-28T16:02:15.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:15.593+0000 7f933b800700 1 --2- 172.21.15.67:0/972189637 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f931c0a2920 0x7f931c0a4de0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:15.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:15.593+0000 7f933b800700 1 --2- 172.21.15.67:0/972189637 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f93341074f0 0x7f933419a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:15.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:15.593+0000 7f933b800700 1 --2- 172.21.15.67:0/972189637 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9334102410 0x7f93341a0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:15.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:15.593+0000 7f933b800700 1 --2- 172.21.15.67:0/972189637 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9334102dc0 0x7f93341a0cc0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:15.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:15.593+0000 7f933b800700 1 -- 172.21.15.67:0/972189637 >> 172.21.15.67:0/972189637 conn(0x7f93340fdc60 msgr2=0x7f93340fff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:02:15.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:15.593+0000 7f933b800700 1 -- 172.21.15.67:0/972189637 shutdown_connections 2024-03-28T16:02:15.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:15.593+0000 7f933b800700 1 -- 172.21.15.67:0/972189637 wait complete. 2024-03-28T16:02:15.843 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:15 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/1108566983' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-28T16:02:15.844 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:15 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/972189637' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-28T16:02:16.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:15 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/1108566983' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-28T16:02:16.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:15 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/972189637' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-28T16:02:16.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:15 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/1108566983' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-28T16:02:16.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:15 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/972189637' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-28T16:02:16.616 INFO:tasks.cephadm.ceph_manager.ceph:need seq 236223201334 got 236223201337 for osd.7 2024-03-28T16:02:16.616 DEBUG:teuthology.parallel:result is None 2024-03-28T16:02:16.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:16 smithi067 ceph-mon[31707]: pgmap v400: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:02:17.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:16 smithi150 ceph-mon[34825]: pgmap v400: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:02:17.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:16 smithi183 ceph-mon[35580]: pgmap v400: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:02:17.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.086+0000 7f05c7000700 1 -- 172.21.15.67:0/1252235144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05c0102df0 msgr2=0x7f05c0106e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:17.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.086+0000 7f05c7000700 1 --2- 172.21.15.67:0/1252235144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05c0102df0 0x7f05c0106e30 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f05ac0099c0 tx=0x7f05ac009cd0 comp rx=0 tx=0).stop 2024-03-28T16:02:17.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.086+0000 7f05c7000700 1 -- 172.21.15.67:0/1252235144 shutdown_connections 2024-03-28T16:02:17.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.086+0000 7f05c7000700 1 --2- 172.21.15.67:0/1252235144 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f05c0107510 0x7f05c010f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:17.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.086+0000 7f05c7000700 1 --2- 172.21.15.67:0/1252235144 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f05c0102440 0x7f05c0102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:17.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.086+0000 7f05c7000700 1 --2- 172.21.15.67:0/1252235144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05c0102df0 0x7f05c0106e30 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:17.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.086+0000 7f05c7000700 1 -- 172.21.15.67:0/1252235144 >> 172.21.15.67:0/1252235144 conn(0x7f05c00792f0 msgr2=0x7f05c0079700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:02:17.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.087+0000 7f05c7000700 1 -- 172.21.15.67:0/1252235144 shutdown_connections 2024-03-28T16:02:17.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.087+0000 7f05c7000700 1 -- 172.21.15.67:0/1252235144 wait complete. 2024-03-28T16:02:17.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.088+0000 7f05c7000700 1 Processor -- start 2024-03-28T16:02:17.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.088+0000 7f05c7000700 1 -- start start 2024-03-28T16:02:17.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.088+0000 7f05c7000700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f05c0102440 0x7f05c0074490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:17.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.088+0000 7f05c7000700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f05c0102df0 0x7f05c00749d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:17.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.088+0000 7f05c7000700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05c0107510 0x7f05c0074f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:17.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.088+0000 7f05c7000700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05c01123d0 con 0x7f05c0107510 2024-03-28T16:02:17.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.088+0000 7f05c7000700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05c0112250 con 0x7f05c0102df0 2024-03-28T16:02:17.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.088+0000 7f05c7000700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05c0112550 con 0x7f05c0102440 2024-03-28T16:02:17.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.089+0000 7f05c5000700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05c0107510 0x7f05c0074f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:17.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.089+0000 7f05c5000700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05c0107510 0x7f05c0074f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56724/0 (socket says 172.21.15.67:56724) 2024-03-28T16:02:17.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.089+0000 7f05c5000700 1 -- 172.21.15.67:0/4058766884 learned_addr learned my addr 172.21.15.67:0/4058766884 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:02:17.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.089+0000 7f05bfe00700 1 --2- 172.21.15.67:0/4058766884 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f05c0102440 0x7f05c0074490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:17.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.089+0000 7f05bf400700 1 --2- 172.21.15.67:0/4058766884 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f05c0102df0 0x7f05c00749d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:17.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.089+0000 7f05c5000700 1 -- 172.21.15.67:0/4058766884 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f05c0102440 msgr2=0x7f05c0074490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:17.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.089+0000 7f05c5000700 1 --2- 172.21.15.67:0/4058766884 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f05c0102440 0x7f05c0074490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:17.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.089+0000 7f05c5000700 1 -- 172.21.15.67:0/4058766884 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f05c0102df0 msgr2=0x7f05c00749d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:17.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.089+0000 7f05c5000700 1 --2- 172.21.15.67:0/4058766884 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f05c0102df0 0x7f05c00749d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:17.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.089+0000 7f05c5000700 1 -- 172.21.15.67:0/4058766884 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05c01a6e50 con 0x7f05c0107510 2024-03-28T16:02:17.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.090+0000 7f05c5000700 1 --2- 172.21.15.67:0/4058766884 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05c0107510 0x7f05c0074f10 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f05b800cca0 tx=0x7f05b800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:02:17.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.090+0000 7f05bcc00700 1 -- 172.21.15.67:0/4058766884 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b8016020 con 0x7f05c0107510 2024-03-28T16:02:17.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.090+0000 7f05bcc00700 1 -- 172.21.15.67:0/4058766884 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f05b8012420 con 0x7f05c0107510 2024-03-28T16:02:17.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.090+0000 7f05c7000700 1 -- 172.21.15.67:0/4058766884 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05c01a7140 con 0x7f05c0107510 2024-03-28T16:02:17.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.090+0000 7f05c7000700 1 -- 172.21.15.67:0/4058766884 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05c01a7600 con 0x7f05c0107510 2024-03-28T16:02:17.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.090+0000 7f05bcc00700 1 -- 172.21.15.67:0/4058766884 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b80115a0 con 0x7f05c0107510 2024-03-28T16:02:17.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.092+0000 7f05c7000700 1 -- 172.21.15.67:0/4058766884 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05c0068e50 con 0x7f05c0107510 2024-03-28T16:02:17.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.094+0000 7f05bcc00700 1 -- 172.21.15.67:0/4058766884 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f05b8019020 con 0x7f05c0107510 2024-03-28T16:02:17.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.095+0000 7f05bcc00700 1 --2- 172.21.15.67:0/4058766884 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f05a40a2920 0x7f05a40a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:17.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.095+0000 7f05bcc00700 1 -- 172.21.15.67:0/4058766884 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f05b80ceca0 con 0x7f05c0107510 2024-03-28T16:02:17.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.095+0000 7f05bfe00700 1 --2- 172.21.15.67:0/4058766884 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f05a40a2920 0x7f05a40a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:17.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.096+0000 7f05bfe00700 1 --2- 172.21.15.67:0/4058766884 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f05a40a2920 0x7f05a40a4de0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f05b0005950 tx=0x7f05b000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:02:17.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.098+0000 7f05bcc00700 1 -- 172.21.15.67:0/4058766884 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f05b8020020 con 0x7f05c0107510 2024-03-28T16:02:17.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.184+0000 7f0d13200700 1 -- 172.21.15.67:0/3512577797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d14102c80 msgr2=0x7f0d14106cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:17.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.184+0000 7f0d13200700 1 --2- 172.21.15.67:0/3512577797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d14102c80 0x7f0d14106cd0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f0d000099c0 tx=0x7f0d00009cd0 comp rx=0 tx=0).stop 2024-03-28T16:02:17.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.185+0000 7f0d13200700 1 -- 172.21.15.67:0/3512577797 shutdown_connections 2024-03-28T16:02:17.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.185+0000 7f0d13200700 1 --2- 172.21.15.67:0/3512577797 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0d141073b0 0x7f0d1410ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:17.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.185+0000 7f0d13200700 1 --2- 172.21.15.67:0/3512577797 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d141022d0 0x7f0d141026b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:17.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.185+0000 7f0d13200700 1 --2- 172.21.15.67:0/3512577797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d14102c80 0x7f0d14106cd0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:17.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.185+0000 7f0d13200700 1 -- 172.21.15.67:0/3512577797 >> 172.21.15.67:0/3512577797 conn(0x7f0d140fdb60 msgr2=0x7f0d140fff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:02:17.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.185+0000 7f0d13200700 1 -- 172.21.15.67:0/3512577797 shutdown_connections 2024-03-28T16:02:17.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.185+0000 7f0d13200700 1 -- 172.21.15.67:0/3512577797 wait complete. 2024-03-28T16:02:17.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.186+0000 7f0d13200700 1 Processor -- start 2024-03-28T16:02:17.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.187+0000 7f0d13200700 1 -- start start 2024-03-28T16:02:17.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.187+0000 7f0d13200700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d141022d0 0x7f0d1419e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:17.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.187+0000 7f0d13200700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0d14102c80 0x7f0d1419e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:17.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.187+0000 7f0d13200700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d141073b0 0x7f0d141985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:17.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.187+0000 7f0d13200700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d14112290 con 0x7f0d141022d0 2024-03-28T16:02:17.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.187+0000 7f0d13200700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d14112110 con 0x7f0d141073b0 2024-03-28T16:02:17.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.187+0000 7f0d13200700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d14112410 con 0x7f0d14102c80 2024-03-28T16:02:17.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.187+0000 7f0d11e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d141022d0 0x7f0d1419e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:17.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.187+0000 7f0d11e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d141022d0 0x7f0d1419e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56748/0 (socket says 172.21.15.67:56748) 2024-03-28T16:02:17.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.187+0000 7f0d11e00700 1 -- 172.21.15.67:0/3231343629 learned_addr learned my addr 172.21.15.67:0/3231343629 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:02:17.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.188+0000 7f0d11400700 1 --2- 172.21.15.67:0/3231343629 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0d14102c80 0x7f0d1419e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:17.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.188+0000 7f0d11e00700 1 -- 172.21.15.67:0/3231343629 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0d14102c80 msgr2=0x7f0d1419e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:17.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.188+0000 7f0d11e00700 1 --2- 172.21.15.67:0/3231343629 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0d14102c80 0x7f0d1419e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:17.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.188+0000 7f0d11e00700 1 -- 172.21.15.67:0/3231343629 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d141073b0 msgr2=0x7f0d141985f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:17.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.188+0000 7f0d12800700 1 --2- 172.21.15.67:0/3231343629 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d141073b0 0x7f0d141985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:17.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.188+0000 7f0d11e00700 1 --2- 172.21.15.67:0/3231343629 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d141073b0 0x7f0d141985f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:17.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.188+0000 7f0d11e00700 1 -- 172.21.15.67:0/3231343629 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d14198d50 con 0x7f0d141022d0 2024-03-28T16:02:17.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.188+0000 7f0d12800700 1 --2- 172.21.15.67:0/3231343629 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d141073b0 0x7f0d141985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:02:17.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.188+0000 7f0d11e00700 1 --2- 172.21.15.67:0/3231343629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d141022d0 0x7f0d1419e430 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0d0400d950 tx=0x7f0d0400dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:02:17.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.189+0000 7f0d0aa00700 1 -- 172.21.15.67:0/3231343629 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d04016070 con 0x7f0d141022d0 2024-03-28T16:02:17.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.189+0000 7f0d0aa00700 1 -- 172.21.15.67:0/3231343629 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0d04010460 con 0x7f0d141022d0 2024-03-28T16:02:17.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.189+0000 7f0d13200700 1 -- 172.21.15.67:0/3231343629 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d14199040 con 0x7f0d141022d0 2024-03-28T16:02:17.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.189+0000 7f0d0aa00700 1 -- 172.21.15.67:0/3231343629 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d0400f620 con 0x7f0d141022d0 2024-03-28T16:02:17.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.189+0000 7f0d13200700 1 -- 172.21.15.67:0/3231343629 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d141a5300 con 0x7f0d141022d0 2024-03-28T16:02:17.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.190+0000 7f0d13200700 1 -- 172.21.15.67:0/3231343629 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d14068e50 con 0x7f0d141022d0 2024-03-28T16:02:17.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.192+0000 7f0d0aa00700 1 -- 172.21.15.67:0/3231343629 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f0d04019070 con 0x7f0d141022d0 2024-03-28T16:02:17.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.195+0000 7f0d0aa00700 1 --2- 172.21.15.67:0/3231343629 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f0cf80a2920 0x7f0cf80a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:17.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.195+0000 7f0d0aa00700 1 -- 172.21.15.67:0/3231343629 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f0d040ce8a0 con 0x7f0d141022d0 2024-03-28T16:02:17.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.195+0000 7f0d11400700 1 --2- 172.21.15.67:0/3231343629 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f0cf80a2920 0x7f0cf80a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:17.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.196+0000 7f0d11400700 1 --2- 172.21.15.67:0/3231343629 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f0cf80a2920 0x7f0cf80a4de0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0d00000c00 tx=0x7f0d00005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:02:17.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.198+0000 7f0d0aa00700 1 -- 172.21.15.67:0/3231343629 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0d0409b790 con 0x7f0d141022d0 2024-03-28T16:02:17.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.337+0000 7f05c7000700 1 -- 172.21.15.67:0/4058766884 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f05c00759c0 con 0x7f05c0107510 2024-03-28T16:02:17.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.337+0000 7f05bcc00700 1 -- 172.21.15.67:0/4058766884 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f05b809bd00 con 0x7f05c0107510 2024-03-28T16:02:17.338 INFO:teuthology.orchestra.run.smithi067.stdout:77309411420 2024-03-28T16:02:17.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.340+0000 7f05c7000700 1 -- 172.21.15.67:0/4058766884 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f05a40a2920 msgr2=0x7f05a40a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:17.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.340+0000 7f05c7000700 1 --2- 172.21.15.67:0/4058766884 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f05a40a2920 0x7f05a40a4de0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f05b0005950 tx=0x7f05b000b410 comp rx=0 tx=0).stop 2024-03-28T16:02:17.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.340+0000 7f05c7000700 1 -- 172.21.15.67:0/4058766884 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05c0107510 msgr2=0x7f05c0074f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:17.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.340+0000 7f05c7000700 1 --2- 172.21.15.67:0/4058766884 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05c0107510 0x7f05c0074f10 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f05b800cca0 tx=0x7f05b800a060 comp rx=0 tx=0).stop 2024-03-28T16:02:17.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.340+0000 7f05c7000700 1 -- 172.21.15.67:0/4058766884 shutdown_connections 2024-03-28T16:02:17.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.340+0000 7f05c7000700 1 --2- 172.21.15.67:0/4058766884 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f05a40a2920 0x7f05a40a4de0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:17.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.340+0000 7f05c7000700 1 --2- 172.21.15.67:0/4058766884 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05c0107510 0x7f05c0074f10 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:17.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.340+0000 7f05c7000700 1 --2- 172.21.15.67:0/4058766884 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f05c0102440 0x7f05c0074490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:17.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.340+0000 7f05c7000700 1 --2- 172.21.15.67:0/4058766884 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f05c0102df0 0x7f05c00749d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:17.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.340+0000 7f05c7000700 1 -- 172.21.15.67:0/4058766884 >> 172.21.15.67:0/4058766884 conn(0x7f05c00792f0 msgr2=0x7f05c0100440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:02:17.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.341+0000 7f05c7000700 1 -- 172.21.15.67:0/4058766884 shutdown_connections 2024-03-28T16:02:17.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.341+0000 7f05c7000700 1 -- 172.21.15.67:0/4058766884 wait complete. 2024-03-28T16:02:17.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.443+0000 7f0d13200700 1 -- 172.21.15.67:0/3231343629 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f0d1402da30 con 0x7f0d141022d0 2024-03-28T16:02:17.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.443+0000 7f0d0aa00700 1 -- 172.21.15.67:0/3231343629 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0d0409b090 con 0x7f0d141022d0 2024-03-28T16:02:17.444 INFO:teuthology.orchestra.run.smithi067.stdout:55834574945 2024-03-28T16:02:17.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.446+0000 7f0d13200700 1 -- 172.21.15.67:0/3231343629 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f0cf80a2920 msgr2=0x7f0cf80a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:17.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.446+0000 7f0d13200700 1 --2- 172.21.15.67:0/3231343629 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f0cf80a2920 0x7f0cf80a4de0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0d00000c00 tx=0x7f0d00005fb0 comp rx=0 tx=0).stop 2024-03-28T16:02:17.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.446+0000 7f0d13200700 1 -- 172.21.15.67:0/3231343629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d141022d0 msgr2=0x7f0d1419e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:17.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.446+0000 7f0d13200700 1 --2- 172.21.15.67:0/3231343629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d141022d0 0x7f0d1419e430 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0d0400d950 tx=0x7f0d0400dd10 comp rx=0 tx=0).stop 2024-03-28T16:02:17.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.446+0000 7f0d13200700 1 -- 172.21.15.67:0/3231343629 shutdown_connections 2024-03-28T16:02:17.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.446+0000 7f0d13200700 1 --2- 172.21.15.67:0/3231343629 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f0cf80a2920 0x7f0cf80a4de0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:17.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.446+0000 7f0d13200700 1 --2- 172.21.15.67:0/3231343629 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d141073b0 0x7f0d141985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:17.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.446+0000 7f0d13200700 1 --2- 172.21.15.67:0/3231343629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d141022d0 0x7f0d1419e430 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:17.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.446+0000 7f0d13200700 1 --2- 172.21.15.67:0/3231343629 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0d14102c80 0x7f0d1419e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:17.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.446+0000 7f0d13200700 1 -- 172.21.15.67:0/3231343629 >> 172.21.15.67:0/3231343629 conn(0x7f0d140fdb60 msgr2=0x7f0d140ffe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:02:17.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.446+0000 7f0d13200700 1 -- 172.21.15.67:0/3231343629 shutdown_connections 2024-03-28T16:02:17.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:17.446+0000 7f0d13200700 1 -- 172.21.15.67:0/3231343629 wait complete. 2024-03-28T16:02:18.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:17 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/4058766884' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-28T16:02:18.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:17 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/3231343629' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-28T16:02:18.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:17 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/4058766884' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-28T16:02:18.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:17 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/3231343629' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-28T16:02:18.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:17 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/4058766884' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-28T16:02:18.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:17 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/3231343629' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-28T16:02:18.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.414+0000 7f1771e00700 1 -- 172.21.15.67:0/438409426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f176c1096c0 msgr2=0x7f176c111290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:18.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.414+0000 7f1771e00700 1 --2- 172.21.15.67:0/438409426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f176c1096c0 0x7f176c111290 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f176400b0d0 tx=0x7f176400b3e0 comp rx=0 tx=0).stop 2024-03-28T16:02:18.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.414+0000 7f1771e00700 1 -- 172.21.15.67:0/438409426 shutdown_connections 2024-03-28T16:02:18.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.414+0000 7f1771e00700 1 --2- 172.21.15.67:0/438409426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f176c1096c0 0x7f176c111290 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:18.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.414+0000 7f1771e00700 1 --2- 172.21.15.67:0/438409426 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f176c1045e0 0x7f176c1049c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:18.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.414+0000 7f1771e00700 1 --2- 172.21.15.67:0/438409426 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f176c104f90 0x7f176c108fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:18.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.414+0000 7f1771e00700 1 -- 172.21.15.67:0/438409426 >> 172.21.15.67:0/438409426 conn(0x7f176c0ffe90 msgr2=0x7f176c1022b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:02:18.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.414+0000 7f1771e00700 1 -- 172.21.15.67:0/438409426 shutdown_connections 2024-03-28T16:02:18.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.414+0000 7f1771e00700 1 -- 172.21.15.67:0/438409426 wait complete. 2024-03-28T16:02:18.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.416+0000 7f1771e00700 1 Processor -- start 2024-03-28T16:02:18.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.416+0000 7f1771e00700 1 -- start start 2024-03-28T16:02:18.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.417+0000 7f1771e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f176c1045e0 0x7f176c1a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:18.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.417+0000 7f1771e00700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f176c104f90 0x7f176c1a0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:18.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.417+0000 7f1771e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f176c1096c0 0x7f176c19a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:18.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.417+0000 7f1771e00700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f176c1145d0 con 0x7f176c1096c0 2024-03-28T16:02:18.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.417+0000 7f1771e00700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f176c114450 con 0x7f176c104f90 2024-03-28T16:02:18.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.417+0000 7f1771e00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f176c114750 con 0x7f176c1045e0 2024-03-28T16:02:18.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.417+0000 7f176b400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f176c1096c0 0x7f176c19a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:18.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.417+0000 7f176b400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f176c1096c0 0x7f176c19a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56772/0 (socket says 172.21.15.67:56772) 2024-03-28T16:02:18.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.417+0000 7f176b400700 1 -- 172.21.15.67:0/2163793930 learned_addr learned my addr 172.21.15.67:0/2163793930 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:02:18.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.417+0000 7f176aa00700 1 --2- 172.21.15.67:0/2163793930 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f176c1045e0 0x7f176c1a06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:18.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.417+0000 7f176a000700 1 --2- 172.21.15.67:0/2163793930 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f176c104f90 0x7f176c1a0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:18.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.417+0000 7f176b400700 1 -- 172.21.15.67:0/2163793930 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f176c1045e0 msgr2=0x7f176c1a06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:18.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.417+0000 7f176b400700 1 --2- 172.21.15.67:0/2163793930 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f176c1045e0 0x7f176c1a06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:18.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.417+0000 7f176b400700 1 -- 172.21.15.67:0/2163793930 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f176c104f90 msgr2=0x7f176c1a0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:18.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.417+0000 7f176b400700 1 --2- 172.21.15.67:0/2163793930 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f176c104f90 0x7f176c1a0bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:18.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.417+0000 7f176b400700 1 -- 172.21.15.67:0/2163793930 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f176c19afd0 con 0x7f176c1096c0 2024-03-28T16:02:18.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.418+0000 7f176b400700 1 --2- 172.21.15.67:0/2163793930 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f176c1096c0 0x7f176c19a870 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f176400d010 tx=0x7f1764007a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:02:18.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.418+0000 7f1770a00700 1 -- 172.21.15.67:0/2163793930 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f176401f070 con 0x7f176c1096c0 2024-03-28T16:02:18.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.418+0000 7f1770a00700 1 -- 172.21.15.67:0/2163793930 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1764003c00 con 0x7f176c1096c0 2024-03-28T16:02:18.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.418+0000 7f1771e00700 1 -- 172.21.15.67:0/2163793930 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f176c19b260 con 0x7f176c1096c0 2024-03-28T16:02:18.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.418+0000 7f1770a00700 1 -- 172.21.15.67:0/2163793930 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f176401b6a0 con 0x7f176c1096c0 2024-03-28T16:02:18.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.418+0000 7f1771e00700 1 -- 172.21.15.67:0/2163793930 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f176c1a7580 con 0x7f176c1096c0 2024-03-28T16:02:18.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.420+0000 7f1771e00700 1 -- 172.21.15.67:0/2163793930 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f176c105ac0 con 0x7f176c1096c0 2024-03-28T16:02:18.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.423+0000 7f1770a00700 1 -- 172.21.15.67:0/2163793930 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f1764022070 con 0x7f176c1096c0 2024-03-28T16:02:18.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.424+0000 7f1770a00700 1 --2- 172.21.15.67:0/2163793930 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f17500a2920 0x7f17500a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:18.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.425+0000 7f1770a00700 1 -- 172.21.15.67:0/2163793930 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f176400f020 con 0x7f176c1096c0 2024-03-28T16:02:18.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.425+0000 7f176aa00700 1 --2- 172.21.15.67:0/2163793930 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f17500a2920 0x7f17500a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:18.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.426+0000 7f176aa00700 1 --2- 172.21.15.67:0/2163793930 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f17500a2920 0x7f17500a4de0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f175c005950 tx=0x7f175c0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:02:18.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.427+0000 7f1770a00700 1 -- 172.21.15.67:0/2163793930 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1764029020 con 0x7f176c1096c0 2024-03-28T16:02:18.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.534+0000 7f2982600700 1 -- 172.21.15.67:0/2201146056 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f297c101090 msgr2=0x7f297c1014f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:18.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.534+0000 7f2982600700 1 --2- 172.21.15.67:0/2201146056 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f297c101090 0x7f297c1014f0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f2968009990 tx=0x7f2968009ca0 comp rx=0 tx=0).stop 2024-03-28T16:02:18.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.534+0000 7f2982600700 1 -- 172.21.15.67:0/2201146056 shutdown_connections 2024-03-28T16:02:18.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.534+0000 7f2982600700 1 --2- 172.21.15.67:0/2201146056 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f297c101a30 0x7f297c10df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:18.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.534+0000 7f2982600700 1 --2- 172.21.15.67:0/2201146056 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f297c1070d0 0x7f297c1074b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:18.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.534+0000 7f2982600700 1 --2- 172.21.15.67:0/2201146056 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f297c101090 0x7f297c1014f0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:18.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.534+0000 7f2982600700 1 -- 172.21.15.67:0/2201146056 >> 172.21.15.67:0/2201146056 conn(0x7f297c077fc0 msgr2=0x7f297c0783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:02:18.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.534+0000 7f2982600700 1 -- 172.21.15.67:0/2201146056 shutdown_connections 2024-03-28T16:02:18.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.534+0000 7f2982600700 1 -- 172.21.15.67:0/2201146056 wait complete. 2024-03-28T16:02:18.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.535+0000 7f2982600700 1 Processor -- start 2024-03-28T16:02:18.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.535+0000 7f2982600700 1 -- start start 2024-03-28T16:02:18.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.536+0000 7f2982600700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f297c101090 0x7f297c1a06f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:18.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.536+0000 7f2982600700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f297c101a30 0x7f297c1a0c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:18.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.536+0000 7f2982600700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f297c1070d0 0x7f297c19a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:18.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.536+0000 7f2982600700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f297c1111e0 con 0x7f297c101090 2024-03-28T16:02:18.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.536+0000 7f2982600700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f297c111060 con 0x7f297c101a30 2024-03-28T16:02:18.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.536+0000 7f2982600700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f297c111360 con 0x7f297c1070d0 2024-03-28T16:02:18.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.536+0000 7f297b400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f297c101090 0x7f297c1a06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:18.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.536+0000 7f297b400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f297c101090 0x7f297c1a06f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56790/0 (socket says 172.21.15.67:56790) 2024-03-28T16:02:18.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.536+0000 7f297b400700 1 -- 172.21.15.67:0/1235642094 learned_addr learned my addr 172.21.15.67:0/1235642094 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:02:18.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.536+0000 7f297aa00700 1 --2- 172.21.15.67:0/1235642094 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f297c101a30 0x7f297c1a0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:18.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.536+0000 7f297be00700 1 --2- 172.21.15.67:0/1235642094 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f297c1070d0 0x7f297c19a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:18.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.536+0000 7f297b400700 1 -- 172.21.15.67:0/1235642094 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f297c1070d0 msgr2=0x7f297c19a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:18.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.536+0000 7f297b400700 1 --2- 172.21.15.67:0/1235642094 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f297c1070d0 0x7f297c19a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:18.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.536+0000 7f297b400700 1 -- 172.21.15.67:0/1235642094 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f297c101a30 msgr2=0x7f297c1a0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:18.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.536+0000 7f297b400700 1 --2- 172.21.15.67:0/1235642094 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f297c101a30 0x7f297c1a0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:18.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.536+0000 7f297b400700 1 -- 172.21.15.67:0/1235642094 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f297c19b010 con 0x7f297c101090 2024-03-28T16:02:18.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.536+0000 7f297aa00700 1 --2- 172.21.15.67:0/1235642094 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f297c101a30 0x7f297c1a0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:02:18.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.537+0000 7f297b400700 1 --2- 172.21.15.67:0/1235642094 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f297c101090 0x7f297c1a06f0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f296c00e9a0 tx=0x7f296c00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:02:18.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.537+0000 7f2973e00700 1 -- 172.21.15.67:0/1235642094 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f296c00cbd0 con 0x7f297c101090 2024-03-28T16:02:18.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.537+0000 7f2973e00700 1 -- 172.21.15.67:0/1235642094 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f296c00cd70 con 0x7f297c101090 2024-03-28T16:02:18.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.537+0000 7f2982600700 1 -- 172.21.15.67:0/1235642094 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f297c19b300 con 0x7f297c101090 2024-03-28T16:02:18.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.537+0000 7f2973e00700 1 -- 172.21.15.67:0/1235642094 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f296c018750 con 0x7f297c101090 2024-03-28T16:02:18.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.537+0000 7f2982600700 1 -- 172.21.15.67:0/1235642094 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f297c1a75c0 con 0x7f297c101090 2024-03-28T16:02:18.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.539+0000 7f2982600700 1 -- 172.21.15.67:0/1235642094 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f297c068e50 con 0x7f297c101090 2024-03-28T16:02:18.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.540+0000 7f2973e00700 1 -- 172.21.15.67:0/1235642094 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f296c010430 con 0x7f297c101090 2024-03-28T16:02:18.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.542+0000 7f2973e00700 1 --2- 172.21.15.67:0/1235642094 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f29600a2920 0x7f29600a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:18.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.542+0000 7f2973e00700 1 -- 172.21.15.67:0/1235642094 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f296c0ceeb0 con 0x7f297c101090 2024-03-28T16:02:18.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.542+0000 7f297aa00700 1 --2- 172.21.15.67:0/1235642094 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f29600a2920 0x7f29600a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:18.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.543+0000 7f297aa00700 1 --2- 172.21.15.67:0/1235642094 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f29600a2920 0x7f29600a4de0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f2968000c00 tx=0x7f2968005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:02:18.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.546+0000 7f2973e00700 1 -- 172.21.15.67:0/1235642094 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f296c09bda0 con 0x7f297c101090 2024-03-28T16:02:18.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.662+0000 7f1771e00700 1 -- 172.21.15.67:0/2163793930 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f176c02da30 con 0x7f176c1096c0 2024-03-28T16:02:18.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.663+0000 7f1770a00700 1 -- 172.21.15.67:0/2163793930 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f176409d200 con 0x7f176c1096c0 2024-03-28T16:02:18.663 INFO:teuthology.orchestra.run.smithi067.stdout:176093659207 2024-03-28T16:02:18.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.666+0000 7f1771e00700 1 -- 172.21.15.67:0/2163793930 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f17500a2920 msgr2=0x7f17500a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:18.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.666+0000 7f1771e00700 1 --2- 172.21.15.67:0/2163793930 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f17500a2920 0x7f17500a4de0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f175c005950 tx=0x7f175c0058e0 comp rx=0 tx=0).stop 2024-03-28T16:02:18.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.666+0000 7f1771e00700 1 -- 172.21.15.67:0/2163793930 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f176c1096c0 msgr2=0x7f176c19a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:18.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.666+0000 7f1771e00700 1 --2- 172.21.15.67:0/2163793930 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f176c1096c0 0x7f176c19a870 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f176400d010 tx=0x7f1764007a50 comp rx=0 tx=0).stop 2024-03-28T16:02:18.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.666+0000 7f1771e00700 1 -- 172.21.15.67:0/2163793930 shutdown_connections 2024-03-28T16:02:18.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.666+0000 7f1771e00700 1 --2- 172.21.15.67:0/2163793930 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f17500a2920 0x7f17500a4de0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:18.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.666+0000 7f1771e00700 1 --2- 172.21.15.67:0/2163793930 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f176c1096c0 0x7f176c19a870 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:18.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.666+0000 7f1771e00700 1 --2- 172.21.15.67:0/2163793930 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f176c1045e0 0x7f176c1a06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:18.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.666+0000 7f1771e00700 1 --2- 172.21.15.67:0/2163793930 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f176c104f90 0x7f176c1a0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:18.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.666+0000 7f1771e00700 1 -- 172.21.15.67:0/2163793930 >> 172.21.15.67:0/2163793930 conn(0x7f176c0ffe90 msgr2=0x7f176c100a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:02:18.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.666+0000 7f1771e00700 1 -- 172.21.15.67:0/2163793930 shutdown_connections 2024-03-28T16:02:18.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.666+0000 7f1771e00700 1 -- 172.21.15.67:0/2163793930 wait complete. 2024-03-28T16:02:18.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.784+0000 7f2982600700 1 -- 172.21.15.67:0/1235642094 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f297c1a7820 con 0x7f297c101090 2024-03-28T16:02:18.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.785+0000 7f2973e00700 1 -- 172.21.15.67:0/1235642094 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f296c09b6a0 con 0x7f297c101090 2024-03-28T16:02:18.785 INFO:teuthology.orchestra.run.smithi067.stdout:210453397568 2024-03-28T16:02:18.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.787+0000 7f2982600700 1 -- 172.21.15.67:0/1235642094 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f29600a2920 msgr2=0x7f29600a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:18.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.787+0000 7f2982600700 1 --2- 172.21.15.67:0/1235642094 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f29600a2920 0x7f29600a4de0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f2968000c00 tx=0x7f2968005fb0 comp rx=0 tx=0).stop 2024-03-28T16:02:18.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.787+0000 7f2982600700 1 -- 172.21.15.67:0/1235642094 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f297c101090 msgr2=0x7f297c1a06f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:18.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.787+0000 7f2982600700 1 --2- 172.21.15.67:0/1235642094 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f297c101090 0x7f297c1a06f0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f296c00e9a0 tx=0x7f296c00ed60 comp rx=0 tx=0).stop 2024-03-28T16:02:18.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.787+0000 7f2982600700 1 -- 172.21.15.67:0/1235642094 shutdown_connections 2024-03-28T16:02:18.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.787+0000 7f2982600700 1 --2- 172.21.15.67:0/1235642094 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f29600a2920 0x7f29600a4de0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:18.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.787+0000 7f2982600700 1 --2- 172.21.15.67:0/1235642094 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f297c1070d0 0x7f297c19a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:18.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.787+0000 7f2982600700 1 --2- 172.21.15.67:0/1235642094 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f297c101090 0x7f297c1a06f0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:18.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.787+0000 7f2982600700 1 --2- 172.21.15.67:0/1235642094 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f297c101a30 0x7f297c1a0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:18.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.787+0000 7f2982600700 1 -- 172.21.15.67:0/1235642094 >> 172.21.15.67:0/1235642094 conn(0x7f297c077fc0 msgr2=0x7f297c069f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:02:18.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.788+0000 7f2982600700 1 -- 172.21.15.67:0/1235642094 shutdown_connections 2024-03-28T16:02:18.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:18.788+0000 7f2982600700 1 -- 172.21.15.67:0/1235642094 wait complete. 2024-03-28T16:02:18.948 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:18 smithi067 ceph-mon[31707]: pgmap v401: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:02:18.948 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:18 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/2163793930' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-28T16:02:19.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:18 smithi150 ceph-mon[34825]: pgmap v401: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:02:19.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:18 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/2163793930' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-28T16:02:19.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:18 smithi183 ceph-mon[35580]: pgmap v401: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:02:19.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:18 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/2163793930' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-28T16:02:19.974 INFO:tasks.cephadm.ceph_manager.ceph:need seq 210453397564 got 210453397568 for osd.6 2024-03-28T16:02:19.974 DEBUG:teuthology.parallel:result is None 2024-03-28T16:02:20.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:19 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/1235642094' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-28T16:02:20.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:19 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/1235642094' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-28T16:02:20.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:19 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/1235642094' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-28T16:02:20.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.174+0000 7f1e7e200700 1 -- 172.21.15.67:0/1422305420 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e78101020 msgr2=0x7f1e78101400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:20.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.174+0000 7f1e7e200700 1 --2- 172.21.15.67:0/1422305420 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e78101020 0x7f1e78101400 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f1e68009990 tx=0x7f1e68009ca0 comp rx=0 tx=0).stop 2024-03-28T16:02:20.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.174+0000 7f1e7e200700 1 -- 172.21.15.67:0/1422305420 shutdown_connections 2024-03-28T16:02:20.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.174+0000 7f1e7e200700 1 --2- 172.21.15.67:0/1422305420 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1e78106430 0x7f1e781068e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:20.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.174+0000 7f1e7e200700 1 --2- 172.21.15.67:0/1422305420 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e78101020 0x7f1e78101400 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:20.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.174+0000 7f1e7e200700 1 --2- 172.21.15.67:0/1422305420 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e78101940 0x7f1e78105e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:20.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.174+0000 7f1e7e200700 1 -- 172.21.15.67:0/1422305420 >> 172.21.15.67:0/1422305420 conn(0x7f1e780fca60 msgr2=0x7f1e780fee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:02:20.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.174+0000 7f1e7e200700 1 -- 172.21.15.67:0/1422305420 shutdown_connections 2024-03-28T16:02:20.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.174+0000 7f1e7e200700 1 -- 172.21.15.67:0/1422305420 wait complete. 2024-03-28T16:02:20.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.176+0000 7f1e7e200700 1 Processor -- start 2024-03-28T16:02:20.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.176+0000 7f1e7e200700 1 -- start start 2024-03-28T16:02:20.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.176+0000 7f1e7e200700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e78101020 0x7f1e7819edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:20.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.176+0000 7f1e7e200700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1e78101940 0x7f1e7819f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:20.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.176+0000 7f1e7e200700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e78106430 0x7f1e781a3630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:20.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.176+0000 7f1e7e200700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e78118880 con 0x7f1e78106430 2024-03-28T16:02:20.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.176+0000 7f1e7e200700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e78118700 con 0x7f1e78101020 2024-03-28T16:02:20.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.176+0000 7f1e7e200700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e78118a00 con 0x7f1e78101940 2024-03-28T16:02:20.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.177+0000 7f1e7d800700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e78106430 0x7f1e781a3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:20.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.177+0000 7f1e7d800700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e78106430 0x7f1e781a3630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58406/0 (socket says 172.21.15.67:58406) 2024-03-28T16:02:20.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.177+0000 7f1e7d800700 1 -- 172.21.15.67:0/1299678844 learned_addr learned my addr 172.21.15.67:0/1299678844 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:02:20.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.177+0000 7f1e6fe00700 1 --2- 172.21.15.67:0/1299678844 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1e78101940 0x7f1e7819f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:20.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.177+0000 7f1e7ce00700 1 --2- 172.21.15.67:0/1299678844 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e78101020 0x7f1e7819edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:20.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.177+0000 7f1e7d800700 1 -- 172.21.15.67:0/1299678844 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1e78101940 msgr2=0x7f1e7819f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:20.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.177+0000 7f1e7d800700 1 --2- 172.21.15.67:0/1299678844 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1e78101940 0x7f1e7819f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:20.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.177+0000 7f1e7d800700 1 -- 172.21.15.67:0/1299678844 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e78101020 msgr2=0x7f1e7819edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:20.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.177+0000 7f1e7d800700 1 --2- 172.21.15.67:0/1299678844 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e78101020 0x7f1e7819edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:20.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.177+0000 7f1e7d800700 1 -- 172.21.15.67:0/1299678844 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e781a3d70 con 0x7f1e78106430 2024-03-28T16:02:20.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.177+0000 7f1e7d800700 1 --2- 172.21.15.67:0/1299678844 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e78106430 0x7f1e781a3630 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f1e70007ee0 tx=0x7f1e7000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:02:20.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.178+0000 7f1e76000700 1 -- 172.21.15.67:0/1299678844 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e7000dbd0 con 0x7f1e78106430 2024-03-28T16:02:20.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.178+0000 7f1e76000700 1 -- 172.21.15.67:0/1299678844 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1e7000dd70 con 0x7f1e78106430 2024-03-28T16:02:20.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.178+0000 7f1e7e200700 1 -- 172.21.15.67:0/1299678844 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e781a4060 con 0x7f1e78106430 2024-03-28T16:02:20.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.178+0000 7f1e76000700 1 -- 172.21.15.67:0/1299678844 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e70005a20 con 0x7f1e78106430 2024-03-28T16:02:20.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.178+0000 7f1e7e200700 1 -- 172.21.15.67:0/1299678844 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e781ab8e0 con 0x7f1e78106430 2024-03-28T16:02:20.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.179+0000 7f1e7e200700 1 -- 172.21.15.67:0/1299678844 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e78068e50 con 0x7f1e78106430 2024-03-28T16:02:20.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.183+0000 7f1e76000700 1 -- 172.21.15.67:0/1299678844 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f1e7001a020 con 0x7f1e78106430 2024-03-28T16:02:20.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.184+0000 7f1e76000700 1 --2- 172.21.15.67:0/1299678844 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f1e5c0a29a0 0x7f1e5c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:20.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.184+0000 7f1e76000700 1 -- 172.21.15.67:0/1299678844 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f1e700ce9b0 con 0x7f1e78106430 2024-03-28T16:02:20.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.184+0000 7f1e7ce00700 1 --2- 172.21.15.67:0/1299678844 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f1e5c0a29a0 0x7f1e5c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:20.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.185+0000 7f1e7ce00700 1 --2- 172.21.15.67:0/1299678844 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f1e5c0a29a0 0x7f1e5c0a4e60 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1e68006010 tx=0x7f1e6801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:02:20.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.187+0000 7f1e76000700 1 -- 172.21.15.67:0/1299678844 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1e70010070 con 0x7f1e78106430 2024-03-28T16:02:20.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.438+0000 7f1e7e200700 1 -- 172.21.15.67:0/1299678844 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) v1 -- 0x7f1e781abb10 con 0x7f1e78106430 2024-03-28T16:02:20.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.438+0000 7f1e76000700 1 -- 172.21.15.67:0/1299678844 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7f1e7009b8a0 con 0x7f1e78106430 2024-03-28T16:02:20.439 INFO:teuthology.orchestra.run.smithi067.stdout:365072220190 2024-03-28T16:02:20.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.441+0000 7f1e7e200700 1 -- 172.21.15.67:0/1299678844 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f1e5c0a29a0 msgr2=0x7f1e5c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:20.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.441+0000 7f1e7e200700 1 --2- 172.21.15.67:0/1299678844 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f1e5c0a29a0 0x7f1e5c0a4e60 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1e68006010 tx=0x7f1e6801a040 comp rx=0 tx=0).stop 2024-03-28T16:02:20.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.442+0000 7f1e7e200700 1 -- 172.21.15.67:0/1299678844 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e78106430 msgr2=0x7f1e781a3630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:20.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.442+0000 7f1e7e200700 1 --2- 172.21.15.67:0/1299678844 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e78106430 0x7f1e781a3630 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f1e70007ee0 tx=0x7f1e7000d3b0 comp rx=0 tx=0).stop 2024-03-28T16:02:20.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.442+0000 7f1e7e200700 1 -- 172.21.15.67:0/1299678844 shutdown_connections 2024-03-28T16:02:20.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.442+0000 7f1e7e200700 1 --2- 172.21.15.67:0/1299678844 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f1e5c0a29a0 0x7f1e5c0a4e60 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:20.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.442+0000 7f1e7e200700 1 --2- 172.21.15.67:0/1299678844 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e78106430 0x7f1e781a3630 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:20.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.442+0000 7f1e7e200700 1 --2- 172.21.15.67:0/1299678844 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e78101020 0x7f1e7819edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:20.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.442+0000 7f1e7e200700 1 --2- 172.21.15.67:0/1299678844 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1e78101940 0x7f1e7819f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:20.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.442+0000 7f1e7e200700 1 -- 172.21.15.67:0/1299678844 >> 172.21.15.67:0/1299678844 conn(0x7f1e780fca60 msgr2=0x7f1e7810fbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:02:20.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.442+0000 7f1e7e200700 1 -- 172.21.15.67:0/1299678844 shutdown_connections 2024-03-28T16:02:20.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.442+0000 7f1e7e200700 1 -- 172.21.15.67:0/1299678844 wait complete. 2024-03-28T16:02:21.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:20 smithi150 ceph-mon[34825]: pgmap v402: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T16:02:21.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:20 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/1299678844' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-28T16:02:21.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:20 smithi067 ceph-mon[31707]: pgmap v402: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T16:02:21.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:20 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/1299678844' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-28T16:02:21.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:20 smithi183 ceph-mon[35580]: pgmap v402: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T16:02:21.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:20 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/1299678844' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-28T16:02:21.356 INFO:tasks.cephadm.ceph_manager.ceph:need seq 176093659202 got 176093659207 for osd.5 2024-03-28T16:02:21.357 DEBUG:teuthology.parallel:result is None 2024-03-28T16:02:21.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.600+0000 7fe20fc00700 1 -- 172.21.15.67:0/1528925129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe20806a7e0 msgr2=0x7fe20810f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:21.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.600+0000 7fe20fc00700 1 --2- 172.21.15.67:0/1528925129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe20806a7e0 0x7fe20810f020 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fe20000b0d0 tx=0x7fe20000b3e0 comp rx=0 tx=0).stop 2024-03-28T16:02:21.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.600+0000 7fe20fc00700 1 -- 172.21.15.67:0/1528925129 shutdown_connections 2024-03-28T16:02:21.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.600+0000 7fe20fc00700 1 --2- 172.21.15.67:0/1528925129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe20806a7e0 0x7fe20810f020 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:21.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.600+0000 7fe20fc00700 1 --2- 172.21.15.67:0/1528925129 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe2081081e0 0x7fe2081085c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:21.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.600+0000 7fe20fc00700 1 --2- 172.21.15.67:0/1528925129 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe208069e40 0x7fe20806a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:21.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.600+0000 7fe20fc00700 1 -- 172.21.15.67:0/1528925129 >> 172.21.15.67:0/1528925129 conn(0x7fe208077fc0 msgr2=0x7fe2080783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:02:21.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.600+0000 7fe20fc00700 1 -- 172.21.15.67:0/1528925129 shutdown_connections 2024-03-28T16:02:21.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.601+0000 7fe20fc00700 1 -- 172.21.15.67:0/1528925129 wait complete. 2024-03-28T16:02:21.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.602+0000 7fe20fc00700 1 Processor -- start 2024-03-28T16:02:21.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.602+0000 7fe20fc00700 1 -- start start 2024-03-28T16:02:21.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.602+0000 7fe20fc00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe208069e40 0x7fe2081a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:21.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.602+0000 7fe20fc00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe20806a7e0 0x7fe2081a0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:21.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.602+0000 7fe20fc00700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2081081e0 0x7fe20819a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:21.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.602+0000 7fe20fc00700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe208112330 con 0x7fe20806a7e0 2024-03-28T16:02:21.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.602+0000 7fe20fc00700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2081121b0 con 0x7fe2081081e0 2024-03-28T16:02:21.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.602+0000 7fe20fc00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2081124b0 con 0x7fe208069e40 2024-03-28T16:02:21.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.603+0000 7fe1ffe00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe20806a7e0 0x7fe2081a0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:21.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.603+0000 7fe1ffe00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe20806a7e0 0x7fe2081a0bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58438/0 (socket says 172.21.15.67:58438) 2024-03-28T16:02:21.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.603+0000 7fe1ffe00700 1 -- 172.21.15.67:0/807556744 learned_addr learned my addr 172.21.15.67:0/807556744 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:02:21.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.603+0000 7fe20d200700 1 --2- 172.21.15.67:0/807556744 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe208069e40 0x7fe2081a06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:21.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.603+0000 7fe20dc00700 1 --2- 172.21.15.67:0/807556744 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2081081e0 0x7fe20819a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:21.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.603+0000 7fe1ffe00700 1 -- 172.21.15.67:0/807556744 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe208069e40 msgr2=0x7fe2081a06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:21.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.603+0000 7fe1ffe00700 1 --2- 172.21.15.67:0/807556744 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe208069e40 0x7fe2081a06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:21.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.603+0000 7fe1ffe00700 1 -- 172.21.15.67:0/807556744 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2081081e0 msgr2=0x7fe20819a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:21.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.603+0000 7fe1ffe00700 1 --2- 172.21.15.67:0/807556744 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2081081e0 0x7fe20819a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:21.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.603+0000 7fe1ffe00700 1 -- 172.21.15.67:0/807556744 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe20819afd0 con 0x7fe20806a7e0 2024-03-28T16:02:21.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.603+0000 7fe20dc00700 1 --2- 172.21.15.67:0/807556744 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2081081e0 0x7fe20819a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:02:21.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.603+0000 7fe1ffe00700 1 --2- 172.21.15.67:0/807556744 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe20806a7e0 0x7fe2081a0bf0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fe1f800d950 tx=0x7fe1f800dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:02:21.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.604+0000 7fe206000700 1 -- 172.21.15.67:0/807556744 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1f8016070 con 0x7fe20806a7e0 2024-03-28T16:02:21.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.604+0000 7fe206000700 1 -- 172.21.15.67:0/807556744 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe1f8010460 con 0x7fe20806a7e0 2024-03-28T16:02:21.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.604+0000 7fe20fc00700 1 -- 172.21.15.67:0/807556744 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe20819b2c0 con 0x7fe20806a7e0 2024-03-28T16:02:21.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.604+0000 7fe206000700 1 -- 172.21.15.67:0/807556744 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1f800f620 con 0x7fe20806a7e0 2024-03-28T16:02:21.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.604+0000 7fe20fc00700 1 -- 172.21.15.67:0/807556744 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2081a7580 con 0x7fe20806a7e0 2024-03-28T16:02:21.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.605+0000 7fe20fc00700 1 -- 172.21.15.67:0/807556744 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe208068e50 con 0x7fe20806a7e0 2024-03-28T16:02:21.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.607+0000 7fe206000700 1 -- 172.21.15.67:0/807556744 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fe1f8019070 con 0x7fe20806a7e0 2024-03-28T16:02:21.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.609+0000 7fe206000700 1 --2- 172.21.15.67:0/807556744 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fe1e80a29a0 0x7fe1e80a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:21.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.609+0000 7fe206000700 1 -- 172.21.15.67:0/807556744 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fe1f80ce830 con 0x7fe20806a7e0 2024-03-28T16:02:21.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.609+0000 7fe20d200700 1 --2- 172.21.15.67:0/807556744 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fe1e80a29a0 0x7fe1e80a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:21.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.610+0000 7fe20d200700 1 --2- 172.21.15.67:0/807556744 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fe1e80a29a0 0x7fe1e80a4e60 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fe20819b9b0 tx=0x7fe1f000b5b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:02:21.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.614+0000 7fe206000700 1 -- 172.21.15.67:0/807556744 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe1f809b720 con 0x7fe20806a7e0 2024-03-28T16:02:21.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.861+0000 7fe20fc00700 1 -- 172.21.15.67:0/807556744 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) v1 -- 0x7fe2081a77e0 con 0x7fe20806a7e0 2024-03-28T16:02:21.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.861+0000 7fe206000700 1 -- 172.21.15.67:0/807556744 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fe1f809b020 con 0x7fe20806a7e0 2024-03-28T16:02:21.785 INFO:teuthology.orchestra.run.smithi067.stdout:270582939699 2024-03-28T16:02:21.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.864+0000 7fe20fc00700 1 -- 172.21.15.67:0/807556744 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fe1e80a29a0 msgr2=0x7fe1e80a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:21.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.864+0000 7fe20fc00700 1 --2- 172.21.15.67:0/807556744 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fe1e80a29a0 0x7fe1e80a4e60 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fe20819b9b0 tx=0x7fe1f000b5b0 comp rx=0 tx=0).stop 2024-03-28T16:02:21.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.864+0000 7fe20fc00700 1 -- 172.21.15.67:0/807556744 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe20806a7e0 msgr2=0x7fe2081a0bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:21.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.864+0000 7fe20fc00700 1 --2- 172.21.15.67:0/807556744 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe20806a7e0 0x7fe2081a0bf0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fe1f800d950 tx=0x7fe1f800dd10 comp rx=0 tx=0).stop 2024-03-28T16:02:21.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.864+0000 7fe20fc00700 1 -- 172.21.15.67:0/807556744 shutdown_connections 2024-03-28T16:02:21.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.864+0000 7fe20fc00700 1 --2- 172.21.15.67:0/807556744 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fe1e80a29a0 0x7fe1e80a4e60 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:21.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.864+0000 7fe20fc00700 1 --2- 172.21.15.67:0/807556744 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2081081e0 0x7fe20819a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:21.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.864+0000 7fe20fc00700 1 --2- 172.21.15.67:0/807556744 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe208069e40 0x7fe2081a06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:21.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.864+0000 7fe20fc00700 1 --2- 172.21.15.67:0/807556744 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe20806a7e0 0x7fe2081a0bf0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:21.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.864+0000 7fe20fc00700 1 -- 172.21.15.67:0/807556744 >> 172.21.15.67:0/807556744 conn(0x7fe208077fc0 msgr2=0x7fe20810d370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:02:21.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.864+0000 7fe20fc00700 1 -- 172.21.15.67:0/807556744 shutdown_connections 2024-03-28T16:02:21.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:20.864+0000 7fe20fc00700 1 -- 172.21.15.67:0/807556744 wait complete. 2024-03-28T16:02:22.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:21 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/807556744' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-28T16:02:22.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:21 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/690445149' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-28T16:02:22.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:21 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/807556744' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-28T16:02:22.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:21 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/690445149' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-28T16:02:22.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:21 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/807556744' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-28T16:02:22.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:21 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/690445149' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-28T16:02:22.125 INFO:tasks.cephadm.ceph_manager.ceph:need seq 330712481823 got 330712481828 for osd.10 2024-03-28T16:02:22.125 DEBUG:teuthology.parallel:result is None 2024-03-28T16:02:22.416 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574940 got 55834574945 for osd.1 2024-03-28T16:02:22.417 DEBUG:teuthology.parallel:result is None 2024-03-28T16:02:22.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:21.115+0000 7f948d200700 1 -- 172.21.15.67:0/2832993506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94881074f0 msgr2=0x7f948810f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:22.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:21.115+0000 7f948d200700 1 --2- 172.21.15.67:0/2832993506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94881074f0 0x7f948810f0c0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f948000b0d0 tx=0x7f948000b3e0 comp rx=0 tx=0).stop 2024-03-28T16:02:22.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:21.115+0000 7f948d200700 1 -- 172.21.15.67:0/2832993506 shutdown_connections 2024-03-28T16:02:22.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:21.115+0000 7f948d200700 1 --2- 172.21.15.67:0/2832993506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94881074f0 0x7f948810f0c0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:22.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:21.115+0000 7f948d200700 1 --2- 172.21.15.67:0/2832993506 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9488102410 0x7f94881027f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:22.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:21.115+0000 7f948d200700 1 --2- 172.21.15.67:0/2832993506 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9488102dc0 0x7f9488106e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:22.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:21.115+0000 7f948d200700 1 -- 172.21.15.67:0/2832993506 >> 172.21.15.67:0/2832993506 conn(0x7f94880fdc60 msgr2=0x7f9488100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:02:22.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:21.115+0000 7f948d200700 1 -- 172.21.15.67:0/2832993506 shutdown_connections 2024-03-28T16:02:22.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:21.115+0000 7f948d200700 1 -- 172.21.15.67:0/2832993506 wait complete. 2024-03-28T16:02:22.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:21.116+0000 7f948d200700 1 Processor -- start 2024-03-28T16:02:22.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:21.116+0000 7f948d200700 1 -- start start 2024-03-28T16:02:22.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:21.117+0000 7f948d200700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9488102410 0x7f948819e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:22.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:21.117+0000 7f948d200700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9488102dc0 0x7f948819e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:22.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:21.117+0000 7f948d200700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f94881074f0 0x7f9488198640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:22.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:21.117+0000 7f948d200700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94881123e0 con 0x7f9488102dc0 2024-03-28T16:02:22.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:21.117+0000 7f948d200700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9488112260 con 0x7f9488102410 2024-03-28T16:02:22.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:21.117+0000 7f948d200700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9488112560 con 0x7f94881074f0 2024-03-28T16:02:22.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:21.117+0000 7f9485400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9488102dc0 0x7f948819e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:22.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:21.117+0000 7f9485400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9488102dc0 0x7f948819e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58448/0 (socket says 172.21.15.67:58448) 2024-03-28T16:02:22.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:21.117+0000 7f9485400700 1 -- 172.21.15.67:0/690445149 learned_addr learned my addr 172.21.15.67:0/690445149 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:02:22.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:21.117+0000 7f9486800700 1 --2- 172.21.15.67:0/690445149 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f94881074f0 0x7f9488198640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:22.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:21.117+0000 7f9485e00700 1 --2- 172.21.15.67:0/690445149 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9488102410 0x7f948819e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:22.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:21.118+0000 7f9485400700 1 -- 172.21.15.67:0/690445149 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f94881074f0 msgr2=0x7f9488198640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:22.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:21.118+0000 7f9485400700 1 --2- 172.21.15.67:0/690445149 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f94881074f0 0x7f9488198640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:22.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:21.118+0000 7f9485400700 1 -- 172.21.15.67:0/690445149 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9488102410 msgr2=0x7f948819e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:22.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:21.118+0000 7f9485400700 1 --2- 172.21.15.67:0/690445149 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9488102410 0x7f948819e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:22.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:21.118+0000 7f9485400700 1 -- 172.21.15.67:0/690445149 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9488198da0 con 0x7f9488102dc0 2024-03-28T16:02:22.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:21.118+0000 7f9486800700 1 --2- 172.21.15.67:0/690445149 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f94881074f0 0x7f9488198640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T16:02:22.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:21.118+0000 7f9485400700 1 --2- 172.21.15.67:0/690445149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9488102dc0 0x7f948819e970 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f947400c9e0 tx=0x7f947400cda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:02:22.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:21.118+0000 7f947ea00700 1 -- 172.21.15.67:0/690445149 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9474007aa0 con 0x7f9488102dc0 2024-03-28T16:02:22.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:21.118+0000 7f947ea00700 1 -- 172.21.15.67:0/690445149 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9474007c40 con 0x7f9488102dc0 2024-03-28T16:02:22.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:21.118+0000 7f948d200700 1 -- 172.21.15.67:0/690445149 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9488199090 con 0x7f9488102dc0 2024-03-28T16:02:22.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:21.118+0000 7f947ea00700 1 -- 172.21.15.67:0/690445149 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9474018660 con 0x7f9488102dc0 2024-03-28T16:02:22.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:21.118+0000 7f948d200700 1 -- 172.21.15.67:0/690445149 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94881a5300 con 0x7f9488102dc0 2024-03-28T16:02:22.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:21.120+0000 7f948d200700 1 -- 172.21.15.67:0/690445149 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9488068e50 con 0x7f9488102dc0 2024-03-28T16:02:22.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:21.122+0000 7f947ea00700 1 -- 172.21.15.67:0/690445149 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f947400f450 con 0x7f9488102dc0 2024-03-28T16:02:22.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:21.124+0000 7f947ea00700 1 --2- 172.21.15.67:0/690445149 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f946c0a2920 0x7f946c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:22.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:21.124+0000 7f947ea00700 1 -- 172.21.15.67:0/690445149 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f94740cebb0 con 0x7f9488102dc0 2024-03-28T16:02:22.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:21.124+0000 7f9485e00700 1 --2- 172.21.15.67:0/690445149 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f946c0a2920 0x7f946c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:22.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:21.125+0000 7f9485e00700 1 --2- 172.21.15.67:0/690445149 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f946c0a2920 0x7f946c0a4de0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f9478005fd0 tx=0x7f947800a560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:02:22.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:21.128+0000 7f947ea00700 1 -- 172.21.15.67:0/690445149 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9474014030 con 0x7f9488102dc0 2024-03-28T16:02:22.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:21.364+0000 7f948d200700 1 -- 172.21.15.67:0/690445149 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f948802da30 con 0x7f9488102dc0 2024-03-28T16:02:22.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:21.365+0000 7f947ea00700 1 -- 172.21.15.67:0/690445149 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f947409baa0 con 0x7f9488102dc0 2024-03-28T16:02:22.691 INFO:teuthology.orchestra.run.smithi067.stdout:150323855438 2024-03-28T16:02:22.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:21.367+0000 7f948d200700 1 -- 172.21.15.67:0/690445149 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f946c0a2920 msgr2=0x7f946c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:22.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:21.367+0000 7f948d200700 1 --2- 172.21.15.67:0/690445149 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f946c0a2920 0x7f946c0a4de0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f9478005fd0 tx=0x7f947800a560 comp rx=0 tx=0).stop 2024-03-28T16:02:22.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:21.367+0000 7f948d200700 1 -- 172.21.15.67:0/690445149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9488102dc0 msgr2=0x7f948819e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:22.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:21.367+0000 7f948d200700 1 --2- 172.21.15.67:0/690445149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9488102dc0 0x7f948819e970 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f947400c9e0 tx=0x7f947400cda0 comp rx=0 tx=0).stop 2024-03-28T16:02:22.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:21.367+0000 7f948d200700 1 -- 172.21.15.67:0/690445149 shutdown_connections 2024-03-28T16:02:22.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:21.367+0000 7f948d200700 1 --2- 172.21.15.67:0/690445149 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f946c0a2920 0x7f946c0a4de0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:22.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:21.367+0000 7f948d200700 1 --2- 172.21.15.67:0/690445149 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f94881074f0 0x7f9488198640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:22.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:21.367+0000 7f948d200700 1 --2- 172.21.15.67:0/690445149 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9488102410 0x7f948819e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:22.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:21.367+0000 7f948d200700 1 --2- 172.21.15.67:0/690445149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9488102dc0 0x7f948819e970 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:22.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:21.367+0000 7f948d200700 1 -- 172.21.15.67:0/690445149 >> 172.21.15.67:0/690445149 conn(0x7f94880fdc60 msgr2=0x7f94880fe750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:02:22.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:21.368+0000 7f948d200700 1 -- 172.21.15.67:0/690445149 shutdown_connections 2024-03-28T16:02:22.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:21.368+0000 7f948d200700 1 -- 172.21.15.67:0/690445149 wait complete. 2024-03-28T16:02:22.995 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:22 smithi067 ceph-mon[31707]: pgmap v403: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:02:23.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:22 smithi150 ceph-mon[34825]: pgmap v403: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:02:23.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:22 smithi183 ceph-mon[35580]: pgmap v403: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:02:23.724 INFO:tasks.cephadm.ceph_manager.ceph:need seq 150323855432 got 150323855438 for osd.4 2024-03-28T16:02:23.724 DEBUG:teuthology.parallel:result is None 2024-03-28T16:02:24.167 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T16:02:24.220 INFO:tasks.cephadm.ceph_manager.ceph:need seq 115964117070 got 115964117075 for osd.3 2024-03-28T16:02:24.220 DEBUG:teuthology.parallel:result is None 2024-03-28T16:02:24.671 INFO:tasks.cephadm.ceph_manager.ceph:need seq 365072220183 got 365072220190 for osd.11 2024-03-28T16:02:24.672 DEBUG:teuthology.parallel:result is None 2024-03-28T16:02:25.004 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411416 got 77309411420 for osd.2 2024-03-28T16:02:25.004 DEBUG:teuthology.parallel:result is None 2024-03-28T16:02:25.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:24 smithi150 ceph-mon[34825]: pgmap v404: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:02:25.054 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:24 smithi067 ceph-mon[31707]: pgmap v404: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:02:25.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:24 smithi183 ceph-mon[35580]: pgmap v404: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:02:25.850 INFO:tasks.cephadm.ceph_manager.ceph:need seq 270582939692 got 270582939699 for osd.8 2024-03-28T16:02:25.850 DEBUG:teuthology.parallel:result is None 2024-03-28T16:02:27.017 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:26 smithi067 ceph-mon[31707]: pgmap v405: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:02:27.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:26 smithi183 ceph-mon[35580]: pgmap v405: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:02:27.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:26 smithi150 ceph-mon[34825]: pgmap v405: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:02:27.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:27.628+0000 7f62fdc00700 1 -- 172.21.15.67:0/1282449575 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62f806add0 msgr2=0x7f62f806b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:27.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:27.628+0000 7f62fdc00700 1 --2- 172.21.15.67:0/1282449575 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62f806add0 0x7f62f806b250 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f62e0009990 tx=0x7f62e0009ca0 comp rx=0 tx=0).stop 2024-03-28T16:02:27.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:27.628+0000 7f62fdc00700 1 -- 172.21.15.67:0/1282449575 shutdown_connections 2024-03-28T16:02:27.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:27.628+0000 7f62fdc00700 1 --2- 172.21.15.67:0/1282449575 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f62f8104cc0 0x7f62f8105120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:27.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:27.628+0000 7f62fdc00700 1 --2- 172.21.15.67:0/1282449575 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62f81071c0 0x7f62f806a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:27.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:27.628+0000 7f62fdc00700 1 --2- 172.21.15.67:0/1282449575 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62f806add0 0x7f62f806b250 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:27.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:27.628+0000 7f62fdc00700 1 -- 172.21.15.67:0/1282449575 >> 172.21.15.67:0/1282449575 conn(0x7f62f80770b0 msgr2=0x7f62f80774c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:02:27.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:27.629+0000 7f62fdc00700 1 -- 172.21.15.67:0/1282449575 shutdown_connections 2024-03-28T16:02:27.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:27.629+0000 7f62fdc00700 1 -- 172.21.15.67:0/1282449575 wait complete. 2024-03-28T16:02:27.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:27.630+0000 7f62fdc00700 1 Processor -- start 2024-03-28T16:02:27.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:27.630+0000 7f62fdc00700 1 -- start start 2024-03-28T16:02:27.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:27.630+0000 7f62fdc00700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62f806add0 0x7f62f819f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:27.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:27.630+0000 7f62fdc00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f62f8104cc0 0x7f62f819f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:27.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:27.631+0000 7f62fdc00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62f81071c0 0x7f62f81a38f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:27.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:27.631+0000 7f62fdc00700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62f8118a50 con 0x7f62f81071c0 2024-03-28T16:02:27.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:27.631+0000 7f62fdc00700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62f81188d0 con 0x7f62f806add0 2024-03-28T16:02:27.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:27.631+0000 7f62fdc00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62f8118bd0 con 0x7f62f8104cc0 2024-03-28T16:02:27.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:27.631+0000 7f62f7400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62f81071c0 0x7f62f81a38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:27.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:27.631+0000 7f62f7400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62f81071c0 0x7f62f81a38f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58460/0 (socket says 172.21.15.67:58460) 2024-03-28T16:02:27.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:27.631+0000 7f62f7400700 1 -- 172.21.15.67:0/1156917776 learned_addr learned my addr 172.21.15.67:0/1156917776 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:02:27.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:27.631+0000 7f62f6a00700 1 --2- 172.21.15.67:0/1156917776 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62f806add0 0x7f62f819f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:27.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:27.631+0000 7f62efe00700 1 --2- 172.21.15.67:0/1156917776 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f62f8104cc0 0x7f62f819f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:27.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:27.632+0000 7f62f7400700 1 -- 172.21.15.67:0/1156917776 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f62f8104cc0 msgr2=0x7f62f819f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:27.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:27.632+0000 7f62f7400700 1 --2- 172.21.15.67:0/1156917776 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f62f8104cc0 0x7f62f819f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:27.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:27.632+0000 7f62f7400700 1 -- 172.21.15.67:0/1156917776 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62f806add0 msgr2=0x7f62f819f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:27.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:27.632+0000 7f62f7400700 1 --2- 172.21.15.67:0/1156917776 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62f806add0 0x7f62f819f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:27.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:27.632+0000 7f62f7400700 1 -- 172.21.15.67:0/1156917776 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62f81a4030 con 0x7f62f81071c0 2024-03-28T16:02:27.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:27.632+0000 7f62f7400700 1 --2- 172.21.15.67:0/1156917776 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62f81071c0 0x7f62f81a38f0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f62f0007ee0 tx=0x7f62f000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:02:27.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:27.632+0000 7f62ef400700 1 -- 172.21.15.67:0/1156917776 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f0004bb0 con 0x7f62f81071c0 2024-03-28T16:02:27.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:27.632+0000 7f62fdc00700 1 -- 172.21.15.67:0/1156917776 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62f81a4320 con 0x7f62f81071c0 2024-03-28T16:02:27.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:27.632+0000 7f62ef400700 1 -- 172.21.15.67:0/1156917776 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f62f0004d50 con 0x7f62f81071c0 2024-03-28T16:02:27.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:27.632+0000 7f62ef400700 1 -- 172.21.15.67:0/1156917776 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f00058e0 con 0x7f62f81071c0 2024-03-28T16:02:27.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:27.632+0000 7f62fdc00700 1 -- 172.21.15.67:0/1156917776 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62f81abba0 con 0x7f62f81071c0 2024-03-28T16:02:27.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:27.633+0000 7f62fdc00700 1 -- 172.21.15.67:0/1156917776 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62f8068e50 con 0x7f62f81071c0 2024-03-28T16:02:27.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:27.636+0000 7f62ef400700 1 -- 172.21.15.67:0/1156917776 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f62f001a020 con 0x7f62f81071c0 2024-03-28T16:02:27.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:27.637+0000 7f62ef400700 1 --2- 172.21.15.67:0/1156917776 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f62dc0a2920 0x7f62dc0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:27.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:27.637+0000 7f62ef400700 1 -- 172.21.15.67:0/1156917776 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f62f00ce950 con 0x7f62f81071c0 2024-03-28T16:02:27.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:27.637+0000 7f62f6a00700 1 --2- 172.21.15.67:0/1156917776 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f62dc0a2920 0x7f62dc0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:27.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:27.638+0000 7f62f6a00700 1 --2- 172.21.15.67:0/1156917776 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f62dc0a2920 0x7f62dc0a4de0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f62e8005950 tx=0x7f62e800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:02:27.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:27.640+0000 7f62ef400700 1 -- 172.21.15.67:0/1156917776 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f62f0010070 con 0x7f62f81071c0 2024-03-28T16:02:27.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:27.874+0000 7f62fdc00700 1 -- 172.21.15.67:0/1156917776 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) v1 -- 0x7f62f81abdd0 con 0x7f62f81071c0 2024-03-28T16:02:27.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:27.875+0000 7f62ef400700 1 -- 172.21.15.67:0/1156917776 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f62f009b8a0 con 0x7f62f81071c0 2024-03-28T16:02:27.876 INFO:teuthology.orchestra.run.smithi067.stdout:304942678061 2024-03-28T16:02:27.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:27.877+0000 7f62fdc00700 1 -- 172.21.15.67:0/1156917776 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f62dc0a2920 msgr2=0x7f62dc0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:27.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:27.877+0000 7f62fdc00700 1 --2- 172.21.15.67:0/1156917776 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f62dc0a2920 0x7f62dc0a4de0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f62e8005950 tx=0x7f62e800b410 comp rx=0 tx=0).stop 2024-03-28T16:02:27.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:27.877+0000 7f62fdc00700 1 -- 172.21.15.67:0/1156917776 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62f81071c0 msgr2=0x7f62f81a38f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:27.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:27.877+0000 7f62fdc00700 1 --2- 172.21.15.67:0/1156917776 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62f81071c0 0x7f62f81a38f0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f62f0007ee0 tx=0x7f62f000d3b0 comp rx=0 tx=0).stop 2024-03-28T16:02:27.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:27.878+0000 7f62fdc00700 1 -- 172.21.15.67:0/1156917776 shutdown_connections 2024-03-28T16:02:27.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:27.878+0000 7f62fdc00700 1 --2- 172.21.15.67:0/1156917776 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f62dc0a2920 0x7f62dc0a4de0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:27.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:27.878+0000 7f62fdc00700 1 --2- 172.21.15.67:0/1156917776 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62f81071c0 0x7f62f81a38f0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:27.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:27.878+0000 7f62fdc00700 1 --2- 172.21.15.67:0/1156917776 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62f806add0 0x7f62f819f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:27.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:27.878+0000 7f62fdc00700 1 --2- 172.21.15.67:0/1156917776 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f62f8104cc0 0x7f62f819f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:27.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:27.878+0000 7f62fdc00700 1 -- 172.21.15.67:0/1156917776 >> 172.21.15.67:0/1156917776 conn(0x7f62f80770b0 msgr2=0x7f62f81000b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:02:27.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:27.878+0000 7f62fdc00700 1 -- 172.21.15.67:0/1156917776 shutdown_connections 2024-03-28T16:02:27.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:27.878+0000 7f62fdc00700 1 -- 172.21.15.67:0/1156917776 wait complete. 2024-03-28T16:02:28.635 INFO:tasks.cephadm.ceph_manager.ceph:need seq 304942678056 got 304942678061 for osd.9 2024-03-28T16:02:28.635 DEBUG:teuthology.parallel:result is None 2024-03-28T16:02:28.635 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-28T16:02:28.635 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-28T16:02:29.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:28 smithi067 ceph-mon[31707]: pgmap v406: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:02:29.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:28 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/1156917776' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-28T16:02:29.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:28 smithi183 ceph-mon[35580]: pgmap v406: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:02:29.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:28 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/1156917776' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-28T16:02:29.114 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T16:02:29.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:28 smithi150 ceph-mon[34825]: pgmap v406: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:02:29.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:28 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/1156917776' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-28T16:02:30.544 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:30 smithi067 ceph-mon[31707]: pgmap v407: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:02:30.545 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:30 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:02:30.545 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:30 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:02:30.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:30 smithi150 ceph-mon[34825]: pgmap v407: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:02:30.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:30 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:02:30.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:30 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:02:30.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:30 smithi183 ceph-mon[35580]: pgmap v407: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:02:30.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:30 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:02:30.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:30 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:02:31.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:31.100+0000 7fd3b3000700 1 -- 172.21.15.67:0/2776159274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3ac10ec40 msgr2=0x7fd3ac111030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:31.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:31.100+0000 7fd3b3000700 1 --2- 172.21.15.67:0/2776159274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3ac10ec40 0x7fd3ac111030 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fd3a400b0a0 tx=0x7fd3a400b3b0 comp rx=0 tx=0).stop 2024-03-28T16:02:31.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:31.100+0000 7fd3b3000700 1 -- 172.21.15.67:0/2776159274 shutdown_connections 2024-03-28T16:02:31.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:31.100+0000 7fd3b3000700 1 --2- 172.21.15.67:0/2776159274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3ac10ec40 0x7fd3ac111030 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:31.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:31.100+0000 7fd3b3000700 1 --2- 172.21.15.67:0/2776159274 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3ac069e40 0x7fd3ac06a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:31.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:31.100+0000 7fd3b3000700 1 --2- 172.21.15.67:0/2776159274 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd3ac06a760 0x7fd3ac10e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:31.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:31.100+0000 7fd3b3000700 1 -- 172.21.15.67:0/2776159274 >> 172.21.15.67:0/2776159274 conn(0x7fd3ac077fc0 msgr2=0x7fd3ac0783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:02:31.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:31.100+0000 7fd3b3000700 1 -- 172.21.15.67:0/2776159274 shutdown_connections 2024-03-28T16:02:31.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:31.100+0000 7fd3b3000700 1 -- 172.21.15.67:0/2776159274 wait complete. 2024-03-28T16:02:31.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:31.101+0000 7fd3b3000700 1 Processor -- start 2024-03-28T16:02:31.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:31.101+0000 7fd3b3000700 1 -- start start 2024-03-28T16:02:31.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:31.102+0000 7fd3b3000700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3ac069e40 0x7fd3ac19ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:31.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:31.102+0000 7fd3b3000700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3ac06a760 0x7fd3ac19b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:31.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:31.102+0000 7fd3b3000700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd3ac10ec40 0x7fd3ac19f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:31.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:31.102+0000 7fd3b3000700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3ac114560 con 0x7fd3ac06a760 2024-03-28T16:02:31.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:31.102+0000 7fd3b3000700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3ac1143e0 con 0x7fd3ac069e40 2024-03-28T16:02:31.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:31.102+0000 7fd3b3000700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3ac1146e0 con 0x7fd3ac10ec40 2024-03-28T16:02:31.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:31.102+0000 7fd3a3400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3ac06a760 0x7fd3ac19b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:31.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:31.102+0000 7fd3a3400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3ac06a760 0x7fd3ac19b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42554/0 (socket says 172.21.15.67:42554) 2024-03-28T16:02:31.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:31.102+0000 7fd3a3400700 1 -- 172.21.15.67:0/3496609476 learned_addr learned my addr 172.21.15.67:0/3496609476 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:02:31.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:31.102+0000 7fd3abe00700 1 --2- 172.21.15.67:0/3496609476 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3ac069e40 0x7fd3ac19ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:31.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:31.102+0000 7fd3b1000700 1 --2- 172.21.15.67:0/3496609476 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd3ac10ec40 0x7fd3ac19f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:31.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:31.103+0000 7fd3a3400700 1 -- 172.21.15.67:0/3496609476 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd3ac10ec40 msgr2=0x7fd3ac19f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:31.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:31.103+0000 7fd3a3400700 1 --2- 172.21.15.67:0/3496609476 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd3ac10ec40 0x7fd3ac19f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:31.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:31.103+0000 7fd3a3400700 1 -- 172.21.15.67:0/3496609476 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3ac069e40 msgr2=0x7fd3ac19ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:31.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:31.103+0000 7fd3a3400700 1 --2- 172.21.15.67:0/3496609476 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3ac069e40 0x7fd3ac19ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:31.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:31.103+0000 7fd3a3400700 1 -- 172.21.15.67:0/3496609476 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3ac19fb60 con 0x7fd3ac06a760 2024-03-28T16:02:31.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:31.103+0000 7fd3a3400700 1 --2- 172.21.15.67:0/3496609476 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3ac06a760 0x7fd3ac19b0a0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fd39400d980 tx=0x7fd39400dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:02:31.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:31.103+0000 7fd3a9600700 1 -- 172.21.15.67:0/3496609476 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd394017070 con 0x7fd3ac06a760 2024-03-28T16:02:31.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:31.103+0000 7fd3a9600700 1 -- 172.21.15.67:0/3496609476 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd394004830 con 0x7fd3ac06a760 2024-03-28T16:02:31.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:31.103+0000 7fd3a9600700 1 -- 172.21.15.67:0/3496609476 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd394005440 con 0x7fd3ac06a760 2024-03-28T16:02:31.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:31.103+0000 7fd3b3000700 1 -- 172.21.15.67:0/3496609476 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3ac19fe50 con 0x7fd3ac06a760 2024-03-28T16:02:31.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:31.103+0000 7fd3b3000700 1 -- 172.21.15.67:0/3496609476 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3ac1a7710 con 0x7fd3ac06a760 2024-03-28T16:02:31.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:31.105+0000 7fd3b3000700 1 -- 172.21.15.67:0/3496609476 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3ac068e50 con 0x7fd3ac06a760 2024-03-28T16:02:31.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:31.107+0000 7fd3a9600700 1 -- 172.21.15.67:0/3496609476 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fd3940055e0 con 0x7fd3ac06a760 2024-03-28T16:02:31.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:31.108+0000 7fd3a9600700 1 --2- 172.21.15.67:0/3496609476 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fd3900a2920 0x7fd3900a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:31.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:31.108+0000 7fd3a9600700 1 -- 172.21.15.67:0/3496609476 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fd3940ceb00 con 0x7fd3ac06a760 2024-03-28T16:02:31.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:31.109+0000 7fd3abe00700 1 --2- 172.21.15.67:0/3496609476 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fd3900a2920 0x7fd3900a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:31.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:31.109+0000 7fd3abe00700 1 --2- 172.21.15.67:0/3496609476 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fd3900a2920 0x7fd3900a4de0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd39c005fd0 tx=0x7fd39c005f60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:02:31.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:31.113+0000 7fd3a9600700 1 -- 172.21.15.67:0/3496609476 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd394020020 con 0x7fd3ac06a760 2024-03-28T16:02:31.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:31.337+0000 7fd3b3000700 1 -- 172.21.15.67:0/3496609476 --> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd3ac0625d0 con 0x7fd3900a2920 2024-03-28T16:02:31.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:31.347+0000 7fd3a9600700 1 -- 172.21.15.67:0/3496609476 <== mgr.14152 v2:172.21.15.67:6800/208726654 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+258651 (secure 0 0 0) 0x7fd3ac0625d0 con 0x7fd3900a2920 2024-03-28T16:02:31.351 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:02:31.354 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:31 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:02:31.354 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:31 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T16:02:31.354 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:31 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:02:31.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:31.352+0000 7fd3b3000700 1 -- 172.21.15.67:0/3496609476 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fd3900a2920 msgr2=0x7fd3900a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:31.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:31.352+0000 7fd3b3000700 1 --2- 172.21.15.67:0/3496609476 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fd3900a2920 0x7fd3900a4de0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd39c005fd0 tx=0x7fd39c005f60 comp rx=0 tx=0).stop 2024-03-28T16:02:31.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:31.352+0000 7fd3b3000700 1 -- 172.21.15.67:0/3496609476 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3ac06a760 msgr2=0x7fd3ac19b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:31.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:31.352+0000 7fd3b3000700 1 --2- 172.21.15.67:0/3496609476 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3ac06a760 0x7fd3ac19b0a0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fd39400d980 tx=0x7fd39400dd40 comp rx=0 tx=0).stop 2024-03-28T16:02:31.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:31.352+0000 7fd3b3000700 1 -- 172.21.15.67:0/3496609476 shutdown_connections 2024-03-28T16:02:31.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:31.352+0000 7fd3b3000700 1 --2- 172.21.15.67:0/3496609476 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fd3900a2920 0x7fd3900a4de0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:31.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:31.352+0000 7fd3b3000700 1 --2- 172.21.15.67:0/3496609476 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd3ac10ec40 0x7fd3ac19f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:31.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:31.352+0000 7fd3b3000700 1 --2- 172.21.15.67:0/3496609476 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3ac069e40 0x7fd3ac19ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:31.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:31.352+0000 7fd3b3000700 1 --2- 172.21.15.67:0/3496609476 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3ac06a760 0x7fd3ac19b0a0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:31.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:31.352+0000 7fd3b3000700 1 -- 172.21.15.67:0/3496609476 >> 172.21.15.67:0/3496609476 conn(0x7fd3ac077fc0 msgr2=0x7fd3ac110090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:02:31.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:31.353+0000 7fd3b3000700 1 -- 172.21.15.67:0/3496609476 shutdown_connections 2024-03-28T16:02:31.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:31.353+0000 7fd3b3000700 1 -- 172.21.15.67:0/3496609476 wait complete. 2024-03-28T16:02:31.357 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-28T16:02:31.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:31 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:02:31.553 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:31 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T16:02:31.553 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:31 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:02:31.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:31 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:02:31.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:31 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T16:02:31.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:31 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:02:32.205 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":408,"stamp":"2024-03-28T16:02:31.073030+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":653296,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":295,"num_read_kb":256,"num_write":262,"num_write_kb":3025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1180758,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":205,"ondisk_log_size":205,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":383664,"kb_used_data":13644,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124558160,"statfs":{"total":1151940427776,"available":1151547555840,"internally_reserved":0,"allocated":13971456,"data_stored":7705812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":378713402},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":30,"num_read_kb":15,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005155"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.663698+0000","last_change":"2024-03-28T16:01:08.268196+0000","last_active":"2024-03-28T16:01:40.663698+0000","last_peered":"2024-03-28T16:01:40.663698+0000","last_clean":"2024-03-28T16:01:40.663698+0000","last_became_active":"2024-03-28T16:01:08.266956+0000","last_became_peered":"2024-03-28T16:01:08.266956+0000","last_unstale":"2024-03-28T16:01:40.663698+0000","last_undegraded":"2024-03-28T16:01:40.663698+0000","last_fullsized":"2024-03-28T16:01:40.663698+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:38:44.296129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.666457+0000","last_change":"2024-03-28T16:01:08.264985+0000","last_active":"2024-03-28T16:01:40.666457+0000","last_peered":"2024-03-28T16:01:40.666457+0000","last_clean":"2024-03-28T16:01:40.666457+0000","last_became_active":"2024-03-28T16:01:08.262484+0000","last_became_peered":"2024-03-28T16:01:08.262484+0000","last_unstale":"2024-03-28T16:01:40.666457+0000","last_undegraded":"2024-03-28T16:01:40.666457+0000","last_fullsized":"2024-03-28T16:01:40.666457+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:33:07.538908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.666725+0000","last_change":"2024-03-28T16:01:08.267423+0000","last_active":"2024-03-28T16:01:40.666725+0000","last_peered":"2024-03-28T16:01:40.666725+0000","last_clean":"2024-03-28T16:01:40.666725+0000","last_became_active":"2024-03-28T16:01:08.263665+0000","last_became_peered":"2024-03-28T16:01:08.263665+0000","last_unstale":"2024-03-28T16:01:40.666725+0000","last_undegraded":"2024-03-28T16:01:40.666725+0000","last_fullsized":"2024-03-28T16:01:40.666725+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:09:43.488003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.656204+0000","last_change":"2024-03-28T16:01:08.274978+0000","last_active":"2024-03-28T16:01:40.656204+0000","last_peered":"2024-03-28T16:01:40.656204+0000","last_clean":"2024-03-28T16:01:40.656204+0000","last_became_active":"2024-03-28T16:01:08.274553+0000","last_became_peered":"2024-03-28T16:01:08.274553+0000","last_unstale":"2024-03-28T16:01:40.656204+0000","last_undegraded":"2024-03-28T16:01:40.656204+0000","last_fullsized":"2024-03-28T16:01:40.656204+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:16:03.997334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.654621+0000","last_change":"2024-03-28T16:01:08.262155+0000","last_active":"2024-03-28T16:01:40.654621+0000","last_peered":"2024-03-28T16:01:40.654621+0000","last_clean":"2024-03-28T16:01:40.654621+0000","last_became_active":"2024-03-28T16:01:08.261814+0000","last_became_peered":"2024-03-28T16:01:08.261814+0000","last_unstale":"2024-03-28T16:01:40.654621+0000","last_undegraded":"2024-03-28T16:01:40.654621+0000","last_fullsized":"2024-03-28T16:01:40.654621+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:41:04.242203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.661462+0000","last_change":"2024-03-28T16:01:08.273342+0000","last_active":"2024-03-28T16:01:40.661462+0000","last_peered":"2024-03-28T16:01:40.661462+0000","last_clean":"2024-03-28T16:01:40.661462+0000","last_became_active":"2024-03-28T16:01:08.272606+0000","last_became_peered":"2024-03-28T16:01:08.272606+0000","last_unstale":"2024-03-28T16:01:40.661462+0000","last_undegraded":"2024-03-28T16:01:40.661462+0000","last_fullsized":"2024-03-28T16:01:40.661462+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:44:48.580214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.666700+0000","last_change":"2024-03-28T16:01:08.278060+0000","last_active":"2024-03-28T16:01:40.666700+0000","last_peered":"2024-03-28T16:01:40.666700+0000","last_clean":"2024-03-28T16:01:40.666700+0000","last_became_active":"2024-03-28T16:01:08.277047+0000","last_became_peered":"2024-03-28T16:01:08.277047+0000","last_unstale":"2024-03-28T16:01:40.666700+0000","last_undegraded":"2024-03-28T16:01:40.666700+0000","last_fullsized":"2024-03-28T16:01:40.666700+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:46:30.005708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.764145+0000","last_change":"2024-03-28T16:01:08.273199+0000","last_active":"2024-03-28T16:01:40.764145+0000","last_peered":"2024-03-28T16:01:40.764145+0000","last_clean":"2024-03-28T16:01:40.764145+0000","last_became_active":"2024-03-28T16:01:08.272822+0000","last_became_peered":"2024-03-28T16:01:08.272822+0000","last_unstale":"2024-03-28T16:01:40.764145+0000","last_undegraded":"2024-03-28T16:01:40.764145+0000","last_fullsized":"2024-03-28T16:01:40.764145+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:28:11.495665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.839751+0000","last_change":"2024-03-28T16:01:08.270262+0000","last_active":"2024-03-28T16:01:40.839751+0000","last_peered":"2024-03-28T16:01:40.839751+0000","last_clean":"2024-03-28T16:01:40.839751+0000","last_became_active":"2024-03-28T16:01:08.269959+0000","last_became_peered":"2024-03-28T16:01:08.269959+0000","last_unstale":"2024-03-28T16:01:40.839751+0000","last_undegraded":"2024-03-28T16:01:40.839751+0000","last_fullsized":"2024-03-28T16:01:40.839751+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:49:02.261893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.671104+0000","last_change":"2024-03-28T16:01:40.671104+0000","last_active":"2024-03-28T16:01:40.671104+0000","last_peered":"2024-03-28T16:01:40.671104+0000","last_clean":"2024-03-28T16:01:40.671104+0000","last_became_active":"2024-03-28T16:01:40.670474+0000","last_became_peered":"2024-03-28T16:01:40.670474+0000","last_unstale":"2024-03-28T16:01:40.671104+0000","last_undegraded":"2024-03-28T16:01:40.671104+0000","last_fullsized":"2024-03-28T16:01:40.671104+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:19:41.651703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.667731+0000","last_change":"2024-03-28T16:01:08.274943+0000","last_active":"2024-03-28T16:01:40.667731+0000","last_peered":"2024-03-28T16:01:40.667731+0000","last_clean":"2024-03-28T16:01:40.667731+0000","last_became_active":"2024-03-28T16:01:08.274267+0000","last_became_peered":"2024-03-28T16:01:08.274267+0000","last_unstale":"2024-03-28T16:01:40.667731+0000","last_undegraded":"2024-03-28T16:01:40.667731+0000","last_fullsized":"2024-03-28T16:01:40.667731+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:41:13.979918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.656132+0000","last_change":"2024-03-28T16:01:08.266817+0000","last_active":"2024-03-28T16:01:40.656132+0000","last_peered":"2024-03-28T16:01:40.656132+0000","last_clean":"2024-03-28T16:01:40.656132+0000","last_became_active":"2024-03-28T16:01:08.266470+0000","last_became_peered":"2024-03-28T16:01:08.266470+0000","last_unstale":"2024-03-28T16:01:40.656132+0000","last_undegraded":"2024-03-28T16:01:40.656132+0000","last_fullsized":"2024-03-28T16:01:40.656132+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:15:28.573153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.920646+0000","last_change":"2024-03-28T16:01:08.269896+0000","last_active":"2024-03-28T16:01:40.920646+0000","last_peered":"2024-03-28T16:01:40.920646+0000","last_clean":"2024-03-28T16:01:40.920646+0000","last_became_active":"2024-03-28T16:01:08.269500+0000","last_became_peered":"2024-03-28T16:01:08.269500+0000","last_unstale":"2024-03-28T16:01:40.920646+0000","last_undegraded":"2024-03-28T16:01:40.920646+0000","last_fullsized":"2024-03-28T16:01:40.920646+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:44:38.867590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.665936+0000","last_change":"2024-03-28T16:01:08.267072+0000","last_active":"2024-03-28T16:01:40.665936+0000","last_peered":"2024-03-28T16:01:40.665936+0000","last_clean":"2024-03-28T16:01:40.665936+0000","last_became_active":"2024-03-28T16:01:08.266688+0000","last_became_peered":"2024-03-28T16:01:08.266688+0000","last_unstale":"2024-03-28T16:01:40.665936+0000","last_undegraded":"2024-03-28T16:01:40.665936+0000","last_fullsized":"2024-03-28T16:01:40.665936+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:07:57.702952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.665434+0000","last_change":"2024-03-28T16:01:08.270912+0000","last_active":"2024-03-28T16:01:40.665434+0000","last_peered":"2024-03-28T16:01:40.665434+0000","last_clean":"2024-03-28T16:01:40.665434+0000","last_became_active":"2024-03-28T16:01:08.270567+0000","last_became_peered":"2024-03-28T16:01:08.270567+0000","last_unstale":"2024-03-28T16:01:40.665434+0000","last_undegraded":"2024-03-28T16:01:40.665434+0000","last_fullsized":"2024-03-28T16:01:40.665434+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:19:29.294502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.837738+0000","last_change":"2024-03-28T16:01:08.266811+0000","last_active":"2024-03-28T16:01:40.837738+0000","last_peered":"2024-03-28T16:01:40.837738+0000","last_clean":"2024-03-28T16:01:40.837738+0000","last_became_active":"2024-03-28T16:01:08.266517+0000","last_became_peered":"2024-03-28T16:01:08.266517+0000","last_unstale":"2024-03-28T16:01:40.837738+0000","last_undegraded":"2024-03-28T16:01:40.837738+0000","last_fullsized":"2024-03-28T16:01:40.837738+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:55:05.972683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.663371+0000","last_change":"2024-03-28T16:01:08.263864+0000","last_active":"2024-03-28T16:01:40.663371+0000","last_peered":"2024-03-28T16:01:40.663371+0000","last_clean":"2024-03-28T16:01:40.663371+0000","last_became_active":"2024-03-28T16:01:08.262956+0000","last_became_peered":"2024-03-28T16:01:08.262956+0000","last_unstale":"2024-03-28T16:01:40.663371+0000","last_undegraded":"2024-03-28T16:01:40.663371+0000","last_fullsized":"2024-03-28T16:01:40.663371+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:05:50.504237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.655131+0000","last_change":"2024-03-28T16:01:08.265151+0000","last_active":"2024-03-28T16:01:40.655131+0000","last_peered":"2024-03-28T16:01:40.655131+0000","last_clean":"2024-03-28T16:01:40.655131+0000","last_became_active":"2024-03-28T16:01:08.264861+0000","last_became_peered":"2024-03-28T16:01:08.264861+0000","last_unstale":"2024-03-28T16:01:40.655131+0000","last_undegraded":"2024-03-28T16:01:40.655131+0000","last_fullsized":"2024-03-28T16:01:40.655131+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:35:45.878983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.667927+0000","last_change":"2024-03-28T16:01:08.265993+0000","last_active":"2024-03-28T16:01:40.667927+0000","last_peered":"2024-03-28T16:01:40.667927+0000","last_clean":"2024-03-28T16:01:40.667927+0000","last_became_active":"2024-03-28T16:01:08.265488+0000","last_became_peered":"2024-03-28T16:01:08.265488+0000","last_unstale":"2024-03-28T16:01:40.667927+0000","last_undegraded":"2024-03-28T16:01:40.667927+0000","last_fullsized":"2024-03-28T16:01:40.667927+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:31:55.920705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'4","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.654757+0000","last_change":"2024-03-28T16:01:07.250361+0000","last_active":"2024-03-28T16:01:40.654757+0000","last_peered":"2024-03-28T16:01:40.654757+0000","last_clean":"2024-03-28T16:01:40.654757+0000","last_became_active":"2024-03-28T16:01:07.250004+0000","last_became_peered":"2024-03-28T16:01:07.250004+0000","last_unstale":"2024-03-28T16:01:40.654757+0000","last_undegraded":"2024-03-28T16:01:40.654757+0000","last_fullsized":"2024-03-28T16:01:40.654757+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:10:32.838716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.765149+0000","last_change":"2024-03-28T16:01:08.274436+0000","last_active":"2024-03-28T16:01:40.765149+0000","last_peered":"2024-03-28T16:01:40.765149+0000","last_clean":"2024-03-28T16:01:40.765149+0000","last_became_active":"2024-03-28T16:01:08.274124+0000","last_became_peered":"2024-03-28T16:01:08.274124+0000","last_unstale":"2024-03-28T16:01:40.765149+0000","last_undegraded":"2024-03-28T16:01:40.765149+0000","last_fullsized":"2024-03-28T16:01:40.765149+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:02:34.037686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"89'1","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.765269+0000","last_change":"2024-03-28T16:01:07.248565+0000","last_active":"2024-03-28T16:01:40.765269+0000","last_peered":"2024-03-28T16:01:40.765269+0000","last_clean":"2024-03-28T16:01:40.765269+0000","last_became_active":"2024-03-28T16:01:07.247366+0000","last_became_peered":"2024-03-28T16:01:07.247366+0000","last_unstale":"2024-03-28T16:01:40.765269+0000","last_undegraded":"2024-03-28T16:01:40.765269+0000","last_fullsized":"2024-03-28T16:01:40.765269+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:48:13.146398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.667251+0000","last_change":"2024-03-28T16:01:08.265639+0000","last_active":"2024-03-28T16:01:40.667251+0000","last_peered":"2024-03-28T16:01:40.667251+0000","last_clean":"2024-03-28T16:01:40.667251+0000","last_became_active":"2024-03-28T16:01:08.263488+0000","last_became_peered":"2024-03-28T16:01:08.263488+0000","last_unstale":"2024-03-28T16:01:40.667251+0000","last_undegraded":"2024-03-28T16:01:40.667251+0000","last_fullsized":"2024-03-28T16:01:40.667251+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:15:06.516016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"89'2","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.850755+0000","last_change":"2024-03-28T16:01:40.850755+0000","last_active":"2024-03-28T16:01:40.850755+0000","last_peered":"2024-03-28T16:01:40.850755+0000","last_clean":"2024-03-28T16:01:40.850755+0000","last_became_active":"2024-03-28T16:01:40.842547+0000","last_became_peered":"2024-03-28T16:01:40.842547+0000","last_unstale":"2024-03-28T16:01:40.850755+0000","last_undegraded":"2024-03-28T16:01:40.850755+0000","last_fullsized":"2024-03-28T16:01:40.850755+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:39:27.862525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.838279+0000","last_change":"2024-03-28T16:01:08.271534+0000","last_active":"2024-03-28T16:01:40.838279+0000","last_peered":"2024-03-28T16:01:40.838279+0000","last_clean":"2024-03-28T16:01:40.838279+0000","last_became_active":"2024-03-28T16:01:08.271104+0000","last_became_peered":"2024-03-28T16:01:08.271104+0000","last_unstale":"2024-03-28T16:01:40.838279+0000","last_undegraded":"2024-03-28T16:01:40.838279+0000","last_fullsized":"2024-03-28T16:01:40.838279+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:29:21.332346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"89'2","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.655555+0000","last_change":"2024-03-28T16:01:07.259175+0000","last_active":"2024-03-28T16:01:40.655555+0000","last_peered":"2024-03-28T16:01:40.655555+0000","last_clean":"2024-03-28T16:01:40.655555+0000","last_became_active":"2024-03-28T16:01:07.258151+0000","last_became_peered":"2024-03-28T16:01:07.258151+0000","last_unstale":"2024-03-28T16:01:40.655555+0000","last_undegraded":"2024-03-28T16:01:40.655555+0000","last_fullsized":"2024-03-28T16:01:40.655555+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:41:07.064800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.843211+0000","last_change":"2024-03-28T16:01:40.843211+0000","last_active":"2024-03-28T16:01:40.843211+0000","last_peered":"2024-03-28T16:01:40.843211+0000","last_clean":"2024-03-28T16:01:40.843211+0000","last_became_active":"2024-03-28T16:01:40.842051+0000","last_became_peered":"2024-03-28T16:01:40.842051+0000","last_unstale":"2024-03-28T16:01:40.843211+0000","last_undegraded":"2024-03-28T16:01:40.843211+0000","last_fullsized":"2024-03-28T16:01:40.843211+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:19:26.437169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"89'1","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.664703+0000","last_change":"2024-03-28T16:01:07.248538+0000","last_active":"2024-03-28T16:01:40.664703+0000","last_peered":"2024-03-28T16:01:40.664703+0000","last_clean":"2024-03-28T16:01:40.664703+0000","last_became_active":"2024-03-28T16:01:07.248116+0000","last_became_peered":"2024-03-28T16:01:07.248116+0000","last_unstale":"2024-03-28T16:01:40.664703+0000","last_undegraded":"2024-03-28T16:01:40.664703+0000","last_fullsized":"2024-03-28T16:01:40.664703+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:44:56.071286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.655028+0000","last_change":"2024-03-28T16:01:08.271059+0000","last_active":"2024-03-28T16:01:40.655028+0000","last_peered":"2024-03-28T16:01:40.655028+0000","last_clean":"2024-03-28T16:01:40.655028+0000","last_became_active":"2024-03-28T16:01:08.270682+0000","last_became_peered":"2024-03-28T16:01:08.270682+0000","last_unstale":"2024-03-28T16:01:40.655028+0000","last_undegraded":"2024-03-28T16:01:40.655028+0000","last_fullsized":"2024-03-28T16:01:40.655028+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:54:48.922125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"89'1","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.661404+0000","last_change":"2024-03-28T16:01:07.261157+0000","last_active":"2024-03-28T16:01:40.661404+0000","last_peered":"2024-03-28T16:01:40.661404+0000","last_clean":"2024-03-28T16:01:40.661404+0000","last_became_active":"2024-03-28T16:01:07.260751+0000","last_became_peered":"2024-03-28T16:01:07.260751+0000","last_unstale":"2024-03-28T16:01:40.661404+0000","last_undegraded":"2024-03-28T16:01:40.661404+0000","last_fullsized":"2024-03-28T16:01:40.661404+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:16:49.622168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.664847+0000","last_change":"2024-03-28T16:01:08.265773+0000","last_active":"2024-03-28T16:01:40.664847+0000","last_peered":"2024-03-28T16:01:40.664847+0000","last_clean":"2024-03-28T16:01:40.664847+0000","last_became_active":"2024-03-28T16:01:08.263724+0000","last_became_peered":"2024-03-28T16:01:08.263724+0000","last_unstale":"2024-03-28T16:01:40.664847+0000","last_undegraded":"2024-03-28T16:01:40.664847+0000","last_fullsized":"2024-03-28T16:01:40.664847+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:42:37.338946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.921045+0000","last_change":"2024-03-28T16:01:07.235361+0000","last_active":"2024-03-28T16:01:40.921045+0000","last_peered":"2024-03-28T16:01:40.921045+0000","last_clean":"2024-03-28T16:01:40.921045+0000","last_became_active":"2024-03-28T16:01:07.234969+0000","last_became_peered":"2024-03-28T16:01:07.234969+0000","last_unstale":"2024-03-28T16:01:40.921045+0000","last_undegraded":"2024-03-28T16:01:40.921045+0000","last_fullsized":"2024-03-28T16:01:40.921045+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:50:35.592147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.764023+0000","last_change":"2024-03-28T16:01:08.263878+0000","last_active":"2024-03-28T16:01:40.764023+0000","last_peered":"2024-03-28T16:01:40.764023+0000","last_clean":"2024-03-28T16:01:40.764023+0000","last_became_active":"2024-03-28T16:01:08.263549+0000","last_became_peered":"2024-03-28T16:01:08.263549+0000","last_unstale":"2024-03-28T16:01:40.764023+0000","last_undegraded":"2024-03-28T16:01:40.764023+0000","last_fullsized":"2024-03-28T16:01:40.764023+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:31:37.669215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.663697+0000","last_change":"2024-03-28T16:01:07.255701+0000","last_active":"2024-03-28T16:01:40.663697+0000","last_peered":"2024-03-28T16:01:40.663697+0000","last_clean":"2024-03-28T16:01:40.663697+0000","last_became_active":"2024-03-28T16:01:07.255247+0000","last_became_peered":"2024-03-28T16:01:07.255247+0000","last_unstale":"2024-03-28T16:01:40.663697+0000","last_undegraded":"2024-03-28T16:01:40.663697+0000","last_fullsized":"2024-03-28T16:01:40.663697+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:08:17.858395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.652417+0000","last_change":"2024-03-28T16:01:08.252949+0000","last_active":"2024-03-28T16:01:40.652417+0000","last_peered":"2024-03-28T16:01:40.652417+0000","last_clean":"2024-03-28T16:01:40.652417+0000","last_became_active":"2024-03-28T16:01:08.252471+0000","last_became_peered":"2024-03-28T16:01:08.252471+0000","last_unstale":"2024-03-28T16:01:40.652417+0000","last_undegraded":"2024-03-28T16:01:40.652417+0000","last_fullsized":"2024-03-28T16:01:40.652417+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:21:15.519020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.661736+0000","last_change":"2024-03-28T16:01:07.263973+0000","last_active":"2024-03-28T16:01:40.661736+0000","last_peered":"2024-03-28T16:01:40.661736+0000","last_clean":"2024-03-28T16:01:40.661736+0000","last_became_active":"2024-03-28T16:01:07.263560+0000","last_became_peered":"2024-03-28T16:01:07.263560+0000","last_unstale":"2024-03-28T16:01:40.661736+0000","last_undegraded":"2024-03-28T16:01:40.661736+0000","last_fullsized":"2024-03-28T16:01:40.661736+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:11.215312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.839899+0000","last_change":"2024-03-28T16:01:08.266122+0000","last_active":"2024-03-28T16:01:40.839899+0000","last_peered":"2024-03-28T16:01:40.839899+0000","last_clean":"2024-03-28T16:01:40.839899+0000","last_became_active":"2024-03-28T16:01:08.265727+0000","last_became_peered":"2024-03-28T16:01:08.265727+0000","last_unstale":"2024-03-28T16:01:40.839899+0000","last_undegraded":"2024-03-28T16:01:40.839899+0000","last_fullsized":"2024-03-28T16:01:40.839899+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:48:38.939777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.666951+0000","last_change":"2024-03-28T16:01:07.251814+0000","last_active":"2024-03-28T16:01:40.666951+0000","last_peered":"2024-03-28T16:01:40.666951+0000","last_clean":"2024-03-28T16:01:40.666951+0000","last_became_active":"2024-03-28T16:01:07.251311+0000","last_became_peered":"2024-03-28T16:01:07.251311+0000","last_unstale":"2024-03-28T16:01:40.666951+0000","last_undegraded":"2024-03-28T16:01:40.666951+0000","last_fullsized":"2024-03-28T16:01:40.666951+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:13:38.344070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.667098+0000","last_change":"2024-03-28T16:01:08.270879+0000","last_active":"2024-03-28T16:01:40.667098+0000","last_peered":"2024-03-28T16:01:40.667098+0000","last_clean":"2024-03-28T16:01:40.667098+0000","last_became_active":"2024-03-28T16:01:08.270513+0000","last_became_peered":"2024-03-28T16:01:08.270513+0000","last_unstale":"2024-03-28T16:01:40.667098+0000","last_undegraded":"2024-03-28T16:01:40.667098+0000","last_fullsized":"2024-03-28T16:01:40.667098+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:53:43.285797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"89'3","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.665900+0000","last_change":"2024-03-28T16:01:07.253595+0000","last_active":"2024-03-28T16:01:40.665900+0000","last_peered":"2024-03-28T16:01:40.665900+0000","last_clean":"2024-03-28T16:01:40.665900+0000","last_became_active":"2024-03-28T16:01:07.253137+0000","last_became_peered":"2024-03-28T16:01:07.253137+0000","last_unstale":"2024-03-28T16:01:40.665900+0000","last_undegraded":"2024-03-28T16:01:40.665900+0000","last_fullsized":"2024-03-28T16:01:40.665900+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:17:35.759791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.838511+0000","last_change":"2024-03-28T16:01:08.261185+0000","last_active":"2024-03-28T16:01:40.838511+0000","last_peered":"2024-03-28T16:01:40.838511+0000","last_clean":"2024-03-28T16:01:40.838511+0000","last_became_active":"2024-03-28T16:01:08.260885+0000","last_became_peered":"2024-03-28T16:01:08.260885+0000","last_unstale":"2024-03-28T16:01:40.838511+0000","last_undegraded":"2024-03-28T16:01:40.838511+0000","last_fullsized":"2024-03-28T16:01:40.838511+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:15:56.640586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"90'26","reported_seq":209,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:28.734728+0000","last_change":"2024-03-28T16:01:07.258655+0000","last_active":"2024-03-28T16:02:28.734728+0000","last_peered":"2024-03-28T16:02:28.734728+0000","last_clean":"2024-03-28T16:02:28.734728+0000","last_became_active":"2024-03-28T16:01:07.258230+0000","last_became_peered":"2024-03-28T16:01:07.258230+0000","last_unstale":"2024-03-28T16:02:28.734728+0000","last_undegraded":"2024-03-28T16:02:28.734728+0000","last_fullsized":"2024-03-28T16:02:28.734728+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:29:36.206061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61896,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":164,"num_read_kb":142,"num_write":26,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.655738+0000","last_change":"2024-03-28T16:01:08.265393+0000","last_active":"2024-03-28T16:01:40.655738+0000","last_peered":"2024-03-28T16:01:40.655738+0000","last_clean":"2024-03-28T16:01:40.655738+0000","last_became_active":"2024-03-28T16:01:08.264560+0000","last_became_peered":"2024-03-28T16:01:08.264560+0000","last_unstale":"2024-03-28T16:01:40.655738+0000","last_undegraded":"2024-03-28T16:01:40.655738+0000","last_fullsized":"2024-03-28T16:01:40.655738+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:43:52.181113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"89'2","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.672781+0000","last_change":"2024-03-28T16:01:07.251428+0000","last_active":"2024-03-28T16:01:40.672781+0000","last_peered":"2024-03-28T16:01:40.672781+0000","last_clean":"2024-03-28T16:01:40.672781+0000","last_became_active":"2024-03-28T16:01:07.251074+0000","last_became_peered":"2024-03-28T16:01:07.251074+0000","last_unstale":"2024-03-28T16:01:40.672781+0000","last_undegraded":"2024-03-28T16:01:40.672781+0000","last_fullsized":"2024-03-28T16:01:40.672781+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:18:41.359338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.920364+0000","last_change":"2024-03-28T16:01:08.268482+0000","last_active":"2024-03-28T16:01:40.920364+0000","last_peered":"2024-03-28T16:01:40.920364+0000","last_clean":"2024-03-28T16:01:40.920364+0000","last_became_active":"2024-03-28T16:01:08.268153+0000","last_became_peered":"2024-03-28T16:01:08.268153+0000","last_unstale":"2024-03-28T16:01:40.920364+0000","last_undegraded":"2024-03-28T16:01:40.920364+0000","last_fullsized":"2024-03-28T16:01:40.920364+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:21:21.874773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.664016+0000","last_change":"2024-03-28T16:01:07.243506+0000","last_active":"2024-03-28T16:01:40.664016+0000","last_peered":"2024-03-28T16:01:40.664016+0000","last_clean":"2024-03-28T16:01:40.664016+0000","last_became_active":"2024-03-28T16:01:07.242793+0000","last_became_peered":"2024-03-28T16:01:07.242793+0000","last_unstale":"2024-03-28T16:01:40.664016+0000","last_undegraded":"2024-03-28T16:01:40.664016+0000","last_fullsized":"2024-03-28T16:01:40.664016+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:31:56.497276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.666759+0000","last_change":"2024-03-28T16:01:08.278848+0000","last_active":"2024-03-28T16:01:40.666759+0000","last_peered":"2024-03-28T16:01:40.666759+0000","last_clean":"2024-03-28T16:01:40.666759+0000","last_became_active":"2024-03-28T16:01:08.278455+0000","last_became_peered":"2024-03-28T16:01:08.278455+0000","last_unstale":"2024-03-28T16:01:40.666759+0000","last_undegraded":"2024-03-28T16:01:40.666759+0000","last_fullsized":"2024-03-28T16:01:40.666759+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:41:17.302538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.655563+0000","last_change":"2024-03-28T16:01:07.255070+0000","last_active":"2024-03-28T16:01:40.655563+0000","last_peered":"2024-03-28T16:01:40.655563+0000","last_clean":"2024-03-28T16:01:40.655563+0000","last_became_active":"2024-03-28T16:01:07.253489+0000","last_became_peered":"2024-03-28T16:01:07.253489+0000","last_unstale":"2024-03-28T16:01:40.655563+0000","last_undegraded":"2024-03-28T16:01:40.655563+0000","last_fullsized":"2024-03-28T16:01:40.655563+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:31:57.501721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.663094+0000","last_change":"2024-03-28T16:01:08.273190+0000","last_active":"2024-03-28T16:01:40.663094+0000","last_peered":"2024-03-28T16:01:40.663094+0000","last_clean":"2024-03-28T16:01:40.663094+0000","last_became_active":"2024-03-28T16:01:08.272870+0000","last_became_peered":"2024-03-28T16:01:08.272870+0000","last_unstale":"2024-03-28T16:01:40.663094+0000","last_undegraded":"2024-03-28T16:01:40.663094+0000","last_fullsized":"2024-03-28T16:01:40.663094+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:26:50.874750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"89'8","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.700937+0000","last_change":"2024-03-28T16:01:40.700157+0000","last_active":"2024-03-28T16:01:40.700937+0000","last_peered":"2024-03-28T16:01:40.700937+0000","last_clean":"2024-03-28T16:01:40.700937+0000","last_became_active":"2024-03-28T16:01:40.674436+0000","last_became_peered":"2024-03-28T16:01:40.674436+0000","last_unstale":"2024-03-28T16:01:40.700937+0000","last_undegraded":"2024-03-28T16:01:40.700937+0000","last_fullsized":"2024-03-28T16:01:40.700937+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:26:48.776506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,4],"acting":[11,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.924563+0000","last_change":"2024-03-28T16:01:40.924563+0000","last_active":"2024-03-28T16:01:40.924563+0000","last_peered":"2024-03-28T16:01:40.924563+0000","last_clean":"2024-03-28T16:01:40.924563+0000","last_became_active":"2024-03-28T16:01:40.923348+0000","last_became_peered":"2024-03-28T16:01:40.923348+0000","last_unstale":"2024-03-28T16:01:40.924563+0000","last_undegraded":"2024-03-28T16:01:40.924563+0000","last_fullsized":"2024-03-28T16:01:40.924563+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:31:46.619651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'1","reported_seq":15,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.681372+0000","last_change":"2024-03-28T16:01:40.680548+0000","last_active":"2024-03-28T16:01:40.681372+0000","last_peered":"2024-03-28T16:01:40.681372+0000","last_clean":"2024-03-28T16:01:40.681372+0000","last_became_active":"2024-03-28T16:01:40.672674+0000","last_became_peered":"2024-03-28T16:01:40.672674+0000","last_unstale":"2024-03-28T16:01:40.681372+0000","last_undegraded":"2024-03-28T16:01:40.681372+0000","last_fullsized":"2024-03-28T16:01:40.681372+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:07:32.767184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.661705+0000","last_change":"2024-03-28T16:01:08.274436+0000","last_active":"2024-03-28T16:01:40.661705+0000","last_peered":"2024-03-28T16:01:40.661705+0000","last_clean":"2024-03-28T16:01:40.661705+0000","last_became_active":"2024-03-28T16:01:08.274081+0000","last_became_peered":"2024-03-28T16:01:08.274081+0000","last_unstale":"2024-03-28T16:01:40.661705+0000","last_undegraded":"2024-03-28T16:01:40.661705+0000","last_fullsized":"2024-03-28T16:01:40.661705+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:57:00.531064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.653093+0000","last_change":"2024-03-28T16:01:07.247542+0000","last_active":"2024-03-28T16:01:40.653093+0000","last_peered":"2024-03-28T16:01:40.653093+0000","last_clean":"2024-03-28T16:01:40.653093+0000","last_became_active":"2024-03-28T16:01:07.247109+0000","last_became_peered":"2024-03-28T16:01:07.247109+0000","last_unstale":"2024-03-28T16:01:40.653093+0000","last_undegraded":"2024-03-28T16:01:40.653093+0000","last_fullsized":"2024-03-28T16:01:40.653093+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:14:19.387896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"90'1","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.662665+0000","last_change":"2024-03-28T16:01:08.264565+0000","last_active":"2024-03-28T16:01:40.662665+0000","last_peered":"2024-03-28T16:01:40.662665+0000","last_clean":"2024-03-28T16:01:40.662665+0000","last_became_active":"2024-03-28T16:01:08.264196+0000","last_became_peered":"2024-03-28T16:01:08.264196+0000","last_unstale":"2024-03-28T16:01:40.662665+0000","last_undegraded":"2024-03-28T16:01:40.662665+0000","last_fullsized":"2024-03-28T16:01:40.662665+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:43:24.750624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.665944+0000","last_change":"2024-03-28T16:01:07.248189+0000","last_active":"2024-03-28T16:01:40.665944+0000","last_peered":"2024-03-28T16:01:40.665944+0000","last_clean":"2024-03-28T16:01:40.665944+0000","last_became_active":"2024-03-28T16:01:07.247295+0000","last_became_peered":"2024-03-28T16:01:07.247295+0000","last_unstale":"2024-03-28T16:01:40.665944+0000","last_undegraded":"2024-03-28T16:01:40.665944+0000","last_fullsized":"2024-03-28T16:01:40.665944+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:46:51.875094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.661465+0000","last_change":"2024-03-28T16:01:08.269724+0000","last_active":"2024-03-28T16:01:40.661465+0000","last_peered":"2024-03-28T16:01:40.661465+0000","last_clean":"2024-03-28T16:01:40.661465+0000","last_became_active":"2024-03-28T16:01:08.269357+0000","last_became_peered":"2024-03-28T16:01:08.269357+0000","last_unstale":"2024-03-28T16:01:40.661465+0000","last_undegraded":"2024-03-28T16:01:40.661465+0000","last_fullsized":"2024-03-28T16:01:40.661465+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:20:12.622238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.664568+0000","last_change":"2024-03-28T16:01:08.266624+0000","last_active":"2024-03-28T16:01:40.664568+0000","last_peered":"2024-03-28T16:01:40.664568+0000","last_clean":"2024-03-28T16:01:40.664568+0000","last_became_active":"2024-03-28T16:01:08.265408+0000","last_became_peered":"2024-03-28T16:01:08.265408+0000","last_unstale":"2024-03-28T16:01:40.664568+0000","last_undegraded":"2024-03-28T16:01:40.664568+0000","last_fullsized":"2024-03-28T16:01:40.664568+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:37:08.831727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.920736+0000","last_change":"2024-03-28T16:01:07.241172+0000","last_active":"2024-03-28T16:01:40.920736+0000","last_peered":"2024-03-28T16:01:40.920736+0000","last_clean":"2024-03-28T16:01:40.920736+0000","last_became_active":"2024-03-28T16:01:07.240754+0000","last_became_peered":"2024-03-28T16:01:07.240754+0000","last_unstale":"2024-03-28T16:01:40.920736+0000","last_undegraded":"2024-03-28T16:01:40.920736+0000","last_fullsized":"2024-03-28T16:01:40.920736+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:52:45.805188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.765231+0000","last_change":"2024-03-28T16:01:08.278429+0000","last_active":"2024-03-28T16:01:40.765231+0000","last_peered":"2024-03-28T16:01:40.765231+0000","last_clean":"2024-03-28T16:01:40.765231+0000","last_became_active":"2024-03-28T16:01:08.275061+0000","last_became_peered":"2024-03-28T16:01:08.275061+0000","last_unstale":"2024-03-28T16:01:40.765231+0000","last_undegraded":"2024-03-28T16:01:40.765231+0000","last_fullsized":"2024-03-28T16:01:40.765231+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:35:42.015979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.655861+0000","last_change":"2024-03-28T16:01:07.252117+0000","last_active":"2024-03-28T16:01:40.655861+0000","last_peered":"2024-03-28T16:01:40.655861+0000","last_clean":"2024-03-28T16:01:40.655861+0000","last_became_active":"2024-03-28T16:01:07.249729+0000","last_became_peered":"2024-03-28T16:01:07.249729+0000","last_unstale":"2024-03-28T16:01:40.655861+0000","last_undegraded":"2024-03-28T16:01:40.655861+0000","last_fullsized":"2024-03-28T16:01:40.655861+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:33:32.719310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":347,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.663525+0000","last_change":"2024-03-28T15:59:32.593730+0000","last_active":"2024-03-28T16:01:40.663525+0000","last_peered":"2024-03-28T16:01:40.663525+0000","last_clean":"2024-03-28T16:01:40.663525+0000","last_became_active":"2024-03-28T15:59:32.592289+0000","last_became_peered":"2024-03-28T15:59:32.592289+0000","last_unstale":"2024-03-28T16:01:40.663525+0000","last_undegraded":"2024-03-28T16:01:40.663525+0000","last_fullsized":"2024-03-28T16:01:40.663525+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:54:55.638193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:54:55.638193+0000","last_clean_scrub_stamp":"2024-03-28T15:54:55.638193+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:20:37.087340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.920353+0000","last_change":"2024-03-28T16:01:08.265513+0000","last_active":"2024-03-28T16:01:40.920353+0000","last_peered":"2024-03-28T16:01:40.920353+0000","last_clean":"2024-03-28T16:01:40.920353+0000","last_became_active":"2024-03-28T16:01:08.264711+0000","last_became_peered":"2024-03-28T16:01:08.264711+0000","last_unstale":"2024-03-28T16:01:40.920353+0000","last_undegraded":"2024-03-28T16:01:40.920353+0000","last_fullsized":"2024-03-28T16:01:40.920353+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:10:01.936780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.664994+0000","last_change":"2024-03-28T16:01:07.252831+0000","last_active":"2024-03-28T16:01:40.664994+0000","last_peered":"2024-03-28T16:01:40.664994+0000","last_clean":"2024-03-28T16:01:40.664994+0000","last_became_active":"2024-03-28T16:01:07.252317+0000","last_became_peered":"2024-03-28T16:01:07.252317+0000","last_unstale":"2024-03-28T16:01:40.664994+0000","last_undegraded":"2024-03-28T16:01:40.664994+0000","last_fullsized":"2024-03-28T16:01:40.664994+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:55:37.461154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.655655+0000","last_change":"2024-03-28T16:01:07.262413+0000","last_active":"2024-03-28T16:01:40.655655+0000","last_peered":"2024-03-28T16:01:40.655655+0000","last_clean":"2024-03-28T16:01:40.655655+0000","last_became_active":"2024-03-28T16:01:07.261875+0000","last_became_peered":"2024-03-28T16:01:07.261875+0000","last_unstale":"2024-03-28T16:01:40.655655+0000","last_undegraded":"2024-03-28T16:01:40.655655+0000","last_fullsized":"2024-03-28T16:01:40.655655+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:25:49.124991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.920743+0000","last_change":"2024-03-28T16:01:08.259506+0000","last_active":"2024-03-28T16:01:40.920743+0000","last_peered":"2024-03-28T16:01:40.920743+0000","last_clean":"2024-03-28T16:01:40.920743+0000","last_became_active":"2024-03-28T16:01:08.258512+0000","last_became_peered":"2024-03-28T16:01:08.258512+0000","last_unstale":"2024-03-28T16:01:40.920743+0000","last_undegraded":"2024-03-28T16:01:40.920743+0000","last_fullsized":"2024-03-28T16:01:40.920743+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:20:17.017763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"90'8","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.653464+0000","last_change":"2024-03-28T16:01:07.254850+0000","last_active":"2024-03-28T16:01:40.653464+0000","last_peered":"2024-03-28T16:01:40.653464+0000","last_clean":"2024-03-28T16:01:40.653464+0000","last_became_active":"2024-03-28T16:01:07.252294+0000","last_became_peered":"2024-03-28T16:01:07.252294+0000","last_unstale":"2024-03-28T16:01:40.653464+0000","last_undegraded":"2024-03-28T16:01:40.653464+0000","last_fullsized":"2024-03-28T16:01:40.653464+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:54:26.700118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.676451+0000","last_change":"2024-03-28T16:01:40.675232+0000","last_active":"2024-03-28T16:01:40.676451+0000","last_peered":"2024-03-28T16:01:40.676451+0000","last_clean":"2024-03-28T16:01:40.676451+0000","last_became_active":"2024-03-28T16:01:40.674454+0000","last_became_peered":"2024-03-28T16:01:40.674454+0000","last_unstale":"2024-03-28T16:01:40.676451+0000","last_undegraded":"2024-03-28T16:01:40.676451+0000","last_fullsized":"2024-03-28T16:01:40.676451+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:47:16.997641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,7],"acting":[9,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.7","version":"89'2","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.657120+0000","last_change":"2024-03-28T16:01:07.245292+0000","last_active":"2024-03-28T16:01:40.657120+0000","last_peered":"2024-03-28T16:01:40.657120+0000","last_clean":"2024-03-28T16:01:40.657120+0000","last_became_active":"2024-03-28T16:01:07.244654+0000","last_became_peered":"2024-03-28T16:01:07.244654+0000","last_unstale":"2024-03-28T16:01:40.657120+0000","last_undegraded":"2024-03-28T16:01:40.657120+0000","last_fullsized":"2024-03-28T16:01:40.657120+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:03:39.027844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.666560+0000","last_change":"2024-03-28T16:01:08.274542+0000","last_active":"2024-03-28T16:01:40.666560+0000","last_peered":"2024-03-28T16:01:40.666560+0000","last_clean":"2024-03-28T16:01:40.666560+0000","last_became_active":"2024-03-28T16:01:08.274202+0000","last_became_peered":"2024-03-28T16:01:08.274202+0000","last_unstale":"2024-03-28T16:01:40.666560+0000","last_undegraded":"2024-03-28T16:01:40.666560+0000","last_fullsized":"2024-03-28T16:01:40.666560+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:09:18.199017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.837891+0000","last_change":"2024-03-28T16:01:07.253274+0000","last_active":"2024-03-28T16:01:40.837891+0000","last_peered":"2024-03-28T16:01:40.837891+0000","last_clean":"2024-03-28T16:01:40.837891+0000","last_became_active":"2024-03-28T16:01:07.250230+0000","last_became_peered":"2024-03-28T16:01:07.250230+0000","last_unstale":"2024-03-28T16:01:40.837891+0000","last_undegraded":"2024-03-28T16:01:40.837891+0000","last_fullsized":"2024-03-28T16:01:40.837891+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:20:31.459983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.654366+0000","last_change":"2024-03-28T16:01:08.266551+0000","last_active":"2024-03-28T16:01:40.654366+0000","last_peered":"2024-03-28T16:01:40.654366+0000","last_clean":"2024-03-28T16:01:40.654366+0000","last_became_active":"2024-03-28T16:01:08.266226+0000","last_became_peered":"2024-03-28T16:01:08.266226+0000","last_unstale":"2024-03-28T16:01:40.654366+0000","last_undegraded":"2024-03-28T16:01:40.654366+0000","last_fullsized":"2024-03-28T16:01:40.654366+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:51:16.520285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.764556+0000","last_change":"2024-03-28T16:01:07.252038+0000","last_active":"2024-03-28T16:01:40.764556+0000","last_peered":"2024-03-28T16:01:40.764556+0000","last_clean":"2024-03-28T16:01:40.764556+0000","last_became_active":"2024-03-28T16:01:07.251703+0000","last_became_peered":"2024-03-28T16:01:07.251703+0000","last_unstale":"2024-03-28T16:01:40.764556+0000","last_undegraded":"2024-03-28T16:01:40.764556+0000","last_fullsized":"2024-03-28T16:01:40.764556+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:35:42.518996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.663843+0000","last_change":"2024-03-28T16:01:08.260025+0000","last_active":"2024-03-28T16:01:40.663843+0000","last_peered":"2024-03-28T16:01:40.663843+0000","last_clean":"2024-03-28T16:01:40.663843+0000","last_became_active":"2024-03-28T16:01:08.259596+0000","last_became_peered":"2024-03-28T16:01:08.259596+0000","last_unstale":"2024-03-28T16:01:40.663843+0000","last_undegraded":"2024-03-28T16:01:40.663843+0000","last_fullsized":"2024-03-28T16:01:40.663843+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:45:29.228188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.667987+0000","last_change":"2024-03-28T16:01:07.259335+0000","last_active":"2024-03-28T16:01:40.667987+0000","last_peered":"2024-03-28T16:01:40.667987+0000","last_clean":"2024-03-28T16:01:40.667987+0000","last_became_active":"2024-03-28T16:01:07.258589+0000","last_became_peered":"2024-03-28T16:01:07.258589+0000","last_unstale":"2024-03-28T16:01:40.667987+0000","last_undegraded":"2024-03-28T16:01:40.667987+0000","last_fullsized":"2024-03-28T16:01:40.667987+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:06:41.742963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.764221+0000","last_change":"2024-03-28T16:01:08.266794+0000","last_active":"2024-03-28T16:01:40.764221+0000","last_peered":"2024-03-28T16:01:40.764221+0000","last_clean":"2024-03-28T16:01:40.764221+0000","last_became_active":"2024-03-28T16:01:08.266502+0000","last_became_peered":"2024-03-28T16:01:08.266502+0000","last_unstale":"2024-03-28T16:01:40.764221+0000","last_undegraded":"2024-03-28T16:01:40.764221+0000","last_fullsized":"2024-03-28T16:01:40.764221+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:17:12.671677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"89'2","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.655500+0000","last_change":"2024-03-28T16:01:07.259239+0000","last_active":"2024-03-28T16:01:40.655500+0000","last_peered":"2024-03-28T16:01:40.655500+0000","last_clean":"2024-03-28T16:01:40.655500+0000","last_became_active":"2024-03-28T16:01:07.258875+0000","last_became_peered":"2024-03-28T16:01:07.258875+0000","last_unstale":"2024-03-28T16:01:40.655500+0000","last_undegraded":"2024-03-28T16:01:40.655500+0000","last_fullsized":"2024-03-28T16:01:40.655500+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:35:12.729173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.666326+0000","last_change":"2024-03-28T16:01:08.278196+0000","last_active":"2024-03-28T16:01:40.666326+0000","last_peered":"2024-03-28T16:01:40.666326+0000","last_clean":"2024-03-28T16:01:40.666326+0000","last_became_active":"2024-03-28T16:01:08.276972+0000","last_became_peered":"2024-03-28T16:01:08.276972+0000","last_unstale":"2024-03-28T16:01:40.666326+0000","last_undegraded":"2024-03-28T16:01:40.666326+0000","last_fullsized":"2024-03-28T16:01:40.666326+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:28:25.231210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.668013+0000","last_change":"2024-03-28T16:01:07.246851+0000","last_active":"2024-03-28T16:01:40.668013+0000","last_peered":"2024-03-28T16:01:40.668013+0000","last_clean":"2024-03-28T16:01:40.668013+0000","last_became_active":"2024-03-28T16:01:07.246365+0000","last_became_peered":"2024-03-28T16:01:07.246365+0000","last_unstale":"2024-03-28T16:01:40.668013+0000","last_undegraded":"2024-03-28T16:01:40.668013+0000","last_fullsized":"2024-03-28T16:01:40.668013+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:16:13.593045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.667751+0000","last_change":"2024-03-28T16:01:08.279468+0000","last_active":"2024-03-28T16:01:40.667751+0000","last_peered":"2024-03-28T16:01:40.667751+0000","last_clean":"2024-03-28T16:01:40.667751+0000","last_became_active":"2024-03-28T16:01:08.274650+0000","last_became_peered":"2024-03-28T16:01:08.274650+0000","last_unstale":"2024-03-28T16:01:40.667751+0000","last_undegraded":"2024-03-28T16:01:40.667751+0000","last_fullsized":"2024-03-28T16:01:40.667751+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:36:32.776085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"89'1","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.664753+0000","last_change":"2024-03-28T16:01:07.258077+0000","last_active":"2024-03-28T16:01:40.664753+0000","last_peered":"2024-03-28T16:01:40.664753+0000","last_clean":"2024-03-28T16:01:40.664753+0000","last_became_active":"2024-03-28T16:01:07.256665+0000","last_became_peered":"2024-03-28T16:01:07.256665+0000","last_unstale":"2024-03-28T16:01:40.664753+0000","last_undegraded":"2024-03-28T16:01:40.664753+0000","last_fullsized":"2024-03-28T16:01:40.664753+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:01:18.852441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"90'1","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.667272+0000","last_change":"2024-03-28T16:01:08.278588+0000","last_active":"2024-03-28T16:01:40.667272+0000","last_peered":"2024-03-28T16:01:40.667272+0000","last_clean":"2024-03-28T16:01:40.667272+0000","last_became_active":"2024-03-28T16:01:08.277303+0000","last_became_peered":"2024-03-28T16:01:08.277303+0000","last_unstale":"2024-03-28T16:01:40.667272+0000","last_undegraded":"2024-03-28T16:01:40.667272+0000","last_fullsized":"2024-03-28T16:01:40.667272+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:01:19.927057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"89'2","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.921153+0000","last_change":"2024-03-28T16:01:07.249382+0000","last_active":"2024-03-28T16:01:40.921153+0000","last_peered":"2024-03-28T16:01:40.921153+0000","last_clean":"2024-03-28T16:01:40.921153+0000","last_became_active":"2024-03-28T16:01:07.249065+0000","last_became_peered":"2024-03-28T16:01:07.249065+0000","last_unstale":"2024-03-28T16:01:40.921153+0000","last_undegraded":"2024-03-28T16:01:40.921153+0000","last_fullsized":"2024-03-28T16:01:40.921153+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:07:22.851862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.667063+0000","last_change":"2024-03-28T16:01:08.274839+0000","last_active":"2024-03-28T16:01:40.667063+0000","last_peered":"2024-03-28T16:01:40.667063+0000","last_clean":"2024-03-28T16:01:40.667063+0000","last_became_active":"2024-03-28T16:01:08.274505+0000","last_became_peered":"2024-03-28T16:01:08.274505+0000","last_unstale":"2024-03-28T16:01:40.667063+0000","last_undegraded":"2024-03-28T16:01:40.667063+0000","last_fullsized":"2024-03-28T16:01:40.667063+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:57:37.635327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.676281+0000","last_change":"2024-03-28T16:01:40.676281+0000","last_active":"2024-03-28T16:01:40.676281+0000","last_peered":"2024-03-28T16:01:40.676281+0000","last_clean":"2024-03-28T16:01:40.676281+0000","last_became_active":"2024-03-28T16:01:40.675734+0000","last_became_peered":"2024-03-28T16:01:40.675734+0000","last_unstale":"2024-03-28T16:01:40.676281+0000","last_undegraded":"2024-03-28T16:01:40.676281+0000","last_fullsized":"2024-03-28T16:01:40.676281+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:23:39.282010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.668431+0000","last_change":"2024-03-28T16:01:08.272816+0000","last_active":"2024-03-28T16:01:40.668431+0000","last_peered":"2024-03-28T16:01:40.668431+0000","last_clean":"2024-03-28T16:01:40.668431+0000","last_became_active":"2024-03-28T16:01:08.270941+0000","last_became_peered":"2024-03-28T16:01:08.270941+0000","last_unstale":"2024-03-28T16:01:40.668431+0000","last_undegraded":"2024-03-28T16:01:40.668431+0000","last_fullsized":"2024-03-28T16:01:40.668431+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:19:00.481809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.667668+0000","last_change":"2024-03-28T16:01:08.258558+0000","last_active":"2024-03-28T16:01:40.667668+0000","last_peered":"2024-03-28T16:01:40.667668+0000","last_clean":"2024-03-28T16:01:40.667668+0000","last_became_active":"2024-03-28T16:01:08.258163+0000","last_became_peered":"2024-03-28T16:01:08.258163+0000","last_unstale":"2024-03-28T16:01:40.667668+0000","last_undegraded":"2024-03-28T16:01:40.667668+0000","last_fullsized":"2024-03-28T16:01:40.667668+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:53:48.358536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.665975+0000","last_change":"2024-03-28T16:01:08.268886+0000","last_active":"2024-03-28T16:01:40.665975+0000","last_peered":"2024-03-28T16:01:40.665975+0000","last_clean":"2024-03-28T16:01:40.665975+0000","last_became_active":"2024-03-28T16:01:08.268519+0000","last_became_peered":"2024-03-28T16:01:08.268519+0000","last_unstale":"2024-03-28T16:01:40.665975+0000","last_undegraded":"2024-03-28T16:01:40.665975+0000","last_fullsized":"2024-03-28T16:01:40.665975+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:49:05.266517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.920852+0000","last_change":"2024-03-28T16:01:08.265039+0000","last_active":"2024-03-28T16:01:40.920852+0000","last_peered":"2024-03-28T16:01:40.920852+0000","last_clean":"2024-03-28T16:01:40.920852+0000","last_became_active":"2024-03-28T16:01:08.264677+0000","last_became_peered":"2024-03-28T16:01:08.264677+0000","last_unstale":"2024-03-28T16:01:40.920852+0000","last_undegraded":"2024-03-28T16:01:40.920852+0000","last_fullsized":"2024-03-28T16:01:40.920852+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:17:27.667324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.661208+0000","last_change":"2024-03-28T16:01:08.271699+0000","last_active":"2024-03-28T16:01:40.661208+0000","last_peered":"2024-03-28T16:01:40.661208+0000","last_clean":"2024-03-28T16:01:40.661208+0000","last_became_active":"2024-03-28T16:01:08.270184+0000","last_became_peered":"2024-03-28T16:01:08.270184+0000","last_unstale":"2024-03-28T16:01:40.661208+0000","last_undegraded":"2024-03-28T16:01:40.661208+0000","last_fullsized":"2024-03-28T16:01:40.661208+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:17:15.699944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.667927+0000","last_change":"2024-03-28T16:01:08.262314+0000","last_active":"2024-03-28T16:01:40.667927+0000","last_peered":"2024-03-28T16:01:40.667927+0000","last_clean":"2024-03-28T16:01:40.667927+0000","last_became_active":"2024-03-28T16:01:08.261992+0000","last_became_peered":"2024-03-28T16:01:08.261992+0000","last_unstale":"2024-03-28T16:01:40.667927+0000","last_undegraded":"2024-03-28T16:01:40.667927+0000","last_fullsized":"2024-03-28T16:01:40.667927+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:40:52.048306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.764605+0000","last_change":"2024-03-28T16:01:08.259407+0000","last_active":"2024-03-28T16:01:40.764605+0000","last_peered":"2024-03-28T16:01:40.764605+0000","last_clean":"2024-03-28T16:01:40.764605+0000","last_became_active":"2024-03-28T16:01:08.259071+0000","last_became_peered":"2024-03-28T16:01:08.259071+0000","last_unstale":"2024-03-28T16:01:40.764605+0000","last_undegraded":"2024-03-28T16:01:40.764605+0000","last_fullsized":"2024-03-28T16:01:40.764605+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:30:17.819868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.663990+0000","last_change":"2024-03-28T16:01:08.271649+0000","last_active":"2024-03-28T16:01:40.663990+0000","last_peered":"2024-03-28T16:01:40.663990+0000","last_clean":"2024-03-28T16:01:40.663990+0000","last_became_active":"2024-03-28T16:01:08.271230+0000","last_became_peered":"2024-03-28T16:01:08.271230+0000","last_unstale":"2024-03-28T16:01:40.663990+0000","last_undegraded":"2024-03-28T16:01:40.663990+0000","last_fullsized":"2024-03-28T16:01:40.663990+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:50:14.404178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.764247+0000","last_change":"2024-03-28T16:01:08.260845+0000","last_active":"2024-03-28T16:01:40.764247+0000","last_peered":"2024-03-28T16:01:40.764247+0000","last_clean":"2024-03-28T16:01:40.764247+0000","last_became_active":"2024-03-28T16:01:08.259997+0000","last_became_peered":"2024-03-28T16:01:08.259997+0000","last_unstale":"2024-03-28T16:01:40.764247+0000","last_undegraded":"2024-03-28T16:01:40.764247+0000","last_fullsized":"2024-03-28T16:01:40.764247+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:08:55.080858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.838139+0000","last_change":"2024-03-28T16:01:08.268881+0000","last_active":"2024-03-28T16:01:40.838139+0000","last_peered":"2024-03-28T16:01:40.838139+0000","last_clean":"2024-03-28T16:01:40.838139+0000","last_became_active":"2024-03-28T16:01:08.268212+0000","last_became_peered":"2024-03-28T16:01:08.268212+0000","last_unstale":"2024-03-28T16:01:40.838139+0000","last_undegraded":"2024-03-28T16:01:40.838139+0000","last_fullsized":"2024-03-28T16:01:40.838139+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:46:22.447763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.920531+0000","last_change":"2024-03-28T16:01:08.265113+0000","last_active":"2024-03-28T16:01:40.920531+0000","last_peered":"2024-03-28T16:01:40.920531+0000","last_clean":"2024-03-28T16:01:40.920531+0000","last_became_active":"2024-03-28T16:01:08.264781+0000","last_became_peered":"2024-03-28T16:01:08.264781+0000","last_unstale":"2024-03-28T16:01:40.920531+0000","last_undegraded":"2024-03-28T16:01:40.920531+0000","last_fullsized":"2024-03-28T16:01:40.920531+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:19:14.720260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.660958+0000","last_change":"2024-03-28T16:01:08.274446+0000","last_active":"2024-03-28T16:01:40.660958+0000","last_peered":"2024-03-28T16:01:40.660958+0000","last_clean":"2024-03-28T16:01:40.660958+0000","last_became_active":"2024-03-28T16:01:08.274027+0000","last_became_peered":"2024-03-28T16:01:08.274027+0000","last_unstale":"2024-03-28T16:01:40.660958+0000","last_undegraded":"2024-03-28T16:01:40.660958+0000","last_fullsized":"2024-03-28T16:01:40.660958+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:33:37.395889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":62694,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":169,"num_read_kb":147,"num_write":79,"num_write_kb":85,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":188082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66,"ondisk_log_size":66,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":85,"seq":365072220192,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":995328,"data_stored":478849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":77,"seq":330712481832,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32812,"kb_used_data":1632,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712340,"statfs":{"total":95995035648,"available":95961436160,"internally_reserved":0,"allocated":1671168,"data_stored":1131402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":71,"seq":304942678062,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":995328,"data_stored":479409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":63,"seq":270582939702,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":479409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":55,"seq":236223201341,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36900,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708252,"statfs":{"total":95995035648,"available":95957250048,"internally_reserved":0,"allocated":1593344,"data_stored":1069894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397571,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36316,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708836,"statfs":{"total":95995035648,"available":95957848064,"internally_reserved":0,"allocated":995328,"data_stored":478944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574948,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1060864,"data_stored":540723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738474,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36316,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708836,"statfs":{"total":95995035648,"available":95957848064,"internally_reserved":0,"allocated":995328,"data_stored":478917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411423,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28056,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717096,"statfs":{"total":95995035648,"available":95966306304,"internally_reserved":0,"allocated":1064960,"data_stored":540757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117079,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36328,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708824,"statfs":{"total":95995035648,"available":95957835776,"internally_reserved":0,"allocated":1007616,"data_stored":479165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855440,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":995328,"data_stored":478849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659210,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36904,"kb_used_data":1564,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708248,"statfs":{"total":95995035648,"available":95957245952,"internally_reserved":0,"allocated":1601536,"data_stored":1069494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":61896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":61930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":62090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T16:02:32.209 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-28T16:02:32.729 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T16:02:32.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:32 smithi150 ceph-mon[34825]: pgmap v408: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:02:32.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:32 smithi150 ceph-mon[34825]: from='client.14946 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T16:02:32.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:32 smithi067 ceph-mon[31707]: pgmap v408: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:02:32.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:32 smithi067 ceph-mon[31707]: from='client.14946 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T16:02:32.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:32 smithi183 ceph-mon[35580]: pgmap v408: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:02:32.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:32 smithi183 ceph-mon[35580]: from='client.14946 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T16:02:34.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:34.670+0000 7fb06d000700 1 -- 172.21.15.67:0/3007663941 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb068069e40 msgr2=0x7fb06806a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:34.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:34.670+0000 7fb06d000700 1 --2- 172.21.15.67:0/3007663941 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb068069e40 0x7fb06806a2a0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fb0540099c0 tx=0x7fb054009cd0 comp rx=0 tx=0).stop 2024-03-28T16:02:34.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:34.670+0000 7fb06d000700 1 -- 172.21.15.67:0/3007663941 shutdown_connections 2024-03-28T16:02:34.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:34.670+0000 7fb06d000700 1 --2- 172.21.15.67:0/3007663941 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb06806a7e0 0x7fb06810f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:34.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:34.670+0000 7fb06d000700 1 --2- 172.21.15.67:0/3007663941 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0681081e0 0x7fb0681085c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:34.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:34.670+0000 7fb06d000700 1 --2- 172.21.15.67:0/3007663941 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb068069e40 0x7fb06806a2a0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:34.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:34.670+0000 7fb06d000700 1 -- 172.21.15.67:0/3007663941 >> 172.21.15.67:0/3007663941 conn(0x7fb068077fc0 msgr2=0x7fb0680783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:02:34.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:34.670+0000 7fb06d000700 1 -- 172.21.15.67:0/3007663941 shutdown_connections 2024-03-28T16:02:34.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:34.670+0000 7fb06d000700 1 -- 172.21.15.67:0/3007663941 wait complete. 2024-03-28T16:02:34.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:34.672+0000 7fb06d000700 1 Processor -- start 2024-03-28T16:02:34.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:34.672+0000 7fb06d000700 1 -- start start 2024-03-28T16:02:34.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:34.672+0000 7fb06d000700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb068069e40 0x7fb0681a0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:34.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:34.672+0000 7fb06d000700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb06806a7e0 0x7fb0681a0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:34.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:34.672+0000 7fb06d000700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb0681081e0 0x7fb06819a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:34.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:34.672+0000 7fb06d000700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb068112340 con 0x7fb06806a7e0 2024-03-28T16:02:34.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:34.672+0000 7fb06d000700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0681121c0 con 0x7fb068069e40 2024-03-28T16:02:34.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:34.672+0000 7fb06d000700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0681124c0 con 0x7fb0681081e0 2024-03-28T16:02:34.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:34.673+0000 7fb065400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb06806a7e0 0x7fb0681a0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:34.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:34.673+0000 7fb065400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb06806a7e0 0x7fb0681a0bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42566/0 (socket says 172.21.15.67:42566) 2024-03-28T16:02:34.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:34.673+0000 7fb065400700 1 -- 172.21.15.67:0/3564944134 learned_addr learned my addr 172.21.15.67:0/3564944134 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:02:34.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:34.673+0000 7fb065e00700 1 --2- 172.21.15.67:0/3564944134 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb068069e40 0x7fb0681a0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:34.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:34.673+0000 7fb066800700 1 --2- 172.21.15.67:0/3564944134 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb0681081e0 0x7fb06819a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:34.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:34.673+0000 7fb065400700 1 -- 172.21.15.67:0/3564944134 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb0681081e0 msgr2=0x7fb06819a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:34.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:34.673+0000 7fb065400700 1 --2- 172.21.15.67:0/3564944134 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb0681081e0 0x7fb06819a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:34.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:34.673+0000 7fb065400700 1 -- 172.21.15.67:0/3564944134 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb068069e40 msgr2=0x7fb0681a0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:34.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:34.673+0000 7fb065400700 1 --2- 172.21.15.67:0/3564944134 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb068069e40 0x7fb0681a0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:34.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:34.673+0000 7fb065400700 1 -- 172.21.15.67:0/3564944134 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb06819afe0 con 0x7fb06806a7e0 2024-03-28T16:02:34.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:34.673+0000 7fb065400700 1 --2- 172.21.15.67:0/3564944134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb06806a7e0 0x7fb0681a0bc0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fb054009990 tx=0x7fb054009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:02:34.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:34.674+0000 7fb05ea00700 1 -- 172.21.15.67:0/3564944134 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb054026070 con 0x7fb06806a7e0 2024-03-28T16:02:34.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:34.674+0000 7fb05ea00700 1 -- 172.21.15.67:0/3564944134 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb054003f50 con 0x7fb06806a7e0 2024-03-28T16:02:34.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:34.674+0000 7fb05ea00700 1 -- 172.21.15.67:0/3564944134 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb05400fbc0 con 0x7fb06806a7e0 2024-03-28T16:02:34.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:34.674+0000 7fb06d000700 1 -- 172.21.15.67:0/3564944134 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb06819b270 con 0x7fb06806a7e0 2024-03-28T16:02:34.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:34.674+0000 7fb06d000700 1 -- 172.21.15.67:0/3564944134 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0681a74e0 con 0x7fb06806a7e0 2024-03-28T16:02:34.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:34.675+0000 7fb06d000700 1 -- 172.21.15.67:0/3564944134 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb068103890 con 0x7fb06806a7e0 2024-03-28T16:02:34.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:34.677+0000 7fb05ea00700 1 -- 172.21.15.67:0/3564944134 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fb054029070 con 0x7fb06806a7e0 2024-03-28T16:02:34.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:34.678+0000 7fb05ea00700 1 --2- 172.21.15.67:0/3564944134 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb04c0a29a0 0x7fb04c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:34.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:34.679+0000 7fb05ea00700 1 -- 172.21.15.67:0/3564944134 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fb0540cfa00 con 0x7fb06806a7e0 2024-03-28T16:02:34.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:34.679+0000 7fb065e00700 1 --2- 172.21.15.67:0/3564944134 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb04c0a29a0 0x7fb04c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:34.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:34.680+0000 7fb065e00700 1 --2- 172.21.15.67:0/3564944134 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb04c0a29a0 0x7fb04c0a4e60 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb058005950 tx=0x7fb05800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:02:34.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:34.682+0000 7fb05ea00700 1 -- 172.21.15.67:0/3564944134 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb054030020 con 0x7fb06806a7e0 2024-03-28T16:02:34.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:34 smithi150 ceph-mon[34825]: pgmap v409: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:02:34.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:34 smithi067 ceph-mon[31707]: pgmap v409: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:02:34.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:34 smithi183 ceph-mon[35580]: pgmap v409: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:02:34.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:34.903+0000 7fb06d000700 1 -- 172.21.15.67:0/3564944134 --> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb0681001f0 con 0x7fb04c0a29a0 2024-03-28T16:02:34.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:34.926+0000 7fb05ea00700 1 -- 172.21.15.67:0/3564944134 <== mgr.14152 v2:172.21.15.67:6800/208726654 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+258651 (secure 0 0 0) 0x7fb0681001f0 con 0x7fb04c0a29a0 2024-03-28T16:02:34.929 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:02:34.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:34.932+0000 7fb06d000700 1 -- 172.21.15.67:0/3564944134 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb04c0a29a0 msgr2=0x7fb04c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:34.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:34.932+0000 7fb06d000700 1 --2- 172.21.15.67:0/3564944134 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb04c0a29a0 0x7fb04c0a4e60 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb058005950 tx=0x7fb05800b410 comp rx=0 tx=0).stop 2024-03-28T16:02:34.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:34.932+0000 7fb06d000700 1 -- 172.21.15.67:0/3564944134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb06806a7e0 msgr2=0x7fb0681a0bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:34.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:34.932+0000 7fb06d000700 1 --2- 172.21.15.67:0/3564944134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb06806a7e0 0x7fb0681a0bc0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fb054009990 tx=0x7fb054009f80 comp rx=0 tx=0).stop 2024-03-28T16:02:34.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:34.932+0000 7fb06d000700 1 -- 172.21.15.67:0/3564944134 shutdown_connections 2024-03-28T16:02:34.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:34.932+0000 7fb06d000700 1 --2- 172.21.15.67:0/3564944134 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb04c0a29a0 0x7fb04c0a4e60 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:34.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:34.932+0000 7fb06d000700 1 --2- 172.21.15.67:0/3564944134 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb0681081e0 0x7fb06819a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:34.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:34.932+0000 7fb06d000700 1 --2- 172.21.15.67:0/3564944134 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb068069e40 0x7fb0681a0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:34.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:34.932+0000 7fb06d000700 1 --2- 172.21.15.67:0/3564944134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb06806a7e0 0x7fb0681a0bc0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:34.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:34.932+0000 7fb06d000700 1 -- 172.21.15.67:0/3564944134 >> 172.21.15.67:0/3564944134 conn(0x7fb068077fc0 msgr2=0x7fb0680ffae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:02:34.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:34.933+0000 7fb06d000700 1 -- 172.21.15.67:0/3564944134 shutdown_connections 2024-03-28T16:02:34.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:34.933+0000 7fb06d000700 1 -- 172.21.15.67:0/3564944134 wait complete. 2024-03-28T16:02:34.948 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-28T16:02:35.701 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":409,"stamp":"2024-03-28T16:02:33.074214+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":653296,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":295,"num_read_kb":256,"num_write":262,"num_write_kb":3025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1180758,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":205,"ondisk_log_size":205,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":383664,"kb_used_data":13644,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124558160,"statfs":{"total":1151940427776,"available":1151547555840,"internally_reserved":0,"allocated":13971456,"data_stored":7705812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":378713402},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005694"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.663698+0000","last_change":"2024-03-28T16:01:08.268196+0000","last_active":"2024-03-28T16:01:40.663698+0000","last_peered":"2024-03-28T16:01:40.663698+0000","last_clean":"2024-03-28T16:01:40.663698+0000","last_became_active":"2024-03-28T16:01:08.266956+0000","last_became_peered":"2024-03-28T16:01:08.266956+0000","last_unstale":"2024-03-28T16:01:40.663698+0000","last_undegraded":"2024-03-28T16:01:40.663698+0000","last_fullsized":"2024-03-28T16:01:40.663698+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:38:44.296129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.666457+0000","last_change":"2024-03-28T16:01:08.264985+0000","last_active":"2024-03-28T16:01:40.666457+0000","last_peered":"2024-03-28T16:01:40.666457+0000","last_clean":"2024-03-28T16:01:40.666457+0000","last_became_active":"2024-03-28T16:01:08.262484+0000","last_became_peered":"2024-03-28T16:01:08.262484+0000","last_unstale":"2024-03-28T16:01:40.666457+0000","last_undegraded":"2024-03-28T16:01:40.666457+0000","last_fullsized":"2024-03-28T16:01:40.666457+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:33:07.538908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.666725+0000","last_change":"2024-03-28T16:01:08.267423+0000","last_active":"2024-03-28T16:01:40.666725+0000","last_peered":"2024-03-28T16:01:40.666725+0000","last_clean":"2024-03-28T16:01:40.666725+0000","last_became_active":"2024-03-28T16:01:08.263665+0000","last_became_peered":"2024-03-28T16:01:08.263665+0000","last_unstale":"2024-03-28T16:01:40.666725+0000","last_undegraded":"2024-03-28T16:01:40.666725+0000","last_fullsized":"2024-03-28T16:01:40.666725+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:09:43.488003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.656204+0000","last_change":"2024-03-28T16:01:08.274978+0000","last_active":"2024-03-28T16:01:40.656204+0000","last_peered":"2024-03-28T16:01:40.656204+0000","last_clean":"2024-03-28T16:01:40.656204+0000","last_became_active":"2024-03-28T16:01:08.274553+0000","last_became_peered":"2024-03-28T16:01:08.274553+0000","last_unstale":"2024-03-28T16:01:40.656204+0000","last_undegraded":"2024-03-28T16:01:40.656204+0000","last_fullsized":"2024-03-28T16:01:40.656204+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:16:03.997334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.654621+0000","last_change":"2024-03-28T16:01:08.262155+0000","last_active":"2024-03-28T16:01:40.654621+0000","last_peered":"2024-03-28T16:01:40.654621+0000","last_clean":"2024-03-28T16:01:40.654621+0000","last_became_active":"2024-03-28T16:01:08.261814+0000","last_became_peered":"2024-03-28T16:01:08.261814+0000","last_unstale":"2024-03-28T16:01:40.654621+0000","last_undegraded":"2024-03-28T16:01:40.654621+0000","last_fullsized":"2024-03-28T16:01:40.654621+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:41:04.242203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.661462+0000","last_change":"2024-03-28T16:01:08.273342+0000","last_active":"2024-03-28T16:01:40.661462+0000","last_peered":"2024-03-28T16:01:40.661462+0000","last_clean":"2024-03-28T16:01:40.661462+0000","last_became_active":"2024-03-28T16:01:08.272606+0000","last_became_peered":"2024-03-28T16:01:08.272606+0000","last_unstale":"2024-03-28T16:01:40.661462+0000","last_undegraded":"2024-03-28T16:01:40.661462+0000","last_fullsized":"2024-03-28T16:01:40.661462+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:44:48.580214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.666700+0000","last_change":"2024-03-28T16:01:08.278060+0000","last_active":"2024-03-28T16:01:40.666700+0000","last_peered":"2024-03-28T16:01:40.666700+0000","last_clean":"2024-03-28T16:01:40.666700+0000","last_became_active":"2024-03-28T16:01:08.277047+0000","last_became_peered":"2024-03-28T16:01:08.277047+0000","last_unstale":"2024-03-28T16:01:40.666700+0000","last_undegraded":"2024-03-28T16:01:40.666700+0000","last_fullsized":"2024-03-28T16:01:40.666700+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:46:30.005708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.764145+0000","last_change":"2024-03-28T16:01:08.273199+0000","last_active":"2024-03-28T16:01:40.764145+0000","last_peered":"2024-03-28T16:01:40.764145+0000","last_clean":"2024-03-28T16:01:40.764145+0000","last_became_active":"2024-03-28T16:01:08.272822+0000","last_became_peered":"2024-03-28T16:01:08.272822+0000","last_unstale":"2024-03-28T16:01:40.764145+0000","last_undegraded":"2024-03-28T16:01:40.764145+0000","last_fullsized":"2024-03-28T16:01:40.764145+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:28:11.495665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.839751+0000","last_change":"2024-03-28T16:01:08.270262+0000","last_active":"2024-03-28T16:01:40.839751+0000","last_peered":"2024-03-28T16:01:40.839751+0000","last_clean":"2024-03-28T16:01:40.839751+0000","last_became_active":"2024-03-28T16:01:08.269959+0000","last_became_peered":"2024-03-28T16:01:08.269959+0000","last_unstale":"2024-03-28T16:01:40.839751+0000","last_undegraded":"2024-03-28T16:01:40.839751+0000","last_fullsized":"2024-03-28T16:01:40.839751+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:49:02.261893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.671104+0000","last_change":"2024-03-28T16:01:40.671104+0000","last_active":"2024-03-28T16:01:40.671104+0000","last_peered":"2024-03-28T16:01:40.671104+0000","last_clean":"2024-03-28T16:01:40.671104+0000","last_became_active":"2024-03-28T16:01:40.670474+0000","last_became_peered":"2024-03-28T16:01:40.670474+0000","last_unstale":"2024-03-28T16:01:40.671104+0000","last_undegraded":"2024-03-28T16:01:40.671104+0000","last_fullsized":"2024-03-28T16:01:40.671104+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:19:41.651703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.667731+0000","last_change":"2024-03-28T16:01:08.274943+0000","last_active":"2024-03-28T16:01:40.667731+0000","last_peered":"2024-03-28T16:01:40.667731+0000","last_clean":"2024-03-28T16:01:40.667731+0000","last_became_active":"2024-03-28T16:01:08.274267+0000","last_became_peered":"2024-03-28T16:01:08.274267+0000","last_unstale":"2024-03-28T16:01:40.667731+0000","last_undegraded":"2024-03-28T16:01:40.667731+0000","last_fullsized":"2024-03-28T16:01:40.667731+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:41:13.979918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.656132+0000","last_change":"2024-03-28T16:01:08.266817+0000","last_active":"2024-03-28T16:01:40.656132+0000","last_peered":"2024-03-28T16:01:40.656132+0000","last_clean":"2024-03-28T16:01:40.656132+0000","last_became_active":"2024-03-28T16:01:08.266470+0000","last_became_peered":"2024-03-28T16:01:08.266470+0000","last_unstale":"2024-03-28T16:01:40.656132+0000","last_undegraded":"2024-03-28T16:01:40.656132+0000","last_fullsized":"2024-03-28T16:01:40.656132+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:15:28.573153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.920646+0000","last_change":"2024-03-28T16:01:08.269896+0000","last_active":"2024-03-28T16:01:40.920646+0000","last_peered":"2024-03-28T16:01:40.920646+0000","last_clean":"2024-03-28T16:01:40.920646+0000","last_became_active":"2024-03-28T16:01:08.269500+0000","last_became_peered":"2024-03-28T16:01:08.269500+0000","last_unstale":"2024-03-28T16:01:40.920646+0000","last_undegraded":"2024-03-28T16:01:40.920646+0000","last_fullsized":"2024-03-28T16:01:40.920646+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:44:38.867590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.665936+0000","last_change":"2024-03-28T16:01:08.267072+0000","last_active":"2024-03-28T16:01:40.665936+0000","last_peered":"2024-03-28T16:01:40.665936+0000","last_clean":"2024-03-28T16:01:40.665936+0000","last_became_active":"2024-03-28T16:01:08.266688+0000","last_became_peered":"2024-03-28T16:01:08.266688+0000","last_unstale":"2024-03-28T16:01:40.665936+0000","last_undegraded":"2024-03-28T16:01:40.665936+0000","last_fullsized":"2024-03-28T16:01:40.665936+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:07:57.702952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.665434+0000","last_change":"2024-03-28T16:01:08.270912+0000","last_active":"2024-03-28T16:01:40.665434+0000","last_peered":"2024-03-28T16:01:40.665434+0000","last_clean":"2024-03-28T16:01:40.665434+0000","last_became_active":"2024-03-28T16:01:08.270567+0000","last_became_peered":"2024-03-28T16:01:08.270567+0000","last_unstale":"2024-03-28T16:01:40.665434+0000","last_undegraded":"2024-03-28T16:01:40.665434+0000","last_fullsized":"2024-03-28T16:01:40.665434+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:19:29.294502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.837738+0000","last_change":"2024-03-28T16:01:08.266811+0000","last_active":"2024-03-28T16:01:40.837738+0000","last_peered":"2024-03-28T16:01:40.837738+0000","last_clean":"2024-03-28T16:01:40.837738+0000","last_became_active":"2024-03-28T16:01:08.266517+0000","last_became_peered":"2024-03-28T16:01:08.266517+0000","last_unstale":"2024-03-28T16:01:40.837738+0000","last_undegraded":"2024-03-28T16:01:40.837738+0000","last_fullsized":"2024-03-28T16:01:40.837738+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:55:05.972683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.663371+0000","last_change":"2024-03-28T16:01:08.263864+0000","last_active":"2024-03-28T16:01:40.663371+0000","last_peered":"2024-03-28T16:01:40.663371+0000","last_clean":"2024-03-28T16:01:40.663371+0000","last_became_active":"2024-03-28T16:01:08.262956+0000","last_became_peered":"2024-03-28T16:01:08.262956+0000","last_unstale":"2024-03-28T16:01:40.663371+0000","last_undegraded":"2024-03-28T16:01:40.663371+0000","last_fullsized":"2024-03-28T16:01:40.663371+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:05:50.504237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.655131+0000","last_change":"2024-03-28T16:01:08.265151+0000","last_active":"2024-03-28T16:01:40.655131+0000","last_peered":"2024-03-28T16:01:40.655131+0000","last_clean":"2024-03-28T16:01:40.655131+0000","last_became_active":"2024-03-28T16:01:08.264861+0000","last_became_peered":"2024-03-28T16:01:08.264861+0000","last_unstale":"2024-03-28T16:01:40.655131+0000","last_undegraded":"2024-03-28T16:01:40.655131+0000","last_fullsized":"2024-03-28T16:01:40.655131+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:35:45.878983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.667927+0000","last_change":"2024-03-28T16:01:08.265993+0000","last_active":"2024-03-28T16:01:40.667927+0000","last_peered":"2024-03-28T16:01:40.667927+0000","last_clean":"2024-03-28T16:01:40.667927+0000","last_became_active":"2024-03-28T16:01:08.265488+0000","last_became_peered":"2024-03-28T16:01:08.265488+0000","last_unstale":"2024-03-28T16:01:40.667927+0000","last_undegraded":"2024-03-28T16:01:40.667927+0000","last_fullsized":"2024-03-28T16:01:40.667927+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:31:55.920705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'4","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.654757+0000","last_change":"2024-03-28T16:01:07.250361+0000","last_active":"2024-03-28T16:01:40.654757+0000","last_peered":"2024-03-28T16:01:40.654757+0000","last_clean":"2024-03-28T16:01:40.654757+0000","last_became_active":"2024-03-28T16:01:07.250004+0000","last_became_peered":"2024-03-28T16:01:07.250004+0000","last_unstale":"2024-03-28T16:01:40.654757+0000","last_undegraded":"2024-03-28T16:01:40.654757+0000","last_fullsized":"2024-03-28T16:01:40.654757+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:10:32.838716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.765149+0000","last_change":"2024-03-28T16:01:08.274436+0000","last_active":"2024-03-28T16:01:40.765149+0000","last_peered":"2024-03-28T16:01:40.765149+0000","last_clean":"2024-03-28T16:01:40.765149+0000","last_became_active":"2024-03-28T16:01:08.274124+0000","last_became_peered":"2024-03-28T16:01:08.274124+0000","last_unstale":"2024-03-28T16:01:40.765149+0000","last_undegraded":"2024-03-28T16:01:40.765149+0000","last_fullsized":"2024-03-28T16:01:40.765149+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:02:34.037686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"89'1","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.765269+0000","last_change":"2024-03-28T16:01:07.248565+0000","last_active":"2024-03-28T16:01:40.765269+0000","last_peered":"2024-03-28T16:01:40.765269+0000","last_clean":"2024-03-28T16:01:40.765269+0000","last_became_active":"2024-03-28T16:01:07.247366+0000","last_became_peered":"2024-03-28T16:01:07.247366+0000","last_unstale":"2024-03-28T16:01:40.765269+0000","last_undegraded":"2024-03-28T16:01:40.765269+0000","last_fullsized":"2024-03-28T16:01:40.765269+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:48:13.146398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.667251+0000","last_change":"2024-03-28T16:01:08.265639+0000","last_active":"2024-03-28T16:01:40.667251+0000","last_peered":"2024-03-28T16:01:40.667251+0000","last_clean":"2024-03-28T16:01:40.667251+0000","last_became_active":"2024-03-28T16:01:08.263488+0000","last_became_peered":"2024-03-28T16:01:08.263488+0000","last_unstale":"2024-03-28T16:01:40.667251+0000","last_undegraded":"2024-03-28T16:01:40.667251+0000","last_fullsized":"2024-03-28T16:01:40.667251+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:15:06.516016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"89'2","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.850755+0000","last_change":"2024-03-28T16:01:40.850755+0000","last_active":"2024-03-28T16:01:40.850755+0000","last_peered":"2024-03-28T16:01:40.850755+0000","last_clean":"2024-03-28T16:01:40.850755+0000","last_became_active":"2024-03-28T16:01:40.842547+0000","last_became_peered":"2024-03-28T16:01:40.842547+0000","last_unstale":"2024-03-28T16:01:40.850755+0000","last_undegraded":"2024-03-28T16:01:40.850755+0000","last_fullsized":"2024-03-28T16:01:40.850755+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:39:27.862525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.838279+0000","last_change":"2024-03-28T16:01:08.271534+0000","last_active":"2024-03-28T16:01:40.838279+0000","last_peered":"2024-03-28T16:01:40.838279+0000","last_clean":"2024-03-28T16:01:40.838279+0000","last_became_active":"2024-03-28T16:01:08.271104+0000","last_became_peered":"2024-03-28T16:01:08.271104+0000","last_unstale":"2024-03-28T16:01:40.838279+0000","last_undegraded":"2024-03-28T16:01:40.838279+0000","last_fullsized":"2024-03-28T16:01:40.838279+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:29:21.332346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"89'2","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.655555+0000","last_change":"2024-03-28T16:01:07.259175+0000","last_active":"2024-03-28T16:01:40.655555+0000","last_peered":"2024-03-28T16:01:40.655555+0000","last_clean":"2024-03-28T16:01:40.655555+0000","last_became_active":"2024-03-28T16:01:07.258151+0000","last_became_peered":"2024-03-28T16:01:07.258151+0000","last_unstale":"2024-03-28T16:01:40.655555+0000","last_undegraded":"2024-03-28T16:01:40.655555+0000","last_fullsized":"2024-03-28T16:01:40.655555+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:41:07.064800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.843211+0000","last_change":"2024-03-28T16:01:40.843211+0000","last_active":"2024-03-28T16:01:40.843211+0000","last_peered":"2024-03-28T16:01:40.843211+0000","last_clean":"2024-03-28T16:01:40.843211+0000","last_became_active":"2024-03-28T16:01:40.842051+0000","last_became_peered":"2024-03-28T16:01:40.842051+0000","last_unstale":"2024-03-28T16:01:40.843211+0000","last_undegraded":"2024-03-28T16:01:40.843211+0000","last_fullsized":"2024-03-28T16:01:40.843211+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:19:26.437169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"89'1","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.664703+0000","last_change":"2024-03-28T16:01:07.248538+0000","last_active":"2024-03-28T16:01:40.664703+0000","last_peered":"2024-03-28T16:01:40.664703+0000","last_clean":"2024-03-28T16:01:40.664703+0000","last_became_active":"2024-03-28T16:01:07.248116+0000","last_became_peered":"2024-03-28T16:01:07.248116+0000","last_unstale":"2024-03-28T16:01:40.664703+0000","last_undegraded":"2024-03-28T16:01:40.664703+0000","last_fullsized":"2024-03-28T16:01:40.664703+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:44:56.071286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.655028+0000","last_change":"2024-03-28T16:01:08.271059+0000","last_active":"2024-03-28T16:01:40.655028+0000","last_peered":"2024-03-28T16:01:40.655028+0000","last_clean":"2024-03-28T16:01:40.655028+0000","last_became_active":"2024-03-28T16:01:08.270682+0000","last_became_peered":"2024-03-28T16:01:08.270682+0000","last_unstale":"2024-03-28T16:01:40.655028+0000","last_undegraded":"2024-03-28T16:01:40.655028+0000","last_fullsized":"2024-03-28T16:01:40.655028+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:54:48.922125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"89'1","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.661404+0000","last_change":"2024-03-28T16:01:07.261157+0000","last_active":"2024-03-28T16:01:40.661404+0000","last_peered":"2024-03-28T16:01:40.661404+0000","last_clean":"2024-03-28T16:01:40.661404+0000","last_became_active":"2024-03-28T16:01:07.260751+0000","last_became_peered":"2024-03-28T16:01:07.260751+0000","last_unstale":"2024-03-28T16:01:40.661404+0000","last_undegraded":"2024-03-28T16:01:40.661404+0000","last_fullsized":"2024-03-28T16:01:40.661404+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:16:49.622168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.664847+0000","last_change":"2024-03-28T16:01:08.265773+0000","last_active":"2024-03-28T16:01:40.664847+0000","last_peered":"2024-03-28T16:01:40.664847+0000","last_clean":"2024-03-28T16:01:40.664847+0000","last_became_active":"2024-03-28T16:01:08.263724+0000","last_became_peered":"2024-03-28T16:01:08.263724+0000","last_unstale":"2024-03-28T16:01:40.664847+0000","last_undegraded":"2024-03-28T16:01:40.664847+0000","last_fullsized":"2024-03-28T16:01:40.664847+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:42:37.338946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.921045+0000","last_change":"2024-03-28T16:01:07.235361+0000","last_active":"2024-03-28T16:01:40.921045+0000","last_peered":"2024-03-28T16:01:40.921045+0000","last_clean":"2024-03-28T16:01:40.921045+0000","last_became_active":"2024-03-28T16:01:07.234969+0000","last_became_peered":"2024-03-28T16:01:07.234969+0000","last_unstale":"2024-03-28T16:01:40.921045+0000","last_undegraded":"2024-03-28T16:01:40.921045+0000","last_fullsized":"2024-03-28T16:01:40.921045+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:50:35.592147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.764023+0000","last_change":"2024-03-28T16:01:08.263878+0000","last_active":"2024-03-28T16:01:40.764023+0000","last_peered":"2024-03-28T16:01:40.764023+0000","last_clean":"2024-03-28T16:01:40.764023+0000","last_became_active":"2024-03-28T16:01:08.263549+0000","last_became_peered":"2024-03-28T16:01:08.263549+0000","last_unstale":"2024-03-28T16:01:40.764023+0000","last_undegraded":"2024-03-28T16:01:40.764023+0000","last_fullsized":"2024-03-28T16:01:40.764023+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:31:37.669215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.663697+0000","last_change":"2024-03-28T16:01:07.255701+0000","last_active":"2024-03-28T16:01:40.663697+0000","last_peered":"2024-03-28T16:01:40.663697+0000","last_clean":"2024-03-28T16:01:40.663697+0000","last_became_active":"2024-03-28T16:01:07.255247+0000","last_became_peered":"2024-03-28T16:01:07.255247+0000","last_unstale":"2024-03-28T16:01:40.663697+0000","last_undegraded":"2024-03-28T16:01:40.663697+0000","last_fullsized":"2024-03-28T16:01:40.663697+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:08:17.858395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.652417+0000","last_change":"2024-03-28T16:01:08.252949+0000","last_active":"2024-03-28T16:01:40.652417+0000","last_peered":"2024-03-28T16:01:40.652417+0000","last_clean":"2024-03-28T16:01:40.652417+0000","last_became_active":"2024-03-28T16:01:08.252471+0000","last_became_peered":"2024-03-28T16:01:08.252471+0000","last_unstale":"2024-03-28T16:01:40.652417+0000","last_undegraded":"2024-03-28T16:01:40.652417+0000","last_fullsized":"2024-03-28T16:01:40.652417+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:21:15.519020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.661736+0000","last_change":"2024-03-28T16:01:07.263973+0000","last_active":"2024-03-28T16:01:40.661736+0000","last_peered":"2024-03-28T16:01:40.661736+0000","last_clean":"2024-03-28T16:01:40.661736+0000","last_became_active":"2024-03-28T16:01:07.263560+0000","last_became_peered":"2024-03-28T16:01:07.263560+0000","last_unstale":"2024-03-28T16:01:40.661736+0000","last_undegraded":"2024-03-28T16:01:40.661736+0000","last_fullsized":"2024-03-28T16:01:40.661736+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:11.215312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.839899+0000","last_change":"2024-03-28T16:01:08.266122+0000","last_active":"2024-03-28T16:01:40.839899+0000","last_peered":"2024-03-28T16:01:40.839899+0000","last_clean":"2024-03-28T16:01:40.839899+0000","last_became_active":"2024-03-28T16:01:08.265727+0000","last_became_peered":"2024-03-28T16:01:08.265727+0000","last_unstale":"2024-03-28T16:01:40.839899+0000","last_undegraded":"2024-03-28T16:01:40.839899+0000","last_fullsized":"2024-03-28T16:01:40.839899+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:48:38.939777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.666951+0000","last_change":"2024-03-28T16:01:07.251814+0000","last_active":"2024-03-28T16:01:40.666951+0000","last_peered":"2024-03-28T16:01:40.666951+0000","last_clean":"2024-03-28T16:01:40.666951+0000","last_became_active":"2024-03-28T16:01:07.251311+0000","last_became_peered":"2024-03-28T16:01:07.251311+0000","last_unstale":"2024-03-28T16:01:40.666951+0000","last_undegraded":"2024-03-28T16:01:40.666951+0000","last_fullsized":"2024-03-28T16:01:40.666951+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:13:38.344070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.667098+0000","last_change":"2024-03-28T16:01:08.270879+0000","last_active":"2024-03-28T16:01:40.667098+0000","last_peered":"2024-03-28T16:01:40.667098+0000","last_clean":"2024-03-28T16:01:40.667098+0000","last_became_active":"2024-03-28T16:01:08.270513+0000","last_became_peered":"2024-03-28T16:01:08.270513+0000","last_unstale":"2024-03-28T16:01:40.667098+0000","last_undegraded":"2024-03-28T16:01:40.667098+0000","last_fullsized":"2024-03-28T16:01:40.667098+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:53:43.285797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"89'3","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.665900+0000","last_change":"2024-03-28T16:01:07.253595+0000","last_active":"2024-03-28T16:01:40.665900+0000","last_peered":"2024-03-28T16:01:40.665900+0000","last_clean":"2024-03-28T16:01:40.665900+0000","last_became_active":"2024-03-28T16:01:07.253137+0000","last_became_peered":"2024-03-28T16:01:07.253137+0000","last_unstale":"2024-03-28T16:01:40.665900+0000","last_undegraded":"2024-03-28T16:01:40.665900+0000","last_fullsized":"2024-03-28T16:01:40.665900+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:17:35.759791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.838511+0000","last_change":"2024-03-28T16:01:08.261185+0000","last_active":"2024-03-28T16:01:40.838511+0000","last_peered":"2024-03-28T16:01:40.838511+0000","last_clean":"2024-03-28T16:01:40.838511+0000","last_became_active":"2024-03-28T16:01:08.260885+0000","last_became_peered":"2024-03-28T16:01:08.260885+0000","last_unstale":"2024-03-28T16:01:40.838511+0000","last_undegraded":"2024-03-28T16:01:40.838511+0000","last_fullsized":"2024-03-28T16:01:40.838511+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:15:56.640586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"90'26","reported_seq":209,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:28.734728+0000","last_change":"2024-03-28T16:01:07.258655+0000","last_active":"2024-03-28T16:02:28.734728+0000","last_peered":"2024-03-28T16:02:28.734728+0000","last_clean":"2024-03-28T16:02:28.734728+0000","last_became_active":"2024-03-28T16:01:07.258230+0000","last_became_peered":"2024-03-28T16:01:07.258230+0000","last_unstale":"2024-03-28T16:02:28.734728+0000","last_undegraded":"2024-03-28T16:02:28.734728+0000","last_fullsized":"2024-03-28T16:02:28.734728+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:29:36.206061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61896,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":164,"num_read_kb":142,"num_write":26,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.655738+0000","last_change":"2024-03-28T16:01:08.265393+0000","last_active":"2024-03-28T16:01:40.655738+0000","last_peered":"2024-03-28T16:01:40.655738+0000","last_clean":"2024-03-28T16:01:40.655738+0000","last_became_active":"2024-03-28T16:01:08.264560+0000","last_became_peered":"2024-03-28T16:01:08.264560+0000","last_unstale":"2024-03-28T16:01:40.655738+0000","last_undegraded":"2024-03-28T16:01:40.655738+0000","last_fullsized":"2024-03-28T16:01:40.655738+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:43:52.181113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"89'2","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.672781+0000","last_change":"2024-03-28T16:01:07.251428+0000","last_active":"2024-03-28T16:01:40.672781+0000","last_peered":"2024-03-28T16:01:40.672781+0000","last_clean":"2024-03-28T16:01:40.672781+0000","last_became_active":"2024-03-28T16:01:07.251074+0000","last_became_peered":"2024-03-28T16:01:07.251074+0000","last_unstale":"2024-03-28T16:01:40.672781+0000","last_undegraded":"2024-03-28T16:01:40.672781+0000","last_fullsized":"2024-03-28T16:01:40.672781+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:18:41.359338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.920364+0000","last_change":"2024-03-28T16:01:08.268482+0000","last_active":"2024-03-28T16:01:40.920364+0000","last_peered":"2024-03-28T16:01:40.920364+0000","last_clean":"2024-03-28T16:01:40.920364+0000","last_became_active":"2024-03-28T16:01:08.268153+0000","last_became_peered":"2024-03-28T16:01:08.268153+0000","last_unstale":"2024-03-28T16:01:40.920364+0000","last_undegraded":"2024-03-28T16:01:40.920364+0000","last_fullsized":"2024-03-28T16:01:40.920364+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:21:21.874773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.664016+0000","last_change":"2024-03-28T16:01:07.243506+0000","last_active":"2024-03-28T16:01:40.664016+0000","last_peered":"2024-03-28T16:01:40.664016+0000","last_clean":"2024-03-28T16:01:40.664016+0000","last_became_active":"2024-03-28T16:01:07.242793+0000","last_became_peered":"2024-03-28T16:01:07.242793+0000","last_unstale":"2024-03-28T16:01:40.664016+0000","last_undegraded":"2024-03-28T16:01:40.664016+0000","last_fullsized":"2024-03-28T16:01:40.664016+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:31:56.497276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.666759+0000","last_change":"2024-03-28T16:01:08.278848+0000","last_active":"2024-03-28T16:01:40.666759+0000","last_peered":"2024-03-28T16:01:40.666759+0000","last_clean":"2024-03-28T16:01:40.666759+0000","last_became_active":"2024-03-28T16:01:08.278455+0000","last_became_peered":"2024-03-28T16:01:08.278455+0000","last_unstale":"2024-03-28T16:01:40.666759+0000","last_undegraded":"2024-03-28T16:01:40.666759+0000","last_fullsized":"2024-03-28T16:01:40.666759+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:41:17.302538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.655563+0000","last_change":"2024-03-28T16:01:07.255070+0000","last_active":"2024-03-28T16:01:40.655563+0000","last_peered":"2024-03-28T16:01:40.655563+0000","last_clean":"2024-03-28T16:01:40.655563+0000","last_became_active":"2024-03-28T16:01:07.253489+0000","last_became_peered":"2024-03-28T16:01:07.253489+0000","last_unstale":"2024-03-28T16:01:40.655563+0000","last_undegraded":"2024-03-28T16:01:40.655563+0000","last_fullsized":"2024-03-28T16:01:40.655563+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:31:57.501721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.663094+0000","last_change":"2024-03-28T16:01:08.273190+0000","last_active":"2024-03-28T16:01:40.663094+0000","last_peered":"2024-03-28T16:01:40.663094+0000","last_clean":"2024-03-28T16:01:40.663094+0000","last_became_active":"2024-03-28T16:01:08.272870+0000","last_became_peered":"2024-03-28T16:01:08.272870+0000","last_unstale":"2024-03-28T16:01:40.663094+0000","last_undegraded":"2024-03-28T16:01:40.663094+0000","last_fullsized":"2024-03-28T16:01:40.663094+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:26:50.874750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"89'8","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.700937+0000","last_change":"2024-03-28T16:01:40.700157+0000","last_active":"2024-03-28T16:01:40.700937+0000","last_peered":"2024-03-28T16:01:40.700937+0000","last_clean":"2024-03-28T16:01:40.700937+0000","last_became_active":"2024-03-28T16:01:40.674436+0000","last_became_peered":"2024-03-28T16:01:40.674436+0000","last_unstale":"2024-03-28T16:01:40.700937+0000","last_undegraded":"2024-03-28T16:01:40.700937+0000","last_fullsized":"2024-03-28T16:01:40.700937+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:26:48.776506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,4],"acting":[11,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.924563+0000","last_change":"2024-03-28T16:01:40.924563+0000","last_active":"2024-03-28T16:01:40.924563+0000","last_peered":"2024-03-28T16:01:40.924563+0000","last_clean":"2024-03-28T16:01:40.924563+0000","last_became_active":"2024-03-28T16:01:40.923348+0000","last_became_peered":"2024-03-28T16:01:40.923348+0000","last_unstale":"2024-03-28T16:01:40.924563+0000","last_undegraded":"2024-03-28T16:01:40.924563+0000","last_fullsized":"2024-03-28T16:01:40.924563+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:31:46.619651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'1","reported_seq":15,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.681372+0000","last_change":"2024-03-28T16:01:40.680548+0000","last_active":"2024-03-28T16:01:40.681372+0000","last_peered":"2024-03-28T16:01:40.681372+0000","last_clean":"2024-03-28T16:01:40.681372+0000","last_became_active":"2024-03-28T16:01:40.672674+0000","last_became_peered":"2024-03-28T16:01:40.672674+0000","last_unstale":"2024-03-28T16:01:40.681372+0000","last_undegraded":"2024-03-28T16:01:40.681372+0000","last_fullsized":"2024-03-28T16:01:40.681372+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:07:32.767184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.661705+0000","last_change":"2024-03-28T16:01:08.274436+0000","last_active":"2024-03-28T16:01:40.661705+0000","last_peered":"2024-03-28T16:01:40.661705+0000","last_clean":"2024-03-28T16:01:40.661705+0000","last_became_active":"2024-03-28T16:01:08.274081+0000","last_became_peered":"2024-03-28T16:01:08.274081+0000","last_unstale":"2024-03-28T16:01:40.661705+0000","last_undegraded":"2024-03-28T16:01:40.661705+0000","last_fullsized":"2024-03-28T16:01:40.661705+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:57:00.531064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.653093+0000","last_change":"2024-03-28T16:01:07.247542+0000","last_active":"2024-03-28T16:01:40.653093+0000","last_peered":"2024-03-28T16:01:40.653093+0000","last_clean":"2024-03-28T16:01:40.653093+0000","last_became_active":"2024-03-28T16:01:07.247109+0000","last_became_peered":"2024-03-28T16:01:07.247109+0000","last_unstale":"2024-03-28T16:01:40.653093+0000","last_undegraded":"2024-03-28T16:01:40.653093+0000","last_fullsized":"2024-03-28T16:01:40.653093+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:14:19.387896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"90'1","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.662665+0000","last_change":"2024-03-28T16:01:08.264565+0000","last_active":"2024-03-28T16:01:40.662665+0000","last_peered":"2024-03-28T16:01:40.662665+0000","last_clean":"2024-03-28T16:01:40.662665+0000","last_became_active":"2024-03-28T16:01:08.264196+0000","last_became_peered":"2024-03-28T16:01:08.264196+0000","last_unstale":"2024-03-28T16:01:40.662665+0000","last_undegraded":"2024-03-28T16:01:40.662665+0000","last_fullsized":"2024-03-28T16:01:40.662665+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:43:24.750624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.665944+0000","last_change":"2024-03-28T16:01:07.248189+0000","last_active":"2024-03-28T16:01:40.665944+0000","last_peered":"2024-03-28T16:01:40.665944+0000","last_clean":"2024-03-28T16:01:40.665944+0000","last_became_active":"2024-03-28T16:01:07.247295+0000","last_became_peered":"2024-03-28T16:01:07.247295+0000","last_unstale":"2024-03-28T16:01:40.665944+0000","last_undegraded":"2024-03-28T16:01:40.665944+0000","last_fullsized":"2024-03-28T16:01:40.665944+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:46:51.875094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.661465+0000","last_change":"2024-03-28T16:01:08.269724+0000","last_active":"2024-03-28T16:01:40.661465+0000","last_peered":"2024-03-28T16:01:40.661465+0000","last_clean":"2024-03-28T16:01:40.661465+0000","last_became_active":"2024-03-28T16:01:08.269357+0000","last_became_peered":"2024-03-28T16:01:08.269357+0000","last_unstale":"2024-03-28T16:01:40.661465+0000","last_undegraded":"2024-03-28T16:01:40.661465+0000","last_fullsized":"2024-03-28T16:01:40.661465+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:20:12.622238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.664568+0000","last_change":"2024-03-28T16:01:08.266624+0000","last_active":"2024-03-28T16:01:40.664568+0000","last_peered":"2024-03-28T16:01:40.664568+0000","last_clean":"2024-03-28T16:01:40.664568+0000","last_became_active":"2024-03-28T16:01:08.265408+0000","last_became_peered":"2024-03-28T16:01:08.265408+0000","last_unstale":"2024-03-28T16:01:40.664568+0000","last_undegraded":"2024-03-28T16:01:40.664568+0000","last_fullsized":"2024-03-28T16:01:40.664568+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:37:08.831727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.920736+0000","last_change":"2024-03-28T16:01:07.241172+0000","last_active":"2024-03-28T16:01:40.920736+0000","last_peered":"2024-03-28T16:01:40.920736+0000","last_clean":"2024-03-28T16:01:40.920736+0000","last_became_active":"2024-03-28T16:01:07.240754+0000","last_became_peered":"2024-03-28T16:01:07.240754+0000","last_unstale":"2024-03-28T16:01:40.920736+0000","last_undegraded":"2024-03-28T16:01:40.920736+0000","last_fullsized":"2024-03-28T16:01:40.920736+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:52:45.805188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.765231+0000","last_change":"2024-03-28T16:01:08.278429+0000","last_active":"2024-03-28T16:01:40.765231+0000","last_peered":"2024-03-28T16:01:40.765231+0000","last_clean":"2024-03-28T16:01:40.765231+0000","last_became_active":"2024-03-28T16:01:08.275061+0000","last_became_peered":"2024-03-28T16:01:08.275061+0000","last_unstale":"2024-03-28T16:01:40.765231+0000","last_undegraded":"2024-03-28T16:01:40.765231+0000","last_fullsized":"2024-03-28T16:01:40.765231+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:35:42.015979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.655861+0000","last_change":"2024-03-28T16:01:07.252117+0000","last_active":"2024-03-28T16:01:40.655861+0000","last_peered":"2024-03-28T16:01:40.655861+0000","last_clean":"2024-03-28T16:01:40.655861+0000","last_became_active":"2024-03-28T16:01:07.249729+0000","last_became_peered":"2024-03-28T16:01:07.249729+0000","last_unstale":"2024-03-28T16:01:40.655861+0000","last_undegraded":"2024-03-28T16:01:40.655861+0000","last_fullsized":"2024-03-28T16:01:40.655861+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:33:32.719310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":347,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.663525+0000","last_change":"2024-03-28T15:59:32.593730+0000","last_active":"2024-03-28T16:01:40.663525+0000","last_peered":"2024-03-28T16:01:40.663525+0000","last_clean":"2024-03-28T16:01:40.663525+0000","last_became_active":"2024-03-28T15:59:32.592289+0000","last_became_peered":"2024-03-28T15:59:32.592289+0000","last_unstale":"2024-03-28T16:01:40.663525+0000","last_undegraded":"2024-03-28T16:01:40.663525+0000","last_fullsized":"2024-03-28T16:01:40.663525+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:54:55.638193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:54:55.638193+0000","last_clean_scrub_stamp":"2024-03-28T15:54:55.638193+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:20:37.087340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.920353+0000","last_change":"2024-03-28T16:01:08.265513+0000","last_active":"2024-03-28T16:01:40.920353+0000","last_peered":"2024-03-28T16:01:40.920353+0000","last_clean":"2024-03-28T16:01:40.920353+0000","last_became_active":"2024-03-28T16:01:08.264711+0000","last_became_peered":"2024-03-28T16:01:08.264711+0000","last_unstale":"2024-03-28T16:01:40.920353+0000","last_undegraded":"2024-03-28T16:01:40.920353+0000","last_fullsized":"2024-03-28T16:01:40.920353+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:10:01.936780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.664994+0000","last_change":"2024-03-28T16:01:07.252831+0000","last_active":"2024-03-28T16:01:40.664994+0000","last_peered":"2024-03-28T16:01:40.664994+0000","last_clean":"2024-03-28T16:01:40.664994+0000","last_became_active":"2024-03-28T16:01:07.252317+0000","last_became_peered":"2024-03-28T16:01:07.252317+0000","last_unstale":"2024-03-28T16:01:40.664994+0000","last_undegraded":"2024-03-28T16:01:40.664994+0000","last_fullsized":"2024-03-28T16:01:40.664994+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:55:37.461154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.655655+0000","last_change":"2024-03-28T16:01:07.262413+0000","last_active":"2024-03-28T16:01:40.655655+0000","last_peered":"2024-03-28T16:01:40.655655+0000","last_clean":"2024-03-28T16:01:40.655655+0000","last_became_active":"2024-03-28T16:01:07.261875+0000","last_became_peered":"2024-03-28T16:01:07.261875+0000","last_unstale":"2024-03-28T16:01:40.655655+0000","last_undegraded":"2024-03-28T16:01:40.655655+0000","last_fullsized":"2024-03-28T16:01:40.655655+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:25:49.124991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.920743+0000","last_change":"2024-03-28T16:01:08.259506+0000","last_active":"2024-03-28T16:01:40.920743+0000","last_peered":"2024-03-28T16:01:40.920743+0000","last_clean":"2024-03-28T16:01:40.920743+0000","last_became_active":"2024-03-28T16:01:08.258512+0000","last_became_peered":"2024-03-28T16:01:08.258512+0000","last_unstale":"2024-03-28T16:01:40.920743+0000","last_undegraded":"2024-03-28T16:01:40.920743+0000","last_fullsized":"2024-03-28T16:01:40.920743+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:20:17.017763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"90'8","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.653464+0000","last_change":"2024-03-28T16:01:07.254850+0000","last_active":"2024-03-28T16:01:40.653464+0000","last_peered":"2024-03-28T16:01:40.653464+0000","last_clean":"2024-03-28T16:01:40.653464+0000","last_became_active":"2024-03-28T16:01:07.252294+0000","last_became_peered":"2024-03-28T16:01:07.252294+0000","last_unstale":"2024-03-28T16:01:40.653464+0000","last_undegraded":"2024-03-28T16:01:40.653464+0000","last_fullsized":"2024-03-28T16:01:40.653464+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:54:26.700118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.676451+0000","last_change":"2024-03-28T16:01:40.675232+0000","last_active":"2024-03-28T16:01:40.676451+0000","last_peered":"2024-03-28T16:01:40.676451+0000","last_clean":"2024-03-28T16:01:40.676451+0000","last_became_active":"2024-03-28T16:01:40.674454+0000","last_became_peered":"2024-03-28T16:01:40.674454+0000","last_unstale":"2024-03-28T16:01:40.676451+0000","last_undegraded":"2024-03-28T16:01:40.676451+0000","last_fullsized":"2024-03-28T16:01:40.676451+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:47:16.997641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,7],"acting":[9,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.7","version":"89'2","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.657120+0000","last_change":"2024-03-28T16:01:07.245292+0000","last_active":"2024-03-28T16:01:40.657120+0000","last_peered":"2024-03-28T16:01:40.657120+0000","last_clean":"2024-03-28T16:01:40.657120+0000","last_became_active":"2024-03-28T16:01:07.244654+0000","last_became_peered":"2024-03-28T16:01:07.244654+0000","last_unstale":"2024-03-28T16:01:40.657120+0000","last_undegraded":"2024-03-28T16:01:40.657120+0000","last_fullsized":"2024-03-28T16:01:40.657120+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:03:39.027844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.666560+0000","last_change":"2024-03-28T16:01:08.274542+0000","last_active":"2024-03-28T16:01:40.666560+0000","last_peered":"2024-03-28T16:01:40.666560+0000","last_clean":"2024-03-28T16:01:40.666560+0000","last_became_active":"2024-03-28T16:01:08.274202+0000","last_became_peered":"2024-03-28T16:01:08.274202+0000","last_unstale":"2024-03-28T16:01:40.666560+0000","last_undegraded":"2024-03-28T16:01:40.666560+0000","last_fullsized":"2024-03-28T16:01:40.666560+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:09:18.199017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.837891+0000","last_change":"2024-03-28T16:01:07.253274+0000","last_active":"2024-03-28T16:01:40.837891+0000","last_peered":"2024-03-28T16:01:40.837891+0000","last_clean":"2024-03-28T16:01:40.837891+0000","last_became_active":"2024-03-28T16:01:07.250230+0000","last_became_peered":"2024-03-28T16:01:07.250230+0000","last_unstale":"2024-03-28T16:01:40.837891+0000","last_undegraded":"2024-03-28T16:01:40.837891+0000","last_fullsized":"2024-03-28T16:01:40.837891+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:20:31.459983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.654366+0000","last_change":"2024-03-28T16:01:08.266551+0000","last_active":"2024-03-28T16:01:40.654366+0000","last_peered":"2024-03-28T16:01:40.654366+0000","last_clean":"2024-03-28T16:01:40.654366+0000","last_became_active":"2024-03-28T16:01:08.266226+0000","last_became_peered":"2024-03-28T16:01:08.266226+0000","last_unstale":"2024-03-28T16:01:40.654366+0000","last_undegraded":"2024-03-28T16:01:40.654366+0000","last_fullsized":"2024-03-28T16:01:40.654366+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:51:16.520285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.764556+0000","last_change":"2024-03-28T16:01:07.252038+0000","last_active":"2024-03-28T16:01:40.764556+0000","last_peered":"2024-03-28T16:01:40.764556+0000","last_clean":"2024-03-28T16:01:40.764556+0000","last_became_active":"2024-03-28T16:01:07.251703+0000","last_became_peered":"2024-03-28T16:01:07.251703+0000","last_unstale":"2024-03-28T16:01:40.764556+0000","last_undegraded":"2024-03-28T16:01:40.764556+0000","last_fullsized":"2024-03-28T16:01:40.764556+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:35:42.518996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.663843+0000","last_change":"2024-03-28T16:01:08.260025+0000","last_active":"2024-03-28T16:01:40.663843+0000","last_peered":"2024-03-28T16:01:40.663843+0000","last_clean":"2024-03-28T16:01:40.663843+0000","last_became_active":"2024-03-28T16:01:08.259596+0000","last_became_peered":"2024-03-28T16:01:08.259596+0000","last_unstale":"2024-03-28T16:01:40.663843+0000","last_undegraded":"2024-03-28T16:01:40.663843+0000","last_fullsized":"2024-03-28T16:01:40.663843+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:45:29.228188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.667987+0000","last_change":"2024-03-28T16:01:07.259335+0000","last_active":"2024-03-28T16:01:40.667987+0000","last_peered":"2024-03-28T16:01:40.667987+0000","last_clean":"2024-03-28T16:01:40.667987+0000","last_became_active":"2024-03-28T16:01:07.258589+0000","last_became_peered":"2024-03-28T16:01:07.258589+0000","last_unstale":"2024-03-28T16:01:40.667987+0000","last_undegraded":"2024-03-28T16:01:40.667987+0000","last_fullsized":"2024-03-28T16:01:40.667987+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:06:41.742963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.764221+0000","last_change":"2024-03-28T16:01:08.266794+0000","last_active":"2024-03-28T16:01:40.764221+0000","last_peered":"2024-03-28T16:01:40.764221+0000","last_clean":"2024-03-28T16:01:40.764221+0000","last_became_active":"2024-03-28T16:01:08.266502+0000","last_became_peered":"2024-03-28T16:01:08.266502+0000","last_unstale":"2024-03-28T16:01:40.764221+0000","last_undegraded":"2024-03-28T16:01:40.764221+0000","last_fullsized":"2024-03-28T16:01:40.764221+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:17:12.671677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"89'2","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.655500+0000","last_change":"2024-03-28T16:01:07.259239+0000","last_active":"2024-03-28T16:01:40.655500+0000","last_peered":"2024-03-28T16:01:40.655500+0000","last_clean":"2024-03-28T16:01:40.655500+0000","last_became_active":"2024-03-28T16:01:07.258875+0000","last_became_peered":"2024-03-28T16:01:07.258875+0000","last_unstale":"2024-03-28T16:01:40.655500+0000","last_undegraded":"2024-03-28T16:01:40.655500+0000","last_fullsized":"2024-03-28T16:01:40.655500+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:35:12.729173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.666326+0000","last_change":"2024-03-28T16:01:08.278196+0000","last_active":"2024-03-28T16:01:40.666326+0000","last_peered":"2024-03-28T16:01:40.666326+0000","last_clean":"2024-03-28T16:01:40.666326+0000","last_became_active":"2024-03-28T16:01:08.276972+0000","last_became_peered":"2024-03-28T16:01:08.276972+0000","last_unstale":"2024-03-28T16:01:40.666326+0000","last_undegraded":"2024-03-28T16:01:40.666326+0000","last_fullsized":"2024-03-28T16:01:40.666326+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:28:25.231210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.668013+0000","last_change":"2024-03-28T16:01:07.246851+0000","last_active":"2024-03-28T16:01:40.668013+0000","last_peered":"2024-03-28T16:01:40.668013+0000","last_clean":"2024-03-28T16:01:40.668013+0000","last_became_active":"2024-03-28T16:01:07.246365+0000","last_became_peered":"2024-03-28T16:01:07.246365+0000","last_unstale":"2024-03-28T16:01:40.668013+0000","last_undegraded":"2024-03-28T16:01:40.668013+0000","last_fullsized":"2024-03-28T16:01:40.668013+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:16:13.593045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.667751+0000","last_change":"2024-03-28T16:01:08.279468+0000","last_active":"2024-03-28T16:01:40.667751+0000","last_peered":"2024-03-28T16:01:40.667751+0000","last_clean":"2024-03-28T16:01:40.667751+0000","last_became_active":"2024-03-28T16:01:08.274650+0000","last_became_peered":"2024-03-28T16:01:08.274650+0000","last_unstale":"2024-03-28T16:01:40.667751+0000","last_undegraded":"2024-03-28T16:01:40.667751+0000","last_fullsized":"2024-03-28T16:01:40.667751+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:36:32.776085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"89'1","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.664753+0000","last_change":"2024-03-28T16:01:07.258077+0000","last_active":"2024-03-28T16:01:40.664753+0000","last_peered":"2024-03-28T16:01:40.664753+0000","last_clean":"2024-03-28T16:01:40.664753+0000","last_became_active":"2024-03-28T16:01:07.256665+0000","last_became_peered":"2024-03-28T16:01:07.256665+0000","last_unstale":"2024-03-28T16:01:40.664753+0000","last_undegraded":"2024-03-28T16:01:40.664753+0000","last_fullsized":"2024-03-28T16:01:40.664753+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:01:18.852441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"90'1","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.667272+0000","last_change":"2024-03-28T16:01:08.278588+0000","last_active":"2024-03-28T16:01:40.667272+0000","last_peered":"2024-03-28T16:01:40.667272+0000","last_clean":"2024-03-28T16:01:40.667272+0000","last_became_active":"2024-03-28T16:01:08.277303+0000","last_became_peered":"2024-03-28T16:01:08.277303+0000","last_unstale":"2024-03-28T16:01:40.667272+0000","last_undegraded":"2024-03-28T16:01:40.667272+0000","last_fullsized":"2024-03-28T16:01:40.667272+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:01:19.927057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"89'2","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.921153+0000","last_change":"2024-03-28T16:01:07.249382+0000","last_active":"2024-03-28T16:01:40.921153+0000","last_peered":"2024-03-28T16:01:40.921153+0000","last_clean":"2024-03-28T16:01:40.921153+0000","last_became_active":"2024-03-28T16:01:07.249065+0000","last_became_peered":"2024-03-28T16:01:07.249065+0000","last_unstale":"2024-03-28T16:01:40.921153+0000","last_undegraded":"2024-03-28T16:01:40.921153+0000","last_fullsized":"2024-03-28T16:01:40.921153+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:07:22.851862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.667063+0000","last_change":"2024-03-28T16:01:08.274839+0000","last_active":"2024-03-28T16:01:40.667063+0000","last_peered":"2024-03-28T16:01:40.667063+0000","last_clean":"2024-03-28T16:01:40.667063+0000","last_became_active":"2024-03-28T16:01:08.274505+0000","last_became_peered":"2024-03-28T16:01:08.274505+0000","last_unstale":"2024-03-28T16:01:40.667063+0000","last_undegraded":"2024-03-28T16:01:40.667063+0000","last_fullsized":"2024-03-28T16:01:40.667063+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:57:37.635327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.676281+0000","last_change":"2024-03-28T16:01:40.676281+0000","last_active":"2024-03-28T16:01:40.676281+0000","last_peered":"2024-03-28T16:01:40.676281+0000","last_clean":"2024-03-28T16:01:40.676281+0000","last_became_active":"2024-03-28T16:01:40.675734+0000","last_became_peered":"2024-03-28T16:01:40.675734+0000","last_unstale":"2024-03-28T16:01:40.676281+0000","last_undegraded":"2024-03-28T16:01:40.676281+0000","last_fullsized":"2024-03-28T16:01:40.676281+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:23:39.282010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.668431+0000","last_change":"2024-03-28T16:01:08.272816+0000","last_active":"2024-03-28T16:01:40.668431+0000","last_peered":"2024-03-28T16:01:40.668431+0000","last_clean":"2024-03-28T16:01:40.668431+0000","last_became_active":"2024-03-28T16:01:08.270941+0000","last_became_peered":"2024-03-28T16:01:08.270941+0000","last_unstale":"2024-03-28T16:01:40.668431+0000","last_undegraded":"2024-03-28T16:01:40.668431+0000","last_fullsized":"2024-03-28T16:01:40.668431+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:19:00.481809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.667668+0000","last_change":"2024-03-28T16:01:08.258558+0000","last_active":"2024-03-28T16:01:40.667668+0000","last_peered":"2024-03-28T16:01:40.667668+0000","last_clean":"2024-03-28T16:01:40.667668+0000","last_became_active":"2024-03-28T16:01:08.258163+0000","last_became_peered":"2024-03-28T16:01:08.258163+0000","last_unstale":"2024-03-28T16:01:40.667668+0000","last_undegraded":"2024-03-28T16:01:40.667668+0000","last_fullsized":"2024-03-28T16:01:40.667668+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:53:48.358536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.665975+0000","last_change":"2024-03-28T16:01:08.268886+0000","last_active":"2024-03-28T16:01:40.665975+0000","last_peered":"2024-03-28T16:01:40.665975+0000","last_clean":"2024-03-28T16:01:40.665975+0000","last_became_active":"2024-03-28T16:01:08.268519+0000","last_became_peered":"2024-03-28T16:01:08.268519+0000","last_unstale":"2024-03-28T16:01:40.665975+0000","last_undegraded":"2024-03-28T16:01:40.665975+0000","last_fullsized":"2024-03-28T16:01:40.665975+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:49:05.266517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.920852+0000","last_change":"2024-03-28T16:01:08.265039+0000","last_active":"2024-03-28T16:01:40.920852+0000","last_peered":"2024-03-28T16:01:40.920852+0000","last_clean":"2024-03-28T16:01:40.920852+0000","last_became_active":"2024-03-28T16:01:08.264677+0000","last_became_peered":"2024-03-28T16:01:08.264677+0000","last_unstale":"2024-03-28T16:01:40.920852+0000","last_undegraded":"2024-03-28T16:01:40.920852+0000","last_fullsized":"2024-03-28T16:01:40.920852+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:17:27.667324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.661208+0000","last_change":"2024-03-28T16:01:08.271699+0000","last_active":"2024-03-28T16:01:40.661208+0000","last_peered":"2024-03-28T16:01:40.661208+0000","last_clean":"2024-03-28T16:01:40.661208+0000","last_became_active":"2024-03-28T16:01:08.270184+0000","last_became_peered":"2024-03-28T16:01:08.270184+0000","last_unstale":"2024-03-28T16:01:40.661208+0000","last_undegraded":"2024-03-28T16:01:40.661208+0000","last_fullsized":"2024-03-28T16:01:40.661208+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:17:15.699944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.667927+0000","last_change":"2024-03-28T16:01:08.262314+0000","last_active":"2024-03-28T16:01:40.667927+0000","last_peered":"2024-03-28T16:01:40.667927+0000","last_clean":"2024-03-28T16:01:40.667927+0000","last_became_active":"2024-03-28T16:01:08.261992+0000","last_became_peered":"2024-03-28T16:01:08.261992+0000","last_unstale":"2024-03-28T16:01:40.667927+0000","last_undegraded":"2024-03-28T16:01:40.667927+0000","last_fullsized":"2024-03-28T16:01:40.667927+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:40:52.048306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.764605+0000","last_change":"2024-03-28T16:01:08.259407+0000","last_active":"2024-03-28T16:01:40.764605+0000","last_peered":"2024-03-28T16:01:40.764605+0000","last_clean":"2024-03-28T16:01:40.764605+0000","last_became_active":"2024-03-28T16:01:08.259071+0000","last_became_peered":"2024-03-28T16:01:08.259071+0000","last_unstale":"2024-03-28T16:01:40.764605+0000","last_undegraded":"2024-03-28T16:01:40.764605+0000","last_fullsized":"2024-03-28T16:01:40.764605+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:30:17.819868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.663990+0000","last_change":"2024-03-28T16:01:08.271649+0000","last_active":"2024-03-28T16:01:40.663990+0000","last_peered":"2024-03-28T16:01:40.663990+0000","last_clean":"2024-03-28T16:01:40.663990+0000","last_became_active":"2024-03-28T16:01:08.271230+0000","last_became_peered":"2024-03-28T16:01:08.271230+0000","last_unstale":"2024-03-28T16:01:40.663990+0000","last_undegraded":"2024-03-28T16:01:40.663990+0000","last_fullsized":"2024-03-28T16:01:40.663990+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:50:14.404178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.764247+0000","last_change":"2024-03-28T16:01:08.260845+0000","last_active":"2024-03-28T16:01:40.764247+0000","last_peered":"2024-03-28T16:01:40.764247+0000","last_clean":"2024-03-28T16:01:40.764247+0000","last_became_active":"2024-03-28T16:01:08.259997+0000","last_became_peered":"2024-03-28T16:01:08.259997+0000","last_unstale":"2024-03-28T16:01:40.764247+0000","last_undegraded":"2024-03-28T16:01:40.764247+0000","last_fullsized":"2024-03-28T16:01:40.764247+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:08:55.080858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.838139+0000","last_change":"2024-03-28T16:01:08.268881+0000","last_active":"2024-03-28T16:01:40.838139+0000","last_peered":"2024-03-28T16:01:40.838139+0000","last_clean":"2024-03-28T16:01:40.838139+0000","last_became_active":"2024-03-28T16:01:08.268212+0000","last_became_peered":"2024-03-28T16:01:08.268212+0000","last_unstale":"2024-03-28T16:01:40.838139+0000","last_undegraded":"2024-03-28T16:01:40.838139+0000","last_fullsized":"2024-03-28T16:01:40.838139+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:46:22.447763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.920531+0000","last_change":"2024-03-28T16:01:08.265113+0000","last_active":"2024-03-28T16:01:40.920531+0000","last_peered":"2024-03-28T16:01:40.920531+0000","last_clean":"2024-03-28T16:01:40.920531+0000","last_became_active":"2024-03-28T16:01:08.264781+0000","last_became_peered":"2024-03-28T16:01:08.264781+0000","last_unstale":"2024-03-28T16:01:40.920531+0000","last_undegraded":"2024-03-28T16:01:40.920531+0000","last_fullsized":"2024-03-28T16:01:40.920531+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:19:14.720260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.660958+0000","last_change":"2024-03-28T16:01:08.274446+0000","last_active":"2024-03-28T16:01:40.660958+0000","last_peered":"2024-03-28T16:01:40.660958+0000","last_clean":"2024-03-28T16:01:40.660958+0000","last_became_active":"2024-03-28T16:01:08.274027+0000","last_became_peered":"2024-03-28T16:01:08.274027+0000","last_unstale":"2024-03-28T16:01:40.660958+0000","last_undegraded":"2024-03-28T16:01:40.660958+0000","last_fullsized":"2024-03-28T16:01:40.660958+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:33:37.395889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":62694,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":169,"num_read_kb":147,"num_write":79,"num_write_kb":85,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":188082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66,"ondisk_log_size":66,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":85,"seq":365072220193,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":995328,"data_stored":478849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":77,"seq":330712481832,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32812,"kb_used_data":1632,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712340,"statfs":{"total":95995035648,"available":95961436160,"internally_reserved":0,"allocated":1671168,"data_stored":1131402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":71,"seq":304942678063,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":995328,"data_stored":479409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":63,"seq":270582939702,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":479409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":55,"seq":236223201341,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36900,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708252,"statfs":{"total":95995035648,"available":95957250048,"internally_reserved":0,"allocated":1593344,"data_stored":1069894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397572,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36316,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708836,"statfs":{"total":95995035648,"available":95957848064,"internally_reserved":0,"allocated":995328,"data_stored":478944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574949,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1060864,"data_stored":540723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738474,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36316,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708836,"statfs":{"total":95995035648,"available":95957848064,"internally_reserved":0,"allocated":995328,"data_stored":478917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411423,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28056,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717096,"statfs":{"total":95995035648,"available":95966306304,"internally_reserved":0,"allocated":1064960,"data_stored":540757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117079,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36328,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708824,"statfs":{"total":95995035648,"available":95957835776,"internally_reserved":0,"allocated":1007616,"data_stored":479165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855440,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":995328,"data_stored":478849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659211,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36904,"kb_used_data":1564,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708248,"statfs":{"total":95995035648,"available":95957245952,"internally_reserved":0,"allocated":1601536,"data_stored":1069494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":61896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":61930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":62090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T16:02:35.705 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-28T16:02:35.705 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-28T16:02:35.705 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-28T16:02:35.705 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph health --format=json 2024-03-28T16:02:35.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:35 smithi067 ceph-mon[31707]: from='client.14952 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T16:02:35.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:35 smithi183 ceph-mon[35580]: from='client.14952 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T16:02:36.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:35 smithi150 ceph-mon[34825]: from='client.14952 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T16:02:36.225 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T16:02:36.579 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:36 smithi067 ceph-mon[31707]: pgmap v410: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:02:36.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:36 smithi183 ceph-mon[35580]: pgmap v410: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:02:37.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:36 smithi150 ceph-mon[34825]: pgmap v410: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:02:38.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:38.208+0000 7fe4e8000700 1 -- 172.21.15.67:0/2629072333 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4e0074900 msgr2=0x7fe4e0074d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:38.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:38.208+0000 7fe4e8000700 1 --2- 172.21.15.67:0/2629072333 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4e0074900 0x7fe4e0074d80 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fe4d0009990 tx=0x7fe4d0009ca0 comp rx=0 tx=0).stop 2024-03-28T16:02:38.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:38.208+0000 7fe4e8000700 1 -- 172.21.15.67:0/2629072333 shutdown_connections 2024-03-28T16:02:38.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:38.208+0000 7fe4e8000700 1 --2- 172.21.15.67:0/2629072333 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4e0112e50 0x7fe4e0115240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:38.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:38.208+0000 7fe4e8000700 1 --2- 172.21.15.67:0/2629072333 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe4e00761a0 0x7fe4e00743c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:38.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:38.208+0000 7fe4e8000700 1 --2- 172.21.15.67:0/2629072333 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4e0074900 0x7fe4e0074d80 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:38.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:38.208+0000 7fe4e8000700 1 -- 172.21.15.67:0/2629072333 >> 172.21.15.67:0/2629072333 conn(0x7fe4e0079220 msgr2=0x7fe4e0079630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:02:38.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:38.208+0000 7fe4e8000700 1 -- 172.21.15.67:0/2629072333 shutdown_connections 2024-03-28T16:02:38.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:38.208+0000 7fe4e8000700 1 -- 172.21.15.67:0/2629072333 wait complete. 2024-03-28T16:02:38.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:38.209+0000 7fe4e8000700 1 Processor -- start 2024-03-28T16:02:38.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:38.209+0000 7fe4e8000700 1 -- start start 2024-03-28T16:02:38.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:38.210+0000 7fe4e8000700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4e0074900 0x7fe4e019ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:38.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:38.210+0000 7fe4e8000700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4e00761a0 0x7fe4e019f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:38.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:38.210+0000 7fe4e8000700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe4e0112e50 0x7fe4e01a3730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:38.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:38.210+0000 7fe4e8000700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e0118990 con 0x7fe4e00761a0 2024-03-28T16:02:38.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:38.210+0000 7fe4e8000700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e0118810 con 0x7fe4e0074900 2024-03-28T16:02:38.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:38.210+0000 7fe4e8000700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e0118b10 con 0x7fe4e0112e50 2024-03-28T16:02:38.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:38.210+0000 7fe4e4c00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4e00761a0 0x7fe4e019f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:38.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:38.210+0000 7fe4e4c00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4e00761a0 0x7fe4e019f3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42586/0 (socket says 172.21.15.67:42586) 2024-03-28T16:02:38.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:38.210+0000 7fe4e4c00700 1 -- 172.21.15.67:0/4259956272 learned_addr learned my addr 172.21.15.67:0/4259956272 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:02:38.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:38.210+0000 7fe4e5600700 1 --2- 172.21.15.67:0/4259956272 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4e0074900 0x7fe4e019ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:38.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:38.210+0000 7fe4e6000700 1 --2- 172.21.15.67:0/4259956272 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe4e0112e50 0x7fe4e01a3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:38.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:38.210+0000 7fe4e4c00700 1 -- 172.21.15.67:0/4259956272 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe4e0112e50 msgr2=0x7fe4e01a3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:38.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:38.210+0000 7fe4e4c00700 1 --2- 172.21.15.67:0/4259956272 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe4e0112e50 0x7fe4e01a3730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:38.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:38.210+0000 7fe4e4c00700 1 -- 172.21.15.67:0/4259956272 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4e0074900 msgr2=0x7fe4e019ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:38.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:38.210+0000 7fe4e4c00700 1 --2- 172.21.15.67:0/4259956272 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4e0074900 0x7fe4e019ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:38.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:38.210+0000 7fe4e4c00700 1 -- 172.21.15.67:0/4259956272 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4e01a3e70 con 0x7fe4e00761a0 2024-03-28T16:02:38.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:38.211+0000 7fe4e4c00700 1 --2- 172.21.15.67:0/4259956272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4e00761a0 0x7fe4e019f3a0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fe4d0003ab0 tx=0x7fe4d0003ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:02:38.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:38.211+0000 7fe4de000700 1 -- 172.21.15.67:0/4259956272 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4d0027070 con 0x7fe4e00761a0 2024-03-28T16:02:38.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:38.211+0000 7fe4de000700 1 -- 172.21.15.67:0/4259956272 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe4d000bdc0 con 0x7fe4e00761a0 2024-03-28T16:02:38.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:38.211+0000 7fe4e8000700 1 -- 172.21.15.67:0/4259956272 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4e01a4100 con 0x7fe4e00761a0 2024-03-28T16:02:38.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:38.211+0000 7fe4de000700 1 -- 172.21.15.67:0/4259956272 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4d0022de0 con 0x7fe4e00761a0 2024-03-28T16:02:38.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:38.211+0000 7fe4e8000700 1 -- 172.21.15.67:0/4259956272 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4e01ab9e0 con 0x7fe4e00761a0 2024-03-28T16:02:38.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:38.212+0000 7fe4e8000700 1 -- 172.21.15.67:0/4259956272 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4e0068e50 con 0x7fe4e00761a0 2024-03-28T16:02:38.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:38.214+0000 7fe4de000700 1 -- 172.21.15.67:0/4259956272 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fe4d002a070 con 0x7fe4e00761a0 2024-03-28T16:02:38.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:38.216+0000 7fe4de000700 1 --2- 172.21.15.67:0/4259956272 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fe4c80a2920 0x7fe4c80a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:38.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:38.216+0000 7fe4de000700 1 -- 172.21.15.67:0/4259956272 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fe4d00cfe80 con 0x7fe4e00761a0 2024-03-28T16:02:38.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:38.216+0000 7fe4e5600700 1 --2- 172.21.15.67:0/4259956272 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fe4c80a2920 0x7fe4c80a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:38.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:38.217+0000 7fe4e5600700 1 --2- 172.21.15.67:0/4259956272 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fe4c80a2920 0x7fe4c80a4de0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fe4d4006fd0 tx=0x7fe4d4008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:02:38.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:38.218+0000 7fe4de000700 1 -- 172.21.15.67:0/4259956272 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe4d0031020 con 0x7fe4e00761a0 2024-03-28T16:02:38.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:38.496+0000 7fe4e8000700 1 -- 172.21.15.67:0/4259956272 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe4e01abbb0 con 0x7fe4e00761a0 2024-03-28T16:02:38.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:38.497+0000 7fe4de000700 1 -- 172.21.15.67:0/4259956272 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fe4d0013070 con 0x7fe4e00761a0 2024-03-28T16:02:38.499 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:02:38.499 INFO:teuthology.orchestra.run.smithi067.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-28T16:02:38.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:38.500+0000 7fe4e8000700 1 -- 172.21.15.67:0/4259956272 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fe4c80a2920 msgr2=0x7fe4c80a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:38.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:38.500+0000 7fe4e8000700 1 --2- 172.21.15.67:0/4259956272 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fe4c80a2920 0x7fe4c80a4de0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fe4d4006fd0 tx=0x7fe4d4008040 comp rx=0 tx=0).stop 2024-03-28T16:02:38.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:38.500+0000 7fe4e8000700 1 -- 172.21.15.67:0/4259956272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4e00761a0 msgr2=0x7fe4e019f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:38.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:38.500+0000 7fe4e8000700 1 --2- 172.21.15.67:0/4259956272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4e00761a0 0x7fe4e019f3a0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fe4d0003ab0 tx=0x7fe4d0003ae0 comp rx=0 tx=0).stop 2024-03-28T16:02:38.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:38.500+0000 7fe4e8000700 1 -- 172.21.15.67:0/4259956272 shutdown_connections 2024-03-28T16:02:38.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:38.500+0000 7fe4e8000700 1 --2- 172.21.15.67:0/4259956272 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fe4c80a2920 0x7fe4c80a4de0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:38.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:38.500+0000 7fe4e8000700 1 --2- 172.21.15.67:0/4259956272 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe4e0112e50 0x7fe4e01a3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:38.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:38.500+0000 7fe4e8000700 1 --2- 172.21.15.67:0/4259956272 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4e0074900 0x7fe4e019ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:38.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:38.500+0000 7fe4e8000700 1 --2- 172.21.15.67:0/4259956272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4e00761a0 0x7fe4e019f3a0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:38.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:38.500+0000 7fe4e8000700 1 -- 172.21.15.67:0/4259956272 >> 172.21.15.67:0/4259956272 conn(0x7fe4e0079220 msgr2=0x7fe4e0111b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:02:38.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:38.500+0000 7fe4e8000700 1 -- 172.21.15.67:0/4259956272 shutdown_connections 2024-03-28T16:02:38.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:38.500+0000 7fe4e8000700 1 -- 172.21.15.67:0/4259956272 wait complete. 2024-03-28T16:02:38.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:38 smithi067 ceph-mon[31707]: pgmap v411: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:02:38.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:38 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/4259956272' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-28T16:02:38.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:38 smithi183 ceph-mon[35580]: pgmap v411: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:02:38.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:38 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/4259956272' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-28T16:02:39.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:38 smithi150 ceph-mon[34825]: pgmap v411: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:02:39.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:38 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/4259956272' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-28T16:02:39.312 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-28T16:02:39.313 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T16:02:39.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:39.544+0000 7f815ba00640 1 -- 172.21.15.67:0/1863722100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8154104ad0 msgr2=0x7f8154104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:39.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:39.544+0000 7f815ba00640 1 --2- 172.21.15.67:0/1863722100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8154104ad0 0x7f8154104f50 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f8140009900 tx=0x7f814002bc40 comp rx=0 tx=0).stop 2024-03-28T16:02:39.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:39.544+0000 7f815ba00640 1 -- 172.21.15.67:0/1863722100 shutdown_connections 2024-03-28T16:02:39.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:39.544+0000 7f815ba00640 1 --2- 172.21.15.67:0/1863722100 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8154105490 0x7f815410bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:39.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:39.544+0000 7f815ba00640 1 --2- 172.21.15.67:0/1863722100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8154104ad0 0x7f8154104f50 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:39.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:39.544+0000 7f815ba00640 1 --2- 172.21.15.67:0/1863722100 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81541038e0 0x7f8154103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:39.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:39.544+0000 7f815ba00640 1 -- 172.21.15.67:0/1863722100 >> 172.21.15.67:0/1863722100 conn(0x7f81540fee90 msgr2=0x7f81541012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:02:39.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:39.544+0000 7f815ba00640 1 -- 172.21.15.67:0/1863722100 shutdown_connections 2024-03-28T16:02:39.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:39.544+0000 7f815ba00640 1 -- 172.21.15.67:0/1863722100 wait complete. 2024-03-28T16:02:39.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:39.545+0000 7f815ba00640 1 Processor -- start 2024-03-28T16:02:39.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:39.546+0000 7f815ba00640 1 -- start start 2024-03-28T16:02:39.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:39.546+0000 7f815ba00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81541038e0 0x7f8154199b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:39.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:39.546+0000 7f815ba00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8154104ad0 0x7f815419a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:39.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:39.546+0000 7f815ba00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8154105490 0x7f81541a10c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:39.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:39.546+0000 7f815ba00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f815410f460 con 0x7f8154104ad0 2024-03-28T16:02:39.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:39.546+0000 7f815ba00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f815410f2e0 con 0x7f81541038e0 2024-03-28T16:02:39.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:39.546+0000 7f815ba00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f815410f5e0 con 0x7f8154105490 2024-03-28T16:02:39.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:39.546+0000 7f8153e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8154104ad0 0x7f815419a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:39.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:39.546+0000 7f8153e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8154104ad0 0x7f815419a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47654/0 (socket says 172.21.15.67:47654) 2024-03-28T16:02:39.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:39.546+0000 7f8153e00640 1 -- 172.21.15.67:0/2784106710 learned_addr learned my addr 172.21.15.67:0/2784106710 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:02:39.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:39.546+0000 7f8159000640 1 --2- 172.21.15.67:0/2784106710 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81541038e0 0x7f8154199b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:39.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:39.547+0000 7f8159a00640 1 --2- 172.21.15.67:0/2784106710 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8154105490 0x7f81541a10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:39.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:39.547+0000 7f8153e00640 1 -- 172.21.15.67:0/2784106710 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8154105490 msgr2=0x7f81541a10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:39.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:39.547+0000 7f8153e00640 1 --2- 172.21.15.67:0/2784106710 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8154105490 0x7f81541a10c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:39.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:39.547+0000 7f8153e00640 1 -- 172.21.15.67:0/2784106710 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81541038e0 msgr2=0x7f8154199b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:39.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:39.547+0000 7f8153e00640 1 --2- 172.21.15.67:0/2784106710 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81541038e0 0x7f8154199b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:39.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:39.547+0000 7f8153e00640 1 -- 172.21.15.67:0/2784106710 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81541a1730 con 0x7f8154104ad0 2024-03-28T16:02:39.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:39.547+0000 7f8159a00640 1 --2- 172.21.15.67:0/2784106710 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8154105490 0x7f81541a10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:02:39.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:39.547+0000 7f8153e00640 1 --2- 172.21.15.67:0/2784106710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8154104ad0 0x7f815419a050 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f8140000c00 tx=0x7f81400042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:02:39.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:39.547+0000 7f8151600640 1 -- 172.21.15.67:0/2784106710 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8140043020 con 0x7f8154104ad0 2024-03-28T16:02:39.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:39.547+0000 7f8151600640 1 -- 172.21.15.67:0/2784106710 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f81400054c0 con 0x7f8154104ad0 2024-03-28T16:02:39.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:39.547+0000 7f815ba00640 1 -- 172.21.15.67:0/2784106710 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81541a19c0 con 0x7f8154104ad0 2024-03-28T16:02:39.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:39.547+0000 7f8151600640 1 -- 172.21.15.67:0/2784106710 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8140003d70 con 0x7f8154104ad0 2024-03-28T16:02:39.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:39.548+0000 7f815ba00640 1 -- 172.21.15.67:0/2784106710 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81541a1f10 con 0x7f8154104ad0 2024-03-28T16:02:39.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:39.549+0000 7f815ba00640 1 -- 172.21.15.67:0/2784106710 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f815410ac80 con 0x7f8154104ad0 2024-03-28T16:02:39.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:39.551+0000 7f8151600640 1 -- 172.21.15.67:0/2784106710 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f8140046020 con 0x7f8154104ad0 2024-03-28T16:02:39.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:39.552+0000 7f8151600640 1 --2- 172.21.15.67:0/2784106710 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f813c0a2890 0x7f813c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:39.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:39.552+0000 7f8151600640 1 -- 172.21.15.67:0/2784106710 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f81400f1400 con 0x7f8154104ad0 2024-03-28T16:02:39.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:39.552+0000 7f8159000640 1 --2- 172.21.15.67:0/2784106710 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f813c0a2890 0x7f813c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:39.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:39.553+0000 7f8159000640 1 --2- 172.21.15.67:0/2784106710 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f813c0a2890 0x7f813c0a4d50 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8148005fd0 tx=0x7f8148005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:02:39.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:39.555+0000 7f8151600640 1 -- 172.21.15.67:0/2784106710 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f814004d020 con 0x7f8154104ad0 2024-03-28T16:02:39.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:39.584+0000 7f8151600640 1 -- 172.21.15.67:0/2784106710 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f81400ba490 con 0x7f8154104ad0 2024-03-28T16:02:39.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:39.780+0000 7f815ba00640 1 -- 172.21.15.67:0/2784106710 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f815402d240 con 0x7f8154104ad0 2024-03-28T16:02:39.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:39.784+0000 7f8151600640 1 -- 172.21.15.67:0/2784106710 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) v1 ==== 74+0+22673 (secure 0 0 0) 0x7f81400be520 con 0x7f8154104ad0 2024-03-28T16:02:39.786 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:02:39.787 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":92,"fsid":"b5f8225a-ed1a-11ee-95d0-87774f69a715","created":"2024-03-28T15:50:21.471924+0000","modified":"2024-03-28T16:01:40.643215+0000","last_up_change":"2024-03-28T16:00:01.369123+0000","last_in_change":"2024-03-28T15:59:37.515483+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T15:54:55.588007+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T16:01:05.719801+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T16:01:07.206303+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7c1791b6-9c0e-4512-a327-cad166bb5581","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6803","nonce":1894681701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6805","nonce":1894681701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6809","nonce":1894681701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6807","nonce":1894681701}]},"public_addr":"172.21.15.67:6803/1894681701","cluster_addr":"172.21.15.67:6805/1894681701","heartbeat_back_addr":"172.21.15.67:6809/1894681701","heartbeat_front_addr":"172.21.15.67:6807/1894681701","state":["exists","up"]},{"osd":1,"uuid":"0cdda7f3-d00c-4436-bcc1-49fb8072b2e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6801","nonce":1734465762}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6803","nonce":1734465762}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6807","nonce":1734465762}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6805","nonce":1734465762}]},"public_addr":"172.21.15.150:6801/1734465762","cluster_addr":"172.21.15.150:6803/1734465762","heartbeat_back_addr":"172.21.15.150:6807/1734465762","heartbeat_front_addr":"172.21.15.150:6805/1734465762","state":["exists","up"]},{"osd":2,"uuid":"a55b9a02-c3fd-4061-95c0-53a24a17ade0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6801","nonce":3398095931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6803","nonce":3398095931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6807","nonce":3398095931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6805","nonce":3398095931}]},"public_addr":"172.21.15.183:6801/3398095931","cluster_addr":"172.21.15.183:6803/3398095931","heartbeat_back_addr":"172.21.15.183:6807/3398095931","heartbeat_front_addr":"172.21.15.183:6805/3398095931","state":["exists","up"]},{"osd":3,"uuid":"d34940c0-5ce9-4b8b-aa36-4872414a043f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6811","nonce":1933541875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6813","nonce":1933541875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6817","nonce":1933541875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6815","nonce":1933541875}]},"public_addr":"172.21.15.67:6811/1933541875","cluster_addr":"172.21.15.67:6813/1933541875","heartbeat_back_addr":"172.21.15.67:6817/1933541875","heartbeat_front_addr":"172.21.15.67:6815/1933541875","state":["exists","up"]},{"osd":4,"uuid":"5cc3e260-83a3-4a4d-aa6a-34a3b12b2b5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6809","nonce":3775269993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6811","nonce":3775269993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6815","nonce":3775269993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6813","nonce":3775269993}]},"public_addr":"172.21.15.150:6809/3775269993","cluster_addr":"172.21.15.150:6811/3775269993","heartbeat_back_addr":"172.21.15.150:6815/3775269993","heartbeat_front_addr":"172.21.15.150:6813/3775269993","state":["exists","up"]},{"osd":5,"uuid":"a2542b21-d6ac-413e-8b9b-4a6548b20840","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6809","nonce":164284137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6811","nonce":164284137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6815","nonce":164284137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6813","nonce":164284137}]},"public_addr":"172.21.15.183:6809/164284137","cluster_addr":"172.21.15.183:6811/164284137","heartbeat_back_addr":"172.21.15.183:6815/164284137","heartbeat_front_addr":"172.21.15.183:6813/164284137","state":["exists","up"]},{"osd":6,"uuid":"8905a2d3-0f1b-4853-a218-22da2635150a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6819","nonce":3565528013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6821","nonce":3565528013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6825","nonce":3565528013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6823","nonce":3565528013}]},"public_addr":"172.21.15.67:6819/3565528013","cluster_addr":"172.21.15.67:6821/3565528013","heartbeat_back_addr":"172.21.15.67:6825/3565528013","heartbeat_front_addr":"172.21.15.67:6823/3565528013","state":["exists","up"]},{"osd":7,"uuid":"b275055b-b451-4180-91b1-e6e61a828c48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6817","nonce":3735115340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6819","nonce":3735115340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6823","nonce":3735115340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6821","nonce":3735115340}]},"public_addr":"172.21.15.150:6817/3735115340","cluster_addr":"172.21.15.150:6819/3735115340","heartbeat_back_addr":"172.21.15.150:6823/3735115340","heartbeat_front_addr":"172.21.15.150:6821/3735115340","state":["exists","up"]},{"osd":8,"uuid":"b6abf7ad-eb34-4f60-a4eb-348a92d7195b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6817","nonce":1025400095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6819","nonce":1025400095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6823","nonce":1025400095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6821","nonce":1025400095}]},"public_addr":"172.21.15.183:6817/1025400095","cluster_addr":"172.21.15.183:6819/1025400095","heartbeat_back_addr":"172.21.15.183:6823/1025400095","heartbeat_front_addr":"172.21.15.183:6821/1025400095","state":["exists","up"]},{"osd":9,"uuid":"28a78e3a-0657-40aa-816f-16a6fadadcbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6827","nonce":76885409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6829","nonce":76885409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6833","nonce":76885409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6831","nonce":76885409}]},"public_addr":"172.21.15.67:6827/76885409","cluster_addr":"172.21.15.67:6829/76885409","heartbeat_back_addr":"172.21.15.67:6833/76885409","heartbeat_front_addr":"172.21.15.67:6831/76885409","state":["exists","up"]},{"osd":10,"uuid":"d287907e-424f-4b49-a06e-4613c6de0474","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6825","nonce":3489982080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6827","nonce":3489982080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6831","nonce":3489982080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6829","nonce":3489982080}]},"public_addr":"172.21.15.150:6825/3489982080","cluster_addr":"172.21.15.150:6827/3489982080","heartbeat_back_addr":"172.21.15.150:6831/3489982080","heartbeat_front_addr":"172.21.15.150:6829/3489982080","state":["exists","up"]},{"osd":11,"uuid":"a3c29c47-a228-4f42-a23c-fbdb0c093699","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":85,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6825","nonce":1892639849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6826","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6827","nonce":1892639849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6831","nonce":1892639849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6829","nonce":1892639849}]},"public_addr":"172.21.15.183:6825/1892639849","cluster_addr":"172.21.15.183:6827/1892639849","heartbeat_back_addr":"172.21.15.183:6831/1892639849","heartbeat_front_addr":"172.21.15.183:6829/1892639849","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:53:45.646211+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:54:18.956590+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:54:52.161646+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:55:27.502822+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:56:01.162932+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:56:34.674452+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:57:09.298679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:57:43.230406+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:58:17.003821+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:58:51.085863+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:59:25.665065+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T16:00:00.263457+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":1}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6801/1631001968":"2024-03-29T15:51:38.875334+0000","172.21.15.67:6800/1631001968":"2024-03-29T15:51:38.875334+0000","172.21.15.67:0/1533056310":"2024-03-29T15:51:38.875334+0000","172.21.15.67:6801/3580491071":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/2030412458":"2024-03-29T15:51:03.859740+0000","172.21.15.67:6800/3580491071":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/2905307533":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/3763759988":"2024-03-29T15:51:38.875334+0000","172.21.15.67:0/4089399880":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/4239108661":"2024-03-29T15:51:38.875334+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T16:02:39.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:39.787+0000 7f815ba00640 1 -- 172.21.15.67:0/2784106710 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f813c0a2890 msgr2=0x7f813c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:39.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:39.787+0000 7f815ba00640 1 --2- 172.21.15.67:0/2784106710 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f813c0a2890 0x7f813c0a4d50 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8148005fd0 tx=0x7f8148005850 comp rx=0 tx=0).stop 2024-03-28T16:02:39.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:39.787+0000 7f815ba00640 1 -- 172.21.15.67:0/2784106710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8154104ad0 msgr2=0x7f815419a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:39.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:39.787+0000 7f815ba00640 1 --2- 172.21.15.67:0/2784106710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8154104ad0 0x7f815419a050 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f8140000c00 tx=0x7f81400042f0 comp rx=0 tx=0).stop 2024-03-28T16:02:39.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:39.787+0000 7f815ba00640 1 -- 172.21.15.67:0/2784106710 shutdown_connections 2024-03-28T16:02:39.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:39.787+0000 7f815ba00640 1 --2- 172.21.15.67:0/2784106710 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f813c0a2890 0x7f813c0a4d50 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:39.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:39.787+0000 7f815ba00640 1 --2- 172.21.15.67:0/2784106710 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8154105490 0x7f81541a10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:39.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:39.787+0000 7f815ba00640 1 --2- 172.21.15.67:0/2784106710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8154104ad0 0x7f815419a050 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:39.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:39.787+0000 7f815ba00640 1 --2- 172.21.15.67:0/2784106710 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81541038e0 0x7f8154199b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:39.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:39.787+0000 7f815ba00640 1 -- 172.21.15.67:0/2784106710 >> 172.21.15.67:0/2784106710 conn(0x7f81540fee90 msgr2=0x7f8154107e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:02:39.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:39.787+0000 7f815ba00640 1 -- 172.21.15.67:0/2784106710 shutdown_connections 2024-03-28T16:02:39.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:39.787+0000 7f815ba00640 1 -- 172.21.15.67:0/2784106710 wait complete. 2024-03-28T16:02:39.810 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T15:54:55.588007+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T16:01:05.719801+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T16:01:07.206303+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T16:02:39.811 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T16:02:40.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.032+0000 7f4f69600640 1 -- 172.21.15.67:0/3430409395 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f64109770 msgr2=0x7f4f64110000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:40.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.032+0000 7f4f69600640 1 --2- 172.21.15.67:0/3430409395 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f64109770 0x7f4f64110000 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c009ca0 tx=0x7f4f5c02c100 comp rx=0 tx=0).stop 2024-03-28T16:02:40.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.033+0000 7f4f69600640 1 -- 172.21.15.67:0/3430409395 shutdown_connections 2024-03-28T16:02:40.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.033+0000 7f4f69600640 1 --2- 172.21.15.67:0/3430409395 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f64109770 0x7f4f64110000 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:40.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.033+0000 7f4f69600640 1 --2- 172.21.15.67:0/3430409395 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4f64108db0 0x7f4f64109230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:40.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.033+0000 7f4f69600640 1 --2- 172.21.15.67:0/3430409395 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f64107bc0 0x7f4f64107fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:40.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.033+0000 7f4f69600640 1 -- 172.21.15.67:0/3430409395 >> 172.21.15.67:0/3430409395 conn(0x7f4f64076b70 msgr2=0x7f4f64078f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:02:40.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.033+0000 7f4f69600640 1 -- 172.21.15.67:0/3430409395 shutdown_connections 2024-03-28T16:02:40.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.033+0000 7f4f69600640 1 -- 172.21.15.67:0/3430409395 wait complete. 2024-03-28T16:02:40.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.034+0000 7f4f69600640 1 Processor -- start 2024-03-28T16:02:40.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.034+0000 7f4f69600640 1 -- start start 2024-03-28T16:02:40.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.034+0000 7f4f69600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4f64107bc0 0x7f4f6419ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:40.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.034+0000 7f4f69600640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f64108db0 0x7f4f6419e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:40.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.034+0000 7f4f69600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f64109770 0x7f4f641a5320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:40.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.034+0000 7f4f69600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f641136f0 con 0x7f4f64109770 2024-03-28T16:02:40.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.034+0000 7f4f69600640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f64113570 con 0x7f4f64108db0 2024-03-28T16:02:40.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.034+0000 7f4f69600640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f64113870 con 0x7f4f64107bc0 2024-03-28T16:02:40.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.034+0000 7f4f68c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f64109770 0x7f4f641a5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:40.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.034+0000 7f4f68c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f64109770 0x7f4f641a5320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47664/0 (socket says 172.21.15.67:47664) 2024-03-28T16:02:40.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.034+0000 7f4f68c00640 1 -- 172.21.15.67:0/3942273547 learned_addr learned my addr 172.21.15.67:0/3942273547 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:02:40.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.035+0000 7f4f68c00640 1 -- 172.21.15.67:0/3942273547 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4f64107bc0 msgr2=0x7f4f6419ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:40.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.035+0000 7f4f68c00640 1 --2- 172.21.15.67:0/3942273547 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4f64107bc0 0x7f4f6419ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:40.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.035+0000 7f4f68c00640 1 -- 172.21.15.67:0/3942273547 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f64108db0 msgr2=0x7f4f6419e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:40.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.035+0000 7f4f68c00640 1 --2- 172.21.15.67:0/3942273547 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f64108db0 0x7f4f6419e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:40.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.035+0000 7f4f68c00640 1 -- 172.21.15.67:0/3942273547 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f641a5990 con 0x7f4f64109770 2024-03-28T16:02:40.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.035+0000 7f4f68c00640 1 --2- 172.21.15.67:0/3942273547 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f64109770 0x7f4f641a5320 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c032520 tx=0x7f4f5c034880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:02:40.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.035+0000 7f4f61600640 1 -- 172.21.15.67:0/3942273547 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f5c039040 con 0x7f4f64109770 2024-03-28T16:02:40.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.035+0000 7f4f61600640 1 -- 172.21.15.67:0/3942273547 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4f5c044070 con 0x7f4f64109770 2024-03-28T16:02:40.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.035+0000 7f4f61600640 1 -- 172.21.15.67:0/3942273547 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f5c0043c0 con 0x7f4f64109770 2024-03-28T16:02:40.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.035+0000 7f4f69600640 1 -- 172.21.15.67:0/3942273547 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f641a5c20 con 0x7f4f64109770 2024-03-28T16:02:40.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.036+0000 7f4f69600640 1 -- 172.21.15.67:0/3942273547 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f641a60c0 con 0x7f4f64109770 2024-03-28T16:02:40.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.037+0000 7f4f69600640 1 -- 172.21.15.67:0/3942273547 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f6410eee0 con 0x7f4f64109770 2024-03-28T16:02:40.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.039+0000 7f4f61600640 1 -- 172.21.15.67:0/3942273547 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f4f5c004560 con 0x7f4f64109770 2024-03-28T16:02:40.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.040+0000 7f4f61600640 1 --2- 172.21.15.67:0/3942273547 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f4f480a2880 0x7f4f480a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:40.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.040+0000 7f4f61600640 1 -- 172.21.15.67:0/3942273547 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f4f5c0f1190 con 0x7f4f64109770 2024-03-28T16:02:40.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.040+0000 7f4f63e00640 1 --2- 172.21.15.67:0/3942273547 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f4f480a2880 0x7f4f480a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:40.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.041+0000 7f4f63e00640 1 --2- 172.21.15.67:0/3942273547 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f4f480a2880 0x7f4f480a4d40 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4f54005fd0 tx=0x7f4f5400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:02:40.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.045+0000 7f4f61600640 1 -- 172.21.15.67:0/3942273547 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4f5c0333a0 con 0x7f4f64109770 2024-03-28T16:02:40.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.247+0000 7f4f69600640 1 -- 172.21.15.67:0/3942273547 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4f64062340 con 0x7f4f64109770 2024-03-28T16:02:40.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.247+0000 7f4f61600640 1 -- 172.21.15.67:0/3942273547 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v92) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4f5c0be1e0 con 0x7f4f64109770 2024-03-28T16:02:40.248 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-03-28T16:02:40.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.250+0000 7f4f69600640 1 -- 172.21.15.67:0/3942273547 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f4f480a2880 msgr2=0x7f4f480a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:40.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.250+0000 7f4f69600640 1 --2- 172.21.15.67:0/3942273547 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f4f480a2880 0x7f4f480a4d40 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4f54005fd0 tx=0x7f4f5400b410 comp rx=0 tx=0).stop 2024-03-28T16:02:40.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.250+0000 7f4f69600640 1 -- 172.21.15.67:0/3942273547 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f64109770 msgr2=0x7f4f641a5320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:40.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.250+0000 7f4f69600640 1 --2- 172.21.15.67:0/3942273547 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f64109770 0x7f4f641a5320 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c032520 tx=0x7f4f5c034880 comp rx=0 tx=0).stop 2024-03-28T16:02:40.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.250+0000 7f4f69600640 1 -- 172.21.15.67:0/3942273547 shutdown_connections 2024-03-28T16:02:40.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.250+0000 7f4f69600640 1 --2- 172.21.15.67:0/3942273547 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f4f480a2880 0x7f4f480a4d40 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:40.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.250+0000 7f4f69600640 1 --2- 172.21.15.67:0/3942273547 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f64109770 0x7f4f641a5320 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:40.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.250+0000 7f4f69600640 1 --2- 172.21.15.67:0/3942273547 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f64108db0 0x7f4f6419e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:40.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.250+0000 7f4f69600640 1 --2- 172.21.15.67:0/3942273547 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4f64107bc0 0x7f4f6419ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:40.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.250+0000 7f4f69600640 1 -- 172.21.15.67:0/3942273547 >> 172.21.15.67:0/3942273547 conn(0x7f4f64076b70 msgr2=0x7f4f6410c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:02:40.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.250+0000 7f4f69600640 1 -- 172.21.15.67:0/3942273547 shutdown_connections 2024-03-28T16:02:40.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.251+0000 7f4f69600640 1 -- 172.21.15.67:0/3942273547 wait complete. 2024-03-28T16:02:40.271 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T16:02:40.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.504+0000 7f794be00640 1 -- 172.21.15.67:0/3501112744 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f794c103150 msgr2=0x7f794c1099e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:40.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.504+0000 7f794be00640 1 --2- 172.21.15.67:0/3501112744 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f794c103150 0x7f794c1099e0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f7944009900 tx=0x7f794402bc30 comp rx=0 tx=0).stop 2024-03-28T16:02:40.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.504+0000 7f794be00640 1 -- 172.21.15.67:0/3501112744 shutdown_connections 2024-03-28T16:02:40.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.504+0000 7f794be00640 1 --2- 172.21.15.67:0/3501112744 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f794c103150 0x7f794c1099e0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:40.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.504+0000 7f794be00640 1 --2- 172.21.15.67:0/3501112744 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f794c102790 0x7f794c102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:40.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.504+0000 7f794be00640 1 --2- 172.21.15.67:0/3501112744 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f794c1015a0 0x7f794c1019c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:40.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.504+0000 7f794be00640 1 -- 172.21.15.67:0/3501112744 >> 172.21.15.67:0/3501112744 conn(0x7f794c0fcb10 msgr2=0x7f794c0fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:02:40.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.505+0000 7f794be00640 1 -- 172.21.15.67:0/3501112744 shutdown_connections 2024-03-28T16:02:40.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.505+0000 7f794be00640 1 -- 172.21.15.67:0/3501112744 wait complete. 2024-03-28T16:02:40.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.506+0000 7f794be00640 1 Processor -- start 2024-03-28T16:02:40.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.506+0000 7f794be00640 1 -- start start 2024-03-28T16:02:40.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.506+0000 7f794be00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f794c1015a0 0x7f794c075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:40.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.506+0000 7f794be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f794c102790 0x7f794c073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:40.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.506+0000 7f794be00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f794c103150 0x7f794c073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:40.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.506+0000 7f794be00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f794c10d100 con 0x7f794c102790 2024-03-28T16:02:40.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.506+0000 7f794be00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f794c10cf80 con 0x7f794c1015a0 2024-03-28T16:02:40.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.506+0000 7f794be00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f794c10d280 con 0x7f794c103150 2024-03-28T16:02:40.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.506+0000 7f794a000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f794c102790 0x7f794c073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:40.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.506+0000 7f794a000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f794c102790 0x7f794c073a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47684/0 (socket says 172.21.15.67:47684) 2024-03-28T16:02:40.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.506+0000 7f794a000640 1 -- 172.21.15.67:0/3660259003 learned_addr learned my addr 172.21.15.67:0/3660259003 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:02:40.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.507+0000 7f794aa00640 1 --2- 172.21.15.67:0/3660259003 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f794c1015a0 0x7f794c075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:40.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.507+0000 7f794b400640 1 --2- 172.21.15.67:0/3660259003 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f794c103150 0x7f794c073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:40.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.507+0000 7f794a000640 1 -- 172.21.15.67:0/3660259003 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f794c103150 msgr2=0x7f794c073fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:40.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.507+0000 7f794a000640 1 --2- 172.21.15.67:0/3660259003 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f794c103150 0x7f794c073fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:40.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.507+0000 7f794a000640 1 -- 172.21.15.67:0/3660259003 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f794c1015a0 msgr2=0x7f794c075450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:40.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.507+0000 7f794a000640 1 --2- 172.21.15.67:0/3660259003 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f794c1015a0 0x7f794c075450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:40.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.507+0000 7f794a000640 1 -- 172.21.15.67:0/3660259003 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f794c074860 con 0x7f794c102790 2024-03-28T16:02:40.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.507+0000 7f794a000640 1 --2- 172.21.15.67:0/3660259003 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f794c102790 0x7f794c073a90 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f793800e9f0 tx=0x7f793800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:02:40.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.507+0000 7f7950a00640 1 -- 172.21.15.67:0/3660259003 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f793800cd80 con 0x7f794c102790 2024-03-28T16:02:40.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.508+0000 7f7950a00640 1 -- 172.21.15.67:0/3660259003 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f79380108e0 con 0x7f794c102790 2024-03-28T16:02:40.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.508+0000 7f7950a00640 1 -- 172.21.15.67:0/3660259003 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7938018920 con 0x7f794c102790 2024-03-28T16:02:40.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.508+0000 7f794be00640 1 -- 172.21.15.67:0/3660259003 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f794c1a1860 con 0x7f794c102790 2024-03-28T16:02:40.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.508+0000 7f794be00640 1 -- 172.21.15.67:0/3660259003 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f794c1a1e00 con 0x7f794c102790 2024-03-28T16:02:40.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.509+0000 7f794be00640 1 -- 172.21.15.67:0/3660259003 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f794c1088c0 con 0x7f794c102790 2024-03-28T16:02:40.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.512+0000 7f7950a00640 1 -- 172.21.15.67:0/3660259003 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f7938010a90 con 0x7f794c102790 2024-03-28T16:02:40.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.513+0000 7f7950a00640 1 --2- 172.21.15.67:0/3660259003 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f79300a2880 0x7f79300a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:40.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.513+0000 7f7950a00640 1 -- 172.21.15.67:0/3660259003 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f79380cee40 con 0x7f794c102790 2024-03-28T16:02:40.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.513+0000 7f794aa00640 1 --2- 172.21.15.67:0/3660259003 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f79300a2880 0x7f79300a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:40.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.514+0000 7f794aa00640 1 --2- 172.21.15.67:0/3660259003 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f79300a2880 0x7f79300a4d40 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f794c1025f0 tx=0x7f793c0098f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:02:40.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.517+0000 7f7950a00640 1 -- 172.21.15.67:0/3660259003 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f793809be90 con 0x7f794c102790 2024-03-28T16:02:40.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.721+0000 7f794be00640 1 -- 172.21.15.67:0/3660259003 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f794c0622a0 con 0x7f794c102790 2024-03-28T16:02:40.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.722+0000 7f7950a00640 1 -- 172.21.15.67:0/3660259003 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v92) v1 ==== 104+0+11 (secure 0 0 0) 0x7f793809b790 con 0x7f794c102790 2024-03-28T16:02:40.723 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-03-28T16:02:40.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.724+0000 7f794be00640 1 -- 172.21.15.67:0/3660259003 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f79300a2880 msgr2=0x7f79300a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:40.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.724+0000 7f794be00640 1 --2- 172.21.15.67:0/3660259003 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f79300a2880 0x7f79300a4d40 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f794c1025f0 tx=0x7f793c0098f0 comp rx=0 tx=0).stop 2024-03-28T16:02:40.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.724+0000 7f794be00640 1 -- 172.21.15.67:0/3660259003 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f794c102790 msgr2=0x7f794c073a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:40.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.724+0000 7f794be00640 1 --2- 172.21.15.67:0/3660259003 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f794c102790 0x7f794c073a90 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f793800e9f0 tx=0x7f793800eeb0 comp rx=0 tx=0).stop 2024-03-28T16:02:40.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.725+0000 7f794be00640 1 -- 172.21.15.67:0/3660259003 shutdown_connections 2024-03-28T16:02:40.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.725+0000 7f794be00640 1 --2- 172.21.15.67:0/3660259003 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f79300a2880 0x7f79300a4d40 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:40.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.725+0000 7f794be00640 1 --2- 172.21.15.67:0/3660259003 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f794c103150 0x7f794c073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:40.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.725+0000 7f794be00640 1 --2- 172.21.15.67:0/3660259003 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f794c102790 0x7f794c073a90 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:40.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.725+0000 7f794be00640 1 --2- 172.21.15.67:0/3660259003 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f794c1015a0 0x7f794c075450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:40.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.725+0000 7f794be00640 1 -- 172.21.15.67:0/3660259003 >> 172.21.15.67:0/3660259003 conn(0x7f794c0fcb10 msgr2=0x7f794c105a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:02:40.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.725+0000 7f794be00640 1 -- 172.21.15.67:0/3660259003 shutdown_connections 2024-03-28T16:02:40.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.725+0000 7f794be00640 1 -- 172.21.15.67:0/3660259003 wait complete. 2024-03-28T16:02:40.747 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T16:02:40.834 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:40 smithi067 ceph-mon[31707]: pgmap v412: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:02:40.834 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:40 smithi067 ceph-mon[31707]: mgrmap e18: x(active, since 11m), standbys: z, y 2024-03-28T16:02:40.835 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:40 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/2784106710' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:02:40.835 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:40 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/3942273547' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T16:02:40.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:40 smithi183 ceph-mon[35580]: pgmap v412: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:02:40.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:40 smithi183 ceph-mon[35580]: mgrmap e18: x(active, since 11m), standbys: z, y 2024-03-28T16:02:40.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:40 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/2784106710' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:02:40.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:40 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/3942273547' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T16:02:40.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.980+0000 7fc5c3e00640 1 -- 172.21.15.67:0/2545761477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5c40756e0 msgr2=0x7fc5c4073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:40.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.980+0000 7fc5c3e00640 1 --2- 172.21.15.67:0/2545761477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5c40756e0 0x7fc5c4073b30 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fc5b4009930 tx=0x7fc5b402bc20 comp rx=0 tx=0).stop 2024-03-28T16:02:40.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.980+0000 7fc5c3e00640 1 -- 172.21.15.67:0/2545761477 shutdown_connections 2024-03-28T16:02:40.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.980+0000 7fc5c3e00640 1 --2- 172.21.15.67:0/2545761477 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5c410db70 0x7fc5c410ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:40.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.980+0000 7fc5c3e00640 1 --2- 172.21.15.67:0/2545761477 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc5c4074100 0x7fc5c4074580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:40.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.980+0000 7fc5c3e00640 1 --2- 172.21.15.67:0/2545761477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5c40756e0 0x7fc5c4073b30 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:40.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.980+0000 7fc5c3e00640 1 -- 172.21.15.67:0/2545761477 >> 172.21.15.67:0/2545761477 conn(0x7fc5c40fcb10 msgr2=0x7fc5c40fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:02:40.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.980+0000 7fc5c3e00640 1 -- 172.21.15.67:0/2545761477 shutdown_connections 2024-03-28T16:02:40.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.980+0000 7fc5c3e00640 1 -- 172.21.15.67:0/2545761477 wait complete. 2024-03-28T16:02:40.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.981+0000 7fc5c3e00640 1 Processor -- start 2024-03-28T16:02:40.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.982+0000 7fc5c3e00640 1 -- start start 2024-03-28T16:02:40.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.982+0000 7fc5c3e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5c4074100 0x7fc5c419de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:40.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.982+0000 7fc5c3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5c40756e0 0x7fc5c419e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:40.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.982+0000 7fc5c3e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc5c410db70 0x7fc5c41a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:40.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.982+0000 7fc5c3e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c4113710 con 0x7fc5c40756e0 2024-03-28T16:02:40.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.982+0000 7fc5c3e00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c4113590 con 0x7fc5c4074100 2024-03-28T16:02:40.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.982+0000 7fc5c3e00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c4113890 con 0x7fc5c410db70 2024-03-28T16:02:40.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.982+0000 7fc5ba000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5c40756e0 0x7fc5c419e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:40.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.982+0000 7fc5ba000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5c40756e0 0x7fc5c419e370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47702/0 (socket says 172.21.15.67:47702) 2024-03-28T16:02:40.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.982+0000 7fc5ba000640 1 -- 172.21.15.67:0/3707005258 learned_addr learned my addr 172.21.15.67:0/3707005258 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:02:40.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.982+0000 7fc5c2a00640 1 --2- 172.21.15.67:0/3707005258 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5c4074100 0x7fc5c419de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:40.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.983+0000 7fc5c3400640 1 --2- 172.21.15.67:0/3707005258 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc5c410db70 0x7fc5c41a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:40.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.983+0000 7fc5ba000640 1 -- 172.21.15.67:0/3707005258 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc5c410db70 msgr2=0x7fc5c41a53e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:40.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.983+0000 7fc5ba000640 1 --2- 172.21.15.67:0/3707005258 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc5c410db70 0x7fc5c41a53e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:40.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.983+0000 7fc5ba000640 1 -- 172.21.15.67:0/3707005258 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5c4074100 msgr2=0x7fc5c419de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:40.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.983+0000 7fc5ba000640 1 --2- 172.21.15.67:0/3707005258 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5c4074100 0x7fc5c419de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:40.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.983+0000 7fc5ba000640 1 -- 172.21.15.67:0/3707005258 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5c41a5a50 con 0x7fc5c40756e0 2024-03-28T16:02:40.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.983+0000 7fc5c3400640 1 --2- 172.21.15.67:0/3707005258 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc5c410db70 0x7fc5c41a53e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:02:40.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.983+0000 7fc5ba000640 1 --2- 172.21.15.67:0/3707005258 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5c40756e0 0x7fc5c419e370 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac00d770 tx=0x7fc5ac00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:02:40.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.983+0000 7fc5bbe00640 1 -- 172.21.15.67:0/3707005258 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5ac017070 con 0x7fc5c40756e0 2024-03-28T16:02:40.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.983+0000 7fc5bbe00640 1 -- 172.21.15.67:0/3707005258 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc5ac0054d0 con 0x7fc5c40756e0 2024-03-28T16:02:40.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.983+0000 7fc5bbe00640 1 -- 172.21.15.67:0/3707005258 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5ac004cf0 con 0x7fc5c40756e0 2024-03-28T16:02:40.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.984+0000 7fc5c3e00640 1 -- 172.21.15.67:0/3707005258 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5c41a5d40 con 0x7fc5c40756e0 2024-03-28T16:02:40.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.984+0000 7fc5c3e00640 1 -- 172.21.15.67:0/3707005258 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5c41a61e0 con 0x7fc5c40756e0 2024-03-28T16:02:40.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.985+0000 7fc5c3e00640 1 -- 172.21.15.67:0/3707005258 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5c4103d40 con 0x7fc5c40756e0 2024-03-28T16:02:40.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.987+0000 7fc5bbe00640 1 -- 172.21.15.67:0/3707005258 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fc5ac004e90 con 0x7fc5c40756e0 2024-03-28T16:02:40.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.988+0000 7fc5bbe00640 1 --2- 172.21.15.67:0/3707005258 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fc5a80a2880 0x7fc5a80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:40.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.988+0000 7fc5bbe00640 1 -- 172.21.15.67:0/3707005258 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fc5ac0ce540 con 0x7fc5c40756e0 2024-03-28T16:02:40.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.988+0000 7fc5c2a00640 1 --2- 172.21.15.67:0/3707005258 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fc5a80a2880 0x7fc5a80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:40.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.989+0000 7fc5c2a00640 1 --2- 172.21.15.67:0/3707005258 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fc5a80a2880 0x7fc5a80a4d40 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fc5b4009900 tx=0x7fc5b4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:02:40.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:40.993+0000 7fc5bbe00640 1 -- 172.21.15.67:0/3707005258 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc5ac09b590 con 0x7fc5c40756e0 2024-03-28T16:02:41.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:40 smithi150 ceph-mon[34825]: pgmap v412: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:02:41.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:40 smithi150 ceph-mon[34825]: mgrmap e18: x(active, since 11m), standbys: z, y 2024-03-28T16:02:41.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:40 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/2784106710' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:02:41.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:40 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/3942273547' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T16:02:41.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.202+0000 7fc5c3e00640 1 -- 172.21.15.67:0/3707005258 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc5c40622a0 con 0x7fc5c40756e0 2024-03-28T16:02:41.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.203+0000 7fc5bbe00640 1 -- 172.21.15.67:0/3707005258 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc5ac09ae90 con 0x7fc5c40756e0 2024-03-28T16:02:41.205 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-03-28T16:02:41.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.206+0000 7fc5c3e00640 1 -- 172.21.15.67:0/3707005258 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fc5a80a2880 msgr2=0x7fc5a80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:41.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.206+0000 7fc5c3e00640 1 --2- 172.21.15.67:0/3707005258 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fc5a80a2880 0x7fc5a80a4d40 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fc5b4009900 tx=0x7fc5b4005fd0 comp rx=0 tx=0).stop 2024-03-28T16:02:41.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.206+0000 7fc5c3e00640 1 -- 172.21.15.67:0/3707005258 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5c40756e0 msgr2=0x7fc5c419e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:41.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.206+0000 7fc5c3e00640 1 --2- 172.21.15.67:0/3707005258 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5c40756e0 0x7fc5c419e370 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac00d770 tx=0x7fc5ac00dc30 comp rx=0 tx=0).stop 2024-03-28T16:02:41.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.206+0000 7fc5c3e00640 1 -- 172.21.15.67:0/3707005258 shutdown_connections 2024-03-28T16:02:41.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.206+0000 7fc5c3e00640 1 --2- 172.21.15.67:0/3707005258 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fc5a80a2880 0x7fc5a80a4d40 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:41.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.206+0000 7fc5c3e00640 1 --2- 172.21.15.67:0/3707005258 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc5c410db70 0x7fc5c41a53e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:41.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.206+0000 7fc5c3e00640 1 --2- 172.21.15.67:0/3707005258 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5c40756e0 0x7fc5c419e370 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:41.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.206+0000 7fc5c3e00640 1 --2- 172.21.15.67:0/3707005258 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5c4074100 0x7fc5c419de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:41.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.206+0000 7fc5c3e00640 1 -- 172.21.15.67:0/3707005258 >> 172.21.15.67:0/3707005258 conn(0x7fc5c40fcb10 msgr2=0x7fc5c4100f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:02:41.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.206+0000 7fc5c3e00640 1 -- 172.21.15.67:0/3707005258 shutdown_connections 2024-03-28T16:02:41.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.207+0000 7fc5c3e00640 1 -- 172.21.15.67:0/3707005258 wait complete. 2024-03-28T16:02:41.227 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T16:02:41.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.449+0000 7fcea1a00640 1 -- 172.21.15.67:0/913103874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce9c109370 msgr2=0x7fce9c10b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:41.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.449+0000 7fcea1a00640 1 --2- 172.21.15.67:0/913103874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce9c109370 0x7fce9c10b810 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fce94009ca0 tx=0x7fce9402c190 comp rx=0 tx=0).stop 2024-03-28T16:02:41.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.449+0000 7fcea1a00640 1 -- 172.21.15.67:0/913103874 shutdown_connections 2024-03-28T16:02:41.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.449+0000 7fcea1a00640 1 --2- 172.21.15.67:0/913103874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce9c109370 0x7fce9c10b810 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:41.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.449+0000 7fcea1a00640 1 --2- 172.21.15.67:0/913103874 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce9c0740a0 0x7fce9c074500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:41.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.449+0000 7fcea1a00640 1 --2- 172.21.15.67:0/913103874 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fce9c075680 0x7fce9c073ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:41.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.449+0000 7fcea1a00640 1 -- 172.21.15.67:0/913103874 >> 172.21.15.67:0/913103874 conn(0x7fce9c0fca30 msgr2=0x7fce9c0fee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:02:41.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.449+0000 7fcea1a00640 1 -- 172.21.15.67:0/913103874 shutdown_connections 2024-03-28T16:02:41.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.449+0000 7fcea1a00640 1 -- 172.21.15.67:0/913103874 wait complete. 2024-03-28T16:02:41.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.450+0000 7fcea1a00640 1 Processor -- start 2024-03-28T16:02:41.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.450+0000 7fcea1a00640 1 -- start start 2024-03-28T16:02:41.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.450+0000 7fcea1a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce9c0740a0 0x7fce9c199900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:41.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.450+0000 7fcea1a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fce9c075680 0x7fce9c199e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:41.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.450+0000 7fcea1a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce9c109370 0x7fce9c1a0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:41.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.450+0000 7fcea1a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce9c10f210 con 0x7fce9c0740a0 2024-03-28T16:02:41.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.450+0000 7fcea1a00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce9c10f090 con 0x7fce9c109370 2024-03-28T16:02:41.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.450+0000 7fcea1a00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce9c10f390 con 0x7fce9c075680 2024-03-28T16:02:41.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.451+0000 7fce9be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce9c0740a0 0x7fce9c199900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:41.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.451+0000 7fce9be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce9c0740a0 0x7fce9c199900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47720/0 (socket says 172.21.15.67:47720) 2024-03-28T16:02:41.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.451+0000 7fce9be00640 1 -- 172.21.15.67:0/1063767549 learned_addr learned my addr 172.21.15.67:0/1063767549 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:02:41.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.451+0000 7fce9be00640 1 -- 172.21.15.67:0/1063767549 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fce9c075680 msgr2=0x7fce9c199e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:41.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.451+0000 7fcea1000640 1 --2- 172.21.15.67:0/1063767549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce9c109370 0x7fce9c1a0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:41.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.451+0000 7fce9b400640 1 --2- 172.21.15.67:0/1063767549 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fce9c075680 0x7fce9c199e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:41.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.451+0000 7fce9be00640 1 --2- 172.21.15.67:0/1063767549 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fce9c075680 0x7fce9c199e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:41.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.451+0000 7fce9be00640 1 -- 172.21.15.67:0/1063767549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce9c109370 msgr2=0x7fce9c1a0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:41.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.451+0000 7fce9be00640 1 --2- 172.21.15.67:0/1063767549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce9c109370 0x7fce9c1a0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:41.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.451+0000 7fce9be00640 1 -- 172.21.15.67:0/1063767549 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce9c1a1520 con 0x7fce9c0740a0 2024-03-28T16:02:41.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.451+0000 7fce9b400640 1 --2- 172.21.15.67:0/1063767549 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fce9c075680 0x7fce9c199e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:02:41.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.451+0000 7fcea1000640 1 --2- 172.21.15.67:0/1063767549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce9c109370 0x7fce9c1a0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:02:41.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.452+0000 7fce9be00640 1 --2- 172.21.15.67:0/1063767549 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce9c0740a0 0x7fce9c199900 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fce8c00d770 tx=0x7fce8c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:02:41.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.452+0000 7fce98c00640 1 -- 172.21.15.67:0/1063767549 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce8c017070 con 0x7fce9c0740a0 2024-03-28T16:02:41.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.452+0000 7fcea1a00640 1 -- 172.21.15.67:0/1063767549 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce9c1a1810 con 0x7fce9c0740a0 2024-03-28T16:02:41.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.452+0000 7fce98c00640 1 -- 172.21.15.67:0/1063767549 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fce8c0054d0 con 0x7fce9c0740a0 2024-03-28T16:02:41.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.452+0000 7fce98c00640 1 -- 172.21.15.67:0/1063767549 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce8c004cf0 con 0x7fce9c0740a0 2024-03-28T16:02:41.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.452+0000 7fcea1a00640 1 -- 172.21.15.67:0/1063767549 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce9c1a1d60 con 0x7fce9c0740a0 2024-03-28T16:02:41.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.453+0000 7fcea1a00640 1 -- 172.21.15.67:0/1063767549 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce9c108130 con 0x7fce9c0740a0 2024-03-28T16:02:41.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.455+0000 7fce98c00640 1 -- 172.21.15.67:0/1063767549 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fce8c004e90 con 0x7fce9c0740a0 2024-03-28T16:02:41.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.456+0000 7fce98c00640 1 --2- 172.21.15.67:0/1063767549 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fce800a2880 0x7fce800a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:41.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.457+0000 7fce98c00640 1 -- 172.21.15.67:0/1063767549 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fce8c0ce540 con 0x7fce9c0740a0 2024-03-28T16:02:41.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.457+0000 7fce9b400640 1 --2- 172.21.15.67:0/1063767549 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fce800a2880 0x7fce800a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:41.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.457+0000 7fce9b400640 1 --2- 172.21.15.67:0/1063767549 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fce800a2880 0x7fce800a4d40 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fce9c19ae10 tx=0x7fce88005d30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:02:41.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.461+0000 7fce98c00640 1 -- 172.21.15.67:0/1063767549 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fce8c09b590 con 0x7fce9c0740a0 2024-03-28T16:02:41.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.711+0000 7fcea1a00640 1 -- 172.21.15.67:0/1063767549 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fce9c02d070 con 0x7fce9c0740a0 2024-03-28T16:02:41.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.713+0000 7fce98c00640 1 -- 172.21.15.67:0/1063767549 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11341 (secure 0 0 0) 0x7fce8c09ae90 con 0x7fce9c0740a0 2024-03-28T16:02:41.714 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:02:41.714 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2008297659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2008297659},{"type":"v1","addr":"172.21.15.67:6835","nonce":2008297659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14532,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/320295745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":320295745},{"type":"v1","addr":"172.21.15.67:6837","nonce":320295745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6839/691821273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":691821273},{"type":"v1","addr":"172.21.15.67:6839","nonce":691821273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6841/3211254875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6840","nonce":3211254875},{"type":"v1","addr":"172.21.15.67:6841","nonce":3211254875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2129665423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2129665423},{"type":"v1","addr":"172.21.15.183:6833","nonce":2129665423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24496,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/1589784425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":1589784425},{"type":"v1","addr":"172.21.15.183:6835","nonce":1589784425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24508,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/2185863835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":2185863835},{"type":"v1","addr":"172.21.15.183:6837","nonce":2185863835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24520,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6839/1242814398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":1242814398},{"type":"v1","addr":"172.21.15.183:6839","nonce":1242814398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":34409,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1008335521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1008335521},{"type":"v1","addr":"172.21.15.150:6833","nonce":1008335521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":34445,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3269935695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3269935695},{"type":"v1","addr":"172.21.15.150:6835","nonce":3269935695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T16:01:07.696824+0000","modified":"2024-03-28T16:01:11.997773+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":34508},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34475":{"gid":34475,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/1112886277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":1112886277},{"type":"v1","addr":"172.21.15.150:6837","nonce":1112886277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_34508":{"gid":34508,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6839/1211011614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":1211011614},{"type":"v1","addr":"172.21.15.150:6839","nonce":1211011614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34508,"qdb_cluster":[34508]},"id":1}]} 2024-03-28T16:02:41.715 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 19 2024-03-28T16:02:41.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.716+0000 7fcea1a00640 1 -- 172.21.15.67:0/1063767549 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fce800a2880 msgr2=0x7fce800a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:41.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.716+0000 7fcea1a00640 1 --2- 172.21.15.67:0/1063767549 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fce800a2880 0x7fce800a4d40 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fce9c19ae10 tx=0x7fce88005d30 comp rx=0 tx=0).stop 2024-03-28T16:02:41.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.716+0000 7fcea1a00640 1 -- 172.21.15.67:0/1063767549 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce9c0740a0 msgr2=0x7fce9c199900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:41.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.716+0000 7fcea1a00640 1 --2- 172.21.15.67:0/1063767549 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce9c0740a0 0x7fce9c199900 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fce8c00d770 tx=0x7fce8c00dc30 comp rx=0 tx=0).stop 2024-03-28T16:02:41.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.716+0000 7fcea1a00640 1 -- 172.21.15.67:0/1063767549 shutdown_connections 2024-03-28T16:02:41.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.716+0000 7fcea1a00640 1 --2- 172.21.15.67:0/1063767549 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fce800a2880 0x7fce800a4d40 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:41.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.716+0000 7fcea1a00640 1 --2- 172.21.15.67:0/1063767549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce9c109370 0x7fce9c1a0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:41.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.716+0000 7fcea1a00640 1 --2- 172.21.15.67:0/1063767549 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fce9c075680 0x7fce9c199e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:41.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.716+0000 7fcea1a00640 1 --2- 172.21.15.67:0/1063767549 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce9c0740a0 0x7fce9c199900 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:41.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.716+0000 7fcea1a00640 1 -- 172.21.15.67:0/1063767549 >> 172.21.15.67:0/1063767549 conn(0x7fce9c0fca30 msgr2=0x7fce9c105300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:02:41.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.716+0000 7fcea1a00640 1 -- 172.21.15.67:0/1063767549 shutdown_connections 2024-03-28T16:02:41.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.716+0000 7fcea1a00640 1 -- 172.21.15.67:0/1063767549 wait complete. 2024-03-28T16:02:41.739 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T16:02:41.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:41 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/3660259003' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T16:02:41.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:41 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/3707005258' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T16:02:41.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.955+0000 7f7cd7200640 1 -- 172.21.15.67:0/139166525 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cd80ffd60 msgr2=0x7f7cd8102190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:41.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.955+0000 7f7cd7200640 1 --2- 172.21.15.67:0/139166525 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cd80ffd60 0x7f7cd8102190 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f7cc8009ca0 tx=0x7f7cc802c0f0 comp rx=0 tx=0).stop 2024-03-28T16:02:41.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.955+0000 7f7cd7200640 1 -- 172.21.15.67:0/139166525 shutdown_connections 2024-03-28T16:02:41.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.955+0000 7f7cd7200640 1 --2- 172.21.15.67:0/139166525 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7cd8105080 0x7f7cd8105500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:41.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.955+0000 7f7cd7200640 1 --2- 172.21.15.67:0/139166525 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7cd81026d0 0x7f7cd8104b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:41.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.955+0000 7f7cd7200640 1 --2- 172.21.15.67:0/139166525 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cd80ffd60 0x7f7cd8102190 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:41.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.955+0000 7f7cd7200640 1 -- 172.21.15.67:0/139166525 >> 172.21.15.67:0/139166525 conn(0x7f7cd80fb980 msgr2=0x7f7cd80fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:02:41.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.956+0000 7f7cd7200640 1 -- 172.21.15.67:0/139166525 shutdown_connections 2024-03-28T16:02:41.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.956+0000 7f7cd7200640 1 -- 172.21.15.67:0/139166525 wait complete. 2024-03-28T16:02:41.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.956+0000 7f7cd7200640 1 Processor -- start 2024-03-28T16:02:41.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.957+0000 7f7cd7200640 1 -- start start 2024-03-28T16:02:41.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.957+0000 7f7cd7200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cd80ffd60 0x7f7cd819dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:41.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.957+0000 7f7cd7200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7cd81026d0 0x7f7cd819e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:41.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.957+0000 7f7cd7200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7cd8105080 0x7f7cd81a52a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:41.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.957+0000 7f7cd7200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd8113610 con 0x7f7cd80ffd60 2024-03-28T16:02:41.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.957+0000 7f7cd7200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd8113490 con 0x7f7cd8105080 2024-03-28T16:02:41.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.957+0000 7f7cd7200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd8113790 con 0x7f7cd81026d0 2024-03-28T16:02:41.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.957+0000 7f7cd5e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cd80ffd60 0x7f7cd819dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:41.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.957+0000 7f7cd5e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cd80ffd60 0x7f7cd819dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47732/0 (socket says 172.21.15.67:47732) 2024-03-28T16:02:41.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.957+0000 7f7cd5e00640 1 -- 172.21.15.67:0/2444950854 learned_addr learned my addr 172.21.15.67:0/2444950854 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:02:41.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.958+0000 7f7cd5e00640 1 -- 172.21.15.67:0/2444950854 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7cd81026d0 msgr2=0x7f7cd819e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:41.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.958+0000 7f7cd5e00640 1 --2- 172.21.15.67:0/2444950854 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7cd81026d0 0x7f7cd819e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:41.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.958+0000 7f7cd5e00640 1 -- 172.21.15.67:0/2444950854 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7cd8105080 msgr2=0x7f7cd81a52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:41.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.958+0000 7f7cd5e00640 1 --2- 172.21.15.67:0/2444950854 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7cd8105080 0x7f7cd81a52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:41.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.958+0000 7f7cd5e00640 1 -- 172.21.15.67:0/2444950854 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cd81a5940 con 0x7f7cd80ffd60 2024-03-28T16:02:41.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.958+0000 7f7cd5e00640 1 --2- 172.21.15.67:0/2444950854 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cd80ffd60 0x7f7cd819dd30 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f7cc8009dc0 tx=0x7f7cc80328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:02:41.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.958+0000 7f7ccea00640 1 -- 172.21.15.67:0/2444950854 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc8041020 con 0x7f7cd80ffd60 2024-03-28T16:02:41.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.958+0000 7f7ccea00640 1 -- 172.21.15.67:0/2444950854 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7cc8003e20 con 0x7f7cd80ffd60 2024-03-28T16:02:41.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.958+0000 7f7cd7200640 1 -- 172.21.15.67:0/2444950854 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cd81a5bd0 con 0x7f7cd80ffd60 2024-03-28T16:02:41.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.958+0000 7f7ccea00640 1 -- 172.21.15.67:0/2444950854 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc8039040 con 0x7f7cd80ffd60 2024-03-28T16:02:41.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.958+0000 7f7cd7200640 1 -- 172.21.15.67:0/2444950854 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cd81a6110 con 0x7f7cd80ffd60 2024-03-28T16:02:41.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.960+0000 7f7cd7200640 1 -- 172.21.15.67:0/2444950854 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cd80fd470 con 0x7f7cd80ffd60 2024-03-28T16:02:41.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.962+0000 7f7ccea00640 1 -- 172.21.15.67:0/2444950854 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f7cc8044020 con 0x7f7cd80ffd60 2024-03-28T16:02:41.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.963+0000 7f7ccea00640 1 --2- 172.21.15.67:0/2444950854 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7cbc0a2880 0x7f7cbc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:41.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.963+0000 7f7ccea00640 1 -- 172.21.15.67:0/2444950854 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f7cc80f0e00 con 0x7f7cd80ffd60 2024-03-28T16:02:41.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.963+0000 7f7cd5400640 1 --2- 172.21.15.67:0/2444950854 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7cbc0a2880 0x7f7cbc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:41.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.964+0000 7f7cd5400640 1 --2- 172.21.15.67:0/2444950854 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7cbc0a2880 0x7f7cbc0a4d40 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7cc0005fd0 tx=0x7f7cc0005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:02:41.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:41.968+0000 7f7ccea00640 1 -- 172.21.15.67:0/2444950854 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7cc804b020 con 0x7f7cd80ffd60 2024-03-28T16:02:42.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:41 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/3660259003' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T16:02:42.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:41 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/3707005258' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T16:02:42.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:41 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/3660259003' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T16:02:42.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:41 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/3707005258' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T16:02:42.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.170+0000 7f7cd7200640 1 -- 172.21.15.67:0/2444950854 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7cd80623e0 con 0x7f7cd80ffd60 2024-03-28T16:02:42.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.174+0000 7f7ccea00640 1 -- 172.21.15.67:0/2444950854 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) v1 ==== 74+0+22673 (secure 0 0 0) 0x7f7cc803b2f0 con 0x7f7cd80ffd60 2024-03-28T16:02:42.176 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:02:42.176 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":92,"fsid":"b5f8225a-ed1a-11ee-95d0-87774f69a715","created":"2024-03-28T15:50:21.471924+0000","modified":"2024-03-28T16:01:40.643215+0000","last_up_change":"2024-03-28T16:00:01.369123+0000","last_in_change":"2024-03-28T15:59:37.515483+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T15:54:55.588007+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T16:01:05.719801+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T16:01:07.206303+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7c1791b6-9c0e-4512-a327-cad166bb5581","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6803","nonce":1894681701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6805","nonce":1894681701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6809","nonce":1894681701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6807","nonce":1894681701}]},"public_addr":"172.21.15.67:6803/1894681701","cluster_addr":"172.21.15.67:6805/1894681701","heartbeat_back_addr":"172.21.15.67:6809/1894681701","heartbeat_front_addr":"172.21.15.67:6807/1894681701","state":["exists","up"]},{"osd":1,"uuid":"0cdda7f3-d00c-4436-bcc1-49fb8072b2e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6801","nonce":1734465762}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6803","nonce":1734465762}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6807","nonce":1734465762}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6805","nonce":1734465762}]},"public_addr":"172.21.15.150:6801/1734465762","cluster_addr":"172.21.15.150:6803/1734465762","heartbeat_back_addr":"172.21.15.150:6807/1734465762","heartbeat_front_addr":"172.21.15.150:6805/1734465762","state":["exists","up"]},{"osd":2,"uuid":"a55b9a02-c3fd-4061-95c0-53a24a17ade0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6801","nonce":3398095931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6803","nonce":3398095931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6807","nonce":3398095931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6805","nonce":3398095931}]},"public_addr":"172.21.15.183:6801/3398095931","cluster_addr":"172.21.15.183:6803/3398095931","heartbeat_back_addr":"172.21.15.183:6807/3398095931","heartbeat_front_addr":"172.21.15.183:6805/3398095931","state":["exists","up"]},{"osd":3,"uuid":"d34940c0-5ce9-4b8b-aa36-4872414a043f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6811","nonce":1933541875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6813","nonce":1933541875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6817","nonce":1933541875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6815","nonce":1933541875}]},"public_addr":"172.21.15.67:6811/1933541875","cluster_addr":"172.21.15.67:6813/1933541875","heartbeat_back_addr":"172.21.15.67:6817/1933541875","heartbeat_front_addr":"172.21.15.67:6815/1933541875","state":["exists","up"]},{"osd":4,"uuid":"5cc3e260-83a3-4a4d-aa6a-34a3b12b2b5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6809","nonce":3775269993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6811","nonce":3775269993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6815","nonce":3775269993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6813","nonce":3775269993}]},"public_addr":"172.21.15.150:6809/3775269993","cluster_addr":"172.21.15.150:6811/3775269993","heartbeat_back_addr":"172.21.15.150:6815/3775269993","heartbeat_front_addr":"172.21.15.150:6813/3775269993","state":["exists","up"]},{"osd":5,"uuid":"a2542b21-d6ac-413e-8b9b-4a6548b20840","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6809","nonce":164284137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6811","nonce":164284137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6815","nonce":164284137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6813","nonce":164284137}]},"public_addr":"172.21.15.183:6809/164284137","cluster_addr":"172.21.15.183:6811/164284137","heartbeat_back_addr":"172.21.15.183:6815/164284137","heartbeat_front_addr":"172.21.15.183:6813/164284137","state":["exists","up"]},{"osd":6,"uuid":"8905a2d3-0f1b-4853-a218-22da2635150a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6819","nonce":3565528013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6821","nonce":3565528013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6825","nonce":3565528013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6823","nonce":3565528013}]},"public_addr":"172.21.15.67:6819/3565528013","cluster_addr":"172.21.15.67:6821/3565528013","heartbeat_back_addr":"172.21.15.67:6825/3565528013","heartbeat_front_addr":"172.21.15.67:6823/3565528013","state":["exists","up"]},{"osd":7,"uuid":"b275055b-b451-4180-91b1-e6e61a828c48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6817","nonce":3735115340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6819","nonce":3735115340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6823","nonce":3735115340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6821","nonce":3735115340}]},"public_addr":"172.21.15.150:6817/3735115340","cluster_addr":"172.21.15.150:6819/3735115340","heartbeat_back_addr":"172.21.15.150:6823/3735115340","heartbeat_front_addr":"172.21.15.150:6821/3735115340","state":["exists","up"]},{"osd":8,"uuid":"b6abf7ad-eb34-4f60-a4eb-348a92d7195b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6817","nonce":1025400095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6819","nonce":1025400095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6823","nonce":1025400095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6821","nonce":1025400095}]},"public_addr":"172.21.15.183:6817/1025400095","cluster_addr":"172.21.15.183:6819/1025400095","heartbeat_back_addr":"172.21.15.183:6823/1025400095","heartbeat_front_addr":"172.21.15.183:6821/1025400095","state":["exists","up"]},{"osd":9,"uuid":"28a78e3a-0657-40aa-816f-16a6fadadcbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6827","nonce":76885409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6829","nonce":76885409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6833","nonce":76885409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6831","nonce":76885409}]},"public_addr":"172.21.15.67:6827/76885409","cluster_addr":"172.21.15.67:6829/76885409","heartbeat_back_addr":"172.21.15.67:6833/76885409","heartbeat_front_addr":"172.21.15.67:6831/76885409","state":["exists","up"]},{"osd":10,"uuid":"d287907e-424f-4b49-a06e-4613c6de0474","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6825","nonce":3489982080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6827","nonce":3489982080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6831","nonce":3489982080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6829","nonce":3489982080}]},"public_addr":"172.21.15.150:6825/3489982080","cluster_addr":"172.21.15.150:6827/3489982080","heartbeat_back_addr":"172.21.15.150:6831/3489982080","heartbeat_front_addr":"172.21.15.150:6829/3489982080","state":["exists","up"]},{"osd":11,"uuid":"a3c29c47-a228-4f42-a23c-fbdb0c093699","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":85,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6825","nonce":1892639849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6826","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6827","nonce":1892639849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6831","nonce":1892639849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6829","nonce":1892639849}]},"public_addr":"172.21.15.183:6825/1892639849","cluster_addr":"172.21.15.183:6827/1892639849","heartbeat_back_addr":"172.21.15.183:6831/1892639849","heartbeat_front_addr":"172.21.15.183:6829/1892639849","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:53:45.646211+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:54:18.956590+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:54:52.161646+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:55:27.502822+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:56:01.162932+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:56:34.674452+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:57:09.298679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:57:43.230406+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:58:17.003821+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:58:51.085863+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:59:25.665065+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T16:00:00.263457+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":1}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6801/1631001968":"2024-03-29T15:51:38.875334+0000","172.21.15.67:6800/1631001968":"2024-03-29T15:51:38.875334+0000","172.21.15.67:0/1533056310":"2024-03-29T15:51:38.875334+0000","172.21.15.67:6801/3580491071":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/2030412458":"2024-03-29T15:51:03.859740+0000","172.21.15.67:6800/3580491071":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/2905307533":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/3763759988":"2024-03-29T15:51:38.875334+0000","172.21.15.67:0/4089399880":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/4239108661":"2024-03-29T15:51:38.875334+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T16:02:42.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.177+0000 7f7cd7200640 1 -- 172.21.15.67:0/2444950854 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7cbc0a2880 msgr2=0x7f7cbc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:42.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.177+0000 7f7cd7200640 1 --2- 172.21.15.67:0/2444950854 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7cbc0a2880 0x7f7cbc0a4d40 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7cc0005fd0 tx=0x7f7cc0005850 comp rx=0 tx=0).stop 2024-03-28T16:02:42.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.177+0000 7f7cd7200640 1 -- 172.21.15.67:0/2444950854 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cd80ffd60 msgr2=0x7f7cd819dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:42.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.177+0000 7f7cd7200640 1 --2- 172.21.15.67:0/2444950854 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cd80ffd60 0x7f7cd819dd30 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f7cc8009dc0 tx=0x7f7cc80328a0 comp rx=0 tx=0).stop 2024-03-28T16:02:42.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.178+0000 7f7cd7200640 1 -- 172.21.15.67:0/2444950854 shutdown_connections 2024-03-28T16:02:42.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.178+0000 7f7cd7200640 1 --2- 172.21.15.67:0/2444950854 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7cbc0a2880 0x7f7cbc0a4d40 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:42.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.178+0000 7f7cd7200640 1 --2- 172.21.15.67:0/2444950854 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7cd8105080 0x7f7cd81a52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:42.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.178+0000 7f7cd7200640 1 --2- 172.21.15.67:0/2444950854 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7cd81026d0 0x7f7cd819e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:42.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.178+0000 7f7cd7200640 1 --2- 172.21.15.67:0/2444950854 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cd80ffd60 0x7f7cd819dd30 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:42.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.178+0000 7f7cd7200640 1 -- 172.21.15.67:0/2444950854 >> 172.21.15.67:0/2444950854 conn(0x7f7cd80fb980 msgr2=0x7f7cd8103200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:02:42.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.178+0000 7f7cd7200640 1 -- 172.21.15.67:0/2444950854 shutdown_connections 2024-03-28T16:02:42.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.178+0000 7f7cd7200640 1 -- 172.21.15.67:0/2444950854 wait complete. 2024-03-28T16:02:42.199 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T15:54:55.588007+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T16:01:05.719801+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T16:01:07.206303+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T16:02:42.199 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T16:02:42.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.428+0000 7efce9400640 1 -- 172.21.15.67:0/923080785 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efce4109770 msgr2=0x7efce4110000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:42.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.428+0000 7efce9400640 1 --2- 172.21.15.67:0/923080785 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efce4109770 0x7efce4110000 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7efcdc009ca0 tx=0x7efcdc02c100 comp rx=0 tx=0).stop 2024-03-28T16:02:42.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.428+0000 7efce9400640 1 -- 172.21.15.67:0/923080785 shutdown_connections 2024-03-28T16:02:42.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.428+0000 7efce9400640 1 --2- 172.21.15.67:0/923080785 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efce4109770 0x7efce4110000 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:42.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.428+0000 7efce9400640 1 --2- 172.21.15.67:0/923080785 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efce4108db0 0x7efce4109230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:42.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.428+0000 7efce9400640 1 --2- 172.21.15.67:0/923080785 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efce4107bc0 0x7efce4107fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:42.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.428+0000 7efce9400640 1 -- 172.21.15.67:0/923080785 >> 172.21.15.67:0/923080785 conn(0x7efce4076b70 msgr2=0x7efce4078f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:02:42.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.429+0000 7efce9400640 1 -- 172.21.15.67:0/923080785 shutdown_connections 2024-03-28T16:02:42.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.429+0000 7efce9400640 1 -- 172.21.15.67:0/923080785 wait complete. 2024-03-28T16:02:42.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.430+0000 7efce9400640 1 Processor -- start 2024-03-28T16:02:42.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.430+0000 7efce9400640 1 -- start start 2024-03-28T16:02:42.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.430+0000 7efce9400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efce4107bc0 0x7efce419ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:42.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.430+0000 7efce9400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efce4108db0 0x7efce419e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:42.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.430+0000 7efce9400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efce4109770 0x7efce41a5320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:42.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.430+0000 7efce9400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efce41136f0 con 0x7efce4109770 2024-03-28T16:02:42.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.430+0000 7efce9400640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efce4113570 con 0x7efce4107bc0 2024-03-28T16:02:42.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.430+0000 7efce9400640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efce4113870 con 0x7efce4108db0 2024-03-28T16:02:42.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.430+0000 7efce8a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efce4109770 0x7efce41a5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:42.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.430+0000 7efce8a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efce4109770 0x7efce41a5320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47752/0 (socket says 172.21.15.67:47752) 2024-03-28T16:02:42.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.430+0000 7efce8a00640 1 -- 172.21.15.67:0/871697918 learned_addr learned my addr 172.21.15.67:0/871697918 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:02:42.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.431+0000 7efce3e00640 1 --2- 172.21.15.67:0/871697918 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efce4107bc0 0x7efce419ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:42.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.431+0000 7efcdbe00640 1 --2- 172.21.15.67:0/871697918 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efce4108db0 0x7efce419e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:42.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.431+0000 7efce8a00640 1 -- 172.21.15.67:0/871697918 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efce4108db0 msgr2=0x7efce419e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:42.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.431+0000 7efce8a00640 1 --2- 172.21.15.67:0/871697918 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efce4108db0 0x7efce419e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:42.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.431+0000 7efce8a00640 1 -- 172.21.15.67:0/871697918 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efce4107bc0 msgr2=0x7efce419ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:42.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.431+0000 7efce8a00640 1 --2- 172.21.15.67:0/871697918 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efce4107bc0 0x7efce419ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:42.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.431+0000 7efce8a00640 1 -- 172.21.15.67:0/871697918 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efce41a5990 con 0x7efce4109770 2024-03-28T16:02:42.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.431+0000 7efce8a00640 1 --2- 172.21.15.67:0/871697918 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efce4109770 0x7efce41a5320 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7efcdc032520 tx=0x7efcdc034880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:02:42.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.431+0000 7efce1600640 1 -- 172.21.15.67:0/871697918 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcdc039040 con 0x7efce4109770 2024-03-28T16:02:42.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.432+0000 7efce9400640 1 -- 172.21.15.67:0/871697918 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efce41a5c20 con 0x7efce4109770 2024-03-28T16:02:42.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.432+0000 7efce1600640 1 -- 172.21.15.67:0/871697918 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7efcdc044070 con 0x7efce4109770 2024-03-28T16:02:42.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.432+0000 7efce1600640 1 -- 172.21.15.67:0/871697918 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcdc004430 con 0x7efce4109770 2024-03-28T16:02:42.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.432+0000 7efce9400640 1 -- 172.21.15.67:0/871697918 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efce41a6160 con 0x7efce4109770 2024-03-28T16:02:42.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.433+0000 7efce9400640 1 -- 172.21.15.67:0/871697918 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efce410eee0 con 0x7efce4109770 2024-03-28T16:02:42.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.435+0000 7efce1600640 1 -- 172.21.15.67:0/871697918 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7efcdc003bc0 con 0x7efce4109770 2024-03-28T16:02:42.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.436+0000 7efce1600640 1 --2- 172.21.15.67:0/871697918 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7efcc80a2880 0x7efcc80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:42.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.436+0000 7efce1600640 1 -- 172.21.15.67:0/871697918 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7efcdc0f11f0 con 0x7efce4109770 2024-03-28T16:02:42.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.436+0000 7efce3e00640 1 --2- 172.21.15.67:0/871697918 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7efcc80a2880 0x7efcc80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:42.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.437+0000 7efce3e00640 1 --2- 172.21.15.67:0/871697918 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7efcc80a2880 0x7efcc80a4d40 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7efcd4005fd0 tx=0x7efcd4005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:02:42.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.440+0000 7efce1600640 1 -- 172.21.15.67:0/871697918 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7efcdc0333a0 con 0x7efce4109770 2024-03-28T16:02:42.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.635+0000 7efce9400640 1 -- 172.21.15.67:0/871697918 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7efce40622a0 con 0x7efce4109770 2024-03-28T16:02:42.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.636+0000 7efce1600640 1 -- 172.21.15.67:0/871697918 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v92) v1 ==== 93+0+10 (secure 0 0 0) 0x7efcdc0be240 con 0x7efce4109770 2024-03-28T16:02:42.637 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-03-28T16:02:42.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.638+0000 7efce9400640 1 -- 172.21.15.67:0/871697918 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7efcc80a2880 msgr2=0x7efcc80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:42.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.638+0000 7efce9400640 1 --2- 172.21.15.67:0/871697918 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7efcc80a2880 0x7efcc80a4d40 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7efcd4005fd0 tx=0x7efcd4005850 comp rx=0 tx=0).stop 2024-03-28T16:02:42.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.638+0000 7efce9400640 1 -- 172.21.15.67:0/871697918 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efce4109770 msgr2=0x7efce41a5320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:42.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.638+0000 7efce9400640 1 --2- 172.21.15.67:0/871697918 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efce4109770 0x7efce41a5320 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7efcdc032520 tx=0x7efcdc034880 comp rx=0 tx=0).stop 2024-03-28T16:02:42.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.638+0000 7efce9400640 1 -- 172.21.15.67:0/871697918 shutdown_connections 2024-03-28T16:02:42.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.638+0000 7efce9400640 1 --2- 172.21.15.67:0/871697918 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7efcc80a2880 0x7efcc80a4d40 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:42.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.638+0000 7efce9400640 1 --2- 172.21.15.67:0/871697918 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efce4109770 0x7efce41a5320 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:42.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.638+0000 7efce9400640 1 --2- 172.21.15.67:0/871697918 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efce4108db0 0x7efce419e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:42.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.638+0000 7efce9400640 1 --2- 172.21.15.67:0/871697918 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efce4107bc0 0x7efce419ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:42.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.638+0000 7efce9400640 1 -- 172.21.15.67:0/871697918 >> 172.21.15.67:0/871697918 conn(0x7efce4076b70 msgr2=0x7efce410c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:02:42.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.639+0000 7efce9400640 1 -- 172.21.15.67:0/871697918 shutdown_connections 2024-03-28T16:02:42.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.639+0000 7efce9400640 1 -- 172.21.15.67:0/871697918 wait complete. 2024-03-28T16:02:42.658 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T16:02:42.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:42 smithi183 ceph-mon[35580]: pgmap v413: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-28T16:02:42.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:42 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/1063767549' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:02:42.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:42 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/2444950854' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:02:42.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.879+0000 7f9335a00640 1 -- 172.21.15.67:0/927669481 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93301049b0 msgr2=0x7f9330104e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:42.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.879+0000 7f9335a00640 1 --2- 172.21.15.67:0/927669481 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93301049b0 0x7f9330104e30 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f9318009900 tx=0x7f931802bc30 comp rx=0 tx=0).stop 2024-03-28T16:02:42.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.879+0000 7f9335a00640 1 -- 172.21.15.67:0/927669481 shutdown_connections 2024-03-28T16:02:42.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.879+0000 7f9335a00640 1 --2- 172.21.15.67:0/927669481 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9330105370 0x7f933010bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:42.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.879+0000 7f9335a00640 1 --2- 172.21.15.67:0/927669481 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93301049b0 0x7f9330104e30 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:42.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.879+0000 7f9335a00640 1 --2- 172.21.15.67:0/927669481 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93301037c0 0x7f9330103be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:42.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.879+0000 7f9335a00640 1 -- 172.21.15.67:0/927669481 >> 172.21.15.67:0/927669481 conn(0x7f93300fed50 msgr2=0x7f9330101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:02:42.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.880+0000 7f9335a00640 1 -- 172.21.15.67:0/927669481 shutdown_connections 2024-03-28T16:02:42.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.880+0000 7f9335a00640 1 -- 172.21.15.67:0/927669481 wait complete. 2024-03-28T16:02:42.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.881+0000 7f9335a00640 1 Processor -- start 2024-03-28T16:02:42.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.881+0000 7f9335a00640 1 -- start start 2024-03-28T16:02:42.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.881+0000 7f9335a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f93301037c0 0x7f93301999b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:42.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.881+0000 7f9335a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93301049b0 0x7f9330199ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:42.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.881+0000 7f9335a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9330105370 0x7f93301a0f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:42.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.881+0000 7f9335a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f933010f3a0 con 0x7f93301049b0 2024-03-28T16:02:42.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.881+0000 7f9335a00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f933010f220 con 0x7f9330105370 2024-03-28T16:02:42.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.881+0000 7f9335a00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f933010f520 con 0x7f93301037c0 2024-03-28T16:02:42.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.882+0000 7f9327e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93301049b0 0x7f9330199ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:42.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.882+0000 7f9327e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93301049b0 0x7f9330199ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47766/0 (socket says 172.21.15.67:47766) 2024-03-28T16:02:42.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.882+0000 7f9327e00640 1 -- 172.21.15.67:0/699862638 learned_addr learned my addr 172.21.15.67:0/699862638 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:02:42.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.882+0000 7f9335000640 1 --2- 172.21.15.67:0/699862638 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9330105370 0x7f93301a0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:42.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.882+0000 7f932fe00640 1 --2- 172.21.15.67:0/699862638 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f93301037c0 0x7f93301999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:42.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.882+0000 7f9327e00640 1 -- 172.21.15.67:0/699862638 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f93301037c0 msgr2=0x7f93301999b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:42.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.882+0000 7f9327e00640 1 --2- 172.21.15.67:0/699862638 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f93301037c0 0x7f93301999b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:42.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.882+0000 7f9327e00640 1 -- 172.21.15.67:0/699862638 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9330105370 msgr2=0x7f93301a0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:42.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.882+0000 7f9327e00640 1 --2- 172.21.15.67:0/699862638 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9330105370 0x7f93301a0f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:42.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.882+0000 7f9327e00640 1 -- 172.21.15.67:0/699862638 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93301a15d0 con 0x7f93301049b0 2024-03-28T16:02:42.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.882+0000 7f9327e00640 1 --2- 172.21.15.67:0/699862638 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93301049b0 0x7f9330199ef0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f9318000c00 tx=0x7f93180042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:02:42.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.883+0000 7f932d600640 1 -- 172.21.15.67:0/699862638 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9318043020 con 0x7f93301049b0 2024-03-28T16:02:42.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.883+0000 7f932d600640 1 -- 172.21.15.67:0/699862638 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f93180054c0 con 0x7f93301049b0 2024-03-28T16:02:42.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.883+0000 7f932d600640 1 -- 172.21.15.67:0/699862638 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9318003d80 con 0x7f93301049b0 2024-03-28T16:02:42.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.883+0000 7f9335a00640 1 -- 172.21.15.67:0/699862638 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93301a1860 con 0x7f93301049b0 2024-03-28T16:02:42.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.883+0000 7f9335a00640 1 -- 172.21.15.67:0/699862638 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93301a1de0 con 0x7f93301049b0 2024-03-28T16:02:42.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.884+0000 7f9335a00640 1 -- 172.21.15.67:0/699862638 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f933010aae0 con 0x7f93301049b0 2024-03-28T16:02:42.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.886+0000 7f932d600640 1 -- 172.21.15.67:0/699862638 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f9318046020 con 0x7f93301049b0 2024-03-28T16:02:42.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.888+0000 7f932d600640 1 --2- 172.21.15.67:0/699862638 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f93140a2880 0x7f93140a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:42.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.889+0000 7f932d600640 1 -- 172.21.15.67:0/699862638 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f93180f0f10 con 0x7f93301049b0 2024-03-28T16:02:42.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.889+0000 7f932fe00640 1 --2- 172.21.15.67:0/699862638 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f93140a2880 0x7f93140a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:42.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.889+0000 7f932fe00640 1 --2- 172.21.15.67:0/699862638 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f93140a2880 0x7f93140a4d40 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f93200058c0 tx=0x7f9320005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:02:42.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:42.893+0000 7f932d600640 1 -- 172.21.15.67:0/699862638 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f931804d020 con 0x7f93301049b0 2024-03-28T16:02:43.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:42 smithi150 ceph-mon[34825]: pgmap v413: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-28T16:02:43.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:42 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/1063767549' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:02:43.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:42 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/2444950854' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:02:43.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:42 smithi067 ceph-mon[31707]: pgmap v413: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-28T16:02:43.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:42 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/1063767549' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:02:43.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:42 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/2444950854' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:02:43.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.098+0000 7f9335a00640 1 -- 172.21.15.67:0/699862638 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f93300622a0 con 0x7f93301049b0 2024-03-28T16:02:43.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.099+0000 7f932d600640 1 -- 172.21.15.67:0/699862638 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v92) v1 ==== 104+0+11 (secure 0 0 0) 0x7f93180be080 con 0x7f93301049b0 2024-03-28T16:02:43.100 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-03-28T16:02:43.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.102+0000 7f9335a00640 1 -- 172.21.15.67:0/699862638 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f93140a2880 msgr2=0x7f93140a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:43.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.102+0000 7f9335a00640 1 --2- 172.21.15.67:0/699862638 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f93140a2880 0x7f93140a4d40 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f93200058c0 tx=0x7f9320005850 comp rx=0 tx=0).stop 2024-03-28T16:02:43.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.102+0000 7f9335a00640 1 -- 172.21.15.67:0/699862638 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93301049b0 msgr2=0x7f9330199ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:43.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.102+0000 7f9335a00640 1 --2- 172.21.15.67:0/699862638 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93301049b0 0x7f9330199ef0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f9318000c00 tx=0x7f93180042f0 comp rx=0 tx=0).stop 2024-03-28T16:02:43.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.102+0000 7f9335a00640 1 -- 172.21.15.67:0/699862638 shutdown_connections 2024-03-28T16:02:43.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.102+0000 7f9335a00640 1 --2- 172.21.15.67:0/699862638 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f93140a2880 0x7f93140a4d40 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:43.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.102+0000 7f9335a00640 1 --2- 172.21.15.67:0/699862638 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9330105370 0x7f93301a0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:43.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.102+0000 7f9335a00640 1 --2- 172.21.15.67:0/699862638 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93301049b0 0x7f9330199ef0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:43.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.102+0000 7f9335a00640 1 --2- 172.21.15.67:0/699862638 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f93301037c0 0x7f93301999b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:43.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.102+0000 7f9335a00640 1 -- 172.21.15.67:0/699862638 >> 172.21.15.67:0/699862638 conn(0x7f93300fed50 msgr2=0x7f9330107cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:02:43.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.103+0000 7f9335a00640 1 -- 172.21.15.67:0/699862638 shutdown_connections 2024-03-28T16:02:43.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.103+0000 7f9335a00640 1 -- 172.21.15.67:0/699862638 wait complete. 2024-03-28T16:02:43.124 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T16:02:43.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.342+0000 7fd749400640 1 -- 172.21.15.67:0/1047431791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd744101090 msgr2=0x7fd744101910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:43.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.342+0000 7fd749400640 1 --2- 172.21.15.67:0/1047431791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd744101090 0x7fd744101910 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fd73c009ca0 tx=0x7fd73c02c0f0 comp rx=0 tx=0).stop 2024-03-28T16:02:43.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.342+0000 7fd749400640 1 -- 172.21.15.67:0/1047431791 shutdown_connections 2024-03-28T16:02:43.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.342+0000 7fd749400640 1 --2- 172.21.15.67:0/1047431791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd744101090 0x7fd744101910 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:43.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.342+0000 7fd749400640 1 --2- 172.21.15.67:0/1047431791 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd7441006d0 0x7fd744100b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:43.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.342+0000 7fd749400640 1 --2- 172.21.15.67:0/1047431791 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7440ff970 0x7fd7440ffd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:43.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.342+0000 7fd749400640 1 -- 172.21.15.67:0/1047431791 >> 172.21.15.67:0/1047431791 conn(0x7fd7440fb2c0 msgr2=0x7fd7440fd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:02:43.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.342+0000 7fd749400640 1 -- 172.21.15.67:0/1047431791 shutdown_connections 2024-03-28T16:02:43.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.343+0000 7fd749400640 1 -- 172.21.15.67:0/1047431791 wait complete. 2024-03-28T16:02:43.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.343+0000 7fd749400640 1 Processor -- start 2024-03-28T16:02:43.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.344+0000 7fd749400640 1 -- start start 2024-03-28T16:02:43.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.344+0000 7fd749400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd7440ff970 0x7fd744199ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:43.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.344+0000 7fd749400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7441006d0 0x7fd74419a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:43.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.344+0000 7fd749400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd744101090 0x7fd7441a1070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:43.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.344+0000 7fd749400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd74410f400 con 0x7fd744101090 2024-03-28T16:02:43.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.344+0000 7fd749400640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd74410f280 con 0x7fd7441006d0 2024-03-28T16:02:43.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.344+0000 7fd749400640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd74410f580 con 0x7fd7440ff970 2024-03-28T16:02:43.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.344+0000 7fd743200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd744101090 0x7fd7441a1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:43.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.344+0000 7fd743200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd744101090 0x7fd7441a1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47794/0 (socket says 172.21.15.67:47794) 2024-03-28T16:02:43.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.344+0000 7fd743200640 1 -- 172.21.15.67:0/296028676 learned_addr learned my addr 172.21.15.67:0/296028676 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:02:43.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.344+0000 7fd741e00640 1 --2- 172.21.15.67:0/296028676 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7441006d0 0x7fd74419a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:43.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.345+0000 7fd742800640 1 --2- 172.21.15.67:0/296028676 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd7440ff970 0x7fd744199ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:43.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.345+0000 7fd743200640 1 -- 172.21.15.67:0/296028676 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd7440ff970 msgr2=0x7fd744199ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:43.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.345+0000 7fd743200640 1 --2- 172.21.15.67:0/296028676 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd7440ff970 0x7fd744199ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:43.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.345+0000 7fd743200640 1 -- 172.21.15.67:0/296028676 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7441006d0 msgr2=0x7fd74419a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:43.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.345+0000 7fd743200640 1 --2- 172.21.15.67:0/296028676 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7441006d0 0x7fd74419a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:43.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.345+0000 7fd743200640 1 -- 172.21.15.67:0/296028676 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7441a16e0 con 0x7fd744101090 2024-03-28T16:02:43.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.345+0000 7fd741e00640 1 --2- 172.21.15.67:0/296028676 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7441006d0 0x7fd74419a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:02:43.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.345+0000 7fd742800640 1 --2- 172.21.15.67:0/296028676 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd7440ff970 0x7fd744199ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:02:43.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.345+0000 7fd743200640 1 --2- 172.21.15.67:0/296028676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd744101090 0x7fd7441a1070 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fd73c032380 tx=0x7fd73c032820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:02:43.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.345+0000 7fd73b400640 1 -- 172.21.15.67:0/296028676 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd73c041020 con 0x7fd744101090 2024-03-28T16:02:43.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.345+0000 7fd73b400640 1 -- 172.21.15.67:0/296028676 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd73c003e20 con 0x7fd744101090 2024-03-28T16:02:43.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.345+0000 7fd73b400640 1 -- 172.21.15.67:0/296028676 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd73c039040 con 0x7fd744101090 2024-03-28T16:02:43.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.345+0000 7fd749400640 1 -- 172.21.15.67:0/296028676 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7441a1970 con 0x7fd744101090 2024-03-28T16:02:43.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.345+0000 7fd749400640 1 -- 172.21.15.67:0/296028676 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7441a1eb0 con 0x7fd744101090 2024-03-28T16:02:43.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.347+0000 7fd749400640 1 -- 172.21.15.67:0/296028676 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7440688e0 con 0x7fd744101090 2024-03-28T16:02:43.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.348+0000 7fd73b400640 1 -- 172.21.15.67:0/296028676 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fd73c044020 con 0x7fd744101090 2024-03-28T16:02:43.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.350+0000 7fd73b400640 1 --2- 172.21.15.67:0/296028676 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fd7280a2880 0x7fd7280a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:43.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.350+0000 7fd73b400640 1 -- 172.21.15.67:0/296028676 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fd73c0f0c30 con 0x7fd744101090 2024-03-28T16:02:43.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.350+0000 7fd742800640 1 --2- 172.21.15.67:0/296028676 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fd7280a2880 0x7fd7280a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:43.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.351+0000 7fd742800640 1 --2- 172.21.15.67:0/296028676 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fd7280a2880 0x7fd7280a4d40 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd734009b50 tx=0x7fd734005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:02:43.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.354+0000 7fd73b400640 1 -- 172.21.15.67:0/296028676 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd73c04b020 con 0x7fd744101090 2024-03-28T16:02:43.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.556+0000 7fd749400640 1 -- 172.21.15.67:0/296028676 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd7440622a0 con 0x7fd744101090 2024-03-28T16:02:43.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.557+0000 7fd73b400640 1 -- 172.21.15.67:0/296028676 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd73c0bdc80 con 0x7fd744101090 2024-03-28T16:02:43.559 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-03-28T16:02:43.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.560+0000 7fd749400640 1 -- 172.21.15.67:0/296028676 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fd7280a2880 msgr2=0x7fd7280a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:43.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.560+0000 7fd749400640 1 --2- 172.21.15.67:0/296028676 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fd7280a2880 0x7fd7280a4d40 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd734009b50 tx=0x7fd734005f90 comp rx=0 tx=0).stop 2024-03-28T16:02:43.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.560+0000 7fd749400640 1 -- 172.21.15.67:0/296028676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd744101090 msgr2=0x7fd7441a1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:43.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.560+0000 7fd749400640 1 --2- 172.21.15.67:0/296028676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd744101090 0x7fd7441a1070 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fd73c032380 tx=0x7fd73c032820 comp rx=0 tx=0).stop 2024-03-28T16:02:43.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.560+0000 7fd749400640 1 -- 172.21.15.67:0/296028676 shutdown_connections 2024-03-28T16:02:43.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.560+0000 7fd749400640 1 --2- 172.21.15.67:0/296028676 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fd7280a2880 0x7fd7280a4d40 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:43.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.560+0000 7fd749400640 1 --2- 172.21.15.67:0/296028676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd744101090 0x7fd7441a1070 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:43.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.560+0000 7fd749400640 1 --2- 172.21.15.67:0/296028676 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7441006d0 0x7fd74419a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:43.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.560+0000 7fd749400640 1 --2- 172.21.15.67:0/296028676 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd7440ff970 0x7fd744199ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:43.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.560+0000 7fd749400640 1 -- 172.21.15.67:0/296028676 >> 172.21.15.67:0/296028676 conn(0x7fd7440fb2c0 msgr2=0x7fd7440fcdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:02:43.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.561+0000 7fd749400640 1 -- 172.21.15.67:0/296028676 shutdown_connections 2024-03-28T16:02:43.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.561+0000 7fd749400640 1 -- 172.21.15.67:0/296028676 wait complete. 2024-03-28T16:02:43.582 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T16:02:43.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.795+0000 7ff944c00640 1 -- 172.21.15.67:0/3627811581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9401071d0 msgr2=0x7ff940109620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:43.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.795+0000 7ff944c00640 1 --2- 172.21.15.67:0/3627811581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9401071d0 0x7ff940109620 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7ff938009900 tx=0x7ff93802bc30 comp rx=0 tx=0).stop 2024-03-28T16:02:43.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.795+0000 7ff944c00640 1 -- 172.21.15.67:0/3627811581 shutdown_connections 2024-03-28T16:02:43.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.795+0000 7ff944c00640 1 --2- 172.21.15.67:0/3627811581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9401071d0 0x7ff940109620 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:43.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.795+0000 7ff944c00640 1 --2- 172.21.15.67:0/3627811581 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff9401048a0 0x7ff940106c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:43.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.795+0000 7ff944c00640 1 --2- 172.21.15.67:0/3627811581 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff940101f70 0x7ff940104360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:43.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.795+0000 7ff944c00640 1 -- 172.21.15.67:0/3627811581 >> 172.21.15.67:0/3627811581 conn(0x7ff9400fb980 msgr2=0x7ff9400fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:02:43.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.795+0000 7ff944c00640 1 -- 172.21.15.67:0/3627811581 shutdown_connections 2024-03-28T16:02:43.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.795+0000 7ff944c00640 1 -- 172.21.15.67:0/3627811581 wait complete. 2024-03-28T16:02:43.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.796+0000 7ff944c00640 1 Processor -- start 2024-03-28T16:02:43.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.796+0000 7ff944c00640 1 -- start start 2024-03-28T16:02:43.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.797+0000 7ff944c00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff940101f70 0x7ff9401018e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:43.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.797+0000 7ff944c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9401048a0 0x7ff9400fff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:43.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.797+0000 7ff944c00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9401071d0 0x7ff940100460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:43.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.797+0000 7ff944c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff94010d080 con 0x7ff9401048a0 2024-03-28T16:02:43.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.797+0000 7ff944c00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff94010cf00 con 0x7ff9401071d0 2024-03-28T16:02:43.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.797+0000 7ff944c00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff94010d200 con 0x7ff940101f70 2024-03-28T16:02:43.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.797+0000 7ff936a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9401048a0 0x7ff9400fff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:43.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.797+0000 7ff936a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9401048a0 0x7ff9400fff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47824/0 (socket says 172.21.15.67:47824) 2024-03-28T16:02:43.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.797+0000 7ff936a00640 1 -- 172.21.15.67:0/981644889 learned_addr learned my addr 172.21.15.67:0/981644889 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:02:43.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.797+0000 7ff93f400640 1 --2- 172.21.15.67:0/981644889 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff940101f70 0x7ff9401018e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:43.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.797+0000 7ff93fe00640 1 --2- 172.21.15.67:0/981644889 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9401071d0 0x7ff940100460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:43.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.797+0000 7ff936a00640 1 -- 172.21.15.67:0/981644889 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff940101f70 msgr2=0x7ff9401018e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:43.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.797+0000 7ff936a00640 1 --2- 172.21.15.67:0/981644889 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff940101f70 0x7ff9401018e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:43.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.797+0000 7ff936a00640 1 -- 172.21.15.67:0/981644889 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9401071d0 msgr2=0x7ff940100460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:43.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.797+0000 7ff936a00640 1 --2- 172.21.15.67:0/981644889 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9401071d0 0x7ff940100460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:43.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.797+0000 7ff936a00640 1 -- 172.21.15.67:0/981644889 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff940100cf0 con 0x7ff9401048a0 2024-03-28T16:02:43.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.798+0000 7ff936a00640 1 --2- 172.21.15.67:0/981644889 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9401048a0 0x7ff9400fff20 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7ff92800d740 tx=0x7ff92800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:02:43.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.798+0000 7ff93cc00640 1 -- 172.21.15.67:0/981644889 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff928017070 con 0x7ff9401048a0 2024-03-28T16:02:43.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.798+0000 7ff93cc00640 1 -- 172.21.15.67:0/981644889 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff92800de50 con 0x7ff9401048a0 2024-03-28T16:02:43.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.798+0000 7ff93cc00640 1 -- 172.21.15.67:0/981644889 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff928004d60 con 0x7ff9401048a0 2024-03-28T16:02:43.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.798+0000 7ff944c00640 1 -- 172.21.15.67:0/981644889 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9401a17e0 con 0x7ff9401048a0 2024-03-28T16:02:43.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.798+0000 7ff944c00640 1 -- 172.21.15.67:0/981644889 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9401a1ce0 con 0x7ff9401048a0 2024-03-28T16:02:43.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.799+0000 7ff944c00640 1 -- 172.21.15.67:0/981644889 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9400fd470 con 0x7ff9401048a0 2024-03-28T16:02:43.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.801+0000 7ff93cc00640 1 -- 172.21.15.67:0/981644889 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ff928004f00 con 0x7ff9401048a0 2024-03-28T16:02:43.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.803+0000 7ff93cc00640 1 --2- 172.21.15.67:0/981644889 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7ff9240a2880 0x7ff9240a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:43.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.803+0000 7ff93cc00640 1 -- 172.21.15.67:0/981644889 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7ff9280ce400 con 0x7ff9401048a0 2024-03-28T16:02:43.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.803+0000 7ff93f400640 1 --2- 172.21.15.67:0/981644889 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7ff9240a2880 0x7ff9240a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:43.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.804+0000 7ff93f400640 1 --2- 172.21.15.67:0/981644889 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7ff9240a2880 0x7ff9240a4d40 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff930005fd0 tx=0x7ff930009520 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:02:43.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:43.807+0000 7ff93cc00640 1 -- 172.21.15.67:0/981644889 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff92809b450 con 0x7ff9401048a0 2024-03-28T16:02:43.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:43 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/871697918' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T16:02:43.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:43 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/699862638' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T16:02:43.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:43 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/296028676' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T16:02:43.889 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:43 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/871697918' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T16:02:43.889 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:43 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/699862638' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T16:02:43.889 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:43 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/296028676' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T16:02:44.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.049+0000 7ff944c00640 1 -- 172.21.15.67:0/981644889 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff9400622a0 con 0x7ff9401048a0 2024-03-28T16:02:44.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.052+0000 7ff93cc00640 1 -- 172.21.15.67:0/981644889 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11341 (secure 0 0 0) 0x7ff92809ad50 con 0x7ff9401048a0 2024-03-28T16:02:44.053 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:02:44.053 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2008297659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2008297659},{"type":"v1","addr":"172.21.15.67:6835","nonce":2008297659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14532,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/320295745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":320295745},{"type":"v1","addr":"172.21.15.67:6837","nonce":320295745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6839/691821273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":691821273},{"type":"v1","addr":"172.21.15.67:6839","nonce":691821273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6841/3211254875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6840","nonce":3211254875},{"type":"v1","addr":"172.21.15.67:6841","nonce":3211254875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2129665423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2129665423},{"type":"v1","addr":"172.21.15.183:6833","nonce":2129665423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24496,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/1589784425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":1589784425},{"type":"v1","addr":"172.21.15.183:6835","nonce":1589784425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24508,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/2185863835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":2185863835},{"type":"v1","addr":"172.21.15.183:6837","nonce":2185863835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24520,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6839/1242814398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":1242814398},{"type":"v1","addr":"172.21.15.183:6839","nonce":1242814398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":34409,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1008335521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1008335521},{"type":"v1","addr":"172.21.15.150:6833","nonce":1008335521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":34445,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3269935695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3269935695},{"type":"v1","addr":"172.21.15.150:6835","nonce":3269935695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T16:01:07.696824+0000","modified":"2024-03-28T16:01:11.997773+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":34508},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34475":{"gid":34475,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/1112886277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":1112886277},{"type":"v1","addr":"172.21.15.150:6837","nonce":1112886277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_34508":{"gid":34508,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6839/1211011614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":1211011614},{"type":"v1","addr":"172.21.15.150:6839","nonce":1211011614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34508,"qdb_cluster":[34508]},"id":1}]} 2024-03-28T16:02:44.053 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 19 2024-03-28T16:02:44.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.053+0000 7ff944c00640 1 -- 172.21.15.67:0/981644889 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7ff9240a2880 msgr2=0x7ff9240a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:44.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.053+0000 7ff944c00640 1 --2- 172.21.15.67:0/981644889 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7ff9240a2880 0x7ff9240a4d40 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff930005fd0 tx=0x7ff930009520 comp rx=0 tx=0).stop 2024-03-28T16:02:44.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.053+0000 7ff944c00640 1 -- 172.21.15.67:0/981644889 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9401048a0 msgr2=0x7ff9400fff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:44.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.053+0000 7ff944c00640 1 --2- 172.21.15.67:0/981644889 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9401048a0 0x7ff9400fff20 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7ff92800d740 tx=0x7ff92800dc00 comp rx=0 tx=0).stop 2024-03-28T16:02:44.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.055+0000 7ff944c00640 1 -- 172.21.15.67:0/981644889 shutdown_connections 2024-03-28T16:02:44.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.055+0000 7ff944c00640 1 --2- 172.21.15.67:0/981644889 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7ff9240a2880 0x7ff9240a4d40 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:44.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.055+0000 7ff944c00640 1 --2- 172.21.15.67:0/981644889 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9401071d0 0x7ff940100460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:44.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.055+0000 7ff944c00640 1 --2- 172.21.15.67:0/981644889 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9401048a0 0x7ff9400fff20 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:44.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.055+0000 7ff944c00640 1 --2- 172.21.15.67:0/981644889 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff940101f70 0x7ff9401018e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:44.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.055+0000 7ff944c00640 1 -- 172.21.15.67:0/981644889 >> 172.21.15.67:0/981644889 conn(0x7ff9400fb980 msgr2=0x7ff9401028b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:02:44.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.055+0000 7ff944c00640 1 -- 172.21.15.67:0/981644889 shutdown_connections 2024-03-28T16:02:44.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.055+0000 7ff944c00640 1 -- 172.21.15.67:0/981644889 wait complete. 2024-03-28T16:02:44.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:43 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/871697918' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T16:02:44.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:43 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/699862638' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T16:02:44.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:43 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/296028676' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T16:02:44.076 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T16:02:44.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.309+0000 7fa0dee00640 1 -- 172.21.15.67:0/2948665849 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0d8074100 msgr2=0x7fa0d8074580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:44.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.309+0000 7fa0dee00640 1 --2- 172.21.15.67:0/2948665849 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0d8074100 0x7fa0d8074580 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fa0c4009930 tx=0x7fa0c402bc20 comp rx=0 tx=0).stop 2024-03-28T16:02:44.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.309+0000 7fa0dee00640 1 -- 172.21.15.67:0/2948665849 shutdown_connections 2024-03-28T16:02:44.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.309+0000 7fa0dee00640 1 --2- 172.21.15.67:0/2948665849 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa0d810dae0 0x7fa0d810fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:44.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.309+0000 7fa0dee00640 1 --2- 172.21.15.67:0/2948665849 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0d8074100 0x7fa0d8074580 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:44.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.309+0000 7fa0dee00640 1 --2- 172.21.15.67:0/2948665849 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0d80756e0 0x7fa0d8073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:44.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.309+0000 7fa0dee00640 1 -- 172.21.15.67:0/2948665849 >> 172.21.15.67:0/2948665849 conn(0x7fa0d80fca80 msgr2=0x7fa0d80feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:02:44.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.310+0000 7fa0dee00640 1 -- 172.21.15.67:0/2948665849 shutdown_connections 2024-03-28T16:02:44.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.310+0000 7fa0dee00640 1 -- 172.21.15.67:0/2948665849 wait complete. 2024-03-28T16:02:44.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.311+0000 7fa0dee00640 1 Processor -- start 2024-03-28T16:02:44.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.311+0000 7fa0dee00640 1 -- start start 2024-03-28T16:02:44.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.311+0000 7fa0dee00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa0d8074100 0x7fa0d819ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:44.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.311+0000 7fa0dee00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0d80756e0 0x7fa0d819e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:44.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.311+0000 7fa0dee00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0d810dae0 0x7fa0d81a5320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:44.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.311+0000 7fa0dee00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0d8113660 con 0x7fa0d80756e0 2024-03-28T16:02:44.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.311+0000 7fa0dee00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0d81134e0 con 0x7fa0d810dae0 2024-03-28T16:02:44.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.311+0000 7fa0dee00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0d81137e0 con 0x7fa0d8074100 2024-03-28T16:02:44.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.311+0000 7fa0dd000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0d80756e0 0x7fa0d819e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:44.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.312+0000 7fa0dd000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0d80756e0 0x7fa0d819e2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47836/0 (socket says 172.21.15.67:47836) 2024-03-28T16:02:44.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.312+0000 7fa0dd000640 1 -- 172.21.15.67:0/2352925141 learned_addr learned my addr 172.21.15.67:0/2352925141 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:02:44.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.312+0000 7fa0dda00640 1 --2- 172.21.15.67:0/2352925141 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa0d8074100 0x7fa0d819ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:44.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.312+0000 7fa0de400640 1 --2- 172.21.15.67:0/2352925141 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0d810dae0 0x7fa0d81a5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:44.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.312+0000 7fa0dd000640 1 -- 172.21.15.67:0/2352925141 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa0d8074100 msgr2=0x7fa0d819ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:44.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.312+0000 7fa0dd000640 1 --2- 172.21.15.67:0/2352925141 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa0d8074100 0x7fa0d819ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:44.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.312+0000 7fa0dd000640 1 -- 172.21.15.67:0/2352925141 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0d810dae0 msgr2=0x7fa0d81a5320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:44.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.312+0000 7fa0dd000640 1 --2- 172.21.15.67:0/2352925141 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0d810dae0 0x7fa0d81a5320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:44.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.312+0000 7fa0dd000640 1 -- 172.21.15.67:0/2352925141 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0d81a59c0 con 0x7fa0d80756e0 2024-03-28T16:02:44.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.312+0000 7fa0de400640 1 --2- 172.21.15.67:0/2352925141 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0d810dae0 0x7fa0d81a5320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:02:44.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.312+0000 7fa0dd000640 1 --2- 172.21.15.67:0/2352925141 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0d80756e0 0x7fa0d819e2f0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fa0c4005ab0 tx=0x7fa0c40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:02:44.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.313+0000 7fa0d6000640 1 -- 172.21.15.67:0/2352925141 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0c4043020 con 0x7fa0d80756e0 2024-03-28T16:02:44.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.313+0000 7fa0d6000640 1 -- 172.21.15.67:0/2352925141 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa0c40054c0 con 0x7fa0d80756e0 2024-03-28T16:02:44.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.313+0000 7fa0dee00640 1 -- 172.21.15.67:0/2352925141 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0d81a5c50 con 0x7fa0d80756e0 2024-03-28T16:02:44.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.313+0000 7fa0d6000640 1 -- 172.21.15.67:0/2352925141 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0c4003d10 con 0x7fa0d80756e0 2024-03-28T16:02:44.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.313+0000 7fa0dee00640 1 -- 172.21.15.67:0/2352925141 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0d81a61a0 con 0x7fa0d80756e0 2024-03-28T16:02:44.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.314+0000 7fa0dee00640 1 -- 172.21.15.67:0/2352925141 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0d8103cb0 con 0x7fa0d80756e0 2024-03-28T16:02:44.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.316+0000 7fa0d6000640 1 -- 172.21.15.67:0/2352925141 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fa0c4046020 con 0x7fa0d80756e0 2024-03-28T16:02:44.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.317+0000 7fa0d6000640 1 --2- 172.21.15.67:0/2352925141 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fa0c00a2800 0x7fa0c00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:44.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.318+0000 7fa0d6000640 1 -- 172.21.15.67:0/2352925141 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fa0c40f0e20 con 0x7fa0d80756e0 2024-03-28T16:02:44.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.318+0000 7fa0dda00640 1 --2- 172.21.15.67:0/2352925141 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fa0c00a2800 0x7fa0c00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:44.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.318+0000 7fa0dda00640 1 --2- 172.21.15.67:0/2352925141 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fa0c00a2800 0x7fa0c00a4cc0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fa0cc005fd0 tx=0x7fa0cc005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:02:44.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.322+0000 7fa0d6000640 1 -- 172.21.15.67:0/2352925141 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa0c404d020 con 0x7fa0d80756e0 2024-03-28T16:02:44.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.531+0000 7fa0dee00640 1 -- 172.21.15.67:0/2352925141 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa0d80622d0 con 0x7fa0d80756e0 2024-03-28T16:02:44.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.536+0000 7fa0d6000640 1 -- 172.21.15.67:0/2352925141 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) v1 ==== 74+0+22673 (secure 0 0 0) 0x7fa0c40bde70 con 0x7fa0d80756e0 2024-03-28T16:02:44.537 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:02:44.537 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":92,"fsid":"b5f8225a-ed1a-11ee-95d0-87774f69a715","created":"2024-03-28T15:50:21.471924+0000","modified":"2024-03-28T16:01:40.643215+0000","last_up_change":"2024-03-28T16:00:01.369123+0000","last_in_change":"2024-03-28T15:59:37.515483+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T15:54:55.588007+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T16:01:05.719801+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T16:01:07.206303+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7c1791b6-9c0e-4512-a327-cad166bb5581","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6803","nonce":1894681701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6805","nonce":1894681701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6809","nonce":1894681701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6807","nonce":1894681701}]},"public_addr":"172.21.15.67:6803/1894681701","cluster_addr":"172.21.15.67:6805/1894681701","heartbeat_back_addr":"172.21.15.67:6809/1894681701","heartbeat_front_addr":"172.21.15.67:6807/1894681701","state":["exists","up"]},{"osd":1,"uuid":"0cdda7f3-d00c-4436-bcc1-49fb8072b2e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6801","nonce":1734465762}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6803","nonce":1734465762}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6807","nonce":1734465762}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6805","nonce":1734465762}]},"public_addr":"172.21.15.150:6801/1734465762","cluster_addr":"172.21.15.150:6803/1734465762","heartbeat_back_addr":"172.21.15.150:6807/1734465762","heartbeat_front_addr":"172.21.15.150:6805/1734465762","state":["exists","up"]},{"osd":2,"uuid":"a55b9a02-c3fd-4061-95c0-53a24a17ade0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6801","nonce":3398095931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6803","nonce":3398095931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6807","nonce":3398095931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6805","nonce":3398095931}]},"public_addr":"172.21.15.183:6801/3398095931","cluster_addr":"172.21.15.183:6803/3398095931","heartbeat_back_addr":"172.21.15.183:6807/3398095931","heartbeat_front_addr":"172.21.15.183:6805/3398095931","state":["exists","up"]},{"osd":3,"uuid":"d34940c0-5ce9-4b8b-aa36-4872414a043f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6811","nonce":1933541875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6813","nonce":1933541875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6817","nonce":1933541875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6815","nonce":1933541875}]},"public_addr":"172.21.15.67:6811/1933541875","cluster_addr":"172.21.15.67:6813/1933541875","heartbeat_back_addr":"172.21.15.67:6817/1933541875","heartbeat_front_addr":"172.21.15.67:6815/1933541875","state":["exists","up"]},{"osd":4,"uuid":"5cc3e260-83a3-4a4d-aa6a-34a3b12b2b5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6809","nonce":3775269993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6811","nonce":3775269993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6815","nonce":3775269993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6813","nonce":3775269993}]},"public_addr":"172.21.15.150:6809/3775269993","cluster_addr":"172.21.15.150:6811/3775269993","heartbeat_back_addr":"172.21.15.150:6815/3775269993","heartbeat_front_addr":"172.21.15.150:6813/3775269993","state":["exists","up"]},{"osd":5,"uuid":"a2542b21-d6ac-413e-8b9b-4a6548b20840","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6809","nonce":164284137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6811","nonce":164284137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6815","nonce":164284137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6813","nonce":164284137}]},"public_addr":"172.21.15.183:6809/164284137","cluster_addr":"172.21.15.183:6811/164284137","heartbeat_back_addr":"172.21.15.183:6815/164284137","heartbeat_front_addr":"172.21.15.183:6813/164284137","state":["exists","up"]},{"osd":6,"uuid":"8905a2d3-0f1b-4853-a218-22da2635150a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6819","nonce":3565528013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6821","nonce":3565528013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6825","nonce":3565528013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6823","nonce":3565528013}]},"public_addr":"172.21.15.67:6819/3565528013","cluster_addr":"172.21.15.67:6821/3565528013","heartbeat_back_addr":"172.21.15.67:6825/3565528013","heartbeat_front_addr":"172.21.15.67:6823/3565528013","state":["exists","up"]},{"osd":7,"uuid":"b275055b-b451-4180-91b1-e6e61a828c48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6817","nonce":3735115340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6819","nonce":3735115340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6823","nonce":3735115340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6821","nonce":3735115340}]},"public_addr":"172.21.15.150:6817/3735115340","cluster_addr":"172.21.15.150:6819/3735115340","heartbeat_back_addr":"172.21.15.150:6823/3735115340","heartbeat_front_addr":"172.21.15.150:6821/3735115340","state":["exists","up"]},{"osd":8,"uuid":"b6abf7ad-eb34-4f60-a4eb-348a92d7195b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6817","nonce":1025400095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6819","nonce":1025400095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6823","nonce":1025400095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6821","nonce":1025400095}]},"public_addr":"172.21.15.183:6817/1025400095","cluster_addr":"172.21.15.183:6819/1025400095","heartbeat_back_addr":"172.21.15.183:6823/1025400095","heartbeat_front_addr":"172.21.15.183:6821/1025400095","state":["exists","up"]},{"osd":9,"uuid":"28a78e3a-0657-40aa-816f-16a6fadadcbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6827","nonce":76885409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6829","nonce":76885409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6833","nonce":76885409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6831","nonce":76885409}]},"public_addr":"172.21.15.67:6827/76885409","cluster_addr":"172.21.15.67:6829/76885409","heartbeat_back_addr":"172.21.15.67:6833/76885409","heartbeat_front_addr":"172.21.15.67:6831/76885409","state":["exists","up"]},{"osd":10,"uuid":"d287907e-424f-4b49-a06e-4613c6de0474","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6825","nonce":3489982080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6827","nonce":3489982080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6831","nonce":3489982080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6829","nonce":3489982080}]},"public_addr":"172.21.15.150:6825/3489982080","cluster_addr":"172.21.15.150:6827/3489982080","heartbeat_back_addr":"172.21.15.150:6831/3489982080","heartbeat_front_addr":"172.21.15.150:6829/3489982080","state":["exists","up"]},{"osd":11,"uuid":"a3c29c47-a228-4f42-a23c-fbdb0c093699","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":85,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6825","nonce":1892639849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6826","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6827","nonce":1892639849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6831","nonce":1892639849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6829","nonce":1892639849}]},"public_addr":"172.21.15.183:6825/1892639849","cluster_addr":"172.21.15.183:6827/1892639849","heartbeat_back_addr":"172.21.15.183:6831/1892639849","heartbeat_front_addr":"172.21.15.183:6829/1892639849","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:53:45.646211+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:54:18.956590+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:54:52.161646+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:55:27.502822+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:56:01.162932+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:56:34.674452+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:57:09.298679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:57:43.230406+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:58:17.003821+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:58:51.085863+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:59:25.665065+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T16:00:00.263457+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":1}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6801/1631001968":"2024-03-29T15:51:38.875334+0000","172.21.15.67:6800/1631001968":"2024-03-29T15:51:38.875334+0000","172.21.15.67:0/1533056310":"2024-03-29T15:51:38.875334+0000","172.21.15.67:6801/3580491071":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/2030412458":"2024-03-29T15:51:03.859740+0000","172.21.15.67:6800/3580491071":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/2905307533":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/3763759988":"2024-03-29T15:51:38.875334+0000","172.21.15.67:0/4089399880":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/4239108661":"2024-03-29T15:51:38.875334+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T16:02:44.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.539+0000 7fa0dee00640 1 -- 172.21.15.67:0/2352925141 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fa0c00a2800 msgr2=0x7fa0c00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:44.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.539+0000 7fa0dee00640 1 --2- 172.21.15.67:0/2352925141 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fa0c00a2800 0x7fa0c00a4cc0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fa0cc005fd0 tx=0x7fa0cc005850 comp rx=0 tx=0).stop 2024-03-28T16:02:44.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.539+0000 7fa0dee00640 1 -- 172.21.15.67:0/2352925141 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0d80756e0 msgr2=0x7fa0d819e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:44.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.539+0000 7fa0dee00640 1 --2- 172.21.15.67:0/2352925141 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0d80756e0 0x7fa0d819e2f0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fa0c4005ab0 tx=0x7fa0c40042f0 comp rx=0 tx=0).stop 2024-03-28T16:02:44.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.539+0000 7fa0dee00640 1 -- 172.21.15.67:0/2352925141 shutdown_connections 2024-03-28T16:02:44.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.539+0000 7fa0dee00640 1 --2- 172.21.15.67:0/2352925141 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fa0c00a2800 0x7fa0c00a4cc0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:44.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.539+0000 7fa0dee00640 1 --2- 172.21.15.67:0/2352925141 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0d810dae0 0x7fa0d81a5320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:44.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.539+0000 7fa0dee00640 1 --2- 172.21.15.67:0/2352925141 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0d80756e0 0x7fa0d819e2f0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:44.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.539+0000 7fa0dee00640 1 --2- 172.21.15.67:0/2352925141 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa0d8074100 0x7fa0d819ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:44.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.539+0000 7fa0dee00640 1 -- 172.21.15.67:0/2352925141 >> 172.21.15.67:0/2352925141 conn(0x7fa0d80fca80 msgr2=0x7fa0d8100e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:02:44.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.540+0000 7fa0dee00640 1 -- 172.21.15.67:0/2352925141 shutdown_connections 2024-03-28T16:02:44.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.540+0000 7fa0dee00640 1 -- 172.21.15.67:0/2352925141 wait complete. 2024-03-28T16:02:44.559 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T16:02:44.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.786+0000 7f7989e00640 1 -- 172.21.15.67:0/2296425173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79841028b0 msgr2=0x7f7984102d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:44.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.786+0000 7f7989e00640 1 --2- 172.21.15.67:0/2296425173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79841028b0 0x7f7984102d30 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f796c009900 tx=0x7f796c02bc40 comp rx=0 tx=0).stop 2024-03-28T16:02:44.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.786+0000 7f7989e00640 1 -- 172.21.15.67:0/2296425173 shutdown_connections 2024-03-28T16:02:44.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.786+0000 7f7989e00640 1 --2- 172.21.15.67:0/2296425173 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7984103270 0x7f7984109b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:44.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.786+0000 7f7989e00640 1 --2- 172.21.15.67:0/2296425173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79841028b0 0x7f7984102d30 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:44.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.786+0000 7f7989e00640 1 --2- 172.21.15.67:0/2296425173 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f79841016c0 0x7f7984101ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:44.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.786+0000 7f7989e00640 1 -- 172.21.15.67:0/2296425173 >> 172.21.15.67:0/2296425173 conn(0x7f79840fcc50 msgr2=0x7f79840ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:02:44.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.787+0000 7f7989e00640 1 -- 172.21.15.67:0/2296425173 shutdown_connections 2024-03-28T16:02:44.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.787+0000 7f7989e00640 1 -- 172.21.15.67:0/2296425173 wait complete. 2024-03-28T16:02:44.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.788+0000 7f7989e00640 1 Processor -- start 2024-03-28T16:02:44.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.788+0000 7f7989e00640 1 -- start start 2024-03-28T16:02:44.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.788+0000 7f7989e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79841016c0 0x7f7984199a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:44.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.788+0000 7f7989e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79841028b0 0x7f7984199fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:44.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.788+0000 7f7989e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7984103270 0x7f79841a1030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:44.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.788+0000 7f7989e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f798410d220 con 0x7f79841028b0 2024-03-28T16:02:44.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.788+0000 7f7989e00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f798410d0a0 con 0x7f79841016c0 2024-03-28T16:02:44.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.788+0000 7f7989e00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f798410d3a0 con 0x7f7984103270 2024-03-28T16:02:44.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.788+0000 7f7982000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79841028b0 0x7f7984199fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:44.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.788+0000 7f7982000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79841028b0 0x7f7984199fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47848/0 (socket says 172.21.15.67:47848) 2024-03-28T16:02:44.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.788+0000 7f7982000640 1 -- 172.21.15.67:0/3070453599 learned_addr learned my addr 172.21.15.67:0/3070453599 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:02:44.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.789+0000 7f7983400640 1 --2- 172.21.15.67:0/3070453599 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7984103270 0x7f79841a1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:44.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.789+0000 7f7982a00640 1 --2- 172.21.15.67:0/3070453599 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79841016c0 0x7f7984199a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:44.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.789+0000 7f7982000640 1 -- 172.21.15.67:0/3070453599 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7984103270 msgr2=0x7f79841a1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:44.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.789+0000 7f7982000640 1 --2- 172.21.15.67:0/3070453599 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7984103270 0x7f79841a1030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:44.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.789+0000 7f7982000640 1 -- 172.21.15.67:0/3070453599 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79841016c0 msgr2=0x7f7984199a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:44.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.789+0000 7f7982000640 1 --2- 172.21.15.67:0/3070453599 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79841016c0 0x7f7984199a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:44.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.789+0000 7f7982000640 1 -- 172.21.15.67:0/3070453599 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79841a16a0 con 0x7f79841028b0 2024-03-28T16:02:44.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.789+0000 7f7983400640 1 --2- 172.21.15.67:0/3070453599 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7984103270 0x7f79841a1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:02:44.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.789+0000 7f7982a00640 1 --2- 172.21.15.67:0/3070453599 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79841016c0 0x7f7984199a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:02:44.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.789+0000 7f7982000640 1 --2- 172.21.15.67:0/3070453599 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79841028b0 0x7f7984199fc0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f796c000c00 tx=0x7f796c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:02:44.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.790+0000 7f7988a00640 1 -- 172.21.15.67:0/3070453599 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f796c043020 con 0x7f79841028b0 2024-03-28T16:02:44.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.790+0000 7f7988a00640 1 -- 172.21.15.67:0/3070453599 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f796c0054c0 con 0x7f79841028b0 2024-03-28T16:02:44.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.790+0000 7f7988a00640 1 -- 172.21.15.67:0/3070453599 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f796c003d80 con 0x7f79841028b0 2024-03-28T16:02:44.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.790+0000 7f7989e00640 1 -- 172.21.15.67:0/3070453599 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79841a1930 con 0x7f79841028b0 2024-03-28T16:02:44.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.790+0000 7f7989e00640 1 -- 172.21.15.67:0/3070453599 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79841a1ed0 con 0x7f79841028b0 2024-03-28T16:02:44.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.791+0000 7f7989e00640 1 -- 172.21.15.67:0/3070453599 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79840688e0 con 0x7f79841028b0 2024-03-28T16:02:44.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.793+0000 7f7988a00640 1 -- 172.21.15.67:0/3070453599 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f796c046020 con 0x7f79841028b0 2024-03-28T16:02:44.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.794+0000 7f7988a00640 1 --2- 172.21.15.67:0/3070453599 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f79680a2880 0x7f79680a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:44.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.794+0000 7f7988a00640 1 -- 172.21.15.67:0/3070453599 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f796c0f0f10 con 0x7f79841028b0 2024-03-28T16:02:44.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.795+0000 7f7982a00640 1 --2- 172.21.15.67:0/3070453599 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f79680a2880 0x7f79680a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:44.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.795+0000 7f7982a00640 1 --2- 172.21.15.67:0/3070453599 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f79680a2880 0x7f79680a4d40 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f7984102710 tx=0x7f7974005db0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:02:44.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:44.799+0000 7f7988a00640 1 -- 172.21.15.67:0/3070453599 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f796c04d020 con 0x7f79841028b0 2024-03-28T16:02:44.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:44 smithi150 ceph-mon[34825]: pgmap v414: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 255 B/s wr, 2 op/s 2024-03-28T16:02:44.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:44 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/981644889' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:02:44.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:44 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/2352925141' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:02:44.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:44 smithi067 ceph-mon[31707]: pgmap v414: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 255 B/s wr, 2 op/s 2024-03-28T16:02:44.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:44 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/981644889' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:02:44.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:44 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/2352925141' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:02:45.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.045+0000 7f7989e00640 1 -- 172.21.15.67:0/3070453599 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f79841a20a0 con 0x7f79841028b0 2024-03-28T16:02:45.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.046+0000 7f7988a00640 1 -- 172.21.15.67:0/3070453599 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11341 (secure 0 0 0) 0x7f796c0be080 con 0x7f79841028b0 2024-03-28T16:02:45.047 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:02:45.047 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2008297659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2008297659},{"type":"v1","addr":"172.21.15.67:6835","nonce":2008297659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14532,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/320295745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":320295745},{"type":"v1","addr":"172.21.15.67:6837","nonce":320295745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6839/691821273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":691821273},{"type":"v1","addr":"172.21.15.67:6839","nonce":691821273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6841/3211254875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6840","nonce":3211254875},{"type":"v1","addr":"172.21.15.67:6841","nonce":3211254875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2129665423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2129665423},{"type":"v1","addr":"172.21.15.183:6833","nonce":2129665423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24496,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/1589784425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":1589784425},{"type":"v1","addr":"172.21.15.183:6835","nonce":1589784425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24508,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/2185863835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":2185863835},{"type":"v1","addr":"172.21.15.183:6837","nonce":2185863835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24520,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6839/1242814398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":1242814398},{"type":"v1","addr":"172.21.15.183:6839","nonce":1242814398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":34409,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1008335521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1008335521},{"type":"v1","addr":"172.21.15.150:6833","nonce":1008335521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":34445,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3269935695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3269935695},{"type":"v1","addr":"172.21.15.150:6835","nonce":3269935695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T16:01:07.696824+0000","modified":"2024-03-28T16:01:11.997773+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":34508},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34475":{"gid":34475,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/1112886277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":1112886277},{"type":"v1","addr":"172.21.15.150:6837","nonce":1112886277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_34508":{"gid":34508,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6839/1211011614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":1211011614},{"type":"v1","addr":"172.21.15.150:6839","nonce":1211011614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34508,"qdb_cluster":[34508]},"id":1}]} 2024-03-28T16:02:45.047 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 19 2024-03-28T16:02:45.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.049+0000 7f7989e00640 1 -- 172.21.15.67:0/3070453599 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f79680a2880 msgr2=0x7f79680a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:45.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.049+0000 7f7989e00640 1 --2- 172.21.15.67:0/3070453599 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f79680a2880 0x7f79680a4d40 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f7984102710 tx=0x7f7974005db0 comp rx=0 tx=0).stop 2024-03-28T16:02:45.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.049+0000 7f7989e00640 1 -- 172.21.15.67:0/3070453599 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79841028b0 msgr2=0x7f7984199fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:45.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.049+0000 7f7989e00640 1 --2- 172.21.15.67:0/3070453599 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79841028b0 0x7f7984199fc0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f796c000c00 tx=0x7f796c0042f0 comp rx=0 tx=0).stop 2024-03-28T16:02:45.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.050+0000 7f7989e00640 1 -- 172.21.15.67:0/3070453599 shutdown_connections 2024-03-28T16:02:45.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.050+0000 7f7989e00640 1 --2- 172.21.15.67:0/3070453599 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f79680a2880 0x7f79680a4d40 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:45.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.050+0000 7f7989e00640 1 --2- 172.21.15.67:0/3070453599 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7984103270 0x7f79841a1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:45.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.050+0000 7f7989e00640 1 --2- 172.21.15.67:0/3070453599 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79841028b0 0x7f7984199fc0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:45.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.050+0000 7f7989e00640 1 --2- 172.21.15.67:0/3070453599 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79841016c0 0x7f7984199a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:45.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.050+0000 7f7989e00640 1 -- 172.21.15.67:0/3070453599 >> 172.21.15.67:0/3070453599 conn(0x7f79840fcc50 msgr2=0x7f79840738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:02:45.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.050+0000 7f7989e00640 1 -- 172.21.15.67:0/3070453599 shutdown_connections 2024-03-28T16:02:45.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.050+0000 7f7989e00640 1 -- 172.21.15.67:0/3070453599 wait complete. 2024-03-28T16:02:45.069 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 19, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-28T16:01:07.696824+0000', 'modified': '2024-03-28T16:01:11.997773+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 34508}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_34475': {'gid': 34475, 'name': 'h', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.150:6837/1112886277', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6836', 'nonce': 1112886277}, {'type': 'v1', 'addr': '172.21.15.150:6837', 'nonce': 1112886277}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_34508': {'gid': 34508, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6839/1211011614', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6838', 'nonce': 1211011614}, {'type': 'v1', 'addr': '172.21.15.150:6839', 'nonce': 1211011614}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 34508, 'qdb_cluster': [34508]} 2024-03-28T16:02:45.069 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-28T16:02:45.069 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-28T16:02:45.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:44 smithi183 ceph-mon[35580]: pgmap v414: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 255 B/s wr, 2 op/s 2024-03-28T16:02:45.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:44 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/981644889' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:02:45.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:44 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/2352925141' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:02:45.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.286+0000 7f4a9f600640 1 -- 172.21.15.67:0/3950229399 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a98107390 msgr2=0x7f4a981097e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:45.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.286+0000 7f4a9f600640 1 --2- 172.21.15.67:0/3950229399 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a98107390 0x7f4a981097e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f4a90009ca0 tx=0x7f4a9002c0f0 comp rx=0 tx=0).stop 2024-03-28T16:02:45.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.286+0000 7f4a9f600640 1 -- 172.21.15.67:0/3950229399 shutdown_connections 2024-03-28T16:02:45.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.286+0000 7f4a9f600640 1 --2- 172.21.15.67:0/3950229399 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a98107390 0x7f4a981097e0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:45.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.286+0000 7f4a9f600640 1 --2- 172.21.15.67:0/3950229399 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4a98104a60 0x7f4a98106e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:45.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.286+0000 7f4a9f600640 1 --2- 172.21.15.67:0/3950229399 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a98102130 0x7f4a98104520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:45.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.286+0000 7f4a9f600640 1 -- 172.21.15.67:0/3950229399 >> 172.21.15.67:0/3950229399 conn(0x7f4a980fbb40 msgr2=0x7f4a980fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:02:45.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.286+0000 7f4a9f600640 1 -- 172.21.15.67:0/3950229399 shutdown_connections 2024-03-28T16:02:45.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.286+0000 7f4a9f600640 1 -- 172.21.15.67:0/3950229399 wait complete. 2024-03-28T16:02:45.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.287+0000 7f4a9f600640 1 Processor -- start 2024-03-28T16:02:45.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.287+0000 7f4a9f600640 1 -- start start 2024-03-28T16:02:45.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.287+0000 7f4a9f600640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a98102130 0x7f4a981978e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:45.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.287+0000 7f4a9f600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4a98104a60 0x7f4a98197e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:45.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.287+0000 7f4a9f600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a98107390 0x7f4a9819ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:45.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.287+0000 7f4a9f600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9810d1e0 con 0x7f4a98107390 2024-03-28T16:02:45.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.287+0000 7f4a9f600640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9810d060 con 0x7f4a98102130 2024-03-28T16:02:45.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.287+0000 7f4a9f600640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9810d360 con 0x7f4a98104a60 2024-03-28T16:02:45.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.287+0000 7f4a9d600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a98107390 0x7f4a9819ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:45.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.288+0000 7f4a9d600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a98107390 0x7f4a9819ee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47872/0 (socket says 172.21.15.67:47872) 2024-03-28T16:02:45.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.288+0000 7f4a9d600640 1 -- 172.21.15.67:0/3620662398 learned_addr learned my addr 172.21.15.67:0/3620662398 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:02:45.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.288+0000 7f4a9d600640 1 -- 172.21.15.67:0/3620662398 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4a98104a60 msgr2=0x7f4a98197e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:45.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.288+0000 7f4a8fe00640 1 --2- 172.21.15.67:0/3620662398 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4a98104a60 0x7f4a98197e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:45.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.288+0000 7f4a9cc00640 1 --2- 172.21.15.67:0/3620662398 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a98102130 0x7f4a981978e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:45.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.288+0000 7f4a9d600640 1 --2- 172.21.15.67:0/3620662398 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4a98104a60 0x7f4a98197e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:45.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.288+0000 7f4a9d600640 1 -- 172.21.15.67:0/3620662398 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a98102130 msgr2=0x7f4a981978e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:45.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.288+0000 7f4a9d600640 1 --2- 172.21.15.67:0/3620662398 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a98102130 0x7f4a981978e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:45.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.288+0000 7f4a9d600640 1 -- 172.21.15.67:0/3620662398 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a9819f500 con 0x7f4a98107390 2024-03-28T16:02:45.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.288+0000 7f4a8fe00640 1 --2- 172.21.15.67:0/3620662398 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4a98104a60 0x7f4a98197e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:02:45.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.288+0000 7f4a9cc00640 1 --2- 172.21.15.67:0/3620662398 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a98102130 0x7f4a981978e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:02:45.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.288+0000 7f4a9d600640 1 --2- 172.21.15.67:0/3620662398 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a98107390 0x7f4a9819ee90 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f4a90032380 tx=0x7f4a90032820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:02:45.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.289+0000 7f4a96000640 1 -- 172.21.15.67:0/3620662398 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a90041020 con 0x7f4a98107390 2024-03-28T16:02:45.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.289+0000 7f4a96000640 1 -- 172.21.15.67:0/3620662398 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4a90003e20 con 0x7f4a98107390 2024-03-28T16:02:45.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.289+0000 7f4a96000640 1 -- 172.21.15.67:0/3620662398 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a90039040 con 0x7f4a98107390 2024-03-28T16:02:45.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.289+0000 7f4a9f600640 1 -- 172.21.15.67:0/3620662398 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a9819f790 con 0x7f4a98107390 2024-03-28T16:02:45.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.289+0000 7f4a9f600640 1 -- 172.21.15.67:0/3620662398 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a9819fcd0 con 0x7f4a98107390 2024-03-28T16:02:45.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.293+0000 7f4aa2243740 1 Processor -- start 2024-03-28T16:02:45.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.293+0000 7f4a96000640 1 -- 172.21.15.67:0/3620662398 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f4a90044020 con 0x7f4a98107390 2024-03-28T16:02:45.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.293+0000 7f4aa2243740 1 -- start start 2024-03-28T16:02:45.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.294+0000 7f4aa2243740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f5e533efb0 0x55f5e533f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:45.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.294+0000 7f4aa2243740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f5e533f9a0 0x55f5e53444a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:45.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.294+0000 7f4aa2243740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f5e5344b10 0x55f5e5346fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:45.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.294+0000 7f4aa2243740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5e529fc90 con 0x55f5e533efb0 2024-03-28T16:02:45.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.294+0000 7f4aa2243740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5e5248980 con 0x55f5e5344b10 2024-03-28T16:02:45.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.294+0000 7f4aa2243740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5e529bd30 con 0x55f5e533f9a0 2024-03-28T16:02:45.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.294+0000 7f4a8fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f5e533efb0 0x55f5e533f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:45.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.294+0000 7f4a8fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f5e533efb0 0x55f5e533f3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47886/0 (socket says 172.21.15.67:47886) 2024-03-28T16:02:45.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.294+0000 7f4a8fe00640 1 -- 172.21.15.67:0/1906458190 learned_addr learned my addr 172.21.15.67:0/1906458190 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:02:45.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.294+0000 7f4a9cc00640 1 --2- 172.21.15.67:0/1906458190 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f5e533f9a0 0x55f5e53444a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:45.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.294+0000 7f4a8fe00640 1 --2- 172.21.15.67:0/1906458190 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f5e5344b10 0x55f5e5346fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:45.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.294+0000 7f4a8fe00640 1 -- 172.21.15.67:0/1906458190 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f5e533f9a0 msgr2=0x55f5e53444a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:02:45.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.294+0000 7f4a8fe00640 1 --2- 172.21.15.67:0/1906458190 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f5e533f9a0 0x55f5e53444a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:45.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.294+0000 7f4a8fe00640 1 -- 172.21.15.67:0/1906458190 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f5e5344b10 msgr2=0x55f5e5346fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:02:45.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.294+0000 7f4a8fe00640 1 --2- 172.21.15.67:0/1906458190 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f5e5344b10 0x55f5e5346fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:45.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.295+0000 7f4a8fe00640 1 -- 172.21.15.67:0/1906458190 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f5e5260510 con 0x55f5e533efb0 2024-03-28T16:02:45.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.295+0000 7f4a8fe00640 1 --2- 172.21.15.67:0/1906458190 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f5e533efb0 0x55f5e533f3d0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f4a8000bd80 tx=0x7f4a8000b500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca35c22bf5ea04a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:02:45.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.295+0000 7f4a96000640 1 --2- 172.21.15.67:0/3620662398 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f4a7c0a2880 0x7f4a7c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:45.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.295+0000 7f4a96000640 1 -- 172.21.15.67:0/3620662398 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f4a900f0e00 con 0x7f4a98107390 2024-03-28T16:02:45.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.295+0000 7f4a8dc00640 1 -- 172.21.15.67:0/1906458190 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a80012650 con 0x55f5e533efb0 2024-03-28T16:02:45.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.295+0000 7f4a8dc00640 1 -- 172.21.15.67:0/1906458190 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4a80012ca0 con 0x55f5e533efb0 2024-03-28T16:02:45.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.295+0000 7f4a8dc00640 1 -- 172.21.15.67:0/1906458190 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a80011c80 con 0x55f5e533efb0 2024-03-28T16:02:45.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.295+0000 7f4a9cc00640 1 --2- 172.21.15.67:0/3620662398 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f4a7c0a2880 0x7f4a7c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:45.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.296+0000 7f4aa2243740 1 -- 172.21.15.67:0/1906458190 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f5e533efb0 msgr2=0x55f5e533f3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:45.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.296+0000 7f4a9cc00640 1 --2- 172.21.15.67:0/3620662398 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f4a7c0a2880 0x7f4a7c0a4d40 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x55f5e533fda0 tx=0x7f4a88009350 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:02:45.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.296+0000 7f4aa2243740 1 --2- 172.21.15.67:0/1906458190 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f5e533efb0 0x55f5e533f3d0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f4a8000bd80 tx=0x7f4a8000b500 comp rx=0 tx=0).stop 2024-03-28T16:02:45.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.296+0000 7f4aa2243740 1 -- 172.21.15.67:0/1906458190 shutdown_connections 2024-03-28T16:02:45.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.296+0000 7f4aa2243740 1 --2- 172.21.15.67:0/1906458190 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f5e5344b10 0x55f5e5346fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:45.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.296+0000 7f4aa2243740 1 --2- 172.21.15.67:0/1906458190 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f5e533f9a0 0x55f5e53444a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:45.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.296+0000 7f4aa2243740 1 --2- 172.21.15.67:0/1906458190 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f5e533efb0 0x55f5e533f3d0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:45.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.296+0000 7f4aa2243740 1 -- 172.21.15.67:0/1906458190 >> 172.21.15.67:0/1906458190 conn(0x55f5e5336110 msgr2=0x55f5e5336510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:02:45.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.296+0000 7f4aa2243740 1 -- 172.21.15.67:0/1906458190 shutdown_connections 2024-03-28T16:02:45.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.296+0000 7f4aa2243740 1 -- 172.21.15.67:0/1906458190 wait complete. 2024-03-28T16:02:45.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.297+0000 7f4aa2243740 1 -- start start 2024-03-28T16:02:45.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.297+0000 7f4aa2243740 1 Processor -- start 2024-03-28T16:02:45.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.297+0000 7f4aa2243740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f5e533efb0 0x55f5e5349570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:45.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.297+0000 7f4aa2243740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f5e533f9a0 0x55f5e5349ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:45.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.297+0000 7f4aa2243740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f5e5344b10 0x55f5e5350af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:45.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.297+0000 7f4aa2243740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5e5347b10 con 0x55f5e5344b10 2024-03-28T16:02:45.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.297+0000 7f4aa2243740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5e525bdd0 con 0x55f5e533efb0 2024-03-28T16:02:45.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.297+0000 7f4aa2243740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5e5347c90 con 0x55f5e533f9a0 2024-03-28T16:02:45.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.297+0000 7f4aa2243740 10 client.0 _ll_register_callbacks cb 0x55f5e4fb3ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:02:45.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.298+0000 7f4a9d600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f5e5344b10 0x55f5e5350af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:45.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.298+0000 7f4a9d600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f5e5344b10 0x55f5e5350af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47888/0 (socket says 172.21.15.67:47888) 2024-03-28T16:02:45.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.298+0000 7f4a9d600640 1 -- 172.21.15.67:0/3472637006 learned_addr learned my addr 172.21.15.67:0/3472637006 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:02:45.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.298+0000 7f4a9cc00640 1 --2- 172.21.15.67:0/3472637006 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f5e533efb0 0x55f5e5349570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:45.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.298+0000 7f4a8fe00640 1 --2- 172.21.15.67:0/3472637006 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f5e533f9a0 0x55f5e5349ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:45.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.298+0000 7f4a9d600640 1 -- 172.21.15.67:0/3472637006 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f5e533f9a0 msgr2=0x55f5e5349ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:02:45.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.298+0000 7f4a9d600640 1 --2- 172.21.15.67:0/3472637006 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f5e533f9a0 0x55f5e5349ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:45.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.298+0000 7f4a9d600640 1 -- 172.21.15.67:0/3472637006 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f5e533efb0 msgr2=0x55f5e5349570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:02:45.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.298+0000 7f4a9d600640 1 --2- 172.21.15.67:0/3472637006 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f5e533efb0 0x55f5e5349570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:45.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.298+0000 7f4a9d600640 1 -- 172.21.15.67:0/3472637006 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f5e4fb42d0 con 0x55f5e5344b10 2024-03-28T16:02:45.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.298+0000 7f4a9cc00640 1 --2- 172.21.15.67:0/3472637006 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f5e533efb0 0x55f5e5349570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T16:02:45.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.298+0000 7f4a9d600640 1 --2- 172.21.15.67:0/3472637006 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f5e5344b10 0x55f5e5350af0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f4a900c59c0 tx=0x7f4a900bd350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fef125a7181fcd8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:02:45.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.298+0000 7f4a86000640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:02:45.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.299+0000 7f4a86000640 1 -- 172.21.15.67:0/3472637006 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a900bde50 con 0x55f5e5344b10 2024-03-28T16:02:45.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.299+0000 7f4a86000640 1 -- 172.21.15.67:0/3472637006 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f4a900a8070 con 0x55f5e5344b10 2024-03-28T16:02:45.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.299+0000 7f4aa2243740 1 -- 172.21.15.67:0/3472637006 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4a80012ca0 con 0x55f5e5344b10 2024-03-28T16:02:45.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.299+0000 7f4a86000640 1 -- 172.21.15.67:0/3472637006 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4a900bd750 con 0x55f5e5344b10 2024-03-28T16:02:45.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.299+0000 7f4aa2243740 10 client.15042 fetch_fsmap learned FSMap version 19 2024-03-28T16:02:45.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.299+0000 7f4aa2243740 1 -- 172.21.15.67:0/3472637006 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55f5e507e6b0 con 0x55f5e5344b10 2024-03-28T16:02:45.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.299+0000 7f4a86000640 1 -- 172.21.15.67:0/3472637006 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4a80012ca0 con 0x55f5e5344b10 2024-03-28T16:02:45.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.299+0000 7f4a86000640 1 -- 172.21.15.67:0/3472637006 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f4a900a6ce0 con 0x55f5e5344b10 2024-03-28T16:02:45.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.299+0000 7f4aa2243740 10 client.15042 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:02:45.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.300+0000 7f4aa2243740 10 client.15042 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T16:02:45.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.300+0000 7f4aa2243740 20 client.15042 populate_metadata read hostname 'smithi067' 2024-03-28T16:02:45.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.300+0000 7f4aa2243740 1 --2- 172.21.15.67:0/3472637006 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55f5e5355d60 0x55f5e5354c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:45.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.300+0000 7f4aa2243740 4 client.15042 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:02:45.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.300+0000 7f4aa2243740 1 -- 172.21.15.67:0/3472637006 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f5e5241520 con 0x55f5e5355d60 2024-03-28T16:02:45.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.300+0000 7f4a9cc00640 1 --2- 172.21.15.67:0/3472637006 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55f5e5355d60 0x55f5e5354c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:45.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.301+0000 7f4a9cc00640 1 --2- 172.21.15.67:0/3472637006 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55f5e5355d60 0x55f5e5354c50 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e72a754b8c6a785 server_cookie=eb8ccb50a2c0ba85 in_seq=0 out_seq=0 2024-03-28T16:02:45.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.301+0000 7f4a86000640 10 client.15042 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:02:45.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.303+0000 7f4a86000640 1 -- 172.21.15.67:0/3472637006 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f5e5241520 con 0x55f5e5355d60 2024-03-28T16:02:45.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.303+0000 7f4a86000640 10 client.15042 handle_command_reply: tid=0 2024-03-28T16:02:45.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.303+0000 7f4aa2243740 1 client.15042 shutdown 2024-03-28T16:02:45.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.303+0000 7f4aa2243740 1 -- 172.21.15.67:0/3472637006 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f5e5344b10 msgr2=0x55f5e5350af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:45.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.303+0000 7f4aa2243740 1 --2- 172.21.15.67:0/3472637006 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f5e5344b10 0x55f5e5350af0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f4a900c59c0 tx=0x7f4a900bd350 comp rx=0 tx=0).stop 2024-03-28T16:02:45.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.303+0000 7f4aa2243740 1 -- 172.21.15.67:0/3472637006 shutdown_connections 2024-03-28T16:02:45.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.303+0000 7f4aa2243740 1 --2- 172.21.15.67:0/3472637006 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55f5e5355d60 0x55f5e5354c50 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:45.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.303+0000 7f4aa2243740 1 --2- 172.21.15.67:0/3472637006 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f5e5344b10 0x55f5e5350af0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:45.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.303+0000 7f4aa2243740 1 --2- 172.21.15.67:0/3472637006 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f5e533f9a0 0x55f5e5349ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:45.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.303+0000 7f4aa2243740 1 --2- 172.21.15.67:0/3472637006 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f5e533efb0 0x55f5e5349570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:45.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.303+0000 7f4aa2243740 1 -- 172.21.15.67:0/3472637006 >> 172.21.15.67:0/3472637006 conn(0x55f5e5336110 msgr2=0x55f5e5335310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:02:45.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.304+0000 7f4a86000640 0 client.15042 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:02:45.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.304+0000 7f4aa2243740 1 -- 172.21.15.67:0/3472637006 shutdown_connections 2024-03-28T16:02:45.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.304+0000 7f4aa2243740 1 -- 172.21.15.67:0/3472637006 wait complete. 2024-03-28T16:02:45.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.304+0000 7f4aa2243740 20 client.15042 trim_cache size 0 max 16384 2024-03-28T16:02:45.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.322+0000 7f4aa2243740 1 Processor -- start 2024-03-28T16:02:45.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.322+0000 7f4aa2243740 1 -- start start 2024-03-28T16:02:45.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.323+0000 7f4aa2243740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f5e533efb0 0x55f5e5349890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:45.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.323+0000 7f4aa2243740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f5e533f9a0 0x55f5e5350ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:45.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.323+0000 7f4aa2243740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f5e5344b10 0x55f5e5348060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:45.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.323+0000 7f4aa2243740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5e5355580 con 0x55f5e533efb0 2024-03-28T16:02:45.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.323+0000 7f4aa2243740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5e5355be0 con 0x55f5e5344b10 2024-03-28T16:02:45.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.323+0000 7f4aa2243740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5e5356240 con 0x55f5e533f9a0 2024-03-28T16:02:45.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.323+0000 7f4a9cc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f5e533efb0 0x55f5e5349890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:45.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.323+0000 7f4a9cc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f5e533efb0 0x55f5e5349890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47892/0 (socket says 172.21.15.67:47892) 2024-03-28T16:02:45.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.323+0000 7f4a9cc00640 1 -- 172.21.15.67:0/3486621819 learned_addr learned my addr 172.21.15.67:0/3486621819 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:02:45.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.323+0000 7f4a9d600640 1 --2- 172.21.15.67:0/3486621819 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f5e5344b10 0x55f5e5348060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:45.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.323+0000 7f4a9cc00640 1 -- 172.21.15.67:0/3486621819 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f5e533f9a0 msgr2=0x55f5e5350ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:02:45.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.323+0000 7f4a8fe00640 1 --2- 172.21.15.67:0/3486621819 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f5e533f9a0 0x55f5e5350ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:45.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.323+0000 7f4a9cc00640 1 --2- 172.21.15.67:0/3486621819 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f5e533f9a0 0x55f5e5350ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:45.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.323+0000 7f4a9cc00640 1 -- 172.21.15.67:0/3486621819 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f5e5344b10 msgr2=0x55f5e5348060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:02:45.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.323+0000 7f4a9cc00640 1 --2- 172.21.15.67:0/3486621819 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f5e5344b10 0x55f5e5348060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:45.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.323+0000 7f4a9cc00640 1 -- 172.21.15.67:0/3486621819 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f5e5336e80 con 0x55f5e533efb0 2024-03-28T16:02:45.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.323+0000 7f4a8fe00640 1 --2- 172.21.15.67:0/3486621819 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f5e533f9a0 0x55f5e5350ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:02:45.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.324+0000 7f4a9cc00640 1 --2- 172.21.15.67:0/3486621819 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f5e533efb0 0x55f5e5349890 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f4a88005fd0 tx=0x7f4a88016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9c0e7b9c859a36f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:02:45.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.324+0000 7f4a8d200640 1 -- 172.21.15.67:0/3486621819 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a88027310 con 0x55f5e533efb0 2024-03-28T16:02:45.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.324+0000 7f4a8d200640 1 -- 172.21.15.67:0/3486621819 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4a880035f0 con 0x55f5e533efb0 2024-03-28T16:02:45.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.324+0000 7f4a8d200640 1 -- 172.21.15.67:0/3486621819 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a88012690 con 0x55f5e533efb0 2024-03-28T16:02:45.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.325+0000 7f4aa2243740 1 -- 172.21.15.67:0/3486621819 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f5e533efb0 msgr2=0x55f5e5349890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:45.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.325+0000 7f4aa2243740 1 --2- 172.21.15.67:0/3486621819 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f5e533efb0 0x55f5e5349890 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f4a88005fd0 tx=0x7f4a88016000 comp rx=0 tx=0).stop 2024-03-28T16:02:45.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.325+0000 7f4aa2243740 1 -- 172.21.15.67:0/3486621819 shutdown_connections 2024-03-28T16:02:45.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.325+0000 7f4aa2243740 1 --2- 172.21.15.67:0/3486621819 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f5e5344b10 0x55f5e5348060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:45.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.325+0000 7f4aa2243740 1 --2- 172.21.15.67:0/3486621819 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f5e533f9a0 0x55f5e5350ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:45.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.325+0000 7f4aa2243740 1 --2- 172.21.15.67:0/3486621819 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f5e533efb0 0x55f5e5349890 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:45.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.325+0000 7f4aa2243740 1 -- 172.21.15.67:0/3486621819 >> 172.21.15.67:0/3486621819 conn(0x55f5e5336110 msgr2=0x55f5e5349180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:02:45.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.325+0000 7f4aa2243740 1 -- 172.21.15.67:0/3486621819 shutdown_connections 2024-03-28T16:02:45.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.325+0000 7f4aa2243740 1 -- 172.21.15.67:0/3486621819 wait complete. 2024-03-28T16:02:45.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.325+0000 7f4aa2243740 1 -- start start 2024-03-28T16:02:45.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.326+0000 7f4aa2243740 1 Processor -- start 2024-03-28T16:02:45.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.326+0000 7f4aa2243740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f5e533f9a0 0x55f5e53503c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:45.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.326+0000 7f4aa2243740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f5e5344b10 0x55f5e5350900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:45.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.326+0000 7f4aa2243740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f5e534ce60 0x55f5e534d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:45.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.326+0000 7f4aa2243740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5e5356f00 con 0x55f5e5344b10 2024-03-28T16:02:45.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.326+0000 7f4aa2243740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5e53568a0 con 0x55f5e534ce60 2024-03-28T16:02:45.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.326+0000 7f4aa2243740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5e5357560 con 0x55f5e533f9a0 2024-03-28T16:02:45.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.326+0000 7f4aa2243740 10 client.0 _ll_register_callbacks cb 0x7f4a90034e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:02:45.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.326+0000 7f4a8fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f5e5344b10 0x55f5e5350900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:45.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.326+0000 7f4a8fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f5e5344b10 0x55f5e5350900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47898/0 (socket says 172.21.15.67:47898) 2024-03-28T16:02:45.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.326+0000 7f4a8fe00640 1 -- 172.21.15.67:0/1778208103 learned_addr learned my addr 172.21.15.67:0/1778208103 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:02:45.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.327+0000 7f4a9cc00640 1 --2- 172.21.15.67:0/1778208103 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f5e533f9a0 0x55f5e53503c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:45.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.327+0000 7f4a9d600640 1 --2- 172.21.15.67:0/1778208103 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f5e534ce60 0x55f5e534d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:45.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.327+0000 7f4a8fe00640 1 -- 172.21.15.67:0/1778208103 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f5e533f9a0 msgr2=0x55f5e53503c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:02:45.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.327+0000 7f4a8fe00640 1 --2- 172.21.15.67:0/1778208103 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f5e533f9a0 0x55f5e53503c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:45.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.327+0000 7f4a8fe00640 1 -- 172.21.15.67:0/1778208103 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f5e534ce60 msgr2=0x55f5e534d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:02:45.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.327+0000 7f4a8fe00640 1 --2- 172.21.15.67:0/1778208103 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f5e534ce60 0x55f5e534d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:45.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.327+0000 7f4a8fe00640 1 -- 172.21.15.67:0/1778208103 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f5e534d9d0 con 0x55f5e5344b10 2024-03-28T16:02:45.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.327+0000 7f4a9d600640 1 --2- 172.21.15.67:0/1778208103 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f5e534ce60 0x55f5e534d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:02:45.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.327+0000 7f4a8fe00640 1 --2- 172.21.15.67:0/1778208103 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f5e5344b10 0x55f5e5350900 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f4a8000bd80 tx=0x7f4a80012c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa455806d41ad9e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:02:45.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.327+0000 7f4a86000640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:02:45.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.327+0000 7f4a86000640 1 -- 172.21.15.67:0/1778208103 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a8002a060 con 0x55f5e5344b10 2024-03-28T16:02:45.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.327+0000 7f4a86000640 1 -- 172.21.15.67:0/1778208103 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f4a8001a600 con 0x55f5e5344b10 2024-03-28T16:02:45.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.328+0000 7f4aa2243740 1 -- 172.21.15.67:0/1778208103 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4a880035f0 con 0x55f5e5344b10 2024-03-28T16:02:45.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.328+0000 7f4a86000640 1 -- 172.21.15.67:0/1778208103 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4a80005c40 con 0x55f5e5344b10 2024-03-28T16:02:45.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.328+0000 7f4aa2243740 10 client.15048 fetch_fsmap learned FSMap version 19 2024-03-28T16:02:45.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.328+0000 7f4aa2243740 1 -- 172.21.15.67:0/1778208103 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55f5e53b1e60 con 0x55f5e5344b10 2024-03-28T16:02:45.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.328+0000 7f4a86000640 1 -- 172.21.15.67:0/1778208103 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4a880035f0 con 0x55f5e5344b10 2024-03-28T16:02:45.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.328+0000 7f4a86000640 1 -- 172.21.15.67:0/1778208103 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f4a80011580 con 0x55f5e5344b10 2024-03-28T16:02:45.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.328+0000 7f4aa2243740 10 client.15048 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:02:45.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.328+0000 7f4aa2243740 10 client.15048 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T16:02:45.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.328+0000 7f4aa2243740 20 client.15048 populate_metadata read hostname 'smithi067' 2024-03-28T16:02:45.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.328+0000 7f4aa2243740 1 --2- 172.21.15.67:0/1778208103 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55f5e53b6170 0x55f5e53b2140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:45.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.328+0000 7f4aa2243740 4 client.15048 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T16:02:45.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.328+0000 7f4aa2243740 1 -- 172.21.15.67:0/1778208103 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55f5e52ad1d0 con 0x55f5e53b6170 2024-03-28T16:02:45.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.329+0000 7f4a9cc00640 1 --2- 172.21.15.67:0/1778208103 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55f5e53b6170 0x55f5e53b2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:45.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.330+0000 7f4a9cc00640 1 --2- 172.21.15.67:0/1778208103 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55f5e53b6170 0x55f5e53b2140 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f8f2866f90e9bc39 server_cookie=8f6bbdc29741c061 in_seq=0 out_seq=0 2024-03-28T16:02:45.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.330+0000 7f4a86000640 10 client.15048 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:02:45.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.331+0000 7f4a86000640 1 -- 172.21.15.67:0/1778208103 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55f5e52ad1d0 con 0x55f5e53b6170 2024-03-28T16:02:45.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.331+0000 7f4a86000640 10 client.15048 handle_command_reply: tid=0 2024-03-28T16:02:45.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.331+0000 7f4aa2243740 1 client.15048 shutdown 2024-03-28T16:02:45.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.331+0000 7f4aa2243740 1 -- 172.21.15.67:0/1778208103 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f5e5344b10 msgr2=0x55f5e5350900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:45.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.331+0000 7f4aa2243740 1 --2- 172.21.15.67:0/1778208103 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f5e5344b10 0x55f5e5350900 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f4a8000bd80 tx=0x7f4a80012c10 comp rx=0 tx=0).stop 2024-03-28T16:02:45.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.331+0000 7f4aa2243740 1 -- 172.21.15.67:0/1778208103 shutdown_connections 2024-03-28T16:02:45.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.331+0000 7f4aa2243740 1 --2- 172.21.15.67:0/1778208103 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55f5e53b6170 0x55f5e53b2140 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:45.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.331+0000 7f4aa2243740 1 --2- 172.21.15.67:0/1778208103 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f5e534ce60 0x55f5e534d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:45.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.331+0000 7f4aa2243740 1 --2- 172.21.15.67:0/1778208103 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f5e5344b10 0x55f5e5350900 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:45.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.331+0000 7f4aa2243740 1 --2- 172.21.15.67:0/1778208103 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f5e533f9a0 0x55f5e53503c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:45.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.331+0000 7f4a86000640 0 client.15048 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:02:45.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.331+0000 7f4aa2243740 1 -- 172.21.15.67:0/1778208103 >> 172.21.15.67:0/1778208103 conn(0x55f5e5336110 msgr2=0x55f5e5339370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:02:45.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.331+0000 7f4aa2243740 1 -- 172.21.15.67:0/1778208103 shutdown_connections 2024-03-28T16:02:45.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.331+0000 7f4aa2243740 1 -- 172.21.15.67:0/1778208103 wait complete. 2024-03-28T16:02:45.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.332+0000 7f4aa2243740 20 client.15048 trim_cache size 0 max 16384 2024-03-28T16:02:45.336 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-28T16:02:45.336 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "b5f8225a-ed1a-11ee-95d0-87774f69a715", 2024-03-28T16:02:45.336 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 0, 2024-03-28T16:02:45.337 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 34508, 2024-03-28T16:02:45.337 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-03-28T16:02:45.337 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-03-28T16:02:45.337 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-03-28T16:02:45.337 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 97.060298424999999, 2024-03-28T16:02:45.337 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 19, 2024-03-28T16:02:45.337 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 92, 2024-03-28T16:02:45.337 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 90, 2024-03-28T16:02:45.337 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 103.589398172 2024-03-28T16:02:45.337 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-28T16:02:45.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.332+0000 7f4a9f600640 1 -- 172.21.15.67:0/3620662398 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f4a7c0a2880 msgr2=0x7f4a7c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:45.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.332+0000 7f4a9f600640 1 --2- 172.21.15.67:0/3620662398 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f4a7c0a2880 0x7f4a7c0a4d40 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x55f5e533fda0 tx=0x7f4a88009350 comp rx=0 tx=0).stop 2024-03-28T16:02:45.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.332+0000 7f4a9f600640 1 -- 172.21.15.67:0/3620662398 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a98107390 msgr2=0x7f4a9819ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:45.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.332+0000 7f4a9f600640 1 --2- 172.21.15.67:0/3620662398 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a98107390 0x7f4a9819ee90 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f4a90032380 tx=0x7f4a90032820 comp rx=0 tx=0).stop 2024-03-28T16:02:45.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.332+0000 7f4a9f600640 1 -- 172.21.15.67:0/3620662398 shutdown_connections 2024-03-28T16:02:45.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.332+0000 7f4a9f600640 1 --2- 172.21.15.67:0/3620662398 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f4a7c0a2880 0x7f4a7c0a4d40 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:45.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.332+0000 7f4a9f600640 1 --2- 172.21.15.67:0/3620662398 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a98107390 0x7f4a9819ee90 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:45.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.332+0000 7f4a9f600640 1 --2- 172.21.15.67:0/3620662398 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4a98104a60 0x7f4a98197e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:45.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.332+0000 7f4a9f600640 1 --2- 172.21.15.67:0/3620662398 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a98102130 0x7f4a981978e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:45.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.332+0000 7f4a9f600640 1 -- 172.21.15.67:0/3620662398 >> 172.21.15.67:0/3620662398 conn(0x7f4a980fbb40 msgr2=0x7f4a98102b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:02:45.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.332+0000 7f4a9f600640 1 -- 172.21.15.67:0/3620662398 shutdown_connections 2024-03-28T16:02:45.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:45.332+0000 7f4a9f600640 1 -- 172.21.15.67:0/3620662398 wait complete. 2024-03-28T16:02:45.350 INFO:tasks.cephadm:Setup complete, yielding 2024-03-28T16:02:45.350 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-28T16:02:45.360 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi067.front.sepia.ceph.com 2024-03-28T16:02:45.360 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- bash -c 'ceph orch status' 2024-03-28T16:02:45.817 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T16:02:45.972 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:45 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/3070453599' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:02:46.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:45 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/3070453599' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:02:46.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:45 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/3070453599' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:02:46.998 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:46 smithi067 ceph-mon[31707]: pgmap v415: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-28T16:02:47.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:46 smithi150 ceph-mon[34825]: pgmap v415: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-28T16:02:47.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:46 smithi183 ceph-mon[35580]: pgmap v415: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-28T16:02:47.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:47.953+0000 7fcf3a600700 1 -- 172.21.15.67:0/313415810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf34109fd0 msgr2=0x7fcf3410c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:47.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:47.953+0000 7fcf3a600700 1 --2- 172.21.15.67:0/313415810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf34109fd0 0x7fcf3410c460 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c00b0a0 tx=0x7fcf2c00b3b0 comp rx=0 tx=0).stop 2024-03-28T16:02:47.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:47.953+0000 7fcf3a600700 1 -- 172.21.15.67:0/313415810 shutdown_connections 2024-03-28T16:02:47.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:47.953+0000 7fcf3a600700 1 --2- 172.21.15.67:0/313415810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf34109fd0 0x7fcf3410c460 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:47.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:47.953+0000 7fcf3a600700 1 --2- 172.21.15.67:0/313415810 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf340761c0 0x7fcf34074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:47.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:47.953+0000 7fcf3a600700 1 --2- 172.21.15.67:0/313415810 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcf34074bf0 0x7fcf34075070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:47.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:47.953+0000 7fcf3a600700 1 -- 172.21.15.67:0/313415810 >> 172.21.15.67:0/313415810 conn(0x7fcf340fd6a0 msgr2=0x7fcf340ffae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:02:47.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:47.953+0000 7fcf3a600700 1 -- 172.21.15.67:0/313415810 shutdown_connections 2024-03-28T16:02:47.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:47.953+0000 7fcf3a600700 1 -- 172.21.15.67:0/313415810 wait complete. 2024-03-28T16:02:47.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:47.955+0000 7fcf3a600700 1 Processor -- start 2024-03-28T16:02:47.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:47.955+0000 7fcf3a600700 1 -- start start 2024-03-28T16:02:47.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:47.955+0000 7fcf3a600700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcf34074bf0 0x7fcf3419e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:47.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:47.955+0000 7fcf3a600700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf340761c0 0x7fcf3419ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:47.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:47.956+0000 7fcf3a600700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf34109fd0 0x7fcf341a5dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:47.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:47.956+0000 7fcf3a600700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf3410fe00 con 0x7fcf34109fd0 2024-03-28T16:02:47.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:47.956+0000 7fcf3a600700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf3410fc80 con 0x7fcf340761c0 2024-03-28T16:02:47.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:47.956+0000 7fcf3a600700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf3410ff80 con 0x7fcf34074bf0 2024-03-28T16:02:47.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:47.956+0000 7fcf39c00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf34109fd0 0x7fcf341a5dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:47.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:47.956+0000 7fcf39c00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf34109fd0 0x7fcf341a5dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47918/0 (socket says 172.21.15.67:47918) 2024-03-28T16:02:47.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:47.956+0000 7fcf39c00700 1 -- 172.21.15.67:0/3355304439 learned_addr learned my addr 172.21.15.67:0/3355304439 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:02:47.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:47.956+0000 7fcf39200700 1 --2- 172.21.15.67:0/3355304439 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcf34074bf0 0x7fcf3419e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:47.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:47.956+0000 7fcf33e00700 1 --2- 172.21.15.67:0/3355304439 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf340761c0 0x7fcf3419ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:47.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:47.957+0000 7fcf33e00700 1 -- 172.21.15.67:0/3355304439 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcf34074bf0 msgr2=0x7fcf3419e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:47.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:47.957+0000 7fcf33e00700 1 --2- 172.21.15.67:0/3355304439 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcf34074bf0 0x7fcf3419e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:47.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:47.957+0000 7fcf33e00700 1 -- 172.21.15.67:0/3355304439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf34109fd0 msgr2=0x7fcf341a5dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:47.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:47.957+0000 7fcf33e00700 1 --2- 172.21.15.67:0/3355304439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf34109fd0 0x7fcf341a5dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:47.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:47.957+0000 7fcf33e00700 1 -- 172.21.15.67:0/3355304439 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf341a6460 con 0x7fcf340761c0 2024-03-28T16:02:47.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:47.957+0000 7fcf39200700 1 --2- 172.21.15.67:0/3355304439 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcf34074bf0 0x7fcf3419e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T16:02:47.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:47.957+0000 7fcf39c00700 1 --2- 172.21.15.67:0/3355304439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf34109fd0 0x7fcf341a5dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T16:02:47.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:47.958+0000 7fcf33e00700 1 --2- 172.21.15.67:0/3355304439 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf340761c0 0x7fcf3419ed40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fcf2000d720 tx=0x7fcf2000da30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:02:47.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:47.958+0000 7fcf31600700 1 -- 172.21.15.67:0/3355304439 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf20016070 con 0x7fcf340761c0 2024-03-28T16:02:47.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:47.958+0000 7fcf31600700 1 -- 172.21.15.67:0/3355304439 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fcf20005020 con 0x7fcf340761c0 2024-03-28T16:02:47.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:47.958+0000 7fcf31600700 1 -- 172.21.15.67:0/3355304439 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf2000faf0 con 0x7fcf340761c0 2024-03-28T16:02:47.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:47.959+0000 7fcf3a600700 1 -- 172.21.15.67:0/3355304439 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf341a6750 con 0x7fcf340761c0 2024-03-28T16:02:47.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:47.959+0000 7fcf3a600700 1 -- 172.21.15.67:0/3355304439 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf341a6c40 con 0x7fcf340761c0 2024-03-28T16:02:47.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:47.960+0000 7fcf3a600700 1 -- 172.21.15.67:0/3355304439 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf34068e50 con 0x7fcf340761c0 2024-03-28T16:02:47.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:47.963+0000 7fcf31600700 1 -- 172.21.15.67:0/3355304439 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fcf200051d0 con 0x7fcf340761c0 2024-03-28T16:02:47.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:47.965+0000 7fcf31600700 1 --2- 172.21.15.67:0/3355304439 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fcf1c0a2910 0x7fcf1c0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:47.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:47.965+0000 7fcf31600700 1 -- 172.21.15.67:0/3355304439 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fcf200ce8c0 con 0x7fcf340761c0 2024-03-28T16:02:47.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:47.965+0000 7fcf39200700 1 --2- 172.21.15.67:0/3355304439 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fcf1c0a2910 0x7fcf1c0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:47.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:47.966+0000 7fcf39200700 1 --2- 172.21.15.67:0/3355304439 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fcf1c0a2910 0x7fcf1c0a4dd0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fcf28005950 tx=0x7fcf2800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:02:47.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:47.985+0000 7fcf31600700 1 -- 172.21.15.67:0/3355304439 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fcf2009b970 con 0x7fcf340761c0 2024-03-28T16:02:48.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:48.234+0000 7fcf3a600700 1 -- 172.21.15.67:0/3355304439 --> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7fcf340625d0 con 0x7fcf1c0a2910 2024-03-28T16:02:48.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:48.235+0000 7fcf31600700 1 -- 172.21.15.67:0/3355304439 <== mgr.14152 v2:172.21.15.67:6800/208726654 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fcf340625d0 con 0x7fcf1c0a2910 2024-03-28T16:02:48.236 INFO:teuthology.orchestra.run.smithi067.stdout:Backend: cephadm 2024-03-28T16:02:48.236 INFO:teuthology.orchestra.run.smithi067.stdout:Available: Yes 2024-03-28T16:02:48.236 INFO:teuthology.orchestra.run.smithi067.stdout:Paused: No 2024-03-28T16:02:48.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:48.238+0000 7fcf3a600700 1 -- 172.21.15.67:0/3355304439 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fcf1c0a2910 msgr2=0x7fcf1c0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:48.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:48.238+0000 7fcf3a600700 1 --2- 172.21.15.67:0/3355304439 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fcf1c0a2910 0x7fcf1c0a4dd0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fcf28005950 tx=0x7fcf2800b410 comp rx=0 tx=0).stop 2024-03-28T16:02:48.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:48.238+0000 7fcf3a600700 1 -- 172.21.15.67:0/3355304439 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf340761c0 msgr2=0x7fcf3419ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:48.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:48.238+0000 7fcf3a600700 1 --2- 172.21.15.67:0/3355304439 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf340761c0 0x7fcf3419ed40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fcf2000d720 tx=0x7fcf2000da30 comp rx=0 tx=0).stop 2024-03-28T16:02:48.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:48.239+0000 7fcf3a600700 1 -- 172.21.15.67:0/3355304439 shutdown_connections 2024-03-28T16:02:48.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:48.239+0000 7fcf3a600700 1 --2- 172.21.15.67:0/3355304439 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fcf1c0a2910 0x7fcf1c0a4dd0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:48.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:48.239+0000 7fcf3a600700 1 --2- 172.21.15.67:0/3355304439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf34109fd0 0x7fcf341a5dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:48.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:48.239+0000 7fcf3a600700 1 --2- 172.21.15.67:0/3355304439 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcf34074bf0 0x7fcf3419e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:48.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:48.239+0000 7fcf3a600700 1 --2- 172.21.15.67:0/3355304439 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf340761c0 0x7fcf3419ed40 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:48.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:48.239+0000 7fcf3a600700 1 -- 172.21.15.67:0/3355304439 >> 172.21.15.67:0/3355304439 conn(0x7fcf340fd6a0 msgr2=0x7fcf34101ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:02:48.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:48.239+0000 7fcf3a600700 1 -- 172.21.15.67:0/3355304439 shutdown_connections 2024-03-28T16:02:48.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:48.239+0000 7fcf3a600700 1 -- 172.21.15.67:0/3355304439 wait complete. 2024-03-28T16:02:48.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:48 smithi067 ceph-mon[31707]: pgmap v416: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-28T16:02:48.968 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-28T16:02:49.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:48 smithi150 ceph-mon[34825]: pgmap v416: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-28T16:02:49.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:48 smithi183 ceph-mon[35580]: pgmap v416: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-28T16:02:49.434 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T16:02:49.910 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:49 smithi067 ceph-mon[31707]: from='client.34778 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T16:02:50.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:49 smithi150 ceph-mon[34825]: from='client.34778 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T16:02:50.102 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:49 smithi183 ceph-mon[35580]: from='client.34778 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T16:02:50.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:50 smithi067 ceph-mon[31707]: pgmap v417: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-28T16:02:51.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:50 smithi150 ceph-mon[34825]: pgmap v417: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-28T16:02:51.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:50 smithi183 ceph-mon[35580]: pgmap v417: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-28T16:02:51.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:51.532+0000 7f3e7b200700 1 -- 172.21.15.67:0/2567675846 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e7c103c80 msgr2=0x7f3e7c10a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:51.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:51.532+0000 7f3e7b200700 1 --2- 172.21.15.67:0/2567675846 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e7c103c80 0x7f3e7c10a530 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f3e7400b0d0 tx=0x7f3e7400b3e0 comp rx=0 tx=0).stop 2024-03-28T16:02:51.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:51.532+0000 7f3e7b200700 1 -- 172.21.15.67:0/2567675846 shutdown_connections 2024-03-28T16:02:51.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:51.532+0000 7f3e7b200700 1 --2- 172.21.15.67:0/2567675846 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e7c103c80 0x7f3e7c10a530 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:51.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:51.532+0000 7f3e7b200700 1 --2- 172.21.15.67:0/2567675846 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e7c1020e0 0x7f3e7c102500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:51.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:51.532+0000 7f3e7b200700 1 --2- 172.21.15.67:0/2567675846 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3e7c1032e0 0x7f3e7c103740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:51.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:51.532+0000 7f3e7b200700 1 -- 172.21.15.67:0/2567675846 >> 172.21.15.67:0/2567675846 conn(0x7f3e7c0fd6a0 msgr2=0x7f3e7c0ffac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:02:51.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:51.532+0000 7f3e7b200700 1 -- 172.21.15.67:0/2567675846 shutdown_connections 2024-03-28T16:02:51.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:51.532+0000 7f3e7b200700 1 -- 172.21.15.67:0/2567675846 wait complete. 2024-03-28T16:02:51.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:51.533+0000 7f3e7b200700 1 Processor -- start 2024-03-28T16:02:51.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:51.533+0000 7f3e7b200700 1 -- start start 2024-03-28T16:02:51.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:51.534+0000 7f3e7b200700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e7c1020e0 0x7f3e7c19a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:51.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:51.534+0000 7f3e7b200700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3e7c1032e0 0x7f3e7c19a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:51.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:51.534+0000 7f3e7b200700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e7c103c80 0x7f3e7c1a1a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:51.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:51.534+0000 7f3e7b200700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e7c10dbe0 con 0x7f3e7c1020e0 2024-03-28T16:02:51.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:51.534+0000 7f3e7b200700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e7c10da60 con 0x7f3e7c103c80 2024-03-28T16:02:51.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:51.534+0000 7f3e7b200700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e7c10dd60 con 0x7f3e7c1032e0 2024-03-28T16:02:51.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:51.534+0000 7f3e79e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e7c1020e0 0x7f3e7c19a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:51.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:51.534+0000 7f3e79e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e7c1020e0 0x7f3e7c19a440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51628/0 (socket says 172.21.15.67:51628) 2024-03-28T16:02:51.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:51.534+0000 7f3e79e00700 1 -- 172.21.15.67:0/710301718 learned_addr learned my addr 172.21.15.67:0/710301718 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:02:51.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:51.535+0000 7f3e79400700 1 --2- 172.21.15.67:0/710301718 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3e7c1032e0 0x7f3e7c19a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:51.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:51.535+0000 7f3e79e00700 1 -- 172.21.15.67:0/710301718 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3e7c1032e0 msgr2=0x7f3e7c19a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:51.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:51.535+0000 7f3e79e00700 1 --2- 172.21.15.67:0/710301718 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3e7c1032e0 0x7f3e7c19a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:51.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:51.535+0000 7f3e79e00700 1 -- 172.21.15.67:0/710301718 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e7c103c80 msgr2=0x7f3e7c1a1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:51.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:51.535+0000 7f3e7a800700 1 --2- 172.21.15.67:0/710301718 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e7c103c80 0x7f3e7c1a1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:51.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:51.535+0000 7f3e79e00700 1 --2- 172.21.15.67:0/710301718 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e7c103c80 0x7f3e7c1a1a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:51.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:51.535+0000 7f3e79e00700 1 -- 172.21.15.67:0/710301718 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e7c1a20a0 con 0x7f3e7c1020e0 2024-03-28T16:02:51.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:51.535+0000 7f3e7a800700 1 --2- 172.21.15.67:0/710301718 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e7c103c80 0x7f3e7c1a1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:02:51.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:51.535+0000 7f3e79e00700 1 --2- 172.21.15.67:0/710301718 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e7c1020e0 0x7f3e7c19a440 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f3e6c00d9b0 tx=0x7f3e6c00dd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:02:51.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:51.535+0000 7f3e72a00700 1 -- 172.21.15.67:0/710301718 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e6c017030 con 0x7f3e7c1020e0 2024-03-28T16:02:51.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:51.535+0000 7f3e72a00700 1 -- 172.21.15.67:0/710301718 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3e6c0044d0 con 0x7f3e7c1020e0 2024-03-28T16:02:51.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:51.535+0000 7f3e7b200700 1 -- 172.21.15.67:0/710301718 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e7c1a2390 con 0x7f3e7c1020e0 2024-03-28T16:02:51.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:51.535+0000 7f3e72a00700 1 -- 172.21.15.67:0/710301718 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e6c0039e0 con 0x7f3e7c1020e0 2024-03-28T16:02:51.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:51.536+0000 7f3e7b200700 1 -- 172.21.15.67:0/710301718 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e7c1a28d0 con 0x7f3e7c1020e0 2024-03-28T16:02:51.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:51.537+0000 7f3e7b200700 1 -- 172.21.15.67:0/710301718 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e7c068e50 con 0x7f3e7c1020e0 2024-03-28T16:02:51.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:51.539+0000 7f3e72a00700 1 -- 172.21.15.67:0/710301718 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f3e6c01a070 con 0x7f3e7c1020e0 2024-03-28T16:02:51.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:51.540+0000 7f3e72a00700 1 --2- 172.21.15.67:0/710301718 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f3e600a2890 0x7f3e600a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:51.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:51.540+0000 7f3e72a00700 1 -- 172.21.15.67:0/710301718 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f3e6c0ce7f0 con 0x7f3e7c1020e0 2024-03-28T16:02:51.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:51.540+0000 7f3e79400700 1 --2- 172.21.15.67:0/710301718 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f3e600a2890 0x7f3e600a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:51.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:51.541+0000 7f3e79400700 1 --2- 172.21.15.67:0/710301718 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f3e600a2890 0x7f3e600a4d50 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f3e68005950 tx=0x7f3e680058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:02:51.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:51.543+0000 7f3e72a00700 1 -- 172.21.15.67:0/710301718 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3e6c09b840 con 0x7f3e7c1020e0 2024-03-28T16:02:51.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:51.784+0000 7f3e7b200700 1 -- 172.21.15.67:0/710301718 --> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f3e7c1a2b90 con 0x7f3e600a2890 2024-03-28T16:02:51.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:51.826+0000 7f3e72a00700 1 -- 172.21.15.67:0/710301718 <== mgr.14152 v2:172.21.15.67:6800/208726654 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+4185 (secure 0 0 0) 0x7f3e7c1a2b90 con 0x7f3e600a2890 2024-03-28T16:02:51.827 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-28T16:02:51.827 INFO:teuthology.orchestra.run.smithi067.stdout:mds.a smithi067 running (114s) 22s ago 113s 12.3M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 0d13172c8488 2024-03-28T16:02:51.827 INFO:teuthology.orchestra.run.smithi067.stdout:mds.b smithi150 running (110s) 37s ago 109s 24.2M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 b6cc8cd55de5 2024-03-28T16:02:51.827 INFO:teuthology.orchestra.run.smithi067.stdout:mds.c smithi183 running (2m) 38s ago 2m 12.3M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 2b20d7441f6a 2024-03-28T16:02:51.827 INFO:teuthology.orchestra.run.smithi067.stdout:mds.d smithi067 running (2m) 22s ago 2m 12.4M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 92b9cad2c191 2024-03-28T16:02:51.827 INFO:teuthology.orchestra.run.smithi067.stdout:mds.e smithi150 running (2m) 37s ago 2m 12.3M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 d7898e35df8d 2024-03-28T16:02:51.827 INFO:teuthology.orchestra.run.smithi067.stdout:mds.f smithi183 running (2m) 38s ago 2m 12.5M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 fc616700c002 2024-03-28T16:02:51.827 INFO:teuthology.orchestra.run.smithi067.stdout:mds.g smithi067 running (118s) 22s ago 117s 12.3M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 a38fb47a48ac 2024-03-28T16:02:51.828 INFO:teuthology.orchestra.run.smithi067.stdout:mds.h smithi150 running (2m) 37s ago 2m 16.9M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 b4dfaf989e08 2024-03-28T16:02:51.828 INFO:teuthology.orchestra.run.smithi067.stdout:mds.i smithi183 running (2m) 38s ago 2m 12.2M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 999fdaab1407 2024-03-28T16:02:51.828 INFO:teuthology.orchestra.run.smithi067.stdout:mds.j smithi067 running (2m) 22s ago 2m 12.5M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 dd754e787437 2024-03-28T16:02:51.828 INFO:teuthology.orchestra.run.smithi067.stdout:mds.k smithi150 running (2m) 37s ago 2m 12.5M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 8b52dd296b4c 2024-03-28T16:02:51.828 INFO:teuthology.orchestra.run.smithi067.stdout:mds.l smithi183 running (2m) 38s ago 2m 12.4M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 4489b0bde908 2024-03-28T16:02:51.828 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.x smithi067 *:9283,8765 running (12m) 22s ago 12m 513M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 a4c2b1cc1e78 2024-03-28T16:02:51.828 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.y smithi150 *:8443,8765 running (9m) 37s ago 9m 451M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 d5b1a132ba2e 2024-03-28T16:02:51.828 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.z smithi183 *:8443,8765 running (9m) 38s ago 9m 451M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 66d2d9e37125 2024-03-28T16:02:51.828 INFO:teuthology.orchestra.run.smithi067.stdout:mon.a smithi067 running (12m) 22s ago 12m 49.5M 2048M 19.0.0-2145-g3be9a5c9 c664afe56bb0 ecc4fb80b575 2024-03-28T16:02:51.828 INFO:teuthology.orchestra.run.smithi067.stdout:mon.b smithi150 running (9m) 37s ago 9m 39.5M 2048M 19.0.0-2145-g3be9a5c9 c664afe56bb0 df1c336724bc 2024-03-28T16:02:51.828 INFO:teuthology.orchestra.run.smithi067.stdout:mon.c smithi183 running (10m) 38s ago 10m 37.1M 2048M 19.0.0-2145-g3be9a5c9 c664afe56bb0 d8279387b55f 2024-03-28T16:02:51.828 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (9m) 22s ago 9m 54.2M 4272M 19.0.0-2145-g3be9a5c9 c664afe56bb0 bbb9444dd28c 2024-03-28T16:02:51.828 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi150 running (8m) 37s ago 8m 48.3M 4272M 19.0.0-2145-g3be9a5c9 c664afe56bb0 4d90ec1f75bd 2024-03-28T16:02:51.828 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi183 running (8m) 38s ago 8m 50.5M 4272M 19.0.0-2145-g3be9a5c9 c664afe56bb0 df90457e5984 2024-03-28T16:02:51.828 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi067 running (7m) 22s ago 7m 53.0M 4272M 19.0.0-2145-g3be9a5c9 c664afe56bb0 b47291b09f6d 2024-03-28T16:02:51.829 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi150 running (6m) 37s ago 6m 44.7M 4272M 19.0.0-2145-g3be9a5c9 c664afe56bb0 94d188816840 2024-03-28T16:02:51.829 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi183 running (6m) 38s ago 6m 52.0M 4272M 19.0.0-2145-g3be9a5c9 c664afe56bb0 8e103876fd53 2024-03-28T16:02:51.829 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6 smithi067 running (5m) 22s ago 5m 50.6M 4272M 19.0.0-2145-g3be9a5c9 c664afe56bb0 65fb70e8afa4 2024-03-28T16:02:51.829 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7 smithi150 running (5m) 37s ago 5m 52.8M 4272M 19.0.0-2145-g3be9a5c9 c664afe56bb0 53c7cd410f36 2024-03-28T16:02:51.829 INFO:teuthology.orchestra.run.smithi067.stdout:osd.8 smithi183 running (4m) 38s ago 4m 45.8M 4272M 19.0.0-2145-g3be9a5c9 c664afe56bb0 c61fd9797c7c 2024-03-28T16:02:51.829 INFO:teuthology.orchestra.run.smithi067.stdout:osd.9 smithi067 running (4m) 22s ago 4m 43.5M 4272M 19.0.0-2145-g3be9a5c9 c664afe56bb0 8fd3cd101854 2024-03-28T16:02:51.829 INFO:teuthology.orchestra.run.smithi067.stdout:osd.10 smithi150 running (3m) 37s ago 3m 52.5M 4272M 19.0.0-2145-g3be9a5c9 c664afe56bb0 7fb4df68f28e 2024-03-28T16:02:51.829 INFO:teuthology.orchestra.run.smithi067.stdout:osd.11 smithi183 running (2m) 38s ago 2m 42.8M 4272M 19.0.0-2145-g3be9a5c9 c664afe56bb0 91ed0d98f521 2024-03-28T16:02:51.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:51.829+0000 7f3e7b200700 1 -- 172.21.15.67:0/710301718 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f3e600a2890 msgr2=0x7f3e600a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:51.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:51.829+0000 7f3e7b200700 1 --2- 172.21.15.67:0/710301718 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f3e600a2890 0x7f3e600a4d50 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f3e68005950 tx=0x7f3e680058e0 comp rx=0 tx=0).stop 2024-03-28T16:02:51.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:51.829+0000 7f3e7b200700 1 -- 172.21.15.67:0/710301718 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e7c1020e0 msgr2=0x7f3e7c19a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:51.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:51.829+0000 7f3e7b200700 1 --2- 172.21.15.67:0/710301718 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e7c1020e0 0x7f3e7c19a440 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f3e6c00d9b0 tx=0x7f3e6c00dd70 comp rx=0 tx=0).stop 2024-03-28T16:02:51.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:51.829+0000 7f3e7b200700 1 -- 172.21.15.67:0/710301718 shutdown_connections 2024-03-28T16:02:51.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:51.829+0000 7f3e7b200700 1 --2- 172.21.15.67:0/710301718 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f3e600a2890 0x7f3e600a4d50 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:51.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:51.829+0000 7f3e7b200700 1 --2- 172.21.15.67:0/710301718 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e7c103c80 0x7f3e7c1a1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:51.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:51.829+0000 7f3e7b200700 1 --2- 172.21.15.67:0/710301718 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e7c1020e0 0x7f3e7c19a440 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:51.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:51.829+0000 7f3e7b200700 1 --2- 172.21.15.67:0/710301718 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3e7c1032e0 0x7f3e7c19a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:51.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:51.829+0000 7f3e7b200700 1 -- 172.21.15.67:0/710301718 >> 172.21.15.67:0/710301718 conn(0x7f3e7c0fd6a0 msgr2=0x7f3e7c0743c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:02:51.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:51.830+0000 7f3e7b200700 1 -- 172.21.15.67:0/710301718 shutdown_connections 2024-03-28T16:02:51.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:51.830+0000 7f3e7b200700 1 -- 172.21.15.67:0/710301718 wait complete. 2024-03-28T16:02:52.649 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-28T16:02:53.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:52 smithi150 ceph-mon[34825]: pgmap v418: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-28T16:02:53.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:52 smithi150 ceph-mon[34825]: from='client.15060 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T16:02:53.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:52 smithi067 ceph-mon[31707]: pgmap v418: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-28T16:02:53.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:52 smithi067 ceph-mon[31707]: from='client.15060 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T16:02:53.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:52 smithi183 ceph-mon[35580]: pgmap v418: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-28T16:02:53.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:52 smithi183 ceph-mon[35580]: from='client.15060 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T16:02:53.114 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T16:02:54.896 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:54 smithi067 ceph-mon[31707]: pgmap v419: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 0 B/s wr, 1 op/s 2024-03-28T16:02:54.903 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:54 smithi183 ceph-mon[35580]: pgmap v419: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 0 B/s wr, 1 op/s 2024-03-28T16:02:55.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:54 smithi150 ceph-mon[34825]: pgmap v419: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 0 B/s wr, 1 op/s 2024-03-28T16:02:55.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:55.152+0000 7fbf7f800700 1 -- 172.21.15.67:0/3839117719 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf78102c60 msgr2=0x7fbf78109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:55.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:55.152+0000 7fbf7f800700 1 --2- 172.21.15.67:0/3839117719 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf78102c60 0x7fbf78109510 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fbf7000b0d0 tx=0x7fbf7000b3e0 comp rx=0 tx=0).stop 2024-03-28T16:02:55.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:55.152+0000 7fbf7f800700 1 -- 172.21.15.67:0/3839117719 shutdown_connections 2024-03-28T16:02:55.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:55.152+0000 7fbf7f800700 1 --2- 172.21.15.67:0/3839117719 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf78102c60 0x7fbf78109510 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:55.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:55.152+0000 7fbf7f800700 1 --2- 172.21.15.67:0/3839117719 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf781010c0 0x7fbf781014e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:55.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:55.152+0000 7fbf7f800700 1 --2- 172.21.15.67:0/3839117719 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbf781022c0 0x7fbf78102720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:55.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:55.152+0000 7fbf7f800700 1 -- 172.21.15.67:0/3839117719 >> 172.21.15.67:0/3839117719 conn(0x7fbf780fc660 msgr2=0x7fbf780feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:02:55.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:55.152+0000 7fbf7f800700 1 -- 172.21.15.67:0/3839117719 shutdown_connections 2024-03-28T16:02:55.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:55.152+0000 7fbf7f800700 1 -- 172.21.15.67:0/3839117719 wait complete. 2024-03-28T16:02:55.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:55.153+0000 7fbf7f800700 1 Processor -- start 2024-03-28T16:02:55.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:55.153+0000 7fbf7f800700 1 -- start start 2024-03-28T16:02:55.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:55.154+0000 7fbf7f800700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf781010c0 0x7fbf7819a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:55.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:55.154+0000 7fbf7f800700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbf781022c0 0x7fbf7819aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:55.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:55.154+0000 7fbf7f800700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf78102c60 0x7fbf781a1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:55.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:55.154+0000 7fbf7f800700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf7810cbb0 con 0x7fbf781010c0 2024-03-28T16:02:55.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:55.154+0000 7fbf7f800700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf7810ca30 con 0x7fbf78102c60 2024-03-28T16:02:55.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:55.154+0000 7fbf7f800700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf7810cd30 con 0x7fbf781022c0 2024-03-28T16:02:55.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:55.154+0000 7fbf7ce00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf781010c0 0x7fbf7819a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:55.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:55.154+0000 7fbf7ce00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf781010c0 0x7fbf7819a4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51648/0 (socket says 172.21.15.67:51648) 2024-03-28T16:02:55.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:55.154+0000 7fbf7ce00700 1 -- 172.21.15.67:0/1450788859 learned_addr learned my addr 172.21.15.67:0/1450788859 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:02:55.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:55.154+0000 7fbf77e00700 1 --2- 172.21.15.67:0/1450788859 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbf781022c0 0x7fbf7819aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:55.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:55.154+0000 7fbf7d800700 1 --2- 172.21.15.67:0/1450788859 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf78102c60 0x7fbf781a1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:55.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:55.155+0000 7fbf7ce00700 1 -- 172.21.15.67:0/1450788859 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbf781022c0 msgr2=0x7fbf7819aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:55.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:55.155+0000 7fbf7ce00700 1 --2- 172.21.15.67:0/1450788859 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbf781022c0 0x7fbf7819aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:55.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:55.155+0000 7fbf7ce00700 1 -- 172.21.15.67:0/1450788859 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf78102c60 msgr2=0x7fbf781a1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:55.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:55.155+0000 7fbf7ce00700 1 --2- 172.21.15.67:0/1450788859 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf78102c60 0x7fbf781a1ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:55.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:55.155+0000 7fbf7ce00700 1 -- 172.21.15.67:0/1450788859 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf781a2150 con 0x7fbf781010c0 2024-03-28T16:02:55.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:55.155+0000 7fbf7ce00700 1 --2- 172.21.15.67:0/1450788859 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf781010c0 0x7fbf7819a4f0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c00c9a0 tx=0x7fbf6c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:02:55.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:55.155+0000 7fbf75600700 1 -- 172.21.15.67:0/1450788859 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6c004220 con 0x7fbf781010c0 2024-03-28T16:02:55.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:55.155+0000 7fbf75600700 1 -- 172.21.15.67:0/1450788859 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbf6c007d10 con 0x7fbf781010c0 2024-03-28T16:02:55.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:55.155+0000 7fbf75600700 1 -- 172.21.15.67:0/1450788859 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6c003d00 con 0x7fbf781010c0 2024-03-28T16:02:55.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:55.155+0000 7fbf7f800700 1 -- 172.21.15.67:0/1450788859 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf781a2440 con 0x7fbf781010c0 2024-03-28T16:02:55.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:55.155+0000 7fbf7f800700 1 -- 172.21.15.67:0/1450788859 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf781a29d0 con 0x7fbf781010c0 2024-03-28T16:02:55.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:55.156+0000 7fbf7f800700 1 -- 172.21.15.67:0/1450788859 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf78068e50 con 0x7fbf781010c0 2024-03-28T16:02:55.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:55.159+0000 7fbf75600700 1 -- 172.21.15.67:0/1450788859 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fbf6c007860 con 0x7fbf781010c0 2024-03-28T16:02:55.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:55.160+0000 7fbf75600700 1 --2- 172.21.15.67:0/1450788859 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fbf600a2910 0x7fbf600a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:55.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:55.160+0000 7fbf75600700 1 -- 172.21.15.67:0/1450788859 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fbf6c0ce9e0 con 0x7fbf781010c0 2024-03-28T16:02:55.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:55.160+0000 7fbf77e00700 1 --2- 172.21.15.67:0/1450788859 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fbf600a2910 0x7fbf600a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:55.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:55.161+0000 7fbf77e00700 1 --2- 172.21.15.67:0/1450788859 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fbf600a2910 0x7fbf600a4dd0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fbf64004630 tx=0x7fbf6400b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:02:55.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:55.163+0000 7fbf75600700 1 -- 172.21.15.67:0/1450788859 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fbf6c014030 con 0x7fbf781010c0 2024-03-28T16:02:55.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:55.400+0000 7fbf7f800700 1 -- 172.21.15.67:0/1450788859 --> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fbf780fe820 con 0x7fbf600a2910 2024-03-28T16:02:55.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:55.408+0000 7fbf75600700 1 -- 172.21.15.67:0/1450788859 <== mgr.14152 v2:172.21.15.67:6800/208726654 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+980 (secure 0 0 0) 0x7fbf780fe820 con 0x7fbf600a2910 2024-03-28T16:02:55.409 INFO:teuthology.orchestra.run.smithi067.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-28T16:02:55.409 INFO:teuthology.orchestra.run.smithi067.stdout:mds.all 12/12 41s ago 2m smithi067=a;smithi067=d;smithi067=g;smithi067=j;smithi150=b;smithi150=e;smithi150=h;smithi150=k;smithi183=c;smithi183=f;smithi183=i;smithi183=l;count:12 2024-03-28T16:02:55.409 INFO:teuthology.orchestra.run.smithi067.stdout:mgr 3/3 41s ago 9m smithi067=x;smithi150=y;smithi183=z;count:3 2024-03-28T16:02:55.409 INFO:teuthology.orchestra.run.smithi067.stdout:mon 3/3 41s ago 10m smithi067:172.21.15.67=a;smithi150:172.21.15.150=b;smithi183:172.21.15.183=c;count:3 2024-03-28T16:02:55.409 INFO:teuthology.orchestra.run.smithi067.stdout:osd 12 41s ago - 2024-03-28T16:02:55.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:55.411+0000 7fbf7f800700 1 -- 172.21.15.67:0/1450788859 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fbf600a2910 msgr2=0x7fbf600a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:55.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:55.411+0000 7fbf7f800700 1 --2- 172.21.15.67:0/1450788859 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fbf600a2910 0x7fbf600a4dd0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fbf64004630 tx=0x7fbf6400b3c0 comp rx=0 tx=0).stop 2024-03-28T16:02:55.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:55.411+0000 7fbf7f800700 1 -- 172.21.15.67:0/1450788859 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf781010c0 msgr2=0x7fbf7819a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:55.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:55.411+0000 7fbf7f800700 1 --2- 172.21.15.67:0/1450788859 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf781010c0 0x7fbf7819a4f0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c00c9a0 tx=0x7fbf6c00ccb0 comp rx=0 tx=0).stop 2024-03-28T16:02:55.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:55.411+0000 7fbf7f800700 1 -- 172.21.15.67:0/1450788859 shutdown_connections 2024-03-28T16:02:55.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:55.411+0000 7fbf7f800700 1 --2- 172.21.15.67:0/1450788859 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fbf600a2910 0x7fbf600a4dd0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:55.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:55.411+0000 7fbf7f800700 1 --2- 172.21.15.67:0/1450788859 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf78102c60 0x7fbf781a1ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:55.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:55.411+0000 7fbf7f800700 1 --2- 172.21.15.67:0/1450788859 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf781010c0 0x7fbf7819a4f0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:55.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:55.411+0000 7fbf7f800700 1 --2- 172.21.15.67:0/1450788859 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbf781022c0 0x7fbf7819aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:55.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:55.411+0000 7fbf7f800700 1 -- 172.21.15.67:0/1450788859 >> 172.21.15.67:0/1450788859 conn(0x7fbf780fc660 msgr2=0x7fbf780fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:02:55.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:55.411+0000 7fbf7f800700 1 -- 172.21.15.67:0/1450788859 shutdown_connections 2024-03-28T16:02:55.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:55.411+0000 7fbf7f800700 1 -- 172.21.15.67:0/1450788859 wait complete. 2024-03-28T16:02:56.186 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-28T16:02:56.663 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T16:02:56.985 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:56 smithi067 ceph-mon[31707]: pgmap v420: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 2 op/s 2024-03-28T16:02:56.985 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:56 smithi067 ceph-mon[31707]: from='client.15066 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T16:02:57.054 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:56 smithi150 ceph-mon[34825]: pgmap v420: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 2 op/s 2024-03-28T16:02:57.055 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:56 smithi150 ceph-mon[34825]: from='client.15066 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T16:02:57.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:56 smithi183 ceph-mon[35580]: pgmap v420: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 2 op/s 2024-03-28T16:02:57.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:56 smithi183 ceph-mon[35580]: from='client.15066 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T16:02:58.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:58.780+0000 7f43b6c00700 1 -- 172.21.15.67:0/4204541829 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43b0102c10 msgr2=0x7f43b0105000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:58.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:58.780+0000 7f43b6c00700 1 --2- 172.21.15.67:0/4204541829 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43b0102c10 0x7f43b0105000 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f43a4009990 tx=0x7f43a4009ca0 comp rx=0 tx=0).stop 2024-03-28T16:02:58.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:58.780+0000 7f43b6c00700 1 -- 172.21.15.67:0/4204541829 shutdown_connections 2024-03-28T16:02:58.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:58.780+0000 7f43b6c00700 1 --2- 172.21.15.67:0/4204541829 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43b0107e70 0x7f43b010a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:58.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:58.780+0000 7f43b6c00700 1 --2- 172.21.15.67:0/4204541829 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43b0102c10 0x7f43b0105000 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:58.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:58.780+0000 7f43b6c00700 1 --2- 172.21.15.67:0/4204541829 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f43b0105540 0x7f43b0107930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:58.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:58.780+0000 7f43b6c00700 1 -- 172.21.15.67:0/4204541829 >> 172.21.15.67:0/4204541829 conn(0x7f43b00fc580 msgr2=0x7f43b00fe9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:02:58.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:58.780+0000 7f43b6c00700 1 -- 172.21.15.67:0/4204541829 shutdown_connections 2024-03-28T16:02:58.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:58.781+0000 7f43b6c00700 1 -- 172.21.15.67:0/4204541829 wait complete. 2024-03-28T16:02:58.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:58.782+0000 7f43b6c00700 1 Processor -- start 2024-03-28T16:02:58.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:58.782+0000 7f43b6c00700 1 -- start start 2024-03-28T16:02:58.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:58.782+0000 7f43b6c00700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43b0102c10 0x7f43b01981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:58.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:58.782+0000 7f43b6c00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43b0105540 0x7f43b01986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:58.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:58.782+0000 7f43b6c00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f43b0107e70 0x7f43b019f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:58.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:58.782+0000 7f43b6c00700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b010db60 con 0x7f43b0105540 2024-03-28T16:02:58.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:58.782+0000 7f43b6c00700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b010d9e0 con 0x7f43b0102c10 2024-03-28T16:02:58.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:58.782+0000 7f43b6c00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b010dce0 con 0x7f43b0107e70 2024-03-28T16:02:58.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:58.783+0000 7f43b4e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43b0105540 0x7f43b01986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:58.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:58.783+0000 7f43b4e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43b0105540 0x7f43b01986f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51674/0 (socket says 172.21.15.67:51674) 2024-03-28T16:02:58.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:58.783+0000 7f43b4e00700 1 -- 172.21.15.67:0/680270073 learned_addr learned my addr 172.21.15.67:0/680270073 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:02:58.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:58.783+0000 7f43b5800700 1 --2- 172.21.15.67:0/680270073 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43b0102c10 0x7f43b01981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:58.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:58.783+0000 7f43b6200700 1 --2- 172.21.15.67:0/680270073 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f43b0107e70 0x7f43b019f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:58.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:58.783+0000 7f43b4e00700 1 -- 172.21.15.67:0/680270073 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f43b0107e70 msgr2=0x7f43b019f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:58.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:58.783+0000 7f43b4e00700 1 --2- 172.21.15.67:0/680270073 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f43b0107e70 0x7f43b019f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:58.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:58.783+0000 7f43b4e00700 1 -- 172.21.15.67:0/680270073 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43b0102c10 msgr2=0x7f43b01981b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:58.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:58.783+0000 7f43b4e00700 1 --2- 172.21.15.67:0/680270073 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43b0102c10 0x7f43b01981b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:58.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:58.783+0000 7f43b4e00700 1 -- 172.21.15.67:0/680270073 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43b019fe10 con 0x7f43b0105540 2024-03-28T16:02:58.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:58.784+0000 7f43b4e00700 1 --2- 172.21.15.67:0/680270073 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43b0105540 0x7f43b01986f0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f439c00e9f0 tx=0x7f439c00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:02:58.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:58.784+0000 7f43ae000700 1 -- 172.21.15.67:0/680270073 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f439c00cbd0 con 0x7f43b0105540 2024-03-28T16:02:58.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:58.784+0000 7f43ae000700 1 -- 172.21.15.67:0/680270073 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f439c00ee90 con 0x7f43b0105540 2024-03-28T16:02:58.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:58.784+0000 7f43b6c00700 1 -- 172.21.15.67:0/680270073 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43b01a0100 con 0x7f43b0105540 2024-03-28T16:02:58.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:58.784+0000 7f43ae000700 1 -- 172.21.15.67:0/680270073 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f439c005590 con 0x7f43b0105540 2024-03-28T16:02:58.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:58.784+0000 7f43b6c00700 1 -- 172.21.15.67:0/680270073 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43b01a0640 con 0x7f43b0105540 2024-03-28T16:02:58.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:58.785+0000 7f43b6c00700 1 -- 172.21.15.67:0/680270073 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43b0068e50 con 0x7f43b0105540 2024-03-28T16:02:58.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:58.787+0000 7f43ae000700 1 -- 172.21.15.67:0/680270073 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f439c004750 con 0x7f43b0105540 2024-03-28T16:02:58.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:58.789+0000 7f43ae000700 1 --2- 172.21.15.67:0/680270073 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f43980a2910 0x7f43980a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:02:58.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:58.789+0000 7f43ae000700 1 -- 172.21.15.67:0/680270073 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f439c0cec90 con 0x7f43b0105540 2024-03-28T16:02:58.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:58.789+0000 7f43b5800700 1 --2- 172.21.15.67:0/680270073 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f43980a2910 0x7f43980a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:02:58.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:58.790+0000 7f43b5800700 1 --2- 172.21.15.67:0/680270073 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f43980a2910 0x7f43980a4dd0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f43a400b710 tx=0x7f43a4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:02:58.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:58.793+0000 7f43ae000700 1 -- 172.21.15.67:0/680270073 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f439c09bce0 con 0x7f43b0105540 2024-03-28T16:02:59.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:59.026+0000 7f43b6c00700 1 -- 172.21.15.67:0/680270073 --> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f43b00625d0 con 0x7f43980a2910 2024-03-28T16:02:59.081 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:02:58 smithi067 ceph-mon[31707]: pgmap v421: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T16:02:59.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:59.031+0000 7f43ae000700 1 -- 172.21.15.67:0/680270073 <== mgr.14152 v2:172.21.15.67:6800/208726654 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+191 (secure 0 0 0) 0x7f43b00625d0 con 0x7f43980a2910 2024-03-28T16:02:59.083 INFO:teuthology.orchestra.run.smithi067.stdout:HOST ADDR LABELS STATUS 2024-03-28T16:02:59.083 INFO:teuthology.orchestra.run.smithi067.stdout:smithi067 172.21.15.67 2024-03-28T16:02:59.083 INFO:teuthology.orchestra.run.smithi067.stdout:smithi150 172.21.15.150 2024-03-28T16:02:59.083 INFO:teuthology.orchestra.run.smithi067.stdout:smithi183 172.21.15.183 2024-03-28T16:02:59.083 INFO:teuthology.orchestra.run.smithi067.stdout:3 hosts in cluster 2024-03-28T16:02:59.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:59.034+0000 7f43b6c00700 1 -- 172.21.15.67:0/680270073 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f43980a2910 msgr2=0x7f43980a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:59.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:59.034+0000 7f43b6c00700 1 --2- 172.21.15.67:0/680270073 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f43980a2910 0x7f43980a4dd0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f43a400b710 tx=0x7f43a4005fd0 comp rx=0 tx=0).stop 2024-03-28T16:02:59.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:59.034+0000 7f43b6c00700 1 -- 172.21.15.67:0/680270073 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43b0105540 msgr2=0x7f43b01986f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:02:59.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:59.034+0000 7f43b6c00700 1 --2- 172.21.15.67:0/680270073 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43b0105540 0x7f43b01986f0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f439c00e9f0 tx=0x7f439c00ed00 comp rx=0 tx=0).stop 2024-03-28T16:02:59.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:59.034+0000 7f43b6c00700 1 -- 172.21.15.67:0/680270073 shutdown_connections 2024-03-28T16:02:59.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:59.034+0000 7f43b6c00700 1 --2- 172.21.15.67:0/680270073 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f43980a2910 0x7f43980a4dd0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:59.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:59.034+0000 7f43b6c00700 1 --2- 172.21.15.67:0/680270073 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f43b0107e70 0x7f43b019f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:59.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:59.034+0000 7f43b6c00700 1 --2- 172.21.15.67:0/680270073 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43b0102c10 0x7f43b01981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:59.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:59.034+0000 7f43b6c00700 1 --2- 172.21.15.67:0/680270073 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43b0105540 0x7f43b01986f0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:02:59.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:59.034+0000 7f43b6c00700 1 -- 172.21.15.67:0/680270073 >> 172.21.15.67:0/680270073 conn(0x7f43b00fc580 msgr2=0x7f43b0103650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:02:59.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:59.034+0000 7f43b6c00700 1 -- 172.21.15.67:0/680270073 shutdown_connections 2024-03-28T16:02:59.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:02:59.035+0000 7f43b6c00700 1 -- 172.21.15.67:0/680270073 wait complete. 2024-03-28T16:02:59.088 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:02:58 smithi150 ceph-mon[34825]: pgmap v421: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T16:02:59.103 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:02:58 smithi183 ceph-mon[35580]: pgmap v421: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T16:02:59.762 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-28T16:03:00.230 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T16:03:01.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:00 smithi150 ceph-mon[34825]: from='client.15072 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T16:03:01.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:00 smithi150 ceph-mon[34825]: pgmap v422: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T16:03:01.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:00 smithi067 ceph-mon[31707]: from='client.15072 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T16:03:01.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:00 smithi067 ceph-mon[31707]: pgmap v422: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T16:03:01.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:00 smithi183 ceph-mon[35580]: from='client.15072 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T16:03:01.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:00 smithi183 ceph-mon[35580]: pgmap v422: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T16:03:02.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:02.440+0000 7fead5600700 1 -- 172.21.15.67:0/1842172497 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fead0107f50 msgr2=0x7fead010a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:02.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:02.440+0000 7fead5600700 1 --2- 172.21.15.67:0/1842172497 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fead0107f50 0x7fead010a340 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7feac800b0a0 tx=0x7feac800b3b0 comp rx=0 tx=0).stop 2024-03-28T16:03:02.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:02.440+0000 7fead5600700 1 -- 172.21.15.67:0/1842172497 shutdown_connections 2024-03-28T16:03:02.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:02.440+0000 7fead5600700 1 --2- 172.21.15.67:0/1842172497 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fead0107f50 0x7fead010a340 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:02.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:02.440+0000 7fead5600700 1 --2- 172.21.15.67:0/1842172497 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fead0102cf0 0x7fead01050e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:02.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:02.440+0000 7fead5600700 1 --2- 172.21.15.67:0/1842172497 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fead0105620 0x7fead0107a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:02.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:02.440+0000 7fead5600700 1 -- 172.21.15.67:0/1842172497 >> 172.21.15.67:0/1842172497 conn(0x7fead00fc660 msgr2=0x7fead00feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:02.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:02.440+0000 7fead5600700 1 -- 172.21.15.67:0/1842172497 shutdown_connections 2024-03-28T16:03:02.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:02.441+0000 7fead5600700 1 -- 172.21.15.67:0/1842172497 wait complete. 2024-03-28T16:03:02.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:02.442+0000 7fead5600700 1 Processor -- start 2024-03-28T16:03:02.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:02.442+0000 7fead5600700 1 -- start start 2024-03-28T16:03:02.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:02.442+0000 7fead5600700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fead0102cf0 0x7fead019a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:02.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:02.442+0000 7fead5600700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fead0105620 0x7fead019aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:02.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:02.442+0000 7fead5600700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fead0107f50 0x7fead01a1b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:02.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:02.442+0000 7fead5600700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fead010dcd0 con 0x7fead0105620 2024-03-28T16:03:02.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:02.442+0000 7fead5600700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fead010db50 con 0x7fead0102cf0 2024-03-28T16:03:02.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:02.442+0000 7fead5600700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fead010de50 con 0x7fead0107f50 2024-03-28T16:03:02.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:02.443+0000 7feace000700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fead0105620 0x7fead019aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:02.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:02.443+0000 7feace000700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fead0105620 0x7fead019aac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50944/0 (socket says 172.21.15.67:50944) 2024-03-28T16:03:02.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:02.443+0000 7feace000700 1 -- 172.21.15.67:0/1194085466 learned_addr learned my addr 172.21.15.67:0/1194085466 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:03:02.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:02.443+0000 7feacea00700 1 --2- 172.21.15.67:0/1194085466 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fead0102cf0 0x7fead019a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:02.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:02.443+0000 7feacf400700 1 --2- 172.21.15.67:0/1194085466 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fead0107f50 0x7fead01a1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:02.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:02.443+0000 7feace000700 1 -- 172.21.15.67:0/1194085466 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fead0107f50 msgr2=0x7fead01a1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:02.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:02.443+0000 7feace000700 1 --2- 172.21.15.67:0/1194085466 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fead0107f50 0x7fead01a1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:02.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:02.443+0000 7feace000700 1 -- 172.21.15.67:0/1194085466 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fead0102cf0 msgr2=0x7fead019a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:02.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:02.443+0000 7feace000700 1 --2- 172.21.15.67:0/1194085466 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fead0102cf0 0x7fead019a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:02.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:02.443+0000 7feace000700 1 -- 172.21.15.67:0/1194085466 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fead01a21e0 con 0x7fead0105620 2024-03-28T16:03:02.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:02.443+0000 7feacea00700 1 --2- 172.21.15.67:0/1194085466 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fead0102cf0 0x7fead019a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:03:02.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:02.444+0000 7feace000700 1 --2- 172.21.15.67:0/1194085466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fead0105620 0x7fead019aac0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7feabc00d9b0 tx=0x7feabc00dd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:02.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:02.444+0000 7feac7400700 1 -- 172.21.15.67:0/1194085466 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feabc017070 con 0x7fead0105620 2024-03-28T16:03:02.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:02.444+0000 7feac7400700 1 -- 172.21.15.67:0/1194085466 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7feabc004830 con 0x7fead0105620 2024-03-28T16:03:02.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:02.444+0000 7fead5600700 1 -- 172.21.15.67:0/1194085466 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fead01a24d0 con 0x7fead0105620 2024-03-28T16:03:02.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:02.444+0000 7feac7400700 1 -- 172.21.15.67:0/1194085466 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feabc0053a0 con 0x7fead0105620 2024-03-28T16:03:02.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:02.444+0000 7fead5600700 1 -- 172.21.15.67:0/1194085466 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fead01a29c0 con 0x7fead0105620 2024-03-28T16:03:02.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:02.445+0000 7fead5600700 1 -- 172.21.15.67:0/1194085466 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fead0068e50 con 0x7fead0105620 2024-03-28T16:03:02.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:02.447+0000 7feac7400700 1 -- 172.21.15.67:0/1194085466 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7feabc0049e0 con 0x7fead0105620 2024-03-28T16:03:02.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:02.449+0000 7feac7400700 1 --2- 172.21.15.67:0/1194085466 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7feab40a2890 0x7feab40a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:02.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:02.449+0000 7feac7400700 1 -- 172.21.15.67:0/1194085466 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7feabc0cee40 con 0x7fead0105620 2024-03-28T16:03:02.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:02.449+0000 7feacea00700 1 --2- 172.21.15.67:0/1194085466 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7feab40a2890 0x7feab40a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:02.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:02.450+0000 7feacea00700 1 --2- 172.21.15.67:0/1194085466 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7feab40a2890 0x7feab40a4d50 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7feac0005fd0 tx=0x7feac0005ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:02.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:02.452+0000 7feac7400700 1 -- 172.21.15.67:0/1194085466 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7feabc020020 con 0x7fead0105620 2024-03-28T16:03:02.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:02.690+0000 7fead5600700 1 -- 172.21.15.67:0/1194085466 --> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fead00625d0 con 0x7feab40a2890 2024-03-28T16:03:02.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:02.693+0000 7feac7400700 1 -- 172.21.15.67:0/1194085466 <== mgr.14152 v2:172.21.15.67:6800/208726654 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+532 (secure 0 0 0) 0x7fead00625d0 con 0x7feab40a2890 2024-03-28T16:03:02.694 INFO:teuthology.orchestra.run.smithi067.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-28T16:03:02.694 INFO:teuthology.orchestra.run.smithi067.stdout:smithi067 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT53440042400BGN 372G No 4m ago Has a FileSystem, LVM detected 2024-03-28T16:03:02.694 INFO:teuthology.orchestra.run.smithi067.stdout:smithi150 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300GX400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-03-28T16:03:02.694 INFO:teuthology.orchestra.run.smithi067.stdout:smithi183 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620500QJ400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-03-28T16:03:02.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:02.696+0000 7fead5600700 1 -- 172.21.15.67:0/1194085466 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7feab40a2890 msgr2=0x7feab40a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:02.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:02.696+0000 7fead5600700 1 --2- 172.21.15.67:0/1194085466 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7feab40a2890 0x7feab40a4d50 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7feac0005fd0 tx=0x7feac0005ee0 comp rx=0 tx=0).stop 2024-03-28T16:03:02.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:02.696+0000 7fead5600700 1 -- 172.21.15.67:0/1194085466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fead0105620 msgr2=0x7fead019aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:02.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:02.696+0000 7fead5600700 1 --2- 172.21.15.67:0/1194085466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fead0105620 0x7fead019aac0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7feabc00d9b0 tx=0x7feabc00dd70 comp rx=0 tx=0).stop 2024-03-28T16:03:02.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:02.696+0000 7fead5600700 1 -- 172.21.15.67:0/1194085466 shutdown_connections 2024-03-28T16:03:02.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:02.696+0000 7fead5600700 1 --2- 172.21.15.67:0/1194085466 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7feab40a2890 0x7feab40a4d50 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:02.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:02.696+0000 7fead5600700 1 --2- 172.21.15.67:0/1194085466 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fead0107f50 0x7fead01a1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:02.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:02.696+0000 7fead5600700 1 --2- 172.21.15.67:0/1194085466 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fead0102cf0 0x7fead019a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:02.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:02.696+0000 7fead5600700 1 --2- 172.21.15.67:0/1194085466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fead0105620 0x7fead019aac0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:02.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:02.696+0000 7fead5600700 1 -- 172.21.15.67:0/1194085466 >> 172.21.15.67:0/1194085466 conn(0x7fead00fc660 msgr2=0x7fead01024f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:02.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:02.697+0000 7fead5600700 1 -- 172.21.15.67:0/1194085466 shutdown_connections 2024-03-28T16:03:02.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:02.697+0000 7fead5600700 1 -- 172.21.15.67:0/1194085466 wait complete. 2024-03-28T16:03:03.015 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:02 smithi067 ceph-mon[31707]: pgmap v423: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:03:03.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:02 smithi150 ceph-mon[34825]: pgmap v423: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:03:03.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:02 smithi183 ceph-mon[35580]: pgmap v423: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:03:03.460 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-28T16:03:03.469 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi067.front.sepia.ceph.com 2024-03-28T16:03:03.470 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-28T16:03:03.980 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T16:03:04.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:03 smithi150 ceph-mon[34825]: from='client.15078 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T16:03:04.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:03 smithi067 ceph-mon[31707]: from='client.15078 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T16:03:04.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:03 smithi183 ceph-mon[35580]: from='client.15078 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T16:03:05.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:04 smithi150 ceph-mon[34825]: pgmap v424: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:03:05.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:04 smithi067 ceph-mon[31707]: pgmap v424: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:03:05.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:04 smithi183 ceph-mon[35580]: pgmap v424: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:03:06.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:06.120+0000 7f968d000700 1 -- 172.21.15.67:0/3726704722 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9688104410 msgr2=0x7f9688104830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:06.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:06.120+0000 7f968d000700 1 --2- 172.21.15.67:0/3726704722 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9688104410 0x7f9688104830 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f96780099c0 tx=0x7f9678009cd0 comp rx=0 tx=0).stop 2024-03-28T16:03:06.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:06.120+0000 7f968d000700 1 -- 172.21.15.67:0/3726704722 shutdown_connections 2024-03-28T16:03:06.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:06.121+0000 7f968d000700 1 --2- 172.21.15.67:0/3726704722 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9688105fb0 0x7f968810c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:06.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:06.121+0000 7f968d000700 1 --2- 172.21.15.67:0/3726704722 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9688104410 0x7f9688104830 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:06.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:06.121+0000 7f968d000700 1 --2- 172.21.15.67:0/3726704722 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9688105610 0x7f9688105a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:06.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:06.121+0000 7f968d000700 1 -- 172.21.15.67:0/3726704722 >> 172.21.15.67:0/3726704722 conn(0x7f96880ff9b0 msgr2=0x7f9688101df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:06.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:06.121+0000 7f968d000700 1 -- 172.21.15.67:0/3726704722 shutdown_connections 2024-03-28T16:03:06.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:06.121+0000 7f968d000700 1 -- 172.21.15.67:0/3726704722 wait complete. 2024-03-28T16:03:06.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:06.122+0000 7f968d000700 1 Processor -- start 2024-03-28T16:03:06.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:06.122+0000 7f968d000700 1 -- start start 2024-03-28T16:03:06.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:06.122+0000 7f968d000700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9688104410 0x7f968819a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:06.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:06.122+0000 7f968d000700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9688105610 0x7f968819aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:06.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:06.122+0000 7f968d000700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9688105fb0 0x7f96881a1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:06.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:06.122+0000 7f968d000700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f968810feb0 con 0x7f9688105610 2024-03-28T16:03:06.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:06.122+0000 7f968d000700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f968810fd30 con 0x7f9688104410 2024-03-28T16:03:06.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:06.122+0000 7f968d000700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9688110030 con 0x7f9688105fb0 2024-03-28T16:03:06.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:06.123+0000 7f9685400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9688105610 0x7f968819aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:06.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:06.123+0000 7f9685400700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9688105610 0x7f968819aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50962/0 (socket says 172.21.15.67:50962) 2024-03-28T16:03:06.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:06.123+0000 7f9685400700 1 -- 172.21.15.67:0/4087358198 learned_addr learned my addr 172.21.15.67:0/4087358198 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:03:06.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:06.123+0000 7f9685e00700 1 --2- 172.21.15.67:0/4087358198 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9688104410 0x7f968819a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:06.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:06.123+0000 7f9686800700 1 --2- 172.21.15.67:0/4087358198 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9688105fb0 0x7f96881a1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:06.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:06.123+0000 7f9685400700 1 -- 172.21.15.67:0/4087358198 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9688105fb0 msgr2=0x7f96881a1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:06.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:06.123+0000 7f9685400700 1 --2- 172.21.15.67:0/4087358198 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9688105fb0 0x7f96881a1ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:06.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:06.123+0000 7f9685400700 1 -- 172.21.15.67:0/4087358198 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9688104410 msgr2=0x7f968819a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:06.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:06.123+0000 7f9685400700 1 --2- 172.21.15.67:0/4087358198 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9688104410 0x7f968819a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:06.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:06.123+0000 7f9685400700 1 -- 172.21.15.67:0/4087358198 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96881a2150 con 0x7f9688105610 2024-03-28T16:03:06.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:06.124+0000 7f9685400700 1 --2- 172.21.15.67:0/4087358198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9688105610 0x7f968819aa30 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f967400d9b0 tx=0x7f967400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:06.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:06.124+0000 7f967ea00700 1 -- 172.21.15.67:0/4087358198 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9674016070 con 0x7f9688105610 2024-03-28T16:03:06.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:06.124+0000 7f967ea00700 1 -- 172.21.15.67:0/4087358198 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9674004830 con 0x7f9688105610 2024-03-28T16:03:06.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:06.124+0000 7f968d000700 1 -- 172.21.15.67:0/4087358198 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96881a2440 con 0x7f9688105610 2024-03-28T16:03:06.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:06.124+0000 7f967ea00700 1 -- 172.21.15.67:0/4087358198 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9674005460 con 0x7f9688105610 2024-03-28T16:03:06.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:06.124+0000 7f968d000700 1 -- 172.21.15.67:0/4087358198 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96881a29d0 con 0x7f9688105610 2024-03-28T16:03:06.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:06.125+0000 7f968d000700 1 -- 172.21.15.67:0/4087358198 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9688068e50 con 0x7f9688105610 2024-03-28T16:03:06.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:06.127+0000 7f967ea00700 1 -- 172.21.15.67:0/4087358198 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f9674005600 con 0x7f9688105610 2024-03-28T16:03:06.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:06.130+0000 7f967ea00700 1 --2- 172.21.15.67:0/4087358198 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f966c0a2910 0x7f966c0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:06.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:06.130+0000 7f967ea00700 1 -- 172.21.15.67:0/4087358198 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f96740ced50 con 0x7f9688105610 2024-03-28T16:03:06.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:06.130+0000 7f9685e00700 1 --2- 172.21.15.67:0/4087358198 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f966c0a2910 0x7f966c0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:06.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:06.131+0000 7f9685e00700 1 --2- 172.21.15.67:0/4087358198 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f966c0a2910 0x7f966c0a4dd0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f9678000c00 tx=0x7f9678005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:06.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:06.133+0000 7f967ea00700 1 -- 172.21.15.67:0/4087358198 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f967401f020 con 0x7f9688105610 2024-03-28T16:03:06.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:06.413+0000 7f968d000700 1 -- 172.21.15.67:0/4087358198 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f968802da60 con 0x7f9688105610 2024-03-28T16:03:06.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:06.413+0000 7f967ea00700 1 -- 172.21.15.67:0/4087358198 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 19 v19) v1 ==== 76+0+2958 (secure 0 0 0) 0x7f967409bda0 con 0x7f9688105610 2024-03-28T16:03:06.415 INFO:teuthology.orchestra.run.smithi067.stdout:e19 2024-03-28T16:03:06.415 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-28T16:03:06.415 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-28T16:03:06.415 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-03-28T16:03:06.415 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:03:06.415 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-03-28T16:03:06.415 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-03-28T16:03:06.415 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 19 2024-03-28T16:03:06.416 INFO:teuthology.orchestra.run.smithi067.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-28T16:03:06.416 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-03-28T16:01:07.696824+0000 2024-03-28T16:03:06.416 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-03-28T16:01:11.997773+0000 2024-03-28T16:03:06.416 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-03-28T16:03:06.416 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-03-28T16:03:06.416 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 300 2024-03-28T16:03:06.416 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-03-28T16:03:06.416 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-03-28T16:03:06.416 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-03-28T16:03:06.416 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-03-28T16:03:06.416 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-03-28T16:03:06.416 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 0 2024-03-28T16:03:06.416 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-28T16:03:06.416 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 1 2024-03-28T16:03:06.416 INFO:teuthology.orchestra.run.smithi067.stdout:in 0 2024-03-28T16:03:06.417 INFO:teuthology.orchestra.run.smithi067.stdout:up {0=34508} 2024-03-28T16:03:06.417 INFO:teuthology.orchestra.run.smithi067.stdout:failed 2024-03-28T16:03:06.417 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-03-28T16:03:06.417 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-03-28T16:03:06.417 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-03-28T16:03:06.417 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-03-28T16:03:06.417 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data disabled 2024-03-28T16:03:06.417 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-03-28T16:03:06.417 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-03-28T16:03:06.417 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 0 2024-03-28T16:03:06.417 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 34508 members: 34508 2024-03-28T16:03:06.417 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.b{0:34508} state up:active seq 3 addr [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T16:03:06.417 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.h{0:34475} state up:standby-replay seq 1 addr [v2:172.21.15.150:6836/1112886277,v1:172.21.15.150:6837/1112886277] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T16:03:06.417 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:03:06.418 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:03:06.418 INFO:teuthology.orchestra.run.smithi067.stdout:Standby daemons: 2024-03-28T16:03:06.418 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:03:06.418 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.j{-1:14514} state up:standby seq 1 addr [v2:172.21.15.67:6834/2008297659,v1:172.21.15.67:6835/2008297659] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T16:03:06.418 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.d{-1:14532} state up:standby seq 1 addr [v2:172.21.15.67:6836/320295745,v1:172.21.15.67:6837/320295745] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T16:03:06.418 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.g{-1:14613} state up:standby seq 1 addr [v2:172.21.15.67:6838/691821273,v1:172.21.15.67:6839/691821273] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T16:03:06.418 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.a{-1:14625} state up:standby seq 1 addr [v2:172.21.15.67:6840/3211254875,v1:172.21.15.67:6841/3211254875] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T16:03:06.418 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.l{-1:24436} state up:standby seq 1 addr [v2:172.21.15.183:6832/2129665423,v1:172.21.15.183:6833/2129665423] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T16:03:06.418 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.f{-1:24496} state up:standby seq 1 addr [v2:172.21.15.183:6834/1589784425,v1:172.21.15.183:6835/1589784425] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T16:03:06.418 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.c{-1:24508} state up:standby seq 1 addr [v2:172.21.15.183:6836/2185863835,v1:172.21.15.183:6837/2185863835] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T16:03:06.418 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.i{-1:24520} state up:standby seq 1 addr [v2:172.21.15.183:6838/1242814398,v1:172.21.15.183:6839/1242814398] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T16:03:06.418 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.k{-1:34409} state up:standby seq 1 addr [v2:172.21.15.150:6832/1008335521,v1:172.21.15.150:6833/1008335521] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T16:03:06.418 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.e{-1:34445} state up:standby seq 1 addr [v2:172.21.15.150:6834/3269935695,v1:172.21.15.150:6835/3269935695] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T16:03:06.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:06.416+0000 7f968d000700 1 -- 172.21.15.67:0/4087358198 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f966c0a2910 msgr2=0x7f966c0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:06.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:06.416+0000 7f968d000700 1 --2- 172.21.15.67:0/4087358198 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f966c0a2910 0x7f966c0a4dd0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f9678000c00 tx=0x7f9678005fb0 comp rx=0 tx=0).stop 2024-03-28T16:03:06.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:06.416+0000 7f968d000700 1 -- 172.21.15.67:0/4087358198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9688105610 msgr2=0x7f968819aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:06.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:06.416+0000 7f968d000700 1 --2- 172.21.15.67:0/4087358198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9688105610 0x7f968819aa30 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f967400d9b0 tx=0x7f967400dcc0 comp rx=0 tx=0).stop 2024-03-28T16:03:06.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:06.417+0000 7f968d000700 1 -- 172.21.15.67:0/4087358198 shutdown_connections 2024-03-28T16:03:06.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:06.417+0000 7f968d000700 1 --2- 172.21.15.67:0/4087358198 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f966c0a2910 0x7f966c0a4dd0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:06.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:06.417+0000 7f968d000700 1 --2- 172.21.15.67:0/4087358198 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9688105fb0 0x7f96881a1ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:06.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:06.417+0000 7f968d000700 1 --2- 172.21.15.67:0/4087358198 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9688104410 0x7f968819a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:06.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:06.417+0000 7f968d000700 1 --2- 172.21.15.67:0/4087358198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9688105610 0x7f968819aa30 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:06.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:06.417+0000 7f968d000700 1 -- 172.21.15.67:0/4087358198 >> 172.21.15.67:0/4087358198 conn(0x7f96880ff9b0 msgr2=0x7f9688108960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:06.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:06.417+0000 7f968d000700 1 -- 172.21.15.67:0/4087358198 shutdown_connections 2024-03-28T16:03:06.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:06.417+0000 7f968d000700 1 -- 172.21.15.67:0/4087358198 wait complete. 2024-03-28T16:03:06.420 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 19 2024-03-28T16:03:07.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:06 smithi150 ceph-mon[34825]: pgmap v425: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:03:07.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:06 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/4087358198' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-28T16:03:07.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:06 smithi067 ceph-mon[31707]: pgmap v425: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:03:07.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:06 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/4087358198' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-28T16:03:07.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:06 smithi183 ceph-mon[35580]: pgmap v425: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:03:07.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:06 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/4087358198' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-28T16:03:07.237 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- bash -c 'ceph osd dump' 2024-03-28T16:03:07.677 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T16:03:08.877 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:08 smithi067 ceph-mon[31707]: pgmap v426: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:03:09.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:08 smithi150 ceph-mon[34825]: pgmap v426: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:03:09.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:08 smithi183 ceph-mon[35580]: pgmap v426: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:03:09.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:09.760+0000 7f8825000700 1 -- 172.21.15.67:0/2306881619 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8820102100 msgr2=0x7f8820102520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:09.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:09.760+0000 7f8825000700 1 --2- 172.21.15.67:0/2306881619 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8820102100 0x7f8820102520 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f88100099c0 tx=0x7f8810009cd0 comp rx=0 tx=0).stop 2024-03-28T16:03:09.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:09.760+0000 7f8825000700 1 -- 172.21.15.67:0/2306881619 shutdown_connections 2024-03-28T16:03:09.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:09.760+0000 7f8825000700 1 --2- 172.21.15.67:0/2306881619 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8820103ca0 0x7f882010a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:09.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:09.760+0000 7f8825000700 1 --2- 172.21.15.67:0/2306881619 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8820102100 0x7f8820102520 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:09.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:09.760+0000 7f8825000700 1 --2- 172.21.15.67:0/2306881619 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8820103300 0x7f8820103760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:09.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:09.760+0000 7f8825000700 1 -- 172.21.15.67:0/2306881619 >> 172.21.15.67:0/2306881619 conn(0x7f88200fd6a0 msgr2=0x7f88200ffae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:09.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:09.761+0000 7f8825000700 1 -- 172.21.15.67:0/2306881619 shutdown_connections 2024-03-28T16:03:09.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:09.761+0000 7f8825000700 1 -- 172.21.15.67:0/2306881619 wait complete. 2024-03-28T16:03:09.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:09.762+0000 7f8825000700 1 Processor -- start 2024-03-28T16:03:09.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:09.762+0000 7f8825000700 1 -- start start 2024-03-28T16:03:09.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:09.763+0000 7f8825000700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8820102100 0x7f8820075f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:09.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:09.763+0000 7f8825000700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8820103300 0x7f8820074580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:09.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:09.763+0000 7f8825000700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8820103ca0 0x7f8820074ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:09.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:09.763+0000 7f8825000700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f882010db80 con 0x7f8820103300 2024-03-28T16:03:09.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:09.763+0000 7f8825000700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f882010da00 con 0x7f8820103ca0 2024-03-28T16:03:09.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:09.763+0000 7f8825000700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f882010dd00 con 0x7f8820102100 2024-03-28T16:03:09.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:09.763+0000 7f8816a00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8820103300 0x7f8820074580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:09.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:09.763+0000 7f8816a00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8820103300 0x7f8820074580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57890/0 (socket says 172.21.15.67:57890) 2024-03-28T16:03:09.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:09.763+0000 7f8816a00700 1 -- 172.21.15.67:0/980676227 learned_addr learned my addr 172.21.15.67:0/980676227 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:03:09.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:09.763+0000 7f881f400700 1 --2- 172.21.15.67:0/980676227 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8820102100 0x7f8820075f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:09.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:09.763+0000 7f881fe00700 1 --2- 172.21.15.67:0/980676227 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8820103ca0 0x7f8820074ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:09.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:09.764+0000 7f8816a00700 1 -- 172.21.15.67:0/980676227 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8820102100 msgr2=0x7f8820075f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:09.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:09.764+0000 7f8816a00700 1 --2- 172.21.15.67:0/980676227 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8820102100 0x7f8820075f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:09.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:09.764+0000 7f8816a00700 1 -- 172.21.15.67:0/980676227 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8820103ca0 msgr2=0x7f8820074ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:09.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:09.764+0000 7f8816a00700 1 --2- 172.21.15.67:0/980676227 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8820103ca0 0x7f8820074ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:09.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:09.764+0000 7f8816a00700 1 -- 172.21.15.67:0/980676227 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8820075370 con 0x7f8820103300 2024-03-28T16:03:09.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:09.764+0000 7f881fe00700 1 --2- 172.21.15.67:0/980676227 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8820103ca0 0x7f8820074ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:03:09.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:09.764+0000 7f8816a00700 1 --2- 172.21.15.67:0/980676227 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8820103300 0x7f8820074580 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f880800d950 tx=0x7f880800dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:09.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:09.764+0000 7f881cc00700 1 -- 172.21.15.67:0/980676227 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8808016070 con 0x7f8820103300 2024-03-28T16:03:09.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:09.764+0000 7f881cc00700 1 -- 172.21.15.67:0/980676227 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8808010460 con 0x7f8820103300 2024-03-28T16:03:09.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:09.764+0000 7f8825000700 1 -- 172.21.15.67:0/980676227 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88201a23b0 con 0x7f8820103300 2024-03-28T16:03:09.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:09.764+0000 7f881cc00700 1 -- 172.21.15.67:0/980676227 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f880800f620 con 0x7f8820103300 2024-03-28T16:03:09.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:09.764+0000 7f8825000700 1 -- 172.21.15.67:0/980676227 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88201a28c0 con 0x7f8820103300 2024-03-28T16:03:09.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:09.766+0000 7f8825000700 1 -- 172.21.15.67:0/980676227 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8820068e50 con 0x7f8820103300 2024-03-28T16:03:09.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:09.768+0000 7f881cc00700 1 -- 172.21.15.67:0/980676227 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f8808019070 con 0x7f8820103300 2024-03-28T16:03:09.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:09.770+0000 7f881cc00700 1 --2- 172.21.15.67:0/980676227 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f88040a2890 0x7f88040a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:09.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:09.770+0000 7f881cc00700 1 -- 172.21.15.67:0/980676227 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f88080ce6c0 con 0x7f8820103300 2024-03-28T16:03:09.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:09.770+0000 7f881f400700 1 --2- 172.21.15.67:0/980676227 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f88040a2890 0x7f88040a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:09.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:09.771+0000 7f881f400700 1 --2- 172.21.15.67:0/980676227 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f88040a2890 0x7f88040a4d50 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f8810000c00 tx=0x7f8810005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:09.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:09.775+0000 7f881cc00700 1 -- 172.21.15.67:0/980676227 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f880809b770 con 0x7f8820103300 2024-03-28T16:03:10.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:10.000+0000 7f8825000700 1 -- 172.21.15.67:0/980676227 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump"} v 0) v1 -- 0x7f8820062da0 con 0x7f8820103300 2024-03-28T16:03:10.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:10.002+0000 7f881cc00700 1 -- 172.21.15.67:0/980676227 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v92) v1 ==== 56+0+5290 (secure 0 0 0) 0x7f880809b070 con 0x7f8820103300 2024-03-28T16:03:10.005 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 92 2024-03-28T16:03:10.005 INFO:teuthology.orchestra.run.smithi067.stdout:fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 2024-03-28T16:03:10.005 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-03-28T15:50:21.471924+0000 2024-03-28T16:03:10.005 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-03-28T16:01:40.643215+0000 2024-03-28T16:03:10.005 INFO:teuthology.orchestra.run.smithi067.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-28T16:03:10.005 INFO:teuthology.orchestra.run.smithi067.stdout:crush_version 26 2024-03-28T16:03:10.005 INFO:teuthology.orchestra.run.smithi067.stdout:full_ratio 0.95 2024-03-28T16:03:10.005 INFO:teuthology.orchestra.run.smithi067.stdout:backfillfull_ratio 0.9 2024-03-28T16:03:10.005 INFO:teuthology.orchestra.run.smithi067.stdout:nearfull_ratio 0.85 2024-03-28T16:03:10.005 INFO:teuthology.orchestra.run.smithi067.stdout:require_min_compat_client luminous 2024-03-28T16:03:10.005 INFO:teuthology.orchestra.run.smithi067.stdout:min_compat_client luminous 2024-03-28T16:03:10.006 INFO:teuthology.orchestra.run.smithi067.stdout:require_osd_release squid 2024-03-28T16:03:10.006 INFO:teuthology.orchestra.run.smithi067.stdout:stretch_mode_enabled false 2024-03-28T16:03:10.006 INFO:teuthology.orchestra.run.smithi067.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-03-28T16:03:10.006 INFO:teuthology.orchestra.run.smithi067.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 89 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-03-28T16:03:10.006 INFO:teuthology.orchestra.run.smithi067.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 90 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2024-03-28T16:03:10.006 INFO:teuthology.orchestra.run.smithi067.stdout:max_osd 12 2024-03-28T16:03:10.006 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 up in weight 1 up_from 8 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.67:6802/1894681701,v1:172.21.15.67:6803/1894681701] [v2:172.21.15.67:6804/1894681701,v1:172.21.15.67:6805/1894681701] exists,up 7c1791b6-9c0e-4512-a327-cad166bb5581 2024-03-28T16:03:10.006 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 up in weight 1 up_from 13 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.150:6800/1734465762,v1:172.21.15.150:6801/1734465762] [v2:172.21.15.150:6802/1734465762,v1:172.21.15.150:6803/1734465762] exists,up 0cdda7f3-d00c-4436-bcc1-49fb8072b2e3 2024-03-28T16:03:10.006 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 up in weight 1 up_from 18 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.183:6800/3398095931,v1:172.21.15.183:6801/3398095931] [v2:172.21.15.183:6802/3398095931,v1:172.21.15.183:6803/3398095931] exists,up a55b9a02-c3fd-4061-95c0-53a24a17ade0 2024-03-28T16:03:10.006 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 up in weight 1 up_from 27 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.67:6810/1933541875,v1:172.21.15.67:6811/1933541875] [v2:172.21.15.67:6812/1933541875,v1:172.21.15.67:6813/1933541875] exists,up d34940c0-5ce9-4b8b-aa36-4872414a043f 2024-03-28T16:03:10.006 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 up in weight 1 up_from 35 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.150:6808/3775269993,v1:172.21.15.150:6809/3775269993] [v2:172.21.15.150:6810/3775269993,v1:172.21.15.150:6811/3775269993] exists,up 5cc3e260-83a3-4a4d-aa6a-34a3b12b2b5f 2024-03-28T16:03:10.006 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 up in weight 1 up_from 41 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.183:6808/164284137,v1:172.21.15.183:6809/164284137] [v2:172.21.15.183:6810/164284137,v1:172.21.15.183:6811/164284137] exists,up a2542b21-d6ac-413e-8b9b-4a6548b20840 2024-03-28T16:03:10.006 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6 up in weight 1 up_from 49 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.67:6818/3565528013,v1:172.21.15.67:6819/3565528013] [v2:172.21.15.67:6820/3565528013,v1:172.21.15.67:6821/3565528013] exists,up 8905a2d3-0f1b-4853-a218-22da2635150a 2024-03-28T16:03:10.007 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7 up in weight 1 up_from 55 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.150:6816/3735115340,v1:172.21.15.150:6817/3735115340] [v2:172.21.15.150:6818/3735115340,v1:172.21.15.150:6819/3735115340] exists,up b275055b-b451-4180-91b1-e6e61a828c48 2024-03-28T16:03:10.007 INFO:teuthology.orchestra.run.smithi067.stdout:osd.8 up in weight 1 up_from 63 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.183:6816/1025400095,v1:172.21.15.183:6817/1025400095] [v2:172.21.15.183:6818/1025400095,v1:172.21.15.183:6819/1025400095] exists,up b6abf7ad-eb34-4f60-a4eb-348a92d7195b 2024-03-28T16:03:10.007 INFO:teuthology.orchestra.run.smithi067.stdout:osd.9 up in weight 1 up_from 71 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.67:6826/76885409,v1:172.21.15.67:6827/76885409] [v2:172.21.15.67:6828/76885409,v1:172.21.15.67:6829/76885409] exists,up 28a78e3a-0657-40aa-816f-16a6fadadcbb 2024-03-28T16:03:10.007 INFO:teuthology.orchestra.run.smithi067.stdout:osd.10 up in weight 1 up_from 77 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.150:6824/3489982080,v1:172.21.15.150:6825/3489982080] [v2:172.21.15.150:6826/3489982080,v1:172.21.15.150:6827/3489982080] exists,up d287907e-424f-4b49-a06e-4613c6de0474 2024-03-28T16:03:10.007 INFO:teuthology.orchestra.run.smithi067.stdout:osd.11 up in weight 1 up_from 85 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.183:6824/1892639849,v1:172.21.15.183:6825/1892639849] [v2:172.21.15.183:6826/1892639849,v1:172.21.15.183:6827/1892639849] exists,up a3c29c47-a228-4f42-a23c-fbdb0c093699 2024-03-28T16:03:10.007 INFO:teuthology.orchestra.run.smithi067.stdout:pg_upmap_items 2.f [8,11] 2024-03-28T16:03:10.007 INFO:teuthology.orchestra.run.smithi067.stdout:pg_upmap_items 2.10 [2,4,8,11] 2024-03-28T16:03:10.007 INFO:teuthology.orchestra.run.smithi067.stdout:pg_upmap_items 2.1d [2,1] 2024-03-28T16:03:10.007 INFO:teuthology.orchestra.run.smithi067.stdout:pg_upmap_items 3.7 [3,9] 2024-03-28T16:03:10.007 INFO:teuthology.orchestra.run.smithi067.stdout:pg_upmap_items 3.11 [4,6] 2024-03-28T16:03:10.007 INFO:teuthology.orchestra.run.smithi067.stdout:pg_upmap_items 3.1d [4,8] 2024-03-28T16:03:10.007 INFO:teuthology.orchestra.run.smithi067.stdout:pg_upmap_items 3.20 [5,9] 2024-03-28T16:03:10.007 INFO:teuthology.orchestra.run.smithi067.stdout:pg_upmap_items 3.36 [4,9] 2024-03-28T16:03:10.007 INFO:teuthology.orchestra.run.smithi067.stdout:blocklist 172.21.15.67:6801/1631001968 expires 2024-03-29T15:51:38.875334+0000 2024-03-28T16:03:10.007 INFO:teuthology.orchestra.run.smithi067.stdout:blocklist 172.21.15.67:6800/1631001968 expires 2024-03-29T15:51:38.875334+0000 2024-03-28T16:03:10.008 INFO:teuthology.orchestra.run.smithi067.stdout:blocklist 172.21.15.67:0/1533056310 expires 2024-03-29T15:51:38.875334+0000 2024-03-28T16:03:10.008 INFO:teuthology.orchestra.run.smithi067.stdout:blocklist 172.21.15.67:6801/3580491071 expires 2024-03-29T15:51:03.859740+0000 2024-03-28T16:03:10.008 INFO:teuthology.orchestra.run.smithi067.stdout:blocklist 172.21.15.67:0/2030412458 expires 2024-03-29T15:51:03.859740+0000 2024-03-28T16:03:10.008 INFO:teuthology.orchestra.run.smithi067.stdout:blocklist 172.21.15.67:6800/3580491071 expires 2024-03-29T15:51:03.859740+0000 2024-03-28T16:03:10.008 INFO:teuthology.orchestra.run.smithi067.stdout:blocklist 172.21.15.67:0/2905307533 expires 2024-03-29T15:51:03.859740+0000 2024-03-28T16:03:10.008 INFO:teuthology.orchestra.run.smithi067.stdout:blocklist 172.21.15.67:0/3763759988 expires 2024-03-29T15:51:38.875334+0000 2024-03-28T16:03:10.008 INFO:teuthology.orchestra.run.smithi067.stdout:blocklist 172.21.15.67:0/4089399880 expires 2024-03-29T15:51:03.859740+0000 2024-03-28T16:03:10.008 INFO:teuthology.orchestra.run.smithi067.stdout:blocklist 172.21.15.67:0/4239108661 expires 2024-03-29T15:51:38.875334+0000 2024-03-28T16:03:10.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:10.006+0000 7f8825000700 1 -- 172.21.15.67:0/980676227 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f88040a2890 msgr2=0x7f88040a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:10.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:10.006+0000 7f8825000700 1 --2- 172.21.15.67:0/980676227 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f88040a2890 0x7f88040a4d50 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f8810000c00 tx=0x7f8810005fd0 comp rx=0 tx=0).stop 2024-03-28T16:03:10.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:10.006+0000 7f8825000700 1 -- 172.21.15.67:0/980676227 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8820103300 msgr2=0x7f8820074580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:10.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:10.006+0000 7f8825000700 1 --2- 172.21.15.67:0/980676227 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8820103300 0x7f8820074580 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f880800d950 tx=0x7f880800dd10 comp rx=0 tx=0).stop 2024-03-28T16:03:10.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:10.007+0000 7f8825000700 1 -- 172.21.15.67:0/980676227 shutdown_connections 2024-03-28T16:03:10.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:10.007+0000 7f8825000700 1 --2- 172.21.15.67:0/980676227 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f88040a2890 0x7f88040a4d50 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:10.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:10.007+0000 7f8825000700 1 --2- 172.21.15.67:0/980676227 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8820103ca0 0x7f8820074ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:10.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:10.007+0000 7f8825000700 1 --2- 172.21.15.67:0/980676227 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8820102100 0x7f8820075f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:10.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:10.007+0000 7f8825000700 1 --2- 172.21.15.67:0/980676227 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8820103300 0x7f8820074580 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:10.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:10.007+0000 7f8825000700 1 -- 172.21.15.67:0/980676227 >> 172.21.15.67:0/980676227 conn(0x7f88200fd6a0 msgr2=0x7f88201066a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:10.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:10.007+0000 7f8825000700 1 -- 172.21.15.67:0/980676227 shutdown_connections 2024-03-28T16:03:10.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:10.007+0000 7f8825000700 1 -- 172.21.15.67:0/980676227 wait complete. 2024-03-28T16:03:10.747 INFO:teuthology.run_tasks:Running task fs.ready... 2024-03-28T16:03:10.761 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T16:03:11.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:10.999+0000 7f065e000640 1 -- 172.21.15.67:0/1903239164 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0658105490 msgr2=0x7f065810bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:11.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:10.999+0000 7f065e000640 1 --2- 172.21.15.67:0/1903239164 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0658105490 0x7f065810bd20 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f0650009ca0 tx=0x7f065002c100 comp rx=0 tx=0).stop 2024-03-28T16:03:11.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:10.999+0000 7f065e000640 1 -- 172.21.15.67:0/1903239164 shutdown_connections 2024-03-28T16:03:11.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:10.999+0000 7f065e000640 1 --2- 172.21.15.67:0/1903239164 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0658105490 0x7f065810bd20 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:11.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:10.999+0000 7f065e000640 1 --2- 172.21.15.67:0/1903239164 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0658104ad0 0x7f0658104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:11.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:10.999+0000 7f065e000640 1 --2- 172.21.15.67:0/1903239164 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06581038e0 0x7f0658103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:11.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:10.999+0000 7f065e000640 1 -- 172.21.15.67:0/1903239164 >> 172.21.15.67:0/1903239164 conn(0x7f06580fee90 msgr2=0x7f06581012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:11.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:10.999+0000 7f065e000640 1 -- 172.21.15.67:0/1903239164 shutdown_connections 2024-03-28T16:03:11.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:10.999+0000 7f065e000640 1 -- 172.21.15.67:0/1903239164 wait complete. 2024-03-28T16:03:11.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.000+0000 7f065e000640 1 Processor -- start 2024-03-28T16:03:11.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.001+0000 7f065e000640 1 -- start start 2024-03-28T16:03:11.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.001+0000 7f065e000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f06581038e0 0x7f0658199a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:11.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.001+0000 7f065e000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0658104ad0 0x7f0658199f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:11.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.001+0000 7f065e000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0658105490 0x7f06581a0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:11.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.001+0000 7f065e000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f065810f3f0 con 0x7f0658105490 2024-03-28T16:03:11.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.001+0000 7f065e000640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f065810f270 con 0x7f0658104ad0 2024-03-28T16:03:11.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.001+0000 7f065e000640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f065810f570 con 0x7f06581038e0 2024-03-28T16:03:11.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.001+0000 7f0657e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0658105490 0x7f06581a0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:11.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.001+0000 7f0657e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0658105490 0x7f06581a0fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57910/0 (socket says 172.21.15.67:57910) 2024-03-28T16:03:11.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.001+0000 7f0657e00640 1 -- 172.21.15.67:0/846851268 learned_addr learned my addr 172.21.15.67:0/846851268 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:03:11.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.002+0000 7f0656a00640 1 --2- 172.21.15.67:0/846851268 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0658104ad0 0x7f0658199f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:11.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.002+0000 7f0657400640 1 --2- 172.21.15.67:0/846851268 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f06581038e0 0x7f0658199a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:11.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.002+0000 7f0657e00640 1 -- 172.21.15.67:0/846851268 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f06581038e0 msgr2=0x7f0658199a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:11.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.002+0000 7f0657e00640 1 --2- 172.21.15.67:0/846851268 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f06581038e0 0x7f0658199a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:11.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.002+0000 7f0657e00640 1 -- 172.21.15.67:0/846851268 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0658104ad0 msgr2=0x7f0658199f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:11.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.002+0000 7f0657e00640 1 --2- 172.21.15.67:0/846851268 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0658104ad0 0x7f0658199f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:11.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.002+0000 7f0657e00640 1 -- 172.21.15.67:0/846851268 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06581a1620 con 0x7f0658105490 2024-03-28T16:03:11.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.002+0000 7f0657400640 1 --2- 172.21.15.67:0/846851268 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f06581038e0 0x7f0658199a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:03:11.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.002+0000 7f0657e00640 1 --2- 172.21.15.67:0/846851268 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0658105490 0x7f06581a0fb0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f0650032380 tx=0x7f0650035e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:11.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.002+0000 7f064fe00640 1 -- 172.21.15.67:0/846851268 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0650039040 con 0x7f0658105490 2024-03-28T16:03:11.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.002+0000 7f064fe00640 1 -- 172.21.15.67:0/846851268 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0650044070 con 0x7f0658105490 2024-03-28T16:03:11.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.002+0000 7f065e000640 1 -- 172.21.15.67:0/846851268 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06581a18b0 con 0x7f0658105490 2024-03-28T16:03:11.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.002+0000 7f064fe00640 1 -- 172.21.15.67:0/846851268 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0650004670 con 0x7f0658105490 2024-03-28T16:03:11.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.003+0000 7f065e000640 1 -- 172.21.15.67:0/846851268 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06581a1da0 con 0x7f0658105490 2024-03-28T16:03:11.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.004+0000 7f065e000640 1 -- 172.21.15.67:0/846851268 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f065810ac00 con 0x7f0658105490 2024-03-28T16:03:11.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.006+0000 7f064fe00640 1 -- 172.21.15.67:0/846851268 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f0650003970 con 0x7f0658105490 2024-03-28T16:03:11.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.007+0000 7f064fe00640 1 --2- 172.21.15.67:0/846851268 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f063c0a2800 0x7f063c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:11.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.007+0000 7f064fe00640 1 -- 172.21.15.67:0/846851268 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f06500f14c0 con 0x7f0658105490 2024-03-28T16:03:11.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.007+0000 7f0657400640 1 --2- 172.21.15.67:0/846851268 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f063c0a2800 0x7f063c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:11.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.008+0000 7f0657400640 1 --2- 172.21.15.67:0/846851268 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f063c0a2800 0x7f063c0a4cc0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f0658104930 tx=0x7f0648005db0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:11.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.011+0000 7f064fe00640 1 -- 172.21.15.67:0/846851268 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f06500be510 con 0x7f0658105490 2024-03-28T16:03:11.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:10 smithi150 ceph-mon[34825]: pgmap v427: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:03:11.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:10 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/980676227' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-28T16:03:11.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:10 smithi067 ceph-mon[31707]: pgmap v427: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:03:11.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:10 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/980676227' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-28T16:03:11.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:10 smithi183 ceph-mon[35580]: pgmap v427: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:03:11.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:10 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/980676227' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-28T16:03:11.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.218+0000 7f065e000640 1 -- 172.21.15.67:0/846851268 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f06580622a0 con 0x7f0658105490 2024-03-28T16:03:11.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.221+0000 7f064fe00640 1 -- 172.21.15.67:0/846851268 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) v1 ==== 74+0+22673 (secure 0 0 0) 0x7f06500098b0 con 0x7f0658105490 2024-03-28T16:03:11.223 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:03:11.223 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":92,"fsid":"b5f8225a-ed1a-11ee-95d0-87774f69a715","created":"2024-03-28T15:50:21.471924+0000","modified":"2024-03-28T16:01:40.643215+0000","last_up_change":"2024-03-28T16:00:01.369123+0000","last_in_change":"2024-03-28T15:59:37.515483+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T15:54:55.588007+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T16:01:05.719801+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T16:01:07.206303+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7c1791b6-9c0e-4512-a327-cad166bb5581","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6803","nonce":1894681701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6805","nonce":1894681701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6809","nonce":1894681701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6807","nonce":1894681701}]},"public_addr":"172.21.15.67:6803/1894681701","cluster_addr":"172.21.15.67:6805/1894681701","heartbeat_back_addr":"172.21.15.67:6809/1894681701","heartbeat_front_addr":"172.21.15.67:6807/1894681701","state":["exists","up"]},{"osd":1,"uuid":"0cdda7f3-d00c-4436-bcc1-49fb8072b2e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6801","nonce":1734465762}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6803","nonce":1734465762}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6807","nonce":1734465762}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6805","nonce":1734465762}]},"public_addr":"172.21.15.150:6801/1734465762","cluster_addr":"172.21.15.150:6803/1734465762","heartbeat_back_addr":"172.21.15.150:6807/1734465762","heartbeat_front_addr":"172.21.15.150:6805/1734465762","state":["exists","up"]},{"osd":2,"uuid":"a55b9a02-c3fd-4061-95c0-53a24a17ade0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6801","nonce":3398095931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6803","nonce":3398095931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6807","nonce":3398095931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6805","nonce":3398095931}]},"public_addr":"172.21.15.183:6801/3398095931","cluster_addr":"172.21.15.183:6803/3398095931","heartbeat_back_addr":"172.21.15.183:6807/3398095931","heartbeat_front_addr":"172.21.15.183:6805/3398095931","state":["exists","up"]},{"osd":3,"uuid":"d34940c0-5ce9-4b8b-aa36-4872414a043f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6811","nonce":1933541875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6813","nonce":1933541875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6817","nonce":1933541875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6815","nonce":1933541875}]},"public_addr":"172.21.15.67:6811/1933541875","cluster_addr":"172.21.15.67:6813/1933541875","heartbeat_back_addr":"172.21.15.67:6817/1933541875","heartbeat_front_addr":"172.21.15.67:6815/1933541875","state":["exists","up"]},{"osd":4,"uuid":"5cc3e260-83a3-4a4d-aa6a-34a3b12b2b5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6809","nonce":3775269993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6811","nonce":3775269993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6815","nonce":3775269993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6813","nonce":3775269993}]},"public_addr":"172.21.15.150:6809/3775269993","cluster_addr":"172.21.15.150:6811/3775269993","heartbeat_back_addr":"172.21.15.150:6815/3775269993","heartbeat_front_addr":"172.21.15.150:6813/3775269993","state":["exists","up"]},{"osd":5,"uuid":"a2542b21-d6ac-413e-8b9b-4a6548b20840","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6809","nonce":164284137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6811","nonce":164284137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6815","nonce":164284137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6813","nonce":164284137}]},"public_addr":"172.21.15.183:6809/164284137","cluster_addr":"172.21.15.183:6811/164284137","heartbeat_back_addr":"172.21.15.183:6815/164284137","heartbeat_front_addr":"172.21.15.183:6813/164284137","state":["exists","up"]},{"osd":6,"uuid":"8905a2d3-0f1b-4853-a218-22da2635150a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6819","nonce":3565528013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6821","nonce":3565528013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6825","nonce":3565528013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6823","nonce":3565528013}]},"public_addr":"172.21.15.67:6819/3565528013","cluster_addr":"172.21.15.67:6821/3565528013","heartbeat_back_addr":"172.21.15.67:6825/3565528013","heartbeat_front_addr":"172.21.15.67:6823/3565528013","state":["exists","up"]},{"osd":7,"uuid":"b275055b-b451-4180-91b1-e6e61a828c48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6817","nonce":3735115340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6819","nonce":3735115340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6823","nonce":3735115340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6821","nonce":3735115340}]},"public_addr":"172.21.15.150:6817/3735115340","cluster_addr":"172.21.15.150:6819/3735115340","heartbeat_back_addr":"172.21.15.150:6823/3735115340","heartbeat_front_addr":"172.21.15.150:6821/3735115340","state":["exists","up"]},{"osd":8,"uuid":"b6abf7ad-eb34-4f60-a4eb-348a92d7195b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6817","nonce":1025400095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6819","nonce":1025400095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6823","nonce":1025400095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6821","nonce":1025400095}]},"public_addr":"172.21.15.183:6817/1025400095","cluster_addr":"172.21.15.183:6819/1025400095","heartbeat_back_addr":"172.21.15.183:6823/1025400095","heartbeat_front_addr":"172.21.15.183:6821/1025400095","state":["exists","up"]},{"osd":9,"uuid":"28a78e3a-0657-40aa-816f-16a6fadadcbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6827","nonce":76885409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6829","nonce":76885409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6833","nonce":76885409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6831","nonce":76885409}]},"public_addr":"172.21.15.67:6827/76885409","cluster_addr":"172.21.15.67:6829/76885409","heartbeat_back_addr":"172.21.15.67:6833/76885409","heartbeat_front_addr":"172.21.15.67:6831/76885409","state":["exists","up"]},{"osd":10,"uuid":"d287907e-424f-4b49-a06e-4613c6de0474","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6825","nonce":3489982080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6827","nonce":3489982080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6831","nonce":3489982080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6829","nonce":3489982080}]},"public_addr":"172.21.15.150:6825/3489982080","cluster_addr":"172.21.15.150:6827/3489982080","heartbeat_back_addr":"172.21.15.150:6831/3489982080","heartbeat_front_addr":"172.21.15.150:6829/3489982080","state":["exists","up"]},{"osd":11,"uuid":"a3c29c47-a228-4f42-a23c-fbdb0c093699","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":85,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6825","nonce":1892639849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6826","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6827","nonce":1892639849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6831","nonce":1892639849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6829","nonce":1892639849}]},"public_addr":"172.21.15.183:6825/1892639849","cluster_addr":"172.21.15.183:6827/1892639849","heartbeat_back_addr":"172.21.15.183:6831/1892639849","heartbeat_front_addr":"172.21.15.183:6829/1892639849","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:53:45.646211+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:54:18.956590+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:54:52.161646+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:55:27.502822+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:56:01.162932+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:56:34.674452+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:57:09.298679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:57:43.230406+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:58:17.003821+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:58:51.085863+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:59:25.665065+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T16:00:00.263457+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":1}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6801/1631001968":"2024-03-29T15:51:38.875334+0000","172.21.15.67:6800/1631001968":"2024-03-29T15:51:38.875334+0000","172.21.15.67:0/1533056310":"2024-03-29T15:51:38.875334+0000","172.21.15.67:6801/3580491071":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/2030412458":"2024-03-29T15:51:03.859740+0000","172.21.15.67:6800/3580491071":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/2905307533":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/3763759988":"2024-03-29T15:51:38.875334+0000","172.21.15.67:0/4089399880":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/4239108661":"2024-03-29T15:51:38.875334+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T16:03:11.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.224+0000 7f065e000640 1 -- 172.21.15.67:0/846851268 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f063c0a2800 msgr2=0x7f063c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:11.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.224+0000 7f065e000640 1 --2- 172.21.15.67:0/846851268 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f063c0a2800 0x7f063c0a4cc0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f0658104930 tx=0x7f0648005db0 comp rx=0 tx=0).stop 2024-03-28T16:03:11.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.224+0000 7f065e000640 1 -- 172.21.15.67:0/846851268 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0658105490 msgr2=0x7f06581a0fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:11.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.224+0000 7f065e000640 1 --2- 172.21.15.67:0/846851268 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0658105490 0x7f06581a0fb0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f0650032380 tx=0x7f0650035e20 comp rx=0 tx=0).stop 2024-03-28T16:03:11.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.224+0000 7f065e000640 1 -- 172.21.15.67:0/846851268 shutdown_connections 2024-03-28T16:03:11.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.224+0000 7f065e000640 1 --2- 172.21.15.67:0/846851268 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f063c0a2800 0x7f063c0a4cc0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:11.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.224+0000 7f065e000640 1 --2- 172.21.15.67:0/846851268 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0658105490 0x7f06581a0fb0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:11.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.224+0000 7f065e000640 1 --2- 172.21.15.67:0/846851268 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0658104ad0 0x7f0658199f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:11.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.224+0000 7f065e000640 1 --2- 172.21.15.67:0/846851268 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f06581038e0 0x7f0658199a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:11.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.224+0000 7f065e000640 1 -- 172.21.15.67:0/846851268 >> 172.21.15.67:0/846851268 conn(0x7f06580fee90 msgr2=0x7f0658107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:11.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.224+0000 7f065e000640 1 -- 172.21.15.67:0/846851268 shutdown_connections 2024-03-28T16:03:11.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.224+0000 7f065e000640 1 -- 172.21.15.67:0/846851268 wait complete. 2024-03-28T16:03:11.245 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T15:54:55.588007+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T16:01:05.719801+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T16:01:07.206303+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T16:03:11.246 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T16:03:11.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.468+0000 7fcf5fe00640 1 -- 172.21.15.67:0/1879789406 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf601049b0 msgr2=0x7fcf60104e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:11.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.468+0000 7fcf5fe00640 1 --2- 172.21.15.67:0/1879789406 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf601049b0 0x7fcf60104e30 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fcf48009900 tx=0x7fcf4802bc30 comp rx=0 tx=0).stop 2024-03-28T16:03:11.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.468+0000 7fcf5fe00640 1 -- 172.21.15.67:0/1879789406 shutdown_connections 2024-03-28T16:03:11.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.468+0000 7fcf5fe00640 1 --2- 172.21.15.67:0/1879789406 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcf60105370 0x7fcf6010bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:11.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.468+0000 7fcf5fe00640 1 --2- 172.21.15.67:0/1879789406 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf601049b0 0x7fcf60104e30 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:11.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.468+0000 7fcf5fe00640 1 --2- 172.21.15.67:0/1879789406 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf601037c0 0x7fcf60103be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:11.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.468+0000 7fcf5fe00640 1 -- 172.21.15.67:0/1879789406 >> 172.21.15.67:0/1879789406 conn(0x7fcf600fed50 msgr2=0x7fcf60101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:11.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.468+0000 7fcf5fe00640 1 -- 172.21.15.67:0/1879789406 shutdown_connections 2024-03-28T16:03:11.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.468+0000 7fcf5fe00640 1 -- 172.21.15.67:0/1879789406 wait complete. 2024-03-28T16:03:11.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.469+0000 7fcf5fe00640 1 Processor -- start 2024-03-28T16:03:11.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.469+0000 7fcf5fe00640 1 -- start start 2024-03-28T16:03:11.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.469+0000 7fcf5fe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcf601037c0 0x7fcf60199a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:11.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.469+0000 7fcf5fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf601049b0 0x7fcf60199f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:11.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.470+0000 7fcf5fe00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf60105370 0x7fcf601a0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:11.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.470+0000 7fcf5fe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf6010f320 con 0x7fcf601049b0 2024-03-28T16:03:11.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.470+0000 7fcf5fe00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf6010f1a0 con 0x7fcf60105370 2024-03-28T16:03:11.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.470+0000 7fcf5fe00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf6010f4a0 con 0x7fcf601037c0 2024-03-28T16:03:11.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.470+0000 7fcf56000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf601049b0 0x7fcf60199f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:11.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.470+0000 7fcf56000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf601049b0 0x7fcf60199f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57920/0 (socket says 172.21.15.67:57920) 2024-03-28T16:03:11.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.470+0000 7fcf56000640 1 -- 172.21.15.67:0/2053804879 learned_addr learned my addr 172.21.15.67:0/2053804879 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:03:11.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.470+0000 7fcf5ea00640 1 --2- 172.21.15.67:0/2053804879 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcf601037c0 0x7fcf60199a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:11.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.470+0000 7fcf5f400640 1 --2- 172.21.15.67:0/2053804879 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf60105370 0x7fcf601a0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:11.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.470+0000 7fcf56000640 1 -- 172.21.15.67:0/2053804879 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcf601037c0 msgr2=0x7fcf60199a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:11.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.470+0000 7fcf56000640 1 --2- 172.21.15.67:0/2053804879 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcf601037c0 0x7fcf60199a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:11.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.470+0000 7fcf56000640 1 -- 172.21.15.67:0/2053804879 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf60105370 msgr2=0x7fcf601a0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:11.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.470+0000 7fcf56000640 1 --2- 172.21.15.67:0/2053804879 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf60105370 0x7fcf601a0fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:11.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.470+0000 7fcf56000640 1 -- 172.21.15.67:0/2053804879 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf601a1620 con 0x7fcf601049b0 2024-03-28T16:03:11.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.470+0000 7fcf5f400640 1 --2- 172.21.15.67:0/2053804879 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf60105370 0x7fcf601a0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:03:11.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.471+0000 7fcf56000640 1 --2- 172.21.15.67:0/2053804879 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf601049b0 0x7fcf60199f40 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fcf48000c00 tx=0x7fcf480042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:11.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.471+0000 7fcf57e00640 1 -- 172.21.15.67:0/2053804879 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf48043020 con 0x7fcf601049b0 2024-03-28T16:03:11.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.471+0000 7fcf57e00640 1 -- 172.21.15.67:0/2053804879 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fcf480054c0 con 0x7fcf601049b0 2024-03-28T16:03:11.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.471+0000 7fcf57e00640 1 -- 172.21.15.67:0/2053804879 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf48003d40 con 0x7fcf601049b0 2024-03-28T16:03:11.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.471+0000 7fcf5fe00640 1 -- 172.21.15.67:0/2053804879 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf601a18b0 con 0x7fcf601049b0 2024-03-28T16:03:11.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.471+0000 7fcf5fe00640 1 -- 172.21.15.67:0/2053804879 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf601a1db0 con 0x7fcf601049b0 2024-03-28T16:03:11.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.472+0000 7fcf5fe00640 1 -- 172.21.15.67:0/2053804879 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf6010ab60 con 0x7fcf601049b0 2024-03-28T16:03:11.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.474+0000 7fcf57e00640 1 -- 172.21.15.67:0/2053804879 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fcf48046020 con 0x7fcf601049b0 2024-03-28T16:03:11.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.476+0000 7fcf57e00640 1 --2- 172.21.15.67:0/2053804879 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fcf440a2880 0x7fcf440a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:11.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.476+0000 7fcf57e00640 1 -- 172.21.15.67:0/2053804879 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fcf480f1080 con 0x7fcf601049b0 2024-03-28T16:03:11.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.476+0000 7fcf5ea00640 1 --2- 172.21.15.67:0/2053804879 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fcf440a2880 0x7fcf440a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:11.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.477+0000 7fcf5ea00640 1 --2- 172.21.15.67:0/2053804879 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fcf440a2880 0x7fcf440a4d40 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fcf60104810 tx=0x7fcf50009870 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:11.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.480+0000 7fcf57e00640 1 -- 172.21.15.67:0/2053804879 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fcf4804d020 con 0x7fcf601049b0 2024-03-28T16:03:11.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.684+0000 7fcf5fe00640 1 -- 172.21.15.67:0/2053804879 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcf600622a0 con 0x7fcf601049b0 2024-03-28T16:03:11.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.685+0000 7fcf57e00640 1 -- 172.21.15.67:0/2053804879 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v92) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcf480be250 con 0x7fcf601049b0 2024-03-28T16:03:11.687 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-03-28T16:03:11.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.688+0000 7fcf5fe00640 1 -- 172.21.15.67:0/2053804879 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fcf440a2880 msgr2=0x7fcf440a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:11.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.688+0000 7fcf5fe00640 1 --2- 172.21.15.67:0/2053804879 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fcf440a2880 0x7fcf440a4d40 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fcf60104810 tx=0x7fcf50009870 comp rx=0 tx=0).stop 2024-03-28T16:03:11.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.688+0000 7fcf5fe00640 1 -- 172.21.15.67:0/2053804879 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf601049b0 msgr2=0x7fcf60199f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:11.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.688+0000 7fcf5fe00640 1 --2- 172.21.15.67:0/2053804879 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf601049b0 0x7fcf60199f40 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fcf48000c00 tx=0x7fcf480042f0 comp rx=0 tx=0).stop 2024-03-28T16:03:11.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.688+0000 7fcf5fe00640 1 -- 172.21.15.67:0/2053804879 shutdown_connections 2024-03-28T16:03:11.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.688+0000 7fcf5fe00640 1 --2- 172.21.15.67:0/2053804879 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fcf440a2880 0x7fcf440a4d40 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:11.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.688+0000 7fcf5fe00640 1 --2- 172.21.15.67:0/2053804879 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf60105370 0x7fcf601a0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:11.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.688+0000 7fcf5fe00640 1 --2- 172.21.15.67:0/2053804879 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf601049b0 0x7fcf60199f40 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:11.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.688+0000 7fcf5fe00640 1 --2- 172.21.15.67:0/2053804879 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcf601037c0 0x7fcf60199a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:11.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.688+0000 7fcf5fe00640 1 -- 172.21.15.67:0/2053804879 >> 172.21.15.67:0/2053804879 conn(0x7fcf600fed50 msgr2=0x7fcf60107d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:11.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.688+0000 7fcf5fe00640 1 -- 172.21.15.67:0/2053804879 shutdown_connections 2024-03-28T16:03:11.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.688+0000 7fcf5fe00640 1 -- 172.21.15.67:0/2053804879 wait complete. 2024-03-28T16:03:11.710 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T16:03:11.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.938+0000 7f2108600640 1 -- 172.21.15.67:0/1234757759 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21001038e0 msgr2=0x7f2100103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:11.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.938+0000 7f2108600640 1 --2- 172.21.15.67:0/1234757759 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21001038e0 0x7f2100103d00 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f20f0009900 tx=0x7f20f002bc30 comp rx=0 tx=0).stop 2024-03-28T16:03:11.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.939+0000 7f2108600640 1 -- 172.21.15.67:0/1234757759 shutdown_connections 2024-03-28T16:03:11.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.939+0000 7f2108600640 1 --2- 172.21.15.67:0/1234757759 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2100105490 0x7f210010bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:11.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.939+0000 7f2108600640 1 --2- 172.21.15.67:0/1234757759 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2100104ad0 0x7f2100104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:11.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.939+0000 7f2108600640 1 --2- 172.21.15.67:0/1234757759 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21001038e0 0x7f2100103d00 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:11.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.939+0000 7f2108600640 1 -- 172.21.15.67:0/1234757759 >> 172.21.15.67:0/1234757759 conn(0x7f21000fee50 msgr2=0x7f2100101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:11.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.939+0000 7f2108600640 1 -- 172.21.15.67:0/1234757759 shutdown_connections 2024-03-28T16:03:11.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.939+0000 7f2108600640 1 -- 172.21.15.67:0/1234757759 wait complete. 2024-03-28T16:03:11.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.940+0000 7f2108600640 1 Processor -- start 2024-03-28T16:03:11.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.940+0000 7f2108600640 1 -- start start 2024-03-28T16:03:11.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.940+0000 7f2108600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f21001038e0 0x7f2100199b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:11.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.940+0000 7f2108600640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2100104ad0 0x7f210019a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:11.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.940+0000 7f2108600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2100105490 0x7f21001a10f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:11.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.940+0000 7f2108600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f210010f450 con 0x7f2100105490 2024-03-28T16:03:11.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.940+0000 7f2108600640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f210010f2d0 con 0x7f2100104ad0 2024-03-28T16:03:11.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.940+0000 7f2108600640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f210010f5d0 con 0x7f21001038e0 2024-03-28T16:03:11.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.940+0000 7f2106600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2100105490 0x7f21001a10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:11.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.940+0000 7f2106600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2100105490 0x7f21001a10f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57938/0 (socket says 172.21.15.67:57938) 2024-03-28T16:03:11.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.940+0000 7f2106600640 1 -- 172.21.15.67:0/2169717936 learned_addr learned my addr 172.21.15.67:0/2169717936 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:03:11.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.941+0000 7f2105200640 1 --2- 172.21.15.67:0/2169717936 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2100104ad0 0x7f210019a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:11.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.941+0000 7f2105c00640 1 --2- 172.21.15.67:0/2169717936 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f21001038e0 0x7f2100199b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:11.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.941+0000 7f2106600640 1 -- 172.21.15.67:0/2169717936 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f21001038e0 msgr2=0x7f2100199b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:11.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.941+0000 7f2106600640 1 --2- 172.21.15.67:0/2169717936 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f21001038e0 0x7f2100199b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:11.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.941+0000 7f2106600640 1 -- 172.21.15.67:0/2169717936 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2100104ad0 msgr2=0x7f210019a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:11.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.941+0000 7f2106600640 1 --2- 172.21.15.67:0/2169717936 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2100104ad0 0x7f210019a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:11.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.941+0000 7f2106600640 1 -- 172.21.15.67:0/2169717936 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21001a1760 con 0x7f2100105490 2024-03-28T16:03:11.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.941+0000 7f2105c00640 1 --2- 172.21.15.67:0/2169717936 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f21001038e0 0x7f2100199b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:03:11.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.941+0000 7f2105200640 1 --2- 172.21.15.67:0/2169717936 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2100104ad0 0x7f210019a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:03:11.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.941+0000 7f2106600640 1 --2- 172.21.15.67:0/2169717936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2100105490 0x7f21001a10f0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f20f800eaa0 tx=0x7f20f800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:11.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.941+0000 7f20fe000640 1 -- 172.21.15.67:0/2169717936 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20f800cdb0 con 0x7f2100105490 2024-03-28T16:03:11.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.941+0000 7f20fe000640 1 -- 172.21.15.67:0/2169717936 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f20f8005020 con 0x7f2100105490 2024-03-28T16:03:11.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.941+0000 7f20fe000640 1 -- 172.21.15.67:0/2169717936 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20f80116d0 con 0x7f2100105490 2024-03-28T16:03:11.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.941+0000 7f2108600640 1 -- 172.21.15.67:0/2169717936 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21001a1a50 con 0x7f2100105490 2024-03-28T16:03:11.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.942+0000 7f2108600640 1 -- 172.21.15.67:0/2169717936 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21001a1fa0 con 0x7f2100105490 2024-03-28T16:03:11.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.943+0000 7f2108600640 1 -- 172.21.15.67:0/2169717936 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f210010ad20 con 0x7f2100105490 2024-03-28T16:03:11.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.945+0000 7f20fe000640 1 -- 172.21.15.67:0/2169717936 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f20f80051d0 con 0x7f2100105490 2024-03-28T16:03:11.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.946+0000 7f20fe000640 1 --2- 172.21.15.67:0/2169717936 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f20e40a2880 0x7f20e40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:11.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.946+0000 7f20fe000640 1 -- 172.21.15.67:0/2169717936 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f20f80ceca0 con 0x7f2100105490 2024-03-28T16:03:11.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.946+0000 7f2105c00640 1 --2- 172.21.15.67:0/2169717936 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f20e40a2880 0x7f20e40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:11.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.947+0000 7f2105c00640 1 --2- 172.21.15.67:0/2169717936 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f20e40a2880 0x7f20e40a4d40 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f20f0036040 tx=0x7f20f0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:11.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:11.951+0000 7f20fe000640 1 -- 172.21.15.67:0/2169717936 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f20f8016030 con 0x7f2100105490 2024-03-28T16:03:12.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:11 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/846851268' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:03:12.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:11 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/2053804879' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T16:03:12.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:11 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/846851268' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:03:12.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:11 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/2053804879' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T16:03:12.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:11 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/846851268' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:03:12.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:11 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/2053804879' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T16:03:12.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.156+0000 7f2108600640 1 -- 172.21.15.67:0/2169717936 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f21000622a0 con 0x7f2100105490 2024-03-28T16:03:12.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.157+0000 7f20fe000640 1 -- 172.21.15.67:0/2169717936 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v92) v1 ==== 104+0+11 (secure 0 0 0) 0x7f20f809bd50 con 0x7f2100105490 2024-03-28T16:03:12.159 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-03-28T16:03:12.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.160+0000 7f2108600640 1 -- 172.21.15.67:0/2169717936 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f20e40a2880 msgr2=0x7f20e40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:12.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.160+0000 7f2108600640 1 --2- 172.21.15.67:0/2169717936 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f20e40a2880 0x7f20e40a4d40 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f20f0036040 tx=0x7f20f0005fd0 comp rx=0 tx=0).stop 2024-03-28T16:03:12.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.160+0000 7f2108600640 1 -- 172.21.15.67:0/2169717936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2100105490 msgr2=0x7f21001a10f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:12.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.160+0000 7f2108600640 1 --2- 172.21.15.67:0/2169717936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2100105490 0x7f21001a10f0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f20f800eaa0 tx=0x7f20f800ef60 comp rx=0 tx=0).stop 2024-03-28T16:03:12.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.160+0000 7f2108600640 1 -- 172.21.15.67:0/2169717936 shutdown_connections 2024-03-28T16:03:12.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.160+0000 7f2108600640 1 --2- 172.21.15.67:0/2169717936 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f20e40a2880 0x7f20e40a4d40 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:12.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.160+0000 7f2108600640 1 --2- 172.21.15.67:0/2169717936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2100105490 0x7f21001a10f0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:12.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.160+0000 7f2108600640 1 --2- 172.21.15.67:0/2169717936 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2100104ad0 0x7f210019a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:12.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.160+0000 7f2108600640 1 --2- 172.21.15.67:0/2169717936 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f21001038e0 0x7f2100199b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:12.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.160+0000 7f2108600640 1 -- 172.21.15.67:0/2169717936 >> 172.21.15.67:0/2169717936 conn(0x7f21000fee50 msgr2=0x7f2100107ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:12.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.161+0000 7f2108600640 1 -- 172.21.15.67:0/2169717936 shutdown_connections 2024-03-28T16:03:12.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.161+0000 7f2108600640 1 -- 172.21.15.67:0/2169717936 wait complete. 2024-03-28T16:03:12.181 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T16:03:12.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.403+0000 7f5b01a00640 1 -- 172.21.15.67:0/530895573 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5afc103150 msgr2=0x7f5afc1099e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:12.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.403+0000 7f5b01a00640 1 --2- 172.21.15.67:0/530895573 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5afc103150 0x7f5afc1099e0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f5af4009900 tx=0x7f5af402bc40 comp rx=0 tx=0).stop 2024-03-28T16:03:12.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.403+0000 7f5b01a00640 1 -- 172.21.15.67:0/530895573 shutdown_connections 2024-03-28T16:03:12.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.403+0000 7f5b01a00640 1 --2- 172.21.15.67:0/530895573 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5afc103150 0x7f5afc1099e0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:12.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.403+0000 7f5b01a00640 1 --2- 172.21.15.67:0/530895573 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5afc102790 0x7f5afc102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:12.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.403+0000 7f5b01a00640 1 --2- 172.21.15.67:0/530895573 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5afc1015a0 0x7f5afc1019c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:12.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.403+0000 7f5b01a00640 1 -- 172.21.15.67:0/530895573 >> 172.21.15.67:0/530895573 conn(0x7f5afc0fcb10 msgr2=0x7f5afc0fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:12.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.403+0000 7f5b01a00640 1 -- 172.21.15.67:0/530895573 shutdown_connections 2024-03-28T16:03:12.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.403+0000 7f5b01a00640 1 -- 172.21.15.67:0/530895573 wait complete. 2024-03-28T16:03:12.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.404+0000 7f5b01a00640 1 Processor -- start 2024-03-28T16:03:12.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.405+0000 7f5b01a00640 1 -- start start 2024-03-28T16:03:12.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.405+0000 7f5b01a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5afc1015a0 0x7f5afc199880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:12.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.405+0000 7f5b01a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5afc102790 0x7f5afc199dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:12.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.405+0000 7f5b01a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5afc103150 0x7f5afc1a0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:12.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.405+0000 7f5b01a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5afc10d100 con 0x7f5afc102790 2024-03-28T16:03:12.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.405+0000 7f5b01a00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5afc10cf80 con 0x7f5afc103150 2024-03-28T16:03:12.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.405+0000 7f5b01a00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5afc10d280 con 0x7f5afc1015a0 2024-03-28T16:03:12.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.405+0000 7f5afb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5afc102790 0x7f5afc199dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:12.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.405+0000 7f5afb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5afc102790 0x7f5afc199dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57950/0 (socket says 172.21.15.67:57950) 2024-03-28T16:03:12.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.405+0000 7f5afb400640 1 -- 172.21.15.67:0/1574054973 learned_addr learned my addr 172.21.15.67:0/1574054973 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:03:12.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.406+0000 7f5b01000640 1 --2- 172.21.15.67:0/1574054973 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5afc103150 0x7f5afc1a0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:12.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.406+0000 7f5afbe00640 1 --2- 172.21.15.67:0/1574054973 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5afc1015a0 0x7f5afc199880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:12.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.406+0000 7f5afb400640 1 -- 172.21.15.67:0/1574054973 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5afc1015a0 msgr2=0x7f5afc199880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:12.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.406+0000 7f5afb400640 1 --2- 172.21.15.67:0/1574054973 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5afc1015a0 0x7f5afc199880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:12.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.406+0000 7f5afb400640 1 -- 172.21.15.67:0/1574054973 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5afc103150 msgr2=0x7f5afc1a0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:12.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.406+0000 7f5afb400640 1 --2- 172.21.15.67:0/1574054973 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5afc103150 0x7f5afc1a0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:12.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.406+0000 7f5afb400640 1 -- 172.21.15.67:0/1574054973 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5afc1a14a0 con 0x7f5afc102790 2024-03-28T16:03:12.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.406+0000 7f5afb400640 1 --2- 172.21.15.67:0/1574054973 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5afc102790 0x7f5afc199dc0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f5ae800c920 tx=0x7f5ae800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:12.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.406+0000 7f5af8c00640 1 -- 172.21.15.67:0/1574054973 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae8007c70 con 0x7f5afc102790 2024-03-28T16:03:12.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.406+0000 7f5af8c00640 1 -- 172.21.15.67:0/1574054973 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5ae8007e10 con 0x7f5afc102790 2024-03-28T16:03:12.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.407+0000 7f5af8c00640 1 -- 172.21.15.67:0/1574054973 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae8018820 con 0x7f5afc102790 2024-03-28T16:03:12.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.407+0000 7f5b01a00640 1 -- 172.21.15.67:0/1574054973 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5afc1a1790 con 0x7f5afc102790 2024-03-28T16:03:12.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.407+0000 7f5b01a00640 1 -- 172.21.15.67:0/1574054973 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5afc1a1c90 con 0x7f5afc102790 2024-03-28T16:03:12.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.408+0000 7f5b01a00640 1 -- 172.21.15.67:0/1574054973 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5afc1a1f50 con 0x7f5afc102790 2024-03-28T16:03:12.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.411+0000 7f5af8c00640 1 -- 172.21.15.67:0/1574054973 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f5ae80189c0 con 0x7f5afc102790 2024-03-28T16:03:12.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.412+0000 7f5af8c00640 1 --2- 172.21.15.67:0/1574054973 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f5ae00a2800 0x7f5ae00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:12.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.412+0000 7f5af8c00640 1 -- 172.21.15.67:0/1574054973 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f5ae80cecb0 con 0x7f5afc102790 2024-03-28T16:03:12.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.412+0000 7f5afbe00640 1 --2- 172.21.15.67:0/1574054973 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f5ae00a2800 0x7f5ae00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:12.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.413+0000 7f5afbe00640 1 --2- 172.21.15.67:0/1574054973 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f5ae00a2800 0x7f5ae00a4cc0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f5afc1025f0 tx=0x7f5aec009870 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:12.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.416+0000 7f5af8c00640 1 -- 172.21.15.67:0/1574054973 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5ae8014030 con 0x7f5afc102790 2024-03-28T16:03:12.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.619+0000 7f5b01a00640 1 -- 172.21.15.67:0/1574054973 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5afc1a2120 con 0x7f5afc102790 2024-03-28T16:03:12.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.620+0000 7f5af8c00640 1 -- 172.21.15.67:0/1574054973 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5ae809bd00 con 0x7f5afc102790 2024-03-28T16:03:12.621 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-03-28T16:03:12.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.623+0000 7f5b01a00640 1 -- 172.21.15.67:0/1574054973 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f5ae00a2800 msgr2=0x7f5ae00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:12.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.623+0000 7f5b01a00640 1 --2- 172.21.15.67:0/1574054973 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f5ae00a2800 0x7f5ae00a4cc0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f5afc1025f0 tx=0x7f5aec009870 comp rx=0 tx=0).stop 2024-03-28T16:03:12.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.623+0000 7f5b01a00640 1 -- 172.21.15.67:0/1574054973 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5afc102790 msgr2=0x7f5afc199dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:12.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.623+0000 7f5b01a00640 1 --2- 172.21.15.67:0/1574054973 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5afc102790 0x7f5afc199dc0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f5ae800c920 tx=0x7f5ae800cde0 comp rx=0 tx=0).stop 2024-03-28T16:03:12.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.623+0000 7f5b01a00640 1 -- 172.21.15.67:0/1574054973 shutdown_connections 2024-03-28T16:03:12.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.623+0000 7f5b01a00640 1 --2- 172.21.15.67:0/1574054973 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f5ae00a2800 0x7f5ae00a4cc0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:12.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.623+0000 7f5b01a00640 1 --2- 172.21.15.67:0/1574054973 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5afc103150 0x7f5afc1a0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:12.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.623+0000 7f5b01a00640 1 --2- 172.21.15.67:0/1574054973 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5afc102790 0x7f5afc199dc0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:12.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.623+0000 7f5b01a00640 1 --2- 172.21.15.67:0/1574054973 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5afc1015a0 0x7f5afc199880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:12.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.623+0000 7f5b01a00640 1 -- 172.21.15.67:0/1574054973 >> 172.21.15.67:0/1574054973 conn(0x7f5afc0fcb10 msgr2=0x7f5afc0738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:12.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.623+0000 7f5b01a00640 1 -- 172.21.15.67:0/1574054973 shutdown_connections 2024-03-28T16:03:12.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.623+0000 7f5b01a00640 1 -- 172.21.15.67:0/1574054973 wait complete. 2024-03-28T16:03:12.644 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T16:03:12.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.848+0000 7f330be00640 1 -- 172.21.15.67:0/4294015368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f330c1048e0 msgr2=0x7f330c104d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:12.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.848+0000 7f330be00640 1 --2- 172.21.15.67:0/4294015368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f330c1048e0 0x7f330c104d60 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f32f4009900 tx=0x7f32f402bc30 comp rx=0 tx=0).stop 2024-03-28T16:03:12.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.848+0000 7f330be00640 1 -- 172.21.15.67:0/4294015368 shutdown_connections 2024-03-28T16:03:12.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.848+0000 7f330be00640 1 --2- 172.21.15.67:0/4294015368 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f330c1052a0 0x7f330c10bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:12.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.848+0000 7f330be00640 1 --2- 172.21.15.67:0/4294015368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f330c1048e0 0x7f330c104d60 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:12.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.848+0000 7f330be00640 1 --2- 172.21.15.67:0/4294015368 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f330c1036f0 0x7f330c103b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:12.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.848+0000 7f330be00640 1 -- 172.21.15.67:0/4294015368 >> 172.21.15.67:0/4294015368 conn(0x7f330c0fec80 msgr2=0x7f330c1010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:12.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.848+0000 7f330be00640 1 -- 172.21.15.67:0/4294015368 shutdown_connections 2024-03-28T16:03:12.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.848+0000 7f330be00640 1 -- 172.21.15.67:0/4294015368 wait complete. 2024-03-28T16:03:12.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.849+0000 7f330be00640 1 Processor -- start 2024-03-28T16:03:12.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.850+0000 7f330be00640 1 -- start start 2024-03-28T16:03:12.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.850+0000 7f330be00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f330c1036f0 0x7f330c1998e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:12.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.850+0000 7f330be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f330c1048e0 0x7f330c199e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:12.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.850+0000 7f330be00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f330c1052a0 0x7f330c1a0e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:12.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.850+0000 7f330be00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f330c10f270 con 0x7f330c1048e0 2024-03-28T16:03:12.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.850+0000 7f330be00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f330c10f0f0 con 0x7f330c1052a0 2024-03-28T16:03:12.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.850+0000 7f330be00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f330c10f3f0 con 0x7f330c1036f0 2024-03-28T16:03:12.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.850+0000 7f3303e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f330c1048e0 0x7f330c199e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:12.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.850+0000 7f3303e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f330c1048e0 0x7f330c199e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57970/0 (socket says 172.21.15.67:57970) 2024-03-28T16:03:12.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.850+0000 7f3303e00640 1 -- 172.21.15.67:0/3346593467 learned_addr learned my addr 172.21.15.67:0/3346593467 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:03:12.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.850+0000 7f330aa00640 1 --2- 172.21.15.67:0/3346593467 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f330c1036f0 0x7f330c1998e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:12.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.850+0000 7f330b400640 1 --2- 172.21.15.67:0/3346593467 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f330c1052a0 0x7f330c1a0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:12.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.851+0000 7f3303e00640 1 -- 172.21.15.67:0/3346593467 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f330c1036f0 msgr2=0x7f330c1998e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:12.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.851+0000 7f3303e00640 1 --2- 172.21.15.67:0/3346593467 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f330c1036f0 0x7f330c1998e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:12.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.851+0000 7f3303e00640 1 -- 172.21.15.67:0/3346593467 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f330c1052a0 msgr2=0x7f330c1a0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:12.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.851+0000 7f3303e00640 1 --2- 172.21.15.67:0/3346593467 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f330c1052a0 0x7f330c1a0e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:12.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.851+0000 7f3303e00640 1 -- 172.21.15.67:0/3346593467 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f330c1a1500 con 0x7f330c1048e0 2024-03-28T16:03:12.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.851+0000 7f3303e00640 1 --2- 172.21.15.67:0/3346593467 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f330c1048e0 0x7f330c199e20 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f32f4000c00 tx=0x7f32f40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:12.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.851+0000 7f3303400640 1 -- 172.21.15.67:0/3346593467 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32f4043020 con 0x7f330c1048e0 2024-03-28T16:03:12.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.851+0000 7f330be00640 1 -- 172.21.15.67:0/3346593467 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f330c1a1790 con 0x7f330c1048e0 2024-03-28T16:03:12.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.851+0000 7f3303400640 1 -- 172.21.15.67:0/3346593467 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f32f40054c0 con 0x7f330c1048e0 2024-03-28T16:03:12.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.851+0000 7f3303400640 1 -- 172.21.15.67:0/3346593467 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32f4003d70 con 0x7f330c1048e0 2024-03-28T16:03:12.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.852+0000 7f330be00640 1 -- 172.21.15.67:0/3346593467 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f330c1a1d30 con 0x7f330c1048e0 2024-03-28T16:03:12.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.853+0000 7f330be00640 1 -- 172.21.15.67:0/3346593467 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f330c10aa10 con 0x7f330c1048e0 2024-03-28T16:03:12.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.855+0000 7f3303400640 1 -- 172.21.15.67:0/3346593467 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f32f4046020 con 0x7f330c1048e0 2024-03-28T16:03:12.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.856+0000 7f3303400640 1 --2- 172.21.15.67:0/3346593467 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f32f00a2800 0x7f32f00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:12.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.856+0000 7f3303400640 1 -- 172.21.15.67:0/3346593467 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f32f40f13e0 con 0x7f330c1048e0 2024-03-28T16:03:12.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.857+0000 7f330aa00640 1 --2- 172.21.15.67:0/3346593467 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f32f00a2800 0x7f32f00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:12.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.857+0000 7f330aa00640 1 --2- 172.21.15.67:0/3346593467 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f32f00a2800 0x7f32f00a4cc0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f32fc005fd0 tx=0x7f32fc005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:12.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:12.859+0000 7f3303400640 1 -- 172.21.15.67:0/3346593467 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f32f404d020 con 0x7f330c1048e0 2024-03-28T16:03:13.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:12 smithi150 ceph-mon[34825]: pgmap v428: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:03:13.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:12 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/2169717936' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T16:03:13.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:12 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/1574054973' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T16:03:13.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:12 smithi067 ceph-mon[31707]: pgmap v428: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:03:13.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:12 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/2169717936' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T16:03:13.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:12 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/1574054973' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T16:03:13.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:12 smithi183 ceph-mon[35580]: pgmap v428: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:03:13.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:12 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/2169717936' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T16:03:13.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:12 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/1574054973' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T16:03:13.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.105+0000 7f330be00640 1 -- 172.21.15.67:0/3346593467 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f330c02d070 con 0x7f330c1048e0 2024-03-28T16:03:13.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.107+0000 7f3303400640 1 -- 172.21.15.67:0/3346593467 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11341 (secure 0 0 0) 0x7f32f40be550 con 0x7f330c1048e0 2024-03-28T16:03:13.108 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:03:13.108 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2008297659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2008297659},{"type":"v1","addr":"172.21.15.67:6835","nonce":2008297659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14532,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/320295745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":320295745},{"type":"v1","addr":"172.21.15.67:6837","nonce":320295745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6839/691821273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":691821273},{"type":"v1","addr":"172.21.15.67:6839","nonce":691821273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6841/3211254875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6840","nonce":3211254875},{"type":"v1","addr":"172.21.15.67:6841","nonce":3211254875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2129665423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2129665423},{"type":"v1","addr":"172.21.15.183:6833","nonce":2129665423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24496,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/1589784425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":1589784425},{"type":"v1","addr":"172.21.15.183:6835","nonce":1589784425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24508,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/2185863835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":2185863835},{"type":"v1","addr":"172.21.15.183:6837","nonce":2185863835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24520,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6839/1242814398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":1242814398},{"type":"v1","addr":"172.21.15.183:6839","nonce":1242814398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":34409,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1008335521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1008335521},{"type":"v1","addr":"172.21.15.150:6833","nonce":1008335521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":34445,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3269935695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3269935695},{"type":"v1","addr":"172.21.15.150:6835","nonce":3269935695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T16:01:07.696824+0000","modified":"2024-03-28T16:01:11.997773+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":34508},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34475":{"gid":34475,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/1112886277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":1112886277},{"type":"v1","addr":"172.21.15.150:6837","nonce":1112886277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_34508":{"gid":34508,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6839/1211011614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":1211011614},{"type":"v1","addr":"172.21.15.150:6839","nonce":1211011614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34508,"qdb_cluster":[34508]},"id":1}]} 2024-03-28T16:03:13.109 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 19 2024-03-28T16:03:13.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.109+0000 7f330be00640 1 -- 172.21.15.67:0/3346593467 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f32f00a2800 msgr2=0x7f32f00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:13.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.109+0000 7f330be00640 1 --2- 172.21.15.67:0/3346593467 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f32f00a2800 0x7f32f00a4cc0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f32fc005fd0 tx=0x7f32fc005850 comp rx=0 tx=0).stop 2024-03-28T16:03:13.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.109+0000 7f330be00640 1 -- 172.21.15.67:0/3346593467 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f330c1048e0 msgr2=0x7f330c199e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:13.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.109+0000 7f330be00640 1 --2- 172.21.15.67:0/3346593467 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f330c1048e0 0x7f330c199e20 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f32f4000c00 tx=0x7f32f40042f0 comp rx=0 tx=0).stop 2024-03-28T16:03:13.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.110+0000 7f330be00640 1 -- 172.21.15.67:0/3346593467 shutdown_connections 2024-03-28T16:03:13.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.110+0000 7f330be00640 1 --2- 172.21.15.67:0/3346593467 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f32f00a2800 0x7f32f00a4cc0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:13.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.110+0000 7f330be00640 1 --2- 172.21.15.67:0/3346593467 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f330c1052a0 0x7f330c1a0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:13.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.110+0000 7f330be00640 1 --2- 172.21.15.67:0/3346593467 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f330c1048e0 0x7f330c199e20 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:13.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.110+0000 7f330be00640 1 --2- 172.21.15.67:0/3346593467 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f330c1036f0 0x7f330c1998e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:13.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.110+0000 7f330be00640 1 -- 172.21.15.67:0/3346593467 >> 172.21.15.67:0/3346593467 conn(0x7f330c0fec80 msgr2=0x7f330c107be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:13.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.110+0000 7f330be00640 1 -- 172.21.15.67:0/3346593467 shutdown_connections 2024-03-28T16:03:13.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.110+0000 7f330be00640 1 -- 172.21.15.67:0/3346593467 wait complete. 2024-03-28T16:03:13.129 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T16:03:13.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.349+0000 7f050be00640 1 -- 172.21.15.67:0/711857936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0504108ed0 msgr2=0x7f0504109350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:13.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.349+0000 7f050be00640 1 --2- 172.21.15.67:0/711857936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0504108ed0 0x7f0504109350 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f04f0009900 tx=0x7f04f002bc30 comp rx=0 tx=0).stop 2024-03-28T16:03:13.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.349+0000 7f050be00640 1 -- 172.21.15.67:0/711857936 shutdown_connections 2024-03-28T16:03:13.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.350+0000 7f050be00640 1 --2- 172.21.15.67:0/711857936 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0504109890 0x7f0504110120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:13.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.350+0000 7f050be00640 1 --2- 172.21.15.67:0/711857936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0504108ed0 0x7f0504109350 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:13.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.350+0000 7f050be00640 1 --2- 172.21.15.67:0/711857936 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0504107ce0 0x7f0504108100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:13.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.350+0000 7f050be00640 1 -- 172.21.15.67:0/711857936 >> 172.21.15.67:0/711857936 conn(0x7f0504076cb0 msgr2=0x7f05040790d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:13.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.350+0000 7f050be00640 1 -- 172.21.15.67:0/711857936 shutdown_connections 2024-03-28T16:03:13.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.350+0000 7f050be00640 1 -- 172.21.15.67:0/711857936 wait complete. 2024-03-28T16:03:13.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.351+0000 7f050be00640 1 Processor -- start 2024-03-28T16:03:13.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.351+0000 7f050be00640 1 -- start start 2024-03-28T16:03:13.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.352+0000 7f050be00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0504107ce0 0x7f050419df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:13.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.352+0000 7f050be00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0504108ed0 0x7f050419e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:13.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.352+0000 7f050be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0504109890 0x7f05041a5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:13.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.352+0000 7f050be00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05041138d0 con 0x7f0504109890 2024-03-28T16:03:13.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.352+0000 7f050be00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0504113750 con 0x7f0504108ed0 2024-03-28T16:03:13.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.352+0000 7f050be00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0504113a50 con 0x7f0504107ce0 2024-03-28T16:03:13.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.352+0000 7f0509e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0504109890 0x7f05041a5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:13.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.352+0000 7f0509e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0504109890 0x7f05041a5540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57988/0 (socket says 172.21.15.67:57988) 2024-03-28T16:03:13.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.352+0000 7f0509e00640 1 -- 172.21.15.67:0/1428025930 learned_addr learned my addr 172.21.15.67:0/1428025930 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:03:13.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.352+0000 7f0509400640 1 --2- 172.21.15.67:0/1428025930 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0504107ce0 0x7f050419df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:13.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.352+0000 7f0508a00640 1 --2- 172.21.15.67:0/1428025930 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0504108ed0 0x7f050419e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:13.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.352+0000 7f0509e00640 1 -- 172.21.15.67:0/1428025930 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0504107ce0 msgr2=0x7f050419df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:13.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.352+0000 7f0509e00640 1 --2- 172.21.15.67:0/1428025930 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0504107ce0 0x7f050419df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:13.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.352+0000 7f0509e00640 1 -- 172.21.15.67:0/1428025930 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0504108ed0 msgr2=0x7f050419e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:13.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.352+0000 7f0509e00640 1 --2- 172.21.15.67:0/1428025930 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0504108ed0 0x7f050419e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:13.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.352+0000 7f0509e00640 1 -- 172.21.15.67:0/1428025930 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05041a5bb0 con 0x7f0504109890 2024-03-28T16:03:13.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.352+0000 7f0508a00640 1 --2- 172.21.15.67:0/1428025930 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0504108ed0 0x7f050419e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:03:13.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.353+0000 7f0509e00640 1 --2- 172.21.15.67:0/1428025930 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0504109890 0x7f05041a5540 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f04fc00eaa0 tx=0x7f04fc00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:13.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.353+0000 7f0502000640 1 -- 172.21.15.67:0/1428025930 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04fc00cdb0 con 0x7f0504109890 2024-03-28T16:03:13.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.353+0000 7f0502000640 1 -- 172.21.15.67:0/1428025930 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f04fc005020 con 0x7f0504109890 2024-03-28T16:03:13.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.353+0000 7f0502000640 1 -- 172.21.15.67:0/1428025930 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04fc0116d0 con 0x7f0504109890 2024-03-28T16:03:13.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.353+0000 7f050be00640 1 -- 172.21.15.67:0/1428025930 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05041a5ea0 con 0x7f0504109890 2024-03-28T16:03:13.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.353+0000 7f050be00640 1 -- 172.21.15.67:0/1428025930 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05041a6390 con 0x7f0504109890 2024-03-28T16:03:13.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.354+0000 7f050be00640 1 -- 172.21.15.67:0/1428025930 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f050410f000 con 0x7f0504109890 2024-03-28T16:03:13.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.356+0000 7f0502000640 1 -- 172.21.15.67:0/1428025930 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f04fc0051d0 con 0x7f0504109890 2024-03-28T16:03:13.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.358+0000 7f0502000640 1 --2- 172.21.15.67:0/1428025930 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f04ec0a2800 0x7f04ec0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:13.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.358+0000 7f0502000640 1 -- 172.21.15.67:0/1428025930 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f04fc0ced40 con 0x7f0504109890 2024-03-28T16:03:13.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.358+0000 7f0509400640 1 --2- 172.21.15.67:0/1428025930 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f04ec0a2800 0x7f04ec0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:13.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.359+0000 7f0509400640 1 --2- 172.21.15.67:0/1428025930 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f04ec0a2800 0x7f04ec0a4cc0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f0504108d30 tx=0x7f04f8008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:13.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.362+0000 7f0502000640 1 -- 172.21.15.67:0/1428025930 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f04fc016030 con 0x7f0504109890 2024-03-28T16:03:13.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.567+0000 7f050be00640 1 -- 172.21.15.67:0/1428025930 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0504062340 con 0x7f0504109890 2024-03-28T16:03:13.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.570+0000 7f0502000640 1 -- 172.21.15.67:0/1428025930 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) v1 ==== 74+0+22673 (secure 0 0 0) 0x7f04fc09bd90 con 0x7f0504109890 2024-03-28T16:03:13.572 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:03:13.572 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":92,"fsid":"b5f8225a-ed1a-11ee-95d0-87774f69a715","created":"2024-03-28T15:50:21.471924+0000","modified":"2024-03-28T16:01:40.643215+0000","last_up_change":"2024-03-28T16:00:01.369123+0000","last_in_change":"2024-03-28T15:59:37.515483+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T15:54:55.588007+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T16:01:05.719801+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T16:01:07.206303+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7c1791b6-9c0e-4512-a327-cad166bb5581","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6803","nonce":1894681701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6805","nonce":1894681701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6809","nonce":1894681701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6807","nonce":1894681701}]},"public_addr":"172.21.15.67:6803/1894681701","cluster_addr":"172.21.15.67:6805/1894681701","heartbeat_back_addr":"172.21.15.67:6809/1894681701","heartbeat_front_addr":"172.21.15.67:6807/1894681701","state":["exists","up"]},{"osd":1,"uuid":"0cdda7f3-d00c-4436-bcc1-49fb8072b2e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6801","nonce":1734465762}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6803","nonce":1734465762}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6807","nonce":1734465762}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6805","nonce":1734465762}]},"public_addr":"172.21.15.150:6801/1734465762","cluster_addr":"172.21.15.150:6803/1734465762","heartbeat_back_addr":"172.21.15.150:6807/1734465762","heartbeat_front_addr":"172.21.15.150:6805/1734465762","state":["exists","up"]},{"osd":2,"uuid":"a55b9a02-c3fd-4061-95c0-53a24a17ade0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6801","nonce":3398095931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6803","nonce":3398095931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6807","nonce":3398095931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6805","nonce":3398095931}]},"public_addr":"172.21.15.183:6801/3398095931","cluster_addr":"172.21.15.183:6803/3398095931","heartbeat_back_addr":"172.21.15.183:6807/3398095931","heartbeat_front_addr":"172.21.15.183:6805/3398095931","state":["exists","up"]},{"osd":3,"uuid":"d34940c0-5ce9-4b8b-aa36-4872414a043f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6811","nonce":1933541875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6813","nonce":1933541875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6817","nonce":1933541875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6815","nonce":1933541875}]},"public_addr":"172.21.15.67:6811/1933541875","cluster_addr":"172.21.15.67:6813/1933541875","heartbeat_back_addr":"172.21.15.67:6817/1933541875","heartbeat_front_addr":"172.21.15.67:6815/1933541875","state":["exists","up"]},{"osd":4,"uuid":"5cc3e260-83a3-4a4d-aa6a-34a3b12b2b5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6809","nonce":3775269993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6811","nonce":3775269993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6815","nonce":3775269993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6813","nonce":3775269993}]},"public_addr":"172.21.15.150:6809/3775269993","cluster_addr":"172.21.15.150:6811/3775269993","heartbeat_back_addr":"172.21.15.150:6815/3775269993","heartbeat_front_addr":"172.21.15.150:6813/3775269993","state":["exists","up"]},{"osd":5,"uuid":"a2542b21-d6ac-413e-8b9b-4a6548b20840","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6809","nonce":164284137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6811","nonce":164284137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6815","nonce":164284137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6813","nonce":164284137}]},"public_addr":"172.21.15.183:6809/164284137","cluster_addr":"172.21.15.183:6811/164284137","heartbeat_back_addr":"172.21.15.183:6815/164284137","heartbeat_front_addr":"172.21.15.183:6813/164284137","state":["exists","up"]},{"osd":6,"uuid":"8905a2d3-0f1b-4853-a218-22da2635150a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6819","nonce":3565528013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6821","nonce":3565528013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6825","nonce":3565528013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6823","nonce":3565528013}]},"public_addr":"172.21.15.67:6819/3565528013","cluster_addr":"172.21.15.67:6821/3565528013","heartbeat_back_addr":"172.21.15.67:6825/3565528013","heartbeat_front_addr":"172.21.15.67:6823/3565528013","state":["exists","up"]},{"osd":7,"uuid":"b275055b-b451-4180-91b1-e6e61a828c48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6817","nonce":3735115340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6819","nonce":3735115340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6823","nonce":3735115340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6821","nonce":3735115340}]},"public_addr":"172.21.15.150:6817/3735115340","cluster_addr":"172.21.15.150:6819/3735115340","heartbeat_back_addr":"172.21.15.150:6823/3735115340","heartbeat_front_addr":"172.21.15.150:6821/3735115340","state":["exists","up"]},{"osd":8,"uuid":"b6abf7ad-eb34-4f60-a4eb-348a92d7195b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6817","nonce":1025400095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6819","nonce":1025400095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6823","nonce":1025400095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6821","nonce":1025400095}]},"public_addr":"172.21.15.183:6817/1025400095","cluster_addr":"172.21.15.183:6819/1025400095","heartbeat_back_addr":"172.21.15.183:6823/1025400095","heartbeat_front_addr":"172.21.15.183:6821/1025400095","state":["exists","up"]},{"osd":9,"uuid":"28a78e3a-0657-40aa-816f-16a6fadadcbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6827","nonce":76885409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6829","nonce":76885409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6833","nonce":76885409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6831","nonce":76885409}]},"public_addr":"172.21.15.67:6827/76885409","cluster_addr":"172.21.15.67:6829/76885409","heartbeat_back_addr":"172.21.15.67:6833/76885409","heartbeat_front_addr":"172.21.15.67:6831/76885409","state":["exists","up"]},{"osd":10,"uuid":"d287907e-424f-4b49-a06e-4613c6de0474","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6825","nonce":3489982080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6827","nonce":3489982080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6831","nonce":3489982080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6829","nonce":3489982080}]},"public_addr":"172.21.15.150:6825/3489982080","cluster_addr":"172.21.15.150:6827/3489982080","heartbeat_back_addr":"172.21.15.150:6831/3489982080","heartbeat_front_addr":"172.21.15.150:6829/3489982080","state":["exists","up"]},{"osd":11,"uuid":"a3c29c47-a228-4f42-a23c-fbdb0c093699","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":85,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6825","nonce":1892639849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6826","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6827","nonce":1892639849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6831","nonce":1892639849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6829","nonce":1892639849}]},"public_addr":"172.21.15.183:6825/1892639849","cluster_addr":"172.21.15.183:6827/1892639849","heartbeat_back_addr":"172.21.15.183:6831/1892639849","heartbeat_front_addr":"172.21.15.183:6829/1892639849","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:53:45.646211+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:54:18.956590+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:54:52.161646+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:55:27.502822+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:56:01.162932+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:56:34.674452+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:57:09.298679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:57:43.230406+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:58:17.003821+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:58:51.085863+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:59:25.665065+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T16:00:00.263457+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":1}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6801/1631001968":"2024-03-29T15:51:38.875334+0000","172.21.15.67:6800/1631001968":"2024-03-29T15:51:38.875334+0000","172.21.15.67:0/1533056310":"2024-03-29T15:51:38.875334+0000","172.21.15.67:6801/3580491071":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/2030412458":"2024-03-29T15:51:03.859740+0000","172.21.15.67:6800/3580491071":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/2905307533":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/3763759988":"2024-03-29T15:51:38.875334+0000","172.21.15.67:0/4089399880":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/4239108661":"2024-03-29T15:51:38.875334+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T16:03:13.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.573+0000 7f050be00640 1 -- 172.21.15.67:0/1428025930 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f04ec0a2800 msgr2=0x7f04ec0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:13.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.573+0000 7f050be00640 1 --2- 172.21.15.67:0/1428025930 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f04ec0a2800 0x7f04ec0a4cc0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f0504108d30 tx=0x7f04f8008040 comp rx=0 tx=0).stop 2024-03-28T16:03:13.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.573+0000 7f050be00640 1 -- 172.21.15.67:0/1428025930 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0504109890 msgr2=0x7f05041a5540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:13.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.573+0000 7f050be00640 1 --2- 172.21.15.67:0/1428025930 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0504109890 0x7f05041a5540 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f04fc00eaa0 tx=0x7f04fc00ef60 comp rx=0 tx=0).stop 2024-03-28T16:03:13.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.573+0000 7f050be00640 1 -- 172.21.15.67:0/1428025930 shutdown_connections 2024-03-28T16:03:13.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.573+0000 7f050be00640 1 --2- 172.21.15.67:0/1428025930 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f04ec0a2800 0x7f04ec0a4cc0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:13.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.573+0000 7f050be00640 1 --2- 172.21.15.67:0/1428025930 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0504109890 0x7f05041a5540 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:13.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.573+0000 7f050be00640 1 --2- 172.21.15.67:0/1428025930 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0504108ed0 0x7f050419e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:13.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.573+0000 7f050be00640 1 --2- 172.21.15.67:0/1428025930 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0504107ce0 0x7f050419df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:13.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.573+0000 7f050be00640 1 -- 172.21.15.67:0/1428025930 >> 172.21.15.67:0/1428025930 conn(0x7f0504076cb0 msgr2=0x7f050410c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:13.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.574+0000 7f050be00640 1 -- 172.21.15.67:0/1428025930 shutdown_connections 2024-03-28T16:03:13.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.574+0000 7f050be00640 1 -- 172.21.15.67:0/1428025930 wait complete. 2024-03-28T16:03:13.595 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T15:54:55.588007+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T16:01:05.719801+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T16:01:07.206303+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T16:03:13.595 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T16:03:13.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.827+0000 7f6ce1000640 1 -- 172.21.15.67:0/3515962365 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cdc104a50 msgr2=0x7f6cdc106e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:13.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.827+0000 7f6ce1000640 1 --2- 172.21.15.67:0/3515962365 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cdc104a50 0x7f6cdc106e40 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f6cc4009900 tx=0x7f6cc402bc30 comp rx=0 tx=0).stop 2024-03-28T16:03:13.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.827+0000 7f6ce1000640 1 -- 172.21.15.67:0/3515962365 shutdown_connections 2024-03-28T16:03:13.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.827+0000 7f6ce1000640 1 --2- 172.21.15.67:0/3515962365 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6cdc107380 0x7f6cdc1097d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:13.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.827+0000 7f6ce1000640 1 --2- 172.21.15.67:0/3515962365 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cdc104a50 0x7f6cdc106e40 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:13.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.827+0000 7f6ce1000640 1 --2- 172.21.15.67:0/3515962365 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6cdc102120 0x7f6cdc104510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:13.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.827+0000 7f6ce1000640 1 -- 172.21.15.67:0/3515962365 >> 172.21.15.67:0/3515962365 conn(0x7f6cdc0fbb30 msgr2=0x7f6cdc0fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:13.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.828+0000 7f6ce1000640 1 -- 172.21.15.67:0/3515962365 shutdown_connections 2024-03-28T16:03:13.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.828+0000 7f6ce1000640 1 -- 172.21.15.67:0/3515962365 wait complete. 2024-03-28T16:03:13.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.829+0000 7f6ce1000640 1 Processor -- start 2024-03-28T16:03:13.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.829+0000 7f6ce1000640 1 -- start start 2024-03-28T16:03:13.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.829+0000 7f6ce1000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6cdc102120 0x7f6cdc101a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:13.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.829+0000 7f6ce1000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cdc104a50 0x7f6cdc1000d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:13.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.829+0000 7f6ce1000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6cdc107380 0x7f6cdc100610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:13.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.829+0000 7f6ce1000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cdc10d230 con 0x7f6cdc104a50 2024-03-28T16:03:13.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.829+0000 7f6ce1000640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cdc10d0b0 con 0x7f6cdc107380 2024-03-28T16:03:13.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.829+0000 7f6ce1000640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cdc10d3b0 con 0x7f6cdc102120 2024-03-28T16:03:13.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.829+0000 7f6cd9400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cdc104a50 0x7f6cdc1000d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:13.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.829+0000 7f6cd9400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cdc104a50 0x7f6cdc1000d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57996/0 (socket says 172.21.15.67:57996) 2024-03-28T16:03:13.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.829+0000 7f6cd9400640 1 -- 172.21.15.67:0/638567553 learned_addr learned my addr 172.21.15.67:0/638567553 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:03:13.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.830+0000 7f6cd9e00640 1 --2- 172.21.15.67:0/638567553 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6cdc102120 0x7f6cdc101a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:13.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.830+0000 7f6cd9400640 1 -- 172.21.15.67:0/638567553 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6cdc102120 msgr2=0x7f6cdc101a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:13.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.830+0000 7f6cd9400640 1 --2- 172.21.15.67:0/638567553 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6cdc102120 0x7f6cdc101a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:13.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.830+0000 7f6cd9400640 1 -- 172.21.15.67:0/638567553 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6cdc107380 msgr2=0x7f6cdc100610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:13.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.830+0000 7f6cd9400640 1 --2- 172.21.15.67:0/638567553 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6cdc107380 0x7f6cdc100610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:13.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.830+0000 7f6cd9400640 1 -- 172.21.15.67:0/638567553 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cdc100ea0 con 0x7f6cdc104a50 2024-03-28T16:03:13.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.830+0000 7f6cd9400640 1 --2- 172.21.15.67:0/638567553 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cdc104a50 0x7f6cdc1000d0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f6cc4005ab0 tx=0x7f6cc40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:13.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.831+0000 7f6cd2a00640 1 -- 172.21.15.67:0/638567553 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cc4043020 con 0x7f6cdc104a50 2024-03-28T16:03:13.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.831+0000 7f6ce1000640 1 -- 172.21.15.67:0/638567553 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cdc1a1a60 con 0x7f6cdc104a50 2024-03-28T16:03:13.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.831+0000 7f6cd2a00640 1 -- 172.21.15.67:0/638567553 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6cc40054c0 con 0x7f6cdc104a50 2024-03-28T16:03:13.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.831+0000 7f6cd2a00640 1 -- 172.21.15.67:0/638567553 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cc4003d10 con 0x7f6cdc104a50 2024-03-28T16:03:13.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.831+0000 7f6ce1000640 1 -- 172.21.15.67:0/638567553 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cdc1a2000 con 0x7f6cdc104a50 2024-03-28T16:03:13.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.832+0000 7f6ce1000640 1 -- 172.21.15.67:0/638567553 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cdc0fd620 con 0x7f6cdc104a50 2024-03-28T16:03:13.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.834+0000 7f6cd2a00640 1 -- 172.21.15.67:0/638567553 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f6cc4046020 con 0x7f6cdc104a50 2024-03-28T16:03:13.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.835+0000 7f6cd2a00640 1 --2- 172.21.15.67:0/638567553 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f6cc00a2800 0x7f6cc00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:13.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.835+0000 7f6cd2a00640 1 -- 172.21.15.67:0/638567553 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f6cc40f0f80 con 0x7f6cdc104a50 2024-03-28T16:03:13.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.835+0000 7f6cd9e00640 1 --2- 172.21.15.67:0/638567553 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f6cc00a2800 0x7f6cc00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:13.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.836+0000 7f6cd9e00640 1 --2- 172.21.15.67:0/638567553 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f6cc00a2800 0x7f6cc00a4cc0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc005fd0 tx=0x7f6ccc009870 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:13.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:13.839+0000 7f6cd2a00640 1 -- 172.21.15.67:0/638567553 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6cc404d020 con 0x7f6cdc104a50 2024-03-28T16:03:14.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.042+0000 7f6ce1000640 1 -- 172.21.15.67:0/638567553 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6cdc0622a0 con 0x7f6cdc104a50 2024-03-28T16:03:14.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.043+0000 7f6cd2a00640 1 -- 172.21.15.67:0/638567553 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v92) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6cc40be0f0 con 0x7f6cdc104a50 2024-03-28T16:03:14.044 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-03-28T16:03:14.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.046+0000 7f6ce1000640 1 -- 172.21.15.67:0/638567553 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f6cc00a2800 msgr2=0x7f6cc00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:14.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.046+0000 7f6ce1000640 1 --2- 172.21.15.67:0/638567553 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f6cc00a2800 0x7f6cc00a4cc0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc005fd0 tx=0x7f6ccc009870 comp rx=0 tx=0).stop 2024-03-28T16:03:14.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.046+0000 7f6ce1000640 1 -- 172.21.15.67:0/638567553 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cdc104a50 msgr2=0x7f6cdc1000d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:14.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.046+0000 7f6ce1000640 1 --2- 172.21.15.67:0/638567553 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cdc104a50 0x7f6cdc1000d0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f6cc4005ab0 tx=0x7f6cc40042f0 comp rx=0 tx=0).stop 2024-03-28T16:03:14.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.046+0000 7f6ce1000640 1 -- 172.21.15.67:0/638567553 shutdown_connections 2024-03-28T16:03:14.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.046+0000 7f6ce1000640 1 --2- 172.21.15.67:0/638567553 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f6cc00a2800 0x7f6cc00a4cc0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:14.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.046+0000 7f6ce1000640 1 --2- 172.21.15.67:0/638567553 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6cdc107380 0x7f6cdc100610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:14.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.046+0000 7f6ce1000640 1 --2- 172.21.15.67:0/638567553 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cdc104a50 0x7f6cdc1000d0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:14.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.046+0000 7f6ce1000640 1 --2- 172.21.15.67:0/638567553 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6cdc102120 0x7f6cdc101a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:14.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.046+0000 7f6ce1000640 1 -- 172.21.15.67:0/638567553 >> 172.21.15.67:0/638567553 conn(0x7f6cdc0fbb30 msgr2=0x7f6cdc102ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:14.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.047+0000 7f6ce1000640 1 -- 172.21.15.67:0/638567553 shutdown_connections 2024-03-28T16:03:14.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.047+0000 7f6ce1000640 1 -- 172.21.15.67:0/638567553 wait complete. 2024-03-28T16:03:14.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:13 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/3346593467' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:03:14.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:13 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/1428025930' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:03:14.065 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:13 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/3346593467' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:03:14.065 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:13 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/1428025930' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:03:14.069 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T16:03:14.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:13 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/3346593467' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:03:14.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:13 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/1428025930' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:03:14.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.284+0000 7f21f3e00640 1 -- 172.21.15.67:0/3484534903 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21f4102030 msgr2=0x7f21f41088c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:14.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.284+0000 7f21f3e00640 1 --2- 172.21.15.67:0/3484534903 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21f4102030 0x7f21f41088c0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f21ec009ca0 tx=0x7f21ec02c0f0 comp rx=0 tx=0).stop 2024-03-28T16:03:14.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.284+0000 7f21f3e00640 1 -- 172.21.15.67:0/3484534903 shutdown_connections 2024-03-28T16:03:14.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.284+0000 7f21f3e00640 1 --2- 172.21.15.67:0/3484534903 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21f4102030 0x7f21f41088c0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:14.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.284+0000 7f21f3e00640 1 --2- 172.21.15.67:0/3484534903 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f21f4101670 0x7f21f4101af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:14.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.284+0000 7f21f3e00640 1 --2- 172.21.15.67:0/3484534903 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21f4100480 0x7f21f41008a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:14.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.284+0000 7f21f3e00640 1 -- 172.21.15.67:0/3484534903 >> 172.21.15.67:0/3484534903 conn(0x7f21f40fba10 msgr2=0x7f21f40fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:14.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.285+0000 7f21f3e00640 1 -- 172.21.15.67:0/3484534903 shutdown_connections 2024-03-28T16:03:14.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.285+0000 7f21f3e00640 1 -- 172.21.15.67:0/3484534903 wait complete. 2024-03-28T16:03:14.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.286+0000 7f21f3e00640 1 Processor -- start 2024-03-28T16:03:14.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.286+0000 7f21f3e00640 1 -- start start 2024-03-28T16:03:14.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.286+0000 7f21f3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21f4100480 0x7f21f41999d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:14.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.286+0000 7f21f3e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21f4101670 0x7f21f4199f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:14.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.286+0000 7f21f3e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f21f4102030 0x7f21f41a0f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:14.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.286+0000 7f21f3e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21f410bfc0 con 0x7f21f4100480 2024-03-28T16:03:14.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.286+0000 7f21f3e00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21f410be40 con 0x7f21f4101670 2024-03-28T16:03:14.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.286+0000 7f21f3e00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21f410c140 con 0x7f21f4102030 2024-03-28T16:03:14.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.286+0000 7f21f2a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21f4100480 0x7f21f41999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:14.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.286+0000 7f21f2a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21f4100480 0x7f21f41999d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58022/0 (socket says 172.21.15.67:58022) 2024-03-28T16:03:14.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.286+0000 7f21f2a00640 1 -- 172.21.15.67:0/2225461524 learned_addr learned my addr 172.21.15.67:0/2225461524 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:03:14.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.287+0000 7f21ea000640 1 --2- 172.21.15.67:0/2225461524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21f4101670 0x7f21f4199f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:14.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.287+0000 7f21f2a00640 1 -- 172.21.15.67:0/2225461524 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f21f4102030 msgr2=0x7f21f41a0f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:14.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.287+0000 7f21f2a00640 1 --2- 172.21.15.67:0/2225461524 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f21f4102030 0x7f21f41a0f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:14.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.287+0000 7f21f2a00640 1 -- 172.21.15.67:0/2225461524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21f4101670 msgr2=0x7f21f4199f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:14.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.287+0000 7f21f2a00640 1 --2- 172.21.15.67:0/2225461524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21f4101670 0x7f21f4199f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:14.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.287+0000 7f21f2a00640 1 -- 172.21.15.67:0/2225461524 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21f41a1580 con 0x7f21f4100480 2024-03-28T16:03:14.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.287+0000 7f21f2a00640 1 --2- 172.21.15.67:0/2225461524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21f4100480 0x7f21f41999d0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f21e400e9c0 tx=0x7f21e400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:14.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.287+0000 7f21ebe00640 1 -- 172.21.15.67:0/2225461524 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e400cd80 con 0x7f21f4100480 2024-03-28T16:03:14.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.287+0000 7f21ebe00640 1 -- 172.21.15.67:0/2225461524 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f21e40035f0 con 0x7f21f4100480 2024-03-28T16:03:14.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.287+0000 7f21ebe00640 1 -- 172.21.15.67:0/2225461524 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4010690 con 0x7f21f4100480 2024-03-28T16:03:14.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.287+0000 7f21f3e00640 1 -- 172.21.15.67:0/2225461524 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21f41a1870 con 0x7f21f4100480 2024-03-28T16:03:14.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.287+0000 7f21f3e00640 1 -- 172.21.15.67:0/2225461524 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21f41a1d70 con 0x7f21f4100480 2024-03-28T16:03:14.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.289+0000 7f21f3e00640 1 -- 172.21.15.67:0/2225461524 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21f41a2030 con 0x7f21f4100480 2024-03-28T16:03:14.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.291+0000 7f21ebe00640 1 -- 172.21.15.67:0/2225461524 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f21e40037a0 con 0x7f21f4100480 2024-03-28T16:03:14.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.292+0000 7f21ebe00640 1 --2- 172.21.15.67:0/2225461524 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f21d80a2880 0x7f21d80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:14.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.292+0000 7f21ebe00640 1 -- 172.21.15.67:0/2225461524 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f21e40ceed0 con 0x7f21f4100480 2024-03-28T16:03:14.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.292+0000 7f21ea000640 1 --2- 172.21.15.67:0/2225461524 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f21d80a2880 0x7f21d80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:14.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.293+0000 7f21ea000640 1 --2- 172.21.15.67:0/2225461524 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f21d80a2880 0x7f21d80a4d40 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f21f419ae70 tx=0x7f21dc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:14.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.296+0000 7f21ebe00640 1 -- 172.21.15.67:0/2225461524 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f21e409bf60 con 0x7f21f4100480 2024-03-28T16:03:14.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.500+0000 7f21f3e00640 1 -- 172.21.15.67:0/2225461524 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f21f41a2200 con 0x7f21f4100480 2024-03-28T16:03:14.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.501+0000 7f21ebe00640 1 -- 172.21.15.67:0/2225461524 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v92) v1 ==== 104+0+11 (secure 0 0 0) 0x7f21e409b860 con 0x7f21f4100480 2024-03-28T16:03:14.502 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-03-28T16:03:14.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.503+0000 7f21f3e00640 1 -- 172.21.15.67:0/2225461524 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f21d80a2880 msgr2=0x7f21d80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:14.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.503+0000 7f21f3e00640 1 --2- 172.21.15.67:0/2225461524 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f21d80a2880 0x7f21d80a4d40 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f21f419ae70 tx=0x7f21dc005f90 comp rx=0 tx=0).stop 2024-03-28T16:03:14.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.503+0000 7f21f3e00640 1 -- 172.21.15.67:0/2225461524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21f4100480 msgr2=0x7f21f41999d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:14.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.503+0000 7f21f3e00640 1 --2- 172.21.15.67:0/2225461524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21f4100480 0x7f21f41999d0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f21e400e9c0 tx=0x7f21e400ee80 comp rx=0 tx=0).stop 2024-03-28T16:03:14.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.504+0000 7f21f3e00640 1 -- 172.21.15.67:0/2225461524 shutdown_connections 2024-03-28T16:03:14.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.504+0000 7f21f3e00640 1 --2- 172.21.15.67:0/2225461524 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f21d80a2880 0x7f21d80a4d40 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:14.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.504+0000 7f21f3e00640 1 --2- 172.21.15.67:0/2225461524 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f21f4102030 0x7f21f41a0f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:14.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.504+0000 7f21f3e00640 1 --2- 172.21.15.67:0/2225461524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21f4101670 0x7f21f4199f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:14.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.504+0000 7f21f3e00640 1 --2- 172.21.15.67:0/2225461524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21f4100480 0x7f21f41999d0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:14.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.504+0000 7f21f3e00640 1 -- 172.21.15.67:0/2225461524 >> 172.21.15.67:0/2225461524 conn(0x7f21f40fba10 msgr2=0x7f21f40fd4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:14.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.504+0000 7f21f3e00640 1 -- 172.21.15.67:0/2225461524 shutdown_connections 2024-03-28T16:03:14.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.504+0000 7f21f3e00640 1 -- 172.21.15.67:0/2225461524 wait complete. 2024-03-28T16:03:14.525 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T16:03:14.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.754+0000 7f647f400640 1 -- 172.21.15.67:0/4182154813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64780fff90 msgr2=0x7f64781003b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:14.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.754+0000 7f647f400640 1 --2- 172.21.15.67:0/4182154813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64780fff90 0x7f64781003b0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f6468009900 tx=0x7f646802bc30 comp rx=0 tx=0).stop 2024-03-28T16:03:14.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.754+0000 7f647f400640 1 -- 172.21.15.67:0/4182154813 shutdown_connections 2024-03-28T16:03:14.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.754+0000 7f647f400640 1 --2- 172.21.15.67:0/4182154813 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6478101290 0x7f6478101710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:14.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.754+0000 7f647f400640 1 --2- 172.21.15.67:0/4182154813 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64781008f0 0x7f6478100d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:14.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.754+0000 7f647f400640 1 --2- 172.21.15.67:0/4182154813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64780fff90 0x7f64781003b0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:14.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.754+0000 7f647f400640 1 -- 172.21.15.67:0/4182154813 >> 172.21.15.67:0/4182154813 conn(0x7f64780fbb50 msgr2=0x7f64780fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:14.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.754+0000 7f647f400640 1 -- 172.21.15.67:0/4182154813 shutdown_connections 2024-03-28T16:03:14.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.754+0000 7f647f400640 1 -- 172.21.15.67:0/4182154813 wait complete. 2024-03-28T16:03:14.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.755+0000 7f647f400640 1 Processor -- start 2024-03-28T16:03:14.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.755+0000 7f647f400640 1 -- start start 2024-03-28T16:03:14.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.756+0000 7f647f400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64780fff90 0x7f6478199ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:14.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.756+0000 7f647f400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f64781008f0 0x7f647819a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:14.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.756+0000 7f647f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6478101290 0x7f64781a1070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:14.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.756+0000 7f647f400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f647810f420 con 0x7f6478101290 2024-03-28T16:03:14.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.756+0000 7f647f400640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f647810f2a0 con 0x7f64780fff90 2024-03-28T16:03:14.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.756+0000 7f647f400640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f647810f5a0 con 0x7f64781008f0 2024-03-28T16:03:14.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.756+0000 7f647d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6478101290 0x7f64781a1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:14.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.756+0000 7f647d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6478101290 0x7f64781a1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58036/0 (socket says 172.21.15.67:58036) 2024-03-28T16:03:14.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.756+0000 7f647d400640 1 -- 172.21.15.67:0/3510524582 learned_addr learned my addr 172.21.15.67:0/3510524582 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:03:14.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.757+0000 7f646fe00640 1 --2- 172.21.15.67:0/3510524582 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f64781008f0 0x7f647819a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:14.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.757+0000 7f647ca00640 1 --2- 172.21.15.67:0/3510524582 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64780fff90 0x7f6478199ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:14.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.757+0000 7f647d400640 1 -- 172.21.15.67:0/3510524582 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f64781008f0 msgr2=0x7f647819a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:14.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.757+0000 7f647d400640 1 --2- 172.21.15.67:0/3510524582 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f64781008f0 0x7f647819a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:14.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.757+0000 7f647d400640 1 -- 172.21.15.67:0/3510524582 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64780fff90 msgr2=0x7f6478199ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:14.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.757+0000 7f647d400640 1 --2- 172.21.15.67:0/3510524582 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64780fff90 0x7f6478199ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:14.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.757+0000 7f647d400640 1 -- 172.21.15.67:0/3510524582 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64781a16e0 con 0x7f6478101290 2024-03-28T16:03:14.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.757+0000 7f647ca00640 1 --2- 172.21.15.67:0/3510524582 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64780fff90 0x7f6478199ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:03:14.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.757+0000 7f647d400640 1 --2- 172.21.15.67:0/3510524582 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6478101290 0x7f64781a1070 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f647000eaa0 tx=0x7f647000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:14.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.757+0000 7f6476000640 1 -- 172.21.15.67:0/3510524582 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f647000cdb0 con 0x7f6478101290 2024-03-28T16:03:14.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.757+0000 7f6476000640 1 -- 172.21.15.67:0/3510524582 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6470005020 con 0x7f6478101290 2024-03-28T16:03:14.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.757+0000 7f6476000640 1 -- 172.21.15.67:0/3510524582 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64700116d0 con 0x7f6478101290 2024-03-28T16:03:14.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.757+0000 7f647f400640 1 -- 172.21.15.67:0/3510524582 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64781a19d0 con 0x7f6478101290 2024-03-28T16:03:14.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.758+0000 7f647f400640 1 -- 172.21.15.67:0/3510524582 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64781a1ed0 con 0x7f6478101290 2024-03-28T16:03:14.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.759+0000 7f647f400640 1 -- 172.21.15.67:0/3510524582 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f647810ace0 con 0x7f6478101290 2024-03-28T16:03:14.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.761+0000 7f6476000640 1 -- 172.21.15.67:0/3510524582 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f64700051d0 con 0x7f6478101290 2024-03-28T16:03:14.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.762+0000 7f6476000640 1 --2- 172.21.15.67:0/3510524582 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f645c0a2880 0x7f645c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:14.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.762+0000 7f6476000640 1 -- 172.21.15.67:0/3510524582 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f64700cecb0 con 0x7f6478101290 2024-03-28T16:03:14.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.762+0000 7f647ca00640 1 --2- 172.21.15.67:0/3510524582 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f645c0a2880 0x7f645c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:14.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.763+0000 7f647ca00640 1 --2- 172.21.15.67:0/3510524582 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f645c0a2880 0x7f645c0a4d40 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f6468036040 tx=0x7f6468005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:14.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.767+0000 7f6476000640 1 -- 172.21.15.67:0/3510524582 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6470016030 con 0x7f6478101290 2024-03-28T16:03:14.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:14 smithi067 ceph-mon[31707]: pgmap v429: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:03:14.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:14 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/638567553' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T16:03:14.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:14 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/2225461524' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T16:03:14.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.972+0000 7f647f400640 1 -- 172.21.15.67:0/3510524582 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6478062340 con 0x7f6478101290 2024-03-28T16:03:14.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.973+0000 7f6476000640 1 -- 172.21.15.67:0/3510524582 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) v1 ==== 100+0+11 (secure 0 0 0) 0x7f647009bd00 con 0x7f6478101290 2024-03-28T16:03:14.975 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-03-28T16:03:14.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.976+0000 7f647f400640 1 -- 172.21.15.67:0/3510524582 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f645c0a2880 msgr2=0x7f645c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:14.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.976+0000 7f647f400640 1 --2- 172.21.15.67:0/3510524582 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f645c0a2880 0x7f645c0a4d40 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f6468036040 tx=0x7f6468005fd0 comp rx=0 tx=0).stop 2024-03-28T16:03:14.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.976+0000 7f647f400640 1 -- 172.21.15.67:0/3510524582 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6478101290 msgr2=0x7f64781a1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:14.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.976+0000 7f647f400640 1 --2- 172.21.15.67:0/3510524582 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6478101290 0x7f64781a1070 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f647000eaa0 tx=0x7f647000ef60 comp rx=0 tx=0).stop 2024-03-28T16:03:14.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.976+0000 7f647f400640 1 -- 172.21.15.67:0/3510524582 shutdown_connections 2024-03-28T16:03:14.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.976+0000 7f647f400640 1 --2- 172.21.15.67:0/3510524582 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f645c0a2880 0x7f645c0a4d40 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:14.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.976+0000 7f647f400640 1 --2- 172.21.15.67:0/3510524582 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6478101290 0x7f64781a1070 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:14.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.976+0000 7f647f400640 1 --2- 172.21.15.67:0/3510524582 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f64781008f0 0x7f647819a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:14.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.976+0000 7f647f400640 1 --2- 172.21.15.67:0/3510524582 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64780fff90 0x7f6478199ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:14.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.976+0000 7f647f400640 1 -- 172.21.15.67:0/3510524582 >> 172.21.15.67:0/3510524582 conn(0x7f64780fbb50 msgr2=0x7f64780fee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:14.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.976+0000 7f647f400640 1 -- 172.21.15.67:0/3510524582 shutdown_connections 2024-03-28T16:03:14.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:14.976+0000 7f647f400640 1 -- 172.21.15.67:0/3510524582 wait complete. 2024-03-28T16:03:14.998 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T16:03:15.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:14 smithi150 ceph-mon[34825]: pgmap v429: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:03:15.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:14 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/638567553' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T16:03:15.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:14 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/2225461524' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T16:03:15.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:14 smithi183 ceph-mon[35580]: pgmap v429: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:03:15.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:14 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/638567553' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T16:03:15.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:14 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/2225461524' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T16:03:15.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.211+0000 7f6d8be00640 1 -- 172.21.15.67:0/2185197239 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d8c074100 msgr2=0x7f6d8c074580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:15.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.211+0000 7f6d8be00640 1 --2- 172.21.15.67:0/2185197239 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d8c074100 0x7f6d8c074580 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f6d74009930 tx=0x7f6d7402bc20 comp rx=0 tx=0).stop 2024-03-28T16:03:15.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.211+0000 7f6d8be00640 1 -- 172.21.15.67:0/2185197239 shutdown_connections 2024-03-28T16:03:15.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.211+0000 7f6d8be00640 1 --2- 172.21.15.67:0/2185197239 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6d8c10dab0 0x7f6d8c10fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:15.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.211+0000 7f6d8be00640 1 --2- 172.21.15.67:0/2185197239 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d8c074100 0x7f6d8c074580 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:15.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.211+0000 7f6d8be00640 1 --2- 172.21.15.67:0/2185197239 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d8c0756e0 0x7f6d8c073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:15.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.211+0000 7f6d8be00640 1 -- 172.21.15.67:0/2185197239 >> 172.21.15.67:0/2185197239 conn(0x7f6d8c0fca30 msgr2=0x7f6d8c0fee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:15.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.211+0000 7f6d8be00640 1 -- 172.21.15.67:0/2185197239 shutdown_connections 2024-03-28T16:03:15.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.211+0000 7f6d8be00640 1 -- 172.21.15.67:0/2185197239 wait complete. 2024-03-28T16:03:15.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.212+0000 7f6d8be00640 1 Processor -- start 2024-03-28T16:03:15.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.212+0000 7f6d8be00640 1 -- start start 2024-03-28T16:03:15.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.212+0000 7f6d8be00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d8c074100 0x7f6d8c19dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:15.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.212+0000 7f6d8be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d8c0756e0 0x7f6d8c19e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:15.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.212+0000 7f6d8be00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6d8c10dab0 0x7f6d8c1a52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:15.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.212+0000 7f6d8be00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d8c113630 con 0x7f6d8c0756e0 2024-03-28T16:03:15.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.212+0000 7f6d8be00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d8c1134b0 con 0x7f6d8c074100 2024-03-28T16:03:15.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.212+0000 7f6d8be00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d8c1137b0 con 0x7f6d8c10dab0 2024-03-28T16:03:15.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.212+0000 7f6d8a000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d8c0756e0 0x7f6d8c19e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:15.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.213+0000 7f6d8a000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d8c0756e0 0x7f6d8c19e290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58062/0 (socket says 172.21.15.67:58062) 2024-03-28T16:03:15.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.213+0000 7f6d8a000640 1 -- 172.21.15.67:0/1340948877 learned_addr learned my addr 172.21.15.67:0/1340948877 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:03:15.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.213+0000 7f6d8aa00640 1 --2- 172.21.15.67:0/1340948877 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d8c074100 0x7f6d8c19dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:15.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.213+0000 7f6d8b400640 1 --2- 172.21.15.67:0/1340948877 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6d8c10dab0 0x7f6d8c1a52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:15.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.213+0000 7f6d8a000640 1 -- 172.21.15.67:0/1340948877 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6d8c10dab0 msgr2=0x7f6d8c1a52c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:15.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.213+0000 7f6d8a000640 1 --2- 172.21.15.67:0/1340948877 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6d8c10dab0 0x7f6d8c1a52c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:15.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.213+0000 7f6d8a000640 1 -- 172.21.15.67:0/1340948877 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d8c074100 msgr2=0x7f6d8c19dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:15.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.213+0000 7f6d8a000640 1 --2- 172.21.15.67:0/1340948877 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d8c074100 0x7f6d8c19dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:15.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.213+0000 7f6d8a000640 1 -- 172.21.15.67:0/1340948877 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d8c1a5960 con 0x7f6d8c0756e0 2024-03-28T16:03:15.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.213+0000 7f6d8aa00640 1 --2- 172.21.15.67:0/1340948877 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d8c074100 0x7f6d8c19dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:03:15.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.213+0000 7f6d8a000640 1 --2- 172.21.15.67:0/1340948877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d8c0756e0 0x7f6d8c19e290 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f6d74005ab0 tx=0x7f6d740042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:15.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.214+0000 7f6d83400640 1 -- 172.21.15.67:0/1340948877 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d74043020 con 0x7f6d8c0756e0 2024-03-28T16:03:15.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.214+0000 7f6d83400640 1 -- 172.21.15.67:0/1340948877 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6d740054c0 con 0x7f6d8c0756e0 2024-03-28T16:03:15.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.214+0000 7f6d8be00640 1 -- 172.21.15.67:0/1340948877 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d8c1a5bf0 con 0x7f6d8c0756e0 2024-03-28T16:03:15.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.214+0000 7f6d83400640 1 -- 172.21.15.67:0/1340948877 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d74003d10 con 0x7f6d8c0756e0 2024-03-28T16:03:15.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.214+0000 7f6d8be00640 1 -- 172.21.15.67:0/1340948877 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d8c1a6190 con 0x7f6d8c0756e0 2024-03-28T16:03:15.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.215+0000 7f6d8be00640 1 -- 172.21.15.67:0/1340948877 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d8c103c80 con 0x7f6d8c0756e0 2024-03-28T16:03:15.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.217+0000 7f6d83400640 1 -- 172.21.15.67:0/1340948877 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f6d74046020 con 0x7f6d8c0756e0 2024-03-28T16:03:15.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.218+0000 7f6d83400640 1 --2- 172.21.15.67:0/1340948877 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f6d700a2880 0x7f6d700a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:15.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.218+0000 7f6d83400640 1 -- 172.21.15.67:0/1340948877 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f6d740f1180 con 0x7f6d8c0756e0 2024-03-28T16:03:15.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.219+0000 7f6d8aa00640 1 --2- 172.21.15.67:0/1340948877 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f6d700a2880 0x7f6d700a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:15.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.219+0000 7f6d8aa00640 1 --2- 172.21.15.67:0/1340948877 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f6d700a2880 0x7f6d700a4d40 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c19f220 tx=0x7f6d7c009400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:15.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.221+0000 7f6d83400640 1 -- 172.21.15.67:0/1340948877 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6d7404d020 con 0x7f6d8c0756e0 2024-03-28T16:03:15.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.467+0000 7f6d8be00640 1 -- 172.21.15.67:0/1340948877 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6d8c02d2d0 con 0x7f6d8c0756e0 2024-03-28T16:03:15.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.469+0000 7f6d83400640 1 -- 172.21.15.67:0/1340948877 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11341 (secure 0 0 0) 0x7f6d740be1d0 con 0x7f6d8c0756e0 2024-03-28T16:03:15.470 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:03:15.470 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2008297659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2008297659},{"type":"v1","addr":"172.21.15.67:6835","nonce":2008297659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14532,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/320295745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":320295745},{"type":"v1","addr":"172.21.15.67:6837","nonce":320295745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6839/691821273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":691821273},{"type":"v1","addr":"172.21.15.67:6839","nonce":691821273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6841/3211254875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6840","nonce":3211254875},{"type":"v1","addr":"172.21.15.67:6841","nonce":3211254875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2129665423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2129665423},{"type":"v1","addr":"172.21.15.183:6833","nonce":2129665423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24496,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/1589784425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":1589784425},{"type":"v1","addr":"172.21.15.183:6835","nonce":1589784425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24508,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/2185863835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":2185863835},{"type":"v1","addr":"172.21.15.183:6837","nonce":2185863835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24520,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6839/1242814398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":1242814398},{"type":"v1","addr":"172.21.15.183:6839","nonce":1242814398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":34409,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1008335521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1008335521},{"type":"v1","addr":"172.21.15.150:6833","nonce":1008335521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":34445,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3269935695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3269935695},{"type":"v1","addr":"172.21.15.150:6835","nonce":3269935695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T16:01:07.696824+0000","modified":"2024-03-28T16:01:11.997773+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":34508},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34475":{"gid":34475,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/1112886277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":1112886277},{"type":"v1","addr":"172.21.15.150:6837","nonce":1112886277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_34508":{"gid":34508,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6839/1211011614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":1211011614},{"type":"v1","addr":"172.21.15.150:6839","nonce":1211011614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34508,"qdb_cluster":[34508]},"id":1}]} 2024-03-28T16:03:15.471 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 19 2024-03-28T16:03:15.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.471+0000 7f6d8be00640 1 -- 172.21.15.67:0/1340948877 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f6d700a2880 msgr2=0x7f6d700a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:15.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.471+0000 7f6d8be00640 1 --2- 172.21.15.67:0/1340948877 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f6d700a2880 0x7f6d700a4d40 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c19f220 tx=0x7f6d7c009400 comp rx=0 tx=0).stop 2024-03-28T16:03:15.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.471+0000 7f6d8be00640 1 -- 172.21.15.67:0/1340948877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d8c0756e0 msgr2=0x7f6d8c19e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:15.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.471+0000 7f6d8be00640 1 --2- 172.21.15.67:0/1340948877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d8c0756e0 0x7f6d8c19e290 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f6d74005ab0 tx=0x7f6d740042f0 comp rx=0 tx=0).stop 2024-03-28T16:03:15.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.471+0000 7f6d8be00640 1 -- 172.21.15.67:0/1340948877 shutdown_connections 2024-03-28T16:03:15.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.471+0000 7f6d8be00640 1 --2- 172.21.15.67:0/1340948877 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f6d700a2880 0x7f6d700a4d40 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:15.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.472+0000 7f6d8be00640 1 --2- 172.21.15.67:0/1340948877 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6d8c10dab0 0x7f6d8c1a52c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:15.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.472+0000 7f6d8be00640 1 --2- 172.21.15.67:0/1340948877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d8c0756e0 0x7f6d8c19e290 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:15.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.472+0000 7f6d8be00640 1 --2- 172.21.15.67:0/1340948877 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d8c074100 0x7f6d8c19dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:15.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.472+0000 7f6d8be00640 1 -- 172.21.15.67:0/1340948877 >> 172.21.15.67:0/1340948877 conn(0x7f6d8c0fca30 msgr2=0x7f6d8c100e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:15.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.472+0000 7f6d8be00640 1 -- 172.21.15.67:0/1340948877 shutdown_connections 2024-03-28T16:03:15.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.472+0000 7f6d8be00640 1 -- 172.21.15.67:0/1340948877 wait complete. 2024-03-28T16:03:15.491 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T16:03:15.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.704+0000 7f49f6200640 1 -- 172.21.15.67:0/2546806763 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49f01036f0 msgr2=0x7f49f0103b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:15.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.704+0000 7f49f6200640 1 --2- 172.21.15.67:0/2546806763 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49f01036f0 0x7f49f0103b10 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f49e0009900 tx=0x7f49e002bc30 comp rx=0 tx=0).stop 2024-03-28T16:03:15.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.704+0000 7f49f6200640 1 -- 172.21.15.67:0/2546806763 shutdown_connections 2024-03-28T16:03:15.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.704+0000 7f49f6200640 1 --2- 172.21.15.67:0/2546806763 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f49f01052a0 0x7f49f010bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:15.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.704+0000 7f49f6200640 1 --2- 172.21.15.67:0/2546806763 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49f01048e0 0x7f49f0104d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:15.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.704+0000 7f49f6200640 1 --2- 172.21.15.67:0/2546806763 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49f01036f0 0x7f49f0103b10 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:15.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.704+0000 7f49f6200640 1 -- 172.21.15.67:0/2546806763 >> 172.21.15.67:0/2546806763 conn(0x7f49f00fec80 msgr2=0x7f49f01010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:15.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.704+0000 7f49f6200640 1 -- 172.21.15.67:0/2546806763 shutdown_connections 2024-03-28T16:03:15.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.704+0000 7f49f6200640 1 -- 172.21.15.67:0/2546806763 wait complete. 2024-03-28T16:03:15.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.705+0000 7f49f6200640 1 Processor -- start 2024-03-28T16:03:15.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.705+0000 7f49f6200640 1 -- start start 2024-03-28T16:03:15.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.706+0000 7f49f6200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49f01036f0 0x7f49f01998e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:15.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.706+0000 7f49f6200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f49f01048e0 0x7f49f0199e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:15.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.706+0000 7f49f6200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49f01052a0 0x7f49f01a0e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:15.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.706+0000 7f49f6200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49f010f270 con 0x7f49f01036f0 2024-03-28T16:03:15.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.706+0000 7f49f6200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49f010f0f0 con 0x7f49f01052a0 2024-03-28T16:03:15.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.706+0000 7f49f6200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49f010f3f0 con 0x7f49f01048e0 2024-03-28T16:03:15.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.706+0000 7f49f4e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49f01036f0 0x7f49f01998e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:15.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.706+0000 7f49f4e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49f01036f0 0x7f49f01998e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58072/0 (socket says 172.21.15.67:58072) 2024-03-28T16:03:15.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.706+0000 7f49f4e00640 1 -- 172.21.15.67:0/2510518779 learned_addr learned my addr 172.21.15.67:0/2510518779 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:03:15.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.706+0000 7f49efe00640 1 --2- 172.21.15.67:0/2510518779 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f49f01048e0 0x7f49f0199e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:15.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.707+0000 7f49f5800640 1 --2- 172.21.15.67:0/2510518779 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49f01052a0 0x7f49f01a0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:15.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.707+0000 7f49f4e00640 1 -- 172.21.15.67:0/2510518779 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f49f01048e0 msgr2=0x7f49f0199e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:15.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.707+0000 7f49f4e00640 1 --2- 172.21.15.67:0/2510518779 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f49f01048e0 0x7f49f0199e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:15.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.707+0000 7f49f4e00640 1 -- 172.21.15.67:0/2510518779 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49f01052a0 msgr2=0x7f49f01a0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:15.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.707+0000 7f49f4e00640 1 --2- 172.21.15.67:0/2510518779 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49f01052a0 0x7f49f01a0e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:15.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.707+0000 7f49f4e00640 1 -- 172.21.15.67:0/2510518779 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49f01a1500 con 0x7f49f01036f0 2024-03-28T16:03:15.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.707+0000 7f49f5800640 1 --2- 172.21.15.67:0/2510518779 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49f01052a0 0x7f49f01a0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:03:15.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.707+0000 7f49f4e00640 1 --2- 172.21.15.67:0/2510518779 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49f01036f0 0x7f49f01998e0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f49e0005ab0 tx=0x7f49e00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:15.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.707+0000 7f49ed600640 1 -- 172.21.15.67:0/2510518779 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49e0043020 con 0x7f49f01036f0 2024-03-28T16:03:15.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.707+0000 7f49ed600640 1 -- 172.21.15.67:0/2510518779 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f49e00054c0 con 0x7f49f01036f0 2024-03-28T16:03:15.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.707+0000 7f49f6200640 1 -- 172.21.15.67:0/2510518779 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49f01a1790 con 0x7f49f01036f0 2024-03-28T16:03:15.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.707+0000 7f49ed600640 1 -- 172.21.15.67:0/2510518779 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49e0003d10 con 0x7f49f01036f0 2024-03-28T16:03:15.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.707+0000 7f49f6200640 1 -- 172.21.15.67:0/2510518779 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49f01a1ce0 con 0x7f49f01036f0 2024-03-28T16:03:15.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.708+0000 7f49f6200640 1 -- 172.21.15.67:0/2510518779 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49f010aa10 con 0x7f49f01036f0 2024-03-28T16:03:15.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.710+0000 7f49ed600640 1 -- 172.21.15.67:0/2510518779 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f49e0046020 con 0x7f49f01036f0 2024-03-28T16:03:15.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.712+0000 7f49ed600640 1 --2- 172.21.15.67:0/2510518779 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f49d40a2800 0x7f49d40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:15.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.713+0000 7f49ed600640 1 -- 172.21.15.67:0/2510518779 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f49e00f0f90 con 0x7f49f01036f0 2024-03-28T16:03:15.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.713+0000 7f49efe00640 1 --2- 172.21.15.67:0/2510518779 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f49d40a2800 0x7f49d40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:15.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.713+0000 7f49efe00640 1 --2- 172.21.15.67:0/2510518779 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f49d40a2800 0x7f49d40a4cc0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f49f019adf0 tx=0x7f49d8009440 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:15.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.717+0000 7f49ed600640 1 -- 172.21.15.67:0/2510518779 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f49e004d020 con 0x7f49f01036f0 2024-03-28T16:03:15.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.927+0000 7f49f6200640 1 -- 172.21.15.67:0/2510518779 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f49f00622a0 con 0x7f49f01036f0 2024-03-28T16:03:15.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.929+0000 7f49ed600640 1 -- 172.21.15.67:0/2510518779 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) v1 ==== 74+0+22673 (secure 0 0 0) 0x7f49e00be100 con 0x7f49f01036f0 2024-03-28T16:03:15.931 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:03:15.931 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":92,"fsid":"b5f8225a-ed1a-11ee-95d0-87774f69a715","created":"2024-03-28T15:50:21.471924+0000","modified":"2024-03-28T16:01:40.643215+0000","last_up_change":"2024-03-28T16:00:01.369123+0000","last_in_change":"2024-03-28T15:59:37.515483+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T15:54:55.588007+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T16:01:05.719801+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T16:01:07.206303+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7c1791b6-9c0e-4512-a327-cad166bb5581","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6803","nonce":1894681701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6805","nonce":1894681701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6809","nonce":1894681701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6807","nonce":1894681701}]},"public_addr":"172.21.15.67:6803/1894681701","cluster_addr":"172.21.15.67:6805/1894681701","heartbeat_back_addr":"172.21.15.67:6809/1894681701","heartbeat_front_addr":"172.21.15.67:6807/1894681701","state":["exists","up"]},{"osd":1,"uuid":"0cdda7f3-d00c-4436-bcc1-49fb8072b2e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6801","nonce":1734465762}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6803","nonce":1734465762}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6807","nonce":1734465762}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6805","nonce":1734465762}]},"public_addr":"172.21.15.150:6801/1734465762","cluster_addr":"172.21.15.150:6803/1734465762","heartbeat_back_addr":"172.21.15.150:6807/1734465762","heartbeat_front_addr":"172.21.15.150:6805/1734465762","state":["exists","up"]},{"osd":2,"uuid":"a55b9a02-c3fd-4061-95c0-53a24a17ade0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6801","nonce":3398095931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6803","nonce":3398095931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6807","nonce":3398095931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6805","nonce":3398095931}]},"public_addr":"172.21.15.183:6801/3398095931","cluster_addr":"172.21.15.183:6803/3398095931","heartbeat_back_addr":"172.21.15.183:6807/3398095931","heartbeat_front_addr":"172.21.15.183:6805/3398095931","state":["exists","up"]},{"osd":3,"uuid":"d34940c0-5ce9-4b8b-aa36-4872414a043f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6811","nonce":1933541875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6813","nonce":1933541875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6817","nonce":1933541875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6815","nonce":1933541875}]},"public_addr":"172.21.15.67:6811/1933541875","cluster_addr":"172.21.15.67:6813/1933541875","heartbeat_back_addr":"172.21.15.67:6817/1933541875","heartbeat_front_addr":"172.21.15.67:6815/1933541875","state":["exists","up"]},{"osd":4,"uuid":"5cc3e260-83a3-4a4d-aa6a-34a3b12b2b5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6809","nonce":3775269993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6811","nonce":3775269993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6815","nonce":3775269993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6813","nonce":3775269993}]},"public_addr":"172.21.15.150:6809/3775269993","cluster_addr":"172.21.15.150:6811/3775269993","heartbeat_back_addr":"172.21.15.150:6815/3775269993","heartbeat_front_addr":"172.21.15.150:6813/3775269993","state":["exists","up"]},{"osd":5,"uuid":"a2542b21-d6ac-413e-8b9b-4a6548b20840","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6809","nonce":164284137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6811","nonce":164284137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6815","nonce":164284137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6813","nonce":164284137}]},"public_addr":"172.21.15.183:6809/164284137","cluster_addr":"172.21.15.183:6811/164284137","heartbeat_back_addr":"172.21.15.183:6815/164284137","heartbeat_front_addr":"172.21.15.183:6813/164284137","state":["exists","up"]},{"osd":6,"uuid":"8905a2d3-0f1b-4853-a218-22da2635150a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6819","nonce":3565528013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6821","nonce":3565528013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6825","nonce":3565528013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6823","nonce":3565528013}]},"public_addr":"172.21.15.67:6819/3565528013","cluster_addr":"172.21.15.67:6821/3565528013","heartbeat_back_addr":"172.21.15.67:6825/3565528013","heartbeat_front_addr":"172.21.15.67:6823/3565528013","state":["exists","up"]},{"osd":7,"uuid":"b275055b-b451-4180-91b1-e6e61a828c48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6817","nonce":3735115340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6819","nonce":3735115340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6823","nonce":3735115340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6821","nonce":3735115340}]},"public_addr":"172.21.15.150:6817/3735115340","cluster_addr":"172.21.15.150:6819/3735115340","heartbeat_back_addr":"172.21.15.150:6823/3735115340","heartbeat_front_addr":"172.21.15.150:6821/3735115340","state":["exists","up"]},{"osd":8,"uuid":"b6abf7ad-eb34-4f60-a4eb-348a92d7195b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6817","nonce":1025400095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6819","nonce":1025400095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6823","nonce":1025400095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6821","nonce":1025400095}]},"public_addr":"172.21.15.183:6817/1025400095","cluster_addr":"172.21.15.183:6819/1025400095","heartbeat_back_addr":"172.21.15.183:6823/1025400095","heartbeat_front_addr":"172.21.15.183:6821/1025400095","state":["exists","up"]},{"osd":9,"uuid":"28a78e3a-0657-40aa-816f-16a6fadadcbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6827","nonce":76885409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6829","nonce":76885409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6833","nonce":76885409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6831","nonce":76885409}]},"public_addr":"172.21.15.67:6827/76885409","cluster_addr":"172.21.15.67:6829/76885409","heartbeat_back_addr":"172.21.15.67:6833/76885409","heartbeat_front_addr":"172.21.15.67:6831/76885409","state":["exists","up"]},{"osd":10,"uuid":"d287907e-424f-4b49-a06e-4613c6de0474","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6825","nonce":3489982080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6827","nonce":3489982080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6831","nonce":3489982080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6829","nonce":3489982080}]},"public_addr":"172.21.15.150:6825/3489982080","cluster_addr":"172.21.15.150:6827/3489982080","heartbeat_back_addr":"172.21.15.150:6831/3489982080","heartbeat_front_addr":"172.21.15.150:6829/3489982080","state":["exists","up"]},{"osd":11,"uuid":"a3c29c47-a228-4f42-a23c-fbdb0c093699","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":85,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6825","nonce":1892639849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6826","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6827","nonce":1892639849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6831","nonce":1892639849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6829","nonce":1892639849}]},"public_addr":"172.21.15.183:6825/1892639849","cluster_addr":"172.21.15.183:6827/1892639849","heartbeat_back_addr":"172.21.15.183:6831/1892639849","heartbeat_front_addr":"172.21.15.183:6829/1892639849","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:53:45.646211+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:54:18.956590+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:54:52.161646+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:55:27.502822+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:56:01.162932+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:56:34.674452+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:57:09.298679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:57:43.230406+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:58:17.003821+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:58:51.085863+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:59:25.665065+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T16:00:00.263457+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":1}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6801/1631001968":"2024-03-29T15:51:38.875334+0000","172.21.15.67:6800/1631001968":"2024-03-29T15:51:38.875334+0000","172.21.15.67:0/1533056310":"2024-03-29T15:51:38.875334+0000","172.21.15.67:6801/3580491071":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/2030412458":"2024-03-29T15:51:03.859740+0000","172.21.15.67:6800/3580491071":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/2905307533":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/3763759988":"2024-03-29T15:51:38.875334+0000","172.21.15.67:0/4089399880":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/4239108661":"2024-03-29T15:51:38.875334+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T16:03:15.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.932+0000 7f49f6200640 1 -- 172.21.15.67:0/2510518779 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f49d40a2800 msgr2=0x7f49d40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:15.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.932+0000 7f49f6200640 1 --2- 172.21.15.67:0/2510518779 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f49d40a2800 0x7f49d40a4cc0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f49f019adf0 tx=0x7f49d8009440 comp rx=0 tx=0).stop 2024-03-28T16:03:15.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.932+0000 7f49f6200640 1 -- 172.21.15.67:0/2510518779 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49f01036f0 msgr2=0x7f49f01998e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:15.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.932+0000 7f49f6200640 1 --2- 172.21.15.67:0/2510518779 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49f01036f0 0x7f49f01998e0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f49e0005ab0 tx=0x7f49e00042f0 comp rx=0 tx=0).stop 2024-03-28T16:03:15.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.932+0000 7f49f6200640 1 -- 172.21.15.67:0/2510518779 shutdown_connections 2024-03-28T16:03:15.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.932+0000 7f49f6200640 1 --2- 172.21.15.67:0/2510518779 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f49d40a2800 0x7f49d40a4cc0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:15.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.932+0000 7f49f6200640 1 --2- 172.21.15.67:0/2510518779 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49f01052a0 0x7f49f01a0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:15.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.932+0000 7f49f6200640 1 --2- 172.21.15.67:0/2510518779 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f49f01048e0 0x7f49f0199e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:15.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.932+0000 7f49f6200640 1 --2- 172.21.15.67:0/2510518779 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49f01036f0 0x7f49f01998e0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:15.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.932+0000 7f49f6200640 1 -- 172.21.15.67:0/2510518779 >> 172.21.15.67:0/2510518779 conn(0x7f49f00fec80 msgr2=0x7f49f0107be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:15.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.932+0000 7f49f6200640 1 -- 172.21.15.67:0/2510518779 shutdown_connections 2024-03-28T16:03:15.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:15.932+0000 7f49f6200640 1 -- 172.21.15.67:0/2510518779 wait complete. 2024-03-28T16:03:15.953 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 19, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-28T16:01:07.696824+0000', 'modified': '2024-03-28T16:01:11.997773+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 34508}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_34475': {'gid': 34475, 'name': 'h', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.150:6837/1112886277', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6836', 'nonce': 1112886277}, {'type': 'v1', 'addr': '172.21.15.150:6837', 'nonce': 1112886277}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_34508': {'gid': 34508, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6839/1211011614', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6838', 'nonce': 1211011614}, {'type': 'v1', 'addr': '172.21.15.150:6839', 'nonce': 1211011614}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 34508, 'qdb_cluster': [34508]} 2024-03-28T16:03:15.953 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-28T16:03:15.953 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-28T16:03:16.029 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:15 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/3510524582' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T16:03:16.029 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:15 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/1340948877' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:03:16.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:15 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/3510524582' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T16:03:16.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:15 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/1340948877' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:03:16.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:15 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/3510524582' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T16:03:16.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:15 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/1340948877' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:03:16.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.166+0000 7fecb2c00640 1 -- 172.21.15.67:0/1864315967 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fecac102770 msgr2=0x7fecac102bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:16.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.166+0000 7fecb2c00640 1 --2- 172.21.15.67:0/1864315967 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fecac102770 0x7fecac102bf0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fec98009900 tx=0x7fec9802bc30 comp rx=0 tx=0).stop 2024-03-28T16:03:16.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.166+0000 7fecb2c00640 1 -- 172.21.15.67:0/1864315967 shutdown_connections 2024-03-28T16:03:16.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.166+0000 7fecb2c00640 1 --2- 172.21.15.67:0/1864315967 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fecac103130 0x7fecac1099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:16.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.166+0000 7fecb2c00640 1 --2- 172.21.15.67:0/1864315967 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fecac102770 0x7fecac102bf0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:16.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.166+0000 7fecb2c00640 1 --2- 172.21.15.67:0/1864315967 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fecac101580 0x7fecac1019a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:16.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.166+0000 7fecb2c00640 1 -- 172.21.15.67:0/1864315967 >> 172.21.15.67:0/1864315967 conn(0x7fecac0fcb10 msgr2=0x7fecac0fef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:16.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.166+0000 7fecb2c00640 1 -- 172.21.15.67:0/1864315967 shutdown_connections 2024-03-28T16:03:16.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.166+0000 7fecb2c00640 1 -- 172.21.15.67:0/1864315967 wait complete. 2024-03-28T16:03:16.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.167+0000 7fecb2c00640 1 Processor -- start 2024-03-28T16:03:16.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.167+0000 7fecb2c00640 1 -- start start 2024-03-28T16:03:16.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.168+0000 7fecb2c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fecac101580 0x7fecac1977a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:16.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.168+0000 7fecb2c00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fecac102770 0x7fecac197ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:16.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.168+0000 7fecb2c00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fecac103130 0x7fecac19ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:16.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.168+0000 7fecb2c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecac10d0c0 con 0x7fecac101580 2024-03-28T16:03:16.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.168+0000 7fecb2c00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecac10cf40 con 0x7fecac103130 2024-03-28T16:03:16.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.168+0000 7fecb2c00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecac10d240 con 0x7fecac102770 2024-03-28T16:03:16.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.168+0000 7fecb1800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fecac101580 0x7fecac1977a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:16.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.168+0000 7fecb1800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fecac101580 0x7fecac1977a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58092/0 (socket says 172.21.15.67:58092) 2024-03-28T16:03:16.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.168+0000 7fecb1800640 1 -- 172.21.15.67:0/2925574350 learned_addr learned my addr 172.21.15.67:0/2925574350 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:03:16.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.168+0000 7fecb0e00640 1 --2- 172.21.15.67:0/2925574350 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fecac102770 0x7fecac197ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:16.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.168+0000 7fecb1800640 1 -- 172.21.15.67:0/2925574350 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fecac102770 msgr2=0x7fecac197ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:16.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.168+0000 7fecb1800640 1 --2- 172.21.15.67:0/2925574350 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fecac102770 0x7fecac197ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:16.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.168+0000 7fecb1800640 1 -- 172.21.15.67:0/2925574350 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fecac103130 msgr2=0x7fecac19ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:16.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.168+0000 7fecb1800640 1 --2- 172.21.15.67:0/2925574350 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fecac103130 0x7fecac19ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:16.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.168+0000 7fecb1800640 1 -- 172.21.15.67:0/2925574350 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecac19f3c0 con 0x7fecac101580 2024-03-28T16:03:16.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.169+0000 7fecb1800640 1 --2- 172.21.15.67:0/2925574350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fecac101580 0x7fecac1977a0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7feca000e9c0 tx=0x7feca000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:16.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.169+0000 7fecaa000640 1 -- 172.21.15.67:0/2925574350 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feca000cd80 con 0x7fecac101580 2024-03-28T16:03:16.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.169+0000 7fecaa000640 1 -- 172.21.15.67:0/2925574350 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7feca00035f0 con 0x7fecac101580 2024-03-28T16:03:16.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.169+0000 7fecaa000640 1 -- 172.21.15.67:0/2925574350 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feca0010690 con 0x7fecac101580 2024-03-28T16:03:16.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.169+0000 7fecb2c00640 1 -- 172.21.15.67:0/2925574350 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecac19f6b0 con 0x7fecac101580 2024-03-28T16:03:16.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.169+0000 7fecb2c00640 1 -- 172.21.15.67:0/2925574350 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecac19fbb0 con 0x7fecac101580 2024-03-28T16:03:16.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.173+0000 7fecb6e21740 1 Processor -- start 2024-03-28T16:03:16.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.173+0000 7fecb6e21740 1 -- start start 2024-03-28T16:03:16.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.173+0000 7fecb6e21740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f19d2f2d80 0x55f19d2f31a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:16.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.173+0000 7fecb6e21740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f19d2f3770 0x55f19d2f8270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:16.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.173+0000 7fecb6e21740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f19d2f88e0 0x55f19d2fad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:16.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.173+0000 7fecb6e21740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f19d257c90 con 0x55f19d2f3770 2024-03-28T16:03:16.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.173+0000 7fecb6e21740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f19d200980 con 0x55f19d2f2d80 2024-03-28T16:03:16.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.173+0000 7fecb6e21740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f19d253d30 con 0x55f19d2f88e0 2024-03-28T16:03:16.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.173+0000 7fecaa000640 1 -- 172.21.15.67:0/2925574350 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7feca00037a0 con 0x7fecac101580 2024-03-28T16:03:16.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.174+0000 7fecb1800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f19d2f3770 0x55f19d2f8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:16.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.174+0000 7fecb1800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f19d2f3770 0x55f19d2f8270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58098/0 (socket says 172.21.15.67:58098) 2024-03-28T16:03:16.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.174+0000 7fecb1800640 1 -- 172.21.15.67:0/362699069 learned_addr learned my addr 172.21.15.67:0/362699069 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:03:16.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.174+0000 7fecb2200640 1 --2- 172.21.15.67:0/362699069 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f19d2f2d80 0x55f19d2f31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:16.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.174+0000 7fecb0e00640 1 --2- 172.21.15.67:0/362699069 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f19d2f88e0 0x55f19d2fad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:16.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.174+0000 7fecb1800640 1 -- 172.21.15.67:0/362699069 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f19d2f88e0 msgr2=0x55f19d2fad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:03:16.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.174+0000 7fecb1800640 1 --2- 172.21.15.67:0/362699069 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f19d2f88e0 0x55f19d2fad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:16.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.174+0000 7fecb1800640 1 -- 172.21.15.67:0/362699069 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f19d2f2d80 msgr2=0x55f19d2f31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:03:16.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.174+0000 7fecb1800640 1 --2- 172.21.15.67:0/362699069 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f19d2f2d80 0x55f19d2f31a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:16.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.174+0000 7fecb1800640 1 -- 172.21.15.67:0/362699069 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f19d218510 con 0x55f19d2f3770 2024-03-28T16:03:16.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.174+0000 7fecb2200640 1 --2- 172.21.15.67:0/362699069 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f19d2f2d80 0x55f19d2f31a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T16:03:16.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.175+0000 7fecb1800640 1 --2- 172.21.15.67:0/362699069 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f19d2f3770 0x55f19d2f8270 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7feca00d2f80 tx=0x7feca00d7570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a011175d03f2ce8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:16.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.175+0000 7fec9ea00640 1 -- 172.21.15.67:0/362699069 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feca00e2070 con 0x55f19d2f3770 2024-03-28T16:03:16.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.175+0000 7fec9ea00640 1 -- 172.21.15.67:0/362699069 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7feca00d12d0 con 0x55f19d2f3770 2024-03-28T16:03:16.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.175+0000 7fec9ea00640 1 -- 172.21.15.67:0/362699069 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feca00dd800 con 0x55f19d2f3770 2024-03-28T16:03:16.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.175+0000 7fecaa000640 1 --2- 172.21.15.67:0/2925574350 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fec940a2880 0x7fec940a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:16.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.175+0000 7fecaa000640 1 -- 172.21.15.67:0/2925574350 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7feca00cf030 con 0x7fecac101580 2024-03-28T16:03:16.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.175+0000 7fecb2200640 1 --2- 172.21.15.67:0/2925574350 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fec940a2880 0x7fec940a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:16.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.176+0000 7fecb6e21740 1 -- 172.21.15.67:0/362699069 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f19d2f3770 msgr2=0x55f19d2f8270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:16.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.176+0000 7fecb6e21740 1 --2- 172.21.15.67:0/362699069 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f19d2f3770 0x55f19d2f8270 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7feca00d2f80 tx=0x7feca00d7570 comp rx=0 tx=0).stop 2024-03-28T16:03:16.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.176+0000 7fecb6e21740 1 -- 172.21.15.67:0/362699069 shutdown_connections 2024-03-28T16:03:16.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.176+0000 7fecb6e21740 1 --2- 172.21.15.67:0/362699069 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f19d2f88e0 0x55f19d2fad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:16.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.176+0000 7fecb6e21740 1 --2- 172.21.15.67:0/362699069 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f19d2f3770 0x55f19d2f8270 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:16.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.176+0000 7fecb6e21740 1 --2- 172.21.15.67:0/362699069 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f19d2f2d80 0x55f19d2f31a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:16.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.176+0000 7fecb6e21740 1 -- 172.21.15.67:0/362699069 >> 172.21.15.67:0/362699069 conn(0x55f19d2ee2f0 msgr2=0x55f19d2ee6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:16.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.176+0000 7fecb2200640 1 --2- 172.21.15.67:0/2925574350 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fec940a2880 0x7fec940a4d40 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7feca4009270 tx=0x7feca4009540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:16.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.176+0000 7fecb6e21740 1 -- 172.21.15.67:0/362699069 shutdown_connections 2024-03-28T16:03:16.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.176+0000 7fecb6e21740 1 -- 172.21.15.67:0/362699069 wait complete. 2024-03-28T16:03:16.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.177+0000 7fecb6e21740 1 -- start start 2024-03-28T16:03:16.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.177+0000 7fecb6e21740 1 Processor -- start 2024-03-28T16:03:16.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.177+0000 7fecb6e21740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f19d2f2d80 0x55f19d3016d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:16.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.177+0000 7fecb6e21740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f19d2f3770 0x55f19d301c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:16.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.177+0000 7fecb6e21740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f19d2f88e0 0x55f19d308c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:16.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.177+0000 7fecb6e21740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f19d2fb890 con 0x55f19d2f3770 2024-03-28T16:03:16.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.177+0000 7fecb6e21740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f19d213dd0 con 0x55f19d2f88e0 2024-03-28T16:03:16.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.177+0000 7fecb6e21740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f19d2fba10 con 0x55f19d2f2d80 2024-03-28T16:03:16.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.178+0000 7fecb6e21740 10 client.0 _ll_register_callbacks cb 0x55f19cf6bac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:03:16.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.178+0000 7fecb0e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f19d2f3770 0x55f19d301c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:16.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.178+0000 7fecb0e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f19d2f3770 0x55f19d301c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58108/0 (socket says 172.21.15.67:58108) 2024-03-28T16:03:16.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.178+0000 7fecb0e00640 1 -- 172.21.15.67:0/3879412721 learned_addr learned my addr 172.21.15.67:0/3879412721 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:03:16.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.178+0000 7fecb1800640 1 --2- 172.21.15.67:0/3879412721 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f19d2f2d80 0x55f19d3016d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:16.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.178+0000 7fecb0e00640 1 -- 172.21.15.67:0/3879412721 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f19d2f2d80 msgr2=0x55f19d3016d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:03:16.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.178+0000 7fecb0e00640 1 --2- 172.21.15.67:0/3879412721 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f19d2f2d80 0x55f19d3016d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:16.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.178+0000 7fecb0e00640 1 -- 172.21.15.67:0/3879412721 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f19d2f88e0 msgr2=0x55f19d308c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:03:16.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.178+0000 7fecb0e00640 1 --2- 172.21.15.67:0/3879412721 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f19d2f88e0 0x55f19d308c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:16.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.178+0000 7fecb0e00640 1 -- 172.21.15.67:0/3879412721 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f19cf6c2d0 con 0x55f19d2f3770 2024-03-28T16:03:16.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.178+0000 7fecb0e00640 1 --2- 172.21.15.67:0/3879412721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f19d2f3770 0x55f19d301c10 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fec98009900 tx=0x7fec98004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd0304d8ae5e4764 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:16.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.178+0000 7fec93400640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:03:16.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.179+0000 7fec93400640 1 -- 172.21.15.67:0/3879412721 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec98005d50 con 0x55f19d2f3770 2024-03-28T16:03:16.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.179+0000 7fec93400640 1 -- 172.21.15.67:0/3879412721 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fec98005030 con 0x55f19d2f3770 2024-03-28T16:03:16.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.179+0000 7fecb6e21740 1 -- 172.21.15.67:0/3879412721 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7feca00d12d0 con 0x55f19d2f3770 2024-03-28T16:03:16.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.179+0000 7fec93400640 1 -- 172.21.15.67:0/3879412721 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fec98050050 con 0x55f19d2f3770 2024-03-28T16:03:16.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.179+0000 7fecb6e21740 10 client.15168 fetch_fsmap learned FSMap version 19 2024-03-28T16:03:16.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.179+0000 7fecb6e21740 1 -- 172.21.15.67:0/3879412721 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55f19d0366b0 con 0x55f19d2f3770 2024-03-28T16:03:16.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.179+0000 7fec93400640 1 -- 172.21.15.67:0/3879412721 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7feca00d12d0 con 0x55f19d2f3770 2024-03-28T16:03:16.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.179+0000 7fec93400640 1 -- 172.21.15.67:0/3879412721 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fec980036f0 con 0x55f19d2f3770 2024-03-28T16:03:16.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.180+0000 7fecb6e21740 10 client.15168 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:03:16.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.180+0000 7fecb6e21740 10 client.15168 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T16:03:16.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.180+0000 7fecb6e21740 20 client.15168 populate_metadata read hostname 'smithi067' 2024-03-28T16:03:16.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.180+0000 7fecb6e21740 1 --2- 172.21.15.67:0/3879412721 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55f19d30df50 0x55f19d30cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:16.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.180+0000 7fecb6e21740 4 client.15168 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:03:16.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.180+0000 7fecb6e21740 1 -- 172.21.15.67:0/3879412721 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f19d1f9520 con 0x55f19d30df50 2024-03-28T16:03:16.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.180+0000 7fecb1800640 1 --2- 172.21.15.67:0/3879412721 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55f19d30df50 0x55f19d30cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:16.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.181+0000 7fecb1800640 1 --2- 172.21.15.67:0/3879412721 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55f19d30df50 0x55f19d30cf90 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ea88cacc4b54c5ad server_cookie=abd80e013ba2d6b in_seq=0 out_seq=0 2024-03-28T16:03:16.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.181+0000 7fec93400640 10 client.15168 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:03:16.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.183+0000 7fec93400640 1 -- 172.21.15.67:0/3879412721 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f19d1f9520 con 0x55f19d30df50 2024-03-28T16:03:16.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.183+0000 7fec93400640 10 client.15168 handle_command_reply: tid=0 2024-03-28T16:03:16.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.184+0000 7fecb6e21740 1 client.15168 shutdown 2024-03-28T16:03:16.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.184+0000 7fecb6e21740 1 -- 172.21.15.67:0/3879412721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f19d2f3770 msgr2=0x55f19d301c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:16.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.184+0000 7fecb6e21740 1 --2- 172.21.15.67:0/3879412721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f19d2f3770 0x55f19d301c10 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fec98009900 tx=0x7fec98004270 comp rx=0 tx=0).stop 2024-03-28T16:03:16.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.184+0000 7fecb6e21740 1 -- 172.21.15.67:0/3879412721 shutdown_connections 2024-03-28T16:03:16.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.184+0000 7fecb6e21740 1 --2- 172.21.15.67:0/3879412721 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55f19d30df50 0x55f19d30cf90 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:16.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.184+0000 7fecb6e21740 1 --2- 172.21.15.67:0/3879412721 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f19d2f88e0 0x55f19d308c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:16.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.184+0000 7fecb6e21740 1 --2- 172.21.15.67:0/3879412721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f19d2f3770 0x55f19d301c10 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:16.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.184+0000 7fecb6e21740 1 --2- 172.21.15.67:0/3879412721 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f19d2f2d80 0x55f19d3016d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:16.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.184+0000 7fec93400640 0 client.15168 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:03:16.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.184+0000 7fecb6e21740 1 -- 172.21.15.67:0/3879412721 >> 172.21.15.67:0/3879412721 conn(0x55f19d2ee2f0 msgr2=0x55f19d2ed4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:16.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.184+0000 7fecb6e21740 1 -- 172.21.15.67:0/3879412721 shutdown_connections 2024-03-28T16:03:16.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.184+0000 7fecb6e21740 1 -- 172.21.15.67:0/3879412721 wait complete. 2024-03-28T16:03:16.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.184+0000 7fecb6e21740 20 client.15168 trim_cache size 0 max 16384 2024-03-28T16:03:16.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.202+0000 7fecb6e21740 1 Processor -- start 2024-03-28T16:03:16.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.202+0000 7fecb6e21740 1 -- start start 2024-03-28T16:03:16.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.203+0000 7fecb6e21740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f19d2f2d80 0x55f19d2e91b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:16.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.203+0000 7fecb6e21740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f19d2f3770 0x55f19d308c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:16.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.203+0000 7fecb6e21740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f19d2f88e0 0x55f19d2facd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:16.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.203+0000 7fecb6e21740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f19d30d770 con 0x55f19d2f2d80 2024-03-28T16:03:16.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.203+0000 7fecb6e21740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f19d30ddd0 con 0x55f19d2f88e0 2024-03-28T16:03:16.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.203+0000 7fecb6e21740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f19d30e430 con 0x55f19d2f3770 2024-03-28T16:03:16.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.203+0000 7fecb1800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f19d2f2d80 0x55f19d2e91b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:16.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.203+0000 7fecb1800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f19d2f2d80 0x55f19d2e91b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58124/0 (socket says 172.21.15.67:58124) 2024-03-28T16:03:16.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.203+0000 7fecb1800640 1 -- 172.21.15.67:0/3676276364 learned_addr learned my addr 172.21.15.67:0/3676276364 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:03:16.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.203+0000 7fecb2200640 1 --2- 172.21.15.67:0/3676276364 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f19d2f88e0 0x55f19d2facd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:16.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.203+0000 7fecb0e00640 1 --2- 172.21.15.67:0/3676276364 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f19d2f3770 0x55f19d308c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:16.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.204+0000 7fecb1800640 1 -- 172.21.15.67:0/3676276364 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f19d2f3770 msgr2=0x55f19d308c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:03:16.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.204+0000 7fecb1800640 1 --2- 172.21.15.67:0/3676276364 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f19d2f3770 0x55f19d308c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:16.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.204+0000 7fecb1800640 1 -- 172.21.15.67:0/3676276364 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f19d2f88e0 msgr2=0x55f19d2facd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:03:16.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.204+0000 7fecb1800640 1 --2- 172.21.15.67:0/3676276364 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f19d2f88e0 0x55f19d2facd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:16.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.204+0000 7fecb1800640 1 -- 172.21.15.67:0/3676276364 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f19d2f8630 con 0x55f19d2f2d80 2024-03-28T16:03:16.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.204+0000 7fecb0e00640 1 --2- 172.21.15.67:0/3676276364 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f19d2f3770 0x55f19d308c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:03:16.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.204+0000 7fecb2200640 1 --2- 172.21.15.67:0/3676276364 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f19d2f88e0 0x55f19d2facd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:03:16.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.204+0000 7fecb1800640 1 --2- 172.21.15.67:0/3676276364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f19d2f2d80 0x55f19d2e91b0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7feca0098390 tx=0x7feca00cfd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55d8565084b5836c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:16.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.204+0000 7fec9e000640 1 -- 172.21.15.67:0/3676276364 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feca009b790 con 0x55f19d2f2d80 2024-03-28T16:03:16.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.204+0000 7fec9e000640 1 -- 172.21.15.67:0/3676276364 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7feca0003b10 con 0x55f19d2f2d80 2024-03-28T16:03:16.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.204+0000 7fec9e000640 1 -- 172.21.15.67:0/3676276364 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feca00d2ba0 con 0x55f19d2f2d80 2024-03-28T16:03:16.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.205+0000 7fecb6e21740 1 -- 172.21.15.67:0/3676276364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f19d2f2d80 msgr2=0x55f19d2e91b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:16.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.205+0000 7fecb6e21740 1 --2- 172.21.15.67:0/3676276364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f19d2f2d80 0x55f19d2e91b0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7feca0098390 tx=0x7feca00cfd10 comp rx=0 tx=0).stop 2024-03-28T16:03:16.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.205+0000 7fecb6e21740 1 -- 172.21.15.67:0/3676276364 shutdown_connections 2024-03-28T16:03:16.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.205+0000 7fecb6e21740 1 --2- 172.21.15.67:0/3676276364 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f19d2f88e0 0x55f19d2facd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:16.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.205+0000 7fecb6e21740 1 --2- 172.21.15.67:0/3676276364 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f19d2f3770 0x55f19d308c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:16.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.205+0000 7fecb6e21740 1 --2- 172.21.15.67:0/3676276364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f19d2f2d80 0x55f19d2e91b0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:16.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.205+0000 7fecb6e21740 1 -- 172.21.15.67:0/3676276364 >> 172.21.15.67:0/3676276364 conn(0x55f19d2ee2f0 msgr2=0x55f19d2f4210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:16.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.205+0000 7fecb6e21740 1 -- 172.21.15.67:0/3676276364 shutdown_connections 2024-03-28T16:03:16.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.205+0000 7fecb6e21740 1 -- 172.21.15.67:0/3676276364 wait complete. 2024-03-28T16:03:16.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.205+0000 7fecb6e21740 1 -- start start 2024-03-28T16:03:16.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.206+0000 7fecb6e21740 1 Processor -- start 2024-03-28T16:03:16.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.206+0000 7fecb6e21740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f19d2f2d80 0x55f19d304480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:16.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.206+0000 7fecb6e21740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f19d2f3770 0x55f19d3049c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:16.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.206+0000 7fecb6e21740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f19d2f88e0 0x55f19d3079e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:16.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.206+0000 7fecb6e21740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f19d30f0f0 con 0x55f19d2f3770 2024-03-28T16:03:16.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.206+0000 7fecb6e21740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f19d30ea90 con 0x55f19d2f88e0 2024-03-28T16:03:16.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.206+0000 7fecb6e21740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f19d30f750 con 0x55f19d2f2d80 2024-03-28T16:03:16.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.206+0000 7fecb6e21740 10 client.0 _ll_register_callbacks cb 0x7fec98009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:03:16.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.206+0000 7fecb0e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f19d2f3770 0x55f19d3049c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:16.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.206+0000 7fecb0e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f19d2f3770 0x55f19d3049c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58126/0 (socket says 172.21.15.67:58126) 2024-03-28T16:03:16.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.206+0000 7fecb0e00640 1 -- 172.21.15.67:0/352456654 learned_addr learned my addr 172.21.15.67:0/352456654 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:03:16.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.207+0000 7fecb1800640 1 --2- 172.21.15.67:0/352456654 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f19d2f2d80 0x55f19d304480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:16.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.207+0000 7fecb0e00640 1 -- 172.21.15.67:0/352456654 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f19d2f2d80 msgr2=0x55f19d304480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:03:16.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.207+0000 7fecb0e00640 1 --2- 172.21.15.67:0/352456654 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f19d2f2d80 0x55f19d304480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:16.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.207+0000 7fecb0e00640 1 -- 172.21.15.67:0/352456654 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f19d2f88e0 msgr2=0x55f19d3079e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:03:16.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.207+0000 7fecb0e00640 1 --2- 172.21.15.67:0/352456654 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f19d2f88e0 0x55f19d3079e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:16.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.207+0000 7fecb0e00640 1 -- 172.21.15.67:0/352456654 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f19d307fd0 con 0x55f19d2f3770 2024-03-28T16:03:16.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.207+0000 7fecb0e00640 1 --2- 172.21.15.67:0/352456654 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f19d2f3770 0x55f19d3049c0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fec98003b90 tx=0x7fec980058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e82f3d0d49fd4f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:16.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.207+0000 7fec93400640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:03:16.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.207+0000 7fec93400640 1 -- 172.21.15.67:0/352456654 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec98059060 con 0x55f19d2f3770 2024-03-28T16:03:16.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.207+0000 7fec93400640 1 -- 172.21.15.67:0/352456654 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fec980035f0 con 0x55f19d2f3770 2024-03-28T16:03:16.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.207+0000 7fecb6e21740 1 -- 172.21.15.67:0/352456654 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f19d2651d0 con 0x55f19d2f3770 2024-03-28T16:03:16.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.208+0000 7fec93400640 1 -- 172.21.15.67:0/352456654 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fec98000e60 con 0x55f19d2f3770 2024-03-28T16:03:16.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.208+0000 7fecb6e21740 10 client.15174 fetch_fsmap learned FSMap version 19 2024-03-28T16:03:16.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.208+0000 7fecb6e21740 1 -- 172.21.15.67:0/352456654 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55f19d369140 con 0x55f19d2f3770 2024-03-28T16:03:16.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.208+0000 7fec93400640 1 -- 172.21.15.67:0/352456654 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55f19d2651d0 con 0x55f19d2f3770 2024-03-28T16:03:16.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.208+0000 7fec93400640 1 -- 172.21.15.67:0/352456654 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fec98004e00 con 0x55f19d2f3770 2024-03-28T16:03:16.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.208+0000 7fecb6e21740 10 client.15174 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:03:16.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.208+0000 7fecb6e21740 10 client.15174 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T16:03:16.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.208+0000 7fecb6e21740 20 client.15174 populate_metadata read hostname 'smithi067' 2024-03-28T16:03:16.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.208+0000 7fecb6e21740 1 --2- 172.21.15.67:0/352456654 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55f19d36d7b0 0x55f19d369370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:16.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.208+0000 7fecb6e21740 4 client.15174 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T16:03:16.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.208+0000 7fecb6e21740 1 -- 172.21.15.67:0/352456654 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55f19d25f520 con 0x55f19d36d7b0 2024-03-28T16:03:16.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.209+0000 7fecb1800640 1 --2- 172.21.15.67:0/352456654 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55f19d36d7b0 0x55f19d369370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:16.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.210+0000 7fecb1800640 1 --2- 172.21.15.67:0/352456654 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55f19d36d7b0 0x55f19d369370 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c43789de239dcdee server_cookie=45d21e61b449fae6 in_seq=0 out_seq=0 2024-03-28T16:03:16.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.210+0000 7fec93400640 10 client.15174 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:03:16.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.210+0000 7fec93400640 1 -- 172.21.15.67:0/352456654 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x55f19d25f520 con 0x55f19d36d7b0 2024-03-28T16:03:16.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.210+0000 7fec93400640 10 client.15174 handle_command_reply: tid=0 2024-03-28T16:03:16.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.210+0000 7fecb6e21740 1 client.15174 shutdown 2024-03-28T16:03:16.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.211+0000 7fecb6e21740 1 -- 172.21.15.67:0/352456654 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f19d2f3770 msgr2=0x55f19d3049c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:16.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.211+0000 7fecb6e21740 1 --2- 172.21.15.67:0/352456654 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f19d2f3770 0x55f19d3049c0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fec98003b90 tx=0x7fec980058d0 comp rx=0 tx=0).stop 2024-03-28T16:03:16.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.211+0000 7fecb6e21740 1 -- 172.21.15.67:0/352456654 shutdown_connections 2024-03-28T16:03:16.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.211+0000 7fecb6e21740 1 --2- 172.21.15.67:0/352456654 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55f19d36d7b0 0x55f19d369370 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:16.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.211+0000 7fecb6e21740 1 --2- 172.21.15.67:0/352456654 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f19d2f88e0 0x55f19d3079e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:16.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.211+0000 7fecb6e21740 1 --2- 172.21.15.67:0/352456654 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f19d2f3770 0x55f19d3049c0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:16.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.211+0000 7fecb6e21740 1 --2- 172.21.15.67:0/352456654 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f19d2f2d80 0x55f19d304480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:16.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.211+0000 7fec93400640 0 client.15174 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:03:16.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.211+0000 7fecb6e21740 1 -- 172.21.15.67:0/352456654 >> 172.21.15.67:0/352456654 conn(0x55f19d2ee2f0 msgr2=0x55f19d2fbde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:16.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.211+0000 7fecb6e21740 1 -- 172.21.15.67:0/352456654 shutdown_connections 2024-03-28T16:03:16.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.211+0000 7fecb6e21740 1 -- 172.21.15.67:0/352456654 wait complete. 2024-03-28T16:03:16.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.211+0000 7fecb6e21740 20 client.15174 trim_cache size 0 max 16384 2024-03-28T16:03:16.228 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-28T16:03:16.228 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "b5f8225a-ed1a-11ee-95d0-87774f69a715", 2024-03-28T16:03:16.228 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 0, 2024-03-28T16:03:16.228 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 34508, 2024-03-28T16:03:16.228 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-03-28T16:03:16.228 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-03-28T16:03:16.228 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-03-28T16:03:16.228 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 127.94059605299999, 2024-03-28T16:03:16.228 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 19, 2024-03-28T16:03:16.229 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 92, 2024-03-28T16:03:16.229 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 90, 2024-03-28T16:03:16.229 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 134.46969530999999 2024-03-28T16:03:16.229 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-28T16:03:16.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.212+0000 7fecb2c00640 1 -- 172.21.15.67:0/2925574350 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fec940a2880 msgr2=0x7fec940a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:16.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.212+0000 7fecb2c00640 1 --2- 172.21.15.67:0/2925574350 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fec940a2880 0x7fec940a4d40 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7feca4009270 tx=0x7feca4009540 comp rx=0 tx=0).stop 2024-03-28T16:03:16.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.212+0000 7fecb2c00640 1 -- 172.21.15.67:0/2925574350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fecac101580 msgr2=0x7fecac1977a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:16.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.212+0000 7fecb2c00640 1 --2- 172.21.15.67:0/2925574350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fecac101580 0x7fecac1977a0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7feca000e9c0 tx=0x7feca000ee80 comp rx=0 tx=0).stop 2024-03-28T16:03:16.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.212+0000 7fecb2c00640 1 -- 172.21.15.67:0/2925574350 shutdown_connections 2024-03-28T16:03:16.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.212+0000 7fecb2c00640 1 --2- 172.21.15.67:0/2925574350 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fec940a2880 0x7fec940a4d40 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:16.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.212+0000 7fecb2c00640 1 --2- 172.21.15.67:0/2925574350 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fecac103130 0x7fecac19ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:16.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.212+0000 7fecb2c00640 1 --2- 172.21.15.67:0/2925574350 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fecac102770 0x7fecac197ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:16.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.212+0000 7fecb2c00640 1 --2- 172.21.15.67:0/2925574350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fecac101580 0x7fecac1977a0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:16.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.212+0000 7fecb2c00640 1 -- 172.21.15.67:0/2925574350 >> 172.21.15.67:0/2925574350 conn(0x7fecac0fcb10 msgr2=0x7fecac105ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:16.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.212+0000 7fecb2c00640 1 -- 172.21.15.67:0/2925574350 shutdown_connections 2024-03-28T16:03:16.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:16.212+0000 7fecb2c00640 1 -- 172.21.15.67:0/2925574350 wait complete. 2024-03-28T16:03:16.231 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-03-28T16:03:16.272 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-03-28T16:03:16.273 INFO:tasks.ceph_fuse:config is {'client.0': {}} 2024-03-28T16:03:16.273 INFO:tasks.ceph_fuse:client.0 config is {'mount_subvol_num': 0} 2024-03-28T16:03:16.273 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-28T16:03:16.273 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-28T16:03:16.273 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T16:03:16.273 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-03-28T16:03:16.332 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T16:03:16.332 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link delete ceph-brx 2024-03-28T16:03:16.430 INFO:teuthology.orchestra.run.smithi067.stderr:Cannot find device "ceph-brx" 2024-03-28T16:03:16.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T16:03:16.434 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-03-28T16:03:16.435 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 -- ceph fs ls 2024-03-28T16:03:16.898 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T16:03:17.050 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:16 smithi067 ceph-mon[31707]: pgmap v430: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:03:17.050 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:16 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/2510518779' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:03:17.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:16 smithi150 ceph-mon[34825]: pgmap v430: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:03:17.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:16 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/2510518779' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:03:17.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:16 smithi183 ceph-mon[35580]: pgmap v430: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:03:17.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:16 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/2510518779' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:03:18.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:18.955+0000 7fb428e00700 1 -- 172.21.15.67:0/564899010 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb424107080 msgr2=0x7fb424069e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:18.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:18.955+0000 7fb428e00700 1 --2- 172.21.15.67:0/564899010 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb424107080 0x7fb424069e40 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fb414009990 tx=0x7fb414009ca0 comp rx=0 tx=0).stop 2024-03-28T16:03:18.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:18.955+0000 7fb428e00700 1 -- 172.21.15.67:0/564899010 shutdown_connections 2024-03-28T16:03:18.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:18.955+0000 7fb428e00700 1 --2- 172.21.15.67:0/564899010 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb424104b90 0x7fb424105020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:18.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:18.955+0000 7fb428e00700 1 --2- 172.21.15.67:0/564899010 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb424107080 0x7fb424069e40 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:18.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:18.955+0000 7fb428e00700 1 --2- 172.21.15.67:0/564899010 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb42406a380 0x7fb42406a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:18.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:18.955+0000 7fb428e00700 1 -- 172.21.15.67:0/564899010 >> 172.21.15.67:0/564899010 conn(0x7fb424077fc0 msgr2=0x7fb4240783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:18.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:18.955+0000 7fb428e00700 1 -- 172.21.15.67:0/564899010 shutdown_connections 2024-03-28T16:03:18.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:18.955+0000 7fb428e00700 1 -- 172.21.15.67:0/564899010 wait complete. 2024-03-28T16:03:18.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:18.956+0000 7fb428e00700 1 Processor -- start 2024-03-28T16:03:18.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:18.957+0000 7fb428e00700 1 -- start start 2024-03-28T16:03:18.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:18.957+0000 7fb428e00700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb42406a380 0x7fb42419ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:18.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:18.957+0000 7fb428e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb424104b90 0x7fb42419f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:18.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:18.957+0000 7fb428e00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb424107080 0x7fb4241a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:18.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:18.957+0000 7fb428e00700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb424118910 con 0x7fb424107080 2024-03-28T16:03:18.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:18.957+0000 7fb428e00700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb424118790 con 0x7fb42406a380 2024-03-28T16:03:18.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:18.957+0000 7fb428e00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb424118a90 con 0x7fb424104b90 2024-03-28T16:03:18.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:18.958+0000 7fb422800700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb424107080 0x7fb4241a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:18.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:18.958+0000 7fb422800700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb424107080 0x7fb4241a3710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58146/0 (socket says 172.21.15.67:58146) 2024-03-28T16:03:18.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:18.958+0000 7fb422800700 1 -- 172.21.15.67:0/1509895712 learned_addr learned my addr 172.21.15.67:0/1509895712 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:03:18.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:18.958+0000 7fb421e00700 1 --2- 172.21.15.67:0/1509895712 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb42406a380 0x7fb42419ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:18.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:18.958+0000 7fb419400700 1 --2- 172.21.15.67:0/1509895712 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb424104b90 0x7fb42419f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:18.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:18.958+0000 7fb422800700 1 -- 172.21.15.67:0/1509895712 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb424104b90 msgr2=0x7fb42419f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:18.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:18.958+0000 7fb422800700 1 --2- 172.21.15.67:0/1509895712 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb424104b90 0x7fb42419f380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:18.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:18.958+0000 7fb422800700 1 -- 172.21.15.67:0/1509895712 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb42406a380 msgr2=0x7fb42419ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:18.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:18.958+0000 7fb422800700 1 --2- 172.21.15.67:0/1509895712 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb42406a380 0x7fb42419ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:18.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:18.958+0000 7fb422800700 1 -- 172.21.15.67:0/1509895712 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4241a3e50 con 0x7fb424107080 2024-03-28T16:03:18.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:18.958+0000 7fb421e00700 1 --2- 172.21.15.67:0/1509895712 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb42406a380 0x7fb42419ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T16:03:18.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:18.959+0000 7fb422800700 1 --2- 172.21.15.67:0/1509895712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb424107080 0x7fb4241a3710 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fb41c007ee0 tx=0x7fb41c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:18.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:18.959+0000 7fb41b400700 1 -- 172.21.15.67:0/1509895712 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb41c00dbd0 con 0x7fb424107080 2024-03-28T16:03:18.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:18.959+0000 7fb41b400700 1 -- 172.21.15.67:0/1509895712 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb41c00dd70 con 0x7fb424107080 2024-03-28T16:03:18.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:18.959+0000 7fb41b400700 1 -- 172.21.15.67:0/1509895712 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb41c005a20 con 0x7fb424107080 2024-03-28T16:03:18.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:18.959+0000 7fb428e00700 1 -- 172.21.15.67:0/1509895712 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4241a4140 con 0x7fb424107080 2024-03-28T16:03:18.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:18.959+0000 7fb428e00700 1 -- 172.21.15.67:0/1509895712 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4241ab9c0 con 0x7fb424107080 2024-03-28T16:03:18.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:18.960+0000 7fb428e00700 1 -- 172.21.15.67:0/1509895712 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb424068e50 con 0x7fb424107080 2024-03-28T16:03:18.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:18.963+0000 7fb41b400700 1 -- 172.21.15.67:0/1509895712 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fb41c01a020 con 0x7fb424107080 2024-03-28T16:03:18.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:18.964+0000 7fb41b400700 1 --2- 172.21.15.67:0/1509895712 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb4080a2910 0x7fb4080a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:18.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:18.964+0000 7fb41b400700 1 -- 172.21.15.67:0/1509895712 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fb41c0ce1f0 con 0x7fb424107080 2024-03-28T16:03:18.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:18.964+0000 7fb421e00700 1 --2- 172.21.15.67:0/1509895712 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb4080a2910 0x7fb4080a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:18.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:18.965+0000 7fb421e00700 1 --2- 172.21.15.67:0/1509895712 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb4080a2910 0x7fb4080a4dd0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fb414006010 tx=0x7fb41401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:18.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:18.969+0000 7fb41b400700 1 -- 172.21.15.67:0/1509895712 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb41c010070 con 0x7fb424107080 2024-03-28T16:03:19.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:18 smithi150 ceph-mon[34825]: pgmap v431: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T16:03:19.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:18 smithi067 ceph-mon[31707]: pgmap v431: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T16:03:19.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:18 smithi183 ceph-mon[35580]: pgmap v431: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T16:03:19.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:19.248+0000 7fb428e00700 1 -- 172.21.15.67:0/1509895712 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fb4241abc30 con 0x7fb424107080 2024-03-28T16:03:19.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:19.249+0000 7fb41b400700 1 -- 172.21.15.67:0/1509895712 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v19) v1 ==== 53+0+73 (secure 0 0 0) 0x7fb41c09b240 con 0x7fb424107080 2024-03-28T16:03:19.251 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-28T16:03:19.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:19.252+0000 7fb428e00700 1 -- 172.21.15.67:0/1509895712 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb4080a2910 msgr2=0x7fb4080a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:19.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:19.252+0000 7fb428e00700 1 --2- 172.21.15.67:0/1509895712 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb4080a2910 0x7fb4080a4dd0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fb414006010 tx=0x7fb41401a040 comp rx=0 tx=0).stop 2024-03-28T16:03:19.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:19.252+0000 7fb428e00700 1 -- 172.21.15.67:0/1509895712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb424107080 msgr2=0x7fb4241a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:19.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:19.252+0000 7fb428e00700 1 --2- 172.21.15.67:0/1509895712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb424107080 0x7fb4241a3710 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fb41c007ee0 tx=0x7fb41c00d3b0 comp rx=0 tx=0).stop 2024-03-28T16:03:19.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:19.253+0000 7fb428e00700 1 -- 172.21.15.67:0/1509895712 shutdown_connections 2024-03-28T16:03:19.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:19.253+0000 7fb428e00700 1 --2- 172.21.15.67:0/1509895712 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb4080a2910 0x7fb4080a4dd0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:19.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:19.253+0000 7fb428e00700 1 --2- 172.21.15.67:0/1509895712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb424107080 0x7fb4241a3710 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:19.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:19.253+0000 7fb428e00700 1 --2- 172.21.15.67:0/1509895712 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb42406a380 0x7fb42419ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:19.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:19.253+0000 7fb428e00700 1 --2- 172.21.15.67:0/1509895712 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb424104b90 0x7fb42419f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:19.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:19.253+0000 7fb428e00700 1 -- 172.21.15.67:0/1509895712 >> 172.21.15.67:0/1509895712 conn(0x7fb424077fc0 msgr2=0x7fb424100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:19.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:19.253+0000 7fb428e00700 1 -- 172.21.15.67:0/1509895712 shutdown_connections 2024-03-28T16:03:19.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:19.253+0000 7fb428e00700 1 -- 172.21.15.67:0/1509895712 wait complete. 2024-03-28T16:03:19.952 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:19 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/1509895712' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-28T16:03:20.024 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-28T16:03:20.024 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-28T16:03:20.024 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi067.front.sepia.ceph.com 2024-03-28T16:03:20.024 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-28T16:03:20.024 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-28T16:03:20.025 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-28T16:03:20.025 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-28T16:03:20.025 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-28T16:03:20.025 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-28T16:03:20.025 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T16:03:20.026 DEBUG:teuthology.orchestra.run.smithi067:> ip addr 2024-03-28T16:03:20.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:19 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/1509895712' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-28T16:03:20.089 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-28T16:03:20.089 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-28T16:03:20.089 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-03-28T16:03:20.089 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-28T16:03:20.090 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-03-28T16:03:20.090 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-28T16:03:20.090 INFO:teuthology.orchestra.run.smithi067.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-28T16:03:20.090 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7e brd ff:ff:ff:ff:ff:ff 2024-03-28T16:03:20.090 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f0 2024-03-28T16:03:20.090 INFO:teuthology.orchestra.run.smithi067.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-28T16:03:20.090 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7f brd ff:ff:ff:ff:ff:ff 2024-03-28T16:03:20.090 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f1 2024-03-28T16:03:20.090 INFO:teuthology.orchestra.run.smithi067.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-28T16:03:20.090 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0a brd ff:ff:ff:ff:ff:ff 2024-03-28T16:03:20.091 INFO:teuthology.orchestra.run.smithi067.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-28T16:03:20.091 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0b brd ff:ff:ff:ff:ff:ff 2024-03-28T16:03:20.091 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.21.15.67/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-28T16:03:20.091 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft 41750sec preferred_lft 41750sec 2024-03-28T16:03:20.091 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::ec4:7aff:feb6:fc0b/64 scope link 2024-03-28T16:03:20.091 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-03-28T16:03:20.093 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-28T16:03:20.093 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T16:03:20.093 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-03-28T16:03:20.093 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-03-28T16:03:20.093 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link add name ceph-brx type bridge 2024-03-28T16:03:20.093 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip addr flush dev ceph-brx 2024-03-28T16:03:20.094 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set ceph-brx up 2024-03-28T16:03:20.094 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-28T16:03:20.094 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-03-28T16:03:20.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:19 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/1509895712' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-28T16:03:20.555 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T16:03:20.556 DEBUG:teuthology.orchestra.run.smithi067:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-28T16:03:20.641 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-03-28T16:03:20.645 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T16:03:20.645 DEBUG:teuthology.orchestra.run.smithi067:> route 2024-03-28T16:03:20.714 INFO:teuthology.orchestra.run.smithi067.stdout:Kernel IP routing table 2024-03-28T16:03:20.714 INFO:teuthology.orchestra.run.smithi067.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-28T16:03:20.714 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-28T16:03:20.714 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-28T16:03:20.714 INFO:teuthology.orchestra.run.smithi067.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-28T16:03:20.716 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T16:03:20.716 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-03-28T16:03:20.716 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-03-28T16:03:20.716 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-28T16:03:20.716 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-28T16:03:20.716 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-28T16:03:20.716 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-03-28T16:03:21.044 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:20 smithi067 ceph-mon[31707]: pgmap v432: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T16:03:21.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:20 smithi150 ceph-mon[34825]: pgmap v432: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T16:03:21.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:20 smithi183 ceph-mon[35580]: pgmap v432: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T16:03:21.440 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T16:03:21.441 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-03-28T16:03:21.500 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T16:03:21.501 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list-id 2024-03-28T16:03:21.556 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T16:03:21.557 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-03-28T16:03:21.557 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-03-28T16:03:21.557 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-28T16:03:21.557 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-28T16:03:21.557 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-03-28T16:03:21.709 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-28T16:03:21.709 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T16:03:21.709 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-03-28T16:03:21.709 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-03-28T16:03:21.709 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-28T16:03:21.710 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-28T16:03:21.710 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-28T16:03:21.710 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-28T16:03:21.710 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-28T16:03:21.710 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-03-28T16:03:22.018 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T16:03:22.019 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-03-28T16:03:22.019 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-03-28T16:03:22.019 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set brx.0 up 2024-03-28T16:03:22.019 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set dev brx.0 master ceph-brx 2024-03-28T16:03:22.019 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-03-28T16:03:22.194 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {'mount_subvol_num': 0} 2024-03-28T16:03:22.194 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-28T16:03:22.194 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-28T16:03:22.248 INFO:teuthology.orchestra.run.smithi067.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-28T16:03:22.248 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-28T16:03:22.248 DEBUG:teuthology.orchestra.run.smithi067:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-28T16:03:22.299 DEBUG:teuthology.orchestra.run.smithi067:> sudo modprobe fuse 2024-03-28T16:03:22.406 DEBUG:teuthology.orchestra.run.smithi067:> cat /proc/self/mounts | awk '{print $2}' 2024-03-28T16:03:22.456 INFO:teuthology.orchestra.run.smithi067.stdout:/proc 2024-03-28T16:03:22.456 INFO:teuthology.orchestra.run.smithi067.stdout:/sys 2024-03-28T16:03:22.457 INFO:teuthology.orchestra.run.smithi067.stdout:/dev 2024-03-28T16:03:22.457 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/kernel/security 2024-03-28T16:03:22.457 INFO:teuthology.orchestra.run.smithi067.stdout:/dev/shm 2024-03-28T16:03:22.457 INFO:teuthology.orchestra.run.smithi067.stdout:/dev/pts 2024-03-28T16:03:22.457 INFO:teuthology.orchestra.run.smithi067.stdout:/run 2024-03-28T16:03:22.457 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/fs/cgroup 2024-03-28T16:03:22.457 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/fs/pstore 2024-03-28T16:03:22.457 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/fs/bpf 2024-03-28T16:03:22.457 INFO:teuthology.orchestra.run.smithi067.stdout:/ 2024-03-28T16:03:22.457 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/fs/selinux 2024-03-28T16:03:22.457 INFO:teuthology.orchestra.run.smithi067.stdout:/proc/sys/fs/binfmt_misc 2024-03-28T16:03:22.458 INFO:teuthology.orchestra.run.smithi067.stdout:/dev/hugepages 2024-03-28T16:03:22.458 INFO:teuthology.orchestra.run.smithi067.stdout:/dev/mqueue 2024-03-28T16:03:22.458 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/kernel/debug 2024-03-28T16:03:22.458 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/kernel/tracing 2024-03-28T16:03:22.458 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/fs/fuse/connections 2024-03-28T16:03:22.458 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/kernel/config 2024-03-28T16:03:22.458 INFO:teuthology.orchestra.run.smithi067.stdout:/run/credentials/systemd-sysctl.service 2024-03-28T16:03:22.458 INFO:teuthology.orchestra.run.smithi067.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-28T16:03:22.458 INFO:teuthology.orchestra.run.smithi067.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-28T16:03:22.459 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/nfs/rpc_pipefs 2024-03-28T16:03:22.459 INFO:teuthology.orchestra.run.smithi067.stdout:/run/user/1000 2024-03-28T16:03:22.459 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/ceph 2024-03-28T16:03:22.459 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/containers/storage/overlay 2024-03-28T16:03:22.459 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/containers/storage/overlay/6a0551a97504bc5b8894f068a898cd13c5b1bcd841c6c4783dddb7ebd0896a1a/merged 2024-03-28T16:03:22.459 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/containers/storage/overlay/d43d86446fd60321247fb6c5ee6b0e5537b03fe91ee5e21635a12d11f290d0c2/merged 2024-03-28T16:03:22.459 INFO:teuthology.orchestra.run.smithi067.stdout:/run/user/0 2024-03-28T16:03:22.459 INFO:teuthology.orchestra.run.smithi067.stdout:/proc/sys/fs/binfmt_misc 2024-03-28T16:03:22.459 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/containers/storage/overlay/30d894f99bed72a32b984503db6cc28f538f7c49fc4c773c383af2cd57bf11fb/merged 2024-03-28T16:03:22.459 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/containers/storage/overlay/1a6a4bff1c35300258cf6439198751bb695a9c5628c713d48a4d9fbea7e23e3e/merged 2024-03-28T16:03:22.459 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/containers/storage/overlay/39ef9de1a3f1e636ffad421832a862f6031bdf7eb717dc017b8419772e8f644f/merged 2024-03-28T16:03:22.459 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/containers/storage/overlay/45b9fdca53df002ccca6e6256a42bd3154a6624370dbf5300c65c383339bb6c5/merged 2024-03-28T16:03:22.459 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/containers/storage/overlay/8169ebff8d96e64432f2137fc3ca2bf8aecb1d8959c67695fcfe7b4e4bbcb20a/merged 2024-03-28T16:03:22.459 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/containers/storage/overlay/4c08a654a70e2302ed4f1c409260eea1d3d8f317c321c0ff2d1c98799dc3c39f/merged 2024-03-28T16:03:22.460 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/containers/storage/overlay/240404dcaa83ad223fd4deb7a8c73b66b8c8ec096c93fc0d1f3b19893df4822a/merged 2024-03-28T16:03:22.460 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/containers/storage/overlay/967eee94b0fc4339bd1ab618c160bf2ecff2a3c7e1ed140771551ffcdb267c93/merged 2024-03-28T16:03:22.460 INFO:teuthology.orchestra.run.smithi067.stdout:/run/netns 2024-03-28T16:03:22.460 INFO:teuthology.orchestra.run.smithi067.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-28T16:03:22.460 INFO:teuthology.orchestra.run.smithi067.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-28T16:03:22.461 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T16:03:22.461 DEBUG:teuthology.orchestra.run.smithi067:> ls /sys/fs/fuse/connections 2024-03-28T16:03:22.505 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-28T16:03:22.506 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0 --client_mountpoint=/volumes/qa/sv_0/7ad6a264-caac-444d-a6a0-0c4d0503695c --client_fs=cephfs) 2024-03-28T16:03:22.552 DEBUG:teuthology.orchestra.run.smithi067:> sudo modprobe fuse 2024-03-28T16:03:22.632 DEBUG:teuthology.orchestra.run.smithi067:> cat /proc/self/mounts | awk '{print $2}' 2024-03-28T16:03:22.677 INFO:teuthology.orchestra.run.smithi067.stdout:/proc 2024-03-28T16:03:22.677 INFO:teuthology.orchestra.run.smithi067.stdout:/sys 2024-03-28T16:03:22.677 INFO:teuthology.orchestra.run.smithi067.stdout:/dev 2024-03-28T16:03:22.677 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/kernel/security 2024-03-28T16:03:22.678 INFO:teuthology.orchestra.run.smithi067.stdout:/dev/shm 2024-03-28T16:03:22.678 INFO:teuthology.orchestra.run.smithi067.stdout:/dev/pts 2024-03-28T16:03:22.678 INFO:teuthology.orchestra.run.smithi067.stdout:/run 2024-03-28T16:03:22.678 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/fs/cgroup 2024-03-28T16:03:22.678 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/fs/pstore 2024-03-28T16:03:22.678 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/fs/bpf 2024-03-28T16:03:22.678 INFO:teuthology.orchestra.run.smithi067.stdout:/ 2024-03-28T16:03:22.678 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/fs/selinux 2024-03-28T16:03:22.678 INFO:teuthology.orchestra.run.smithi067.stdout:/proc/sys/fs/binfmt_misc 2024-03-28T16:03:22.678 INFO:teuthology.orchestra.run.smithi067.stdout:/dev/hugepages 2024-03-28T16:03:22.678 INFO:teuthology.orchestra.run.smithi067.stdout:/dev/mqueue 2024-03-28T16:03:22.678 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/kernel/debug 2024-03-28T16:03:22.678 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/kernel/tracing 2024-03-28T16:03:22.678 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/fs/fuse/connections 2024-03-28T16:03:22.678 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/kernel/config 2024-03-28T16:03:22.679 INFO:teuthology.orchestra.run.smithi067.stdout:/run/credentials/systemd-sysctl.service 2024-03-28T16:03:22.679 INFO:teuthology.orchestra.run.smithi067.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-28T16:03:22.679 INFO:teuthology.orchestra.run.smithi067.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-28T16:03:22.679 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/nfs/rpc_pipefs 2024-03-28T16:03:22.679 INFO:teuthology.orchestra.run.smithi067.stdout:/run/user/1000 2024-03-28T16:03:22.679 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/ceph 2024-03-28T16:03:22.679 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/containers/storage/overlay 2024-03-28T16:03:22.679 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/containers/storage/overlay/6a0551a97504bc5b8894f068a898cd13c5b1bcd841c6c4783dddb7ebd0896a1a/merged 2024-03-28T16:03:22.679 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/containers/storage/overlay/d43d86446fd60321247fb6c5ee6b0e5537b03fe91ee5e21635a12d11f290d0c2/merged 2024-03-28T16:03:22.679 INFO:teuthology.orchestra.run.smithi067.stdout:/run/user/0 2024-03-28T16:03:22.679 INFO:teuthology.orchestra.run.smithi067.stdout:/proc/sys/fs/binfmt_misc 2024-03-28T16:03:22.679 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/containers/storage/overlay/30d894f99bed72a32b984503db6cc28f538f7c49fc4c773c383af2cd57bf11fb/merged 2024-03-28T16:03:22.679 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/containers/storage/overlay/1a6a4bff1c35300258cf6439198751bb695a9c5628c713d48a4d9fbea7e23e3e/merged 2024-03-28T16:03:22.679 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/containers/storage/overlay/39ef9de1a3f1e636ffad421832a862f6031bdf7eb717dc017b8419772e8f644f/merged 2024-03-28T16:03:22.680 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/containers/storage/overlay/45b9fdca53df002ccca6e6256a42bd3154a6624370dbf5300c65c383339bb6c5/merged 2024-03-28T16:03:22.680 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/containers/storage/overlay/8169ebff8d96e64432f2137fc3ca2bf8aecb1d8959c67695fcfe7b4e4bbcb20a/merged 2024-03-28T16:03:22.680 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/containers/storage/overlay/4c08a654a70e2302ed4f1c409260eea1d3d8f317c321c0ff2d1c98799dc3c39f/merged 2024-03-28T16:03:22.680 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/containers/storage/overlay/240404dcaa83ad223fd4deb7a8c73b66b8c8ec096c93fc0d1f3b19893df4822a/merged 2024-03-28T16:03:22.680 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/containers/storage/overlay/967eee94b0fc4339bd1ab618c160bf2ecff2a3c7e1ed140771551ffcdb267c93/merged 2024-03-28T16:03:22.680 INFO:teuthology.orchestra.run.smithi067.stdout:/run/netns 2024-03-28T16:03:22.680 INFO:teuthology.orchestra.run.smithi067.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-28T16:03:22.680 INFO:teuthology.orchestra.run.smithi067.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-28T16:03:22.681 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T16:03:22.681 DEBUG:teuthology.orchestra.run.smithi067:> ls /sys/fs/fuse/connections 2024-03-28T16:03:22.764 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi067.stderr:2024-03-28T16:03:22.762+0000 7fd41b235480 -1 init, newargv = 0x557dcbd07fb0 newargc=15 2024-03-28T16:03:22.764 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi067.stderr:ceph-fuse[83688]: starting ceph client 2024-03-28T16:03:22.786 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi067.stderr:ceph-fuse[83688]: starting fuse 2024-03-28T16:03:23.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:22 smithi150 ceph-mon[34825]: pgmap v433: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:03:23.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:22 smithi067 ceph-mon[31707]: pgmap v433: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:03:23.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:22 smithi183 ceph-mon[35580]: pgmap v433: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:03:23.872 DEBUG:teuthology.orchestra.run.smithi067:> sudo modprobe fuse 2024-03-28T16:03:23.966 DEBUG:teuthology.orchestra.run.smithi067:> cat /proc/self/mounts | awk '{print $2}' 2024-03-28T16:03:24.017 INFO:teuthology.orchestra.run.smithi067.stdout:/proc 2024-03-28T16:03:24.017 INFO:teuthology.orchestra.run.smithi067.stdout:/sys 2024-03-28T16:03:24.017 INFO:teuthology.orchestra.run.smithi067.stdout:/dev 2024-03-28T16:03:24.017 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/kernel/security 2024-03-28T16:03:24.017 INFO:teuthology.orchestra.run.smithi067.stdout:/dev/shm 2024-03-28T16:03:24.017 INFO:teuthology.orchestra.run.smithi067.stdout:/dev/pts 2024-03-28T16:03:24.017 INFO:teuthology.orchestra.run.smithi067.stdout:/run 2024-03-28T16:03:24.017 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/fs/cgroup 2024-03-28T16:03:24.018 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/fs/pstore 2024-03-28T16:03:24.018 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/fs/bpf 2024-03-28T16:03:24.018 INFO:teuthology.orchestra.run.smithi067.stdout:/ 2024-03-28T16:03:24.018 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/fs/selinux 2024-03-28T16:03:24.018 INFO:teuthology.orchestra.run.smithi067.stdout:/proc/sys/fs/binfmt_misc 2024-03-28T16:03:24.018 INFO:teuthology.orchestra.run.smithi067.stdout:/dev/hugepages 2024-03-28T16:03:24.018 INFO:teuthology.orchestra.run.smithi067.stdout:/dev/mqueue 2024-03-28T16:03:24.018 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/kernel/debug 2024-03-28T16:03:24.018 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/kernel/tracing 2024-03-28T16:03:24.018 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/fs/fuse/connections 2024-03-28T16:03:24.018 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/kernel/config 2024-03-28T16:03:24.018 INFO:teuthology.orchestra.run.smithi067.stdout:/run/credentials/systemd-sysctl.service 2024-03-28T16:03:24.018 INFO:teuthology.orchestra.run.smithi067.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-28T16:03:24.018 INFO:teuthology.orchestra.run.smithi067.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-28T16:03:24.019 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/nfs/rpc_pipefs 2024-03-28T16:03:24.019 INFO:teuthology.orchestra.run.smithi067.stdout:/run/user/1000 2024-03-28T16:03:24.019 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/ceph 2024-03-28T16:03:24.019 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/containers/storage/overlay 2024-03-28T16:03:24.019 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/containers/storage/overlay/6a0551a97504bc5b8894f068a898cd13c5b1bcd841c6c4783dddb7ebd0896a1a/merged 2024-03-28T16:03:24.019 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/containers/storage/overlay/d43d86446fd60321247fb6c5ee6b0e5537b03fe91ee5e21635a12d11f290d0c2/merged 2024-03-28T16:03:24.019 INFO:teuthology.orchestra.run.smithi067.stdout:/run/user/0 2024-03-28T16:03:24.019 INFO:teuthology.orchestra.run.smithi067.stdout:/proc/sys/fs/binfmt_misc 2024-03-28T16:03:24.019 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/containers/storage/overlay/30d894f99bed72a32b984503db6cc28f538f7c49fc4c773c383af2cd57bf11fb/merged 2024-03-28T16:03:24.019 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/containers/storage/overlay/1a6a4bff1c35300258cf6439198751bb695a9c5628c713d48a4d9fbea7e23e3e/merged 2024-03-28T16:03:24.019 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/containers/storage/overlay/39ef9de1a3f1e636ffad421832a862f6031bdf7eb717dc017b8419772e8f644f/merged 2024-03-28T16:03:24.019 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/containers/storage/overlay/45b9fdca53df002ccca6e6256a42bd3154a6624370dbf5300c65c383339bb6c5/merged 2024-03-28T16:03:24.019 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/containers/storage/overlay/8169ebff8d96e64432f2137fc3ca2bf8aecb1d8959c67695fcfe7b4e4bbcb20a/merged 2024-03-28T16:03:24.020 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/containers/storage/overlay/4c08a654a70e2302ed4f1c409260eea1d3d8f317c321c0ff2d1c98799dc3c39f/merged 2024-03-28T16:03:24.020 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/containers/storage/overlay/240404dcaa83ad223fd4deb7a8c73b66b8c8ec096c93fc0d1f3b19893df4822a/merged 2024-03-28T16:03:24.020 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/containers/storage/overlay/967eee94b0fc4339bd1ab618c160bf2ecff2a3c7e1ed140771551ffcdb267c93/merged 2024-03-28T16:03:24.020 INFO:teuthology.orchestra.run.smithi067.stdout:/run/netns 2024-03-28T16:03:24.020 INFO:teuthology.orchestra.run.smithi067.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-28T16:03:24.020 INFO:teuthology.orchestra.run.smithi067.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-28T16:03:24.020 INFO:teuthology.orchestra.run.smithi067.stdout:/home/ubuntu/cephtest/mnt.0 2024-03-28T16:03:24.021 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T16:03:24.021 DEBUG:teuthology.orchestra.run.smithi067:> ls /sys/fs/fuse/connections 2024-03-28T16:03:24.065 INFO:teuthology.orchestra.run.smithi067.stdout:118 2024-03-28T16:03:24.065 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [118] 2024-03-28T16:03:24.065 DEBUG:teuthology.orchestra.run.smithi067:> sudo stdin-killer -- python3 -c ' 2024-03-28T16:03:24.065 DEBUG:teuthology.orchestra.run.smithi067:> import glob 2024-03-28T16:03:24.066 DEBUG:teuthology.orchestra.run.smithi067:> import re 2024-03-28T16:03:24.066 DEBUG:teuthology.orchestra.run.smithi067:> import os 2024-03-28T16:03:24.066 DEBUG:teuthology.orchestra.run.smithi067:> import subprocess 2024-03-28T16:03:24.066 DEBUG:teuthology.orchestra.run.smithi067:> 2024-03-28T16:03:24.066 DEBUG:teuthology.orchestra.run.smithi067:> def _find_admin_socket(client_name): 2024-03-28T16:03:24.066 DEBUG:teuthology.orchestra.run.smithi067:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-28T16:03:24.066 DEBUG:teuthology.orchestra.run.smithi067:> files = glob.glob(asok_path) 2024-03-28T16:03:24.066 DEBUG:teuthology.orchestra.run.smithi067:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-28T16:03:24.066 DEBUG:teuthology.orchestra.run.smithi067:> 2024-03-28T16:03:24.066 DEBUG:teuthology.orchestra.run.smithi067:> # Given a non-glob path, it better be there 2024-03-28T16:03:24.066 DEBUG:teuthology.orchestra.run.smithi067:> if "*" not in asok_path: 2024-03-28T16:03:24.066 DEBUG:teuthology.orchestra.run.smithi067:> assert(len(files) == 1) 2024-03-28T16:03:24.066 DEBUG:teuthology.orchestra.run.smithi067:> return files[0] 2024-03-28T16:03:24.066 DEBUG:teuthology.orchestra.run.smithi067:> 2024-03-28T16:03:24.066 DEBUG:teuthology.orchestra.run.smithi067:> for f in files: 2024-03-28T16:03:24.067 DEBUG:teuthology.orchestra.run.smithi067:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-28T16:03:24.067 DEBUG:teuthology.orchestra.run.smithi067:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-28T16:03:24.067 DEBUG:teuthology.orchestra.run.smithi067:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-28T16:03:24.067 DEBUG:teuthology.orchestra.run.smithi067:> contents = proc_f.read() 2024-03-28T16:03:24.067 DEBUG:teuthology.orchestra.run.smithi067:> if mountpoint in contents: 2024-03-28T16:03:24.067 DEBUG:teuthology.orchestra.run.smithi067:> return f 2024-03-28T16:03:24.067 DEBUG:teuthology.orchestra.run.smithi067:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-28T16:03:24.067 DEBUG:teuthology.orchestra.run.smithi067:> 2024-03-28T16:03:24.067 DEBUG:teuthology.orchestra.run.smithi067:> print(_find_admin_socket("client.0")) 2024-03-28T16:03:24.067 DEBUG:teuthology.orchestra.run.smithi067:> ' 2024-03-28T16:03:24.296 INFO:teuthology.orchestra.run.smithi067.stdout:/var/run/ceph/ceph-client.0.83688.asok 2024-03-28T16:03:24.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:24 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-28T16:03:24.315 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.83688.asok 2024-03-28T16:03:24.315 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T16:03:24.315 DEBUG:teuthology.orchestra.run.smithi067:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.83688.asok status 2024-03-28T16:03:24.566 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-28T16:03:24.566 INFO:teuthology.orchestra.run.smithi067.stdout: "metadata": { 2024-03-28T16:03:24.566 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph_sha1": "3be9a5c9bc793e11e2800a8c0c696e8b46742033", 2024-03-28T16:03:24.566 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-28T16:03:24.566 INFO:teuthology.orchestra.run.smithi067.stdout: "entity_id": "0", 2024-03-28T16:03:24.567 INFO:teuthology.orchestra.run.smithi067.stdout: "hostname": "smithi067", 2024-03-28T16:03:24.567 INFO:teuthology.orchestra.run.smithi067.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-28T16:03:24.567 INFO:teuthology.orchestra.run.smithi067.stdout: "pid": "83688", 2024-03-28T16:03:24.567 INFO:teuthology.orchestra.run.smithi067.stdout: "root": "/volumes/qa/sv_0/7ad6a264-caac-444d-a6a0-0c4d0503695c" 2024-03-28T16:03:24.567 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:03:24.567 INFO:teuthology.orchestra.run.smithi067.stdout: "dentry_count": 0, 2024-03-28T16:03:24.567 INFO:teuthology.orchestra.run.smithi067.stdout: "dentry_pinned_count": 0, 2024-03-28T16:03:24.567 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 15186, 2024-03-28T16:03:24.567 INFO:teuthology.orchestra.run.smithi067.stdout: "inst": { 2024-03-28T16:03:24.567 INFO:teuthology.orchestra.run.smithi067.stdout: "name": { 2024-03-28T16:03:24.567 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "client", 2024-03-28T16:03:24.567 INFO:teuthology.orchestra.run.smithi067.stdout: "num": 15186 2024-03-28T16:03:24.567 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:03:24.567 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": { 2024-03-28T16:03:24.567 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-03-28T16:03:24.568 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "192.168.0.1:0", 2024-03-28T16:03:24.568 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 4188646177 2024-03-28T16:03:24.568 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:03:24.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:03:24.568 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": { 2024-03-28T16:03:24.568 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-03-28T16:03:24.568 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "192.168.0.1:0", 2024-03-28T16:03:24.568 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 4188646177 2024-03-28T16:03:24.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:03:24.568 INFO:teuthology.orchestra.run.smithi067.stdout: "inst_str": "client.15186 192.168.0.1:0/4188646177", 2024-03-28T16:03:24.568 INFO:teuthology.orchestra.run.smithi067.stdout: "addr_str": "192.168.0.1:0/4188646177", 2024-03-28T16:03:24.568 INFO:teuthology.orchestra.run.smithi067.stdout: "inode_count": 5, 2024-03-28T16:03:24.568 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_epoch": 19, 2024-03-28T16:03:24.568 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_epoch": 92, 2024-03-28T16:03:24.568 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_epoch_barrier": 0, 2024-03-28T16:03:24.569 INFO:teuthology.orchestra.run.smithi067.stdout: "blocklisted": false, 2024-03-28T16:03:24.569 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs" 2024-03-28T16:03:24.569 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-28T16:03:24.582 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T16:03:24.582 DEBUG:teuthology.orchestra.run.smithi067:> stat --file-system '--printf=%T 2024-03-28T16:03:24.582 DEBUG:teuthology.orchestra.run.smithi067:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-28T16:03:24.634 INFO:teuthology.orchestra.run.smithi067.stdout:fuseblk 2024-03-28T16:03:24.635 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-28T16:03:24.635 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T16:03:24.635 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-28T16:03:24.723 INFO:teuthology.run_tasks:Running task check-counter... 2024-03-28T16:03:24.734 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-03-28T16:03:24.745 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T16:03:24.833 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:24 smithi067 ceph-mon[31707]: pgmap v434: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T16:03:24.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:24.972+0000 7f58ed200640 1 -- 172.21.15.67:0/2883658389 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58e81016c0 msgr2=0x7f58e8101ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:24.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:24.972+0000 7f58ed200640 1 --2- 172.21.15.67:0/2883658389 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58e81016c0 0x7f58e8101ae0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f58d8009900 tx=0x7f58d802bc30 comp rx=0 tx=0).stop 2024-03-28T16:03:24.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:24.972+0000 7f58ed200640 1 -- 172.21.15.67:0/2883658389 shutdown_connections 2024-03-28T16:03:24.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:24.972+0000 7f58ed200640 1 --2- 172.21.15.67:0/2883658389 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f58e8103270 0x7f58e8109b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:24.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:24.972+0000 7f58ed200640 1 --2- 172.21.15.67:0/2883658389 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f58e81028b0 0x7f58e8102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:24.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:24.972+0000 7f58ed200640 1 --2- 172.21.15.67:0/2883658389 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58e81016c0 0x7f58e8101ae0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:24.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:24.972+0000 7f58ed200640 1 -- 172.21.15.67:0/2883658389 >> 172.21.15.67:0/2883658389 conn(0x7f58e80fcc50 msgr2=0x7f58e80ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:24.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:24.972+0000 7f58ed200640 1 -- 172.21.15.67:0/2883658389 shutdown_connections 2024-03-28T16:03:24.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:24.972+0000 7f58ed200640 1 -- 172.21.15.67:0/2883658389 wait complete. 2024-03-28T16:03:24.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:24.973+0000 7f58ed200640 1 Processor -- start 2024-03-28T16:03:24.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:24.973+0000 7f58ed200640 1 -- start start 2024-03-28T16:03:24.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:24.974+0000 7f58ed200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f58e81016c0 0x7f58e81978f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:24.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:24.974+0000 7f58ed200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f58e81028b0 0x7f58e8197e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:24.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:24.974+0000 7f58ed200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58e8103270 0x7f58e819eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:24.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:24.974+0000 7f58ed200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58e810d240 con 0x7f58e8103270 2024-03-28T16:03:24.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:24.974+0000 7f58ed200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58e810d0c0 con 0x7f58e81028b0 2024-03-28T16:03:24.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:24.974+0000 7f58ed200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58e810d3c0 con 0x7f58e81016c0 2024-03-28T16:03:24.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:24.974+0000 7f58e6800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58e8103270 0x7f58e819eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:24.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:24.974+0000 7f58e6800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58e8103270 0x7f58e819eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53844/0 (socket says 172.21.15.67:53844) 2024-03-28T16:03:24.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:24.974+0000 7f58e6800640 1 -- 172.21.15.67:0/2039367970 learned_addr learned my addr 172.21.15.67:0/2039367970 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:03:24.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:24.974+0000 7f58dd400640 1 --2- 172.21.15.67:0/2039367970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f58e81028b0 0x7f58e8197e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:24.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:24.974+0000 7f58e5e00640 1 --2- 172.21.15.67:0/2039367970 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f58e81016c0 0x7f58e81978f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:24.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:24.975+0000 7f58e6800640 1 -- 172.21.15.67:0/2039367970 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f58e81016c0 msgr2=0x7f58e81978f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:24.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:24.975+0000 7f58e6800640 1 --2- 172.21.15.67:0/2039367970 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f58e81016c0 0x7f58e81978f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:24.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:24.975+0000 7f58e6800640 1 -- 172.21.15.67:0/2039367970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f58e81028b0 msgr2=0x7f58e8197e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:24.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:24.975+0000 7f58e6800640 1 --2- 172.21.15.67:0/2039367970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f58e81028b0 0x7f58e8197e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:24.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:24.975+0000 7f58e6800640 1 -- 172.21.15.67:0/2039367970 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58e819f510 con 0x7f58e8103270 2024-03-28T16:03:24.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:24.975+0000 7f58e6800640 1 --2- 172.21.15.67:0/2039367970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58e8103270 0x7f58e819eea0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f58e000eaa0 tx=0x7f58e000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:24.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:24.975+0000 7f58df400640 1 -- 172.21.15.67:0/2039367970 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e000cdb0 con 0x7f58e8103270 2024-03-28T16:03:24.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:24.975+0000 7f58df400640 1 -- 172.21.15.67:0/2039367970 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f58e0005020 con 0x7f58e8103270 2024-03-28T16:03:24.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:24.975+0000 7f58ed200640 1 -- 172.21.15.67:0/2039367970 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58e819f800 con 0x7f58e8103270 2024-03-28T16:03:24.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:24.975+0000 7f58df400640 1 -- 172.21.15.67:0/2039367970 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e00116d0 con 0x7f58e8103270 2024-03-28T16:03:24.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:24.975+0000 7f58ed200640 1 -- 172.21.15.67:0/2039367970 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58e819fd50 con 0x7f58e8103270 2024-03-28T16:03:24.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:24.977+0000 7f58ed200640 1 -- 172.21.15.67:0/2039367970 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58e81089e0 con 0x7f58e8103270 2024-03-28T16:03:24.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:24.979+0000 7f58df400640 1 -- 172.21.15.67:0/2039367970 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f58e00051d0 con 0x7f58e8103270 2024-03-28T16:03:24.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:24.980+0000 7f58df400640 1 --2- 172.21.15.67:0/2039367970 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f58cc0a2800 0x7f58cc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:24.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:24.980+0000 7f58df400640 1 -- 172.21.15.67:0/2039367970 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f58e00cecb0 con 0x7f58e8103270 2024-03-28T16:03:24.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:24.980+0000 7f58e5e00640 1 --2- 172.21.15.67:0/2039367970 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f58cc0a2800 0x7f58cc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:24.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:24.981+0000 7f58e5e00640 1 --2- 172.21.15.67:0/2039367970 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f58cc0a2800 0x7f58cc0a4cc0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f58d802c240 tx=0x7f58d8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:24.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:24.985+0000 7f58df400640 1 -- 172.21.15.67:0/2039367970 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f58e0016030 con 0x7f58e8103270 2024-03-28T16:03:25.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:24 smithi150 ceph-mon[34825]: pgmap v434: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T16:03:25.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:24 smithi183 ceph-mon[35580]: pgmap v434: 97 pgs: 97 active+clean; 638 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T16:03:25.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.189+0000 7f58ed200640 1 -- 172.21.15.67:0/2039367970 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f58e80622a0 con 0x7f58e8103270 2024-03-28T16:03:25.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.192+0000 7f58df400640 1 -- 172.21.15.67:0/2039367970 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) v1 ==== 74+0+22673 (secure 0 0 0) 0x7f58e009bd00 con 0x7f58e8103270 2024-03-28T16:03:25.194 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:03:25.194 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":92,"fsid":"b5f8225a-ed1a-11ee-95d0-87774f69a715","created":"2024-03-28T15:50:21.471924+0000","modified":"2024-03-28T16:01:40.643215+0000","last_up_change":"2024-03-28T16:00:01.369123+0000","last_in_change":"2024-03-28T15:59:37.515483+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T15:54:55.588007+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T16:01:05.719801+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T16:01:07.206303+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7c1791b6-9c0e-4512-a327-cad166bb5581","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6803","nonce":1894681701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6805","nonce":1894681701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6809","nonce":1894681701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6807","nonce":1894681701}]},"public_addr":"172.21.15.67:6803/1894681701","cluster_addr":"172.21.15.67:6805/1894681701","heartbeat_back_addr":"172.21.15.67:6809/1894681701","heartbeat_front_addr":"172.21.15.67:6807/1894681701","state":["exists","up"]},{"osd":1,"uuid":"0cdda7f3-d00c-4436-bcc1-49fb8072b2e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6801","nonce":1734465762}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6803","nonce":1734465762}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6807","nonce":1734465762}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6805","nonce":1734465762}]},"public_addr":"172.21.15.150:6801/1734465762","cluster_addr":"172.21.15.150:6803/1734465762","heartbeat_back_addr":"172.21.15.150:6807/1734465762","heartbeat_front_addr":"172.21.15.150:6805/1734465762","state":["exists","up"]},{"osd":2,"uuid":"a55b9a02-c3fd-4061-95c0-53a24a17ade0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6801","nonce":3398095931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6803","nonce":3398095931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6807","nonce":3398095931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6805","nonce":3398095931}]},"public_addr":"172.21.15.183:6801/3398095931","cluster_addr":"172.21.15.183:6803/3398095931","heartbeat_back_addr":"172.21.15.183:6807/3398095931","heartbeat_front_addr":"172.21.15.183:6805/3398095931","state":["exists","up"]},{"osd":3,"uuid":"d34940c0-5ce9-4b8b-aa36-4872414a043f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6811","nonce":1933541875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6813","nonce":1933541875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6817","nonce":1933541875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6815","nonce":1933541875}]},"public_addr":"172.21.15.67:6811/1933541875","cluster_addr":"172.21.15.67:6813/1933541875","heartbeat_back_addr":"172.21.15.67:6817/1933541875","heartbeat_front_addr":"172.21.15.67:6815/1933541875","state":["exists","up"]},{"osd":4,"uuid":"5cc3e260-83a3-4a4d-aa6a-34a3b12b2b5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6809","nonce":3775269993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6811","nonce":3775269993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6815","nonce":3775269993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6813","nonce":3775269993}]},"public_addr":"172.21.15.150:6809/3775269993","cluster_addr":"172.21.15.150:6811/3775269993","heartbeat_back_addr":"172.21.15.150:6815/3775269993","heartbeat_front_addr":"172.21.15.150:6813/3775269993","state":["exists","up"]},{"osd":5,"uuid":"a2542b21-d6ac-413e-8b9b-4a6548b20840","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6809","nonce":164284137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6811","nonce":164284137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6815","nonce":164284137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6813","nonce":164284137}]},"public_addr":"172.21.15.183:6809/164284137","cluster_addr":"172.21.15.183:6811/164284137","heartbeat_back_addr":"172.21.15.183:6815/164284137","heartbeat_front_addr":"172.21.15.183:6813/164284137","state":["exists","up"]},{"osd":6,"uuid":"8905a2d3-0f1b-4853-a218-22da2635150a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6819","nonce":3565528013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6821","nonce":3565528013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6825","nonce":3565528013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6823","nonce":3565528013}]},"public_addr":"172.21.15.67:6819/3565528013","cluster_addr":"172.21.15.67:6821/3565528013","heartbeat_back_addr":"172.21.15.67:6825/3565528013","heartbeat_front_addr":"172.21.15.67:6823/3565528013","state":["exists","up"]},{"osd":7,"uuid":"b275055b-b451-4180-91b1-e6e61a828c48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6817","nonce":3735115340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6819","nonce":3735115340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6823","nonce":3735115340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6821","nonce":3735115340}]},"public_addr":"172.21.15.150:6817/3735115340","cluster_addr":"172.21.15.150:6819/3735115340","heartbeat_back_addr":"172.21.15.150:6823/3735115340","heartbeat_front_addr":"172.21.15.150:6821/3735115340","state":["exists","up"]},{"osd":8,"uuid":"b6abf7ad-eb34-4f60-a4eb-348a92d7195b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6817","nonce":1025400095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6819","nonce":1025400095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6823","nonce":1025400095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6821","nonce":1025400095}]},"public_addr":"172.21.15.183:6817/1025400095","cluster_addr":"172.21.15.183:6819/1025400095","heartbeat_back_addr":"172.21.15.183:6823/1025400095","heartbeat_front_addr":"172.21.15.183:6821/1025400095","state":["exists","up"]},{"osd":9,"uuid":"28a78e3a-0657-40aa-816f-16a6fadadcbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6827","nonce":76885409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6829","nonce":76885409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6833","nonce":76885409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6831","nonce":76885409}]},"public_addr":"172.21.15.67:6827/76885409","cluster_addr":"172.21.15.67:6829/76885409","heartbeat_back_addr":"172.21.15.67:6833/76885409","heartbeat_front_addr":"172.21.15.67:6831/76885409","state":["exists","up"]},{"osd":10,"uuid":"d287907e-424f-4b49-a06e-4613c6de0474","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6825","nonce":3489982080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6827","nonce":3489982080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6831","nonce":3489982080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6829","nonce":3489982080}]},"public_addr":"172.21.15.150:6825/3489982080","cluster_addr":"172.21.15.150:6827/3489982080","heartbeat_back_addr":"172.21.15.150:6831/3489982080","heartbeat_front_addr":"172.21.15.150:6829/3489982080","state":["exists","up"]},{"osd":11,"uuid":"a3c29c47-a228-4f42-a23c-fbdb0c093699","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":85,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6825","nonce":1892639849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6826","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6827","nonce":1892639849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6831","nonce":1892639849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6829","nonce":1892639849}]},"public_addr":"172.21.15.183:6825/1892639849","cluster_addr":"172.21.15.183:6827/1892639849","heartbeat_back_addr":"172.21.15.183:6831/1892639849","heartbeat_front_addr":"172.21.15.183:6829/1892639849","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:53:45.646211+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:54:18.956590+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:54:52.161646+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:55:27.502822+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:56:01.162932+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:56:34.674452+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:57:09.298679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:57:43.230406+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:58:17.003821+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:58:51.085863+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:59:25.665065+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T16:00:00.263457+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":1}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6801/1631001968":"2024-03-29T15:51:38.875334+0000","172.21.15.67:6800/1631001968":"2024-03-29T15:51:38.875334+0000","172.21.15.67:0/1533056310":"2024-03-29T15:51:38.875334+0000","172.21.15.67:6801/3580491071":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/2030412458":"2024-03-29T15:51:03.859740+0000","172.21.15.67:6800/3580491071":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/2905307533":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/3763759988":"2024-03-29T15:51:38.875334+0000","172.21.15.67:0/4089399880":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/4239108661":"2024-03-29T15:51:38.875334+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T16:03:25.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.195+0000 7f58ed200640 1 -- 172.21.15.67:0/2039367970 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f58cc0a2800 msgr2=0x7f58cc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:25.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.195+0000 7f58ed200640 1 --2- 172.21.15.67:0/2039367970 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f58cc0a2800 0x7f58cc0a4cc0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f58d802c240 tx=0x7f58d8005fd0 comp rx=0 tx=0).stop 2024-03-28T16:03:25.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.195+0000 7f58ed200640 1 -- 172.21.15.67:0/2039367970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58e8103270 msgr2=0x7f58e819eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:25.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.195+0000 7f58ed200640 1 --2- 172.21.15.67:0/2039367970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58e8103270 0x7f58e819eea0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f58e000eaa0 tx=0x7f58e000ef60 comp rx=0 tx=0).stop 2024-03-28T16:03:25.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.195+0000 7f58ed200640 1 -- 172.21.15.67:0/2039367970 shutdown_connections 2024-03-28T16:03:25.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.195+0000 7f58ed200640 1 --2- 172.21.15.67:0/2039367970 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f58cc0a2800 0x7f58cc0a4cc0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:25.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.195+0000 7f58ed200640 1 --2- 172.21.15.67:0/2039367970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58e8103270 0x7f58e819eea0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:25.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.195+0000 7f58ed200640 1 --2- 172.21.15.67:0/2039367970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f58e81028b0 0x7f58e8197e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:25.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.195+0000 7f58ed200640 1 --2- 172.21.15.67:0/2039367970 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f58e81016c0 0x7f58e81978f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:25.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.195+0000 7f58ed200640 1 -- 172.21.15.67:0/2039367970 >> 172.21.15.67:0/2039367970 conn(0x7f58e80fcc50 msgr2=0x7f58e8105bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:25.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.196+0000 7f58ed200640 1 -- 172.21.15.67:0/2039367970 shutdown_connections 2024-03-28T16:03:25.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.196+0000 7f58ed200640 1 -- 172.21.15.67:0/2039367970 wait complete. 2024-03-28T16:03:25.217 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T15:54:55.588007+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T16:01:05.719801+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T16:01:07.206303+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T16:03:25.218 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T16:03:25.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.447+0000 7fb881200640 1 -- 172.21.15.67:0/3179961564 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb87c109770 msgr2=0x7fb87c110000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:25.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.447+0000 7fb881200640 1 --2- 172.21.15.67:0/3179961564 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb87c109770 0x7fb87c110000 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fb874009ca0 tx=0x7fb87402c100 comp rx=0 tx=0).stop 2024-03-28T16:03:25.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.447+0000 7fb881200640 1 -- 172.21.15.67:0/3179961564 shutdown_connections 2024-03-28T16:03:25.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.447+0000 7fb881200640 1 --2- 172.21.15.67:0/3179961564 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb87c109770 0x7fb87c110000 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:25.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.447+0000 7fb881200640 1 --2- 172.21.15.67:0/3179961564 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb87c108db0 0x7fb87c109230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:25.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.447+0000 7fb881200640 1 --2- 172.21.15.67:0/3179961564 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb87c107bc0 0x7fb87c107fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:25.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.447+0000 7fb881200640 1 -- 172.21.15.67:0/3179961564 >> 172.21.15.67:0/3179961564 conn(0x7fb87c076b70 msgr2=0x7fb87c078f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:25.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.447+0000 7fb881200640 1 -- 172.21.15.67:0/3179961564 shutdown_connections 2024-03-28T16:03:25.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.447+0000 7fb881200640 1 -- 172.21.15.67:0/3179961564 wait complete. 2024-03-28T16:03:25.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.448+0000 7fb881200640 1 Processor -- start 2024-03-28T16:03:25.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.448+0000 7fb881200640 1 -- start start 2024-03-28T16:03:25.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.448+0000 7fb881200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb87c107bc0 0x7fb87c19ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:25.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.449+0000 7fb881200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb87c108db0 0x7fb87c19e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:25.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.449+0000 7fb881200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb87c109770 0x7fb87c1a5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:25.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.449+0000 7fb881200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb87c1136d0 con 0x7fb87c109770 2024-03-28T16:03:25.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.449+0000 7fb881200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb87c113550 con 0x7fb87c108db0 2024-03-28T16:03:25.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.449+0000 7fb881200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb87c113850 con 0x7fb87c107bc0 2024-03-28T16:03:25.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.449+0000 7fb87be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb87c109770 0x7fb87c1a5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:25.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.449+0000 7fb87be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb87c109770 0x7fb87c1a5290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53860/0 (socket says 172.21.15.67:53860) 2024-03-28T16:03:25.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.449+0000 7fb87be00640 1 -- 172.21.15.67:0/4009836903 learned_addr learned my addr 172.21.15.67:0/4009836903 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:03:25.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.449+0000 7fb873e00640 1 --2- 172.21.15.67:0/4009836903 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb87c108db0 0x7fb87c19e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:25.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.449+0000 7fb87b400640 1 --2- 172.21.15.67:0/4009836903 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb87c107bc0 0x7fb87c19ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:25.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.449+0000 7fb87be00640 1 -- 172.21.15.67:0/4009836903 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb87c107bc0 msgr2=0x7fb87c19ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:25.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.449+0000 7fb87be00640 1 --2- 172.21.15.67:0/4009836903 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb87c107bc0 0x7fb87c19ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:25.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.449+0000 7fb87be00640 1 -- 172.21.15.67:0/4009836903 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb87c108db0 msgr2=0x7fb87c19e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:25.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.449+0000 7fb87be00640 1 --2- 172.21.15.67:0/4009836903 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb87c108db0 0x7fb87c19e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:25.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.449+0000 7fb87be00640 1 -- 172.21.15.67:0/4009836903 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb87c1a5930 con 0x7fb87c109770 2024-03-28T16:03:25.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.450+0000 7fb87be00640 1 --2- 172.21.15.67:0/4009836903 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb87c109770 0x7fb87c1a5290 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fb874032520 tx=0x7fb874034880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:25.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.450+0000 7fb878c00640 1 -- 172.21.15.67:0/4009836903 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb874039040 con 0x7fb87c109770 2024-03-28T16:03:25.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.450+0000 7fb878c00640 1 -- 172.21.15.67:0/4009836903 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb874044070 con 0x7fb87c109770 2024-03-28T16:03:25.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.450+0000 7fb881200640 1 -- 172.21.15.67:0/4009836903 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb87c1a5bc0 con 0x7fb87c109770 2024-03-28T16:03:25.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.450+0000 7fb878c00640 1 -- 172.21.15.67:0/4009836903 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8740043c0 con 0x7fb87c109770 2024-03-28T16:03:25.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.450+0000 7fb881200640 1 -- 172.21.15.67:0/4009836903 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb87c1a6100 con 0x7fb87c109770 2024-03-28T16:03:25.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.452+0000 7fb881200640 1 -- 172.21.15.67:0/4009836903 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb87c10eee0 con 0x7fb87c109770 2024-03-28T16:03:25.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.454+0000 7fb878c00640 1 -- 172.21.15.67:0/4009836903 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fb874004560 con 0x7fb87c109770 2024-03-28T16:03:25.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.455+0000 7fb878c00640 1 --2- 172.21.15.67:0/4009836903 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb8600a2880 0x7fb8600a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:25.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.455+0000 7fb878c00640 1 -- 172.21.15.67:0/4009836903 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fb8740f1200 con 0x7fb87c109770 2024-03-28T16:03:25.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.455+0000 7fb87b400640 1 --2- 172.21.15.67:0/4009836903 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb8600a2880 0x7fb8600a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:25.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.456+0000 7fb87b400640 1 --2- 172.21.15.67:0/4009836903 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb8600a2880 0x7fb8600a4d40 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fb86c005fd0 tx=0x7fb86c005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:25.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.460+0000 7fb878c00640 1 -- 172.21.15.67:0/4009836903 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb8740333a0 con 0x7fb87c109770 2024-03-28T16:03:25.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.662+0000 7fb881200640 1 -- 172.21.15.67:0/4009836903 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb87c0622a0 con 0x7fb87c109770 2024-03-28T16:03:25.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.663+0000 7fb878c00640 1 -- 172.21.15.67:0/4009836903 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v92) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb8740be250 con 0x7fb87c109770 2024-03-28T16:03:25.664 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-03-28T16:03:25.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.665+0000 7fb881200640 1 -- 172.21.15.67:0/4009836903 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb8600a2880 msgr2=0x7fb8600a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:25.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.665+0000 7fb881200640 1 --2- 172.21.15.67:0/4009836903 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb8600a2880 0x7fb8600a4d40 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fb86c005fd0 tx=0x7fb86c005850 comp rx=0 tx=0).stop 2024-03-28T16:03:25.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.665+0000 7fb881200640 1 -- 172.21.15.67:0/4009836903 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb87c109770 msgr2=0x7fb87c1a5290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:25.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.665+0000 7fb881200640 1 --2- 172.21.15.67:0/4009836903 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb87c109770 0x7fb87c1a5290 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fb874032520 tx=0x7fb874034880 comp rx=0 tx=0).stop 2024-03-28T16:03:25.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.665+0000 7fb881200640 1 -- 172.21.15.67:0/4009836903 shutdown_connections 2024-03-28T16:03:25.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.665+0000 7fb881200640 1 --2- 172.21.15.67:0/4009836903 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb8600a2880 0x7fb8600a4d40 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:25.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.665+0000 7fb881200640 1 --2- 172.21.15.67:0/4009836903 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb87c109770 0x7fb87c1a5290 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:25.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.665+0000 7fb881200640 1 --2- 172.21.15.67:0/4009836903 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb87c108db0 0x7fb87c19e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:25.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.665+0000 7fb881200640 1 --2- 172.21.15.67:0/4009836903 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb87c107bc0 0x7fb87c19ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:25.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.665+0000 7fb881200640 1 -- 172.21.15.67:0/4009836903 >> 172.21.15.67:0/4009836903 conn(0x7fb87c076b70 msgr2=0x7fb87c10c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:25.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.666+0000 7fb881200640 1 -- 172.21.15.67:0/4009836903 shutdown_connections 2024-03-28T16:03:25.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.666+0000 7fb881200640 1 -- 172.21.15.67:0/4009836903 wait complete. 2024-03-28T16:03:25.684 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T16:03:25.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.904+0000 7fd00a200640 1 -- 172.21.15.67:0/3149705206 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd004105370 msgr2=0x7fd00410bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:25.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.904+0000 7fd00a200640 1 --2- 172.21.15.67:0/3149705206 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd004105370 0x7fd00410bc00 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fcffc009ca0 tx=0x7fcffc02c0f0 comp rx=0 tx=0).stop 2024-03-28T16:03:25.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.904+0000 7fd00a200640 1 -- 172.21.15.67:0/3149705206 shutdown_connections 2024-03-28T16:03:25.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.904+0000 7fd00a200640 1 --2- 172.21.15.67:0/3149705206 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd004105370 0x7fd00410bc00 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:25.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.904+0000 7fd00a200640 1 --2- 172.21.15.67:0/3149705206 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd0041049b0 0x7fd004104e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:25.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.904+0000 7fd00a200640 1 --2- 172.21.15.67:0/3149705206 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd0041037c0 0x7fd004103be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:25.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.904+0000 7fd00a200640 1 -- 172.21.15.67:0/3149705206 >> 172.21.15.67:0/3149705206 conn(0x7fd0040fed50 msgr2=0x7fd004101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:25.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.904+0000 7fd00a200640 1 -- 172.21.15.67:0/3149705206 shutdown_connections 2024-03-28T16:03:25.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.904+0000 7fd00a200640 1 -- 172.21.15.67:0/3149705206 wait complete. 2024-03-28T16:03:25.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.905+0000 7fd00a200640 1 Processor -- start 2024-03-28T16:03:25.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.906+0000 7fd00a200640 1 -- start start 2024-03-28T16:03:25.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.906+0000 7fd00a200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd0041037c0 0x7fd0041998c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:25.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.906+0000 7fd00a200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd0041049b0 0x7fd004199e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:25.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.906+0000 7fd00a200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd004105370 0x7fd0041a0e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:25.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.906+0000 7fd00a200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd00410f2d0 con 0x7fd004105370 2024-03-28T16:03:25.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.906+0000 7fd00a200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd00410f150 con 0x7fd0041049b0 2024-03-28T16:03:25.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.906+0000 7fd00a200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd00410f450 con 0x7fd0041037c0 2024-03-28T16:03:25.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.906+0000 7fd009800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd004105370 0x7fd0041a0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:25.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.906+0000 7fd009800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd004105370 0x7fd0041a0e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53878/0 (socket says 172.21.15.67:53878) 2024-03-28T16:03:25.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.906+0000 7fd009800640 1 -- 172.21.15.67:0/3671695072 learned_addr learned my addr 172.21.15.67:0/3671695072 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:03:25.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.906+0000 7fd003e00640 1 --2- 172.21.15.67:0/3671695072 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd0041049b0 0x7fd004199e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:25.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.907+0000 7fd008e00640 1 --2- 172.21.15.67:0/3671695072 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd0041037c0 0x7fd0041998c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:25.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.907+0000 7fd009800640 1 -- 172.21.15.67:0/3671695072 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd0041037c0 msgr2=0x7fd0041998c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:25.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.907+0000 7fd009800640 1 --2- 172.21.15.67:0/3671695072 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd0041037c0 0x7fd0041998c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:25.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.907+0000 7fd009800640 1 -- 172.21.15.67:0/3671695072 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd0041049b0 msgr2=0x7fd004199e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:25.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.907+0000 7fd009800640 1 --2- 172.21.15.67:0/3671695072 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd0041049b0 0x7fd004199e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:25.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.907+0000 7fd009800640 1 -- 172.21.15.67:0/3671695072 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0041a14e0 con 0x7fd004105370 2024-03-28T16:03:25.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.907+0000 7fd009800640 1 --2- 172.21.15.67:0/3671695072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd004105370 0x7fd0041a0e70 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fcffc02c700 tx=0x7fcffc0328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:25.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.907+0000 7fd001600640 1 -- 172.21.15.67:0/3671695072 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcffc041020 con 0x7fd004105370 2024-03-28T16:03:25.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.907+0000 7fd00a200640 1 -- 172.21.15.67:0/3671695072 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0041a1770 con 0x7fd004105370 2024-03-28T16:03:25.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.907+0000 7fd001600640 1 -- 172.21.15.67:0/3671695072 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fcffc003e20 con 0x7fd004105370 2024-03-28T16:03:25.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.907+0000 7fd001600640 1 -- 172.21.15.67:0/3671695072 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcffc039040 con 0x7fd004105370 2024-03-28T16:03:25.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.908+0000 7fd00a200640 1 -- 172.21.15.67:0/3671695072 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0041a1c60 con 0x7fd004105370 2024-03-28T16:03:25.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.909+0000 7fd00a200640 1 -- 172.21.15.67:0/3671695072 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd00410aae0 con 0x7fd004105370 2024-03-28T16:03:25.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.911+0000 7fd001600640 1 -- 172.21.15.67:0/3671695072 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fcffc044020 con 0x7fd004105370 2024-03-28T16:03:25.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.912+0000 7fd001600640 1 --2- 172.21.15.67:0/3671695072 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fcfec0a2800 0x7fcfec0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:25.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.912+0000 7fd001600640 1 -- 172.21.15.67:0/3671695072 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fcffc0f0fa0 con 0x7fd004105370 2024-03-28T16:03:25.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.913+0000 7fd008e00640 1 --2- 172.21.15.67:0/3671695072 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fcfec0a2800 0x7fcfec0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:25.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.913+0000 7fd008e00640 1 --2- 172.21.15.67:0/3671695072 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fcfec0a2800 0x7fcfec0a4cc0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fcff80058c0 tx=0x7fcff8005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:25.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:25.917+0000 7fd001600640 1 -- 172.21.15.67:0/3671695072 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fcffc04b020 con 0x7fd004105370 2024-03-28T16:03:26.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:25 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/2039367970' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:03:26.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:25 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/4009836903' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T16:03:26.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:25 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/2039367970' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:03:26.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:25 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/4009836903' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T16:03:26.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:25 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/2039367970' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:03:26.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:25 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/4009836903' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T16:03:26.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.125+0000 7fd00a200640 1 -- 172.21.15.67:0/3671695072 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd0040622a0 con 0x7fd004105370 2024-03-28T16:03:26.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.126+0000 7fd001600640 1 -- 172.21.15.67:0/3671695072 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v92) v1 ==== 104+0+11 (secure 0 0 0) 0x7fcffc0bdff0 con 0x7fd004105370 2024-03-28T16:03:26.128 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-03-28T16:03:26.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.129+0000 7fd00a200640 1 -- 172.21.15.67:0/3671695072 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fcfec0a2800 msgr2=0x7fcfec0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:26.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.129+0000 7fd00a200640 1 --2- 172.21.15.67:0/3671695072 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fcfec0a2800 0x7fcfec0a4cc0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fcff80058c0 tx=0x7fcff8005850 comp rx=0 tx=0).stop 2024-03-28T16:03:26.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.129+0000 7fd00a200640 1 -- 172.21.15.67:0/3671695072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd004105370 msgr2=0x7fd0041a0e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:26.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.129+0000 7fd00a200640 1 --2- 172.21.15.67:0/3671695072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd004105370 0x7fd0041a0e70 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fcffc02c700 tx=0x7fcffc0328a0 comp rx=0 tx=0).stop 2024-03-28T16:03:26.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.129+0000 7fd00a200640 1 -- 172.21.15.67:0/3671695072 shutdown_connections 2024-03-28T16:03:26.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.129+0000 7fd00a200640 1 --2- 172.21.15.67:0/3671695072 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fcfec0a2800 0x7fcfec0a4cc0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:26.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.129+0000 7fd00a200640 1 --2- 172.21.15.67:0/3671695072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd004105370 0x7fd0041a0e70 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:26.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.129+0000 7fd00a200640 1 --2- 172.21.15.67:0/3671695072 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd0041049b0 0x7fd004199e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:26.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.129+0000 7fd00a200640 1 --2- 172.21.15.67:0/3671695072 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd0041037c0 0x7fd0041998c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:26.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.129+0000 7fd00a200640 1 -- 172.21.15.67:0/3671695072 >> 172.21.15.67:0/3671695072 conn(0x7fd0040fed50 msgr2=0x7fd004107cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:26.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.129+0000 7fd00a200640 1 -- 172.21.15.67:0/3671695072 shutdown_connections 2024-03-28T16:03:26.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.129+0000 7fd00a200640 1 -- 172.21.15.67:0/3671695072 wait complete. 2024-03-28T16:03:26.149 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T16:03:26.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.375+0000 7f71e1600640 1 -- 172.21.15.67:0/2361472259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71dc102790 msgr2=0x7f71dc102c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:26.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.375+0000 7f71e1600640 1 --2- 172.21.15.67:0/2361472259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71dc102790 0x7f71dc102c10 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f71c4009900 tx=0x7f71c402bc40 comp rx=0 tx=0).stop 2024-03-28T16:03:26.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.375+0000 7f71e1600640 1 -- 172.21.15.67:0/2361472259 shutdown_connections 2024-03-28T16:03:26.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.375+0000 7f71e1600640 1 --2- 172.21.15.67:0/2361472259 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f71dc103150 0x7f71dc1099e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:26.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.375+0000 7f71e1600640 1 --2- 172.21.15.67:0/2361472259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71dc102790 0x7f71dc102c10 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:26.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.375+0000 7f71e1600640 1 --2- 172.21.15.67:0/2361472259 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71dc1015a0 0x7f71dc1019c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:26.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.375+0000 7f71e1600640 1 -- 172.21.15.67:0/2361472259 >> 172.21.15.67:0/2361472259 conn(0x7f71dc0fcb10 msgr2=0x7f71dc0fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:26.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.375+0000 7f71e1600640 1 -- 172.21.15.67:0/2361472259 shutdown_connections 2024-03-28T16:03:26.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.375+0000 7f71e1600640 1 -- 172.21.15.67:0/2361472259 wait complete. 2024-03-28T16:03:26.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.376+0000 7f71e1600640 1 Processor -- start 2024-03-28T16:03:26.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.376+0000 7f71e1600640 1 -- start start 2024-03-28T16:03:26.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.377+0000 7f71e1600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71dc1015a0 0x7f71dc199a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:26.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.377+0000 7f71e1600640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71dc102790 0x7f71dc199f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:26.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.377+0000 7f71e1600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f71dc103150 0x7f71dc1a0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:26.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.377+0000 7f71e1600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71dc10d180 con 0x7f71dc1015a0 2024-03-28T16:03:26.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.377+0000 7f71e1600640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71dc10d000 con 0x7f71dc102790 2024-03-28T16:03:26.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.377+0000 7f71e1600640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71dc10d300 con 0x7f71dc103150 2024-03-28T16:03:26.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.377+0000 7f71dbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71dc1015a0 0x7f71dc199a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:26.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.377+0000 7f71dbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71dc1015a0 0x7f71dc199a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53898/0 (socket says 172.21.15.67:53898) 2024-03-28T16:03:26.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.377+0000 7f71dbe00640 1 -- 172.21.15.67:0/1824313103 learned_addr learned my addr 172.21.15.67:0/1824313103 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:03:26.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.377+0000 7f71db400640 1 --2- 172.21.15.67:0/1824313103 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71dc102790 0x7f71dc199f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:26.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.377+0000 7f71e0c00640 1 --2- 172.21.15.67:0/1824313103 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f71dc103150 0x7f71dc1a0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:26.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.378+0000 7f71dbe00640 1 -- 172.21.15.67:0/1824313103 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f71dc103150 msgr2=0x7f71dc1a0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:26.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.378+0000 7f71dbe00640 1 --2- 172.21.15.67:0/1824313103 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f71dc103150 0x7f71dc1a0fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:26.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.378+0000 7f71dbe00640 1 -- 172.21.15.67:0/1824313103 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71dc102790 msgr2=0x7f71dc199f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:26.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.378+0000 7f71dbe00640 1 --2- 172.21.15.67:0/1824313103 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71dc102790 0x7f71dc199f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:26.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.378+0000 7f71dbe00640 1 -- 172.21.15.67:0/1824313103 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71dc1a1650 con 0x7f71dc1015a0 2024-03-28T16:03:26.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.378+0000 7f71e0c00640 1 --2- 172.21.15.67:0/1824313103 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f71dc103150 0x7f71dc1a0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:03:26.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.378+0000 7f71dbe00640 1 --2- 172.21.15.67:0/1824313103 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71dc1015a0 0x7f71dc199a30 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f71cc00ea80 tx=0x7f71cc00ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:26.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.378+0000 7f71d8c00640 1 -- 172.21.15.67:0/1824313103 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71cc00ce40 con 0x7f71dc1015a0 2024-03-28T16:03:26.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.378+0000 7f71d8c00640 1 -- 172.21.15.67:0/1824313103 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f71cc011920 con 0x7f71dc1015a0 2024-03-28T16:03:26.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.378+0000 7f71e1600640 1 -- 172.21.15.67:0/1824313103 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71dc1a1940 con 0x7f71dc1015a0 2024-03-28T16:03:26.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.378+0000 7f71d8c00640 1 -- 172.21.15.67:0/1824313103 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71cc01aaa0 con 0x7f71dc1015a0 2024-03-28T16:03:26.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.378+0000 7f71e1600640 1 -- 172.21.15.67:0/1824313103 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71dc1a1ee0 con 0x7f71dc1015a0 2024-03-28T16:03:26.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.380+0000 7f71e1600640 1 -- 172.21.15.67:0/1824313103 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71dc0683b0 con 0x7f71dc1015a0 2024-03-28T16:03:26.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.382+0000 7f71d8c00640 1 -- 172.21.15.67:0/1824313103 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f71cc011ad0 con 0x7f71dc1015a0 2024-03-28T16:03:26.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.383+0000 7f71d8c00640 1 --2- 172.21.15.67:0/1824313103 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f71c00a2800 0x7f71c00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:26.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.383+0000 7f71d8c00640 1 -- 172.21.15.67:0/1824313103 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f71cc0cf130 con 0x7f71dc1015a0 2024-03-28T16:03:26.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.383+0000 7f71db400640 1 --2- 172.21.15.67:0/1824313103 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f71c00a2800 0x7f71c00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:26.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.384+0000 7f71db400640 1 --2- 172.21.15.67:0/1824313103 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f71c00a2800 0x7f71c00a4cc0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f71c4006010 tx=0x7f71c4000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:26.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.388+0000 7f71d8c00640 1 -- 172.21.15.67:0/1824313103 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f71cc016030 con 0x7f71dc1015a0 2024-03-28T16:03:26.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.596+0000 7f71e1600640 1 -- 172.21.15.67:0/1824313103 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f71dc1a28b0 con 0x7f71dc1015a0 2024-03-28T16:03:26.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.597+0000 7f71d8c00640 1 -- 172.21.15.67:0/1824313103 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) v1 ==== 100+0+11 (secure 0 0 0) 0x7f71cc09c180 con 0x7f71dc1015a0 2024-03-28T16:03:26.599 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-03-28T16:03:26.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.600+0000 7f71e1600640 1 -- 172.21.15.67:0/1824313103 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f71c00a2800 msgr2=0x7f71c00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:26.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.600+0000 7f71e1600640 1 --2- 172.21.15.67:0/1824313103 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f71c00a2800 0x7f71c00a4cc0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f71c4006010 tx=0x7f71c4000bc0 comp rx=0 tx=0).stop 2024-03-28T16:03:26.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.600+0000 7f71e1600640 1 -- 172.21.15.67:0/1824313103 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71dc1015a0 msgr2=0x7f71dc199a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:26.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.600+0000 7f71e1600640 1 --2- 172.21.15.67:0/1824313103 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71dc1015a0 0x7f71dc199a30 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f71cc00ea80 tx=0x7f71cc00ef40 comp rx=0 tx=0).stop 2024-03-28T16:03:26.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.600+0000 7f71e1600640 1 -- 172.21.15.67:0/1824313103 shutdown_connections 2024-03-28T16:03:26.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.600+0000 7f71e1600640 1 --2- 172.21.15.67:0/1824313103 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f71c00a2800 0x7f71c00a4cc0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:26.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.600+0000 7f71e1600640 1 --2- 172.21.15.67:0/1824313103 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f71dc103150 0x7f71dc1a0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:26.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.600+0000 7f71e1600640 1 --2- 172.21.15.67:0/1824313103 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71dc102790 0x7f71dc199f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:26.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.600+0000 7f71e1600640 1 --2- 172.21.15.67:0/1824313103 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71dc1015a0 0x7f71dc199a30 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:26.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.600+0000 7f71e1600640 1 -- 172.21.15.67:0/1824313103 >> 172.21.15.67:0/1824313103 conn(0x7f71dc0fcb10 msgr2=0x7f71dc0738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:26.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.600+0000 7f71e1600640 1 -- 172.21.15.67:0/1824313103 shutdown_connections 2024-03-28T16:03:26.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.601+0000 7f71e1600640 1 -- 172.21.15.67:0/1824313103 wait complete. 2024-03-28T16:03:26.621 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T16:03:26.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.839+0000 7fa433e00640 1 -- 172.21.15.67:0/4117710267 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa434108cf0 msgr2=0x7fa434109170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:26.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.839+0000 7fa433e00640 1 --2- 172.21.15.67:0/4117710267 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa434108cf0 0x7fa434109170 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fa420009900 tx=0x7fa42002bc30 comp rx=0 tx=0).stop 2024-03-28T16:03:26.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.840+0000 7fa433e00640 1 -- 172.21.15.67:0/4117710267 shutdown_connections 2024-03-28T16:03:26.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.840+0000 7fa433e00640 1 --2- 172.21.15.67:0/4117710267 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa4341096b0 0x7fa43410ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:26.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.840+0000 7fa433e00640 1 --2- 172.21.15.67:0/4117710267 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa434108cf0 0x7fa434109170 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:26.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.840+0000 7fa433e00640 1 --2- 172.21.15.67:0/4117710267 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa434107b00 0x7fa434107f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:26.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.840+0000 7fa433e00640 1 -- 172.21.15.67:0/4117710267 >> 172.21.15.67:0/4117710267 conn(0x7fa434076ab0 msgr2=0x7fa434078ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:26.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.840+0000 7fa433e00640 1 -- 172.21.15.67:0/4117710267 shutdown_connections 2024-03-28T16:03:26.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.840+0000 7fa433e00640 1 -- 172.21.15.67:0/4117710267 wait complete. 2024-03-28T16:03:26.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.841+0000 7fa433e00640 1 Processor -- start 2024-03-28T16:03:26.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.841+0000 7fa433e00640 1 -- start start 2024-03-28T16:03:26.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.841+0000 7fa433e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa434107b00 0x7fa43419dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:26.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.841+0000 7fa433e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa434108cf0 0x7fa43419e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:26.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.841+0000 7fa433e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4341096b0 0x7fa4341a5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:26.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.841+0000 7fa433e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa434113610 con 0x7fa434107b00 2024-03-28T16:03:26.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.841+0000 7fa433e00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa434113490 con 0x7fa4341096b0 2024-03-28T16:03:26.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.841+0000 7fa433e00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa434113790 con 0x7fa434108cf0 2024-03-28T16:03:26.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.841+0000 7fa432a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa434107b00 0x7fa43419dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:26.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.841+0000 7fa432a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa434107b00 0x7fa43419dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53904/0 (socket says 172.21.15.67:53904) 2024-03-28T16:03:26.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.841+0000 7fa432a00640 1 -- 172.21.15.67:0/3678841440 learned_addr learned my addr 172.21.15.67:0/3678841440 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:03:26.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.842+0000 7fa432000640 1 --2- 172.21.15.67:0/3678841440 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa434108cf0 0x7fa43419e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:26.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.842+0000 7fa433400640 1 --2- 172.21.15.67:0/3678841440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4341096b0 0x7fa4341a5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:26.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.842+0000 7fa432a00640 1 -- 172.21.15.67:0/3678841440 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa434108cf0 msgr2=0x7fa43419e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:26.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.842+0000 7fa432a00640 1 --2- 172.21.15.67:0/3678841440 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa434108cf0 0x7fa43419e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:26.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.842+0000 7fa432a00640 1 -- 172.21.15.67:0/3678841440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4341096b0 msgr2=0x7fa4341a5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:26.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.842+0000 7fa432a00640 1 --2- 172.21.15.67:0/3678841440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4341096b0 0x7fa4341a5270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:26.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.842+0000 7fa432a00640 1 -- 172.21.15.67:0/3678841440 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4341a5910 con 0x7fa434107b00 2024-03-28T16:03:26.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.842+0000 7fa432a00640 1 --2- 172.21.15.67:0/3678841440 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa434107b00 0x7fa43419dd00 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fa42400d770 tx=0x7fa42400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:26.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.843+0000 7fa42b400640 1 -- 172.21.15.67:0/3678841440 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa424017070 con 0x7fa434107b00 2024-03-28T16:03:26.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.843+0000 7fa42b400640 1 -- 172.21.15.67:0/3678841440 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa424009c30 con 0x7fa434107b00 2024-03-28T16:03:26.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.843+0000 7fa433e00640 1 -- 172.21.15.67:0/3678841440 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4341a5c00 con 0x7fa434107b00 2024-03-28T16:03:26.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.843+0000 7fa42b400640 1 -- 172.21.15.67:0/3678841440 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa424003d00 con 0x7fa434107b00 2024-03-28T16:03:26.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.843+0000 7fa433e00640 1 -- 172.21.15.67:0/3678841440 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4341a6150 con 0x7fa434107b00 2024-03-28T16:03:26.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.844+0000 7fa433e00640 1 -- 172.21.15.67:0/3678841440 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa43410ee70 con 0x7fa434107b00 2024-03-28T16:03:26.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.846+0000 7fa42b400640 1 -- 172.21.15.67:0/3678841440 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fa424003ea0 con 0x7fa434107b00 2024-03-28T16:03:26.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.847+0000 7fa42b400640 1 --2- 172.21.15.67:0/3678841440 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fa4180a2800 0x7fa4180a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:26.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.848+0000 7fa42b400640 1 -- 172.21.15.67:0/3678841440 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fa4240ce920 con 0x7fa434107b00 2024-03-28T16:03:26.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.848+0000 7fa432000640 1 --2- 172.21.15.67:0/3678841440 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fa4180a2800 0x7fa4180a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:26.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.849+0000 7fa432000640 1 --2- 172.21.15.67:0/3678841440 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fa4180a2800 0x7fa4180a4cc0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fa420000c00 tx=0x7fa420005be0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:26.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:26.852+0000 7fa42b400640 1 -- 172.21.15.67:0/3678841440 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa42409b970 con 0x7fa434107b00 2024-03-28T16:03:27.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:26 smithi150 ceph-mon[34825]: pgmap v435: 97 pgs: 97 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 170 B/s wr, 2 op/s 2024-03-28T16:03:27.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:26 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/3671695072' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T16:03:27.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:26 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/1824313103' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T16:03:27.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.055+0000 7fa433e00640 1 -- 172.21.15.67:0/3678841440 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa4340622a0 con 0x7fa434107b00 2024-03-28T16:03:27.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.058+0000 7fa42b400640 1 -- 172.21.15.67:0/3678841440 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) v1 ==== 74+0+22673 (secure 0 0 0) 0x7fa424009de0 con 0x7fa434107b00 2024-03-28T16:03:27.060 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:03:27.060 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":92,"fsid":"b5f8225a-ed1a-11ee-95d0-87774f69a715","created":"2024-03-28T15:50:21.471924+0000","modified":"2024-03-28T16:01:40.643215+0000","last_up_change":"2024-03-28T16:00:01.369123+0000","last_in_change":"2024-03-28T15:59:37.515483+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T15:54:55.588007+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T16:01:05.719801+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T16:01:07.206303+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7c1791b6-9c0e-4512-a327-cad166bb5581","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6803","nonce":1894681701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6805","nonce":1894681701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6809","nonce":1894681701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6807","nonce":1894681701}]},"public_addr":"172.21.15.67:6803/1894681701","cluster_addr":"172.21.15.67:6805/1894681701","heartbeat_back_addr":"172.21.15.67:6809/1894681701","heartbeat_front_addr":"172.21.15.67:6807/1894681701","state":["exists","up"]},{"osd":1,"uuid":"0cdda7f3-d00c-4436-bcc1-49fb8072b2e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6801","nonce":1734465762}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6803","nonce":1734465762}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6807","nonce":1734465762}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6805","nonce":1734465762}]},"public_addr":"172.21.15.150:6801/1734465762","cluster_addr":"172.21.15.150:6803/1734465762","heartbeat_back_addr":"172.21.15.150:6807/1734465762","heartbeat_front_addr":"172.21.15.150:6805/1734465762","state":["exists","up"]},{"osd":2,"uuid":"a55b9a02-c3fd-4061-95c0-53a24a17ade0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6801","nonce":3398095931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6803","nonce":3398095931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6807","nonce":3398095931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6805","nonce":3398095931}]},"public_addr":"172.21.15.183:6801/3398095931","cluster_addr":"172.21.15.183:6803/3398095931","heartbeat_back_addr":"172.21.15.183:6807/3398095931","heartbeat_front_addr":"172.21.15.183:6805/3398095931","state":["exists","up"]},{"osd":3,"uuid":"d34940c0-5ce9-4b8b-aa36-4872414a043f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6811","nonce":1933541875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6813","nonce":1933541875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6817","nonce":1933541875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6815","nonce":1933541875}]},"public_addr":"172.21.15.67:6811/1933541875","cluster_addr":"172.21.15.67:6813/1933541875","heartbeat_back_addr":"172.21.15.67:6817/1933541875","heartbeat_front_addr":"172.21.15.67:6815/1933541875","state":["exists","up"]},{"osd":4,"uuid":"5cc3e260-83a3-4a4d-aa6a-34a3b12b2b5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6809","nonce":3775269993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6811","nonce":3775269993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6815","nonce":3775269993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6813","nonce":3775269993}]},"public_addr":"172.21.15.150:6809/3775269993","cluster_addr":"172.21.15.150:6811/3775269993","heartbeat_back_addr":"172.21.15.150:6815/3775269993","heartbeat_front_addr":"172.21.15.150:6813/3775269993","state":["exists","up"]},{"osd":5,"uuid":"a2542b21-d6ac-413e-8b9b-4a6548b20840","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6809","nonce":164284137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6811","nonce":164284137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6815","nonce":164284137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6813","nonce":164284137}]},"public_addr":"172.21.15.183:6809/164284137","cluster_addr":"172.21.15.183:6811/164284137","heartbeat_back_addr":"172.21.15.183:6815/164284137","heartbeat_front_addr":"172.21.15.183:6813/164284137","state":["exists","up"]},{"osd":6,"uuid":"8905a2d3-0f1b-4853-a218-22da2635150a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6819","nonce":3565528013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6821","nonce":3565528013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6825","nonce":3565528013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6823","nonce":3565528013}]},"public_addr":"172.21.15.67:6819/3565528013","cluster_addr":"172.21.15.67:6821/3565528013","heartbeat_back_addr":"172.21.15.67:6825/3565528013","heartbeat_front_addr":"172.21.15.67:6823/3565528013","state":["exists","up"]},{"osd":7,"uuid":"b275055b-b451-4180-91b1-e6e61a828c48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6817","nonce":3735115340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6819","nonce":3735115340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6823","nonce":3735115340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6821","nonce":3735115340}]},"public_addr":"172.21.15.150:6817/3735115340","cluster_addr":"172.21.15.150:6819/3735115340","heartbeat_back_addr":"172.21.15.150:6823/3735115340","heartbeat_front_addr":"172.21.15.150:6821/3735115340","state":["exists","up"]},{"osd":8,"uuid":"b6abf7ad-eb34-4f60-a4eb-348a92d7195b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6817","nonce":1025400095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6819","nonce":1025400095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6823","nonce":1025400095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6821","nonce":1025400095}]},"public_addr":"172.21.15.183:6817/1025400095","cluster_addr":"172.21.15.183:6819/1025400095","heartbeat_back_addr":"172.21.15.183:6823/1025400095","heartbeat_front_addr":"172.21.15.183:6821/1025400095","state":["exists","up"]},{"osd":9,"uuid":"28a78e3a-0657-40aa-816f-16a6fadadcbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6827","nonce":76885409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6829","nonce":76885409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6833","nonce":76885409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6831","nonce":76885409}]},"public_addr":"172.21.15.67:6827/76885409","cluster_addr":"172.21.15.67:6829/76885409","heartbeat_back_addr":"172.21.15.67:6833/76885409","heartbeat_front_addr":"172.21.15.67:6831/76885409","state":["exists","up"]},{"osd":10,"uuid":"d287907e-424f-4b49-a06e-4613c6de0474","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6825","nonce":3489982080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6827","nonce":3489982080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6831","nonce":3489982080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6829","nonce":3489982080}]},"public_addr":"172.21.15.150:6825/3489982080","cluster_addr":"172.21.15.150:6827/3489982080","heartbeat_back_addr":"172.21.15.150:6831/3489982080","heartbeat_front_addr":"172.21.15.150:6829/3489982080","state":["exists","up"]},{"osd":11,"uuid":"a3c29c47-a228-4f42-a23c-fbdb0c093699","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":85,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6825","nonce":1892639849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6826","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6827","nonce":1892639849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6831","nonce":1892639849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6829","nonce":1892639849}]},"public_addr":"172.21.15.183:6825/1892639849","cluster_addr":"172.21.15.183:6827/1892639849","heartbeat_back_addr":"172.21.15.183:6831/1892639849","heartbeat_front_addr":"172.21.15.183:6829/1892639849","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:53:45.646211+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:54:18.956590+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:54:52.161646+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:55:27.502822+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:56:01.162932+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:56:34.674452+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:57:09.298679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:57:43.230406+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:58:17.003821+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:58:51.085863+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:59:25.665065+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T16:00:00.263457+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":1}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6801/1631001968":"2024-03-29T15:51:38.875334+0000","172.21.15.67:6800/1631001968":"2024-03-29T15:51:38.875334+0000","172.21.15.67:0/1533056310":"2024-03-29T15:51:38.875334+0000","172.21.15.67:6801/3580491071":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/2030412458":"2024-03-29T15:51:03.859740+0000","172.21.15.67:6800/3580491071":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/2905307533":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/3763759988":"2024-03-29T15:51:38.875334+0000","172.21.15.67:0/4089399880":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/4239108661":"2024-03-29T15:51:38.875334+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T16:03:27.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.061+0000 7fa433e00640 1 -- 172.21.15.67:0/3678841440 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fa4180a2800 msgr2=0x7fa4180a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:27.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.061+0000 7fa433e00640 1 --2- 172.21.15.67:0/3678841440 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fa4180a2800 0x7fa4180a4cc0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fa420000c00 tx=0x7fa420005be0 comp rx=0 tx=0).stop 2024-03-28T16:03:27.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.061+0000 7fa433e00640 1 -- 172.21.15.67:0/3678841440 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa434107b00 msgr2=0x7fa43419dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:27.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.061+0000 7fa433e00640 1 --2- 172.21.15.67:0/3678841440 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa434107b00 0x7fa43419dd00 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fa42400d770 tx=0x7fa42400dc30 comp rx=0 tx=0).stop 2024-03-28T16:03:27.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.061+0000 7fa433e00640 1 -- 172.21.15.67:0/3678841440 shutdown_connections 2024-03-28T16:03:27.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.061+0000 7fa433e00640 1 --2- 172.21.15.67:0/3678841440 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fa4180a2800 0x7fa4180a4cc0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:27.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.061+0000 7fa433e00640 1 --2- 172.21.15.67:0/3678841440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4341096b0 0x7fa4341a5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:27.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.061+0000 7fa433e00640 1 --2- 172.21.15.67:0/3678841440 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa434108cf0 0x7fa43419e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:27.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.061+0000 7fa433e00640 1 --2- 172.21.15.67:0/3678841440 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa434107b00 0x7fa43419dd00 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:27.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.061+0000 7fa433e00640 1 -- 172.21.15.67:0/3678841440 >> 172.21.15.67:0/3678841440 conn(0x7fa434076ab0 msgr2=0x7fa43410c040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:27.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.061+0000 7fa433e00640 1 -- 172.21.15.67:0/3678841440 shutdown_connections 2024-03-28T16:03:27.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.061+0000 7fa433e00640 1 -- 172.21.15.67:0/3678841440 wait complete. 2024-03-28T16:03:27.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:26 smithi067 ceph-mon[31707]: pgmap v435: 97 pgs: 97 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 170 B/s wr, 2 op/s 2024-03-28T16:03:27.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:26 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/3671695072' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T16:03:27.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:26 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/1824313103' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T16:03:27.082 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T15:54:55.588007+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T16:01:05.719801+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T16:01:07.206303+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T16:03:27.082 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T16:03:27.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:26 smithi183 ceph-mon[35580]: pgmap v435: 97 pgs: 97 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 170 B/s wr, 2 op/s 2024-03-28T16:03:27.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:26 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/3671695072' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T16:03:27.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:26 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/1824313103' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T16:03:27.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.303+0000 7f4f82600640 1 -- 172.21.15.67:0/3467775040 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f7c076060 msgr2=0x7f4f7c076480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:27.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.303+0000 7f4f82600640 1 --2- 172.21.15.67:0/3467775040 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f7c076060 0x7f4f7c076480 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f4f70009900 tx=0x7f4f7002bc30 comp rx=0 tx=0).stop 2024-03-28T16:03:27.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.303+0000 7f4f82600640 1 -- 172.21.15.67:0/3467775040 shutdown_connections 2024-03-28T16:03:27.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.303+0000 7f4f82600640 1 --2- 172.21.15.67:0/3467775040 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f7c077c40 0x7f4f7c0784e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:27.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.303+0000 7f4f82600640 1 --2- 172.21.15.67:0/3467775040 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4f7c0772a0 0x7f4f7c077700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:27.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.303+0000 7f4f82600640 1 --2- 172.21.15.67:0/3467775040 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f7c076060 0x7f4f7c076480 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:27.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.303+0000 7f4f82600640 1 -- 172.21.15.67:0/3467775040 >> 172.21.15.67:0/3467775040 conn(0x7f4f7c0fed10 msgr2=0x7f4f7c101150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:27.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.304+0000 7f4f82600640 1 -- 172.21.15.67:0/3467775040 shutdown_connections 2024-03-28T16:03:27.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.304+0000 7f4f82600640 1 -- 172.21.15.67:0/3467775040 wait complete. 2024-03-28T16:03:27.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.304+0000 7f4f82600640 1 Processor -- start 2024-03-28T16:03:27.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.305+0000 7f4f82600640 1 -- start start 2024-03-28T16:03:27.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.305+0000 7f4f82600640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f7c076060 0x7f4f7c19ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:27.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.305+0000 7f4f82600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4f7c0772a0 0x7f4f7c19e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:27.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.305+0000 7f4f82600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f7c077c40 0x7f4f7c1a5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:27.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.305+0000 7f4f82600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7c113720 con 0x7f4f7c077c40 2024-03-28T16:03:27.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.305+0000 7f4f82600640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7c1135a0 con 0x7f4f7c076060 2024-03-28T16:03:27.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.305+0000 7f4f82600640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7c1138a0 con 0x7f4f7c0772a0 2024-03-28T16:03:27.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.305+0000 7f4f81c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f7c077c40 0x7f4f7c1a5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:27.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.305+0000 7f4f81c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f7c077c40 0x7f4f7c1a5380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53922/0 (socket says 172.21.15.67:53922) 2024-03-28T16:03:27.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.305+0000 7f4f81c00640 1 -- 172.21.15.67:0/4006049804 learned_addr learned my addr 172.21.15.67:0/4006049804 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:03:27.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.306+0000 7f4f81200640 1 --2- 172.21.15.67:0/4006049804 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f7c076060 0x7f4f7c19ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:27.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.306+0000 7f4f7be00640 1 --2- 172.21.15.67:0/4006049804 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4f7c0772a0 0x7f4f7c19e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:27.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.306+0000 7f4f81c00640 1 -- 172.21.15.67:0/4006049804 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4f7c0772a0 msgr2=0x7f4f7c19e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:27.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.306+0000 7f4f81c00640 1 --2- 172.21.15.67:0/4006049804 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4f7c0772a0 0x7f4f7c19e310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:27.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.306+0000 7f4f81c00640 1 -- 172.21.15.67:0/4006049804 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f7c076060 msgr2=0x7f4f7c19ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:27.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.306+0000 7f4f81c00640 1 --2- 172.21.15.67:0/4006049804 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f7c076060 0x7f4f7c19ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:27.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.306+0000 7f4f81c00640 1 -- 172.21.15.67:0/4006049804 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f7c1a59f0 con 0x7f4f7c077c40 2024-03-28T16:03:27.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.306+0000 7f4f81c00640 1 --2- 172.21.15.67:0/4006049804 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f7c077c40 0x7f4f7c1a5380 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f4f7400eb10 tx=0x7f4f7400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:27.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.306+0000 7f4f79600640 1 -- 172.21.15.67:0/4006049804 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f74004b20 con 0x7f4f7c077c40 2024-03-28T16:03:27.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.306+0000 7f4f79600640 1 -- 172.21.15.67:0/4006049804 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4f74004cc0 con 0x7f4f7c077c40 2024-03-28T16:03:27.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.306+0000 7f4f79600640 1 -- 172.21.15.67:0/4006049804 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f74010690 con 0x7f4f7c077c40 2024-03-28T16:03:27.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.306+0000 7f4f82600640 1 -- 172.21.15.67:0/4006049804 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f7c1a5ce0 con 0x7f4f7c077c40 2024-03-28T16:03:27.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.307+0000 7f4f82600640 1 -- 172.21.15.67:0/4006049804 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f7c1a6180 con 0x7f4f7c077c40 2024-03-28T16:03:27.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.308+0000 7f4f82600640 1 -- 172.21.15.67:0/4006049804 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f7c110110 con 0x7f4f7c077c40 2024-03-28T16:03:27.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.310+0000 7f4f79600640 1 -- 172.21.15.67:0/4006049804 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f4f74005000 con 0x7f4f7c077c40 2024-03-28T16:03:27.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.311+0000 7f4f79600640 1 --2- 172.21.15.67:0/4006049804 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f4f640a2800 0x7f4f640a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:27.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.312+0000 7f4f79600640 1 -- 172.21.15.67:0/4006049804 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f4f740cf3a0 con 0x7f4f7c077c40 2024-03-28T16:03:27.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.312+0000 7f4f81200640 1 --2- 172.21.15.67:0/4006049804 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f4f640a2800 0x7f4f640a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:27.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.312+0000 7f4f81200640 1 --2- 172.21.15.67:0/4006049804 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f4f640a2800 0x7f4f640a4cc0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4f70006010 tx=0x7f4f70000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:27.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.314+0000 7f4f79600640 1 -- 172.21.15.67:0/4006049804 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4f74053740 con 0x7f4f7c077c40 2024-03-28T16:03:27.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.517+0000 7f4f82600640 1 -- 172.21.15.67:0/4006049804 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4f7c0622a0 con 0x7f4f7c077c40 2024-03-28T16:03:27.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.518+0000 7f4f79600640 1 -- 172.21.15.67:0/4006049804 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v92) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4f7409c3f0 con 0x7f4f7c077c40 2024-03-28T16:03:27.519 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-03-28T16:03:27.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.520+0000 7f4f82600640 1 -- 172.21.15.67:0/4006049804 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f4f640a2800 msgr2=0x7f4f640a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:27.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.520+0000 7f4f82600640 1 --2- 172.21.15.67:0/4006049804 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f4f640a2800 0x7f4f640a4cc0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4f70006010 tx=0x7f4f70000bc0 comp rx=0 tx=0).stop 2024-03-28T16:03:27.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.520+0000 7f4f82600640 1 -- 172.21.15.67:0/4006049804 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f7c077c40 msgr2=0x7f4f7c1a5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:27.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.520+0000 7f4f82600640 1 --2- 172.21.15.67:0/4006049804 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f7c077c40 0x7f4f7c1a5380 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f4f7400eb10 tx=0x7f4f7400c2d0 comp rx=0 tx=0).stop 2024-03-28T16:03:27.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.520+0000 7f4f82600640 1 -- 172.21.15.67:0/4006049804 shutdown_connections 2024-03-28T16:03:27.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.520+0000 7f4f82600640 1 --2- 172.21.15.67:0/4006049804 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f4f640a2800 0x7f4f640a4cc0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:27.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.520+0000 7f4f82600640 1 --2- 172.21.15.67:0/4006049804 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f7c077c40 0x7f4f7c1a5380 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:27.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.520+0000 7f4f82600640 1 --2- 172.21.15.67:0/4006049804 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4f7c0772a0 0x7f4f7c19e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:27.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.520+0000 7f4f82600640 1 --2- 172.21.15.67:0/4006049804 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f7c076060 0x7f4f7c19ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:27.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.520+0000 7f4f82600640 1 -- 172.21.15.67:0/4006049804 >> 172.21.15.67:0/4006049804 conn(0x7f4f7c0fed10 msgr2=0x7f4f7c102070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:27.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.521+0000 7f4f82600640 1 -- 172.21.15.67:0/4006049804 shutdown_connections 2024-03-28T16:03:27.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.521+0000 7f4f82600640 1 -- 172.21.15.67:0/4006049804 wait complete. 2024-03-28T16:03:27.542 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T16:03:27.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.766+0000 7f2231200640 1 -- 172.21.15.67:0/3125709484 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f222c105370 msgr2=0x7f222c10bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:27.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.766+0000 7f2231200640 1 --2- 172.21.15.67:0/3125709484 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f222c105370 0x7f222c10bc00 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f2224009ca0 tx=0x7f222402c100 comp rx=0 tx=0).stop 2024-03-28T16:03:27.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.766+0000 7f2231200640 1 -- 172.21.15.67:0/3125709484 shutdown_connections 2024-03-28T16:03:27.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.766+0000 7f2231200640 1 --2- 172.21.15.67:0/3125709484 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f222c105370 0x7f222c10bc00 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:27.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.766+0000 7f2231200640 1 --2- 172.21.15.67:0/3125709484 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f222c1049b0 0x7f222c104e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:27.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.766+0000 7f2231200640 1 --2- 172.21.15.67:0/3125709484 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f222c1037c0 0x7f222c103be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:27.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.766+0000 7f2231200640 1 -- 172.21.15.67:0/3125709484 >> 172.21.15.67:0/3125709484 conn(0x7f222c0fed50 msgr2=0x7f222c101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:27.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.767+0000 7f2231200640 1 -- 172.21.15.67:0/3125709484 shutdown_connections 2024-03-28T16:03:27.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.767+0000 7f2231200640 1 -- 172.21.15.67:0/3125709484 wait complete. 2024-03-28T16:03:27.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.768+0000 7f2231200640 1 Processor -- start 2024-03-28T16:03:27.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.768+0000 7f2231200640 1 -- start start 2024-03-28T16:03:27.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.768+0000 7f2231200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f222c1037c0 0x7f222c199960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:27.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.769+0000 7f2231200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f222c1049b0 0x7f222c199ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:27.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.769+0000 7f2231200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f222c105370 0x7f222c1a0f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:27.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.769+0000 7f2231200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f222c10f2f0 con 0x7f222c105370 2024-03-28T16:03:27.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.769+0000 7f2231200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f222c10f170 con 0x7f222c1037c0 2024-03-28T16:03:27.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.769+0000 7f2231200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f222c10f470 con 0x7f222c1049b0 2024-03-28T16:03:27.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.769+0000 7f222be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f222c105370 0x7f222c1a0f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:27.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.769+0000 7f222be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f222c105370 0x7f222c1a0f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53932/0 (socket says 172.21.15.67:53932) 2024-03-28T16:03:27.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.769+0000 7f222be00640 1 -- 172.21.15.67:0/3176289571 learned_addr learned my addr 172.21.15.67:0/3176289571 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:03:27.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.769+0000 7f2222a00640 1 --2- 172.21.15.67:0/3176289571 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f222c1049b0 0x7f222c199ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:27.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.769+0000 7f222b400640 1 --2- 172.21.15.67:0/3176289571 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f222c1037c0 0x7f222c199960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:27.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.769+0000 7f222be00640 1 -- 172.21.15.67:0/3176289571 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f222c1049b0 msgr2=0x7f222c199ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:27.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.769+0000 7f222be00640 1 --2- 172.21.15.67:0/3176289571 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f222c1049b0 0x7f222c199ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:27.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.769+0000 7f222be00640 1 -- 172.21.15.67:0/3176289571 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f222c1037c0 msgr2=0x7f222c199960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:27.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.769+0000 7f222be00640 1 --2- 172.21.15.67:0/3176289571 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f222c1037c0 0x7f222c199960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:27.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.769+0000 7f222be00640 1 -- 172.21.15.67:0/3176289571 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f222c1a1580 con 0x7f222c105370 2024-03-28T16:03:27.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.770+0000 7f222b400640 1 --2- 172.21.15.67:0/3176289571 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f222c1037c0 0x7f222c199960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T16:03:27.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.770+0000 7f222be00640 1 --2- 172.21.15.67:0/3176289571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f222c105370 0x7f222c1a0f10 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f2224032520 tx=0x7f2224034880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:27.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.770+0000 7f2228c00640 1 -- 172.21.15.67:0/3176289571 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2224039040 con 0x7f222c105370 2024-03-28T16:03:27.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.770+0000 7f2228c00640 1 -- 172.21.15.67:0/3176289571 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f2224044070 con 0x7f222c105370 2024-03-28T16:03:27.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.770+0000 7f2231200640 1 -- 172.21.15.67:0/3176289571 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f222c1a1810 con 0x7f222c105370 2024-03-28T16:03:27.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.770+0000 7f2228c00640 1 -- 172.21.15.67:0/3176289571 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22240043c0 con 0x7f222c105370 2024-03-28T16:03:27.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.770+0000 7f2231200640 1 -- 172.21.15.67:0/3176289571 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f222c1a1d00 con 0x7f222c105370 2024-03-28T16:03:27.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.772+0000 7f2231200640 1 -- 172.21.15.67:0/3176289571 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f222c10aae0 con 0x7f222c105370 2024-03-28T16:03:27.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.774+0000 7f2228c00640 1 -- 172.21.15.67:0/3176289571 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f2224004560 con 0x7f222c105370 2024-03-28T16:03:27.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.775+0000 7f2228c00640 1 --2- 172.21.15.67:0/3176289571 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f22100a2800 0x7f22100a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:27.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.775+0000 7f2228c00640 1 -- 172.21.15.67:0/3176289571 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f22240f1190 con 0x7f222c105370 2024-03-28T16:03:27.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.775+0000 7f222b400640 1 --2- 172.21.15.67:0/3176289571 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f22100a2800 0x7f22100a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:27.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.776+0000 7f222b400640 1 --2- 172.21.15.67:0/3176289571 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f22100a2800 0x7f22100a4cc0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f221c0058c0 tx=0x7f221c005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:27.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.780+0000 7f2228c00640 1 -- 172.21.15.67:0/3176289571 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f22240be1e0 con 0x7f222c105370 2024-03-28T16:03:27.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.985+0000 7f2231200640 1 -- 172.21.15.67:0/3176289571 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f222c0622a0 con 0x7f222c105370 2024-03-28T16:03:27.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.986+0000 7f2228c00640 1 -- 172.21.15.67:0/3176289571 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v92) v1 ==== 104+0+11 (secure 0 0 0) 0x7f22240bdae0 con 0x7f222c105370 2024-03-28T16:03:27.987 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-03-28T16:03:27.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.989+0000 7f2231200640 1 -- 172.21.15.67:0/3176289571 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f22100a2800 msgr2=0x7f22100a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:27.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.989+0000 7f2231200640 1 --2- 172.21.15.67:0/3176289571 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f22100a2800 0x7f22100a4cc0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f221c0058c0 tx=0x7f221c005850 comp rx=0 tx=0).stop 2024-03-28T16:03:27.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.989+0000 7f2231200640 1 -- 172.21.15.67:0/3176289571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f222c105370 msgr2=0x7f222c1a0f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:27.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.989+0000 7f2231200640 1 --2- 172.21.15.67:0/3176289571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f222c105370 0x7f222c1a0f10 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f2224032520 tx=0x7f2224034880 comp rx=0 tx=0).stop 2024-03-28T16:03:27.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.989+0000 7f2231200640 1 -- 172.21.15.67:0/3176289571 shutdown_connections 2024-03-28T16:03:27.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.989+0000 7f2231200640 1 --2- 172.21.15.67:0/3176289571 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f22100a2800 0x7f22100a4cc0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:27.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.989+0000 7f2231200640 1 --2- 172.21.15.67:0/3176289571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f222c105370 0x7f222c1a0f10 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:27.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.989+0000 7f2231200640 1 --2- 172.21.15.67:0/3176289571 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f222c1049b0 0x7f222c199ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:27.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.989+0000 7f2231200640 1 --2- 172.21.15.67:0/3176289571 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f222c1037c0 0x7f222c199960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:27.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.989+0000 7f2231200640 1 -- 172.21.15.67:0/3176289571 >> 172.21.15.67:0/3176289571 conn(0x7f222c0fed50 msgr2=0x7f222c107cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:27.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.989+0000 7f2231200640 1 -- 172.21.15.67:0/3176289571 shutdown_connections 2024-03-28T16:03:27.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:27.989+0000 7f2231200640 1 -- 172.21.15.67:0/3176289571 wait complete. 2024-03-28T16:03:28.011 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T16:03:28.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:27 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/3678841440' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:03:28.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:27 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/4006049804' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T16:03:28.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:27 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/3678841440' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:03:28.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:27 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/4006049804' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T16:03:28.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:27 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/3678841440' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:03:28.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:27 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/4006049804' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T16:03:28.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.240+0000 7eff02c00640 1 -- 172.21.15.67:0/4070376660 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efefc100480 msgr2=0x7efefc1008a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:28.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.240+0000 7eff02c00640 1 --2- 172.21.15.67:0/4070376660 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efefc100480 0x7efefc1008a0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7efef0009900 tx=0x7efef002bc30 comp rx=0 tx=0).stop 2024-03-28T16:03:28.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.240+0000 7eff02c00640 1 -- 172.21.15.67:0/4070376660 shutdown_connections 2024-03-28T16:03:28.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.240+0000 7eff02c00640 1 --2- 172.21.15.67:0/4070376660 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efefc102030 0x7efefc1088c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:28.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.240+0000 7eff02c00640 1 --2- 172.21.15.67:0/4070376660 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efefc101670 0x7efefc101af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:28.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.240+0000 7eff02c00640 1 --2- 172.21.15.67:0/4070376660 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efefc100480 0x7efefc1008a0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:28.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.240+0000 7eff02c00640 1 -- 172.21.15.67:0/4070376660 >> 172.21.15.67:0/4070376660 conn(0x7efefc0fba10 msgr2=0x7efefc0fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:28.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.240+0000 7eff02c00640 1 -- 172.21.15.67:0/4070376660 shutdown_connections 2024-03-28T16:03:28.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.240+0000 7eff02c00640 1 -- 172.21.15.67:0/4070376660 wait complete. 2024-03-28T16:03:28.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.241+0000 7eff02c00640 1 Processor -- start 2024-03-28T16:03:28.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.242+0000 7eff02c00640 1 -- start start 2024-03-28T16:03:28.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.242+0000 7eff02c00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efefc100480 0x7efefc197800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:28.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.242+0000 7eff02c00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efefc101670 0x7efefc197d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:28.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.242+0000 7eff02c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efefc102030 0x7efefc19edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:28.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.242+0000 7eff02c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efefc10c010 con 0x7efefc102030 2024-03-28T16:03:28.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.242+0000 7eff02c00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efefc10be90 con 0x7efefc100480 2024-03-28T16:03:28.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.242+0000 7eff02c00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efefc10c190 con 0x7efefc101670 2024-03-28T16:03:28.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.242+0000 7eff02200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efefc102030 0x7efefc19edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:28.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.242+0000 7eff02200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efefc102030 0x7efefc19edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53948/0 (socket says 172.21.15.67:53948) 2024-03-28T16:03:28.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.242+0000 7eff02200640 1 -- 172.21.15.67:0/2971395460 learned_addr learned my addr 172.21.15.67:0/2971395460 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:03:28.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.243+0000 7eff00e00640 1 --2- 172.21.15.67:0/2971395460 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efefc101670 0x7efefc197d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:28.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.243+0000 7eff01800640 1 --2- 172.21.15.67:0/2971395460 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efefc100480 0x7efefc197800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:28.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.243+0000 7eff02200640 1 -- 172.21.15.67:0/2971395460 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efefc101670 msgr2=0x7efefc197d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:28.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.243+0000 7eff02200640 1 --2- 172.21.15.67:0/2971395460 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efefc101670 0x7efefc197d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:28.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.243+0000 7eff02200640 1 -- 172.21.15.67:0/2971395460 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efefc100480 msgr2=0x7efefc197800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:28.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.243+0000 7eff02200640 1 --2- 172.21.15.67:0/2971395460 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efefc100480 0x7efefc197800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:28.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.243+0000 7eff02200640 1 -- 172.21.15.67:0/2971395460 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efefc19f420 con 0x7efefc102030 2024-03-28T16:03:28.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.243+0000 7eff02200640 1 --2- 172.21.15.67:0/2971395460 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efefc102030 0x7efefc19edb0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7efef400eaa0 tx=0x7efef400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:28.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.243+0000 7efefa000640 1 -- 172.21.15.67:0/2971395460 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef400cdb0 con 0x7efefc102030 2024-03-28T16:03:28.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.244+0000 7efefa000640 1 -- 172.21.15.67:0/2971395460 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7efef4005020 con 0x7efefc102030 2024-03-28T16:03:28.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.244+0000 7eff02c00640 1 -- 172.21.15.67:0/2971395460 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efefc19f710 con 0x7efefc102030 2024-03-28T16:03:28.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.244+0000 7efefa000640 1 -- 172.21.15.67:0/2971395460 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef40116d0 con 0x7efefc102030 2024-03-28T16:03:28.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.244+0000 7eff02c00640 1 -- 172.21.15.67:0/2971395460 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efefc19fc10 con 0x7efefc102030 2024-03-28T16:03:28.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.245+0000 7eff02c00640 1 -- 172.21.15.67:0/2971395460 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efefc19fed0 con 0x7efefc102030 2024-03-28T16:03:28.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.247+0000 7efefa000640 1 -- 172.21.15.67:0/2971395460 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7efef40051d0 con 0x7efefc102030 2024-03-28T16:03:28.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.248+0000 7efefa000640 1 --2- 172.21.15.67:0/2971395460 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7efee40a2800 0x7efee40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:28.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.248+0000 7efefa000640 1 -- 172.21.15.67:0/2971395460 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7efef40cf130 con 0x7efefc102030 2024-03-28T16:03:28.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.248+0000 7eff01800640 1 --2- 172.21.15.67:0/2971395460 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7efee40a2800 0x7efee40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:28.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.249+0000 7eff01800640 1 --2- 172.21.15.67:0/2971395460 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7efee40a2800 0x7efee40a4cc0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7efef002c240 tx=0x7efef0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:28.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.251+0000 7efefa000640 1 -- 172.21.15.67:0/2971395460 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7efef4016030 con 0x7efefc102030 2024-03-28T16:03:28.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.455+0000 7eff02c00640 1 -- 172.21.15.67:0/2971395460 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7efefc1a00a0 con 0x7efefc102030 2024-03-28T16:03:28.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.456+0000 7efefa000640 1 -- 172.21.15.67:0/2971395460 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) v1 ==== 100+0+11 (secure 0 0 0) 0x7efef409c180 con 0x7efefc102030 2024-03-28T16:03:28.457 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-03-28T16:03:28.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.459+0000 7eff02c00640 1 -- 172.21.15.67:0/2971395460 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7efee40a2800 msgr2=0x7efee40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:28.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.459+0000 7eff02c00640 1 --2- 172.21.15.67:0/2971395460 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7efee40a2800 0x7efee40a4cc0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7efef002c240 tx=0x7efef0005fd0 comp rx=0 tx=0).stop 2024-03-28T16:03:28.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.459+0000 7eff02c00640 1 -- 172.21.15.67:0/2971395460 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efefc102030 msgr2=0x7efefc19edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:28.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.459+0000 7eff02c00640 1 --2- 172.21.15.67:0/2971395460 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efefc102030 0x7efefc19edb0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7efef400eaa0 tx=0x7efef400ef60 comp rx=0 tx=0).stop 2024-03-28T16:03:28.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.459+0000 7eff02c00640 1 -- 172.21.15.67:0/2971395460 shutdown_connections 2024-03-28T16:03:28.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.459+0000 7eff02c00640 1 --2- 172.21.15.67:0/2971395460 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7efee40a2800 0x7efee40a4cc0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:28.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.459+0000 7eff02c00640 1 --2- 172.21.15.67:0/2971395460 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efefc102030 0x7efefc19edb0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:28.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.459+0000 7eff02c00640 1 --2- 172.21.15.67:0/2971395460 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efefc101670 0x7efefc197d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:28.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.459+0000 7eff02c00640 1 --2- 172.21.15.67:0/2971395460 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efefc100480 0x7efefc197800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:28.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.459+0000 7eff02c00640 1 -- 172.21.15.67:0/2971395460 >> 172.21.15.67:0/2971395460 conn(0x7efefc0fba10 msgr2=0x7efefc068800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:28.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.459+0000 7eff02c00640 1 -- 172.21.15.67:0/2971395460 shutdown_connections 2024-03-28T16:03:28.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.460+0000 7eff02c00640 1 -- 172.21.15.67:0/2971395460 wait complete. 2024-03-28T16:03:28.482 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-03-28T16:03:28.482 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T16:03:28.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.702+0000 7f98d4e00640 1 -- 172.21.15.67:0/327036663 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98d01052a0 msgr2=0x7f98d010bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:28.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.702+0000 7f98d4e00640 1 --2- 172.21.15.67:0/327036663 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98d01052a0 0x7f98d010bb30 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f98c8009ca0 tx=0x7f98c802c0f0 comp rx=0 tx=0).stop 2024-03-28T16:03:28.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.702+0000 7f98d4e00640 1 -- 172.21.15.67:0/327036663 shutdown_connections 2024-03-28T16:03:28.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.702+0000 7f98d4e00640 1 --2- 172.21.15.67:0/327036663 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98d01052a0 0x7f98d010bb30 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:28.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.702+0000 7f98d4e00640 1 --2- 172.21.15.67:0/327036663 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f98d01048e0 0x7f98d0104d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:28.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.702+0000 7f98d4e00640 1 --2- 172.21.15.67:0/327036663 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f98d01036f0 0x7f98d0103b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:28.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.702+0000 7f98d4e00640 1 -- 172.21.15.67:0/327036663 >> 172.21.15.67:0/327036663 conn(0x7f98d00fec80 msgr2=0x7f98d01010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:28.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.702+0000 7f98d4e00640 1 -- 172.21.15.67:0/327036663 shutdown_connections 2024-03-28T16:03:28.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.702+0000 7f98d4e00640 1 -- 172.21.15.67:0/327036663 wait complete. 2024-03-28T16:03:28.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.703+0000 7f98d4e00640 1 Processor -- start 2024-03-28T16:03:28.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.703+0000 7f98d4e00640 1 -- start start 2024-03-28T16:03:28.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.703+0000 7f98d4e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f98d01036f0 0x7f98d0199870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:28.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.704+0000 7f98d4e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98d01048e0 0x7f98d0199db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:28.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.704+0000 7f98d4e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f98d01052a0 0x7f98d01a0e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:28.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.704+0000 7f98d4e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d010f220 con 0x7f98d01048e0 2024-03-28T16:03:28.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.704+0000 7f98d4e00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d010f0a0 con 0x7f98d01052a0 2024-03-28T16:03:28.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.704+0000 7f98d4e00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d010f3a0 con 0x7f98d01036f0 2024-03-28T16:03:28.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.704+0000 7f98c7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98d01048e0 0x7f98d0199db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:28.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.704+0000 7f98c7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98d01048e0 0x7f98d0199db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53964/0 (socket says 172.21.15.67:53964) 2024-03-28T16:03:28.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.704+0000 7f98c7e00640 1 -- 172.21.15.67:0/3397472828 learned_addr learned my addr 172.21.15.67:0/3397472828 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:03:28.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.704+0000 7f98cf400640 1 --2- 172.21.15.67:0/3397472828 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f98d01036f0 0x7f98d0199870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:28.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.704+0000 7f98cfe00640 1 --2- 172.21.15.67:0/3397472828 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f98d01052a0 0x7f98d01a0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:28.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.704+0000 7f98c7e00640 1 -- 172.21.15.67:0/3397472828 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f98d01036f0 msgr2=0x7f98d0199870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:28.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.704+0000 7f98c7e00640 1 --2- 172.21.15.67:0/3397472828 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f98d01036f0 0x7f98d0199870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:28.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.704+0000 7f98c7e00640 1 -- 172.21.15.67:0/3397472828 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f98d01052a0 msgr2=0x7f98d01a0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:28.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.704+0000 7f98c7e00640 1 --2- 172.21.15.67:0/3397472828 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f98d01052a0 0x7f98d01a0e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:28.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.704+0000 7f98c7e00640 1 -- 172.21.15.67:0/3397472828 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98d01a1490 con 0x7f98d01048e0 2024-03-28T16:03:28.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.705+0000 7f98c7e00640 1 --2- 172.21.15.67:0/3397472828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98d01048e0 0x7f98d0199db0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f98b800e9f0 tx=0x7f98b800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:28.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.705+0000 7f98ccc00640 1 -- 172.21.15.67:0/3397472828 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b800cd80 con 0x7f98d01048e0 2024-03-28T16:03:28.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.705+0000 7f98ccc00640 1 -- 172.21.15.67:0/3397472828 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f98b80108e0 con 0x7f98d01048e0 2024-03-28T16:03:28.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.705+0000 7f98d4e00640 1 -- 172.21.15.67:0/3397472828 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98d01a1780 con 0x7f98d01048e0 2024-03-28T16:03:28.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.705+0000 7f98ccc00640 1 -- 172.21.15.67:0/3397472828 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b8018920 con 0x7f98d01048e0 2024-03-28T16:03:28.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.705+0000 7f98d4e00640 1 -- 172.21.15.67:0/3397472828 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98d01a1cd0 con 0x7f98d01048e0 2024-03-28T16:03:28.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.706+0000 7f98d4e00640 1 -- 172.21.15.67:0/3397472828 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98d010aa90 con 0x7f98d01048e0 2024-03-28T16:03:28.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.708+0000 7f98ccc00640 1 -- 172.21.15.67:0/3397472828 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f98b8010a90 con 0x7f98d01048e0 2024-03-28T16:03:28.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.709+0000 7f98ccc00640 1 --2- 172.21.15.67:0/3397472828 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f98b40a2800 0x7f98b40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:28.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.709+0000 7f98ccc00640 1 -- 172.21.15.67:0/3397472828 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f98b80cef00 con 0x7f98d01048e0 2024-03-28T16:03:28.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.710+0000 7f98cf400640 1 --2- 172.21.15.67:0/3397472828 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f98b40a2800 0x7f98b40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:28.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.710+0000 7f98cf400640 1 --2- 172.21.15.67:0/3397472828 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f98b40a2800 0x7f98b40a4cc0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f98d0104740 tx=0x7f98c0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:28.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.714+0000 7f98ccc00640 1 -- 172.21.15.67:0/3397472828 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f98b809bf50 con 0x7f98d01048e0 2024-03-28T16:03:28.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.962+0000 7f98d4e00640 1 -- 172.21.15.67:0/3397472828 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f98d002d070 con 0x7f98d01048e0 2024-03-28T16:03:28.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.965+0000 7f98ccc00640 1 -- 172.21.15.67:0/3397472828 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11341 (secure 0 0 0) 0x7f98b809b850 con 0x7f98d01048e0 2024-03-28T16:03:28.966 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:03:28.966 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2008297659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2008297659},{"type":"v1","addr":"172.21.15.67:6835","nonce":2008297659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14532,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/320295745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":320295745},{"type":"v1","addr":"172.21.15.67:6837","nonce":320295745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6839/691821273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":691821273},{"type":"v1","addr":"172.21.15.67:6839","nonce":691821273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6841/3211254875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6840","nonce":3211254875},{"type":"v1","addr":"172.21.15.67:6841","nonce":3211254875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2129665423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2129665423},{"type":"v1","addr":"172.21.15.183:6833","nonce":2129665423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24496,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/1589784425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":1589784425},{"type":"v1","addr":"172.21.15.183:6835","nonce":1589784425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24508,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/2185863835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":2185863835},{"type":"v1","addr":"172.21.15.183:6837","nonce":2185863835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24520,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6839/1242814398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":1242814398},{"type":"v1","addr":"172.21.15.183:6839","nonce":1242814398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":34409,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1008335521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1008335521},{"type":"v1","addr":"172.21.15.150:6833","nonce":1008335521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":34445,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3269935695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3269935695},{"type":"v1","addr":"172.21.15.150:6835","nonce":3269935695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T16:01:07.696824+0000","modified":"2024-03-28T16:01:11.997773+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":34508},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34475":{"gid":34475,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/1112886277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":1112886277},{"type":"v1","addr":"172.21.15.150:6837","nonce":1112886277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_34508":{"gid":34508,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6839/1211011614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":1211011614},{"type":"v1","addr":"172.21.15.150:6839","nonce":1211011614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34508,"qdb_cluster":[34508]},"id":1}]} 2024-03-28T16:03:28.966 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 19 2024-03-28T16:03:28.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.968+0000 7f98d4e00640 1 -- 172.21.15.67:0/3397472828 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f98b40a2800 msgr2=0x7f98b40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:28.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.968+0000 7f98d4e00640 1 --2- 172.21.15.67:0/3397472828 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f98b40a2800 0x7f98b40a4cc0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f98d0104740 tx=0x7f98c0005f90 comp rx=0 tx=0).stop 2024-03-28T16:03:28.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.968+0000 7f98d4e00640 1 -- 172.21.15.67:0/3397472828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98d01048e0 msgr2=0x7f98d0199db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:28.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.968+0000 7f98d4e00640 1 --2- 172.21.15.67:0/3397472828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98d01048e0 0x7f98d0199db0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f98b800e9f0 tx=0x7f98b800eeb0 comp rx=0 tx=0).stop 2024-03-28T16:03:28.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.968+0000 7f98d4e00640 1 -- 172.21.15.67:0/3397472828 shutdown_connections 2024-03-28T16:03:28.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.968+0000 7f98d4e00640 1 --2- 172.21.15.67:0/3397472828 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f98b40a2800 0x7f98b40a4cc0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:28.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.968+0000 7f98d4e00640 1 --2- 172.21.15.67:0/3397472828 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f98d01052a0 0x7f98d01a0e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:28.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.968+0000 7f98d4e00640 1 --2- 172.21.15.67:0/3397472828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98d01048e0 0x7f98d0199db0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:28.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.968+0000 7f98d4e00640 1 --2- 172.21.15.67:0/3397472828 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f98d01036f0 0x7f98d0199870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:28.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.968+0000 7f98d4e00640 1 -- 172.21.15.67:0/3397472828 >> 172.21.15.67:0/3397472828 conn(0x7f98d00fec80 msgr2=0x7f98d0107c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:28.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.968+0000 7f98d4e00640 1 -- 172.21.15.67:0/3397472828 shutdown_connections 2024-03-28T16:03:28.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:28.968+0000 7f98d4e00640 1 -- 172.21.15.67:0/3397472828 wait complete. 2024-03-28T16:03:28.987 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-03-28T16:03:28.988 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-03-28T16:03:28.988 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T16:03:29.066 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:28 smithi067 ceph-mon[31707]: pgmap v436: 97 pgs: 97 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 170 B/s wr, 1 op/s 2024-03-28T16:03:29.066 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:28 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/3176289571' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T16:03:29.066 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:28 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/2971395460' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T16:03:29.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:28 smithi183 ceph-mon[35580]: pgmap v436: 97 pgs: 97 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 170 B/s wr, 1 op/s 2024-03-28T16:03:29.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:28 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/3176289571' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T16:03:29.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:28 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/2971395460' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T16:03:29.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.205+0000 7fa4d7e00640 1 -- 172.21.15.67:0/3435078018 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4d81026d0 msgr2=0x7fa4d8102b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:29.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.205+0000 7fa4d7e00640 1 --2- 172.21.15.67:0/3435078018 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4d81026d0 0x7fa4d8102b50 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fa4c4009900 tx=0x7fa4c402bc30 comp rx=0 tx=0).stop 2024-03-28T16:03:29.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.205+0000 7fa4d7e00640 1 -- 172.21.15.67:0/3435078018 shutdown_connections 2024-03-28T16:03:29.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.205+0000 7fa4d7e00640 1 --2- 172.21.15.67:0/3435078018 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa4d8103090 0x7fa4d8109920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:29.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.205+0000 7fa4d7e00640 1 --2- 172.21.15.67:0/3435078018 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4d81026d0 0x7fa4d8102b50 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:29.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.205+0000 7fa4d7e00640 1 --2- 172.21.15.67:0/3435078018 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4d81014e0 0x7fa4d8101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:29.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.205+0000 7fa4d7e00640 1 -- 172.21.15.67:0/3435078018 >> 172.21.15.67:0/3435078018 conn(0x7fa4d80fca30 msgr2=0x7fa4d80fee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:29.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.205+0000 7fa4d7e00640 1 -- 172.21.15.67:0/3435078018 shutdown_connections 2024-03-28T16:03:29.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.205+0000 7fa4d7e00640 1 -- 172.21.15.67:0/3435078018 wait complete. 2024-03-28T16:03:29.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.207+0000 7fa4d7e00640 1 Processor -- start 2024-03-28T16:03:29.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.207+0000 7fa4d7e00640 1 -- start start 2024-03-28T16:03:29.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.207+0000 7fa4d7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4d81014e0 0x7fa4d81999c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:29.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.207+0000 7fa4d7e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4d81026d0 0x7fa4d8199f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:29.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.207+0000 7fa4d7e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa4d8103090 0x7fa4d81a0f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:29.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.207+0000 7fa4d7e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d810d0b0 con 0x7fa4d81014e0 2024-03-28T16:03:29.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.207+0000 7fa4d7e00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d810cf30 con 0x7fa4d81026d0 2024-03-28T16:03:29.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.207+0000 7fa4d7e00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d810d230 con 0x7fa4d8103090 2024-03-28T16:03:29.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.208+0000 7fa4d6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4d81014e0 0x7fa4d81999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:29.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.208+0000 7fa4d6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4d81014e0 0x7fa4d81999c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53980/0 (socket says 172.21.15.67:53980) 2024-03-28T16:03:29.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.208+0000 7fa4d6a00640 1 -- 172.21.15.67:0/2768428942 learned_addr learned my addr 172.21.15.67:0/2768428942 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:03:29.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.208+0000 7fa4d6000640 1 --2- 172.21.15.67:0/2768428942 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4d81026d0 0x7fa4d8199f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:29.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.208+0000 7fa4d6a00640 1 -- 172.21.15.67:0/2768428942 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa4d8103090 msgr2=0x7fa4d81a0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:29.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.208+0000 7fa4d7400640 1 --2- 172.21.15.67:0/2768428942 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa4d8103090 0x7fa4d81a0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:29.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.208+0000 7fa4d6a00640 1 --2- 172.21.15.67:0/2768428942 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa4d8103090 0x7fa4d81a0f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:29.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.208+0000 7fa4d6a00640 1 -- 172.21.15.67:0/2768428942 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4d81026d0 msgr2=0x7fa4d8199f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:29.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.208+0000 7fa4d6a00640 1 --2- 172.21.15.67:0/2768428942 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4d81026d0 0x7fa4d8199f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:29.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.208+0000 7fa4d6a00640 1 -- 172.21.15.67:0/2768428942 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4d81a15e0 con 0x7fa4d81014e0 2024-03-28T16:03:29.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.208+0000 7fa4d6000640 1 --2- 172.21.15.67:0/2768428942 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4d81026d0 0x7fa4d8199f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:03:29.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.208+0000 7fa4d7400640 1 --2- 172.21.15.67:0/2768428942 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa4d8103090 0x7fa4d81a0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:03:29.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.209+0000 7fa4d6a00640 1 --2- 172.21.15.67:0/2768428942 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4d81014e0 0x7fa4d81999c0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fa4c800d770 tx=0x7fa4c800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:29.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.209+0000 7fa4cf400640 1 -- 172.21.15.67:0/2768428942 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4c8017070 con 0x7fa4d81014e0 2024-03-28T16:03:29.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.209+0000 7fa4cf400640 1 -- 172.21.15.67:0/2768428942 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa4c8009c30 con 0x7fa4d81014e0 2024-03-28T16:03:29.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.209+0000 7fa4cf400640 1 -- 172.21.15.67:0/2768428942 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4c8003d00 con 0x7fa4d81014e0 2024-03-28T16:03:29.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.209+0000 7fa4d7e00640 1 -- 172.21.15.67:0/2768428942 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4d81a18d0 con 0x7fa4d81014e0 2024-03-28T16:03:29.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.209+0000 7fa4d7e00640 1 -- 172.21.15.67:0/2768428942 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4d81a1e50 con 0x7fa4d81014e0 2024-03-28T16:03:29.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.210+0000 7fa4d7e00640 1 -- 172.21.15.67:0/2768428942 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4d81a2080 con 0x7fa4d81014e0 2024-03-28T16:03:29.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.212+0000 7fa4cf400640 1 -- 172.21.15.67:0/2768428942 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fa4c8003ea0 con 0x7fa4d81014e0 2024-03-28T16:03:29.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.215+0000 7fa4cf400640 1 --2- 172.21.15.67:0/2768428942 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fa4bc0a2880 0x7fa4bc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:29.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.215+0000 7fa4cf400640 1 -- 172.21.15.67:0/2768428942 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fa4c80ce830 con 0x7fa4d81014e0 2024-03-28T16:03:29.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.215+0000 7fa4d6000640 1 --2- 172.21.15.67:0/2768428942 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fa4bc0a2880 0x7fa4bc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:29.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.216+0000 7fa4d6000640 1 --2- 172.21.15.67:0/2768428942 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fa4bc0a2880 0x7fa4bc0a4d40 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fa4c4036040 tx=0x7fa4c4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:29.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.219+0000 7fa4cf400640 1 -- 172.21.15.67:0/2768428942 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa4c809b880 con 0x7fa4d81014e0 2024-03-28T16:03:29.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:28 smithi150 ceph-mon[34825]: pgmap v436: 97 pgs: 97 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 170 B/s wr, 1 op/s 2024-03-28T16:03:29.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:28 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/3176289571' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T16:03:29.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:28 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/2971395460' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T16:03:29.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.422+0000 7fa4d7e00640 1 -- 172.21.15.67:0/2768428942 --> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa4d8074ff0 con 0x7fa4bc0a2880 2024-03-28T16:03:29.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.447+0000 7fa4cf400640 1 -- 172.21.15.67:0/2768428942 <== mgr.14152 v2:172.21.15.67:6800/208726654 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+258821 (secure 0 0 0) 0x7fa4d8074ff0 con 0x7fa4bc0a2880 2024-03-28T16:03:29.452 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:03:29.453 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-28T16:03:29.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.454+0000 7fa4d7e00640 1 -- 172.21.15.67:0/2768428942 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fa4bc0a2880 msgr2=0x7fa4bc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:29.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.454+0000 7fa4d7e00640 1 --2- 172.21.15.67:0/2768428942 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fa4bc0a2880 0x7fa4bc0a4d40 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fa4c4036040 tx=0x7fa4c4005fd0 comp rx=0 tx=0).stop 2024-03-28T16:03:29.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.454+0000 7fa4d7e00640 1 -- 172.21.15.67:0/2768428942 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4d81014e0 msgr2=0x7fa4d81999c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:29.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.454+0000 7fa4d7e00640 1 --2- 172.21.15.67:0/2768428942 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4d81014e0 0x7fa4d81999c0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fa4c800d770 tx=0x7fa4c800dc30 comp rx=0 tx=0).stop 2024-03-28T16:03:29.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.454+0000 7fa4d7e00640 1 -- 172.21.15.67:0/2768428942 shutdown_connections 2024-03-28T16:03:29.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.454+0000 7fa4d7e00640 1 --2- 172.21.15.67:0/2768428942 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fa4bc0a2880 0x7fa4bc0a4d40 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:29.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.454+0000 7fa4d7e00640 1 --2- 172.21.15.67:0/2768428942 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa4d8103090 0x7fa4d81a0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:29.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.454+0000 7fa4d7e00640 1 --2- 172.21.15.67:0/2768428942 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4d81026d0 0x7fa4d8199f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:29.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.454+0000 7fa4d7e00640 1 --2- 172.21.15.67:0/2768428942 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4d81014e0 0x7fa4d81999c0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:29.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.454+0000 7fa4d7e00640 1 -- 172.21.15.67:0/2768428942 >> 172.21.15.67:0/2768428942 conn(0x7fa4d80fca30 msgr2=0x7fa4d80738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:29.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.454+0000 7fa4d7e00640 1 -- 172.21.15.67:0/2768428942 shutdown_connections 2024-03-28T16:03:29.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.454+0000 7fa4d7e00640 1 -- 172.21.15.67:0/2768428942 wait complete. 2024-03-28T16:03:29.473 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":437,"stamp":"2024-03-28T16:03:29.097067+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":654038,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":407,"num_read_kb":313,"num_write":271,"num_write_kb":3031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1180758,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":712,"ondisk_log_size":712,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":383608,"kb_used_data":13656,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124558216,"statfs":{"total":1151940427776,"available":1151547613184,"internally_reserved":0,"allocated":13983744,"data_stored":7708038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":378713402},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":493,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":11,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004831"},"pg_stats":[{"pgid":"3.3f","version":"92'8","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.049941+0000","last_change":"2024-03-28T16:01:08.268196+0000","last_active":"2024-03-28T16:02:39.049941+0000","last_peered":"2024-03-28T16:02:39.049941+0000","last_clean":"2024-03-28T16:02:39.049941+0000","last_became_active":"2024-03-28T16:01:08.266956+0000","last_became_peered":"2024-03-28T16:01:08.266956+0000","last_unstale":"2024-03-28T16:02:39.049941+0000","last_undegraded":"2024-03-28T16:02:39.049941+0000","last_fullsized":"2024-03-28T16:02:39.049941+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:38:44.296129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"92'4","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.048446+0000","last_change":"2024-03-28T16:01:08.264985+0000","last_active":"2024-03-28T16:02:39.048446+0000","last_peered":"2024-03-28T16:02:39.048446+0000","last_clean":"2024-03-28T16:02:39.048446+0000","last_became_active":"2024-03-28T16:01:08.262484+0000","last_became_peered":"2024-03-28T16:01:08.262484+0000","last_unstale":"2024-03-28T16:02:39.048446+0000","last_undegraded":"2024-03-28T16:02:39.048446+0000","last_fullsized":"2024-03-28T16:02:39.048446+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:33:07.538908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"92'7","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.051488+0000","last_change":"2024-03-28T16:01:08.267423+0000","last_active":"2024-03-28T16:02:39.051488+0000","last_peered":"2024-03-28T16:02:39.051488+0000","last_clean":"2024-03-28T16:02:39.051488+0000","last_became_active":"2024-03-28T16:01:08.263665+0000","last_became_peered":"2024-03-28T16:01:08.263665+0000","last_unstale":"2024-03-28T16:02:39.051488+0000","last_undegraded":"2024-03-28T16:02:39.051488+0000","last_fullsized":"2024-03-28T16:02:39.051488+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:09:43.488003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"92'8","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.028028+0000","last_change":"2024-03-28T16:01:08.274978+0000","last_active":"2024-03-28T16:02:39.028028+0000","last_peered":"2024-03-28T16:02:39.028028+0000","last_clean":"2024-03-28T16:02:39.028028+0000","last_became_active":"2024-03-28T16:01:08.274553+0000","last_became_peered":"2024-03-28T16:01:08.274553+0000","last_unstale":"2024-03-28T16:02:39.028028+0000","last_undegraded":"2024-03-28T16:02:39.028028+0000","last_fullsized":"2024-03-28T16:02:39.028028+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:16:03.997334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"92'4","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.042170+0000","last_change":"2024-03-28T16:01:08.262155+0000","last_active":"2024-03-28T16:02:39.042170+0000","last_peered":"2024-03-28T16:02:39.042170+0000","last_clean":"2024-03-28T16:02:39.042170+0000","last_became_active":"2024-03-28T16:01:08.261814+0000","last_became_peered":"2024-03-28T16:01:08.261814+0000","last_unstale":"2024-03-28T16:02:39.042170+0000","last_undegraded":"2024-03-28T16:02:39.042170+0000","last_fullsized":"2024-03-28T16:02:39.042170+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:41:04.242203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"92'6","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.023511+0000","last_change":"2024-03-28T16:01:08.273342+0000","last_active":"2024-03-28T16:02:39.023511+0000","last_peered":"2024-03-28T16:02:39.023511+0000","last_clean":"2024-03-28T16:02:39.023511+0000","last_became_active":"2024-03-28T16:01:08.272606+0000","last_became_peered":"2024-03-28T16:01:08.272606+0000","last_unstale":"2024-03-28T16:02:39.023511+0000","last_undegraded":"2024-03-28T16:02:39.023511+0000","last_fullsized":"2024-03-28T16:02:39.023511+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:44:48.580214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"92'5","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.040848+0000","last_change":"2024-03-28T16:01:08.278060+0000","last_active":"2024-03-28T16:02:39.040848+0000","last_peered":"2024-03-28T16:02:39.040848+0000","last_clean":"2024-03-28T16:02:39.040848+0000","last_became_active":"2024-03-28T16:01:08.277047+0000","last_became_peered":"2024-03-28T16:01:08.277047+0000","last_unstale":"2024-03-28T16:02:39.040848+0000","last_undegraded":"2024-03-28T16:02:39.040848+0000","last_fullsized":"2024-03-28T16:02:39.040848+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:46:30.005708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"92'9","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.036559+0000","last_change":"2024-03-28T16:01:08.273199+0000","last_active":"2024-03-28T16:02:39.036559+0000","last_peered":"2024-03-28T16:02:39.036559+0000","last_clean":"2024-03-28T16:02:39.036559+0000","last_became_active":"2024-03-28T16:01:08.272822+0000","last_became_peered":"2024-03-28T16:01:08.272822+0000","last_unstale":"2024-03-28T16:02:39.036559+0000","last_undegraded":"2024-03-28T16:02:39.036559+0000","last_fullsized":"2024-03-28T16:02:39.036559+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:28:11.495665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"92'11","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.050965+0000","last_change":"2024-03-28T16:01:08.270262+0000","last_active":"2024-03-28T16:02:39.050965+0000","last_peered":"2024-03-28T16:02:39.050965+0000","last_clean":"2024-03-28T16:02:39.050965+0000","last_became_active":"2024-03-28T16:01:08.269959+0000","last_became_peered":"2024-03-28T16:01:08.269959+0000","last_unstale":"2024-03-28T16:02:39.050965+0000","last_undegraded":"2024-03-28T16:02:39.050965+0000","last_fullsized":"2024-03-28T16:02:39.050965+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:49:02.261893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"92'16","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.028205+0000","last_change":"2024-03-28T16:01:40.671104+0000","last_active":"2024-03-28T16:02:39.028205+0000","last_peered":"2024-03-28T16:02:39.028205+0000","last_clean":"2024-03-28T16:02:39.028205+0000","last_became_active":"2024-03-28T16:01:40.670474+0000","last_became_peered":"2024-03-28T16:01:40.670474+0000","last_unstale":"2024-03-28T16:02:39.028205+0000","last_undegraded":"2024-03-28T16:02:39.028205+0000","last_fullsized":"2024-03-28T16:02:39.028205+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:19:41.651703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"92'6","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.052808+0000","last_change":"2024-03-28T16:01:08.274943+0000","last_active":"2024-03-28T16:02:39.052808+0000","last_peered":"2024-03-28T16:02:39.052808+0000","last_clean":"2024-03-28T16:02:39.052808+0000","last_became_active":"2024-03-28T16:01:08.274267+0000","last_became_peered":"2024-03-28T16:01:08.274267+0000","last_unstale":"2024-03-28T16:02:39.052808+0000","last_undegraded":"2024-03-28T16:02:39.052808+0000","last_fullsized":"2024-03-28T16:02:39.052808+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:41:13.979918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"92'7","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.020548+0000","last_change":"2024-03-28T16:01:08.266817+0000","last_active":"2024-03-28T16:02:39.020548+0000","last_peered":"2024-03-28T16:02:39.020548+0000","last_clean":"2024-03-28T16:02:39.020548+0000","last_became_active":"2024-03-28T16:01:08.266470+0000","last_became_peered":"2024-03-28T16:01:08.266470+0000","last_unstale":"2024-03-28T16:02:39.020548+0000","last_undegraded":"2024-03-28T16:02:39.020548+0000","last_fullsized":"2024-03-28T16:02:39.020548+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:15:28.573153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"92'10","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.052174+0000","last_change":"2024-03-28T16:01:08.269896+0000","last_active":"2024-03-28T16:02:39.052174+0000","last_peered":"2024-03-28T16:02:39.052174+0000","last_clean":"2024-03-28T16:02:39.052174+0000","last_became_active":"2024-03-28T16:01:08.269500+0000","last_became_peered":"2024-03-28T16:01:08.269500+0000","last_unstale":"2024-03-28T16:02:39.052174+0000","last_undegraded":"2024-03-28T16:02:39.052174+0000","last_fullsized":"2024-03-28T16:02:39.052174+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:44:38.867590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"92'7","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.047544+0000","last_change":"2024-03-28T16:01:08.267072+0000","last_active":"2024-03-28T16:02:39.047544+0000","last_peered":"2024-03-28T16:02:39.047544+0000","last_clean":"2024-03-28T16:02:39.047544+0000","last_became_active":"2024-03-28T16:01:08.266688+0000","last_became_peered":"2024-03-28T16:01:08.266688+0000","last_unstale":"2024-03-28T16:02:39.047544+0000","last_undegraded":"2024-03-28T16:02:39.047544+0000","last_fullsized":"2024-03-28T16:02:39.047544+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:07:57.702952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"92'7","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.045559+0000","last_change":"2024-03-28T16:01:08.270912+0000","last_active":"2024-03-28T16:02:39.045559+0000","last_peered":"2024-03-28T16:02:39.045559+0000","last_clean":"2024-03-28T16:02:39.045559+0000","last_became_active":"2024-03-28T16:01:08.270567+0000","last_became_peered":"2024-03-28T16:01:08.270567+0000","last_unstale":"2024-03-28T16:02:39.045559+0000","last_undegraded":"2024-03-28T16:02:39.045559+0000","last_fullsized":"2024-03-28T16:02:39.045559+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:19:29.294502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"92'9","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.050241+0000","last_change":"2024-03-28T16:01:08.266811+0000","last_active":"2024-03-28T16:02:39.050241+0000","last_peered":"2024-03-28T16:02:39.050241+0000","last_clean":"2024-03-28T16:02:39.050241+0000","last_became_active":"2024-03-28T16:01:08.266517+0000","last_became_peered":"2024-03-28T16:01:08.266517+0000","last_unstale":"2024-03-28T16:02:39.050241+0000","last_undegraded":"2024-03-28T16:02:39.050241+0000","last_fullsized":"2024-03-28T16:02:39.050241+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:55:05.972683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"92'8","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.050243+0000","last_change":"2024-03-28T16:01:08.263864+0000","last_active":"2024-03-28T16:02:39.050243+0000","last_peered":"2024-03-28T16:02:39.050243+0000","last_clean":"2024-03-28T16:02:39.050243+0000","last_became_active":"2024-03-28T16:01:08.262956+0000","last_became_peered":"2024-03-28T16:01:08.262956+0000","last_unstale":"2024-03-28T16:02:39.050243+0000","last_undegraded":"2024-03-28T16:02:39.050243+0000","last_fullsized":"2024-03-28T16:02:39.050243+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:05:50.504237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"92'10","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.045902+0000","last_change":"2024-03-28T16:01:08.265151+0000","last_active":"2024-03-28T16:02:39.045902+0000","last_peered":"2024-03-28T16:02:39.045902+0000","last_clean":"2024-03-28T16:02:39.045902+0000","last_became_active":"2024-03-28T16:01:08.264861+0000","last_became_peered":"2024-03-28T16:01:08.264861+0000","last_unstale":"2024-03-28T16:02:39.045902+0000","last_undegraded":"2024-03-28T16:02:39.045902+0000","last_fullsized":"2024-03-28T16:02:39.045902+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:35:45.878983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"92'7","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.037162+0000","last_change":"2024-03-28T16:01:08.265993+0000","last_active":"2024-03-28T16:02:39.037162+0000","last_peered":"2024-03-28T16:02:39.037162+0000","last_clean":"2024-03-28T16:02:39.037162+0000","last_became_active":"2024-03-28T16:01:08.265488+0000","last_became_peered":"2024-03-28T16:01:08.265488+0000","last_unstale":"2024-03-28T16:02:39.037162+0000","last_undegraded":"2024-03-28T16:02:39.037162+0000","last_fullsized":"2024-03-28T16:02:39.037162+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:31:55.920705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'4","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.654831+0000","last_change":"2024-03-28T16:01:07.250361+0000","last_active":"2024-03-28T16:01:40.654831+0000","last_peered":"2024-03-28T16:01:40.654831+0000","last_clean":"2024-03-28T16:01:40.654831+0000","last_became_active":"2024-03-28T16:01:07.250004+0000","last_became_peered":"2024-03-28T16:01:07.250004+0000","last_unstale":"2024-03-28T16:01:40.654831+0000","last_undegraded":"2024-03-28T16:01:40.654831+0000","last_fullsized":"2024-03-28T16:01:40.654831+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:10:32.838716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"92'5","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.032768+0000","last_change":"2024-03-28T16:01:08.274436+0000","last_active":"2024-03-28T16:02:39.032768+0000","last_peered":"2024-03-28T16:02:39.032768+0000","last_clean":"2024-03-28T16:02:39.032768+0000","last_became_active":"2024-03-28T16:01:08.274124+0000","last_became_peered":"2024-03-28T16:01:08.274124+0000","last_unstale":"2024-03-28T16:02:39.032768+0000","last_undegraded":"2024-03-28T16:02:39.032768+0000","last_fullsized":"2024-03-28T16:02:39.032768+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:02:34.037686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"89'1","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.765443+0000","last_change":"2024-03-28T16:01:07.248565+0000","last_active":"2024-03-28T16:01:40.765443+0000","last_peered":"2024-03-28T16:01:40.765443+0000","last_clean":"2024-03-28T16:01:40.765443+0000","last_became_active":"2024-03-28T16:01:07.247366+0000","last_became_peered":"2024-03-28T16:01:07.247366+0000","last_unstale":"2024-03-28T16:01:40.765443+0000","last_undegraded":"2024-03-28T16:01:40.765443+0000","last_fullsized":"2024-03-28T16:01:40.765443+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:48:13.146398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"92'6","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.051966+0000","last_change":"2024-03-28T16:01:08.265639+0000","last_active":"2024-03-28T16:02:39.051966+0000","last_peered":"2024-03-28T16:02:39.051966+0000","last_clean":"2024-03-28T16:02:39.051966+0000","last_became_active":"2024-03-28T16:01:08.263488+0000","last_became_peered":"2024-03-28T16:01:08.263488+0000","last_unstale":"2024-03-28T16:02:39.051966+0000","last_undegraded":"2024-03-28T16:02:39.051966+0000","last_fullsized":"2024-03-28T16:02:39.051966+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:15:06.516016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"89'2","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.850755+0000","last_change":"2024-03-28T16:01:40.850755+0000","last_active":"2024-03-28T16:01:40.850755+0000","last_peered":"2024-03-28T16:01:40.850755+0000","last_clean":"2024-03-28T16:01:40.850755+0000","last_became_active":"2024-03-28T16:01:40.842547+0000","last_became_peered":"2024-03-28T16:01:40.842547+0000","last_unstale":"2024-03-28T16:01:40.850755+0000","last_undegraded":"2024-03-28T16:01:40.850755+0000","last_fullsized":"2024-03-28T16:01:40.850755+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:39:27.862525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"92'9","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.051889+0000","last_change":"2024-03-28T16:01:08.271534+0000","last_active":"2024-03-28T16:02:39.051889+0000","last_peered":"2024-03-28T16:02:39.051889+0000","last_clean":"2024-03-28T16:02:39.051889+0000","last_became_active":"2024-03-28T16:01:08.271104+0000","last_became_peered":"2024-03-28T16:01:08.271104+0000","last_unstale":"2024-03-28T16:02:39.051889+0000","last_undegraded":"2024-03-28T16:02:39.051889+0000","last_fullsized":"2024-03-28T16:02:39.051889+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:29:21.332346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"89'2","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.655555+0000","last_change":"2024-03-28T16:01:07.259175+0000","last_active":"2024-03-28T16:01:40.655555+0000","last_peered":"2024-03-28T16:01:40.655555+0000","last_clean":"2024-03-28T16:01:40.655555+0000","last_became_active":"2024-03-28T16:01:07.258151+0000","last_became_peered":"2024-03-28T16:01:07.258151+0000","last_unstale":"2024-03-28T16:01:40.655555+0000","last_undegraded":"2024-03-28T16:01:40.655555+0000","last_fullsized":"2024-03-28T16:01:40.655555+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:41:07.064800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"92'15","reported_seq":38,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.046924+0000","last_change":"2024-03-28T16:01:40.843211+0000","last_active":"2024-03-28T16:02:39.046924+0000","last_peered":"2024-03-28T16:02:39.046924+0000","last_clean":"2024-03-28T16:02:39.046924+0000","last_became_active":"2024-03-28T16:01:40.842051+0000","last_became_peered":"2024-03-28T16:01:40.842051+0000","last_unstale":"2024-03-28T16:02:39.046924+0000","last_undegraded":"2024-03-28T16:02:39.046924+0000","last_fullsized":"2024-03-28T16:02:39.046924+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:19:26.437169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"89'1","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.664703+0000","last_change":"2024-03-28T16:01:07.248538+0000","last_active":"2024-03-28T16:01:40.664703+0000","last_peered":"2024-03-28T16:01:40.664703+0000","last_clean":"2024-03-28T16:01:40.664703+0000","last_became_active":"2024-03-28T16:01:07.248116+0000","last_became_peered":"2024-03-28T16:01:07.248116+0000","last_unstale":"2024-03-28T16:01:40.664703+0000","last_undegraded":"2024-03-28T16:01:40.664703+0000","last_fullsized":"2024-03-28T16:01:40.664703+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:44:56.071286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"92'7","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.033579+0000","last_change":"2024-03-28T16:01:08.271059+0000","last_active":"2024-03-28T16:02:39.033579+0000","last_peered":"2024-03-28T16:02:39.033579+0000","last_clean":"2024-03-28T16:02:39.033579+0000","last_became_active":"2024-03-28T16:01:08.270682+0000","last_became_peered":"2024-03-28T16:01:08.270682+0000","last_unstale":"2024-03-28T16:02:39.033579+0000","last_undegraded":"2024-03-28T16:02:39.033579+0000","last_fullsized":"2024-03-28T16:02:39.033579+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:54:48.922125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"89'1","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.661404+0000","last_change":"2024-03-28T16:01:07.261157+0000","last_active":"2024-03-28T16:01:40.661404+0000","last_peered":"2024-03-28T16:01:40.661404+0000","last_clean":"2024-03-28T16:01:40.661404+0000","last_became_active":"2024-03-28T16:01:07.260751+0000","last_became_peered":"2024-03-28T16:01:07.260751+0000","last_unstale":"2024-03-28T16:01:40.661404+0000","last_undegraded":"2024-03-28T16:01:40.661404+0000","last_fullsized":"2024-03-28T16:01:40.661404+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:16:49.622168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"92'8","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.038730+0000","last_change":"2024-03-28T16:01:08.265773+0000","last_active":"2024-03-28T16:02:39.038730+0000","last_peered":"2024-03-28T16:02:39.038730+0000","last_clean":"2024-03-28T16:02:39.038730+0000","last_became_active":"2024-03-28T16:01:08.263724+0000","last_became_peered":"2024-03-28T16:01:08.263724+0000","last_unstale":"2024-03-28T16:02:39.038730+0000","last_undegraded":"2024-03-28T16:02:39.038730+0000","last_fullsized":"2024-03-28T16:02:39.038730+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:42:37.338946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.921045+0000","last_change":"2024-03-28T16:01:07.235361+0000","last_active":"2024-03-28T16:01:40.921045+0000","last_peered":"2024-03-28T16:01:40.921045+0000","last_clean":"2024-03-28T16:01:40.921045+0000","last_became_active":"2024-03-28T16:01:07.234969+0000","last_became_peered":"2024-03-28T16:01:07.234969+0000","last_unstale":"2024-03-28T16:01:40.921045+0000","last_undegraded":"2024-03-28T16:01:40.921045+0000","last_fullsized":"2024-03-28T16:01:40.921045+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:50:35.592147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"92'8","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.039851+0000","last_change":"2024-03-28T16:01:08.263878+0000","last_active":"2024-03-28T16:02:39.039851+0000","last_peered":"2024-03-28T16:02:39.039851+0000","last_clean":"2024-03-28T16:02:39.039851+0000","last_became_active":"2024-03-28T16:01:08.263549+0000","last_became_peered":"2024-03-28T16:01:08.263549+0000","last_unstale":"2024-03-28T16:02:39.039851+0000","last_undegraded":"2024-03-28T16:02:39.039851+0000","last_fullsized":"2024-03-28T16:02:39.039851+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:31:37.669215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.663697+0000","last_change":"2024-03-28T16:01:07.255701+0000","last_active":"2024-03-28T16:01:40.663697+0000","last_peered":"2024-03-28T16:01:40.663697+0000","last_clean":"2024-03-28T16:01:40.663697+0000","last_became_active":"2024-03-28T16:01:07.255247+0000","last_became_peered":"2024-03-28T16:01:07.255247+0000","last_unstale":"2024-03-28T16:01:40.663697+0000","last_undegraded":"2024-03-28T16:01:40.663697+0000","last_fullsized":"2024-03-28T16:01:40.663697+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:08:17.858395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"92'11","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.034304+0000","last_change":"2024-03-28T16:01:08.252949+0000","last_active":"2024-03-28T16:02:39.034304+0000","last_peered":"2024-03-28T16:02:39.034304+0000","last_clean":"2024-03-28T16:02:39.034304+0000","last_became_active":"2024-03-28T16:01:08.252471+0000","last_became_peered":"2024-03-28T16:01:08.252471+0000","last_unstale":"2024-03-28T16:02:39.034304+0000","last_undegraded":"2024-03-28T16:02:39.034304+0000","last_fullsized":"2024-03-28T16:02:39.034304+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:21:15.519020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.661736+0000","last_change":"2024-03-28T16:01:07.263973+0000","last_active":"2024-03-28T16:01:40.661736+0000","last_peered":"2024-03-28T16:01:40.661736+0000","last_clean":"2024-03-28T16:01:40.661736+0000","last_became_active":"2024-03-28T16:01:07.263560+0000","last_became_peered":"2024-03-28T16:01:07.263560+0000","last_unstale":"2024-03-28T16:01:40.661736+0000","last_undegraded":"2024-03-28T16:01:40.661736+0000","last_fullsized":"2024-03-28T16:01:40.661736+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:11.215312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"92'10","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.051393+0000","last_change":"2024-03-28T16:01:08.266122+0000","last_active":"2024-03-28T16:02:39.051393+0000","last_peered":"2024-03-28T16:02:39.051393+0000","last_clean":"2024-03-28T16:02:39.051393+0000","last_became_active":"2024-03-28T16:01:08.265727+0000","last_became_peered":"2024-03-28T16:01:08.265727+0000","last_unstale":"2024-03-28T16:02:39.051393+0000","last_undegraded":"2024-03-28T16:02:39.051393+0000","last_fullsized":"2024-03-28T16:02:39.051393+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:48:38.939777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.666951+0000","last_change":"2024-03-28T16:01:07.251814+0000","last_active":"2024-03-28T16:01:40.666951+0000","last_peered":"2024-03-28T16:01:40.666951+0000","last_clean":"2024-03-28T16:01:40.666951+0000","last_became_active":"2024-03-28T16:01:07.251311+0000","last_became_peered":"2024-03-28T16:01:07.251311+0000","last_unstale":"2024-03-28T16:01:40.666951+0000","last_undegraded":"2024-03-28T16:01:40.666951+0000","last_fullsized":"2024-03-28T16:01:40.666951+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:13:38.344070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"92'9","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.049556+0000","last_change":"2024-03-28T16:01:08.270879+0000","last_active":"2024-03-28T16:02:39.049556+0000","last_peered":"2024-03-28T16:02:39.049556+0000","last_clean":"2024-03-28T16:02:39.049556+0000","last_became_active":"2024-03-28T16:01:08.270513+0000","last_became_peered":"2024-03-28T16:01:08.270513+0000","last_unstale":"2024-03-28T16:02:39.049556+0000","last_undegraded":"2024-03-28T16:02:39.049556+0000","last_fullsized":"2024-03-28T16:02:39.049556+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:53:43.285797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"89'3","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.665900+0000","last_change":"2024-03-28T16:01:07.253595+0000","last_active":"2024-03-28T16:01:40.665900+0000","last_peered":"2024-03-28T16:01:40.665900+0000","last_clean":"2024-03-28T16:01:40.665900+0000","last_became_active":"2024-03-28T16:01:07.253137+0000","last_became_peered":"2024-03-28T16:01:07.253137+0000","last_unstale":"2024-03-28T16:01:40.665900+0000","last_undegraded":"2024-03-28T16:01:40.665900+0000","last_fullsized":"2024-03-28T16:01:40.665900+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:17:35.759791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"92'11","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.050280+0000","last_change":"2024-03-28T16:01:08.261185+0000","last_active":"2024-03-28T16:02:39.050280+0000","last_peered":"2024-03-28T16:02:39.050280+0000","last_clean":"2024-03-28T16:02:39.050280+0000","last_became_active":"2024-03-28T16:01:08.260885+0000","last_became_peered":"2024-03-28T16:01:08.260885+0000","last_unstale":"2024-03-28T16:02:39.050280+0000","last_undegraded":"2024-03-28T16:02:39.050280+0000","last_fullsized":"2024-03-28T16:02:39.050280+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:15:56.640586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"92'31","reported_seq":326,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:03:23.893167+0000","last_change":"2024-03-28T16:01:07.258655+0000","last_active":"2024-03-28T16:03:23.893167+0000","last_peered":"2024-03-28T16:03:23.893167+0000","last_clean":"2024-03-28T16:03:23.893167+0000","last_became_active":"2024-03-28T16:01:07.258230+0000","last_became_peered":"2024-03-28T16:01:07.258230+0000","last_unstale":"2024-03-28T16:03:23.893167+0000","last_undegraded":"2024-03-28T16:03:23.893167+0000","last_fullsized":"2024-03-28T16:03:23.893167+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:29:36.206061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62638,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":276,"num_read_kb":199,"num_write":31,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"92'6","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.038343+0000","last_change":"2024-03-28T16:01:08.265393+0000","last_active":"2024-03-28T16:02:39.038343+0000","last_peered":"2024-03-28T16:02:39.038343+0000","last_clean":"2024-03-28T16:02:39.038343+0000","last_became_active":"2024-03-28T16:01:08.264560+0000","last_became_peered":"2024-03-28T16:01:08.264560+0000","last_unstale":"2024-03-28T16:02:39.038343+0000","last_undegraded":"2024-03-28T16:02:39.038343+0000","last_fullsized":"2024-03-28T16:02:39.038343+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:43:52.181113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"89'2","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.672781+0000","last_change":"2024-03-28T16:01:07.251428+0000","last_active":"2024-03-28T16:01:40.672781+0000","last_peered":"2024-03-28T16:01:40.672781+0000","last_clean":"2024-03-28T16:01:40.672781+0000","last_became_active":"2024-03-28T16:01:07.251074+0000","last_became_peered":"2024-03-28T16:01:07.251074+0000","last_unstale":"2024-03-28T16:01:40.672781+0000","last_undegraded":"2024-03-28T16:01:40.672781+0000","last_fullsized":"2024-03-28T16:01:40.672781+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:18:41.359338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"92'9","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.037592+0000","last_change":"2024-03-28T16:01:08.268482+0000","last_active":"2024-03-28T16:02:39.037592+0000","last_peered":"2024-03-28T16:02:39.037592+0000","last_clean":"2024-03-28T16:02:39.037592+0000","last_became_active":"2024-03-28T16:01:08.268153+0000","last_became_peered":"2024-03-28T16:01:08.268153+0000","last_unstale":"2024-03-28T16:02:39.037592+0000","last_undegraded":"2024-03-28T16:02:39.037592+0000","last_fullsized":"2024-03-28T16:02:39.037592+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:21:21.874773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.664138+0000","last_change":"2024-03-28T16:01:07.243506+0000","last_active":"2024-03-28T16:01:40.664138+0000","last_peered":"2024-03-28T16:01:40.664138+0000","last_clean":"2024-03-28T16:01:40.664138+0000","last_became_active":"2024-03-28T16:01:07.242793+0000","last_became_peered":"2024-03-28T16:01:07.242793+0000","last_unstale":"2024-03-28T16:01:40.664138+0000","last_undegraded":"2024-03-28T16:01:40.664138+0000","last_fullsized":"2024-03-28T16:01:40.664138+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:31:56.497276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"92'15","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.044862+0000","last_change":"2024-03-28T16:01:08.278848+0000","last_active":"2024-03-28T16:02:39.044862+0000","last_peered":"2024-03-28T16:02:39.044862+0000","last_clean":"2024-03-28T16:02:39.044862+0000","last_became_active":"2024-03-28T16:01:08.278455+0000","last_became_peered":"2024-03-28T16:01:08.278455+0000","last_unstale":"2024-03-28T16:02:39.044862+0000","last_undegraded":"2024-03-28T16:02:39.044862+0000","last_fullsized":"2024-03-28T16:02:39.044862+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:41:17.302538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.655733+0000","last_change":"2024-03-28T16:01:07.255070+0000","last_active":"2024-03-28T16:01:40.655733+0000","last_peered":"2024-03-28T16:01:40.655733+0000","last_clean":"2024-03-28T16:01:40.655733+0000","last_became_active":"2024-03-28T16:01:07.253489+0000","last_became_peered":"2024-03-28T16:01:07.253489+0000","last_unstale":"2024-03-28T16:01:40.655733+0000","last_undegraded":"2024-03-28T16:01:40.655733+0000","last_fullsized":"2024-03-28T16:01:40.655733+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:31:57.501721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"92'6","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.036271+0000","last_change":"2024-03-28T16:01:08.273190+0000","last_active":"2024-03-28T16:02:39.036271+0000","last_peered":"2024-03-28T16:02:39.036271+0000","last_clean":"2024-03-28T16:02:39.036271+0000","last_became_active":"2024-03-28T16:01:08.272870+0000","last_became_peered":"2024-03-28T16:01:08.272870+0000","last_unstale":"2024-03-28T16:02:39.036271+0000","last_undegraded":"2024-03-28T16:02:39.036271+0000","last_fullsized":"2024-03-28T16:02:39.036271+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:26:50.874750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"89'8","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.700937+0000","last_change":"2024-03-28T16:01:40.700157+0000","last_active":"2024-03-28T16:01:40.700937+0000","last_peered":"2024-03-28T16:01:40.700937+0000","last_clean":"2024-03-28T16:01:40.700937+0000","last_became_active":"2024-03-28T16:01:40.674436+0000","last_became_peered":"2024-03-28T16:01:40.674436+0000","last_unstale":"2024-03-28T16:01:40.700937+0000","last_undegraded":"2024-03-28T16:01:40.700937+0000","last_fullsized":"2024-03-28T16:01:40.700937+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:26:48.776506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,4],"acting":[11,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"92'6","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.049436+0000","last_change":"2024-03-28T16:01:40.924563+0000","last_active":"2024-03-28T16:02:39.049436+0000","last_peered":"2024-03-28T16:02:39.049436+0000","last_clean":"2024-03-28T16:02:39.049436+0000","last_became_active":"2024-03-28T16:01:40.923348+0000","last_became_peered":"2024-03-28T16:01:40.923348+0000","last_unstale":"2024-03-28T16:02:39.049436+0000","last_undegraded":"2024-03-28T16:02:39.049436+0000","last_fullsized":"2024-03-28T16:02:39.049436+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:31:46.619651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'1","reported_seq":15,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.681372+0000","last_change":"2024-03-28T16:01:40.680548+0000","last_active":"2024-03-28T16:01:40.681372+0000","last_peered":"2024-03-28T16:01:40.681372+0000","last_clean":"2024-03-28T16:01:40.681372+0000","last_became_active":"2024-03-28T16:01:40.672674+0000","last_became_peered":"2024-03-28T16:01:40.672674+0000","last_unstale":"2024-03-28T16:01:40.681372+0000","last_undegraded":"2024-03-28T16:01:40.681372+0000","last_fullsized":"2024-03-28T16:01:40.681372+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:07:32.767184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"92'5","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.035565+0000","last_change":"2024-03-28T16:01:08.274436+0000","last_active":"2024-03-28T16:02:39.035565+0000","last_peered":"2024-03-28T16:02:39.035565+0000","last_clean":"2024-03-28T16:02:39.035565+0000","last_became_active":"2024-03-28T16:01:08.274081+0000","last_became_peered":"2024-03-28T16:01:08.274081+0000","last_unstale":"2024-03-28T16:02:39.035565+0000","last_undegraded":"2024-03-28T16:02:39.035565+0000","last_fullsized":"2024-03-28T16:02:39.035565+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:57:00.531064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.653153+0000","last_change":"2024-03-28T16:01:07.247542+0000","last_active":"2024-03-28T16:01:40.653153+0000","last_peered":"2024-03-28T16:01:40.653153+0000","last_clean":"2024-03-28T16:01:40.653153+0000","last_became_active":"2024-03-28T16:01:07.247109+0000","last_became_peered":"2024-03-28T16:01:07.247109+0000","last_unstale":"2024-03-28T16:01:40.653153+0000","last_undegraded":"2024-03-28T16:01:40.653153+0000","last_fullsized":"2024-03-28T16:01:40.653153+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:14:19.387896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"92'5","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.037067+0000","last_change":"2024-03-28T16:01:08.264565+0000","last_active":"2024-03-28T16:02:39.037067+0000","last_peered":"2024-03-28T16:02:39.037067+0000","last_clean":"2024-03-28T16:02:39.037067+0000","last_became_active":"2024-03-28T16:01:08.264196+0000","last_became_peered":"2024-03-28T16:01:08.264196+0000","last_unstale":"2024-03-28T16:02:39.037067+0000","last_undegraded":"2024-03-28T16:02:39.037067+0000","last_fullsized":"2024-03-28T16:02:39.037067+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:43:24.750624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.666071+0000","last_change":"2024-03-28T16:01:07.248189+0000","last_active":"2024-03-28T16:01:40.666071+0000","last_peered":"2024-03-28T16:01:40.666071+0000","last_clean":"2024-03-28T16:01:40.666071+0000","last_became_active":"2024-03-28T16:01:07.247295+0000","last_became_peered":"2024-03-28T16:01:07.247295+0000","last_unstale":"2024-03-28T16:01:40.666071+0000","last_undegraded":"2024-03-28T16:01:40.666071+0000","last_fullsized":"2024-03-28T16:01:40.666071+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:46:51.875094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"92'7","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.031820+0000","last_change":"2024-03-28T16:01:08.269724+0000","last_active":"2024-03-28T16:02:39.031820+0000","last_peered":"2024-03-28T16:02:39.031820+0000","last_clean":"2024-03-28T16:02:39.031820+0000","last_became_active":"2024-03-28T16:01:08.269357+0000","last_became_peered":"2024-03-28T16:01:08.269357+0000","last_unstale":"2024-03-28T16:02:39.031820+0000","last_undegraded":"2024-03-28T16:02:39.031820+0000","last_fullsized":"2024-03-28T16:02:39.031820+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:20:12.622238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"92'8","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.035129+0000","last_change":"2024-03-28T16:01:08.266624+0000","last_active":"2024-03-28T16:02:39.035129+0000","last_peered":"2024-03-28T16:02:39.035129+0000","last_clean":"2024-03-28T16:02:39.035129+0000","last_became_active":"2024-03-28T16:01:08.265408+0000","last_became_peered":"2024-03-28T16:01:08.265408+0000","last_unstale":"2024-03-28T16:02:39.035129+0000","last_undegraded":"2024-03-28T16:02:39.035129+0000","last_fullsized":"2024-03-28T16:02:39.035129+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:37:08.831727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.920935+0000","last_change":"2024-03-28T16:01:07.241172+0000","last_active":"2024-03-28T16:01:40.920935+0000","last_peered":"2024-03-28T16:01:40.920935+0000","last_clean":"2024-03-28T16:01:40.920935+0000","last_became_active":"2024-03-28T16:01:07.240754+0000","last_became_peered":"2024-03-28T16:01:07.240754+0000","last_unstale":"2024-03-28T16:01:40.920935+0000","last_undegraded":"2024-03-28T16:01:40.920935+0000","last_fullsized":"2024-03-28T16:01:40.920935+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:52:45.805188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"92'8","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.047153+0000","last_change":"2024-03-28T16:01:08.278429+0000","last_active":"2024-03-28T16:02:39.047153+0000","last_peered":"2024-03-28T16:02:39.047153+0000","last_clean":"2024-03-28T16:02:39.047153+0000","last_became_active":"2024-03-28T16:01:08.275061+0000","last_became_peered":"2024-03-28T16:01:08.275061+0000","last_unstale":"2024-03-28T16:02:39.047153+0000","last_undegraded":"2024-03-28T16:02:39.047153+0000","last_fullsized":"2024-03-28T16:02:39.047153+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:35:42.015979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.655996+0000","last_change":"2024-03-28T16:01:07.252117+0000","last_active":"2024-03-28T16:01:40.655996+0000","last_peered":"2024-03-28T16:01:40.655996+0000","last_clean":"2024-03-28T16:01:40.655996+0000","last_became_active":"2024-03-28T16:01:07.249729+0000","last_became_peered":"2024-03-28T16:01:07.249729+0000","last_unstale":"2024-03-28T16:01:40.655996+0000","last_undegraded":"2024-03-28T16:01:40.655996+0000","last_fullsized":"2024-03-28T16:01:40.655996+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:33:32.719310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":347,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.663525+0000","last_change":"2024-03-28T15:59:32.593730+0000","last_active":"2024-03-28T16:01:40.663525+0000","last_peered":"2024-03-28T16:01:40.663525+0000","last_clean":"2024-03-28T16:01:40.663525+0000","last_became_active":"2024-03-28T15:59:32.592289+0000","last_became_peered":"2024-03-28T15:59:32.592289+0000","last_unstale":"2024-03-28T16:01:40.663525+0000","last_undegraded":"2024-03-28T16:01:40.663525+0000","last_fullsized":"2024-03-28T16:01:40.663525+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:54:55.638193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:54:55.638193+0000","last_clean_scrub_stamp":"2024-03-28T15:54:55.638193+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:20:37.087340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"92'5","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.025916+0000","last_change":"2024-03-28T16:01:08.265513+0000","last_active":"2024-03-28T16:02:39.025916+0000","last_peered":"2024-03-28T16:02:39.025916+0000","last_clean":"2024-03-28T16:02:39.025916+0000","last_became_active":"2024-03-28T16:01:08.264711+0000","last_became_peered":"2024-03-28T16:01:08.264711+0000","last_unstale":"2024-03-28T16:02:39.025916+0000","last_undegraded":"2024-03-28T16:02:39.025916+0000","last_fullsized":"2024-03-28T16:02:39.025916+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:10:01.936780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.665127+0000","last_change":"2024-03-28T16:01:07.252831+0000","last_active":"2024-03-28T16:01:40.665127+0000","last_peered":"2024-03-28T16:01:40.665127+0000","last_clean":"2024-03-28T16:01:40.665127+0000","last_became_active":"2024-03-28T16:01:07.252317+0000","last_became_peered":"2024-03-28T16:01:07.252317+0000","last_unstale":"2024-03-28T16:01:40.665127+0000","last_undegraded":"2024-03-28T16:01:40.665127+0000","last_fullsized":"2024-03-28T16:01:40.665127+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:55:37.461154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.655837+0000","last_change":"2024-03-28T16:01:07.262413+0000","last_active":"2024-03-28T16:01:40.655837+0000","last_peered":"2024-03-28T16:01:40.655837+0000","last_clean":"2024-03-28T16:01:40.655837+0000","last_became_active":"2024-03-28T16:01:07.261875+0000","last_became_peered":"2024-03-28T16:01:07.261875+0000","last_unstale":"2024-03-28T16:01:40.655837+0000","last_undegraded":"2024-03-28T16:01:40.655837+0000","last_fullsized":"2024-03-28T16:01:40.655837+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:25:49.124991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"92'9","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.042204+0000","last_change":"2024-03-28T16:01:08.259506+0000","last_active":"2024-03-28T16:02:39.042204+0000","last_peered":"2024-03-28T16:02:39.042204+0000","last_clean":"2024-03-28T16:02:39.042204+0000","last_became_active":"2024-03-28T16:01:08.258512+0000","last_became_peered":"2024-03-28T16:01:08.258512+0000","last_unstale":"2024-03-28T16:02:39.042204+0000","last_undegraded":"2024-03-28T16:02:39.042204+0000","last_fullsized":"2024-03-28T16:02:39.042204+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:20:17.017763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"92'10","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:03:17.778725+0000","last_change":"2024-03-28T16:01:07.254850+0000","last_active":"2024-03-28T16:03:17.778725+0000","last_peered":"2024-03-28T16:03:17.778725+0000","last_clean":"2024-03-28T16:03:17.778725+0000","last_became_active":"2024-03-28T16:01:07.252294+0000","last_became_peered":"2024-03-28T16:01:07.252294+0000","last_unstale":"2024-03-28T16:03:17.778725+0000","last_undegraded":"2024-03-28T16:03:17.778725+0000","last_fullsized":"2024-03-28T16:03:17.778725+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:54:26.700118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"92'5","reported_seq":16,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.039125+0000","last_change":"2024-03-28T16:01:40.675232+0000","last_active":"2024-03-28T16:02:39.039125+0000","last_peered":"2024-03-28T16:02:39.039125+0000","last_clean":"2024-03-28T16:02:39.039125+0000","last_became_active":"2024-03-28T16:01:40.674454+0000","last_became_peered":"2024-03-28T16:01:40.674454+0000","last_unstale":"2024-03-28T16:02:39.039125+0000","last_undegraded":"2024-03-28T16:02:39.039125+0000","last_fullsized":"2024-03-28T16:02:39.039125+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:47:16.997641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,7],"acting":[9,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.7","version":"89'2","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.657272+0000","last_change":"2024-03-28T16:01:07.245292+0000","last_active":"2024-03-28T16:01:40.657272+0000","last_peered":"2024-03-28T16:01:40.657272+0000","last_clean":"2024-03-28T16:01:40.657272+0000","last_became_active":"2024-03-28T16:01:07.244654+0000","last_became_peered":"2024-03-28T16:01:07.244654+0000","last_unstale":"2024-03-28T16:01:40.657272+0000","last_undegraded":"2024-03-28T16:01:40.657272+0000","last_fullsized":"2024-03-28T16:01:40.657272+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:03:39.027844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"92'7","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.049641+0000","last_change":"2024-03-28T16:01:08.274542+0000","last_active":"2024-03-28T16:02:39.049641+0000","last_peered":"2024-03-28T16:02:39.049641+0000","last_clean":"2024-03-28T16:02:39.049641+0000","last_became_active":"2024-03-28T16:01:08.274202+0000","last_became_peered":"2024-03-28T16:01:08.274202+0000","last_unstale":"2024-03-28T16:02:39.049641+0000","last_undegraded":"2024-03-28T16:02:39.049641+0000","last_fullsized":"2024-03-28T16:02:39.049641+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:09:18.199017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.838038+0000","last_change":"2024-03-28T16:01:07.253274+0000","last_active":"2024-03-28T16:01:40.838038+0000","last_peered":"2024-03-28T16:01:40.838038+0000","last_clean":"2024-03-28T16:01:40.838038+0000","last_became_active":"2024-03-28T16:01:07.250230+0000","last_became_peered":"2024-03-28T16:01:07.250230+0000","last_unstale":"2024-03-28T16:01:40.838038+0000","last_undegraded":"2024-03-28T16:01:40.838038+0000","last_fullsized":"2024-03-28T16:01:40.838038+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:20:31.459983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"92'2","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.026627+0000","last_change":"2024-03-28T16:01:08.266551+0000","last_active":"2024-03-28T16:02:39.026627+0000","last_peered":"2024-03-28T16:02:39.026627+0000","last_clean":"2024-03-28T16:02:39.026627+0000","last_became_active":"2024-03-28T16:01:08.266226+0000","last_became_peered":"2024-03-28T16:01:08.266226+0000","last_unstale":"2024-03-28T16:02:39.026627+0000","last_undegraded":"2024-03-28T16:02:39.026627+0000","last_fullsized":"2024-03-28T16:02:39.026627+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:51:16.520285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.764663+0000","last_change":"2024-03-28T16:01:07.252038+0000","last_active":"2024-03-28T16:01:40.764663+0000","last_peered":"2024-03-28T16:01:40.764663+0000","last_clean":"2024-03-28T16:01:40.764663+0000","last_became_active":"2024-03-28T16:01:07.251703+0000","last_became_peered":"2024-03-28T16:01:07.251703+0000","last_unstale":"2024-03-28T16:01:40.764663+0000","last_undegraded":"2024-03-28T16:01:40.764663+0000","last_fullsized":"2024-03-28T16:01:40.764663+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:35:42.518996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"92'9","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.048402+0000","last_change":"2024-03-28T16:01:08.260025+0000","last_active":"2024-03-28T16:02:39.048402+0000","last_peered":"2024-03-28T16:02:39.048402+0000","last_clean":"2024-03-28T16:02:39.048402+0000","last_became_active":"2024-03-28T16:01:08.259596+0000","last_became_peered":"2024-03-28T16:01:08.259596+0000","last_unstale":"2024-03-28T16:02:39.048402+0000","last_undegraded":"2024-03-28T16:02:39.048402+0000","last_fullsized":"2024-03-28T16:02:39.048402+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:45:29.228188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.668132+0000","last_change":"2024-03-28T16:01:07.259335+0000","last_active":"2024-03-28T16:01:40.668132+0000","last_peered":"2024-03-28T16:01:40.668132+0000","last_clean":"2024-03-28T16:01:40.668132+0000","last_became_active":"2024-03-28T16:01:07.258589+0000","last_became_peered":"2024-03-28T16:01:07.258589+0000","last_unstale":"2024-03-28T16:01:40.668132+0000","last_undegraded":"2024-03-28T16:01:40.668132+0000","last_fullsized":"2024-03-28T16:01:40.668132+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:06:41.742963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"92'8","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.048367+0000","last_change":"2024-03-28T16:01:08.266794+0000","last_active":"2024-03-28T16:02:39.048367+0000","last_peered":"2024-03-28T16:02:39.048367+0000","last_clean":"2024-03-28T16:02:39.048367+0000","last_became_active":"2024-03-28T16:01:08.266502+0000","last_became_peered":"2024-03-28T16:01:08.266502+0000","last_unstale":"2024-03-28T16:02:39.048367+0000","last_undegraded":"2024-03-28T16:02:39.048367+0000","last_fullsized":"2024-03-28T16:02:39.048367+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:17:12.671677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"89'2","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.655656+0000","last_change":"2024-03-28T16:01:07.259239+0000","last_active":"2024-03-28T16:01:40.655656+0000","last_peered":"2024-03-28T16:01:40.655656+0000","last_clean":"2024-03-28T16:01:40.655656+0000","last_became_active":"2024-03-28T16:01:07.258875+0000","last_became_peered":"2024-03-28T16:01:07.258875+0000","last_unstale":"2024-03-28T16:01:40.655656+0000","last_undegraded":"2024-03-28T16:01:40.655656+0000","last_fullsized":"2024-03-28T16:01:40.655656+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:35:12.729173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"92'5","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.052410+0000","last_change":"2024-03-28T16:01:08.278196+0000","last_active":"2024-03-28T16:02:39.052410+0000","last_peered":"2024-03-28T16:02:39.052410+0000","last_clean":"2024-03-28T16:02:39.052410+0000","last_became_active":"2024-03-28T16:01:08.276972+0000","last_became_peered":"2024-03-28T16:01:08.276972+0000","last_unstale":"2024-03-28T16:02:39.052410+0000","last_undegraded":"2024-03-28T16:02:39.052410+0000","last_fullsized":"2024-03-28T16:02:39.052410+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:28:25.231210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.668117+0000","last_change":"2024-03-28T16:01:07.246851+0000","last_active":"2024-03-28T16:01:40.668117+0000","last_peered":"2024-03-28T16:01:40.668117+0000","last_clean":"2024-03-28T16:01:40.668117+0000","last_became_active":"2024-03-28T16:01:07.246365+0000","last_became_peered":"2024-03-28T16:01:07.246365+0000","last_unstale":"2024-03-28T16:01:40.668117+0000","last_undegraded":"2024-03-28T16:01:40.668117+0000","last_fullsized":"2024-03-28T16:01:40.668117+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:16:13.593045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"92'7","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.053165+0000","last_change":"2024-03-28T16:01:08.279468+0000","last_active":"2024-03-28T16:02:39.053165+0000","last_peered":"2024-03-28T16:02:39.053165+0000","last_clean":"2024-03-28T16:02:39.053165+0000","last_became_active":"2024-03-28T16:01:08.274650+0000","last_became_peered":"2024-03-28T16:01:08.274650+0000","last_unstale":"2024-03-28T16:02:39.053165+0000","last_undegraded":"2024-03-28T16:02:39.053165+0000","last_fullsized":"2024-03-28T16:02:39.053165+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:36:32.776085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"89'1","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.665556+0000","last_change":"2024-03-28T16:01:07.258077+0000","last_active":"2024-03-28T16:01:40.665556+0000","last_peered":"2024-03-28T16:01:40.665556+0000","last_clean":"2024-03-28T16:01:40.665556+0000","last_became_active":"2024-03-28T16:01:07.256665+0000","last_became_peered":"2024-03-28T16:01:07.256665+0000","last_unstale":"2024-03-28T16:01:40.665556+0000","last_undegraded":"2024-03-28T16:01:40.665556+0000","last_fullsized":"2024-03-28T16:01:40.665556+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:01:18.852441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"92'7","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.040959+0000","last_change":"2024-03-28T16:01:08.278588+0000","last_active":"2024-03-28T16:02:39.040959+0000","last_peered":"2024-03-28T16:02:39.040959+0000","last_clean":"2024-03-28T16:02:39.040959+0000","last_became_active":"2024-03-28T16:01:08.277303+0000","last_became_peered":"2024-03-28T16:01:08.277303+0000","last_unstale":"2024-03-28T16:02:39.040959+0000","last_undegraded":"2024-03-28T16:02:39.040959+0000","last_fullsized":"2024-03-28T16:02:39.040959+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:01:19.927057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"89'2","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.921309+0000","last_change":"2024-03-28T16:01:07.249382+0000","last_active":"2024-03-28T16:01:40.921309+0000","last_peered":"2024-03-28T16:01:40.921309+0000","last_clean":"2024-03-28T16:01:40.921309+0000","last_became_active":"2024-03-28T16:01:07.249065+0000","last_became_peered":"2024-03-28T16:01:07.249065+0000","last_unstale":"2024-03-28T16:01:40.921309+0000","last_undegraded":"2024-03-28T16:01:40.921309+0000","last_fullsized":"2024-03-28T16:01:40.921309+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:07:22.851862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"92'9","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.038073+0000","last_change":"2024-03-28T16:01:08.274839+0000","last_active":"2024-03-28T16:02:39.038073+0000","last_peered":"2024-03-28T16:02:39.038073+0000","last_clean":"2024-03-28T16:02:39.038073+0000","last_became_active":"2024-03-28T16:01:08.274505+0000","last_became_peered":"2024-03-28T16:01:08.274505+0000","last_unstale":"2024-03-28T16:02:39.038073+0000","last_undegraded":"2024-03-28T16:02:39.038073+0000","last_fullsized":"2024-03-28T16:02:39.038073+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:57:37.635327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"92'9","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.037953+0000","last_change":"2024-03-28T16:01:40.676281+0000","last_active":"2024-03-28T16:02:39.037953+0000","last_peered":"2024-03-28T16:02:39.037953+0000","last_clean":"2024-03-28T16:02:39.037953+0000","last_became_active":"2024-03-28T16:01:40.675734+0000","last_became_peered":"2024-03-28T16:01:40.675734+0000","last_unstale":"2024-03-28T16:02:39.037953+0000","last_undegraded":"2024-03-28T16:02:39.037953+0000","last_fullsized":"2024-03-28T16:02:39.037953+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:23:39.282010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"92'12","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.030636+0000","last_change":"2024-03-28T16:01:08.272816+0000","last_active":"2024-03-28T16:02:39.030636+0000","last_peered":"2024-03-28T16:02:39.030636+0000","last_clean":"2024-03-28T16:02:39.030636+0000","last_became_active":"2024-03-28T16:01:08.270941+0000","last_became_peered":"2024-03-28T16:01:08.270941+0000","last_unstale":"2024-03-28T16:02:39.030636+0000","last_undegraded":"2024-03-28T16:02:39.030636+0000","last_fullsized":"2024-03-28T16:02:39.030636+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:19:00.481809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"92'8","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.047233+0000","last_change":"2024-03-28T16:01:08.258558+0000","last_active":"2024-03-28T16:02:39.047233+0000","last_peered":"2024-03-28T16:02:39.047233+0000","last_clean":"2024-03-28T16:02:39.047233+0000","last_became_active":"2024-03-28T16:01:08.258163+0000","last_became_peered":"2024-03-28T16:01:08.258163+0000","last_unstale":"2024-03-28T16:02:39.047233+0000","last_undegraded":"2024-03-28T16:02:39.047233+0000","last_fullsized":"2024-03-28T16:02:39.047233+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:53:48.358536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"92'7","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.048041+0000","last_change":"2024-03-28T16:01:08.268886+0000","last_active":"2024-03-28T16:02:39.048041+0000","last_peered":"2024-03-28T16:02:39.048041+0000","last_clean":"2024-03-28T16:02:39.048041+0000","last_became_active":"2024-03-28T16:01:08.268519+0000","last_became_peered":"2024-03-28T16:01:08.268519+0000","last_unstale":"2024-03-28T16:02:39.048041+0000","last_undegraded":"2024-03-28T16:02:39.048041+0000","last_fullsized":"2024-03-28T16:02:39.048041+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:49:05.266517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"92'4","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.049066+0000","last_change":"2024-03-28T16:01:08.265039+0000","last_active":"2024-03-28T16:02:39.049066+0000","last_peered":"2024-03-28T16:02:39.049066+0000","last_clean":"2024-03-28T16:02:39.049066+0000","last_became_active":"2024-03-28T16:01:08.264677+0000","last_became_peered":"2024-03-28T16:01:08.264677+0000","last_unstale":"2024-03-28T16:02:39.049066+0000","last_undegraded":"2024-03-28T16:02:39.049066+0000","last_fullsized":"2024-03-28T16:02:39.049066+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:17:27.667324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"92'6","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.035821+0000","last_change":"2024-03-28T16:01:08.271699+0000","last_active":"2024-03-28T16:02:39.035821+0000","last_peered":"2024-03-28T16:02:39.035821+0000","last_clean":"2024-03-28T16:02:39.035821+0000","last_became_active":"2024-03-28T16:01:08.270184+0000","last_became_peered":"2024-03-28T16:01:08.270184+0000","last_unstale":"2024-03-28T16:02:39.035821+0000","last_undegraded":"2024-03-28T16:02:39.035821+0000","last_fullsized":"2024-03-28T16:02:39.035821+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:17:15.699944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"92'15","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.038913+0000","last_change":"2024-03-28T16:01:08.262314+0000","last_active":"2024-03-28T16:02:39.038913+0000","last_peered":"2024-03-28T16:02:39.038913+0000","last_clean":"2024-03-28T16:02:39.038913+0000","last_became_active":"2024-03-28T16:01:08.261992+0000","last_became_peered":"2024-03-28T16:01:08.261992+0000","last_unstale":"2024-03-28T16:02:39.038913+0000","last_undegraded":"2024-03-28T16:02:39.038913+0000","last_fullsized":"2024-03-28T16:02:39.038913+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:40:52.048306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"92'2","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.027024+0000","last_change":"2024-03-28T16:01:08.259407+0000","last_active":"2024-03-28T16:02:39.027024+0000","last_peered":"2024-03-28T16:02:39.027024+0000","last_clean":"2024-03-28T16:02:39.027024+0000","last_became_active":"2024-03-28T16:01:08.259071+0000","last_became_peered":"2024-03-28T16:01:08.259071+0000","last_unstale":"2024-03-28T16:02:39.027024+0000","last_undegraded":"2024-03-28T16:02:39.027024+0000","last_fullsized":"2024-03-28T16:02:39.027024+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:30:17.819868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"92'6","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.039238+0000","last_change":"2024-03-28T16:01:08.271649+0000","last_active":"2024-03-28T16:02:39.039238+0000","last_peered":"2024-03-28T16:02:39.039238+0000","last_clean":"2024-03-28T16:02:39.039238+0000","last_became_active":"2024-03-28T16:01:08.271230+0000","last_became_peered":"2024-03-28T16:01:08.271230+0000","last_unstale":"2024-03-28T16:02:39.039238+0000","last_undegraded":"2024-03-28T16:02:39.039238+0000","last_fullsized":"2024-03-28T16:02:39.039238+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:50:14.404178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"92'8","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.031938+0000","last_change":"2024-03-28T16:01:08.260845+0000","last_active":"2024-03-28T16:02:39.031938+0000","last_peered":"2024-03-28T16:02:39.031938+0000","last_clean":"2024-03-28T16:02:39.031938+0000","last_became_active":"2024-03-28T16:01:08.259997+0000","last_became_peered":"2024-03-28T16:01:08.259997+0000","last_unstale":"2024-03-28T16:02:39.031938+0000","last_undegraded":"2024-03-28T16:02:39.031938+0000","last_fullsized":"2024-03-28T16:02:39.031938+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:08:55.080858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"92'10","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.050870+0000","last_change":"2024-03-28T16:01:08.268881+0000","last_active":"2024-03-28T16:02:39.050870+0000","last_peered":"2024-03-28T16:02:39.050870+0000","last_clean":"2024-03-28T16:02:39.050870+0000","last_became_active":"2024-03-28T16:01:08.268212+0000","last_became_peered":"2024-03-28T16:01:08.268212+0000","last_unstale":"2024-03-28T16:02:39.050870+0000","last_undegraded":"2024-03-28T16:02:39.050870+0000","last_fullsized":"2024-03-28T16:02:39.050870+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:46:22.447763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"92'11","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.051690+0000","last_change":"2024-03-28T16:01:08.265113+0000","last_active":"2024-03-28T16:02:39.051690+0000","last_peered":"2024-03-28T16:02:39.051690+0000","last_clean":"2024-03-28T16:02:39.051690+0000","last_became_active":"2024-03-28T16:01:08.264781+0000","last_became_peered":"2024-03-28T16:01:08.264781+0000","last_unstale":"2024-03-28T16:02:39.051690+0000","last_undegraded":"2024-03-28T16:02:39.051690+0000","last_fullsized":"2024-03-28T16:02:39.051690+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:19:14.720260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"92'8","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.030628+0000","last_change":"2024-03-28T16:01:08.274446+0000","last_active":"2024-03-28T16:02:39.030628+0000","last_peered":"2024-03-28T16:02:39.030628+0000","last_clean":"2024-03-28T16:02:39.030628+0000","last_became_active":"2024-03-28T16:01:08.274027+0000","last_became_peered":"2024-03-28T16:01:08.274027+0000","last_unstale":"2024-03-28T16:02:39.030628+0000","last_undegraded":"2024-03-28T16:02:39.030628+0000","last_fullsized":"2024-03-28T16:02:39.030628+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:33:37.395889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":63436,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":281,"num_read_kb":204,"num_write":88,"num_write_kb":91,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":190308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":85,"seq":365072220204,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":478849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":77,"seq":330712481843,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32812,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712340,"statfs":{"total":95995035648,"available":95961436160,"internally_reserved":0,"allocated":1675264,"data_stored":1132144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":71,"seq":304942678074,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":479409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":63,"seq":270582939713,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":479409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":55,"seq":236223201352,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36892,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708260,"statfs":{"total":95995035648,"available":95957258240,"internally_reserved":0,"allocated":1593344,"data_stored":1069894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397583,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36308,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708844,"statfs":{"total":95995035648,"available":95957856256,"internally_reserved":0,"allocated":995328,"data_stored":478944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574960,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28056,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717096,"statfs":{"total":95995035648,"available":95966306304,"internally_reserved":0,"allocated":1064960,"data_stored":541465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738485,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36308,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708844,"statfs":{"total":95995035648,"available":95957856256,"internally_reserved":0,"allocated":995328,"data_stored":478917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411434,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1069056,"data_stored":541499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117090,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36320,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708832,"statfs":{"total":95995035648,"available":95957843968,"internally_reserved":0,"allocated":1007616,"data_stored":479165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855452,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":478849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659222,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36900,"kb_used_data":1564,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708252,"statfs":{"total":95995035648,"available":95957250048,"internally_reserved":0,"allocated":1601536,"data_stored":1069494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":62638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":62672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":62832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T16:03:29.478 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T16:03:29.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.745+0000 7f502e000640 1 -- 172.21.15.67:0/3576870063 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50281048e0 msgr2=0x7f5028104d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:29.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.745+0000 7f502e000640 1 --2- 172.21.15.67:0/3576870063 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50281048e0 0x7f5028104d60 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f5014009900 tx=0x7f501402bc50 comp rx=0 tx=0).stop 2024-03-28T16:03:29.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.745+0000 7f502e000640 1 -- 172.21.15.67:0/3576870063 shutdown_connections 2024-03-28T16:03:29.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.745+0000 7f502e000640 1 --2- 172.21.15.67:0/3576870063 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f50281052a0 0x7f502810bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:29.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.745+0000 7f502e000640 1 --2- 172.21.15.67:0/3576870063 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50281048e0 0x7f5028104d60 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:29.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.745+0000 7f502e000640 1 --2- 172.21.15.67:0/3576870063 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f50281036f0 0x7f5028103b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:29.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.745+0000 7f502e000640 1 -- 172.21.15.67:0/3576870063 >> 172.21.15.67:0/3576870063 conn(0x7f50280fec80 msgr2=0x7f50281010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:29.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.746+0000 7f502e000640 1 -- 172.21.15.67:0/3576870063 shutdown_connections 2024-03-28T16:03:29.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.746+0000 7f502e000640 1 -- 172.21.15.67:0/3576870063 wait complete. 2024-03-28T16:03:29.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.747+0000 7f502e000640 1 Processor -- start 2024-03-28T16:03:29.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.747+0000 7f502e000640 1 -- start start 2024-03-28T16:03:29.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.747+0000 7f502e000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f50281036f0 0x7f50281998e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:29.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.748+0000 7f502e000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50281048e0 0x7f5028199e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:29.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.748+0000 7f502e000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f50281052a0 0x7f50281a0e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:29.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.748+0000 7f502e000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f502810f220 con 0x7f50281048e0 2024-03-28T16:03:29.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.748+0000 7f502e000640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f502810f0a0 con 0x7f50281052a0 2024-03-28T16:03:29.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.748+0000 7f502e000640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f502810f3a0 con 0x7f50281036f0 2024-03-28T16:03:29.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.748+0000 7f5027e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50281048e0 0x7f5028199e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:29.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.748+0000 7f5027e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50281048e0 0x7f5028199e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50608/0 (socket says 172.21.15.67:50608) 2024-03-28T16:03:29.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.748+0000 7f5027e00640 1 -- 172.21.15.67:0/2343388401 learned_addr learned my addr 172.21.15.67:0/2343388401 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:03:29.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.748+0000 7f502cc00640 1 --2- 172.21.15.67:0/2343388401 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f50281036f0 0x7f50281998e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:29.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.748+0000 7f502d600640 1 --2- 172.21.15.67:0/2343388401 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f50281052a0 0x7f50281a0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:29.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.748+0000 7f5027e00640 1 -- 172.21.15.67:0/2343388401 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f50281036f0 msgr2=0x7f50281998e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:29.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.748+0000 7f5027e00640 1 --2- 172.21.15.67:0/2343388401 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f50281036f0 0x7f50281998e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:29.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.748+0000 7f5027e00640 1 -- 172.21.15.67:0/2343388401 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f50281052a0 msgr2=0x7f50281a0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:29.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.748+0000 7f5027e00640 1 --2- 172.21.15.67:0/2343388401 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f50281052a0 0x7f50281a0e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:29.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.748+0000 7f5027e00640 1 -- 172.21.15.67:0/2343388401 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50281a1500 con 0x7f50281048e0 2024-03-28T16:03:29.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.748+0000 7f502d600640 1 --2- 172.21.15.67:0/2343388401 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f50281052a0 0x7f50281a0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:03:29.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.749+0000 7f5027e00640 1 --2- 172.21.15.67:0/2343388401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50281048e0 0x7f5028199e20 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f5014006010 tx=0x7f5014004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:29.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.749+0000 7f5025600640 1 -- 172.21.15.67:0/2343388401 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5014043020 con 0x7f50281048e0 2024-03-28T16:03:29.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.749+0000 7f5025600640 1 -- 172.21.15.67:0/2343388401 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5014004440 con 0x7f50281048e0 2024-03-28T16:03:29.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.749+0000 7f502e000640 1 -- 172.21.15.67:0/2343388401 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50281a1790 con 0x7f50281048e0 2024-03-28T16:03:29.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.749+0000 7f5025600640 1 -- 172.21.15.67:0/2343388401 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5014003bf0 con 0x7f50281048e0 2024-03-28T16:03:29.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.749+0000 7f502e000640 1 -- 172.21.15.67:0/2343388401 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50281a1d30 con 0x7f50281048e0 2024-03-28T16:03:29.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.751+0000 7f502e000640 1 -- 172.21.15.67:0/2343388401 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f502810aa10 con 0x7f50281048e0 2024-03-28T16:03:29.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.753+0000 7f5025600640 1 -- 172.21.15.67:0/2343388401 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f5014046020 con 0x7f50281048e0 2024-03-28T16:03:29.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.755+0000 7f5025600640 1 --2- 172.21.15.67:0/2343388401 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f50100a2880 0x7f50100a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:29.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.755+0000 7f5025600640 1 -- 172.21.15.67:0/2343388401 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f50140f0ea0 con 0x7f50281048e0 2024-03-28T16:03:29.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.755+0000 7f502cc00640 1 --2- 172.21.15.67:0/2343388401 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f50100a2880 0x7f50100a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:29.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.756+0000 7f502cc00640 1 --2- 172.21.15.67:0/2343388401 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f50100a2880 0x7f50100a4d40 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f5028104740 tx=0x7f501c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:29.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.759+0000 7f5025600640 1 -- 172.21.15.67:0/2343388401 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f501404d020 con 0x7f50281048e0 2024-03-28T16:03:29.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.963+0000 7f502e000640 1 -- 172.21.15.67:0/2343388401 --> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5028061ae0 con 0x7f50100a2880 2024-03-28T16:03:29.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.983+0000 7f5025600640 1 -- 172.21.15.67:0/2343388401 <== mgr.14152 v2:172.21.15.67:6800/208726654 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+258821 (secure 0 0 0) 0x7f5028061ae0 con 0x7f50100a2880 2024-03-28T16:03:29.987 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:03:29.988 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-28T16:03:29.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.991+0000 7f502e000640 1 -- 172.21.15.67:0/2343388401 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f50100a2880 msgr2=0x7f50100a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:29.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.991+0000 7f502e000640 1 --2- 172.21.15.67:0/2343388401 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f50100a2880 0x7f50100a4d40 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f5028104740 tx=0x7f501c005f90 comp rx=0 tx=0).stop 2024-03-28T16:03:29.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.991+0000 7f502e000640 1 -- 172.21.15.67:0/2343388401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50281048e0 msgr2=0x7f5028199e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:29.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.991+0000 7f502e000640 1 --2- 172.21.15.67:0/2343388401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50281048e0 0x7f5028199e20 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f5014006010 tx=0x7f5014004040 comp rx=0 tx=0).stop 2024-03-28T16:03:29.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.991+0000 7f502e000640 1 -- 172.21.15.67:0/2343388401 shutdown_connections 2024-03-28T16:03:29.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.991+0000 7f502e000640 1 --2- 172.21.15.67:0/2343388401 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f50100a2880 0x7f50100a4d40 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:29.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.991+0000 7f502e000640 1 --2- 172.21.15.67:0/2343388401 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f50281052a0 0x7f50281a0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:29.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.991+0000 7f502e000640 1 --2- 172.21.15.67:0/2343388401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50281048e0 0x7f5028199e20 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:29.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.991+0000 7f502e000640 1 --2- 172.21.15.67:0/2343388401 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f50281036f0 0x7f50281998e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:29.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.992+0000 7f502e000640 1 -- 172.21.15.67:0/2343388401 >> 172.21.15.67:0/2343388401 conn(0x7f50280fec80 msgr2=0x7f5028107be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:29.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.992+0000 7f502e000640 1 -- 172.21.15.67:0/2343388401 shutdown_connections 2024-03-28T16:03:29.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:29.992+0000 7f502e000640 1 -- 172.21.15.67:0/2343388401 wait complete. 2024-03-28T16:03:30.016 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":437,"stamp":"2024-03-28T16:03:29.097067+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":654038,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":407,"num_read_kb":313,"num_write":271,"num_write_kb":3031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1180758,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":712,"ondisk_log_size":712,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":383608,"kb_used_data":13656,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124558216,"statfs":{"total":1151940427776,"available":1151547613184,"internally_reserved":0,"allocated":13983744,"data_stored":7708038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":378713402},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":493,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":11,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004831"},"pg_stats":[{"pgid":"3.3f","version":"92'8","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.049941+0000","last_change":"2024-03-28T16:01:08.268196+0000","last_active":"2024-03-28T16:02:39.049941+0000","last_peered":"2024-03-28T16:02:39.049941+0000","last_clean":"2024-03-28T16:02:39.049941+0000","last_became_active":"2024-03-28T16:01:08.266956+0000","last_became_peered":"2024-03-28T16:01:08.266956+0000","last_unstale":"2024-03-28T16:02:39.049941+0000","last_undegraded":"2024-03-28T16:02:39.049941+0000","last_fullsized":"2024-03-28T16:02:39.049941+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:38:44.296129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"92'4","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.048446+0000","last_change":"2024-03-28T16:01:08.264985+0000","last_active":"2024-03-28T16:02:39.048446+0000","last_peered":"2024-03-28T16:02:39.048446+0000","last_clean":"2024-03-28T16:02:39.048446+0000","last_became_active":"2024-03-28T16:01:08.262484+0000","last_became_peered":"2024-03-28T16:01:08.262484+0000","last_unstale":"2024-03-28T16:02:39.048446+0000","last_undegraded":"2024-03-28T16:02:39.048446+0000","last_fullsized":"2024-03-28T16:02:39.048446+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:33:07.538908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"92'7","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.051488+0000","last_change":"2024-03-28T16:01:08.267423+0000","last_active":"2024-03-28T16:02:39.051488+0000","last_peered":"2024-03-28T16:02:39.051488+0000","last_clean":"2024-03-28T16:02:39.051488+0000","last_became_active":"2024-03-28T16:01:08.263665+0000","last_became_peered":"2024-03-28T16:01:08.263665+0000","last_unstale":"2024-03-28T16:02:39.051488+0000","last_undegraded":"2024-03-28T16:02:39.051488+0000","last_fullsized":"2024-03-28T16:02:39.051488+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:09:43.488003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"92'8","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.028028+0000","last_change":"2024-03-28T16:01:08.274978+0000","last_active":"2024-03-28T16:02:39.028028+0000","last_peered":"2024-03-28T16:02:39.028028+0000","last_clean":"2024-03-28T16:02:39.028028+0000","last_became_active":"2024-03-28T16:01:08.274553+0000","last_became_peered":"2024-03-28T16:01:08.274553+0000","last_unstale":"2024-03-28T16:02:39.028028+0000","last_undegraded":"2024-03-28T16:02:39.028028+0000","last_fullsized":"2024-03-28T16:02:39.028028+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:16:03.997334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"92'4","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.042170+0000","last_change":"2024-03-28T16:01:08.262155+0000","last_active":"2024-03-28T16:02:39.042170+0000","last_peered":"2024-03-28T16:02:39.042170+0000","last_clean":"2024-03-28T16:02:39.042170+0000","last_became_active":"2024-03-28T16:01:08.261814+0000","last_became_peered":"2024-03-28T16:01:08.261814+0000","last_unstale":"2024-03-28T16:02:39.042170+0000","last_undegraded":"2024-03-28T16:02:39.042170+0000","last_fullsized":"2024-03-28T16:02:39.042170+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:41:04.242203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"92'6","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.023511+0000","last_change":"2024-03-28T16:01:08.273342+0000","last_active":"2024-03-28T16:02:39.023511+0000","last_peered":"2024-03-28T16:02:39.023511+0000","last_clean":"2024-03-28T16:02:39.023511+0000","last_became_active":"2024-03-28T16:01:08.272606+0000","last_became_peered":"2024-03-28T16:01:08.272606+0000","last_unstale":"2024-03-28T16:02:39.023511+0000","last_undegraded":"2024-03-28T16:02:39.023511+0000","last_fullsized":"2024-03-28T16:02:39.023511+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:44:48.580214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"92'5","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.040848+0000","last_change":"2024-03-28T16:01:08.278060+0000","last_active":"2024-03-28T16:02:39.040848+0000","last_peered":"2024-03-28T16:02:39.040848+0000","last_clean":"2024-03-28T16:02:39.040848+0000","last_became_active":"2024-03-28T16:01:08.277047+0000","last_became_peered":"2024-03-28T16:01:08.277047+0000","last_unstale":"2024-03-28T16:02:39.040848+0000","last_undegraded":"2024-03-28T16:02:39.040848+0000","last_fullsized":"2024-03-28T16:02:39.040848+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:46:30.005708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"92'9","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.036559+0000","last_change":"2024-03-28T16:01:08.273199+0000","last_active":"2024-03-28T16:02:39.036559+0000","last_peered":"2024-03-28T16:02:39.036559+0000","last_clean":"2024-03-28T16:02:39.036559+0000","last_became_active":"2024-03-28T16:01:08.272822+0000","last_became_peered":"2024-03-28T16:01:08.272822+0000","last_unstale":"2024-03-28T16:02:39.036559+0000","last_undegraded":"2024-03-28T16:02:39.036559+0000","last_fullsized":"2024-03-28T16:02:39.036559+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:28:11.495665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"92'11","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.050965+0000","last_change":"2024-03-28T16:01:08.270262+0000","last_active":"2024-03-28T16:02:39.050965+0000","last_peered":"2024-03-28T16:02:39.050965+0000","last_clean":"2024-03-28T16:02:39.050965+0000","last_became_active":"2024-03-28T16:01:08.269959+0000","last_became_peered":"2024-03-28T16:01:08.269959+0000","last_unstale":"2024-03-28T16:02:39.050965+0000","last_undegraded":"2024-03-28T16:02:39.050965+0000","last_fullsized":"2024-03-28T16:02:39.050965+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:49:02.261893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"92'16","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.028205+0000","last_change":"2024-03-28T16:01:40.671104+0000","last_active":"2024-03-28T16:02:39.028205+0000","last_peered":"2024-03-28T16:02:39.028205+0000","last_clean":"2024-03-28T16:02:39.028205+0000","last_became_active":"2024-03-28T16:01:40.670474+0000","last_became_peered":"2024-03-28T16:01:40.670474+0000","last_unstale":"2024-03-28T16:02:39.028205+0000","last_undegraded":"2024-03-28T16:02:39.028205+0000","last_fullsized":"2024-03-28T16:02:39.028205+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:19:41.651703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"92'6","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.052808+0000","last_change":"2024-03-28T16:01:08.274943+0000","last_active":"2024-03-28T16:02:39.052808+0000","last_peered":"2024-03-28T16:02:39.052808+0000","last_clean":"2024-03-28T16:02:39.052808+0000","last_became_active":"2024-03-28T16:01:08.274267+0000","last_became_peered":"2024-03-28T16:01:08.274267+0000","last_unstale":"2024-03-28T16:02:39.052808+0000","last_undegraded":"2024-03-28T16:02:39.052808+0000","last_fullsized":"2024-03-28T16:02:39.052808+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:41:13.979918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"92'7","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.020548+0000","last_change":"2024-03-28T16:01:08.266817+0000","last_active":"2024-03-28T16:02:39.020548+0000","last_peered":"2024-03-28T16:02:39.020548+0000","last_clean":"2024-03-28T16:02:39.020548+0000","last_became_active":"2024-03-28T16:01:08.266470+0000","last_became_peered":"2024-03-28T16:01:08.266470+0000","last_unstale":"2024-03-28T16:02:39.020548+0000","last_undegraded":"2024-03-28T16:02:39.020548+0000","last_fullsized":"2024-03-28T16:02:39.020548+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:15:28.573153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"92'10","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.052174+0000","last_change":"2024-03-28T16:01:08.269896+0000","last_active":"2024-03-28T16:02:39.052174+0000","last_peered":"2024-03-28T16:02:39.052174+0000","last_clean":"2024-03-28T16:02:39.052174+0000","last_became_active":"2024-03-28T16:01:08.269500+0000","last_became_peered":"2024-03-28T16:01:08.269500+0000","last_unstale":"2024-03-28T16:02:39.052174+0000","last_undegraded":"2024-03-28T16:02:39.052174+0000","last_fullsized":"2024-03-28T16:02:39.052174+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:44:38.867590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"92'7","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.047544+0000","last_change":"2024-03-28T16:01:08.267072+0000","last_active":"2024-03-28T16:02:39.047544+0000","last_peered":"2024-03-28T16:02:39.047544+0000","last_clean":"2024-03-28T16:02:39.047544+0000","last_became_active":"2024-03-28T16:01:08.266688+0000","last_became_peered":"2024-03-28T16:01:08.266688+0000","last_unstale":"2024-03-28T16:02:39.047544+0000","last_undegraded":"2024-03-28T16:02:39.047544+0000","last_fullsized":"2024-03-28T16:02:39.047544+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:07:57.702952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"92'7","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.045559+0000","last_change":"2024-03-28T16:01:08.270912+0000","last_active":"2024-03-28T16:02:39.045559+0000","last_peered":"2024-03-28T16:02:39.045559+0000","last_clean":"2024-03-28T16:02:39.045559+0000","last_became_active":"2024-03-28T16:01:08.270567+0000","last_became_peered":"2024-03-28T16:01:08.270567+0000","last_unstale":"2024-03-28T16:02:39.045559+0000","last_undegraded":"2024-03-28T16:02:39.045559+0000","last_fullsized":"2024-03-28T16:02:39.045559+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:19:29.294502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"92'9","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.050241+0000","last_change":"2024-03-28T16:01:08.266811+0000","last_active":"2024-03-28T16:02:39.050241+0000","last_peered":"2024-03-28T16:02:39.050241+0000","last_clean":"2024-03-28T16:02:39.050241+0000","last_became_active":"2024-03-28T16:01:08.266517+0000","last_became_peered":"2024-03-28T16:01:08.266517+0000","last_unstale":"2024-03-28T16:02:39.050241+0000","last_undegraded":"2024-03-28T16:02:39.050241+0000","last_fullsized":"2024-03-28T16:02:39.050241+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:55:05.972683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"92'8","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.050243+0000","last_change":"2024-03-28T16:01:08.263864+0000","last_active":"2024-03-28T16:02:39.050243+0000","last_peered":"2024-03-28T16:02:39.050243+0000","last_clean":"2024-03-28T16:02:39.050243+0000","last_became_active":"2024-03-28T16:01:08.262956+0000","last_became_peered":"2024-03-28T16:01:08.262956+0000","last_unstale":"2024-03-28T16:02:39.050243+0000","last_undegraded":"2024-03-28T16:02:39.050243+0000","last_fullsized":"2024-03-28T16:02:39.050243+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:05:50.504237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"92'10","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.045902+0000","last_change":"2024-03-28T16:01:08.265151+0000","last_active":"2024-03-28T16:02:39.045902+0000","last_peered":"2024-03-28T16:02:39.045902+0000","last_clean":"2024-03-28T16:02:39.045902+0000","last_became_active":"2024-03-28T16:01:08.264861+0000","last_became_peered":"2024-03-28T16:01:08.264861+0000","last_unstale":"2024-03-28T16:02:39.045902+0000","last_undegraded":"2024-03-28T16:02:39.045902+0000","last_fullsized":"2024-03-28T16:02:39.045902+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:35:45.878983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"92'7","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.037162+0000","last_change":"2024-03-28T16:01:08.265993+0000","last_active":"2024-03-28T16:02:39.037162+0000","last_peered":"2024-03-28T16:02:39.037162+0000","last_clean":"2024-03-28T16:02:39.037162+0000","last_became_active":"2024-03-28T16:01:08.265488+0000","last_became_peered":"2024-03-28T16:01:08.265488+0000","last_unstale":"2024-03-28T16:02:39.037162+0000","last_undegraded":"2024-03-28T16:02:39.037162+0000","last_fullsized":"2024-03-28T16:02:39.037162+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:31:55.920705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'4","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.654831+0000","last_change":"2024-03-28T16:01:07.250361+0000","last_active":"2024-03-28T16:01:40.654831+0000","last_peered":"2024-03-28T16:01:40.654831+0000","last_clean":"2024-03-28T16:01:40.654831+0000","last_became_active":"2024-03-28T16:01:07.250004+0000","last_became_peered":"2024-03-28T16:01:07.250004+0000","last_unstale":"2024-03-28T16:01:40.654831+0000","last_undegraded":"2024-03-28T16:01:40.654831+0000","last_fullsized":"2024-03-28T16:01:40.654831+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:10:32.838716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"92'5","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.032768+0000","last_change":"2024-03-28T16:01:08.274436+0000","last_active":"2024-03-28T16:02:39.032768+0000","last_peered":"2024-03-28T16:02:39.032768+0000","last_clean":"2024-03-28T16:02:39.032768+0000","last_became_active":"2024-03-28T16:01:08.274124+0000","last_became_peered":"2024-03-28T16:01:08.274124+0000","last_unstale":"2024-03-28T16:02:39.032768+0000","last_undegraded":"2024-03-28T16:02:39.032768+0000","last_fullsized":"2024-03-28T16:02:39.032768+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:02:34.037686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"89'1","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.765443+0000","last_change":"2024-03-28T16:01:07.248565+0000","last_active":"2024-03-28T16:01:40.765443+0000","last_peered":"2024-03-28T16:01:40.765443+0000","last_clean":"2024-03-28T16:01:40.765443+0000","last_became_active":"2024-03-28T16:01:07.247366+0000","last_became_peered":"2024-03-28T16:01:07.247366+0000","last_unstale":"2024-03-28T16:01:40.765443+0000","last_undegraded":"2024-03-28T16:01:40.765443+0000","last_fullsized":"2024-03-28T16:01:40.765443+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:48:13.146398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"92'6","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.051966+0000","last_change":"2024-03-28T16:01:08.265639+0000","last_active":"2024-03-28T16:02:39.051966+0000","last_peered":"2024-03-28T16:02:39.051966+0000","last_clean":"2024-03-28T16:02:39.051966+0000","last_became_active":"2024-03-28T16:01:08.263488+0000","last_became_peered":"2024-03-28T16:01:08.263488+0000","last_unstale":"2024-03-28T16:02:39.051966+0000","last_undegraded":"2024-03-28T16:02:39.051966+0000","last_fullsized":"2024-03-28T16:02:39.051966+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:15:06.516016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"89'2","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.850755+0000","last_change":"2024-03-28T16:01:40.850755+0000","last_active":"2024-03-28T16:01:40.850755+0000","last_peered":"2024-03-28T16:01:40.850755+0000","last_clean":"2024-03-28T16:01:40.850755+0000","last_became_active":"2024-03-28T16:01:40.842547+0000","last_became_peered":"2024-03-28T16:01:40.842547+0000","last_unstale":"2024-03-28T16:01:40.850755+0000","last_undegraded":"2024-03-28T16:01:40.850755+0000","last_fullsized":"2024-03-28T16:01:40.850755+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:39:27.862525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"92'9","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.051889+0000","last_change":"2024-03-28T16:01:08.271534+0000","last_active":"2024-03-28T16:02:39.051889+0000","last_peered":"2024-03-28T16:02:39.051889+0000","last_clean":"2024-03-28T16:02:39.051889+0000","last_became_active":"2024-03-28T16:01:08.271104+0000","last_became_peered":"2024-03-28T16:01:08.271104+0000","last_unstale":"2024-03-28T16:02:39.051889+0000","last_undegraded":"2024-03-28T16:02:39.051889+0000","last_fullsized":"2024-03-28T16:02:39.051889+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:29:21.332346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"89'2","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.655555+0000","last_change":"2024-03-28T16:01:07.259175+0000","last_active":"2024-03-28T16:01:40.655555+0000","last_peered":"2024-03-28T16:01:40.655555+0000","last_clean":"2024-03-28T16:01:40.655555+0000","last_became_active":"2024-03-28T16:01:07.258151+0000","last_became_peered":"2024-03-28T16:01:07.258151+0000","last_unstale":"2024-03-28T16:01:40.655555+0000","last_undegraded":"2024-03-28T16:01:40.655555+0000","last_fullsized":"2024-03-28T16:01:40.655555+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:41:07.064800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"92'15","reported_seq":38,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.046924+0000","last_change":"2024-03-28T16:01:40.843211+0000","last_active":"2024-03-28T16:02:39.046924+0000","last_peered":"2024-03-28T16:02:39.046924+0000","last_clean":"2024-03-28T16:02:39.046924+0000","last_became_active":"2024-03-28T16:01:40.842051+0000","last_became_peered":"2024-03-28T16:01:40.842051+0000","last_unstale":"2024-03-28T16:02:39.046924+0000","last_undegraded":"2024-03-28T16:02:39.046924+0000","last_fullsized":"2024-03-28T16:02:39.046924+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:19:26.437169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"89'1","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.664703+0000","last_change":"2024-03-28T16:01:07.248538+0000","last_active":"2024-03-28T16:01:40.664703+0000","last_peered":"2024-03-28T16:01:40.664703+0000","last_clean":"2024-03-28T16:01:40.664703+0000","last_became_active":"2024-03-28T16:01:07.248116+0000","last_became_peered":"2024-03-28T16:01:07.248116+0000","last_unstale":"2024-03-28T16:01:40.664703+0000","last_undegraded":"2024-03-28T16:01:40.664703+0000","last_fullsized":"2024-03-28T16:01:40.664703+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:44:56.071286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"92'7","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.033579+0000","last_change":"2024-03-28T16:01:08.271059+0000","last_active":"2024-03-28T16:02:39.033579+0000","last_peered":"2024-03-28T16:02:39.033579+0000","last_clean":"2024-03-28T16:02:39.033579+0000","last_became_active":"2024-03-28T16:01:08.270682+0000","last_became_peered":"2024-03-28T16:01:08.270682+0000","last_unstale":"2024-03-28T16:02:39.033579+0000","last_undegraded":"2024-03-28T16:02:39.033579+0000","last_fullsized":"2024-03-28T16:02:39.033579+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:54:48.922125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"89'1","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.661404+0000","last_change":"2024-03-28T16:01:07.261157+0000","last_active":"2024-03-28T16:01:40.661404+0000","last_peered":"2024-03-28T16:01:40.661404+0000","last_clean":"2024-03-28T16:01:40.661404+0000","last_became_active":"2024-03-28T16:01:07.260751+0000","last_became_peered":"2024-03-28T16:01:07.260751+0000","last_unstale":"2024-03-28T16:01:40.661404+0000","last_undegraded":"2024-03-28T16:01:40.661404+0000","last_fullsized":"2024-03-28T16:01:40.661404+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:16:49.622168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"92'8","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.038730+0000","last_change":"2024-03-28T16:01:08.265773+0000","last_active":"2024-03-28T16:02:39.038730+0000","last_peered":"2024-03-28T16:02:39.038730+0000","last_clean":"2024-03-28T16:02:39.038730+0000","last_became_active":"2024-03-28T16:01:08.263724+0000","last_became_peered":"2024-03-28T16:01:08.263724+0000","last_unstale":"2024-03-28T16:02:39.038730+0000","last_undegraded":"2024-03-28T16:02:39.038730+0000","last_fullsized":"2024-03-28T16:02:39.038730+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:42:37.338946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.921045+0000","last_change":"2024-03-28T16:01:07.235361+0000","last_active":"2024-03-28T16:01:40.921045+0000","last_peered":"2024-03-28T16:01:40.921045+0000","last_clean":"2024-03-28T16:01:40.921045+0000","last_became_active":"2024-03-28T16:01:07.234969+0000","last_became_peered":"2024-03-28T16:01:07.234969+0000","last_unstale":"2024-03-28T16:01:40.921045+0000","last_undegraded":"2024-03-28T16:01:40.921045+0000","last_fullsized":"2024-03-28T16:01:40.921045+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:50:35.592147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"92'8","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.039851+0000","last_change":"2024-03-28T16:01:08.263878+0000","last_active":"2024-03-28T16:02:39.039851+0000","last_peered":"2024-03-28T16:02:39.039851+0000","last_clean":"2024-03-28T16:02:39.039851+0000","last_became_active":"2024-03-28T16:01:08.263549+0000","last_became_peered":"2024-03-28T16:01:08.263549+0000","last_unstale":"2024-03-28T16:02:39.039851+0000","last_undegraded":"2024-03-28T16:02:39.039851+0000","last_fullsized":"2024-03-28T16:02:39.039851+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:31:37.669215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.663697+0000","last_change":"2024-03-28T16:01:07.255701+0000","last_active":"2024-03-28T16:01:40.663697+0000","last_peered":"2024-03-28T16:01:40.663697+0000","last_clean":"2024-03-28T16:01:40.663697+0000","last_became_active":"2024-03-28T16:01:07.255247+0000","last_became_peered":"2024-03-28T16:01:07.255247+0000","last_unstale":"2024-03-28T16:01:40.663697+0000","last_undegraded":"2024-03-28T16:01:40.663697+0000","last_fullsized":"2024-03-28T16:01:40.663697+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:08:17.858395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"92'11","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.034304+0000","last_change":"2024-03-28T16:01:08.252949+0000","last_active":"2024-03-28T16:02:39.034304+0000","last_peered":"2024-03-28T16:02:39.034304+0000","last_clean":"2024-03-28T16:02:39.034304+0000","last_became_active":"2024-03-28T16:01:08.252471+0000","last_became_peered":"2024-03-28T16:01:08.252471+0000","last_unstale":"2024-03-28T16:02:39.034304+0000","last_undegraded":"2024-03-28T16:02:39.034304+0000","last_fullsized":"2024-03-28T16:02:39.034304+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:21:15.519020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.661736+0000","last_change":"2024-03-28T16:01:07.263973+0000","last_active":"2024-03-28T16:01:40.661736+0000","last_peered":"2024-03-28T16:01:40.661736+0000","last_clean":"2024-03-28T16:01:40.661736+0000","last_became_active":"2024-03-28T16:01:07.263560+0000","last_became_peered":"2024-03-28T16:01:07.263560+0000","last_unstale":"2024-03-28T16:01:40.661736+0000","last_undegraded":"2024-03-28T16:01:40.661736+0000","last_fullsized":"2024-03-28T16:01:40.661736+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:11.215312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"92'10","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.051393+0000","last_change":"2024-03-28T16:01:08.266122+0000","last_active":"2024-03-28T16:02:39.051393+0000","last_peered":"2024-03-28T16:02:39.051393+0000","last_clean":"2024-03-28T16:02:39.051393+0000","last_became_active":"2024-03-28T16:01:08.265727+0000","last_became_peered":"2024-03-28T16:01:08.265727+0000","last_unstale":"2024-03-28T16:02:39.051393+0000","last_undegraded":"2024-03-28T16:02:39.051393+0000","last_fullsized":"2024-03-28T16:02:39.051393+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:48:38.939777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.666951+0000","last_change":"2024-03-28T16:01:07.251814+0000","last_active":"2024-03-28T16:01:40.666951+0000","last_peered":"2024-03-28T16:01:40.666951+0000","last_clean":"2024-03-28T16:01:40.666951+0000","last_became_active":"2024-03-28T16:01:07.251311+0000","last_became_peered":"2024-03-28T16:01:07.251311+0000","last_unstale":"2024-03-28T16:01:40.666951+0000","last_undegraded":"2024-03-28T16:01:40.666951+0000","last_fullsized":"2024-03-28T16:01:40.666951+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:13:38.344070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"92'9","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.049556+0000","last_change":"2024-03-28T16:01:08.270879+0000","last_active":"2024-03-28T16:02:39.049556+0000","last_peered":"2024-03-28T16:02:39.049556+0000","last_clean":"2024-03-28T16:02:39.049556+0000","last_became_active":"2024-03-28T16:01:08.270513+0000","last_became_peered":"2024-03-28T16:01:08.270513+0000","last_unstale":"2024-03-28T16:02:39.049556+0000","last_undegraded":"2024-03-28T16:02:39.049556+0000","last_fullsized":"2024-03-28T16:02:39.049556+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:53:43.285797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"89'3","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.665900+0000","last_change":"2024-03-28T16:01:07.253595+0000","last_active":"2024-03-28T16:01:40.665900+0000","last_peered":"2024-03-28T16:01:40.665900+0000","last_clean":"2024-03-28T16:01:40.665900+0000","last_became_active":"2024-03-28T16:01:07.253137+0000","last_became_peered":"2024-03-28T16:01:07.253137+0000","last_unstale":"2024-03-28T16:01:40.665900+0000","last_undegraded":"2024-03-28T16:01:40.665900+0000","last_fullsized":"2024-03-28T16:01:40.665900+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:17:35.759791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"92'11","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.050280+0000","last_change":"2024-03-28T16:01:08.261185+0000","last_active":"2024-03-28T16:02:39.050280+0000","last_peered":"2024-03-28T16:02:39.050280+0000","last_clean":"2024-03-28T16:02:39.050280+0000","last_became_active":"2024-03-28T16:01:08.260885+0000","last_became_peered":"2024-03-28T16:01:08.260885+0000","last_unstale":"2024-03-28T16:02:39.050280+0000","last_undegraded":"2024-03-28T16:02:39.050280+0000","last_fullsized":"2024-03-28T16:02:39.050280+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:15:56.640586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"92'31","reported_seq":326,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:03:23.893167+0000","last_change":"2024-03-28T16:01:07.258655+0000","last_active":"2024-03-28T16:03:23.893167+0000","last_peered":"2024-03-28T16:03:23.893167+0000","last_clean":"2024-03-28T16:03:23.893167+0000","last_became_active":"2024-03-28T16:01:07.258230+0000","last_became_peered":"2024-03-28T16:01:07.258230+0000","last_unstale":"2024-03-28T16:03:23.893167+0000","last_undegraded":"2024-03-28T16:03:23.893167+0000","last_fullsized":"2024-03-28T16:03:23.893167+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:29:36.206061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62638,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":276,"num_read_kb":199,"num_write":31,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"92'6","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.038343+0000","last_change":"2024-03-28T16:01:08.265393+0000","last_active":"2024-03-28T16:02:39.038343+0000","last_peered":"2024-03-28T16:02:39.038343+0000","last_clean":"2024-03-28T16:02:39.038343+0000","last_became_active":"2024-03-28T16:01:08.264560+0000","last_became_peered":"2024-03-28T16:01:08.264560+0000","last_unstale":"2024-03-28T16:02:39.038343+0000","last_undegraded":"2024-03-28T16:02:39.038343+0000","last_fullsized":"2024-03-28T16:02:39.038343+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:43:52.181113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"89'2","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.672781+0000","last_change":"2024-03-28T16:01:07.251428+0000","last_active":"2024-03-28T16:01:40.672781+0000","last_peered":"2024-03-28T16:01:40.672781+0000","last_clean":"2024-03-28T16:01:40.672781+0000","last_became_active":"2024-03-28T16:01:07.251074+0000","last_became_peered":"2024-03-28T16:01:07.251074+0000","last_unstale":"2024-03-28T16:01:40.672781+0000","last_undegraded":"2024-03-28T16:01:40.672781+0000","last_fullsized":"2024-03-28T16:01:40.672781+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:18:41.359338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"92'9","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.037592+0000","last_change":"2024-03-28T16:01:08.268482+0000","last_active":"2024-03-28T16:02:39.037592+0000","last_peered":"2024-03-28T16:02:39.037592+0000","last_clean":"2024-03-28T16:02:39.037592+0000","last_became_active":"2024-03-28T16:01:08.268153+0000","last_became_peered":"2024-03-28T16:01:08.268153+0000","last_unstale":"2024-03-28T16:02:39.037592+0000","last_undegraded":"2024-03-28T16:02:39.037592+0000","last_fullsized":"2024-03-28T16:02:39.037592+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:21:21.874773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.664138+0000","last_change":"2024-03-28T16:01:07.243506+0000","last_active":"2024-03-28T16:01:40.664138+0000","last_peered":"2024-03-28T16:01:40.664138+0000","last_clean":"2024-03-28T16:01:40.664138+0000","last_became_active":"2024-03-28T16:01:07.242793+0000","last_became_peered":"2024-03-28T16:01:07.242793+0000","last_unstale":"2024-03-28T16:01:40.664138+0000","last_undegraded":"2024-03-28T16:01:40.664138+0000","last_fullsized":"2024-03-28T16:01:40.664138+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:31:56.497276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"92'15","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.044862+0000","last_change":"2024-03-28T16:01:08.278848+0000","last_active":"2024-03-28T16:02:39.044862+0000","last_peered":"2024-03-28T16:02:39.044862+0000","last_clean":"2024-03-28T16:02:39.044862+0000","last_became_active":"2024-03-28T16:01:08.278455+0000","last_became_peered":"2024-03-28T16:01:08.278455+0000","last_unstale":"2024-03-28T16:02:39.044862+0000","last_undegraded":"2024-03-28T16:02:39.044862+0000","last_fullsized":"2024-03-28T16:02:39.044862+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:41:17.302538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.655733+0000","last_change":"2024-03-28T16:01:07.255070+0000","last_active":"2024-03-28T16:01:40.655733+0000","last_peered":"2024-03-28T16:01:40.655733+0000","last_clean":"2024-03-28T16:01:40.655733+0000","last_became_active":"2024-03-28T16:01:07.253489+0000","last_became_peered":"2024-03-28T16:01:07.253489+0000","last_unstale":"2024-03-28T16:01:40.655733+0000","last_undegraded":"2024-03-28T16:01:40.655733+0000","last_fullsized":"2024-03-28T16:01:40.655733+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:31:57.501721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"92'6","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.036271+0000","last_change":"2024-03-28T16:01:08.273190+0000","last_active":"2024-03-28T16:02:39.036271+0000","last_peered":"2024-03-28T16:02:39.036271+0000","last_clean":"2024-03-28T16:02:39.036271+0000","last_became_active":"2024-03-28T16:01:08.272870+0000","last_became_peered":"2024-03-28T16:01:08.272870+0000","last_unstale":"2024-03-28T16:02:39.036271+0000","last_undegraded":"2024-03-28T16:02:39.036271+0000","last_fullsized":"2024-03-28T16:02:39.036271+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:26:50.874750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"89'8","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.700937+0000","last_change":"2024-03-28T16:01:40.700157+0000","last_active":"2024-03-28T16:01:40.700937+0000","last_peered":"2024-03-28T16:01:40.700937+0000","last_clean":"2024-03-28T16:01:40.700937+0000","last_became_active":"2024-03-28T16:01:40.674436+0000","last_became_peered":"2024-03-28T16:01:40.674436+0000","last_unstale":"2024-03-28T16:01:40.700937+0000","last_undegraded":"2024-03-28T16:01:40.700937+0000","last_fullsized":"2024-03-28T16:01:40.700937+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:26:48.776506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,4],"acting":[11,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"92'6","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.049436+0000","last_change":"2024-03-28T16:01:40.924563+0000","last_active":"2024-03-28T16:02:39.049436+0000","last_peered":"2024-03-28T16:02:39.049436+0000","last_clean":"2024-03-28T16:02:39.049436+0000","last_became_active":"2024-03-28T16:01:40.923348+0000","last_became_peered":"2024-03-28T16:01:40.923348+0000","last_unstale":"2024-03-28T16:02:39.049436+0000","last_undegraded":"2024-03-28T16:02:39.049436+0000","last_fullsized":"2024-03-28T16:02:39.049436+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:31:46.619651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'1","reported_seq":15,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.681372+0000","last_change":"2024-03-28T16:01:40.680548+0000","last_active":"2024-03-28T16:01:40.681372+0000","last_peered":"2024-03-28T16:01:40.681372+0000","last_clean":"2024-03-28T16:01:40.681372+0000","last_became_active":"2024-03-28T16:01:40.672674+0000","last_became_peered":"2024-03-28T16:01:40.672674+0000","last_unstale":"2024-03-28T16:01:40.681372+0000","last_undegraded":"2024-03-28T16:01:40.681372+0000","last_fullsized":"2024-03-28T16:01:40.681372+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:07:32.767184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"92'5","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.035565+0000","last_change":"2024-03-28T16:01:08.274436+0000","last_active":"2024-03-28T16:02:39.035565+0000","last_peered":"2024-03-28T16:02:39.035565+0000","last_clean":"2024-03-28T16:02:39.035565+0000","last_became_active":"2024-03-28T16:01:08.274081+0000","last_became_peered":"2024-03-28T16:01:08.274081+0000","last_unstale":"2024-03-28T16:02:39.035565+0000","last_undegraded":"2024-03-28T16:02:39.035565+0000","last_fullsized":"2024-03-28T16:02:39.035565+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:57:00.531064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.653153+0000","last_change":"2024-03-28T16:01:07.247542+0000","last_active":"2024-03-28T16:01:40.653153+0000","last_peered":"2024-03-28T16:01:40.653153+0000","last_clean":"2024-03-28T16:01:40.653153+0000","last_became_active":"2024-03-28T16:01:07.247109+0000","last_became_peered":"2024-03-28T16:01:07.247109+0000","last_unstale":"2024-03-28T16:01:40.653153+0000","last_undegraded":"2024-03-28T16:01:40.653153+0000","last_fullsized":"2024-03-28T16:01:40.653153+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:14:19.387896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"92'5","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.037067+0000","last_change":"2024-03-28T16:01:08.264565+0000","last_active":"2024-03-28T16:02:39.037067+0000","last_peered":"2024-03-28T16:02:39.037067+0000","last_clean":"2024-03-28T16:02:39.037067+0000","last_became_active":"2024-03-28T16:01:08.264196+0000","last_became_peered":"2024-03-28T16:01:08.264196+0000","last_unstale":"2024-03-28T16:02:39.037067+0000","last_undegraded":"2024-03-28T16:02:39.037067+0000","last_fullsized":"2024-03-28T16:02:39.037067+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:43:24.750624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.666071+0000","last_change":"2024-03-28T16:01:07.248189+0000","last_active":"2024-03-28T16:01:40.666071+0000","last_peered":"2024-03-28T16:01:40.666071+0000","last_clean":"2024-03-28T16:01:40.666071+0000","last_became_active":"2024-03-28T16:01:07.247295+0000","last_became_peered":"2024-03-28T16:01:07.247295+0000","last_unstale":"2024-03-28T16:01:40.666071+0000","last_undegraded":"2024-03-28T16:01:40.666071+0000","last_fullsized":"2024-03-28T16:01:40.666071+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:46:51.875094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"92'7","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.031820+0000","last_change":"2024-03-28T16:01:08.269724+0000","last_active":"2024-03-28T16:02:39.031820+0000","last_peered":"2024-03-28T16:02:39.031820+0000","last_clean":"2024-03-28T16:02:39.031820+0000","last_became_active":"2024-03-28T16:01:08.269357+0000","last_became_peered":"2024-03-28T16:01:08.269357+0000","last_unstale":"2024-03-28T16:02:39.031820+0000","last_undegraded":"2024-03-28T16:02:39.031820+0000","last_fullsized":"2024-03-28T16:02:39.031820+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:20:12.622238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"92'8","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.035129+0000","last_change":"2024-03-28T16:01:08.266624+0000","last_active":"2024-03-28T16:02:39.035129+0000","last_peered":"2024-03-28T16:02:39.035129+0000","last_clean":"2024-03-28T16:02:39.035129+0000","last_became_active":"2024-03-28T16:01:08.265408+0000","last_became_peered":"2024-03-28T16:01:08.265408+0000","last_unstale":"2024-03-28T16:02:39.035129+0000","last_undegraded":"2024-03-28T16:02:39.035129+0000","last_fullsized":"2024-03-28T16:02:39.035129+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:37:08.831727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.920935+0000","last_change":"2024-03-28T16:01:07.241172+0000","last_active":"2024-03-28T16:01:40.920935+0000","last_peered":"2024-03-28T16:01:40.920935+0000","last_clean":"2024-03-28T16:01:40.920935+0000","last_became_active":"2024-03-28T16:01:07.240754+0000","last_became_peered":"2024-03-28T16:01:07.240754+0000","last_unstale":"2024-03-28T16:01:40.920935+0000","last_undegraded":"2024-03-28T16:01:40.920935+0000","last_fullsized":"2024-03-28T16:01:40.920935+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:52:45.805188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"92'8","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.047153+0000","last_change":"2024-03-28T16:01:08.278429+0000","last_active":"2024-03-28T16:02:39.047153+0000","last_peered":"2024-03-28T16:02:39.047153+0000","last_clean":"2024-03-28T16:02:39.047153+0000","last_became_active":"2024-03-28T16:01:08.275061+0000","last_became_peered":"2024-03-28T16:01:08.275061+0000","last_unstale":"2024-03-28T16:02:39.047153+0000","last_undegraded":"2024-03-28T16:02:39.047153+0000","last_fullsized":"2024-03-28T16:02:39.047153+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:35:42.015979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.655996+0000","last_change":"2024-03-28T16:01:07.252117+0000","last_active":"2024-03-28T16:01:40.655996+0000","last_peered":"2024-03-28T16:01:40.655996+0000","last_clean":"2024-03-28T16:01:40.655996+0000","last_became_active":"2024-03-28T16:01:07.249729+0000","last_became_peered":"2024-03-28T16:01:07.249729+0000","last_unstale":"2024-03-28T16:01:40.655996+0000","last_undegraded":"2024-03-28T16:01:40.655996+0000","last_fullsized":"2024-03-28T16:01:40.655996+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:33:32.719310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":347,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.663525+0000","last_change":"2024-03-28T15:59:32.593730+0000","last_active":"2024-03-28T16:01:40.663525+0000","last_peered":"2024-03-28T16:01:40.663525+0000","last_clean":"2024-03-28T16:01:40.663525+0000","last_became_active":"2024-03-28T15:59:32.592289+0000","last_became_peered":"2024-03-28T15:59:32.592289+0000","last_unstale":"2024-03-28T16:01:40.663525+0000","last_undegraded":"2024-03-28T16:01:40.663525+0000","last_fullsized":"2024-03-28T16:01:40.663525+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:54:55.638193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:54:55.638193+0000","last_clean_scrub_stamp":"2024-03-28T15:54:55.638193+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:20:37.087340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"92'5","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.025916+0000","last_change":"2024-03-28T16:01:08.265513+0000","last_active":"2024-03-28T16:02:39.025916+0000","last_peered":"2024-03-28T16:02:39.025916+0000","last_clean":"2024-03-28T16:02:39.025916+0000","last_became_active":"2024-03-28T16:01:08.264711+0000","last_became_peered":"2024-03-28T16:01:08.264711+0000","last_unstale":"2024-03-28T16:02:39.025916+0000","last_undegraded":"2024-03-28T16:02:39.025916+0000","last_fullsized":"2024-03-28T16:02:39.025916+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:10:01.936780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.665127+0000","last_change":"2024-03-28T16:01:07.252831+0000","last_active":"2024-03-28T16:01:40.665127+0000","last_peered":"2024-03-28T16:01:40.665127+0000","last_clean":"2024-03-28T16:01:40.665127+0000","last_became_active":"2024-03-28T16:01:07.252317+0000","last_became_peered":"2024-03-28T16:01:07.252317+0000","last_unstale":"2024-03-28T16:01:40.665127+0000","last_undegraded":"2024-03-28T16:01:40.665127+0000","last_fullsized":"2024-03-28T16:01:40.665127+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:55:37.461154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.655837+0000","last_change":"2024-03-28T16:01:07.262413+0000","last_active":"2024-03-28T16:01:40.655837+0000","last_peered":"2024-03-28T16:01:40.655837+0000","last_clean":"2024-03-28T16:01:40.655837+0000","last_became_active":"2024-03-28T16:01:07.261875+0000","last_became_peered":"2024-03-28T16:01:07.261875+0000","last_unstale":"2024-03-28T16:01:40.655837+0000","last_undegraded":"2024-03-28T16:01:40.655837+0000","last_fullsized":"2024-03-28T16:01:40.655837+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:25:49.124991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"92'9","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.042204+0000","last_change":"2024-03-28T16:01:08.259506+0000","last_active":"2024-03-28T16:02:39.042204+0000","last_peered":"2024-03-28T16:02:39.042204+0000","last_clean":"2024-03-28T16:02:39.042204+0000","last_became_active":"2024-03-28T16:01:08.258512+0000","last_became_peered":"2024-03-28T16:01:08.258512+0000","last_unstale":"2024-03-28T16:02:39.042204+0000","last_undegraded":"2024-03-28T16:02:39.042204+0000","last_fullsized":"2024-03-28T16:02:39.042204+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:20:17.017763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"92'10","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:03:17.778725+0000","last_change":"2024-03-28T16:01:07.254850+0000","last_active":"2024-03-28T16:03:17.778725+0000","last_peered":"2024-03-28T16:03:17.778725+0000","last_clean":"2024-03-28T16:03:17.778725+0000","last_became_active":"2024-03-28T16:01:07.252294+0000","last_became_peered":"2024-03-28T16:01:07.252294+0000","last_unstale":"2024-03-28T16:03:17.778725+0000","last_undegraded":"2024-03-28T16:03:17.778725+0000","last_fullsized":"2024-03-28T16:03:17.778725+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:54:26.700118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"92'5","reported_seq":16,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.039125+0000","last_change":"2024-03-28T16:01:40.675232+0000","last_active":"2024-03-28T16:02:39.039125+0000","last_peered":"2024-03-28T16:02:39.039125+0000","last_clean":"2024-03-28T16:02:39.039125+0000","last_became_active":"2024-03-28T16:01:40.674454+0000","last_became_peered":"2024-03-28T16:01:40.674454+0000","last_unstale":"2024-03-28T16:02:39.039125+0000","last_undegraded":"2024-03-28T16:02:39.039125+0000","last_fullsized":"2024-03-28T16:02:39.039125+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:47:16.997641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,7],"acting":[9,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.7","version":"89'2","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.657272+0000","last_change":"2024-03-28T16:01:07.245292+0000","last_active":"2024-03-28T16:01:40.657272+0000","last_peered":"2024-03-28T16:01:40.657272+0000","last_clean":"2024-03-28T16:01:40.657272+0000","last_became_active":"2024-03-28T16:01:07.244654+0000","last_became_peered":"2024-03-28T16:01:07.244654+0000","last_unstale":"2024-03-28T16:01:40.657272+0000","last_undegraded":"2024-03-28T16:01:40.657272+0000","last_fullsized":"2024-03-28T16:01:40.657272+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:03:39.027844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"92'7","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.049641+0000","last_change":"2024-03-28T16:01:08.274542+0000","last_active":"2024-03-28T16:02:39.049641+0000","last_peered":"2024-03-28T16:02:39.049641+0000","last_clean":"2024-03-28T16:02:39.049641+0000","last_became_active":"2024-03-28T16:01:08.274202+0000","last_became_peered":"2024-03-28T16:01:08.274202+0000","last_unstale":"2024-03-28T16:02:39.049641+0000","last_undegraded":"2024-03-28T16:02:39.049641+0000","last_fullsized":"2024-03-28T16:02:39.049641+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:09:18.199017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.838038+0000","last_change":"2024-03-28T16:01:07.253274+0000","last_active":"2024-03-28T16:01:40.838038+0000","last_peered":"2024-03-28T16:01:40.838038+0000","last_clean":"2024-03-28T16:01:40.838038+0000","last_became_active":"2024-03-28T16:01:07.250230+0000","last_became_peered":"2024-03-28T16:01:07.250230+0000","last_unstale":"2024-03-28T16:01:40.838038+0000","last_undegraded":"2024-03-28T16:01:40.838038+0000","last_fullsized":"2024-03-28T16:01:40.838038+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:20:31.459983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"92'2","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.026627+0000","last_change":"2024-03-28T16:01:08.266551+0000","last_active":"2024-03-28T16:02:39.026627+0000","last_peered":"2024-03-28T16:02:39.026627+0000","last_clean":"2024-03-28T16:02:39.026627+0000","last_became_active":"2024-03-28T16:01:08.266226+0000","last_became_peered":"2024-03-28T16:01:08.266226+0000","last_unstale":"2024-03-28T16:02:39.026627+0000","last_undegraded":"2024-03-28T16:02:39.026627+0000","last_fullsized":"2024-03-28T16:02:39.026627+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:51:16.520285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.764663+0000","last_change":"2024-03-28T16:01:07.252038+0000","last_active":"2024-03-28T16:01:40.764663+0000","last_peered":"2024-03-28T16:01:40.764663+0000","last_clean":"2024-03-28T16:01:40.764663+0000","last_became_active":"2024-03-28T16:01:07.251703+0000","last_became_peered":"2024-03-28T16:01:07.251703+0000","last_unstale":"2024-03-28T16:01:40.764663+0000","last_undegraded":"2024-03-28T16:01:40.764663+0000","last_fullsized":"2024-03-28T16:01:40.764663+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:35:42.518996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"92'9","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.048402+0000","last_change":"2024-03-28T16:01:08.260025+0000","last_active":"2024-03-28T16:02:39.048402+0000","last_peered":"2024-03-28T16:02:39.048402+0000","last_clean":"2024-03-28T16:02:39.048402+0000","last_became_active":"2024-03-28T16:01:08.259596+0000","last_became_peered":"2024-03-28T16:01:08.259596+0000","last_unstale":"2024-03-28T16:02:39.048402+0000","last_undegraded":"2024-03-28T16:02:39.048402+0000","last_fullsized":"2024-03-28T16:02:39.048402+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:45:29.228188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.668132+0000","last_change":"2024-03-28T16:01:07.259335+0000","last_active":"2024-03-28T16:01:40.668132+0000","last_peered":"2024-03-28T16:01:40.668132+0000","last_clean":"2024-03-28T16:01:40.668132+0000","last_became_active":"2024-03-28T16:01:07.258589+0000","last_became_peered":"2024-03-28T16:01:07.258589+0000","last_unstale":"2024-03-28T16:01:40.668132+0000","last_undegraded":"2024-03-28T16:01:40.668132+0000","last_fullsized":"2024-03-28T16:01:40.668132+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:06:41.742963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"92'8","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.048367+0000","last_change":"2024-03-28T16:01:08.266794+0000","last_active":"2024-03-28T16:02:39.048367+0000","last_peered":"2024-03-28T16:02:39.048367+0000","last_clean":"2024-03-28T16:02:39.048367+0000","last_became_active":"2024-03-28T16:01:08.266502+0000","last_became_peered":"2024-03-28T16:01:08.266502+0000","last_unstale":"2024-03-28T16:02:39.048367+0000","last_undegraded":"2024-03-28T16:02:39.048367+0000","last_fullsized":"2024-03-28T16:02:39.048367+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:17:12.671677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"89'2","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.655656+0000","last_change":"2024-03-28T16:01:07.259239+0000","last_active":"2024-03-28T16:01:40.655656+0000","last_peered":"2024-03-28T16:01:40.655656+0000","last_clean":"2024-03-28T16:01:40.655656+0000","last_became_active":"2024-03-28T16:01:07.258875+0000","last_became_peered":"2024-03-28T16:01:07.258875+0000","last_unstale":"2024-03-28T16:01:40.655656+0000","last_undegraded":"2024-03-28T16:01:40.655656+0000","last_fullsized":"2024-03-28T16:01:40.655656+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:35:12.729173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"92'5","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.052410+0000","last_change":"2024-03-28T16:01:08.278196+0000","last_active":"2024-03-28T16:02:39.052410+0000","last_peered":"2024-03-28T16:02:39.052410+0000","last_clean":"2024-03-28T16:02:39.052410+0000","last_became_active":"2024-03-28T16:01:08.276972+0000","last_became_peered":"2024-03-28T16:01:08.276972+0000","last_unstale":"2024-03-28T16:02:39.052410+0000","last_undegraded":"2024-03-28T16:02:39.052410+0000","last_fullsized":"2024-03-28T16:02:39.052410+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:28:25.231210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.668117+0000","last_change":"2024-03-28T16:01:07.246851+0000","last_active":"2024-03-28T16:01:40.668117+0000","last_peered":"2024-03-28T16:01:40.668117+0000","last_clean":"2024-03-28T16:01:40.668117+0000","last_became_active":"2024-03-28T16:01:07.246365+0000","last_became_peered":"2024-03-28T16:01:07.246365+0000","last_unstale":"2024-03-28T16:01:40.668117+0000","last_undegraded":"2024-03-28T16:01:40.668117+0000","last_fullsized":"2024-03-28T16:01:40.668117+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:16:13.593045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"92'7","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.053165+0000","last_change":"2024-03-28T16:01:08.279468+0000","last_active":"2024-03-28T16:02:39.053165+0000","last_peered":"2024-03-28T16:02:39.053165+0000","last_clean":"2024-03-28T16:02:39.053165+0000","last_became_active":"2024-03-28T16:01:08.274650+0000","last_became_peered":"2024-03-28T16:01:08.274650+0000","last_unstale":"2024-03-28T16:02:39.053165+0000","last_undegraded":"2024-03-28T16:02:39.053165+0000","last_fullsized":"2024-03-28T16:02:39.053165+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:36:32.776085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"89'1","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.665556+0000","last_change":"2024-03-28T16:01:07.258077+0000","last_active":"2024-03-28T16:01:40.665556+0000","last_peered":"2024-03-28T16:01:40.665556+0000","last_clean":"2024-03-28T16:01:40.665556+0000","last_became_active":"2024-03-28T16:01:07.256665+0000","last_became_peered":"2024-03-28T16:01:07.256665+0000","last_unstale":"2024-03-28T16:01:40.665556+0000","last_undegraded":"2024-03-28T16:01:40.665556+0000","last_fullsized":"2024-03-28T16:01:40.665556+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:01:18.852441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"92'7","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.040959+0000","last_change":"2024-03-28T16:01:08.278588+0000","last_active":"2024-03-28T16:02:39.040959+0000","last_peered":"2024-03-28T16:02:39.040959+0000","last_clean":"2024-03-28T16:02:39.040959+0000","last_became_active":"2024-03-28T16:01:08.277303+0000","last_became_peered":"2024-03-28T16:01:08.277303+0000","last_unstale":"2024-03-28T16:02:39.040959+0000","last_undegraded":"2024-03-28T16:02:39.040959+0000","last_fullsized":"2024-03-28T16:02:39.040959+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:01:19.927057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"89'2","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.921309+0000","last_change":"2024-03-28T16:01:07.249382+0000","last_active":"2024-03-28T16:01:40.921309+0000","last_peered":"2024-03-28T16:01:40.921309+0000","last_clean":"2024-03-28T16:01:40.921309+0000","last_became_active":"2024-03-28T16:01:07.249065+0000","last_became_peered":"2024-03-28T16:01:07.249065+0000","last_unstale":"2024-03-28T16:01:40.921309+0000","last_undegraded":"2024-03-28T16:01:40.921309+0000","last_fullsized":"2024-03-28T16:01:40.921309+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:07:22.851862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"92'9","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.038073+0000","last_change":"2024-03-28T16:01:08.274839+0000","last_active":"2024-03-28T16:02:39.038073+0000","last_peered":"2024-03-28T16:02:39.038073+0000","last_clean":"2024-03-28T16:02:39.038073+0000","last_became_active":"2024-03-28T16:01:08.274505+0000","last_became_peered":"2024-03-28T16:01:08.274505+0000","last_unstale":"2024-03-28T16:02:39.038073+0000","last_undegraded":"2024-03-28T16:02:39.038073+0000","last_fullsized":"2024-03-28T16:02:39.038073+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:57:37.635327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"92'9","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.037953+0000","last_change":"2024-03-28T16:01:40.676281+0000","last_active":"2024-03-28T16:02:39.037953+0000","last_peered":"2024-03-28T16:02:39.037953+0000","last_clean":"2024-03-28T16:02:39.037953+0000","last_became_active":"2024-03-28T16:01:40.675734+0000","last_became_peered":"2024-03-28T16:01:40.675734+0000","last_unstale":"2024-03-28T16:02:39.037953+0000","last_undegraded":"2024-03-28T16:02:39.037953+0000","last_fullsized":"2024-03-28T16:02:39.037953+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:23:39.282010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"92'12","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.030636+0000","last_change":"2024-03-28T16:01:08.272816+0000","last_active":"2024-03-28T16:02:39.030636+0000","last_peered":"2024-03-28T16:02:39.030636+0000","last_clean":"2024-03-28T16:02:39.030636+0000","last_became_active":"2024-03-28T16:01:08.270941+0000","last_became_peered":"2024-03-28T16:01:08.270941+0000","last_unstale":"2024-03-28T16:02:39.030636+0000","last_undegraded":"2024-03-28T16:02:39.030636+0000","last_fullsized":"2024-03-28T16:02:39.030636+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:19:00.481809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"92'8","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.047233+0000","last_change":"2024-03-28T16:01:08.258558+0000","last_active":"2024-03-28T16:02:39.047233+0000","last_peered":"2024-03-28T16:02:39.047233+0000","last_clean":"2024-03-28T16:02:39.047233+0000","last_became_active":"2024-03-28T16:01:08.258163+0000","last_became_peered":"2024-03-28T16:01:08.258163+0000","last_unstale":"2024-03-28T16:02:39.047233+0000","last_undegraded":"2024-03-28T16:02:39.047233+0000","last_fullsized":"2024-03-28T16:02:39.047233+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:53:48.358536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"92'7","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.048041+0000","last_change":"2024-03-28T16:01:08.268886+0000","last_active":"2024-03-28T16:02:39.048041+0000","last_peered":"2024-03-28T16:02:39.048041+0000","last_clean":"2024-03-28T16:02:39.048041+0000","last_became_active":"2024-03-28T16:01:08.268519+0000","last_became_peered":"2024-03-28T16:01:08.268519+0000","last_unstale":"2024-03-28T16:02:39.048041+0000","last_undegraded":"2024-03-28T16:02:39.048041+0000","last_fullsized":"2024-03-28T16:02:39.048041+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:49:05.266517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"92'4","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.049066+0000","last_change":"2024-03-28T16:01:08.265039+0000","last_active":"2024-03-28T16:02:39.049066+0000","last_peered":"2024-03-28T16:02:39.049066+0000","last_clean":"2024-03-28T16:02:39.049066+0000","last_became_active":"2024-03-28T16:01:08.264677+0000","last_became_peered":"2024-03-28T16:01:08.264677+0000","last_unstale":"2024-03-28T16:02:39.049066+0000","last_undegraded":"2024-03-28T16:02:39.049066+0000","last_fullsized":"2024-03-28T16:02:39.049066+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:17:27.667324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"92'6","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.035821+0000","last_change":"2024-03-28T16:01:08.271699+0000","last_active":"2024-03-28T16:02:39.035821+0000","last_peered":"2024-03-28T16:02:39.035821+0000","last_clean":"2024-03-28T16:02:39.035821+0000","last_became_active":"2024-03-28T16:01:08.270184+0000","last_became_peered":"2024-03-28T16:01:08.270184+0000","last_unstale":"2024-03-28T16:02:39.035821+0000","last_undegraded":"2024-03-28T16:02:39.035821+0000","last_fullsized":"2024-03-28T16:02:39.035821+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:17:15.699944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"92'15","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.038913+0000","last_change":"2024-03-28T16:01:08.262314+0000","last_active":"2024-03-28T16:02:39.038913+0000","last_peered":"2024-03-28T16:02:39.038913+0000","last_clean":"2024-03-28T16:02:39.038913+0000","last_became_active":"2024-03-28T16:01:08.261992+0000","last_became_peered":"2024-03-28T16:01:08.261992+0000","last_unstale":"2024-03-28T16:02:39.038913+0000","last_undegraded":"2024-03-28T16:02:39.038913+0000","last_fullsized":"2024-03-28T16:02:39.038913+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:40:52.048306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"92'2","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.027024+0000","last_change":"2024-03-28T16:01:08.259407+0000","last_active":"2024-03-28T16:02:39.027024+0000","last_peered":"2024-03-28T16:02:39.027024+0000","last_clean":"2024-03-28T16:02:39.027024+0000","last_became_active":"2024-03-28T16:01:08.259071+0000","last_became_peered":"2024-03-28T16:01:08.259071+0000","last_unstale":"2024-03-28T16:02:39.027024+0000","last_undegraded":"2024-03-28T16:02:39.027024+0000","last_fullsized":"2024-03-28T16:02:39.027024+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:30:17.819868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"92'6","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.039238+0000","last_change":"2024-03-28T16:01:08.271649+0000","last_active":"2024-03-28T16:02:39.039238+0000","last_peered":"2024-03-28T16:02:39.039238+0000","last_clean":"2024-03-28T16:02:39.039238+0000","last_became_active":"2024-03-28T16:01:08.271230+0000","last_became_peered":"2024-03-28T16:01:08.271230+0000","last_unstale":"2024-03-28T16:02:39.039238+0000","last_undegraded":"2024-03-28T16:02:39.039238+0000","last_fullsized":"2024-03-28T16:02:39.039238+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:50:14.404178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"92'8","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.031938+0000","last_change":"2024-03-28T16:01:08.260845+0000","last_active":"2024-03-28T16:02:39.031938+0000","last_peered":"2024-03-28T16:02:39.031938+0000","last_clean":"2024-03-28T16:02:39.031938+0000","last_became_active":"2024-03-28T16:01:08.259997+0000","last_became_peered":"2024-03-28T16:01:08.259997+0000","last_unstale":"2024-03-28T16:02:39.031938+0000","last_undegraded":"2024-03-28T16:02:39.031938+0000","last_fullsized":"2024-03-28T16:02:39.031938+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:08:55.080858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"92'10","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.050870+0000","last_change":"2024-03-28T16:01:08.268881+0000","last_active":"2024-03-28T16:02:39.050870+0000","last_peered":"2024-03-28T16:02:39.050870+0000","last_clean":"2024-03-28T16:02:39.050870+0000","last_became_active":"2024-03-28T16:01:08.268212+0000","last_became_peered":"2024-03-28T16:01:08.268212+0000","last_unstale":"2024-03-28T16:02:39.050870+0000","last_undegraded":"2024-03-28T16:02:39.050870+0000","last_fullsized":"2024-03-28T16:02:39.050870+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:46:22.447763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"92'11","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.051690+0000","last_change":"2024-03-28T16:01:08.265113+0000","last_active":"2024-03-28T16:02:39.051690+0000","last_peered":"2024-03-28T16:02:39.051690+0000","last_clean":"2024-03-28T16:02:39.051690+0000","last_became_active":"2024-03-28T16:01:08.264781+0000","last_became_peered":"2024-03-28T16:01:08.264781+0000","last_unstale":"2024-03-28T16:02:39.051690+0000","last_undegraded":"2024-03-28T16:02:39.051690+0000","last_fullsized":"2024-03-28T16:02:39.051690+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:19:14.720260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"92'8","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.030628+0000","last_change":"2024-03-28T16:01:08.274446+0000","last_active":"2024-03-28T16:02:39.030628+0000","last_peered":"2024-03-28T16:02:39.030628+0000","last_clean":"2024-03-28T16:02:39.030628+0000","last_became_active":"2024-03-28T16:01:08.274027+0000","last_became_peered":"2024-03-28T16:01:08.274027+0000","last_unstale":"2024-03-28T16:02:39.030628+0000","last_undegraded":"2024-03-28T16:02:39.030628+0000","last_fullsized":"2024-03-28T16:02:39.030628+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:33:37.395889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":63436,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":281,"num_read_kb":204,"num_write":88,"num_write_kb":91,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":190308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":85,"seq":365072220204,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":478849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":77,"seq":330712481843,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32812,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712340,"statfs":{"total":95995035648,"available":95961436160,"internally_reserved":0,"allocated":1675264,"data_stored":1132144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":71,"seq":304942678074,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":479409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":63,"seq":270582939713,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":479409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":55,"seq":236223201352,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36892,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708260,"statfs":{"total":95995035648,"available":95957258240,"internally_reserved":0,"allocated":1593344,"data_stored":1069894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397583,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36308,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708844,"statfs":{"total":95995035648,"available":95957856256,"internally_reserved":0,"allocated":995328,"data_stored":478944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574960,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28056,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717096,"statfs":{"total":95995035648,"available":95966306304,"internally_reserved":0,"allocated":1064960,"data_stored":541465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738485,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36308,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708844,"statfs":{"total":95995035648,"available":95957856256,"internally_reserved":0,"allocated":995328,"data_stored":478917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411434,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1069056,"data_stored":541499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117090,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36320,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708832,"statfs":{"total":95995035648,"available":95957843968,"internally_reserved":0,"allocated":1007616,"data_stored":479165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855452,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":478849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659222,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36900,"kb_used_data":1564,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708252,"statfs":{"total":95995035648,"available":95957250048,"internally_reserved":0,"allocated":1601536,"data_stored":1069494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":62638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":62672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":62832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T16:03:30.020 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-03-28T16:03:30.021 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T16:03:30.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:29 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/3397472828' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:03:30.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:29 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/3397472828' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:03:30.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.283+0000 7fce7d600640 1 -- 172.21.15.67:0/3410695609 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce78074100 msgr2=0x7fce78074580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:30.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.283+0000 7fce7d600640 1 --2- 172.21.15.67:0/3410695609 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce78074100 0x7fce78074580 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fce60009930 tx=0x7fce6002bc30 comp rx=0 tx=0).stop 2024-03-28T16:03:30.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.284+0000 7fce7d600640 1 -- 172.21.15.67:0/3410695609 shutdown_connections 2024-03-28T16:03:30.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.284+0000 7fce7d600640 1 --2- 172.21.15.67:0/3410695609 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fce7810dab0 0x7fce7810fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:30.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.284+0000 7fce7d600640 1 --2- 172.21.15.67:0/3410695609 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce78074100 0x7fce78074580 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:30.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.284+0000 7fce7d600640 1 --2- 172.21.15.67:0/3410695609 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce780756e0 0x7fce78073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:30.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.284+0000 7fce7d600640 1 -- 172.21.15.67:0/3410695609 >> 172.21.15.67:0/3410695609 conn(0x7fce780fca30 msgr2=0x7fce780fee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:30.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.284+0000 7fce7d600640 1 -- 172.21.15.67:0/3410695609 shutdown_connections 2024-03-28T16:03:30.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.284+0000 7fce7d600640 1 -- 172.21.15.67:0/3410695609 wait complete. 2024-03-28T16:03:30.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.285+0000 7fce7d600640 1 Processor -- start 2024-03-28T16:03:30.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.285+0000 7fce7d600640 1 -- start start 2024-03-28T16:03:30.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.285+0000 7fce7d600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fce78074100 0x7fce7819dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:30.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.285+0000 7fce7d600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce780756e0 0x7fce7819e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:30.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.285+0000 7fce7d600640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce7810dab0 0x7fce781a5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:30.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.285+0000 7fce7d600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce78113650 con 0x7fce780756e0 2024-03-28T16:03:30.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.285+0000 7fce7d600640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce781134d0 con 0x7fce7810dab0 2024-03-28T16:03:30.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.285+0000 7fce7d600640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce781137d0 con 0x7fce78074100 2024-03-28T16:03:30.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.285+0000 7fce6f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce780756e0 0x7fce7819e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:30.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.285+0000 7fce6f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce780756e0 0x7fce7819e2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50620/0 (socket says 172.21.15.67:50620) 2024-03-28T16:03:30.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.285+0000 7fce6f400640 1 -- 172.21.15.67:0/2702377993 learned_addr learned my addr 172.21.15.67:0/2702377993 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:03:30.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.286+0000 7fce7cc00640 1 --2- 172.21.15.67:0/2702377993 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce7810dab0 0x7fce781a5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:30.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.286+0000 7fce77e00640 1 --2- 172.21.15.67:0/2702377993 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fce78074100 0x7fce7819dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:30.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.286+0000 7fce6f400640 1 -- 172.21.15.67:0/2702377993 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fce78074100 msgr2=0x7fce7819dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:30.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.286+0000 7fce6f400640 1 --2- 172.21.15.67:0/2702377993 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fce78074100 0x7fce7819dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:30.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.286+0000 7fce6f400640 1 -- 172.21.15.67:0/2702377993 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce7810dab0 msgr2=0x7fce781a5310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:30.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.286+0000 7fce6f400640 1 --2- 172.21.15.67:0/2702377993 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce7810dab0 0x7fce781a5310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:30.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.286+0000 7fce6f400640 1 -- 172.21.15.67:0/2702377993 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce781a59b0 con 0x7fce780756e0 2024-03-28T16:03:30.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.286+0000 7fce6f400640 1 --2- 172.21.15.67:0/2702377993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce780756e0 0x7fce7819e2e0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fce60005ab0 tx=0x7fce600042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:30.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.287+0000 7fce75600640 1 -- 172.21.15.67:0/2702377993 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce60043020 con 0x7fce780756e0 2024-03-28T16:03:30.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.287+0000 7fce75600640 1 -- 172.21.15.67:0/2702377993 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fce600054c0 con 0x7fce780756e0 2024-03-28T16:03:30.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.287+0000 7fce75600640 1 -- 172.21.15.67:0/2702377993 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce60003d10 con 0x7fce780756e0 2024-03-28T16:03:30.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.287+0000 7fce7d600640 1 -- 172.21.15.67:0/2702377993 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce781a5c40 con 0x7fce780756e0 2024-03-28T16:03:30.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.287+0000 7fce7d600640 1 -- 172.21.15.67:0/2702377993 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce781a6140 con 0x7fce780756e0 2024-03-28T16:03:30.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.288+0000 7fce7d600640 1 -- 172.21.15.67:0/2702377993 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce78103c80 con 0x7fce780756e0 2024-03-28T16:03:30.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.290+0000 7fce75600640 1 -- 172.21.15.67:0/2702377993 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fce60046020 con 0x7fce780756e0 2024-03-28T16:03:30.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.291+0000 7fce75600640 1 --2- 172.21.15.67:0/2702377993 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fce5c0a2800 0x7fce5c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:30.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.292+0000 7fce75600640 1 -- 172.21.15.67:0/2702377993 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fce600f0f70 con 0x7fce780756e0 2024-03-28T16:03:30.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.292+0000 7fce77e00640 1 --2- 172.21.15.67:0/2702377993 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fce5c0a2800 0x7fce5c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:30.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.293+0000 7fce77e00640 1 --2- 172.21.15.67:0/2702377993 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fce5c0a2800 0x7fce5c0a4cc0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fce68005fd0 tx=0x7fce68005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:30.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.296+0000 7fce75600640 1 -- 172.21.15.67:0/2702377993 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fce6004d020 con 0x7fce780756e0 2024-03-28T16:03:30.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:29 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/3397472828' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:03:30.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.497+0000 7fce7d600640 1 -- 172.21.15.67:0/2702377993 --> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fce78061ae0 con 0x7fce5c0a2800 2024-03-28T16:03:30.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.519+0000 7fce75600640 1 -- 172.21.15.67:0/2702377993 <== mgr.14152 v2:172.21.15.67:6800/208726654 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+258821 (secure 0 0 0) 0x7fce78061ae0 con 0x7fce5c0a2800 2024-03-28T16:03:30.523 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:03:30.524 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-28T16:03:30.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.525+0000 7fce7d600640 1 -- 172.21.15.67:0/2702377993 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fce5c0a2800 msgr2=0x7fce5c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:30.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.525+0000 7fce7d600640 1 --2- 172.21.15.67:0/2702377993 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fce5c0a2800 0x7fce5c0a4cc0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fce68005fd0 tx=0x7fce68005850 comp rx=0 tx=0).stop 2024-03-28T16:03:30.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.525+0000 7fce7d600640 1 -- 172.21.15.67:0/2702377993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce780756e0 msgr2=0x7fce7819e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:30.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.525+0000 7fce7d600640 1 --2- 172.21.15.67:0/2702377993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce780756e0 0x7fce7819e2e0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fce60005ab0 tx=0x7fce600042f0 comp rx=0 tx=0).stop 2024-03-28T16:03:30.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.525+0000 7fce7d600640 1 -- 172.21.15.67:0/2702377993 shutdown_connections 2024-03-28T16:03:30.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.525+0000 7fce7d600640 1 --2- 172.21.15.67:0/2702377993 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fce5c0a2800 0x7fce5c0a4cc0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:30.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.525+0000 7fce7d600640 1 --2- 172.21.15.67:0/2702377993 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce7810dab0 0x7fce781a5310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:30.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.525+0000 7fce7d600640 1 --2- 172.21.15.67:0/2702377993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce780756e0 0x7fce7819e2e0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:30.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.525+0000 7fce7d600640 1 --2- 172.21.15.67:0/2702377993 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fce78074100 0x7fce7819dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:30.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.525+0000 7fce7d600640 1 -- 172.21.15.67:0/2702377993 >> 172.21.15.67:0/2702377993 conn(0x7fce780fca30 msgr2=0x7fce78100e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:30.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.526+0000 7fce7d600640 1 -- 172.21.15.67:0/2702377993 shutdown_connections 2024-03-28T16:03:30.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.526+0000 7fce7d600640 1 -- 172.21.15.67:0/2702377993 wait complete. 2024-03-28T16:03:30.547 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":437,"stamp":"2024-03-28T16:03:29.097067+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":654038,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":407,"num_read_kb":313,"num_write":271,"num_write_kb":3031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1180758,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":712,"ondisk_log_size":712,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":383608,"kb_used_data":13656,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124558216,"statfs":{"total":1151940427776,"available":1151547613184,"internally_reserved":0,"allocated":13983744,"data_stored":7708038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":378713402},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":493,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":11,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004831"},"pg_stats":[{"pgid":"3.3f","version":"92'8","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.049941+0000","last_change":"2024-03-28T16:01:08.268196+0000","last_active":"2024-03-28T16:02:39.049941+0000","last_peered":"2024-03-28T16:02:39.049941+0000","last_clean":"2024-03-28T16:02:39.049941+0000","last_became_active":"2024-03-28T16:01:08.266956+0000","last_became_peered":"2024-03-28T16:01:08.266956+0000","last_unstale":"2024-03-28T16:02:39.049941+0000","last_undegraded":"2024-03-28T16:02:39.049941+0000","last_fullsized":"2024-03-28T16:02:39.049941+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:38:44.296129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"92'4","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.048446+0000","last_change":"2024-03-28T16:01:08.264985+0000","last_active":"2024-03-28T16:02:39.048446+0000","last_peered":"2024-03-28T16:02:39.048446+0000","last_clean":"2024-03-28T16:02:39.048446+0000","last_became_active":"2024-03-28T16:01:08.262484+0000","last_became_peered":"2024-03-28T16:01:08.262484+0000","last_unstale":"2024-03-28T16:02:39.048446+0000","last_undegraded":"2024-03-28T16:02:39.048446+0000","last_fullsized":"2024-03-28T16:02:39.048446+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:33:07.538908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"92'7","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.051488+0000","last_change":"2024-03-28T16:01:08.267423+0000","last_active":"2024-03-28T16:02:39.051488+0000","last_peered":"2024-03-28T16:02:39.051488+0000","last_clean":"2024-03-28T16:02:39.051488+0000","last_became_active":"2024-03-28T16:01:08.263665+0000","last_became_peered":"2024-03-28T16:01:08.263665+0000","last_unstale":"2024-03-28T16:02:39.051488+0000","last_undegraded":"2024-03-28T16:02:39.051488+0000","last_fullsized":"2024-03-28T16:02:39.051488+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:09:43.488003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"92'8","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.028028+0000","last_change":"2024-03-28T16:01:08.274978+0000","last_active":"2024-03-28T16:02:39.028028+0000","last_peered":"2024-03-28T16:02:39.028028+0000","last_clean":"2024-03-28T16:02:39.028028+0000","last_became_active":"2024-03-28T16:01:08.274553+0000","last_became_peered":"2024-03-28T16:01:08.274553+0000","last_unstale":"2024-03-28T16:02:39.028028+0000","last_undegraded":"2024-03-28T16:02:39.028028+0000","last_fullsized":"2024-03-28T16:02:39.028028+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:16:03.997334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"92'4","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.042170+0000","last_change":"2024-03-28T16:01:08.262155+0000","last_active":"2024-03-28T16:02:39.042170+0000","last_peered":"2024-03-28T16:02:39.042170+0000","last_clean":"2024-03-28T16:02:39.042170+0000","last_became_active":"2024-03-28T16:01:08.261814+0000","last_became_peered":"2024-03-28T16:01:08.261814+0000","last_unstale":"2024-03-28T16:02:39.042170+0000","last_undegraded":"2024-03-28T16:02:39.042170+0000","last_fullsized":"2024-03-28T16:02:39.042170+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:41:04.242203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"92'6","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.023511+0000","last_change":"2024-03-28T16:01:08.273342+0000","last_active":"2024-03-28T16:02:39.023511+0000","last_peered":"2024-03-28T16:02:39.023511+0000","last_clean":"2024-03-28T16:02:39.023511+0000","last_became_active":"2024-03-28T16:01:08.272606+0000","last_became_peered":"2024-03-28T16:01:08.272606+0000","last_unstale":"2024-03-28T16:02:39.023511+0000","last_undegraded":"2024-03-28T16:02:39.023511+0000","last_fullsized":"2024-03-28T16:02:39.023511+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:44:48.580214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"92'5","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.040848+0000","last_change":"2024-03-28T16:01:08.278060+0000","last_active":"2024-03-28T16:02:39.040848+0000","last_peered":"2024-03-28T16:02:39.040848+0000","last_clean":"2024-03-28T16:02:39.040848+0000","last_became_active":"2024-03-28T16:01:08.277047+0000","last_became_peered":"2024-03-28T16:01:08.277047+0000","last_unstale":"2024-03-28T16:02:39.040848+0000","last_undegraded":"2024-03-28T16:02:39.040848+0000","last_fullsized":"2024-03-28T16:02:39.040848+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:46:30.005708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"92'9","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.036559+0000","last_change":"2024-03-28T16:01:08.273199+0000","last_active":"2024-03-28T16:02:39.036559+0000","last_peered":"2024-03-28T16:02:39.036559+0000","last_clean":"2024-03-28T16:02:39.036559+0000","last_became_active":"2024-03-28T16:01:08.272822+0000","last_became_peered":"2024-03-28T16:01:08.272822+0000","last_unstale":"2024-03-28T16:02:39.036559+0000","last_undegraded":"2024-03-28T16:02:39.036559+0000","last_fullsized":"2024-03-28T16:02:39.036559+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:28:11.495665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"92'11","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.050965+0000","last_change":"2024-03-28T16:01:08.270262+0000","last_active":"2024-03-28T16:02:39.050965+0000","last_peered":"2024-03-28T16:02:39.050965+0000","last_clean":"2024-03-28T16:02:39.050965+0000","last_became_active":"2024-03-28T16:01:08.269959+0000","last_became_peered":"2024-03-28T16:01:08.269959+0000","last_unstale":"2024-03-28T16:02:39.050965+0000","last_undegraded":"2024-03-28T16:02:39.050965+0000","last_fullsized":"2024-03-28T16:02:39.050965+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:49:02.261893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"92'16","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.028205+0000","last_change":"2024-03-28T16:01:40.671104+0000","last_active":"2024-03-28T16:02:39.028205+0000","last_peered":"2024-03-28T16:02:39.028205+0000","last_clean":"2024-03-28T16:02:39.028205+0000","last_became_active":"2024-03-28T16:01:40.670474+0000","last_became_peered":"2024-03-28T16:01:40.670474+0000","last_unstale":"2024-03-28T16:02:39.028205+0000","last_undegraded":"2024-03-28T16:02:39.028205+0000","last_fullsized":"2024-03-28T16:02:39.028205+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:19:41.651703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"92'6","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.052808+0000","last_change":"2024-03-28T16:01:08.274943+0000","last_active":"2024-03-28T16:02:39.052808+0000","last_peered":"2024-03-28T16:02:39.052808+0000","last_clean":"2024-03-28T16:02:39.052808+0000","last_became_active":"2024-03-28T16:01:08.274267+0000","last_became_peered":"2024-03-28T16:01:08.274267+0000","last_unstale":"2024-03-28T16:02:39.052808+0000","last_undegraded":"2024-03-28T16:02:39.052808+0000","last_fullsized":"2024-03-28T16:02:39.052808+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:41:13.979918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"92'7","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.020548+0000","last_change":"2024-03-28T16:01:08.266817+0000","last_active":"2024-03-28T16:02:39.020548+0000","last_peered":"2024-03-28T16:02:39.020548+0000","last_clean":"2024-03-28T16:02:39.020548+0000","last_became_active":"2024-03-28T16:01:08.266470+0000","last_became_peered":"2024-03-28T16:01:08.266470+0000","last_unstale":"2024-03-28T16:02:39.020548+0000","last_undegraded":"2024-03-28T16:02:39.020548+0000","last_fullsized":"2024-03-28T16:02:39.020548+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:15:28.573153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"92'10","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.052174+0000","last_change":"2024-03-28T16:01:08.269896+0000","last_active":"2024-03-28T16:02:39.052174+0000","last_peered":"2024-03-28T16:02:39.052174+0000","last_clean":"2024-03-28T16:02:39.052174+0000","last_became_active":"2024-03-28T16:01:08.269500+0000","last_became_peered":"2024-03-28T16:01:08.269500+0000","last_unstale":"2024-03-28T16:02:39.052174+0000","last_undegraded":"2024-03-28T16:02:39.052174+0000","last_fullsized":"2024-03-28T16:02:39.052174+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:44:38.867590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"92'7","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.047544+0000","last_change":"2024-03-28T16:01:08.267072+0000","last_active":"2024-03-28T16:02:39.047544+0000","last_peered":"2024-03-28T16:02:39.047544+0000","last_clean":"2024-03-28T16:02:39.047544+0000","last_became_active":"2024-03-28T16:01:08.266688+0000","last_became_peered":"2024-03-28T16:01:08.266688+0000","last_unstale":"2024-03-28T16:02:39.047544+0000","last_undegraded":"2024-03-28T16:02:39.047544+0000","last_fullsized":"2024-03-28T16:02:39.047544+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:07:57.702952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"92'7","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.045559+0000","last_change":"2024-03-28T16:01:08.270912+0000","last_active":"2024-03-28T16:02:39.045559+0000","last_peered":"2024-03-28T16:02:39.045559+0000","last_clean":"2024-03-28T16:02:39.045559+0000","last_became_active":"2024-03-28T16:01:08.270567+0000","last_became_peered":"2024-03-28T16:01:08.270567+0000","last_unstale":"2024-03-28T16:02:39.045559+0000","last_undegraded":"2024-03-28T16:02:39.045559+0000","last_fullsized":"2024-03-28T16:02:39.045559+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:19:29.294502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"92'9","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.050241+0000","last_change":"2024-03-28T16:01:08.266811+0000","last_active":"2024-03-28T16:02:39.050241+0000","last_peered":"2024-03-28T16:02:39.050241+0000","last_clean":"2024-03-28T16:02:39.050241+0000","last_became_active":"2024-03-28T16:01:08.266517+0000","last_became_peered":"2024-03-28T16:01:08.266517+0000","last_unstale":"2024-03-28T16:02:39.050241+0000","last_undegraded":"2024-03-28T16:02:39.050241+0000","last_fullsized":"2024-03-28T16:02:39.050241+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:55:05.972683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"92'8","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.050243+0000","last_change":"2024-03-28T16:01:08.263864+0000","last_active":"2024-03-28T16:02:39.050243+0000","last_peered":"2024-03-28T16:02:39.050243+0000","last_clean":"2024-03-28T16:02:39.050243+0000","last_became_active":"2024-03-28T16:01:08.262956+0000","last_became_peered":"2024-03-28T16:01:08.262956+0000","last_unstale":"2024-03-28T16:02:39.050243+0000","last_undegraded":"2024-03-28T16:02:39.050243+0000","last_fullsized":"2024-03-28T16:02:39.050243+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:05:50.504237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"92'10","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.045902+0000","last_change":"2024-03-28T16:01:08.265151+0000","last_active":"2024-03-28T16:02:39.045902+0000","last_peered":"2024-03-28T16:02:39.045902+0000","last_clean":"2024-03-28T16:02:39.045902+0000","last_became_active":"2024-03-28T16:01:08.264861+0000","last_became_peered":"2024-03-28T16:01:08.264861+0000","last_unstale":"2024-03-28T16:02:39.045902+0000","last_undegraded":"2024-03-28T16:02:39.045902+0000","last_fullsized":"2024-03-28T16:02:39.045902+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:35:45.878983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"92'7","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.037162+0000","last_change":"2024-03-28T16:01:08.265993+0000","last_active":"2024-03-28T16:02:39.037162+0000","last_peered":"2024-03-28T16:02:39.037162+0000","last_clean":"2024-03-28T16:02:39.037162+0000","last_became_active":"2024-03-28T16:01:08.265488+0000","last_became_peered":"2024-03-28T16:01:08.265488+0000","last_unstale":"2024-03-28T16:02:39.037162+0000","last_undegraded":"2024-03-28T16:02:39.037162+0000","last_fullsized":"2024-03-28T16:02:39.037162+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:31:55.920705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'4","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.654831+0000","last_change":"2024-03-28T16:01:07.250361+0000","last_active":"2024-03-28T16:01:40.654831+0000","last_peered":"2024-03-28T16:01:40.654831+0000","last_clean":"2024-03-28T16:01:40.654831+0000","last_became_active":"2024-03-28T16:01:07.250004+0000","last_became_peered":"2024-03-28T16:01:07.250004+0000","last_unstale":"2024-03-28T16:01:40.654831+0000","last_undegraded":"2024-03-28T16:01:40.654831+0000","last_fullsized":"2024-03-28T16:01:40.654831+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:10:32.838716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"92'5","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.032768+0000","last_change":"2024-03-28T16:01:08.274436+0000","last_active":"2024-03-28T16:02:39.032768+0000","last_peered":"2024-03-28T16:02:39.032768+0000","last_clean":"2024-03-28T16:02:39.032768+0000","last_became_active":"2024-03-28T16:01:08.274124+0000","last_became_peered":"2024-03-28T16:01:08.274124+0000","last_unstale":"2024-03-28T16:02:39.032768+0000","last_undegraded":"2024-03-28T16:02:39.032768+0000","last_fullsized":"2024-03-28T16:02:39.032768+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:02:34.037686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"89'1","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.765443+0000","last_change":"2024-03-28T16:01:07.248565+0000","last_active":"2024-03-28T16:01:40.765443+0000","last_peered":"2024-03-28T16:01:40.765443+0000","last_clean":"2024-03-28T16:01:40.765443+0000","last_became_active":"2024-03-28T16:01:07.247366+0000","last_became_peered":"2024-03-28T16:01:07.247366+0000","last_unstale":"2024-03-28T16:01:40.765443+0000","last_undegraded":"2024-03-28T16:01:40.765443+0000","last_fullsized":"2024-03-28T16:01:40.765443+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:48:13.146398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"92'6","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.051966+0000","last_change":"2024-03-28T16:01:08.265639+0000","last_active":"2024-03-28T16:02:39.051966+0000","last_peered":"2024-03-28T16:02:39.051966+0000","last_clean":"2024-03-28T16:02:39.051966+0000","last_became_active":"2024-03-28T16:01:08.263488+0000","last_became_peered":"2024-03-28T16:01:08.263488+0000","last_unstale":"2024-03-28T16:02:39.051966+0000","last_undegraded":"2024-03-28T16:02:39.051966+0000","last_fullsized":"2024-03-28T16:02:39.051966+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:15:06.516016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"89'2","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.850755+0000","last_change":"2024-03-28T16:01:40.850755+0000","last_active":"2024-03-28T16:01:40.850755+0000","last_peered":"2024-03-28T16:01:40.850755+0000","last_clean":"2024-03-28T16:01:40.850755+0000","last_became_active":"2024-03-28T16:01:40.842547+0000","last_became_peered":"2024-03-28T16:01:40.842547+0000","last_unstale":"2024-03-28T16:01:40.850755+0000","last_undegraded":"2024-03-28T16:01:40.850755+0000","last_fullsized":"2024-03-28T16:01:40.850755+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:39:27.862525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"92'9","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.051889+0000","last_change":"2024-03-28T16:01:08.271534+0000","last_active":"2024-03-28T16:02:39.051889+0000","last_peered":"2024-03-28T16:02:39.051889+0000","last_clean":"2024-03-28T16:02:39.051889+0000","last_became_active":"2024-03-28T16:01:08.271104+0000","last_became_peered":"2024-03-28T16:01:08.271104+0000","last_unstale":"2024-03-28T16:02:39.051889+0000","last_undegraded":"2024-03-28T16:02:39.051889+0000","last_fullsized":"2024-03-28T16:02:39.051889+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:29:21.332346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"89'2","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.655555+0000","last_change":"2024-03-28T16:01:07.259175+0000","last_active":"2024-03-28T16:01:40.655555+0000","last_peered":"2024-03-28T16:01:40.655555+0000","last_clean":"2024-03-28T16:01:40.655555+0000","last_became_active":"2024-03-28T16:01:07.258151+0000","last_became_peered":"2024-03-28T16:01:07.258151+0000","last_unstale":"2024-03-28T16:01:40.655555+0000","last_undegraded":"2024-03-28T16:01:40.655555+0000","last_fullsized":"2024-03-28T16:01:40.655555+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:41:07.064800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"92'15","reported_seq":38,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.046924+0000","last_change":"2024-03-28T16:01:40.843211+0000","last_active":"2024-03-28T16:02:39.046924+0000","last_peered":"2024-03-28T16:02:39.046924+0000","last_clean":"2024-03-28T16:02:39.046924+0000","last_became_active":"2024-03-28T16:01:40.842051+0000","last_became_peered":"2024-03-28T16:01:40.842051+0000","last_unstale":"2024-03-28T16:02:39.046924+0000","last_undegraded":"2024-03-28T16:02:39.046924+0000","last_fullsized":"2024-03-28T16:02:39.046924+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:19:26.437169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"89'1","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.664703+0000","last_change":"2024-03-28T16:01:07.248538+0000","last_active":"2024-03-28T16:01:40.664703+0000","last_peered":"2024-03-28T16:01:40.664703+0000","last_clean":"2024-03-28T16:01:40.664703+0000","last_became_active":"2024-03-28T16:01:07.248116+0000","last_became_peered":"2024-03-28T16:01:07.248116+0000","last_unstale":"2024-03-28T16:01:40.664703+0000","last_undegraded":"2024-03-28T16:01:40.664703+0000","last_fullsized":"2024-03-28T16:01:40.664703+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:44:56.071286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"92'7","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.033579+0000","last_change":"2024-03-28T16:01:08.271059+0000","last_active":"2024-03-28T16:02:39.033579+0000","last_peered":"2024-03-28T16:02:39.033579+0000","last_clean":"2024-03-28T16:02:39.033579+0000","last_became_active":"2024-03-28T16:01:08.270682+0000","last_became_peered":"2024-03-28T16:01:08.270682+0000","last_unstale":"2024-03-28T16:02:39.033579+0000","last_undegraded":"2024-03-28T16:02:39.033579+0000","last_fullsized":"2024-03-28T16:02:39.033579+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:54:48.922125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"89'1","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.661404+0000","last_change":"2024-03-28T16:01:07.261157+0000","last_active":"2024-03-28T16:01:40.661404+0000","last_peered":"2024-03-28T16:01:40.661404+0000","last_clean":"2024-03-28T16:01:40.661404+0000","last_became_active":"2024-03-28T16:01:07.260751+0000","last_became_peered":"2024-03-28T16:01:07.260751+0000","last_unstale":"2024-03-28T16:01:40.661404+0000","last_undegraded":"2024-03-28T16:01:40.661404+0000","last_fullsized":"2024-03-28T16:01:40.661404+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:16:49.622168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"92'8","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.038730+0000","last_change":"2024-03-28T16:01:08.265773+0000","last_active":"2024-03-28T16:02:39.038730+0000","last_peered":"2024-03-28T16:02:39.038730+0000","last_clean":"2024-03-28T16:02:39.038730+0000","last_became_active":"2024-03-28T16:01:08.263724+0000","last_became_peered":"2024-03-28T16:01:08.263724+0000","last_unstale":"2024-03-28T16:02:39.038730+0000","last_undegraded":"2024-03-28T16:02:39.038730+0000","last_fullsized":"2024-03-28T16:02:39.038730+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:42:37.338946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.921045+0000","last_change":"2024-03-28T16:01:07.235361+0000","last_active":"2024-03-28T16:01:40.921045+0000","last_peered":"2024-03-28T16:01:40.921045+0000","last_clean":"2024-03-28T16:01:40.921045+0000","last_became_active":"2024-03-28T16:01:07.234969+0000","last_became_peered":"2024-03-28T16:01:07.234969+0000","last_unstale":"2024-03-28T16:01:40.921045+0000","last_undegraded":"2024-03-28T16:01:40.921045+0000","last_fullsized":"2024-03-28T16:01:40.921045+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:50:35.592147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"92'8","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.039851+0000","last_change":"2024-03-28T16:01:08.263878+0000","last_active":"2024-03-28T16:02:39.039851+0000","last_peered":"2024-03-28T16:02:39.039851+0000","last_clean":"2024-03-28T16:02:39.039851+0000","last_became_active":"2024-03-28T16:01:08.263549+0000","last_became_peered":"2024-03-28T16:01:08.263549+0000","last_unstale":"2024-03-28T16:02:39.039851+0000","last_undegraded":"2024-03-28T16:02:39.039851+0000","last_fullsized":"2024-03-28T16:02:39.039851+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:31:37.669215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.663697+0000","last_change":"2024-03-28T16:01:07.255701+0000","last_active":"2024-03-28T16:01:40.663697+0000","last_peered":"2024-03-28T16:01:40.663697+0000","last_clean":"2024-03-28T16:01:40.663697+0000","last_became_active":"2024-03-28T16:01:07.255247+0000","last_became_peered":"2024-03-28T16:01:07.255247+0000","last_unstale":"2024-03-28T16:01:40.663697+0000","last_undegraded":"2024-03-28T16:01:40.663697+0000","last_fullsized":"2024-03-28T16:01:40.663697+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:08:17.858395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"92'11","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.034304+0000","last_change":"2024-03-28T16:01:08.252949+0000","last_active":"2024-03-28T16:02:39.034304+0000","last_peered":"2024-03-28T16:02:39.034304+0000","last_clean":"2024-03-28T16:02:39.034304+0000","last_became_active":"2024-03-28T16:01:08.252471+0000","last_became_peered":"2024-03-28T16:01:08.252471+0000","last_unstale":"2024-03-28T16:02:39.034304+0000","last_undegraded":"2024-03-28T16:02:39.034304+0000","last_fullsized":"2024-03-28T16:02:39.034304+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:21:15.519020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.661736+0000","last_change":"2024-03-28T16:01:07.263973+0000","last_active":"2024-03-28T16:01:40.661736+0000","last_peered":"2024-03-28T16:01:40.661736+0000","last_clean":"2024-03-28T16:01:40.661736+0000","last_became_active":"2024-03-28T16:01:07.263560+0000","last_became_peered":"2024-03-28T16:01:07.263560+0000","last_unstale":"2024-03-28T16:01:40.661736+0000","last_undegraded":"2024-03-28T16:01:40.661736+0000","last_fullsized":"2024-03-28T16:01:40.661736+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:11.215312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"92'10","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.051393+0000","last_change":"2024-03-28T16:01:08.266122+0000","last_active":"2024-03-28T16:02:39.051393+0000","last_peered":"2024-03-28T16:02:39.051393+0000","last_clean":"2024-03-28T16:02:39.051393+0000","last_became_active":"2024-03-28T16:01:08.265727+0000","last_became_peered":"2024-03-28T16:01:08.265727+0000","last_unstale":"2024-03-28T16:02:39.051393+0000","last_undegraded":"2024-03-28T16:02:39.051393+0000","last_fullsized":"2024-03-28T16:02:39.051393+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:48:38.939777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.666951+0000","last_change":"2024-03-28T16:01:07.251814+0000","last_active":"2024-03-28T16:01:40.666951+0000","last_peered":"2024-03-28T16:01:40.666951+0000","last_clean":"2024-03-28T16:01:40.666951+0000","last_became_active":"2024-03-28T16:01:07.251311+0000","last_became_peered":"2024-03-28T16:01:07.251311+0000","last_unstale":"2024-03-28T16:01:40.666951+0000","last_undegraded":"2024-03-28T16:01:40.666951+0000","last_fullsized":"2024-03-28T16:01:40.666951+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:13:38.344070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"92'9","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.049556+0000","last_change":"2024-03-28T16:01:08.270879+0000","last_active":"2024-03-28T16:02:39.049556+0000","last_peered":"2024-03-28T16:02:39.049556+0000","last_clean":"2024-03-28T16:02:39.049556+0000","last_became_active":"2024-03-28T16:01:08.270513+0000","last_became_peered":"2024-03-28T16:01:08.270513+0000","last_unstale":"2024-03-28T16:02:39.049556+0000","last_undegraded":"2024-03-28T16:02:39.049556+0000","last_fullsized":"2024-03-28T16:02:39.049556+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:53:43.285797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"89'3","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.665900+0000","last_change":"2024-03-28T16:01:07.253595+0000","last_active":"2024-03-28T16:01:40.665900+0000","last_peered":"2024-03-28T16:01:40.665900+0000","last_clean":"2024-03-28T16:01:40.665900+0000","last_became_active":"2024-03-28T16:01:07.253137+0000","last_became_peered":"2024-03-28T16:01:07.253137+0000","last_unstale":"2024-03-28T16:01:40.665900+0000","last_undegraded":"2024-03-28T16:01:40.665900+0000","last_fullsized":"2024-03-28T16:01:40.665900+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:17:35.759791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"92'11","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.050280+0000","last_change":"2024-03-28T16:01:08.261185+0000","last_active":"2024-03-28T16:02:39.050280+0000","last_peered":"2024-03-28T16:02:39.050280+0000","last_clean":"2024-03-28T16:02:39.050280+0000","last_became_active":"2024-03-28T16:01:08.260885+0000","last_became_peered":"2024-03-28T16:01:08.260885+0000","last_unstale":"2024-03-28T16:02:39.050280+0000","last_undegraded":"2024-03-28T16:02:39.050280+0000","last_fullsized":"2024-03-28T16:02:39.050280+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:15:56.640586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"92'31","reported_seq":326,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:03:23.893167+0000","last_change":"2024-03-28T16:01:07.258655+0000","last_active":"2024-03-28T16:03:23.893167+0000","last_peered":"2024-03-28T16:03:23.893167+0000","last_clean":"2024-03-28T16:03:23.893167+0000","last_became_active":"2024-03-28T16:01:07.258230+0000","last_became_peered":"2024-03-28T16:01:07.258230+0000","last_unstale":"2024-03-28T16:03:23.893167+0000","last_undegraded":"2024-03-28T16:03:23.893167+0000","last_fullsized":"2024-03-28T16:03:23.893167+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:29:36.206061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62638,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":276,"num_read_kb":199,"num_write":31,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"92'6","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.038343+0000","last_change":"2024-03-28T16:01:08.265393+0000","last_active":"2024-03-28T16:02:39.038343+0000","last_peered":"2024-03-28T16:02:39.038343+0000","last_clean":"2024-03-28T16:02:39.038343+0000","last_became_active":"2024-03-28T16:01:08.264560+0000","last_became_peered":"2024-03-28T16:01:08.264560+0000","last_unstale":"2024-03-28T16:02:39.038343+0000","last_undegraded":"2024-03-28T16:02:39.038343+0000","last_fullsized":"2024-03-28T16:02:39.038343+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:43:52.181113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"89'2","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.672781+0000","last_change":"2024-03-28T16:01:07.251428+0000","last_active":"2024-03-28T16:01:40.672781+0000","last_peered":"2024-03-28T16:01:40.672781+0000","last_clean":"2024-03-28T16:01:40.672781+0000","last_became_active":"2024-03-28T16:01:07.251074+0000","last_became_peered":"2024-03-28T16:01:07.251074+0000","last_unstale":"2024-03-28T16:01:40.672781+0000","last_undegraded":"2024-03-28T16:01:40.672781+0000","last_fullsized":"2024-03-28T16:01:40.672781+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:18:41.359338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"92'9","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.037592+0000","last_change":"2024-03-28T16:01:08.268482+0000","last_active":"2024-03-28T16:02:39.037592+0000","last_peered":"2024-03-28T16:02:39.037592+0000","last_clean":"2024-03-28T16:02:39.037592+0000","last_became_active":"2024-03-28T16:01:08.268153+0000","last_became_peered":"2024-03-28T16:01:08.268153+0000","last_unstale":"2024-03-28T16:02:39.037592+0000","last_undegraded":"2024-03-28T16:02:39.037592+0000","last_fullsized":"2024-03-28T16:02:39.037592+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:21:21.874773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.664138+0000","last_change":"2024-03-28T16:01:07.243506+0000","last_active":"2024-03-28T16:01:40.664138+0000","last_peered":"2024-03-28T16:01:40.664138+0000","last_clean":"2024-03-28T16:01:40.664138+0000","last_became_active":"2024-03-28T16:01:07.242793+0000","last_became_peered":"2024-03-28T16:01:07.242793+0000","last_unstale":"2024-03-28T16:01:40.664138+0000","last_undegraded":"2024-03-28T16:01:40.664138+0000","last_fullsized":"2024-03-28T16:01:40.664138+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:31:56.497276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"92'15","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.044862+0000","last_change":"2024-03-28T16:01:08.278848+0000","last_active":"2024-03-28T16:02:39.044862+0000","last_peered":"2024-03-28T16:02:39.044862+0000","last_clean":"2024-03-28T16:02:39.044862+0000","last_became_active":"2024-03-28T16:01:08.278455+0000","last_became_peered":"2024-03-28T16:01:08.278455+0000","last_unstale":"2024-03-28T16:02:39.044862+0000","last_undegraded":"2024-03-28T16:02:39.044862+0000","last_fullsized":"2024-03-28T16:02:39.044862+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:41:17.302538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.655733+0000","last_change":"2024-03-28T16:01:07.255070+0000","last_active":"2024-03-28T16:01:40.655733+0000","last_peered":"2024-03-28T16:01:40.655733+0000","last_clean":"2024-03-28T16:01:40.655733+0000","last_became_active":"2024-03-28T16:01:07.253489+0000","last_became_peered":"2024-03-28T16:01:07.253489+0000","last_unstale":"2024-03-28T16:01:40.655733+0000","last_undegraded":"2024-03-28T16:01:40.655733+0000","last_fullsized":"2024-03-28T16:01:40.655733+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:31:57.501721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"92'6","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.036271+0000","last_change":"2024-03-28T16:01:08.273190+0000","last_active":"2024-03-28T16:02:39.036271+0000","last_peered":"2024-03-28T16:02:39.036271+0000","last_clean":"2024-03-28T16:02:39.036271+0000","last_became_active":"2024-03-28T16:01:08.272870+0000","last_became_peered":"2024-03-28T16:01:08.272870+0000","last_unstale":"2024-03-28T16:02:39.036271+0000","last_undegraded":"2024-03-28T16:02:39.036271+0000","last_fullsized":"2024-03-28T16:02:39.036271+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:26:50.874750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"89'8","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.700937+0000","last_change":"2024-03-28T16:01:40.700157+0000","last_active":"2024-03-28T16:01:40.700937+0000","last_peered":"2024-03-28T16:01:40.700937+0000","last_clean":"2024-03-28T16:01:40.700937+0000","last_became_active":"2024-03-28T16:01:40.674436+0000","last_became_peered":"2024-03-28T16:01:40.674436+0000","last_unstale":"2024-03-28T16:01:40.700937+0000","last_undegraded":"2024-03-28T16:01:40.700937+0000","last_fullsized":"2024-03-28T16:01:40.700937+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:26:48.776506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,4],"acting":[11,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"92'6","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.049436+0000","last_change":"2024-03-28T16:01:40.924563+0000","last_active":"2024-03-28T16:02:39.049436+0000","last_peered":"2024-03-28T16:02:39.049436+0000","last_clean":"2024-03-28T16:02:39.049436+0000","last_became_active":"2024-03-28T16:01:40.923348+0000","last_became_peered":"2024-03-28T16:01:40.923348+0000","last_unstale":"2024-03-28T16:02:39.049436+0000","last_undegraded":"2024-03-28T16:02:39.049436+0000","last_fullsized":"2024-03-28T16:02:39.049436+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:31:46.619651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'1","reported_seq":15,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.681372+0000","last_change":"2024-03-28T16:01:40.680548+0000","last_active":"2024-03-28T16:01:40.681372+0000","last_peered":"2024-03-28T16:01:40.681372+0000","last_clean":"2024-03-28T16:01:40.681372+0000","last_became_active":"2024-03-28T16:01:40.672674+0000","last_became_peered":"2024-03-28T16:01:40.672674+0000","last_unstale":"2024-03-28T16:01:40.681372+0000","last_undegraded":"2024-03-28T16:01:40.681372+0000","last_fullsized":"2024-03-28T16:01:40.681372+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:07:32.767184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"92'5","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.035565+0000","last_change":"2024-03-28T16:01:08.274436+0000","last_active":"2024-03-28T16:02:39.035565+0000","last_peered":"2024-03-28T16:02:39.035565+0000","last_clean":"2024-03-28T16:02:39.035565+0000","last_became_active":"2024-03-28T16:01:08.274081+0000","last_became_peered":"2024-03-28T16:01:08.274081+0000","last_unstale":"2024-03-28T16:02:39.035565+0000","last_undegraded":"2024-03-28T16:02:39.035565+0000","last_fullsized":"2024-03-28T16:02:39.035565+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:57:00.531064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.653153+0000","last_change":"2024-03-28T16:01:07.247542+0000","last_active":"2024-03-28T16:01:40.653153+0000","last_peered":"2024-03-28T16:01:40.653153+0000","last_clean":"2024-03-28T16:01:40.653153+0000","last_became_active":"2024-03-28T16:01:07.247109+0000","last_became_peered":"2024-03-28T16:01:07.247109+0000","last_unstale":"2024-03-28T16:01:40.653153+0000","last_undegraded":"2024-03-28T16:01:40.653153+0000","last_fullsized":"2024-03-28T16:01:40.653153+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:14:19.387896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"92'5","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.037067+0000","last_change":"2024-03-28T16:01:08.264565+0000","last_active":"2024-03-28T16:02:39.037067+0000","last_peered":"2024-03-28T16:02:39.037067+0000","last_clean":"2024-03-28T16:02:39.037067+0000","last_became_active":"2024-03-28T16:01:08.264196+0000","last_became_peered":"2024-03-28T16:01:08.264196+0000","last_unstale":"2024-03-28T16:02:39.037067+0000","last_undegraded":"2024-03-28T16:02:39.037067+0000","last_fullsized":"2024-03-28T16:02:39.037067+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:43:24.750624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.666071+0000","last_change":"2024-03-28T16:01:07.248189+0000","last_active":"2024-03-28T16:01:40.666071+0000","last_peered":"2024-03-28T16:01:40.666071+0000","last_clean":"2024-03-28T16:01:40.666071+0000","last_became_active":"2024-03-28T16:01:07.247295+0000","last_became_peered":"2024-03-28T16:01:07.247295+0000","last_unstale":"2024-03-28T16:01:40.666071+0000","last_undegraded":"2024-03-28T16:01:40.666071+0000","last_fullsized":"2024-03-28T16:01:40.666071+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:46:51.875094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"92'7","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.031820+0000","last_change":"2024-03-28T16:01:08.269724+0000","last_active":"2024-03-28T16:02:39.031820+0000","last_peered":"2024-03-28T16:02:39.031820+0000","last_clean":"2024-03-28T16:02:39.031820+0000","last_became_active":"2024-03-28T16:01:08.269357+0000","last_became_peered":"2024-03-28T16:01:08.269357+0000","last_unstale":"2024-03-28T16:02:39.031820+0000","last_undegraded":"2024-03-28T16:02:39.031820+0000","last_fullsized":"2024-03-28T16:02:39.031820+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:20:12.622238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"92'8","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.035129+0000","last_change":"2024-03-28T16:01:08.266624+0000","last_active":"2024-03-28T16:02:39.035129+0000","last_peered":"2024-03-28T16:02:39.035129+0000","last_clean":"2024-03-28T16:02:39.035129+0000","last_became_active":"2024-03-28T16:01:08.265408+0000","last_became_peered":"2024-03-28T16:01:08.265408+0000","last_unstale":"2024-03-28T16:02:39.035129+0000","last_undegraded":"2024-03-28T16:02:39.035129+0000","last_fullsized":"2024-03-28T16:02:39.035129+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:37:08.831727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.920935+0000","last_change":"2024-03-28T16:01:07.241172+0000","last_active":"2024-03-28T16:01:40.920935+0000","last_peered":"2024-03-28T16:01:40.920935+0000","last_clean":"2024-03-28T16:01:40.920935+0000","last_became_active":"2024-03-28T16:01:07.240754+0000","last_became_peered":"2024-03-28T16:01:07.240754+0000","last_unstale":"2024-03-28T16:01:40.920935+0000","last_undegraded":"2024-03-28T16:01:40.920935+0000","last_fullsized":"2024-03-28T16:01:40.920935+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:52:45.805188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"92'8","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.047153+0000","last_change":"2024-03-28T16:01:08.278429+0000","last_active":"2024-03-28T16:02:39.047153+0000","last_peered":"2024-03-28T16:02:39.047153+0000","last_clean":"2024-03-28T16:02:39.047153+0000","last_became_active":"2024-03-28T16:01:08.275061+0000","last_became_peered":"2024-03-28T16:01:08.275061+0000","last_unstale":"2024-03-28T16:02:39.047153+0000","last_undegraded":"2024-03-28T16:02:39.047153+0000","last_fullsized":"2024-03-28T16:02:39.047153+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:35:42.015979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.655996+0000","last_change":"2024-03-28T16:01:07.252117+0000","last_active":"2024-03-28T16:01:40.655996+0000","last_peered":"2024-03-28T16:01:40.655996+0000","last_clean":"2024-03-28T16:01:40.655996+0000","last_became_active":"2024-03-28T16:01:07.249729+0000","last_became_peered":"2024-03-28T16:01:07.249729+0000","last_unstale":"2024-03-28T16:01:40.655996+0000","last_undegraded":"2024-03-28T16:01:40.655996+0000","last_fullsized":"2024-03-28T16:01:40.655996+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:33:32.719310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":347,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.663525+0000","last_change":"2024-03-28T15:59:32.593730+0000","last_active":"2024-03-28T16:01:40.663525+0000","last_peered":"2024-03-28T16:01:40.663525+0000","last_clean":"2024-03-28T16:01:40.663525+0000","last_became_active":"2024-03-28T15:59:32.592289+0000","last_became_peered":"2024-03-28T15:59:32.592289+0000","last_unstale":"2024-03-28T16:01:40.663525+0000","last_undegraded":"2024-03-28T16:01:40.663525+0000","last_fullsized":"2024-03-28T16:01:40.663525+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:54:55.638193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:54:55.638193+0000","last_clean_scrub_stamp":"2024-03-28T15:54:55.638193+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:20:37.087340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"92'5","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.025916+0000","last_change":"2024-03-28T16:01:08.265513+0000","last_active":"2024-03-28T16:02:39.025916+0000","last_peered":"2024-03-28T16:02:39.025916+0000","last_clean":"2024-03-28T16:02:39.025916+0000","last_became_active":"2024-03-28T16:01:08.264711+0000","last_became_peered":"2024-03-28T16:01:08.264711+0000","last_unstale":"2024-03-28T16:02:39.025916+0000","last_undegraded":"2024-03-28T16:02:39.025916+0000","last_fullsized":"2024-03-28T16:02:39.025916+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:10:01.936780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.665127+0000","last_change":"2024-03-28T16:01:07.252831+0000","last_active":"2024-03-28T16:01:40.665127+0000","last_peered":"2024-03-28T16:01:40.665127+0000","last_clean":"2024-03-28T16:01:40.665127+0000","last_became_active":"2024-03-28T16:01:07.252317+0000","last_became_peered":"2024-03-28T16:01:07.252317+0000","last_unstale":"2024-03-28T16:01:40.665127+0000","last_undegraded":"2024-03-28T16:01:40.665127+0000","last_fullsized":"2024-03-28T16:01:40.665127+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:55:37.461154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.655837+0000","last_change":"2024-03-28T16:01:07.262413+0000","last_active":"2024-03-28T16:01:40.655837+0000","last_peered":"2024-03-28T16:01:40.655837+0000","last_clean":"2024-03-28T16:01:40.655837+0000","last_became_active":"2024-03-28T16:01:07.261875+0000","last_became_peered":"2024-03-28T16:01:07.261875+0000","last_unstale":"2024-03-28T16:01:40.655837+0000","last_undegraded":"2024-03-28T16:01:40.655837+0000","last_fullsized":"2024-03-28T16:01:40.655837+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:25:49.124991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"92'9","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.042204+0000","last_change":"2024-03-28T16:01:08.259506+0000","last_active":"2024-03-28T16:02:39.042204+0000","last_peered":"2024-03-28T16:02:39.042204+0000","last_clean":"2024-03-28T16:02:39.042204+0000","last_became_active":"2024-03-28T16:01:08.258512+0000","last_became_peered":"2024-03-28T16:01:08.258512+0000","last_unstale":"2024-03-28T16:02:39.042204+0000","last_undegraded":"2024-03-28T16:02:39.042204+0000","last_fullsized":"2024-03-28T16:02:39.042204+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:20:17.017763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"92'10","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:03:17.778725+0000","last_change":"2024-03-28T16:01:07.254850+0000","last_active":"2024-03-28T16:03:17.778725+0000","last_peered":"2024-03-28T16:03:17.778725+0000","last_clean":"2024-03-28T16:03:17.778725+0000","last_became_active":"2024-03-28T16:01:07.252294+0000","last_became_peered":"2024-03-28T16:01:07.252294+0000","last_unstale":"2024-03-28T16:03:17.778725+0000","last_undegraded":"2024-03-28T16:03:17.778725+0000","last_fullsized":"2024-03-28T16:03:17.778725+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:54:26.700118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"92'5","reported_seq":16,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.039125+0000","last_change":"2024-03-28T16:01:40.675232+0000","last_active":"2024-03-28T16:02:39.039125+0000","last_peered":"2024-03-28T16:02:39.039125+0000","last_clean":"2024-03-28T16:02:39.039125+0000","last_became_active":"2024-03-28T16:01:40.674454+0000","last_became_peered":"2024-03-28T16:01:40.674454+0000","last_unstale":"2024-03-28T16:02:39.039125+0000","last_undegraded":"2024-03-28T16:02:39.039125+0000","last_fullsized":"2024-03-28T16:02:39.039125+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:47:16.997641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,7],"acting":[9,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.7","version":"89'2","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.657272+0000","last_change":"2024-03-28T16:01:07.245292+0000","last_active":"2024-03-28T16:01:40.657272+0000","last_peered":"2024-03-28T16:01:40.657272+0000","last_clean":"2024-03-28T16:01:40.657272+0000","last_became_active":"2024-03-28T16:01:07.244654+0000","last_became_peered":"2024-03-28T16:01:07.244654+0000","last_unstale":"2024-03-28T16:01:40.657272+0000","last_undegraded":"2024-03-28T16:01:40.657272+0000","last_fullsized":"2024-03-28T16:01:40.657272+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:03:39.027844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"92'7","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.049641+0000","last_change":"2024-03-28T16:01:08.274542+0000","last_active":"2024-03-28T16:02:39.049641+0000","last_peered":"2024-03-28T16:02:39.049641+0000","last_clean":"2024-03-28T16:02:39.049641+0000","last_became_active":"2024-03-28T16:01:08.274202+0000","last_became_peered":"2024-03-28T16:01:08.274202+0000","last_unstale":"2024-03-28T16:02:39.049641+0000","last_undegraded":"2024-03-28T16:02:39.049641+0000","last_fullsized":"2024-03-28T16:02:39.049641+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:09:18.199017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.838038+0000","last_change":"2024-03-28T16:01:07.253274+0000","last_active":"2024-03-28T16:01:40.838038+0000","last_peered":"2024-03-28T16:01:40.838038+0000","last_clean":"2024-03-28T16:01:40.838038+0000","last_became_active":"2024-03-28T16:01:07.250230+0000","last_became_peered":"2024-03-28T16:01:07.250230+0000","last_unstale":"2024-03-28T16:01:40.838038+0000","last_undegraded":"2024-03-28T16:01:40.838038+0000","last_fullsized":"2024-03-28T16:01:40.838038+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:20:31.459983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"92'2","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.026627+0000","last_change":"2024-03-28T16:01:08.266551+0000","last_active":"2024-03-28T16:02:39.026627+0000","last_peered":"2024-03-28T16:02:39.026627+0000","last_clean":"2024-03-28T16:02:39.026627+0000","last_became_active":"2024-03-28T16:01:08.266226+0000","last_became_peered":"2024-03-28T16:01:08.266226+0000","last_unstale":"2024-03-28T16:02:39.026627+0000","last_undegraded":"2024-03-28T16:02:39.026627+0000","last_fullsized":"2024-03-28T16:02:39.026627+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:51:16.520285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.764663+0000","last_change":"2024-03-28T16:01:07.252038+0000","last_active":"2024-03-28T16:01:40.764663+0000","last_peered":"2024-03-28T16:01:40.764663+0000","last_clean":"2024-03-28T16:01:40.764663+0000","last_became_active":"2024-03-28T16:01:07.251703+0000","last_became_peered":"2024-03-28T16:01:07.251703+0000","last_unstale":"2024-03-28T16:01:40.764663+0000","last_undegraded":"2024-03-28T16:01:40.764663+0000","last_fullsized":"2024-03-28T16:01:40.764663+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:35:42.518996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"92'9","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.048402+0000","last_change":"2024-03-28T16:01:08.260025+0000","last_active":"2024-03-28T16:02:39.048402+0000","last_peered":"2024-03-28T16:02:39.048402+0000","last_clean":"2024-03-28T16:02:39.048402+0000","last_became_active":"2024-03-28T16:01:08.259596+0000","last_became_peered":"2024-03-28T16:01:08.259596+0000","last_unstale":"2024-03-28T16:02:39.048402+0000","last_undegraded":"2024-03-28T16:02:39.048402+0000","last_fullsized":"2024-03-28T16:02:39.048402+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:45:29.228188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.668132+0000","last_change":"2024-03-28T16:01:07.259335+0000","last_active":"2024-03-28T16:01:40.668132+0000","last_peered":"2024-03-28T16:01:40.668132+0000","last_clean":"2024-03-28T16:01:40.668132+0000","last_became_active":"2024-03-28T16:01:07.258589+0000","last_became_peered":"2024-03-28T16:01:07.258589+0000","last_unstale":"2024-03-28T16:01:40.668132+0000","last_undegraded":"2024-03-28T16:01:40.668132+0000","last_fullsized":"2024-03-28T16:01:40.668132+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:06:41.742963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"92'8","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.048367+0000","last_change":"2024-03-28T16:01:08.266794+0000","last_active":"2024-03-28T16:02:39.048367+0000","last_peered":"2024-03-28T16:02:39.048367+0000","last_clean":"2024-03-28T16:02:39.048367+0000","last_became_active":"2024-03-28T16:01:08.266502+0000","last_became_peered":"2024-03-28T16:01:08.266502+0000","last_unstale":"2024-03-28T16:02:39.048367+0000","last_undegraded":"2024-03-28T16:02:39.048367+0000","last_fullsized":"2024-03-28T16:02:39.048367+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:17:12.671677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"89'2","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.655656+0000","last_change":"2024-03-28T16:01:07.259239+0000","last_active":"2024-03-28T16:01:40.655656+0000","last_peered":"2024-03-28T16:01:40.655656+0000","last_clean":"2024-03-28T16:01:40.655656+0000","last_became_active":"2024-03-28T16:01:07.258875+0000","last_became_peered":"2024-03-28T16:01:07.258875+0000","last_unstale":"2024-03-28T16:01:40.655656+0000","last_undegraded":"2024-03-28T16:01:40.655656+0000","last_fullsized":"2024-03-28T16:01:40.655656+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:35:12.729173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"92'5","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.052410+0000","last_change":"2024-03-28T16:01:08.278196+0000","last_active":"2024-03-28T16:02:39.052410+0000","last_peered":"2024-03-28T16:02:39.052410+0000","last_clean":"2024-03-28T16:02:39.052410+0000","last_became_active":"2024-03-28T16:01:08.276972+0000","last_became_peered":"2024-03-28T16:01:08.276972+0000","last_unstale":"2024-03-28T16:02:39.052410+0000","last_undegraded":"2024-03-28T16:02:39.052410+0000","last_fullsized":"2024-03-28T16:02:39.052410+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:28:25.231210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.668117+0000","last_change":"2024-03-28T16:01:07.246851+0000","last_active":"2024-03-28T16:01:40.668117+0000","last_peered":"2024-03-28T16:01:40.668117+0000","last_clean":"2024-03-28T16:01:40.668117+0000","last_became_active":"2024-03-28T16:01:07.246365+0000","last_became_peered":"2024-03-28T16:01:07.246365+0000","last_unstale":"2024-03-28T16:01:40.668117+0000","last_undegraded":"2024-03-28T16:01:40.668117+0000","last_fullsized":"2024-03-28T16:01:40.668117+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:16:13.593045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"92'7","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.053165+0000","last_change":"2024-03-28T16:01:08.279468+0000","last_active":"2024-03-28T16:02:39.053165+0000","last_peered":"2024-03-28T16:02:39.053165+0000","last_clean":"2024-03-28T16:02:39.053165+0000","last_became_active":"2024-03-28T16:01:08.274650+0000","last_became_peered":"2024-03-28T16:01:08.274650+0000","last_unstale":"2024-03-28T16:02:39.053165+0000","last_undegraded":"2024-03-28T16:02:39.053165+0000","last_fullsized":"2024-03-28T16:02:39.053165+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:36:32.776085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"89'1","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.665556+0000","last_change":"2024-03-28T16:01:07.258077+0000","last_active":"2024-03-28T16:01:40.665556+0000","last_peered":"2024-03-28T16:01:40.665556+0000","last_clean":"2024-03-28T16:01:40.665556+0000","last_became_active":"2024-03-28T16:01:07.256665+0000","last_became_peered":"2024-03-28T16:01:07.256665+0000","last_unstale":"2024-03-28T16:01:40.665556+0000","last_undegraded":"2024-03-28T16:01:40.665556+0000","last_fullsized":"2024-03-28T16:01:40.665556+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:01:18.852441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"92'7","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.040959+0000","last_change":"2024-03-28T16:01:08.278588+0000","last_active":"2024-03-28T16:02:39.040959+0000","last_peered":"2024-03-28T16:02:39.040959+0000","last_clean":"2024-03-28T16:02:39.040959+0000","last_became_active":"2024-03-28T16:01:08.277303+0000","last_became_peered":"2024-03-28T16:01:08.277303+0000","last_unstale":"2024-03-28T16:02:39.040959+0000","last_undegraded":"2024-03-28T16:02:39.040959+0000","last_fullsized":"2024-03-28T16:02:39.040959+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:01:19.927057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"89'2","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:01:40.921309+0000","last_change":"2024-03-28T16:01:07.249382+0000","last_active":"2024-03-28T16:01:40.921309+0000","last_peered":"2024-03-28T16:01:40.921309+0000","last_clean":"2024-03-28T16:01:40.921309+0000","last_became_active":"2024-03-28T16:01:07.249065+0000","last_became_peered":"2024-03-28T16:01:07.249065+0000","last_unstale":"2024-03-28T16:01:40.921309+0000","last_undegraded":"2024-03-28T16:01:40.921309+0000","last_fullsized":"2024-03-28T16:01:40.921309+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:06.159717+0000","last_clean_scrub_stamp":"2024-03-28T16:01:06.159717+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:07:22.851862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"92'9","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.038073+0000","last_change":"2024-03-28T16:01:08.274839+0000","last_active":"2024-03-28T16:02:39.038073+0000","last_peered":"2024-03-28T16:02:39.038073+0000","last_clean":"2024-03-28T16:02:39.038073+0000","last_became_active":"2024-03-28T16:01:08.274505+0000","last_became_peered":"2024-03-28T16:01:08.274505+0000","last_unstale":"2024-03-28T16:02:39.038073+0000","last_undegraded":"2024-03-28T16:02:39.038073+0000","last_fullsized":"2024-03-28T16:02:39.038073+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:57:37.635327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"92'9","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.037953+0000","last_change":"2024-03-28T16:01:40.676281+0000","last_active":"2024-03-28T16:02:39.037953+0000","last_peered":"2024-03-28T16:02:39.037953+0000","last_clean":"2024-03-28T16:02:39.037953+0000","last_became_active":"2024-03-28T16:01:40.675734+0000","last_became_peered":"2024-03-28T16:01:40.675734+0000","last_unstale":"2024-03-28T16:02:39.037953+0000","last_undegraded":"2024-03-28T16:02:39.037953+0000","last_fullsized":"2024-03-28T16:02:39.037953+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:23:39.282010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"92'12","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.030636+0000","last_change":"2024-03-28T16:01:08.272816+0000","last_active":"2024-03-28T16:02:39.030636+0000","last_peered":"2024-03-28T16:02:39.030636+0000","last_clean":"2024-03-28T16:02:39.030636+0000","last_became_active":"2024-03-28T16:01:08.270941+0000","last_became_peered":"2024-03-28T16:01:08.270941+0000","last_unstale":"2024-03-28T16:02:39.030636+0000","last_undegraded":"2024-03-28T16:02:39.030636+0000","last_fullsized":"2024-03-28T16:02:39.030636+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:19:00.481809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"92'8","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.047233+0000","last_change":"2024-03-28T16:01:08.258558+0000","last_active":"2024-03-28T16:02:39.047233+0000","last_peered":"2024-03-28T16:02:39.047233+0000","last_clean":"2024-03-28T16:02:39.047233+0000","last_became_active":"2024-03-28T16:01:08.258163+0000","last_became_peered":"2024-03-28T16:01:08.258163+0000","last_unstale":"2024-03-28T16:02:39.047233+0000","last_undegraded":"2024-03-28T16:02:39.047233+0000","last_fullsized":"2024-03-28T16:02:39.047233+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:53:48.358536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"92'7","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.048041+0000","last_change":"2024-03-28T16:01:08.268886+0000","last_active":"2024-03-28T16:02:39.048041+0000","last_peered":"2024-03-28T16:02:39.048041+0000","last_clean":"2024-03-28T16:02:39.048041+0000","last_became_active":"2024-03-28T16:01:08.268519+0000","last_became_peered":"2024-03-28T16:01:08.268519+0000","last_unstale":"2024-03-28T16:02:39.048041+0000","last_undegraded":"2024-03-28T16:02:39.048041+0000","last_fullsized":"2024-03-28T16:02:39.048041+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:49:05.266517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"92'4","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.049066+0000","last_change":"2024-03-28T16:01:08.265039+0000","last_active":"2024-03-28T16:02:39.049066+0000","last_peered":"2024-03-28T16:02:39.049066+0000","last_clean":"2024-03-28T16:02:39.049066+0000","last_became_active":"2024-03-28T16:01:08.264677+0000","last_became_peered":"2024-03-28T16:01:08.264677+0000","last_unstale":"2024-03-28T16:02:39.049066+0000","last_undegraded":"2024-03-28T16:02:39.049066+0000","last_fullsized":"2024-03-28T16:02:39.049066+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:17:27.667324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"92'6","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.035821+0000","last_change":"2024-03-28T16:01:08.271699+0000","last_active":"2024-03-28T16:02:39.035821+0000","last_peered":"2024-03-28T16:02:39.035821+0000","last_clean":"2024-03-28T16:02:39.035821+0000","last_became_active":"2024-03-28T16:01:08.270184+0000","last_became_peered":"2024-03-28T16:01:08.270184+0000","last_unstale":"2024-03-28T16:02:39.035821+0000","last_undegraded":"2024-03-28T16:02:39.035821+0000","last_fullsized":"2024-03-28T16:02:39.035821+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:17:15.699944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"92'15","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.038913+0000","last_change":"2024-03-28T16:01:08.262314+0000","last_active":"2024-03-28T16:02:39.038913+0000","last_peered":"2024-03-28T16:02:39.038913+0000","last_clean":"2024-03-28T16:02:39.038913+0000","last_became_active":"2024-03-28T16:01:08.261992+0000","last_became_peered":"2024-03-28T16:01:08.261992+0000","last_unstale":"2024-03-28T16:02:39.038913+0000","last_undegraded":"2024-03-28T16:02:39.038913+0000","last_fullsized":"2024-03-28T16:02:39.038913+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:40:52.048306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"92'2","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.027024+0000","last_change":"2024-03-28T16:01:08.259407+0000","last_active":"2024-03-28T16:02:39.027024+0000","last_peered":"2024-03-28T16:02:39.027024+0000","last_clean":"2024-03-28T16:02:39.027024+0000","last_became_active":"2024-03-28T16:01:08.259071+0000","last_became_peered":"2024-03-28T16:01:08.259071+0000","last_unstale":"2024-03-28T16:02:39.027024+0000","last_undegraded":"2024-03-28T16:02:39.027024+0000","last_fullsized":"2024-03-28T16:02:39.027024+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:30:17.819868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"92'6","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.039238+0000","last_change":"2024-03-28T16:01:08.271649+0000","last_active":"2024-03-28T16:02:39.039238+0000","last_peered":"2024-03-28T16:02:39.039238+0000","last_clean":"2024-03-28T16:02:39.039238+0000","last_became_active":"2024-03-28T16:01:08.271230+0000","last_became_peered":"2024-03-28T16:01:08.271230+0000","last_unstale":"2024-03-28T16:02:39.039238+0000","last_undegraded":"2024-03-28T16:02:39.039238+0000","last_fullsized":"2024-03-28T16:02:39.039238+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:50:14.404178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"92'8","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.031938+0000","last_change":"2024-03-28T16:01:08.260845+0000","last_active":"2024-03-28T16:02:39.031938+0000","last_peered":"2024-03-28T16:02:39.031938+0000","last_clean":"2024-03-28T16:02:39.031938+0000","last_became_active":"2024-03-28T16:01:08.259997+0000","last_became_peered":"2024-03-28T16:01:08.259997+0000","last_unstale":"2024-03-28T16:02:39.031938+0000","last_undegraded":"2024-03-28T16:02:39.031938+0000","last_fullsized":"2024-03-28T16:02:39.031938+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:08:55.080858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"92'10","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.050870+0000","last_change":"2024-03-28T16:01:08.268881+0000","last_active":"2024-03-28T16:02:39.050870+0000","last_peered":"2024-03-28T16:02:39.050870+0000","last_clean":"2024-03-28T16:02:39.050870+0000","last_became_active":"2024-03-28T16:01:08.268212+0000","last_became_peered":"2024-03-28T16:01:08.268212+0000","last_unstale":"2024-03-28T16:02:39.050870+0000","last_undegraded":"2024-03-28T16:02:39.050870+0000","last_fullsized":"2024-03-28T16:02:39.050870+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:46:22.447763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"92'11","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.051690+0000","last_change":"2024-03-28T16:01:08.265113+0000","last_active":"2024-03-28T16:02:39.051690+0000","last_peered":"2024-03-28T16:02:39.051690+0000","last_clean":"2024-03-28T16:02:39.051690+0000","last_became_active":"2024-03-28T16:01:08.264781+0000","last_became_peered":"2024-03-28T16:01:08.264781+0000","last_unstale":"2024-03-28T16:02:39.051690+0000","last_undegraded":"2024-03-28T16:02:39.051690+0000","last_fullsized":"2024-03-28T16:02:39.051690+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:19:14.720260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"92'8","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T16:02:39.030628+0000","last_change":"2024-03-28T16:01:08.274446+0000","last_active":"2024-03-28T16:02:39.030628+0000","last_peered":"2024-03-28T16:02:39.030628+0000","last_clean":"2024-03-28T16:02:39.030628+0000","last_became_active":"2024-03-28T16:01:08.274027+0000","last_became_peered":"2024-03-28T16:01:08.274027+0000","last_unstale":"2024-03-28T16:02:39.030628+0000","last_undegraded":"2024-03-28T16:02:39.030628+0000","last_fullsized":"2024-03-28T16:02:39.030628+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:01:07.206696+0000","last_clean_scrub_stamp":"2024-03-28T16:01:07.206696+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:33:37.395889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":63436,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":281,"num_read_kb":204,"num_write":88,"num_write_kb":91,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":190308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":85,"seq":365072220204,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":478849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":77,"seq":330712481843,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32812,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712340,"statfs":{"total":95995035648,"available":95961436160,"internally_reserved":0,"allocated":1675264,"data_stored":1132144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":71,"seq":304942678074,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":479409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":63,"seq":270582939713,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":479409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":55,"seq":236223201352,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36892,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708260,"statfs":{"total":95995035648,"available":95957258240,"internally_reserved":0,"allocated":1593344,"data_stored":1069894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397583,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36308,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708844,"statfs":{"total":95995035648,"available":95957856256,"internally_reserved":0,"allocated":995328,"data_stored":478944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574960,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28056,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717096,"statfs":{"total":95995035648,"available":95966306304,"internally_reserved":0,"allocated":1064960,"data_stored":541465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738485,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36308,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708844,"statfs":{"total":95995035648,"available":95957856256,"internally_reserved":0,"allocated":995328,"data_stored":478917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411434,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1069056,"data_stored":541499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117090,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36320,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708832,"statfs":{"total":95995035648,"available":95957843968,"internally_reserved":0,"allocated":1007616,"data_stored":479165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855452,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":478849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659222,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36900,"kb_used_data":1564,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708252,"statfs":{"total":95995035648,"available":95957250048,"internally_reserved":0,"allocated":1601536,"data_stored":1069494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":62638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":62672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":62832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T16:03:30.551 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T16:03:30.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.817+0000 7ff881800640 1 -- 172.21.15.67:0/2146789887 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff87c1096b0 msgr2=0x7ff87c10ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:30.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.817+0000 7ff881800640 1 --2- 172.21.15.67:0/2146789887 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff87c1096b0 0x7ff87c10ff40 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7ff874009900 tx=0x7ff87402bc30 comp rx=0 tx=0).stop 2024-03-28T16:03:30.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.817+0000 7ff881800640 1 -- 172.21.15.67:0/2146789887 shutdown_connections 2024-03-28T16:03:30.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.817+0000 7ff881800640 1 --2- 172.21.15.67:0/2146789887 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff87c1096b0 0x7ff87c10ff40 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:30.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.817+0000 7ff881800640 1 --2- 172.21.15.67:0/2146789887 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff87c108cf0 0x7ff87c109170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:30.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.817+0000 7ff881800640 1 --2- 172.21.15.67:0/2146789887 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff87c107b00 0x7ff87c107f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:30.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.817+0000 7ff881800640 1 -- 172.21.15.67:0/2146789887 >> 172.21.15.67:0/2146789887 conn(0x7ff87c076ab0 msgr2=0x7ff87c078ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:30.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.817+0000 7ff881800640 1 -- 172.21.15.67:0/2146789887 shutdown_connections 2024-03-28T16:03:30.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.817+0000 7ff881800640 1 -- 172.21.15.67:0/2146789887 wait complete. 2024-03-28T16:03:30.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.818+0000 7ff881800640 1 Processor -- start 2024-03-28T16:03:30.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.818+0000 7ff881800640 1 -- start start 2024-03-28T16:03:30.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.819+0000 7ff881800640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff87c107b00 0x7ff87c19dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:30.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.819+0000 7ff881800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff87c108cf0 0x7ff87c19e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:30.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.819+0000 7ff881800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff87c1096b0 0x7ff87c1a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:30.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.819+0000 7ff881800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff87c113610 con 0x7ff87c108cf0 2024-03-28T16:03:30.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.819+0000 7ff881800640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff87c113490 con 0x7ff87c107b00 2024-03-28T16:03:30.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.819+0000 7ff881800640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff87c113790 con 0x7ff87c1096b0 2024-03-28T16:03:30.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.819+0000 7ff87b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff87c108cf0 0x7ff87c19e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:30.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.819+0000 7ff87b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff87c108cf0 0x7ff87c19e1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50638/0 (socket says 172.21.15.67:50638) 2024-03-28T16:03:30.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.819+0000 7ff87b400640 1 -- 172.21.15.67:0/4214300768 learned_addr learned my addr 172.21.15.67:0/4214300768 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:03:30.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.819+0000 7ff880e00640 1 --2- 172.21.15.67:0/4214300768 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff87c1096b0 0x7ff87c1a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:30.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.820+0000 7ff87b400640 1 -- 172.21.15.67:0/4214300768 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff87c1096b0 msgr2=0x7ff87c1a51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:30.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.820+0000 7ff87b400640 1 --2- 172.21.15.67:0/4214300768 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff87c1096b0 0x7ff87c1a51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:30.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.820+0000 7ff87b400640 1 -- 172.21.15.67:0/4214300768 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff87c107b00 msgr2=0x7ff87c19dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:30.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.820+0000 7ff87b400640 1 --2- 172.21.15.67:0/4214300768 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff87c107b00 0x7ff87c19dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:30.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.820+0000 7ff87b400640 1 -- 172.21.15.67:0/4214300768 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff87c1a5890 con 0x7ff87c108cf0 2024-03-28T16:03:30.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.820+0000 7ff880e00640 1 --2- 172.21.15.67:0/4214300768 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff87c1096b0 0x7ff87c1a51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:03:30.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.820+0000 7ff87b400640 1 --2- 172.21.15.67:0/4214300768 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff87c108cf0 0x7ff87c19e1c0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7ff86c00ebc0 tx=0x7ff86c00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:30.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.827+0000 7ff878c00640 1 -- 172.21.15.67:0/4214300768 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff86c010630 con 0x7ff87c108cf0 2024-03-28T16:03:30.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.827+0000 7ff878c00640 1 -- 172.21.15.67:0/4214300768 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff86c010c80 con 0x7ff87c108cf0 2024-03-28T16:03:30.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.827+0000 7ff878c00640 1 -- 172.21.15.67:0/4214300768 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff86c018b30 con 0x7ff87c108cf0 2024-03-28T16:03:30.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.827+0000 7ff881800640 1 -- 172.21.15.67:0/4214300768 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff87c1a5b80 con 0x7ff87c108cf0 2024-03-28T16:03:30.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.828+0000 7ff881800640 1 -- 172.21.15.67:0/4214300768 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff87c1a6120 con 0x7ff87c108cf0 2024-03-28T16:03:30.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.829+0000 7ff881800640 1 -- 172.21.15.67:0/4214300768 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff87c10ee20 con 0x7ff87c108cf0 2024-03-28T16:03:30.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.831+0000 7ff878c00640 1 -- 172.21.15.67:0/4214300768 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ff86c018cd0 con 0x7ff87c108cf0 2024-03-28T16:03:30.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.832+0000 7ff878c00640 1 --2- 172.21.15.67:0/4214300768 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7ff8600a2880 0x7ff8600a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:30.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.832+0000 7ff878c00640 1 -- 172.21.15.67:0/4214300768 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7ff86c0cf1f0 con 0x7ff87c108cf0 2024-03-28T16:03:30.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.833+0000 7ff87be00640 1 --2- 172.21.15.67:0/4214300768 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7ff8600a2880 0x7ff8600a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:30.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.833+0000 7ff87be00640 1 --2- 172.21.15.67:0/4214300768 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7ff8600a2880 0x7ff8600a4d40 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7ff864005fd0 tx=0x7ff86400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:30.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:30.837+0000 7ff878c00640 1 -- 172.21.15.67:0/4214300768 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff86c09c240 con 0x7ff87c108cf0 2024-03-28T16:03:31.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.042+0000 7ff881800640 1 -- 172.21.15.67:0/4214300768 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff87c02d3c0 con 0x7ff87c108cf0 2024-03-28T16:03:31.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.044+0000 7ff878c00640 1 -- 172.21.15.67:0/4214300768 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) v1 ==== 74+0+22673 (secure 0 0 0) 0x7ff87c02d3c0 con 0x7ff87c108cf0 2024-03-28T16:03:31.045 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:03:31.045 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":92,"fsid":"b5f8225a-ed1a-11ee-95d0-87774f69a715","created":"2024-03-28T15:50:21.471924+0000","modified":"2024-03-28T16:01:40.643215+0000","last_up_change":"2024-03-28T16:00:01.369123+0000","last_in_change":"2024-03-28T15:59:37.515483+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T15:54:55.588007+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T16:01:05.719801+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T16:01:07.206303+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7c1791b6-9c0e-4512-a327-cad166bb5581","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6803","nonce":1894681701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6805","nonce":1894681701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6809","nonce":1894681701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6807","nonce":1894681701}]},"public_addr":"172.21.15.67:6803/1894681701","cluster_addr":"172.21.15.67:6805/1894681701","heartbeat_back_addr":"172.21.15.67:6809/1894681701","heartbeat_front_addr":"172.21.15.67:6807/1894681701","state":["exists","up"]},{"osd":1,"uuid":"0cdda7f3-d00c-4436-bcc1-49fb8072b2e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6801","nonce":1734465762}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6803","nonce":1734465762}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6807","nonce":1734465762}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6805","nonce":1734465762}]},"public_addr":"172.21.15.150:6801/1734465762","cluster_addr":"172.21.15.150:6803/1734465762","heartbeat_back_addr":"172.21.15.150:6807/1734465762","heartbeat_front_addr":"172.21.15.150:6805/1734465762","state":["exists","up"]},{"osd":2,"uuid":"a55b9a02-c3fd-4061-95c0-53a24a17ade0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6801","nonce":3398095931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6803","nonce":3398095931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6807","nonce":3398095931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6805","nonce":3398095931}]},"public_addr":"172.21.15.183:6801/3398095931","cluster_addr":"172.21.15.183:6803/3398095931","heartbeat_back_addr":"172.21.15.183:6807/3398095931","heartbeat_front_addr":"172.21.15.183:6805/3398095931","state":["exists","up"]},{"osd":3,"uuid":"d34940c0-5ce9-4b8b-aa36-4872414a043f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6811","nonce":1933541875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6813","nonce":1933541875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6817","nonce":1933541875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6815","nonce":1933541875}]},"public_addr":"172.21.15.67:6811/1933541875","cluster_addr":"172.21.15.67:6813/1933541875","heartbeat_back_addr":"172.21.15.67:6817/1933541875","heartbeat_front_addr":"172.21.15.67:6815/1933541875","state":["exists","up"]},{"osd":4,"uuid":"5cc3e260-83a3-4a4d-aa6a-34a3b12b2b5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6809","nonce":3775269993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6811","nonce":3775269993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6815","nonce":3775269993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6813","nonce":3775269993}]},"public_addr":"172.21.15.150:6809/3775269993","cluster_addr":"172.21.15.150:6811/3775269993","heartbeat_back_addr":"172.21.15.150:6815/3775269993","heartbeat_front_addr":"172.21.15.150:6813/3775269993","state":["exists","up"]},{"osd":5,"uuid":"a2542b21-d6ac-413e-8b9b-4a6548b20840","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6809","nonce":164284137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6811","nonce":164284137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6815","nonce":164284137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6813","nonce":164284137}]},"public_addr":"172.21.15.183:6809/164284137","cluster_addr":"172.21.15.183:6811/164284137","heartbeat_back_addr":"172.21.15.183:6815/164284137","heartbeat_front_addr":"172.21.15.183:6813/164284137","state":["exists","up"]},{"osd":6,"uuid":"8905a2d3-0f1b-4853-a218-22da2635150a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6819","nonce":3565528013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6821","nonce":3565528013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6825","nonce":3565528013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6823","nonce":3565528013}]},"public_addr":"172.21.15.67:6819/3565528013","cluster_addr":"172.21.15.67:6821/3565528013","heartbeat_back_addr":"172.21.15.67:6825/3565528013","heartbeat_front_addr":"172.21.15.67:6823/3565528013","state":["exists","up"]},{"osd":7,"uuid":"b275055b-b451-4180-91b1-e6e61a828c48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6817","nonce":3735115340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6819","nonce":3735115340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6823","nonce":3735115340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6821","nonce":3735115340}]},"public_addr":"172.21.15.150:6817/3735115340","cluster_addr":"172.21.15.150:6819/3735115340","heartbeat_back_addr":"172.21.15.150:6823/3735115340","heartbeat_front_addr":"172.21.15.150:6821/3735115340","state":["exists","up"]},{"osd":8,"uuid":"b6abf7ad-eb34-4f60-a4eb-348a92d7195b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6817","nonce":1025400095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6819","nonce":1025400095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6823","nonce":1025400095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6821","nonce":1025400095}]},"public_addr":"172.21.15.183:6817/1025400095","cluster_addr":"172.21.15.183:6819/1025400095","heartbeat_back_addr":"172.21.15.183:6823/1025400095","heartbeat_front_addr":"172.21.15.183:6821/1025400095","state":["exists","up"]},{"osd":9,"uuid":"28a78e3a-0657-40aa-816f-16a6fadadcbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6827","nonce":76885409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6829","nonce":76885409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6833","nonce":76885409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6831","nonce":76885409}]},"public_addr":"172.21.15.67:6827/76885409","cluster_addr":"172.21.15.67:6829/76885409","heartbeat_back_addr":"172.21.15.67:6833/76885409","heartbeat_front_addr":"172.21.15.67:6831/76885409","state":["exists","up"]},{"osd":10,"uuid":"d287907e-424f-4b49-a06e-4613c6de0474","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6825","nonce":3489982080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6827","nonce":3489982080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6831","nonce":3489982080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6829","nonce":3489982080}]},"public_addr":"172.21.15.150:6825/3489982080","cluster_addr":"172.21.15.150:6827/3489982080","heartbeat_back_addr":"172.21.15.150:6831/3489982080","heartbeat_front_addr":"172.21.15.150:6829/3489982080","state":["exists","up"]},{"osd":11,"uuid":"a3c29c47-a228-4f42-a23c-fbdb0c093699","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":85,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6825","nonce":1892639849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6826","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6827","nonce":1892639849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6831","nonce":1892639849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6829","nonce":1892639849}]},"public_addr":"172.21.15.183:6825/1892639849","cluster_addr":"172.21.15.183:6827/1892639849","heartbeat_back_addr":"172.21.15.183:6831/1892639849","heartbeat_front_addr":"172.21.15.183:6829/1892639849","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:53:45.646211+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:54:18.956590+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:54:52.161646+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:55:27.502822+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:56:01.162932+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:56:34.674452+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:57:09.298679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:57:43.230406+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:58:17.003821+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:58:51.085863+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:59:25.665065+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T16:00:00.263457+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":1}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6801/1631001968":"2024-03-29T15:51:38.875334+0000","172.21.15.67:6800/1631001968":"2024-03-29T15:51:38.875334+0000","172.21.15.67:0/1533056310":"2024-03-29T15:51:38.875334+0000","172.21.15.67:6801/3580491071":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/2030412458":"2024-03-29T15:51:03.859740+0000","172.21.15.67:6800/3580491071":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/2905307533":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/3763759988":"2024-03-29T15:51:38.875334+0000","172.21.15.67:0/4089399880":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/4239108661":"2024-03-29T15:51:38.875334+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T16:03:31.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.047+0000 7ff881800640 1 -- 172.21.15.67:0/4214300768 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7ff8600a2880 msgr2=0x7ff8600a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:31.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.047+0000 7ff881800640 1 --2- 172.21.15.67:0/4214300768 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7ff8600a2880 0x7ff8600a4d40 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7ff864005fd0 tx=0x7ff86400b410 comp rx=0 tx=0).stop 2024-03-28T16:03:31.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.047+0000 7ff881800640 1 -- 172.21.15.67:0/4214300768 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff87c108cf0 msgr2=0x7ff87c19e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:31.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.047+0000 7ff881800640 1 --2- 172.21.15.67:0/4214300768 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff87c108cf0 0x7ff87c19e1c0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7ff86c00ebc0 tx=0x7ff86c00c6a0 comp rx=0 tx=0).stop 2024-03-28T16:03:31.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.047+0000 7ff881800640 1 -- 172.21.15.67:0/4214300768 shutdown_connections 2024-03-28T16:03:31.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.047+0000 7ff881800640 1 --2- 172.21.15.67:0/4214300768 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7ff8600a2880 0x7ff8600a4d40 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:31.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.047+0000 7ff881800640 1 --2- 172.21.15.67:0/4214300768 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff87c1096b0 0x7ff87c1a51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:31.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.047+0000 7ff881800640 1 --2- 172.21.15.67:0/4214300768 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff87c108cf0 0x7ff87c19e1c0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:31.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.047+0000 7ff881800640 1 --2- 172.21.15.67:0/4214300768 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff87c107b00 0x7ff87c19dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:31.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.047+0000 7ff881800640 1 -- 172.21.15.67:0/4214300768 >> 172.21.15.67:0/4214300768 conn(0x7ff87c076ab0 msgr2=0x7ff87c10bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:31.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.047+0000 7ff881800640 1 -- 172.21.15.67:0/4214300768 shutdown_connections 2024-03-28T16:03:31.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.047+0000 7ff881800640 1 -- 172.21.15.67:0/4214300768 wait complete. 2024-03-28T16:03:31.066 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T15:54:55.588007+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T16:01:05.719801+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T16:01:07.206303+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T16:03:31.067 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T16:03:31.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:30 smithi183 ceph-mon[35580]: pgmap v437: 97 pgs: 97 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-28T16:03:31.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:30 smithi183 ceph-mon[35580]: from='client.15246 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T16:03:31.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:30 smithi183 ceph-mon[35580]: from='client.15252 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T16:03:31.135 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:30 smithi067 ceph-mon[31707]: pgmap v437: 97 pgs: 97 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-28T16:03:31.135 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:30 smithi067 ceph-mon[31707]: from='client.15246 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T16:03:31.136 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:30 smithi067 ceph-mon[31707]: from='client.15252 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T16:03:31.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.270+0000 7fa310800640 1 -- 172.21.15.67:0/2471244162 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa308103260 msgr2=0x7fa308109af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:31.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.270+0000 7fa310800640 1 --2- 172.21.15.67:0/2471244162 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa308103260 0x7fa308109af0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fa304009ca0 tx=0x7fa30402c0f0 comp rx=0 tx=0).stop 2024-03-28T16:03:31.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.271+0000 7fa310800640 1 -- 172.21.15.67:0/2471244162 shutdown_connections 2024-03-28T16:03:31.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.271+0000 7fa310800640 1 --2- 172.21.15.67:0/2471244162 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa308103260 0x7fa308109af0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:31.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.271+0000 7fa310800640 1 --2- 172.21.15.67:0/2471244162 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa3081028a0 0x7fa308102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:31.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.271+0000 7fa310800640 1 --2- 172.21.15.67:0/2471244162 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3081016b0 0x7fa308101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:31.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.271+0000 7fa310800640 1 -- 172.21.15.67:0/2471244162 >> 172.21.15.67:0/2471244162 conn(0x7fa3080fcc40 msgr2=0x7fa3080ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:31.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.271+0000 7fa310800640 1 -- 172.21.15.67:0/2471244162 shutdown_connections 2024-03-28T16:03:31.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.271+0000 7fa310800640 1 -- 172.21.15.67:0/2471244162 wait complete. 2024-03-28T16:03:31.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.272+0000 7fa310800640 1 Processor -- start 2024-03-28T16:03:31.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.272+0000 7fa310800640 1 -- start start 2024-03-28T16:03:31.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.272+0000 7fa310800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3081016b0 0x7fa308199aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:31.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.272+0000 7fa310800640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3081028a0 0x7fa308199fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:31.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.272+0000 7fa310800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa308103260 0x7fa3081a1050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:31.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.272+0000 7fa310800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa30810d1c0 con 0x7fa3081016b0 2024-03-28T16:03:31.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.272+0000 7fa310800640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa30810d040 con 0x7fa3081028a0 2024-03-28T16:03:31.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.272+0000 7fa310800640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa30810d340 con 0x7fa308103260 2024-03-28T16:03:31.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.273+0000 7fa30de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3081016b0 0x7fa308199aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:31.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.273+0000 7fa30de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3081016b0 0x7fa308199aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50656/0 (socket says 172.21.15.67:50656) 2024-03-28T16:03:31.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.273+0000 7fa30de00640 1 -- 172.21.15.67:0/2112670624 learned_addr learned my addr 172.21.15.67:0/2112670624 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:03:31.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.273+0000 7fa30d400640 1 --2- 172.21.15.67:0/2112670624 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3081028a0 0x7fa308199fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:31.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.273+0000 7fa30e800640 1 --2- 172.21.15.67:0/2112670624 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa308103260 0x7fa3081a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:31.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.273+0000 7fa30de00640 1 -- 172.21.15.67:0/2112670624 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa308103260 msgr2=0x7fa3081a1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:31.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.273+0000 7fa30de00640 1 --2- 172.21.15.67:0/2112670624 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa308103260 0x7fa3081a1050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:31.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.273+0000 7fa30de00640 1 -- 172.21.15.67:0/2112670624 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3081028a0 msgr2=0x7fa308199fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:31.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.273+0000 7fa30de00640 1 --2- 172.21.15.67:0/2112670624 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3081028a0 0x7fa308199fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:31.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.273+0000 7fa30de00640 1 -- 172.21.15.67:0/2112670624 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3081a16c0 con 0x7fa3081016b0 2024-03-28T16:03:31.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.273+0000 7fa30e800640 1 --2- 172.21.15.67:0/2112670624 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa308103260 0x7fa3081a1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:03:31.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.273+0000 7fa30de00640 1 --2- 172.21.15.67:0/2112670624 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3081016b0 0x7fa308199aa0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc00e9c0 tx=0x7fa2fc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:31.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.274+0000 7fa302a00640 1 -- 172.21.15.67:0/2112670624 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2fc00cd80 con 0x7fa3081016b0 2024-03-28T16:03:31.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.274+0000 7fa310800640 1 -- 172.21.15.67:0/2112670624 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3081a19b0 con 0x7fa3081016b0 2024-03-28T16:03:31.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.274+0000 7fa302a00640 1 -- 172.21.15.67:0/2112670624 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa2fc0035f0 con 0x7fa3081016b0 2024-03-28T16:03:31.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.274+0000 7fa302a00640 1 -- 172.21.15.67:0/2112670624 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2fc010690 con 0x7fa3081016b0 2024-03-28T16:03:31.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.274+0000 7fa310800640 1 -- 172.21.15.67:0/2112670624 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3081a1f50 con 0x7fa3081016b0 2024-03-28T16:03:31.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.275+0000 7fa310800640 1 -- 172.21.15.67:0/2112670624 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3080688e0 con 0x7fa3081016b0 2024-03-28T16:03:31.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.277+0000 7fa302a00640 1 -- 172.21.15.67:0/2112670624 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fa2fc0037a0 con 0x7fa3081016b0 2024-03-28T16:03:31.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.278+0000 7fa302a00640 1 --2- 172.21.15.67:0/2112670624 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fa2f00a2800 0x7fa2f00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:31.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.279+0000 7fa302a00640 1 -- 172.21.15.67:0/2112670624 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fa2fc0cef90 con 0x7fa3081016b0 2024-03-28T16:03:31.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.279+0000 7fa30d400640 1 --2- 172.21.15.67:0/2112670624 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fa2f00a2800 0x7fa2f00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:31.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.279+0000 7fa30d400640 1 --2- 172.21.15.67:0/2112670624 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fa2f00a2800 0x7fa2f00a4cc0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fa30819afb0 tx=0x7fa2f800b590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:31.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.287+0000 7fa302a00640 1 -- 172.21.15.67:0/2112670624 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa2fc09bfe0 con 0x7fa3081016b0 2024-03-28T16:03:31.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:30 smithi150 ceph-mon[34825]: pgmap v437: 97 pgs: 97 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-28T16:03:31.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:30 smithi150 ceph-mon[34825]: from='client.15246 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T16:03:31.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:30 smithi150 ceph-mon[34825]: from='client.15252 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T16:03:31.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.492+0000 7fa310800640 1 -- 172.21.15.67:0/2112670624 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa3081a2990 con 0x7fa3081016b0 2024-03-28T16:03:31.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.493+0000 7fa302a00640 1 -- 172.21.15.67:0/2112670624 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v92) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa2fc09b8e0 con 0x7fa3081016b0 2024-03-28T16:03:31.493 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-03-28T16:03:31.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.496+0000 7fa310800640 1 -- 172.21.15.67:0/2112670624 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fa2f00a2800 msgr2=0x7fa2f00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:31.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.496+0000 7fa310800640 1 --2- 172.21.15.67:0/2112670624 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fa2f00a2800 0x7fa2f00a4cc0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fa30819afb0 tx=0x7fa2f800b590 comp rx=0 tx=0).stop 2024-03-28T16:03:31.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.496+0000 7fa310800640 1 -- 172.21.15.67:0/2112670624 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3081016b0 msgr2=0x7fa308199aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:31.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.496+0000 7fa310800640 1 --2- 172.21.15.67:0/2112670624 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3081016b0 0x7fa308199aa0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc00e9c0 tx=0x7fa2fc00ee80 comp rx=0 tx=0).stop 2024-03-28T16:03:31.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.496+0000 7fa310800640 1 -- 172.21.15.67:0/2112670624 shutdown_connections 2024-03-28T16:03:31.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.496+0000 7fa310800640 1 --2- 172.21.15.67:0/2112670624 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fa2f00a2800 0x7fa2f00a4cc0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:31.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.496+0000 7fa310800640 1 --2- 172.21.15.67:0/2112670624 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa308103260 0x7fa3081a1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:31.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.496+0000 7fa310800640 1 --2- 172.21.15.67:0/2112670624 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3081028a0 0x7fa308199fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:31.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.496+0000 7fa310800640 1 --2- 172.21.15.67:0/2112670624 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3081016b0 0x7fa308199aa0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:31.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.496+0000 7fa310800640 1 -- 172.21.15.67:0/2112670624 >> 172.21.15.67:0/2112670624 conn(0x7fa3080fcc40 msgr2=0x7fa3080738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:31.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.496+0000 7fa310800640 1 -- 172.21.15.67:0/2112670624 shutdown_connections 2024-03-28T16:03:31.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.496+0000 7fa310800640 1 -- 172.21.15.67:0/2112670624 wait complete. 2024-03-28T16:03:31.516 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T16:03:31.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.722+0000 7f8918000640 1 -- 172.21.15.67:0/975545307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8910102140 msgr2=0x7f8910104530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:31.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.722+0000 7f8918000640 1 --2- 172.21.15.67:0/975545307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8910102140 0x7f8910104530 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f8904009900 tx=0x7f890402bc50 comp rx=0 tx=0).stop 2024-03-28T16:03:31.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.722+0000 7f8918000640 1 -- 172.21.15.67:0/975545307 shutdown_connections 2024-03-28T16:03:31.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.722+0000 7f8918000640 1 --2- 172.21.15.67:0/975545307 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89101073a0 0x7f89101097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:31.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.722+0000 7f8918000640 1 --2- 172.21.15.67:0/975545307 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8910104a70 0x7f8910106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:31.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.722+0000 7f8918000640 1 --2- 172.21.15.67:0/975545307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8910102140 0x7f8910104530 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:31.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.722+0000 7f8918000640 1 -- 172.21.15.67:0/975545307 >> 172.21.15.67:0/975545307 conn(0x7f89100fbb30 msgr2=0x7f89100fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:31.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.722+0000 7f8918000640 1 -- 172.21.15.67:0/975545307 shutdown_connections 2024-03-28T16:03:31.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.723+0000 7f8918000640 1 -- 172.21.15.67:0/975545307 wait complete. 2024-03-28T16:03:31.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.723+0000 7f8918000640 1 Processor -- start 2024-03-28T16:03:31.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.724+0000 7f8918000640 1 -- start start 2024-03-28T16:03:31.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.724+0000 7f8918000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8910102140 0x7f8910199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:31.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.724+0000 7f8918000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8910104a70 0x7f891019a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:31.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.724+0000 7f8918000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89101073a0 0x7f89101a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:31.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.724+0000 7f8918000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f891010d270 con 0x7f8910102140 2024-03-28T16:03:31.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.724+0000 7f8918000640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f891010d0f0 con 0x7f89101073a0 2024-03-28T16:03:31.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.724+0000 7f8918000640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f891010d3f0 con 0x7f8910104a70 2024-03-28T16:03:31.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.724+0000 7f8915600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8910102140 0x7f8910199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:31.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.724+0000 7f8915600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8910102140 0x7f8910199bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50668/0 (socket says 172.21.15.67:50668) 2024-03-28T16:03:31.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.724+0000 7f8915600640 1 -- 172.21.15.67:0/405269719 learned_addr learned my addr 172.21.15.67:0/405269719 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:03:31.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.724+0000 7f8914c00640 1 --2- 172.21.15.67:0/405269719 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8910104a70 0x7f891019a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:31.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.724+0000 7f8916000640 1 --2- 172.21.15.67:0/405269719 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89101073a0 0x7f89101a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:31.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.725+0000 7f8915600640 1 -- 172.21.15.67:0/405269719 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8910104a70 msgr2=0x7f891019a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:31.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.725+0000 7f8915600640 1 --2- 172.21.15.67:0/405269719 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8910104a70 0x7f891019a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:31.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.725+0000 7f8915600640 1 -- 172.21.15.67:0/405269719 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89101073a0 msgr2=0x7f89101a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:31.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.725+0000 7f8915600640 1 --2- 172.21.15.67:0/405269719 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89101073a0 0x7f89101a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:31.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.725+0000 7f8915600640 1 -- 172.21.15.67:0/405269719 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89101a1810 con 0x7f8910102140 2024-03-28T16:03:31.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.725+0000 7f8915600640 1 --2- 172.21.15.67:0/405269719 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8910102140 0x7f8910199bf0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f890402c260 tx=0x7f89040042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:31.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.725+0000 7f890e000640 1 -- 172.21.15.67:0/405269719 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8904043020 con 0x7f8910102140 2024-03-28T16:03:31.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.725+0000 7f890e000640 1 -- 172.21.15.67:0/405269719 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8904004aa0 con 0x7f8910102140 2024-03-28T16:03:31.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.725+0000 7f8918000640 1 -- 172.21.15.67:0/405269719 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89101a1aa0 con 0x7f8910102140 2024-03-28T16:03:31.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.725+0000 7f890e000640 1 -- 172.21.15.67:0/405269719 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8904003d40 con 0x7f8910102140 2024-03-28T16:03:31.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.725+0000 7f8918000640 1 -- 172.21.15.67:0/405269719 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89101a1ff0 con 0x7f8910102140 2024-03-28T16:03:31.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.727+0000 7f8918000640 1 -- 172.21.15.67:0/405269719 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8910108ef0 con 0x7f8910102140 2024-03-28T16:03:31.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.730+0000 7f890e000640 1 -- 172.21.15.67:0/405269719 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f8904046020 con 0x7f8910102140 2024-03-28T16:03:31.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.731+0000 7f890e000640 1 --2- 172.21.15.67:0/405269719 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f88f80a2800 0x7f88f80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:31.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.731+0000 7f890e000640 1 -- 172.21.15.67:0/405269719 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f89040f0e70 con 0x7f8910102140 2024-03-28T16:03:31.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.731+0000 7f8914c00640 1 --2- 172.21.15.67:0/405269719 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f88f80a2800 0x7f88f80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:31.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.732+0000 7f8914c00640 1 --2- 172.21.15.67:0/405269719 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f88f80a2800 0x7f88f80a4cc0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f891019b100 tx=0x7f88fc00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:31.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.735+0000 7f890e000640 1 -- 172.21.15.67:0/405269719 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f890404d020 con 0x7f8910102140 2024-03-28T16:03:31.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.949+0000 7f8918000640 1 -- 172.21.15.67:0/405269719 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f89100622a0 con 0x7f8910102140 2024-03-28T16:03:31.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.949+0000 7f890e000640 1 -- 172.21.15.67:0/405269719 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v92) v1 ==== 104+0+11 (secure 0 0 0) 0x7f89040bdec0 con 0x7f8910102140 2024-03-28T16:03:31.950 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-03-28T16:03:31.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.951+0000 7f8918000640 1 -- 172.21.15.67:0/405269719 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f88f80a2800 msgr2=0x7f88f80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:31.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.951+0000 7f8918000640 1 --2- 172.21.15.67:0/405269719 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f88f80a2800 0x7f88f80a4cc0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f891019b100 tx=0x7f88fc00b390 comp rx=0 tx=0).stop 2024-03-28T16:03:31.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.952+0000 7f8918000640 1 -- 172.21.15.67:0/405269719 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8910102140 msgr2=0x7f8910199bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:31.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.952+0000 7f8918000640 1 --2- 172.21.15.67:0/405269719 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8910102140 0x7f8910199bf0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f890402c260 tx=0x7f89040042f0 comp rx=0 tx=0).stop 2024-03-28T16:03:31.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.952+0000 7f8918000640 1 -- 172.21.15.67:0/405269719 shutdown_connections 2024-03-28T16:03:31.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.952+0000 7f8918000640 1 --2- 172.21.15.67:0/405269719 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f88f80a2800 0x7f88f80a4cc0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:31.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.952+0000 7f8918000640 1 --2- 172.21.15.67:0/405269719 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89101073a0 0x7f89101a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:31.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.952+0000 7f8918000640 1 --2- 172.21.15.67:0/405269719 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8910104a70 0x7f891019a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:31.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.952+0000 7f8918000640 1 --2- 172.21.15.67:0/405269719 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8910102140 0x7f8910199bf0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:31.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.952+0000 7f8918000640 1 -- 172.21.15.67:0/405269719 >> 172.21.15.67:0/405269719 conn(0x7f89100fbb30 msgr2=0x7f89101018b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:31.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.952+0000 7f8918000640 1 -- 172.21.15.67:0/405269719 shutdown_connections 2024-03-28T16:03:31.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:31.952+0000 7f8918000640 1 -- 172.21.15.67:0/405269719 wait complete. 2024-03-28T16:03:31.969 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T16:03:32.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:31 smithi183 ceph-mon[35580]: from='client.15258 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T16:03:32.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:31 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T16:03:32.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:31 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/4214300768' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:03:32.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:31 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/2112670624' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T16:03:32.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:31 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:03:32.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.157+0000 7f01d6400640 1 -- 172.21.15.67:0/3774864270 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01d0107bc0 msgr2=0x7f01d0107fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:32.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.157+0000 7f01d6400640 1 --2- 172.21.15.67:0/3774864270 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01d0107bc0 0x7f01d0107fe0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f01c4009900 tx=0x7f01c402bc40 comp rx=0 tx=0).stop 2024-03-28T16:03:32.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.158+0000 7f01d6400640 1 -- 172.21.15.67:0/3774864270 shutdown_connections 2024-03-28T16:03:32.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.158+0000 7f01d6400640 1 --2- 172.21.15.67:0/3774864270 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f01d0109770 0x7f01d0110000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:32.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.158+0000 7f01d6400640 1 --2- 172.21.15.67:0/3774864270 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01d0108db0 0x7f01d0109230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:32.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.158+0000 7f01d6400640 1 --2- 172.21.15.67:0/3774864270 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01d0107bc0 0x7f01d0107fe0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:32.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.158+0000 7f01d6400640 1 -- 172.21.15.67:0/3774864270 >> 172.21.15.67:0/3774864270 conn(0x7f01d0076b70 msgr2=0x7f01d0078f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:32.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.158+0000 7f01d6400640 1 -- 172.21.15.67:0/3774864270 shutdown_connections 2024-03-28T16:03:32.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.158+0000 7f01d6400640 1 -- 172.21.15.67:0/3774864270 wait complete. 2024-03-28T16:03:32.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.159+0000 7f01d6400640 1 Processor -- start 2024-03-28T16:03:32.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.159+0000 7f01d6400640 1 -- start start 2024-03-28T16:03:32.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.159+0000 7f01d6400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01d0107bc0 0x7f01d019de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:32.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.159+0000 7f01d6400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01d0108db0 0x7f01d019e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:32.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.159+0000 7f01d6400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f01d0109770 0x7f01d01a53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:32.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.159+0000 7f01d6400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d0113790 con 0x7f01d0107bc0 2024-03-28T16:03:32.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.159+0000 7f01d6400640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d0113610 con 0x7f01d0108db0 2024-03-28T16:03:32.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.159+0000 7f01d6400640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d0113910 con 0x7f01d0109770 2024-03-28T16:03:32.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.160+0000 7f01d5000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01d0107bc0 0x7f01d019de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:32.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.160+0000 7f01d5000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01d0107bc0 0x7f01d019de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50692/0 (socket says 172.21.15.67:50692) 2024-03-28T16:03:32.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.160+0000 7f01d5000640 1 -- 172.21.15.67:0/1903065113 learned_addr learned my addr 172.21.15.67:0/1903065113 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:03:32.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.160+0000 7f01cfe00640 1 --2- 172.21.15.67:0/1903065113 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01d0108db0 0x7f01d019e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:32.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.160+0000 7f01d5a00640 1 --2- 172.21.15.67:0/1903065113 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f01d0109770 0x7f01d01a53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:32.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.160+0000 7f01d5000640 1 -- 172.21.15.67:0/1903065113 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f01d0109770 msgr2=0x7f01d01a53b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:32.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.160+0000 7f01d5000640 1 --2- 172.21.15.67:0/1903065113 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f01d0109770 0x7f01d01a53b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:32.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.160+0000 7f01d5000640 1 -- 172.21.15.67:0/1903065113 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01d0108db0 msgr2=0x7f01d019e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:32.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.160+0000 7f01d5000640 1 --2- 172.21.15.67:0/1903065113 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01d0108db0 0x7f01d019e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:32.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.160+0000 7f01d5000640 1 -- 172.21.15.67:0/1903065113 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01d01a5a20 con 0x7f01d0107bc0 2024-03-28T16:03:32.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.160+0000 7f01d5a00640 1 --2- 172.21.15.67:0/1903065113 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f01d0109770 0x7f01d01a53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:03:32.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.160+0000 7f01d5000640 1 --2- 172.21.15.67:0/1903065113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01d0107bc0 0x7f01d019de00 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f01c4005ab0 tx=0x7f01c40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:32.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.161+0000 7f01cd600640 1 -- 172.21.15.67:0/1903065113 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01c4043020 con 0x7f01d0107bc0 2024-03-28T16:03:32.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.161+0000 7f01cd600640 1 -- 172.21.15.67:0/1903065113 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f01c40054c0 con 0x7f01d0107bc0 2024-03-28T16:03:32.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.161+0000 7f01cd600640 1 -- 172.21.15.67:0/1903065113 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01c4003de0 con 0x7f01d0107bc0 2024-03-28T16:03:32.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.161+0000 7f01d6400640 1 -- 172.21.15.67:0/1903065113 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01d01a5cb0 con 0x7f01d0107bc0 2024-03-28T16:03:32.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.161+0000 7f01d6400640 1 -- 172.21.15.67:0/1903065113 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01d01a6180 con 0x7f01d0107bc0 2024-03-28T16:03:32.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.162+0000 7f01d6400640 1 -- 172.21.15.67:0/1903065113 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01d0068800 con 0x7f01d0107bc0 2024-03-28T16:03:32.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.164+0000 7f01cd600640 1 -- 172.21.15.67:0/1903065113 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f01c4046020 con 0x7f01d0107bc0 2024-03-28T16:03:32.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.165+0000 7f01cd600640 1 --2- 172.21.15.67:0/1903065113 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f01b80a2800 0x7f01b80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:32.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.165+0000 7f01cd600640 1 -- 172.21.15.67:0/1903065113 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f01c40f1420 con 0x7f01d0107bc0 2024-03-28T16:03:32.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.165+0000 7f01cfe00640 1 --2- 172.21.15.67:0/1903065113 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f01b80a2800 0x7f01b80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:32.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.166+0000 7f01cfe00640 1 --2- 172.21.15.67:0/1903065113 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f01b80a2800 0x7f01b80a4cc0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f01bc005e10 tx=0x7f01bc005da0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:32.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.168+0000 7f01cd600640 1 -- 172.21.15.67:0/1903065113 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f01c404d020 con 0x7f01d0107bc0 2024-03-28T16:03:32.194 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:31 smithi067 ceph-mon[31707]: from='client.15258 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T16:03:32.194 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:31 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T16:03:32.194 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:31 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/4214300768' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:03:32.194 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:31 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/2112670624' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T16:03:32.195 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:31 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:03:32.203 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:31 smithi150 ceph-mon[34825]: from='client.15258 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T16:03:32.203 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:31 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T16:03:32.204 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:31 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/4214300768' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:03:32.204 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:31 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/2112670624' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T16:03:32.204 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:31 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:03:32.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.377+0000 7f01d6400640 1 -- 172.21.15.67:0/1903065113 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f01d00622a0 con 0x7f01d0107bc0 2024-03-28T16:03:32.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.377+0000 7f01cd600640 1 -- 172.21.15.67:0/1903065113 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) v1 ==== 100+0+11 (secure 0 0 0) 0x7f01c40be470 con 0x7f01d0107bc0 2024-03-28T16:03:32.378 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-03-28T16:03:32.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.379+0000 7f01d6400640 1 -- 172.21.15.67:0/1903065113 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f01b80a2800 msgr2=0x7f01b80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:32.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.379+0000 7f01d6400640 1 --2- 172.21.15.67:0/1903065113 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f01b80a2800 0x7f01b80a4cc0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f01bc005e10 tx=0x7f01bc005da0 comp rx=0 tx=0).stop 2024-03-28T16:03:32.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.379+0000 7f01d6400640 1 -- 172.21.15.67:0/1903065113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01d0107bc0 msgr2=0x7f01d019de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:32.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.379+0000 7f01d6400640 1 --2- 172.21.15.67:0/1903065113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01d0107bc0 0x7f01d019de00 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f01c4005ab0 tx=0x7f01c40042f0 comp rx=0 tx=0).stop 2024-03-28T16:03:32.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.380+0000 7f01d6400640 1 -- 172.21.15.67:0/1903065113 shutdown_connections 2024-03-28T16:03:32.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.380+0000 7f01d6400640 1 --2- 172.21.15.67:0/1903065113 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f01b80a2800 0x7f01b80a4cc0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:32.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.380+0000 7f01d6400640 1 --2- 172.21.15.67:0/1903065113 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f01d0109770 0x7f01d01a53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:32.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.380+0000 7f01d6400640 1 --2- 172.21.15.67:0/1903065113 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01d0108db0 0x7f01d019e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:32.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.380+0000 7f01d6400640 1 --2- 172.21.15.67:0/1903065113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01d0107bc0 0x7f01d019de00 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:32.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.380+0000 7f01d6400640 1 -- 172.21.15.67:0/1903065113 >> 172.21.15.67:0/1903065113 conn(0x7f01d0076b70 msgr2=0x7f01d010c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:32.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.380+0000 7f01d6400640 1 -- 172.21.15.67:0/1903065113 shutdown_connections 2024-03-28T16:03:32.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.380+0000 7f01d6400640 1 -- 172.21.15.67:0/1903065113 wait complete. 2024-03-28T16:03:32.399 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T16:03:32.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.606+0000 7f0c14a00640 1 -- 172.21.15.67:0/2337396482 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c101036f0 msgr2=0x7f0c10103b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:32.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.606+0000 7f0c14a00640 1 --2- 172.21.15.67:0/2337396482 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c101036f0 0x7f0c10103b10 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f0c00009900 tx=0x7f0c0002bc50 comp rx=0 tx=0).stop 2024-03-28T16:03:32.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.606+0000 7f0c14a00640 1 -- 172.21.15.67:0/2337396482 shutdown_connections 2024-03-28T16:03:32.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.606+0000 7f0c14a00640 1 --2- 172.21.15.67:0/2337396482 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0c101052a0 0x7f0c1010bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:32.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.606+0000 7f0c14a00640 1 --2- 172.21.15.67:0/2337396482 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0c101048e0 0x7f0c10104d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:32.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.606+0000 7f0c14a00640 1 --2- 172.21.15.67:0/2337396482 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c101036f0 0x7f0c10103b10 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:32.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.606+0000 7f0c14a00640 1 -- 172.21.15.67:0/2337396482 >> 172.21.15.67:0/2337396482 conn(0x7f0c100fec80 msgr2=0x7f0c101010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:32.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.606+0000 7f0c14a00640 1 -- 172.21.15.67:0/2337396482 shutdown_connections 2024-03-28T16:03:32.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.606+0000 7f0c14a00640 1 -- 172.21.15.67:0/2337396482 wait complete. 2024-03-28T16:03:32.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.607+0000 7f0c14a00640 1 Processor -- start 2024-03-28T16:03:32.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.607+0000 7f0c14a00640 1 -- start start 2024-03-28T16:03:32.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.607+0000 7f0c14a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0c101036f0 0x7f0c10199900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:32.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.607+0000 7f0c14a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c101048e0 0x7f0c10199e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:32.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.607+0000 7f0c14a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0c101052a0 0x7f0c101a0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:32.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.607+0000 7f0c14a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c1010f260 con 0x7f0c101048e0 2024-03-28T16:03:32.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.607+0000 7f0c14a00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c1010f0e0 con 0x7f0c101052a0 2024-03-28T16:03:32.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.607+0000 7f0c14a00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c1010f3e0 con 0x7f0c101036f0 2024-03-28T16:03:32.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.608+0000 7f0c07e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c101048e0 0x7f0c10199e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:32.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.608+0000 7f0c07e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c101048e0 0x7f0c10199e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50714/0 (socket says 172.21.15.67:50714) 2024-03-28T16:03:32.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.608+0000 7f0c07e00640 1 -- 172.21.15.67:0/3462454923 learned_addr learned my addr 172.21.15.67:0/3462454923 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:03:32.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.608+0000 7f0c0e800640 1 --2- 172.21.15.67:0/3462454923 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0c101036f0 0x7f0c10199900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:32.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.608+0000 7f0c0f200640 1 --2- 172.21.15.67:0/3462454923 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0c101052a0 0x7f0c101a0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:32.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.608+0000 7f0c07e00640 1 -- 172.21.15.67:0/3462454923 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0c101036f0 msgr2=0x7f0c10199900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:32.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.608+0000 7f0c07e00640 1 --2- 172.21.15.67:0/3462454923 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0c101036f0 0x7f0c10199900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:32.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.608+0000 7f0c07e00640 1 -- 172.21.15.67:0/3462454923 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0c101052a0 msgr2=0x7f0c101a0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:32.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.608+0000 7f0c07e00640 1 --2- 172.21.15.67:0/3462454923 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0c101052a0 0x7f0c101a0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:32.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.608+0000 7f0c07e00640 1 -- 172.21.15.67:0/3462454923 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c101a1520 con 0x7f0c101048e0 2024-03-28T16:03:32.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.608+0000 7f0c07e00640 1 --2- 172.21.15.67:0/3462454923 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c101048e0 0x7f0c10199e40 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f0bf800e9f0 tx=0x7f0bf800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:32.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.609+0000 7f0c07400640 1 -- 172.21.15.67:0/3462454923 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf800cd80 con 0x7f0c101048e0 2024-03-28T16:03:32.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.609+0000 7f0c07400640 1 -- 172.21.15.67:0/3462454923 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0bf80054d0 con 0x7f0c101048e0 2024-03-28T16:03:32.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.609+0000 7f0c14a00640 1 -- 172.21.15.67:0/3462454923 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c101a1810 con 0x7f0c101048e0 2024-03-28T16:03:32.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.609+0000 7f0c07400640 1 -- 172.21.15.67:0/3462454923 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf8010490 con 0x7f0c101048e0 2024-03-28T16:03:32.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.609+0000 7f0c14a00640 1 -- 172.21.15.67:0/3462454923 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c101a1d60 con 0x7f0c101048e0 2024-03-28T16:03:32.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.610+0000 7f0c14a00640 1 -- 172.21.15.67:0/3462454923 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c1010aa90 con 0x7f0c101048e0 2024-03-28T16:03:32.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.612+0000 7f0c07400640 1 -- 172.21.15.67:0/3462454923 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f0bf8005020 con 0x7f0c101048e0 2024-03-28T16:03:32.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.615+0000 7f0c07400640 1 --2- 172.21.15.67:0/3462454923 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f0bf40a2880 0x7f0bf40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:32.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.615+0000 7f0c07400640 1 -- 172.21.15.67:0/3462454923 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f0bf80cee60 con 0x7f0c101048e0 2024-03-28T16:03:32.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.615+0000 7f0c0e800640 1 --2- 172.21.15.67:0/3462454923 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f0bf40a2880 0x7f0bf40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:32.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.616+0000 7f0c0e800640 1 --2- 172.21.15.67:0/3462454923 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f0bf40a2880 0x7f0bf40a4d40 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f0c0002c260 tx=0x7f0c0002c800 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:32.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.619+0000 7f0c07400640 1 -- 172.21.15.67:0/3462454923 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0bf809beb0 con 0x7f0c101048e0 2024-03-28T16:03:32.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.884+0000 7f0c14a00640 1 -- 172.21.15.67:0/3462454923 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0c1002d070 con 0x7f0c101048e0 2024-03-28T16:03:32.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.885+0000 7f0c07400640 1 -- 172.21.15.67:0/3462454923 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11341 (secure 0 0 0) 0x7f0bf809b7b0 con 0x7f0c101048e0 2024-03-28T16:03:32.886 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:03:32.886 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2008297659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2008297659},{"type":"v1","addr":"172.21.15.67:6835","nonce":2008297659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14532,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/320295745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":320295745},{"type":"v1","addr":"172.21.15.67:6837","nonce":320295745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6839/691821273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":691821273},{"type":"v1","addr":"172.21.15.67:6839","nonce":691821273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6841/3211254875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6840","nonce":3211254875},{"type":"v1","addr":"172.21.15.67:6841","nonce":3211254875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2129665423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2129665423},{"type":"v1","addr":"172.21.15.183:6833","nonce":2129665423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24496,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/1589784425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":1589784425},{"type":"v1","addr":"172.21.15.183:6835","nonce":1589784425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24508,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/2185863835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":2185863835},{"type":"v1","addr":"172.21.15.183:6837","nonce":2185863835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24520,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6839/1242814398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":1242814398},{"type":"v1","addr":"172.21.15.183:6839","nonce":1242814398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":34409,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1008335521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1008335521},{"type":"v1","addr":"172.21.15.150:6833","nonce":1008335521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":34445,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3269935695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3269935695},{"type":"v1","addr":"172.21.15.150:6835","nonce":3269935695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T16:01:07.696824+0000","modified":"2024-03-28T16:01:11.997773+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":34508},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34475":{"gid":34475,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/1112886277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":1112886277},{"type":"v1","addr":"172.21.15.150:6837","nonce":1112886277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_34508":{"gid":34508,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6839/1211011614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":1211011614},{"type":"v1","addr":"172.21.15.150:6839","nonce":1211011614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34508,"qdb_cluster":[34508]},"id":1}]} 2024-03-28T16:03:32.886 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 19 2024-03-28T16:03:32.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.888+0000 7f0c14a00640 1 -- 172.21.15.67:0/3462454923 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f0bf40a2880 msgr2=0x7f0bf40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:32.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.888+0000 7f0c14a00640 1 --2- 172.21.15.67:0/3462454923 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f0bf40a2880 0x7f0bf40a4d40 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f0c0002c260 tx=0x7f0c0002c800 comp rx=0 tx=0).stop 2024-03-28T16:03:32.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.888+0000 7f0c14a00640 1 -- 172.21.15.67:0/3462454923 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c101048e0 msgr2=0x7f0c10199e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:32.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.888+0000 7f0c14a00640 1 --2- 172.21.15.67:0/3462454923 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c101048e0 0x7f0c10199e40 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f0bf800e9f0 tx=0x7f0bf800edf0 comp rx=0 tx=0).stop 2024-03-28T16:03:32.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.888+0000 7f0c14a00640 1 -- 172.21.15.67:0/3462454923 shutdown_connections 2024-03-28T16:03:32.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.888+0000 7f0c14a00640 1 --2- 172.21.15.67:0/3462454923 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f0bf40a2880 0x7f0bf40a4d40 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:32.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.888+0000 7f0c14a00640 1 --2- 172.21.15.67:0/3462454923 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0c101052a0 0x7f0c101a0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:32.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.888+0000 7f0c14a00640 1 --2- 172.21.15.67:0/3462454923 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c101048e0 0x7f0c10199e40 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:32.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.888+0000 7f0c14a00640 1 --2- 172.21.15.67:0/3462454923 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0c101036f0 0x7f0c10199900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:32.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.888+0000 7f0c14a00640 1 -- 172.21.15.67:0/3462454923 >> 172.21.15.67:0/3462454923 conn(0x7f0c100fec80 msgr2=0x7f0c10107c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:32.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.888+0000 7f0c14a00640 1 -- 172.21.15.67:0/3462454923 shutdown_connections 2024-03-28T16:03:32.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:32.889+0000 7f0c14a00640 1 -- 172.21.15.67:0/3462454923 wait complete. 2024-03-28T16:03:32.907 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T16:03:33.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:32 smithi183 ceph-mon[35580]: pgmap v438: 97 pgs: 97 active+clean; 643 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 682 B/s wr, 3 op/s 2024-03-28T16:03:33.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:32 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:03:33.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:32 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/405269719' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T16:03:33.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:32 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/1903065113' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T16:03:33.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.106+0000 7f96a3e00640 1 -- 172.21.15.67:0/269465019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96a41026e0 msgr2=0x7f96a4102b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:33.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.106+0000 7f96a3e00640 1 --2- 172.21.15.67:0/269465019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96a41026e0 0x7f96a4102b60 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f9690009900 tx=0x7f969002bc30 comp rx=0 tx=0).stop 2024-03-28T16:03:33.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.107+0000 7f96a3e00640 1 -- 172.21.15.67:0/269465019 shutdown_connections 2024-03-28T16:03:33.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.107+0000 7f96a3e00640 1 --2- 172.21.15.67:0/269465019 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f96a41030a0 0x7f96a4109930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.107+0000 7f96a3e00640 1 --2- 172.21.15.67:0/269465019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96a41026e0 0x7f96a4102b60 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.107+0000 7f96a3e00640 1 --2- 172.21.15.67:0/269465019 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96a41014f0 0x7f96a4101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.107+0000 7f96a3e00640 1 -- 172.21.15.67:0/269465019 >> 172.21.15.67:0/269465019 conn(0x7f96a40fca80 msgr2=0x7f96a40feea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:33.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.107+0000 7f96a3e00640 1 -- 172.21.15.67:0/269465019 shutdown_connections 2024-03-28T16:03:33.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.107+0000 7f96a3e00640 1 -- 172.21.15.67:0/269465019 wait complete. 2024-03-28T16:03:33.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.108+0000 7f96a3e00640 1 Processor -- start 2024-03-28T16:03:33.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.108+0000 7f96a3e00640 1 -- start start 2024-03-28T16:03:33.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.108+0000 7f96a3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96a41014f0 0x7f96a4199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:33.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.108+0000 7f96a3e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f96a41026e0 0x7f96a4199eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:33.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.108+0000 7f96a3e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96a41030a0 0x7f96a41a0f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:33.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.108+0000 7f96a3e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96a410d080 con 0x7f96a41014f0 2024-03-28T16:03:33.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.108+0000 7f96a3e00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96a410cf00 con 0x7f96a41030a0 2024-03-28T16:03:33.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.108+0000 7f96a3e00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96a410d200 con 0x7f96a41026e0 2024-03-28T16:03:33.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.109+0000 7f96a3400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96a41030a0 0x7f96a41a0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:33.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.109+0000 7f96a3400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96a41030a0 0x7f96a41a0f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.67:53658/0 (socket says 172.21.15.67:53658) 2024-03-28T16:03:33.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.109+0000 7f96a3400640 1 -- 172.21.15.67:0/1457962225 learned_addr learned my addr 172.21.15.67:0/1457962225 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:03:33.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.109+0000 7f96a2000640 1 --2- 172.21.15.67:0/1457962225 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f96a41026e0 0x7f96a4199eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:33.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.109+0000 7f96a2a00640 1 --2- 172.21.15.67:0/1457962225 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96a41014f0 0x7f96a4199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:33.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.109+0000 7f96a3400640 1 -- 172.21.15.67:0/1457962225 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f96a41026e0 msgr2=0x7f96a4199eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:33.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.109+0000 7f96a3400640 1 --2- 172.21.15.67:0/1457962225 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f96a41026e0 0x7f96a4199eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.109+0000 7f96a3400640 1 -- 172.21.15.67:0/1457962225 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96a41014f0 msgr2=0x7f96a4199970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:33.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.109+0000 7f96a3400640 1 --2- 172.21.15.67:0/1457962225 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96a41014f0 0x7f96a4199970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.109+0000 7f96a3400640 1 -- 172.21.15.67:0/1457962225 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96a41a1590 con 0x7f96a41030a0 2024-03-28T16:03:33.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.109+0000 7f96a2000640 1 --2- 172.21.15.67:0/1457962225 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f96a41026e0 0x7f96a4199eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T16:03:33.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.109+0000 7f96a2a00640 1 --2- 172.21.15.67:0/1457962225 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96a41014f0 0x7f96a4199970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:03:33.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.110+0000 7f96a3400640 1 --2- 172.21.15.67:0/1457962225 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96a41030a0 0x7f96a41a0f20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f969c00eaa0 tx=0x7f969c00ef60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:33.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.110+0000 7f969b400640 1 -- 172.21.15.67:0/1457962225 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f969c00cdb0 con 0x7f96a41030a0 2024-03-28T16:03:33.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.110+0000 7f969b400640 1 -- 172.21.15.67:0/1457962225 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f969c005020 con 0x7f96a41030a0 2024-03-28T16:03:33.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.110+0000 7f969b400640 1 -- 172.21.15.67:0/1457962225 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f969c0116d0 con 0x7f96a41030a0 2024-03-28T16:03:33.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.110+0000 7f96a3e00640 1 -- 172.21.15.67:0/1457962225 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96a41a1880 con 0x7f96a41030a0 2024-03-28T16:03:33.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.110+0000 7f96a3e00640 1 -- 172.21.15.67:0/1457962225 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96a41a1da0 con 0x7f96a41030a0 2024-03-28T16:03:33.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.111+0000 7f96a3e00640 1 -- 172.21.15.67:0/1457962225 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96a41a2000 con 0x7f96a41030a0 2024-03-28T16:03:33.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.113+0000 7f969b400640 1 -- 172.21.15.67:0/1457962225 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f969c0051d0 con 0x7f96a41030a0 2024-03-28T16:03:33.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.114+0000 7f969b400640 1 --2- 172.21.15.67:0/1457962225 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f96880a2800 0x7f96880a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:33.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.114+0000 7f969b400640 1 -- 172.21.15.67:0/1457962225 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f969c0ced80 con 0x7f96a41030a0 2024-03-28T16:03:33.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.115+0000 7f96a2a00640 1 --2- 172.21.15.67:0/1457962225 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f96880a2800 0x7f96880a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:33.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.115+0000 7f96a2a00640 1 --2- 172.21.15.67:0/1457962225 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f96880a2800 0x7f96880a4cc0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f96a4102540 tx=0x7f9694005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:33.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.119+0000 7f969b400640 1 -- 172.21.15.67:0/1457962225 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f969c016030 con 0x7f96a41030a0 2024-03-28T16:03:33.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:32 smithi150 ceph-mon[34825]: pgmap v438: 97 pgs: 97 active+clean; 643 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 682 B/s wr, 3 op/s 2024-03-28T16:03:33.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:32 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:03:33.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:32 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/405269719' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T16:03:33.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:32 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/1903065113' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T16:03:33.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:32 smithi067 ceph-mon[31707]: pgmap v438: 97 pgs: 97 active+clean; 643 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 682 B/s wr, 3 op/s 2024-03-28T16:03:33.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:32 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:03:33.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:32 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/405269719' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T16:03:33.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:32 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/1903065113' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T16:03:33.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.333+0000 7f96a3e00640 1 -- 172.21.15.67:0/1457962225 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f96a41a2270 con 0x7f96a41030a0 2024-03-28T16:03:33.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.336+0000 7f969b400640 1 -- 172.21.15.67:0/1457962225 <== mon.2 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) v1 ==== 74+0+22673 (secure 0 0 0) 0x7f969c09bdd0 con 0x7f96a41030a0 2024-03-28T16:03:33.337 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:03:33.337 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":92,"fsid":"b5f8225a-ed1a-11ee-95d0-87774f69a715","created":"2024-03-28T15:50:21.471924+0000","modified":"2024-03-28T16:01:40.643215+0000","last_up_change":"2024-03-28T16:00:01.369123+0000","last_in_change":"2024-03-28T15:59:37.515483+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T15:54:55.588007+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T16:01:05.719801+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T16:01:07.206303+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7c1791b6-9c0e-4512-a327-cad166bb5581","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6803","nonce":1894681701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6805","nonce":1894681701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6809","nonce":1894681701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6807","nonce":1894681701}]},"public_addr":"172.21.15.67:6803/1894681701","cluster_addr":"172.21.15.67:6805/1894681701","heartbeat_back_addr":"172.21.15.67:6809/1894681701","heartbeat_front_addr":"172.21.15.67:6807/1894681701","state":["exists","up"]},{"osd":1,"uuid":"0cdda7f3-d00c-4436-bcc1-49fb8072b2e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6801","nonce":1734465762}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6803","nonce":1734465762}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6807","nonce":1734465762}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6805","nonce":1734465762}]},"public_addr":"172.21.15.150:6801/1734465762","cluster_addr":"172.21.15.150:6803/1734465762","heartbeat_back_addr":"172.21.15.150:6807/1734465762","heartbeat_front_addr":"172.21.15.150:6805/1734465762","state":["exists","up"]},{"osd":2,"uuid":"a55b9a02-c3fd-4061-95c0-53a24a17ade0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6801","nonce":3398095931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6803","nonce":3398095931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6807","nonce":3398095931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6805","nonce":3398095931}]},"public_addr":"172.21.15.183:6801/3398095931","cluster_addr":"172.21.15.183:6803/3398095931","heartbeat_back_addr":"172.21.15.183:6807/3398095931","heartbeat_front_addr":"172.21.15.183:6805/3398095931","state":["exists","up"]},{"osd":3,"uuid":"d34940c0-5ce9-4b8b-aa36-4872414a043f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6811","nonce":1933541875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6813","nonce":1933541875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6817","nonce":1933541875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6815","nonce":1933541875}]},"public_addr":"172.21.15.67:6811/1933541875","cluster_addr":"172.21.15.67:6813/1933541875","heartbeat_back_addr":"172.21.15.67:6817/1933541875","heartbeat_front_addr":"172.21.15.67:6815/1933541875","state":["exists","up"]},{"osd":4,"uuid":"5cc3e260-83a3-4a4d-aa6a-34a3b12b2b5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6809","nonce":3775269993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6811","nonce":3775269993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6815","nonce":3775269993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6813","nonce":3775269993}]},"public_addr":"172.21.15.150:6809/3775269993","cluster_addr":"172.21.15.150:6811/3775269993","heartbeat_back_addr":"172.21.15.150:6815/3775269993","heartbeat_front_addr":"172.21.15.150:6813/3775269993","state":["exists","up"]},{"osd":5,"uuid":"a2542b21-d6ac-413e-8b9b-4a6548b20840","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6809","nonce":164284137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6811","nonce":164284137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6815","nonce":164284137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6813","nonce":164284137}]},"public_addr":"172.21.15.183:6809/164284137","cluster_addr":"172.21.15.183:6811/164284137","heartbeat_back_addr":"172.21.15.183:6815/164284137","heartbeat_front_addr":"172.21.15.183:6813/164284137","state":["exists","up"]},{"osd":6,"uuid":"8905a2d3-0f1b-4853-a218-22da2635150a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6819","nonce":3565528013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6821","nonce":3565528013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6825","nonce":3565528013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6823","nonce":3565528013}]},"public_addr":"172.21.15.67:6819/3565528013","cluster_addr":"172.21.15.67:6821/3565528013","heartbeat_back_addr":"172.21.15.67:6825/3565528013","heartbeat_front_addr":"172.21.15.67:6823/3565528013","state":["exists","up"]},{"osd":7,"uuid":"b275055b-b451-4180-91b1-e6e61a828c48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6817","nonce":3735115340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6819","nonce":3735115340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6823","nonce":3735115340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6821","nonce":3735115340}]},"public_addr":"172.21.15.150:6817/3735115340","cluster_addr":"172.21.15.150:6819/3735115340","heartbeat_back_addr":"172.21.15.150:6823/3735115340","heartbeat_front_addr":"172.21.15.150:6821/3735115340","state":["exists","up"]},{"osd":8,"uuid":"b6abf7ad-eb34-4f60-a4eb-348a92d7195b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6817","nonce":1025400095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6819","nonce":1025400095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6823","nonce":1025400095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6821","nonce":1025400095}]},"public_addr":"172.21.15.183:6817/1025400095","cluster_addr":"172.21.15.183:6819/1025400095","heartbeat_back_addr":"172.21.15.183:6823/1025400095","heartbeat_front_addr":"172.21.15.183:6821/1025400095","state":["exists","up"]},{"osd":9,"uuid":"28a78e3a-0657-40aa-816f-16a6fadadcbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6827","nonce":76885409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6829","nonce":76885409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6833","nonce":76885409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6831","nonce":76885409}]},"public_addr":"172.21.15.67:6827/76885409","cluster_addr":"172.21.15.67:6829/76885409","heartbeat_back_addr":"172.21.15.67:6833/76885409","heartbeat_front_addr":"172.21.15.67:6831/76885409","state":["exists","up"]},{"osd":10,"uuid":"d287907e-424f-4b49-a06e-4613c6de0474","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6825","nonce":3489982080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6827","nonce":3489982080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6831","nonce":3489982080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6829","nonce":3489982080}]},"public_addr":"172.21.15.150:6825/3489982080","cluster_addr":"172.21.15.150:6827/3489982080","heartbeat_back_addr":"172.21.15.150:6831/3489982080","heartbeat_front_addr":"172.21.15.150:6829/3489982080","state":["exists","up"]},{"osd":11,"uuid":"a3c29c47-a228-4f42-a23c-fbdb0c093699","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":85,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6825","nonce":1892639849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6826","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6827","nonce":1892639849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6831","nonce":1892639849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6829","nonce":1892639849}]},"public_addr":"172.21.15.183:6825/1892639849","cluster_addr":"172.21.15.183:6827/1892639849","heartbeat_back_addr":"172.21.15.183:6831/1892639849","heartbeat_front_addr":"172.21.15.183:6829/1892639849","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:53:45.646211+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:54:18.956590+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:54:52.161646+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:55:27.502822+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:56:01.162932+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:56:34.674452+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:57:09.298679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:57:43.230406+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:58:17.003821+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:58:51.085863+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:59:25.665065+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T16:00:00.263457+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":1}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6801/1631001968":"2024-03-29T15:51:38.875334+0000","172.21.15.67:6800/1631001968":"2024-03-29T15:51:38.875334+0000","172.21.15.67:0/1533056310":"2024-03-29T15:51:38.875334+0000","172.21.15.67:6801/3580491071":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/2030412458":"2024-03-29T15:51:03.859740+0000","172.21.15.67:6800/3580491071":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/2905307533":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/3763759988":"2024-03-29T15:51:38.875334+0000","172.21.15.67:0/4089399880":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/4239108661":"2024-03-29T15:51:38.875334+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T16:03:33.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.339+0000 7f96a3e00640 1 -- 172.21.15.67:0/1457962225 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f96880a2800 msgr2=0x7f96880a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:33.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.339+0000 7f96a3e00640 1 --2- 172.21.15.67:0/1457962225 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f96880a2800 0x7f96880a4cc0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f96a4102540 tx=0x7f9694005f90 comp rx=0 tx=0).stop 2024-03-28T16:03:33.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.339+0000 7f96a3e00640 1 -- 172.21.15.67:0/1457962225 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96a41030a0 msgr2=0x7f96a41a0f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:33.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.339+0000 7f96a3e00640 1 --2- 172.21.15.67:0/1457962225 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96a41030a0 0x7f96a41a0f20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f969c00eaa0 tx=0x7f969c00ef60 comp rx=0 tx=0).stop 2024-03-28T16:03:33.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.339+0000 7f96a3e00640 1 -- 172.21.15.67:0/1457962225 shutdown_connections 2024-03-28T16:03:33.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.339+0000 7f96a3e00640 1 --2- 172.21.15.67:0/1457962225 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f96880a2800 0x7f96880a4cc0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.339+0000 7f96a3e00640 1 --2- 172.21.15.67:0/1457962225 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96a41030a0 0x7f96a41a0f20 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.339+0000 7f96a3e00640 1 --2- 172.21.15.67:0/1457962225 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f96a41026e0 0x7f96a4199eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.339+0000 7f96a3e00640 1 --2- 172.21.15.67:0/1457962225 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96a41014f0 0x7f96a4199970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.339+0000 7f96a3e00640 1 -- 172.21.15.67:0/1457962225 >> 172.21.15.67:0/1457962225 conn(0x7f96a40fca80 msgr2=0x7f96a40738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:33.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.339+0000 7f96a3e00640 1 -- 172.21.15.67:0/1457962225 shutdown_connections 2024-03-28T16:03:33.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.339+0000 7f96a3e00640 1 -- 172.21.15.67:0/1457962225 wait complete. 2024-03-28T16:03:33.358 DEBUG:tasks.fwd_scrub:Yielding 2024-03-28T16:03:33.358 INFO:teuthology.run_tasks:Running task workunit... 2024-03-28T16:03:33.372 INFO:tasks.workunit:Pulling workunits from ref 3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T16:03:33.372 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-28T16:03:33.372 INFO:tasks.workunit:timeout=3h 2024-03-28T16:03:33.372 INFO:tasks.workunit:cleanup=True 2024-03-28T16:03:33.372 DEBUG:teuthology.orchestra.run.smithi067:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-28T16:03:33.373 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-03-28T16:03:33.373 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T16:03:33.373 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T16:03:33.373 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T16:03:33.418 INFO:teuthology.orchestra.run.smithi067.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-28T16:03:33.418 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-03-28T16:03:33.418 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 76h/118d Inode: 1 Links: 2 2024-03-28T16:03:33.418 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T16:03:33.418 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-28T16:03:33.418 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-03-28 16:01:13.551817064 +0000 2024-03-28T16:03:33.418 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-03-28 16:01:13.551817064 +0000 2024-03-28T16:03:33.419 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-03-28 16:03:24.716526364 +0000 2024-03-28T16:03:33.419 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: - 2024-03-28T16:03:33.419 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-28T16:03:33.419 DEBUG:teuthology.orchestra.run.smithi067:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-28T16:03:33.552 DEBUG:teuthology.orchestra.run.smithi067:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T16:03:33.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.588+0000 7fb05fe00640 1 -- 172.21.15.67:0/3358780146 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb058104a50 msgr2=0x7fb058106e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:33.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.588+0000 7fb05fe00640 1 --2- 172.21.15.67:0/3358780146 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb058104a50 0x7fb058106e40 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fb044009900 tx=0x7fb04402bc30 comp rx=0 tx=0).stop 2024-03-28T16:03:33.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.588+0000 7fb05fe00640 1 -- 172.21.15.67:0/3358780146 shutdown_connections 2024-03-28T16:03:33.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.588+0000 7fb05fe00640 1 --2- 172.21.15.67:0/3358780146 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb058107380 0x7fb0581097d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.588+0000 7fb05fe00640 1 --2- 172.21.15.67:0/3358780146 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb058104a50 0x7fb058106e40 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.588+0000 7fb05fe00640 1 --2- 172.21.15.67:0/3358780146 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb058102120 0x7fb058104510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.588+0000 7fb05fe00640 1 -- 172.21.15.67:0/3358780146 >> 172.21.15.67:0/3358780146 conn(0x7fb0580fbb30 msgr2=0x7fb0580fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:33.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.588+0000 7fb05fe00640 1 -- 172.21.15.67:0/3358780146 shutdown_connections 2024-03-28T16:03:33.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.588+0000 7fb05fe00640 1 -- 172.21.15.67:0/3358780146 wait complete. 2024-03-28T16:03:33.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.589+0000 7fb05fe00640 1 Processor -- start 2024-03-28T16:03:33.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.590+0000 7fb05fe00640 1 -- start start 2024-03-28T16:03:33.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.590+0000 7fb05fe00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb058102120 0x7fb058197920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:33.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.590+0000 7fb05fe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb058104a50 0x7fb058197e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:33.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.590+0000 7fb05fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb058107380 0x7fb05819eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:33.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.590+0000 7fb05fe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb05810d210 con 0x7fb058107380 2024-03-28T16:03:33.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.590+0000 7fb05fe00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb05810d090 con 0x7fb058102120 2024-03-28T16:03:33.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.590+0000 7fb05fe00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb05810d390 con 0x7fb058104a50 2024-03-28T16:03:33.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.591+0000 7fb05d400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb058102120 0x7fb058197920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:33.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.591+0000 7fb05ca00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb058104a50 0x7fb058197e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:33.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.591+0000 7fb05de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb058107380 0x7fb05819eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:33.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.591+0000 7fb05ca00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb058104a50 0x7fb058197e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.67:34146/0 (socket says 172.21.15.67:34146) 2024-03-28T16:03:33.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.591+0000 7fb05d400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb058102120 0x7fb058197920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.67:53666/0 (socket says 172.21.15.67:53666) 2024-03-28T16:03:33.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.591+0000 7fb05ca00640 1 -- 172.21.15.67:0/2602184188 learned_addr learned my addr 172.21.15.67:0/2602184188 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:03:33.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.591+0000 7fb05ca00640 1 -- 172.21.15.67:0/2602184188 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb058102120 msgr2=0x7fb058197920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:33.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.591+0000 7fb05ca00640 1 --2- 172.21.15.67:0/2602184188 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb058102120 0x7fb058197920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.591+0000 7fb05ca00640 1 -- 172.21.15.67:0/2602184188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb058107380 msgr2=0x7fb05819eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:33.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.591+0000 7fb05ca00640 1 --2- 172.21.15.67:0/2602184188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb058107380 0x7fb05819eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.591+0000 7fb05ca00640 1 -- 172.21.15.67:0/2602184188 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb05819f540 con 0x7fb058104a50 2024-03-28T16:03:33.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.591+0000 7fb05de00640 1 --2- 172.21.15.67:0/2602184188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb058107380 0x7fb05819eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T16:03:33.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.592+0000 7fb05ca00640 1 --2- 172.21.15.67:0/2602184188 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb058104a50 0x7fb058197e60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb044000c00 tx=0x7fb044005e60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:33.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.592+0000 7fb056000640 1 -- 172.21.15.67:0/2602184188 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb044043020 con 0x7fb058104a50 2024-03-28T16:03:33.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.592+0000 7fb056000640 1 -- 172.21.15.67:0/2602184188 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb0440043d0 con 0x7fb058104a50 2024-03-28T16:03:33.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.592+0000 7fb05fe00640 1 -- 172.21.15.67:0/2602184188 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb05819f7d0 con 0x7fb058104a50 2024-03-28T16:03:33.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.592+0000 7fb056000640 1 -- 172.21.15.67:0/2602184188 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb044003bb0 con 0x7fb058104a50 2024-03-28T16:03:33.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.592+0000 7fb05fe00640 1 -- 172.21.15.67:0/2602184188 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb05819fd70 con 0x7fb058104a50 2024-03-28T16:03:33.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.596+0000 7fb062bd9740 1 Processor -- start 2024-03-28T16:03:33.596 INFO:tasks.workunit.client.0.smithi067.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-28T16:03:33.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.596+0000 7fb062bd9740 1 -- start start 2024-03-28T16:03:33.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.596+0000 7fb062bd9740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bfccb3a4b0 0x55bfccb3a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:33.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.596+0000 7fb062bd9740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bfccb3aea0 0x55bfccb3f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:33.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.596+0000 7fb062bd9740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bfccb40010 0x55bfccb424a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:33.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.596+0000 7fb062bd9740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bfcca9beb0 con 0x55bfccb3a4b0 2024-03-28T16:03:33.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.596+0000 7fb062bd9740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bfcca44a60 con 0x55bfccb3aea0 2024-03-28T16:03:33.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.596+0000 7fb062bd9740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bfcca97f50 con 0x55bfccb40010 2024-03-28T16:03:33.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.597+0000 7fb05de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bfccb3a4b0 0x55bfccb3a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:33.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.597+0000 7fb05de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bfccb3a4b0 0x55bfccb3a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50756/0 (socket says 172.21.15.67:50756) 2024-03-28T16:03:33.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.597+0000 7fb05de00640 1 -- 172.21.15.67:0/3943922388 learned_addr learned my addr 172.21.15.67:0/3943922388 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:03:33.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.597+0000 7fb05d400640 1 --2- 172.21.15.67:0/3943922388 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bfccb3aea0 0x55bfccb3f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:33.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.597+0000 7fb056000640 1 -- 172.21.15.67:0/2602184188 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fb044046020 con 0x7fb058104a50 2024-03-28T16:03:33.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.597+0000 7fb05de00640 1 -- 172.21.15.67:0/3943922388 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bfccb40010 msgr2=0x55bfccb424a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T16:03:33.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.597+0000 7fb05de00640 1 --2- 172.21.15.67:0/3943922388 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bfccb40010 0x55bfccb424a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.597+0000 7fb05de00640 1 -- 172.21.15.67:0/3943922388 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bfccb3aea0 msgr2=0x55bfccb3f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:03:33.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.597+0000 7fb05de00640 1 --2- 172.21.15.67:0/3943922388 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bfccb3aea0 0x55bfccb3f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.597+0000 7fb05de00640 1 -- 172.21.15.67:0/3943922388 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bfcc8787e0 con 0x55bfccb3a4b0 2024-03-28T16:03:33.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.597+0000 7fb05d400640 1 --2- 172.21.15.67:0/3943922388 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bfccb3aea0 0x55bfccb3f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T16:03:33.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.598+0000 7fb05de00640 1 --2- 172.21.15.67:0/3943922388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bfccb3a4b0 0x55bfccb3a8d0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fb0500117e0 tx=0x7fb050011ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=684269ff1f71d3e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:33.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.598+0000 7fb04a600640 1 -- 172.21.15.67:0/3943922388 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb050011310 con 0x55bfccb3a4b0 2024-03-28T16:03:33.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.598+0000 7fb04a600640 1 -- 172.21.15.67:0/3943922388 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb050003e20 con 0x55bfccb3a4b0 2024-03-28T16:03:33.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.598+0000 7fb04a600640 1 -- 172.21.15.67:0/3943922388 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0500044f0 con 0x55bfccb3a4b0 2024-03-28T16:03:33.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.599+0000 7fb062bd9740 1 -- 172.21.15.67:0/3943922388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bfccb3a4b0 msgr2=0x55bfccb3a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:33.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.599+0000 7fb062bd9740 1 --2- 172.21.15.67:0/3943922388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bfccb3a4b0 0x55bfccb3a8d0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fb0500117e0 tx=0x7fb050011ca0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.599+0000 7fb062bd9740 1 -- 172.21.15.67:0/3943922388 shutdown_connections 2024-03-28T16:03:33.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.599+0000 7fb062bd9740 1 --2- 172.21.15.67:0/3943922388 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bfccb40010 0x55bfccb424a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.599+0000 7fb062bd9740 1 --2- 172.21.15.67:0/3943922388 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bfccb3aea0 0x55bfccb3f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.599+0000 7fb062bd9740 1 --2- 172.21.15.67:0/3943922388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bfccb3a4b0 0x55bfccb3a8d0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.599+0000 7fb062bd9740 1 -- 172.21.15.67:0/3943922388 >> 172.21.15.67:0/3943922388 conn(0x55bfccb31610 msgr2=0x55bfccb31a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:33.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.599+0000 7fb062bd9740 1 -- 172.21.15.67:0/3943922388 shutdown_connections 2024-03-28T16:03:33.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.599+0000 7fb062bd9740 1 -- 172.21.15.67:0/3943922388 wait complete. 2024-03-28T16:03:33.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.600+0000 7fb056000640 1 --2- 172.21.15.67:0/2602184188 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb0400a2790 0x7fb0400a4c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:33.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.600+0000 7fb062bd9740 1 -- start start 2024-03-28T16:03:33.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.600+0000 7fb056000640 1 -- 172.21.15.67:0/2602184188 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fb0440f0b10 con 0x7fb058104a50 2024-03-28T16:03:33.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.600+0000 7fb05d400640 1 --2- 172.21.15.67:0/2602184188 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb0400a2790 0x7fb0400a4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:33.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.600+0000 7fb062bd9740 1 Processor -- start 2024-03-28T16:03:33.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.601+0000 7fb062bd9740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bfccb3a4b0 0x55bfccb446d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:33.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.601+0000 7fb062bd9740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bfccb3aea0 0x55bfccb44c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:33.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.601+0000 7fb062bd9740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bfccb40010 0x55bfccb4bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:33.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.601+0000 7fb062bd9740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bfccb42fc0 con 0x55bfccb3aea0 2024-03-28T16:03:33.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.601+0000 7fb062bd9740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bfcca57eb0 con 0x55bfccb40010 2024-03-28T16:03:33.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.601+0000 7fb062bd9740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bfccb43140 con 0x55bfccb3a4b0 2024-03-28T16:03:33.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.601+0000 7fb062bd9740 10 client.0 _ll_register_callbacks cb 0x55bfcca4c150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:03:33.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.601+0000 7fb05ca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bfccb3aea0 0x55bfccb44c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:33.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.601+0000 7fb05ca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bfccb3aea0 0x55bfccb44c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50762/0 (socket says 172.21.15.67:50762) 2024-03-28T16:03:33.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.601+0000 7fb05ca00640 1 -- 172.21.15.67:0/779246699 learned_addr learned my addr 172.21.15.67:0/779246699 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:03:33.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.601+0000 7fb05de00640 1 --2- 172.21.15.67:0/779246699 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bfccb40010 0x55bfccb4bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:33.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.601+0000 7fb05d400640 1 --2- 172.21.15.67:0/779246699 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bfccb3a4b0 0x55bfccb446d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:33.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.601+0000 7fb05d400640 1 --2- 172.21.15.67:0/2602184188 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb0400a2790 0x7fb0400a4c50 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x55bfccb32580 tx=0x7fb04c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:33.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.602+0000 7fb05ca00640 1 -- 172.21.15.67:0/779246699 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bfccb3a4b0 msgr2=0x55bfccb446d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:03:33.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.602+0000 7fb05ca00640 1 --2- 172.21.15.67:0/779246699 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bfccb3a4b0 0x55bfccb446d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.602+0000 7fb05ca00640 1 -- 172.21.15.67:0/779246699 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bfccb40010 msgr2=0x55bfccb4bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:03:33.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.602+0000 7fb05ca00640 1 --2- 172.21.15.67:0/779246699 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bfccb40010 0x55bfccb4bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.602+0000 7fb05ca00640 1 -- 172.21.15.67:0/779246699 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bfcc8d8f70 con 0x55bfccb3aea0 2024-03-28T16:03:33.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.602+0000 7fb05ca00640 1 --2- 172.21.15.67:0/779246699 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bfccb3aea0 0x55bfccb44c10 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fb0440f0e70 tx=0x7fb0440b9820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d16f43127984a4d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:33.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.602+0000 7fb03ea00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:03:33.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.602+0000 7fb03ea00640 1 -- 172.21.15.67:0/779246699 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0440bdbc0 con 0x55bfccb3aea0 2024-03-28T16:03:33.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.602+0000 7fb03ea00640 1 -- 172.21.15.67:0/779246699 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fb044003d50 con 0x55bfccb3aea0 2024-03-28T16:03:33.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.602+0000 7fb062bd9740 1 -- 172.21.15.67:0/779246699 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb050003e20 con 0x55bfccb3aea0 2024-03-28T16:03:33.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.603+0000 7fb03ea00640 1 -- 172.21.15.67:0/779246699 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb0440bd4c0 con 0x55bfccb3aea0 2024-03-28T16:03:33.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.603+0000 7fb062bd9740 10 client.15303 fetch_fsmap learned FSMap version 19 2024-03-28T16:03:33.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.603+0000 7fb062bd9740 1 -- 172.21.15.67:0/779246699 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55bfcc770170 con 0x55bfccb3aea0 2024-03-28T16:03:33.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.603+0000 7fb03ea00640 1 -- 172.21.15.67:0/779246699 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb0440c1610 con 0x55bfccb3aea0 2024-03-28T16:03:33.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.603+0000 7fb03ea00640 1 -- 172.21.15.67:0/779246699 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fb0440c1170 con 0x55bfccb3aea0 2024-03-28T16:03:33.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.603+0000 7fb062bd9740 10 client.15303 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:03:33.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.603+0000 7fb062bd9740 10 client.15303 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:03:33.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.603+0000 7fb062bd9740 20 client.15303 populate_metadata read hostname 'smithi067' 2024-03-28T16:03:33.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.603+0000 7fb062bd9740 1 --2- 172.21.15.67:0/779246699 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55bfccb504d0 0x55bfccb528f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:33.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.603+0000 7fb062bd9740 4 client.15303 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:03:33.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.603+0000 7fb062bd9740 1 -- 172.21.15.67:0/779246699 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb050003e20 con 0x55bfccb504d0 2024-03-28T16:03:33.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.604+0000 7fb05d400640 1 --2- 172.21.15.67:0/779246699 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55bfccb504d0 0x55bfccb528f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:33.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.605+0000 7fb05d400640 1 --2- 172.21.15.67:0/779246699 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55bfccb504d0 0x55bfccb528f0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b18aa4d844713363 server_cookie=413869ccb873841c in_seq=0 out_seq=0 2024-03-28T16:03:33.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.605+0000 7fb03ea00640 10 client.15303 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:03:33.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.606+0000 7fb03ea00640 1 -- 172.21.15.67:0/779246699 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x7fb050003e20 con 0x55bfccb504d0 2024-03-28T16:03:33.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.606+0000 7fb03ea00640 10 client.15303 handle_command_reply: tid=0 2024-03-28T16:03:33.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.606+0000 7fb062bd9740 1 client.15303 shutdown 2024-03-28T16:03:33.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.606+0000 7fb062bd9740 1 -- 172.21.15.67:0/779246699 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bfccb3aea0 msgr2=0x55bfccb44c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:33.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.606+0000 7fb062bd9740 1 --2- 172.21.15.67:0/779246699 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bfccb3aea0 0x55bfccb44c10 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fb0440f0e70 tx=0x7fb0440b9820 comp rx=0 tx=0).stop 2024-03-28T16:03:33.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.606+0000 7fb062bd9740 1 -- 172.21.15.67:0/779246699 shutdown_connections 2024-03-28T16:03:33.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.606+0000 7fb062bd9740 1 --2- 172.21.15.67:0/779246699 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55bfccb504d0 0x55bfccb528f0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.606+0000 7fb062bd9740 1 --2- 172.21.15.67:0/779246699 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bfccb40010 0x55bfccb4bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.606+0000 7fb062bd9740 1 --2- 172.21.15.67:0/779246699 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bfccb3aea0 0x55bfccb44c10 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.606+0000 7fb062bd9740 1 --2- 172.21.15.67:0/779246699 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bfccb3a4b0 0x55bfccb446d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.606+0000 7fb062bd9740 1 -- 172.21.15.67:0/779246699 >> 172.21.15.67:0/779246699 conn(0x55bfccb31610 msgr2=0x55bfccb30830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:33.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.606+0000 7fb03ea00640 0 client.15303 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:03:33.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.607+0000 7fb062bd9740 1 -- 172.21.15.67:0/779246699 shutdown_connections 2024-03-28T16:03:33.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.607+0000 7fb062bd9740 1 -- 172.21.15.67:0/779246699 wait complete. 2024-03-28T16:03:33.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.607+0000 7fb062bd9740 20 client.15303 trim_cache size 0 max 16384 2024-03-28T16:03:33.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.634+0000 7fb062bd9740 1 Processor -- start 2024-03-28T16:03:33.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.634+0000 7fb062bd9740 1 -- start start 2024-03-28T16:03:33.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.634+0000 7fb062bd9740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bfccb31610 0x55bfccb2ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:33.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.634+0000 7fb062bd9740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bfccb3a4b0 0x55bfccb4bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:33.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.634+0000 7fb062bd9740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bfccb3aea0 0x55bfccb304e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:33.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.634+0000 7fb062bd9740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0581a4790 con 0x55bfccb31610 2024-03-28T16:03:33.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.634+0000 7fb062bd9740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0581a6d50 con 0x55bfccb3aea0 2024-03-28T16:03:33.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.635+0000 7fb062bd9740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0581a4d90 con 0x55bfccb3a4b0 2024-03-28T16:03:33.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.635+0000 7fb05d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bfccb31610 0x55bfccb2ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:33.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.635+0000 7fb05d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bfccb31610 0x55bfccb2ffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50768/0 (socket says 172.21.15.67:50768) 2024-03-28T16:03:33.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.635+0000 7fb05d400640 1 -- 172.21.15.67:0/3722085755 learned_addr learned my addr 172.21.15.67:0/3722085755 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:03:33.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.635+0000 7fb05ca00640 1 --2- 172.21.15.67:0/3722085755 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bfccb3a4b0 0x55bfccb4bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:33.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.635+0000 7fb05de00640 1 --2- 172.21.15.67:0/3722085755 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bfccb3aea0 0x55bfccb304e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:33.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.635+0000 7fb05d400640 1 -- 172.21.15.67:0/3722085755 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bfccb3a4b0 msgr2=0x55bfccb4bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:03:33.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.635+0000 7fb05d400640 1 --2- 172.21.15.67:0/3722085755 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bfccb3a4b0 0x55bfccb4bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.635+0000 7fb05d400640 1 -- 172.21.15.67:0/3722085755 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bfccb3aea0 msgr2=0x55bfccb304e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:03:33.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.635+0000 7fb05d400640 1 --2- 172.21.15.67:0/3722085755 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bfccb3aea0 0x55bfccb304e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.635+0000 7fb05d400640 1 -- 172.21.15.67:0/3722085755 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bfccb52c60 con 0x55bfccb31610 2024-03-28T16:03:33.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.636+0000 7fb05d400640 1 --2- 172.21.15.67:0/3722085755 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bfccb31610 0x55bfccb2ffa0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x55bfccb53140 tx=0x7fb04c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81b23a33314f3c52 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:33.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.636+0000 7fb049c00640 1 -- 172.21.15.67:0/3722085755 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb04c005020 con 0x55bfccb31610 2024-03-28T16:03:33.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.636+0000 7fb049c00640 1 -- 172.21.15.67:0/3722085755 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb04c0047a0 con 0x55bfccb31610 2024-03-28T16:03:33.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.636+0000 7fb049c00640 1 -- 172.21.15.67:0/3722085755 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb04c003850 con 0x55bfccb31610 2024-03-28T16:03:33.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.637+0000 7fb062bd9740 1 -- 172.21.15.67:0/3722085755 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bfccb31610 msgr2=0x55bfccb2ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:33.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.637+0000 7fb062bd9740 1 --2- 172.21.15.67:0/3722085755 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bfccb31610 0x55bfccb2ffa0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x55bfccb53140 tx=0x7fb04c018000 comp rx=0 tx=0).stop 2024-03-28T16:03:33.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.637+0000 7fb062bd9740 1 -- 172.21.15.67:0/3722085755 shutdown_connections 2024-03-28T16:03:33.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.637+0000 7fb062bd9740 1 --2- 172.21.15.67:0/3722085755 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bfccb3aea0 0x55bfccb304e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.637+0000 7fb062bd9740 1 --2- 172.21.15.67:0/3722085755 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bfccb3a4b0 0x55bfccb4bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.637+0000 7fb062bd9740 1 --2- 172.21.15.67:0/3722085755 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bfccb31610 0x55bfccb2ffa0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.637+0000 7fb062bd9740 1 -- 172.21.15.67:0/3722085755 >> 172.21.15.67:0/3722085755 conn(0x55bfccb354a0 msgr2=0x55bfccb3c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:33.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.637+0000 7fb062bd9740 1 -- 172.21.15.67:0/3722085755 shutdown_connections 2024-03-28T16:03:33.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.637+0000 7fb062bd9740 1 -- 172.21.15.67:0/3722085755 wait complete. 2024-03-28T16:03:33.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.637+0000 7fb062bd9740 1 -- start start 2024-03-28T16:03:33.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.638+0000 7fb062bd9740 1 Processor -- start 2024-03-28T16:03:33.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.638+0000 7fb062bd9740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bfccb31610 0x55bfccb4a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:33.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.638+0000 7fb062bd9740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bfccb3a4b0 0x55bfccb4a820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:33.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.638+0000 7fb062bd9740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bfccb3aea0 0x55bfccb4ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:33.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.638+0000 7fb062bd9740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0581a53d0 con 0x55bfccb31610 2024-03-28T16:03:33.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.638+0000 7fb062bd9740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0581a66f0 con 0x55bfccb3a4b0 2024-03-28T16:03:33.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.638+0000 7fb062bd9740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0581a5a30 con 0x55bfccb3aea0 2024-03-28T16:03:33.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.638+0000 7fb062bd9740 10 client.0 _ll_register_callbacks cb 0x7fb0440b90a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:03:33.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.638+0000 7fb05d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bfccb31610 0x55bfccb4a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:33.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.638+0000 7fb05d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bfccb31610 0x55bfccb4a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50778/0 (socket says 172.21.15.67:50778) 2024-03-28T16:03:33.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.638+0000 7fb05d400640 1 -- 172.21.15.67:0/1884422362 learned_addr learned my addr 172.21.15.67:0/1884422362 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:03:33.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.639+0000 7fb05de00640 1 --2- 172.21.15.67:0/1884422362 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bfccb3aea0 0x55bfccb4ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:33.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.639+0000 7fb05ca00640 1 --2- 172.21.15.67:0/1884422362 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bfccb3a4b0 0x55bfccb4a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:33.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.639+0000 7fb05d400640 1 -- 172.21.15.67:0/1884422362 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bfccb3aea0 msgr2=0x55bfccb4ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:03:33.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.639+0000 7fb05d400640 1 --2- 172.21.15.67:0/1884422362 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bfccb3aea0 0x55bfccb4ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.639+0000 7fb05d400640 1 -- 172.21.15.67:0/1884422362 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bfccb3a4b0 msgr2=0x55bfccb4a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:03:33.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.639+0000 7fb05d400640 1 --2- 172.21.15.67:0/1884422362 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bfccb3a4b0 0x55bfccb4a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.639+0000 7fb05d400640 1 -- 172.21.15.67:0/1884422362 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bfccbabd60 con 0x55bfccb31610 2024-03-28T16:03:33.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.639+0000 7fb05d400640 1 --2- 172.21.15.67:0/1884422362 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bfccb31610 0x55bfccb4a2e0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fb04c012e60 tx=0x7fb04c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c87cd252d3b8617c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:33.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.639+0000 7fb03ea00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:03:33.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.640+0000 7fb03ea00640 1 -- 172.21.15.67:0/1884422362 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb04c03b020 con 0x55bfccb31610 2024-03-28T16:03:33.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.640+0000 7fb03ea00640 1 -- 172.21.15.67:0/1884422362 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fb04c004ca0 con 0x55bfccb31610 2024-03-28T16:03:33.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.640+0000 7fb062bd9740 1 -- 172.21.15.67:0/1884422362 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb04c0047a0 con 0x55bfccb31610 2024-03-28T16:03:33.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.640+0000 7fb03ea00640 1 -- 172.21.15.67:0/1884422362 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb04c004fe0 con 0x55bfccb31610 2024-03-28T16:03:33.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.640+0000 7fb062bd9740 10 client.15309 fetch_fsmap learned FSMap version 19 2024-03-28T16:03:33.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.640+0000 7fb062bd9740 1 -- 172.21.15.67:0/1884422362 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55bfccbacfb0 con 0x55bfccb31610 2024-03-28T16:03:33.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.640+0000 7fb03ea00640 1 -- 172.21.15.67:0/1884422362 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb04c0047a0 con 0x55bfccb31610 2024-03-28T16:03:33.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.640+0000 7fb03ea00640 1 -- 172.21.15.67:0/1884422362 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fb04c037470 con 0x55bfccb31610 2024-03-28T16:03:33.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.640+0000 7fb062bd9740 10 client.15309 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:03:33.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.640+0000 7fb062bd9740 10 client.15309 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:03:33.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.640+0000 7fb062bd9740 20 client.15309 populate_metadata read hostname 'smithi067' 2024-03-28T16:03:33.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.640+0000 7fb062bd9740 1 --2- 172.21.15.67:0/1884422362 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55bfccb40010 0x55bfccbaf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:33.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.640+0000 7fb062bd9740 4 client.15309 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T16:03:33.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.640+0000 7fb062bd9740 1 -- 172.21.15.67:0/1884422362 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55bfccaa93f0 con 0x55bfccb40010 2024-03-28T16:03:33.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.641+0000 7fb05ca00640 1 --2- 172.21.15.67:0/1884422362 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55bfccb40010 0x55bfccbaf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:33.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.642+0000 7fb05ca00640 1 --2- 172.21.15.67:0/1884422362 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55bfccb40010 0x55bfccbaf980 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba3f8b9b5ad1b9af server_cookie=73cdd41bcd0b6320 in_seq=0 out_seq=0 2024-03-28T16:03:33.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.642+0000 7fb03ea00640 10 client.15309 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:03:33.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.646+0000 7fb03ea00640 1 -- 172.21.15.67:0/1884422362 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55bfccaa93f0 con 0x55bfccb40010 2024-03-28T16:03:33.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.646+0000 7fb03ea00640 10 client.15309 handle_command_reply: tid=0 2024-03-28T16:03:33.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.646+0000 7fb062bd9740 1 client.15309 shutdown 2024-03-28T16:03:33.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.646+0000 7fb062bd9740 1 -- 172.21.15.67:0/1884422362 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bfccb31610 msgr2=0x55bfccb4a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:33.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.646+0000 7fb062bd9740 1 --2- 172.21.15.67:0/1884422362 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bfccb31610 0x55bfccb4a2e0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fb04c012e60 tx=0x7fb04c004ab0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.646+0000 7fb062bd9740 1 -- 172.21.15.67:0/1884422362 shutdown_connections 2024-03-28T16:03:33.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.646+0000 7fb062bd9740 1 --2- 172.21.15.67:0/1884422362 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55bfccb40010 0x55bfccbaf980 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.646+0000 7fb062bd9740 1 --2- 172.21.15.67:0/1884422362 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bfccb3aea0 0x55bfccb4ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.646+0000 7fb062bd9740 1 --2- 172.21.15.67:0/1884422362 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bfccb3a4b0 0x55bfccb4a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.646+0000 7fb03ea00640 0 client.15309 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:03:33.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.646+0000 7fb062bd9740 1 --2- 172.21.15.67:0/1884422362 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bfccb31610 0x55bfccb4a2e0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.646+0000 7fb062bd9740 1 -- 172.21.15.67:0/1884422362 >> 172.21.15.67:0/1884422362 conn(0x55bfccb354a0 msgr2=0x55bfccb57190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:33.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.646+0000 7fb062bd9740 1 -- 172.21.15.67:0/1884422362 shutdown_connections 2024-03-28T16:03:33.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.646+0000 7fb062bd9740 1 -- 172.21.15.67:0/1884422362 wait complete. 2024-03-28T16:03:33.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.646+0000 7fb062bd9740 20 client.15309 trim_cache size 0 max 16384 2024-03-28T16:03:33.648 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-28T16:03:33.648 INFO:teuthology.orchestra.run.smithi067.stdout: "return_code": 0, 2024-03-28T16:03:33.648 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_tag": "c9eba687-ebd9-4c8d-bdae-70c79824dbc4", 2024-03-28T16:03:33.648 INFO:teuthology.orchestra.run.smithi067.stdout: "mode": "asynchronous" 2024-03-28T16:03:33.648 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-28T16:03:33.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.647+0000 7fb05fe00640 1 -- 172.21.15.67:0/2602184188 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb0400a2790 msgr2=0x7fb0400a4c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:33.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.647+0000 7fb05fe00640 1 --2- 172.21.15.67:0/2602184188 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb0400a2790 0x7fb0400a4c50 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x55bfccb32580 tx=0x7fb04c005f90 comp rx=0 tx=0).stop 2024-03-28T16:03:33.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.647+0000 7fb05fe00640 1 -- 172.21.15.67:0/2602184188 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb058104a50 msgr2=0x7fb058197e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:33.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.647+0000 7fb05fe00640 1 --2- 172.21.15.67:0/2602184188 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb058104a50 0x7fb058197e60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb044000c00 tx=0x7fb044005e60 comp rx=0 tx=0).stop 2024-03-28T16:03:33.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.647+0000 7fb05fe00640 1 -- 172.21.15.67:0/2602184188 shutdown_connections 2024-03-28T16:03:33.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.647+0000 7fb05fe00640 1 --2- 172.21.15.67:0/2602184188 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb0400a2790 0x7fb0400a4c50 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.647+0000 7fb05fe00640 1 --2- 172.21.15.67:0/2602184188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb058107380 0x7fb05819eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.647+0000 7fb05fe00640 1 --2- 172.21.15.67:0/2602184188 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb058104a50 0x7fb058197e60 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.647+0000 7fb05fe00640 1 --2- 172.21.15.67:0/2602184188 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb058102120 0x7fb058197920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.647+0000 7fb05fe00640 1 -- 172.21.15.67:0/2602184188 >> 172.21.15.67:0/2602184188 conn(0x7fb0580fbb30 msgr2=0x7fb058102b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:33.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.647+0000 7fb05fe00640 1 -- 172.21.15.67:0/2602184188 shutdown_connections 2024-03-28T16:03:33.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.647+0000 7fb05fe00640 1 -- 172.21.15.67:0/2602184188 wait complete. 2024-03-28T16:03:33.666 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T16:03:33.667 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T16:03:33.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.934+0000 7f90b6a00640 1 -- 172.21.15.67:0/111367921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90b00740a0 msgr2=0x7f90b0074500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:33.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.934+0000 7f90b6a00640 1 --2- 172.21.15.67:0/111367921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90b00740a0 0x7f90b0074500 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f909c009860 tx=0x7f909c02bc00 comp rx=0 tx=0).stop 2024-03-28T16:03:33.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.934+0000 7f90b6a00640 1 -- 172.21.15.67:0/111367921 shutdown_connections 2024-03-28T16:03:33.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.934+0000 7f90b6a00640 1 --2- 172.21.15.67:0/111367921 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90b0109430 0x7f90b010b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.934+0000 7f90b6a00640 1 --2- 172.21.15.67:0/111367921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90b00740a0 0x7f90b0074500 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.934+0000 7f90b6a00640 1 --2- 172.21.15.67:0/111367921 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f90b0075680 0x7f90b0073ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.934+0000 7f90b6a00640 1 -- 172.21.15.67:0/111367921 >> 172.21.15.67:0/111367921 conn(0x7f90b00fcb10 msgr2=0x7f90b00fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:33.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.934+0000 7f90b6a00640 1 -- 172.21.15.67:0/111367921 shutdown_connections 2024-03-28T16:03:33.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.934+0000 7f90b6a00640 1 -- 172.21.15.67:0/111367921 wait complete. 2024-03-28T16:03:33.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.935+0000 7f90b6a00640 1 Processor -- start 2024-03-28T16:03:33.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.936+0000 7f90b6a00640 1 -- start start 2024-03-28T16:03:33.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.936+0000 7f90b6a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f90b00740a0 0x7f90b0199a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:33.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.936+0000 7f90b6a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90b0075680 0x7f90b0199f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:33.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.936+0000 7f90b6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90b0109430 0x7f90b01a0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:33.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.936+0000 7f90b6a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b010f2b0 con 0x7f90b0109430 2024-03-28T16:03:33.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.936+0000 7f90b6a00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b010f130 con 0x7f90b0075680 2024-03-28T16:03:33.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.936+0000 7f90b6a00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b010f430 con 0x7f90b00740a0 2024-03-28T16:03:33.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.936+0000 7f90b5600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f90b00740a0 0x7f90b0199a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:33.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.936+0000 7f90b6000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90b0109430 0x7f90b01a0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:33.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.936+0000 7f90b5600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f90b00740a0 0x7f90b0199a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.67:34196/0 (socket says 172.21.15.67:34196) 2024-03-28T16:03:33.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.936+0000 7f90b5600640 1 -- 172.21.15.67:0/586999171 learned_addr learned my addr 172.21.15.67:0/586999171 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:03:33.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.936+0000 7f90b4c00640 1 --2- 172.21.15.67:0/586999171 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90b0075680 0x7f90b0199f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:33.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.937+0000 7f90b5600640 1 -- 172.21.15.67:0/586999171 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90b0075680 msgr2=0x7f90b0199f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:33.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.937+0000 7f90b5600640 1 --2- 172.21.15.67:0/586999171 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90b0075680 0x7f90b0199f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.937+0000 7f90b5600640 1 -- 172.21.15.67:0/586999171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90b0109430 msgr2=0x7f90b01a0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:33.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.937+0000 7f90b5600640 1 --2- 172.21.15.67:0/586999171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90b0109430 0x7f90b01a0fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.937+0000 7f90b5600640 1 -- 172.21.15.67:0/586999171 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90b01a1620 con 0x7f90b00740a0 2024-03-28T16:03:33.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.937+0000 7f90b6000640 1 --2- 172.21.15.67:0/586999171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90b0109430 0x7f90b01a0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T16:03:33.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.937+0000 7f90b5600640 1 --2- 172.21.15.67:0/586999171 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f90b00740a0 0x7f90b0199a00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f90a400efb0 tx=0x7f90a400c740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:33.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.938+0000 7f90ae000640 1 -- 172.21.15.67:0/586999171 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a401b070 con 0x7f90b00740a0 2024-03-28T16:03:33.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.938+0000 7f90ae000640 1 -- 172.21.15.67:0/586999171 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f90a400ba20 con 0x7f90b00740a0 2024-03-28T16:03:33.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.938+0000 7f90b6a00640 1 -- 172.21.15.67:0/586999171 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90b01a1910 con 0x7f90b00740a0 2024-03-28T16:03:33.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.938+0000 7f90ae000640 1 -- 172.21.15.67:0/586999171 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a4016a80 con 0x7f90b00740a0 2024-03-28T16:03:33.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.938+0000 7f90b6a00640 1 -- 172.21.15.67:0/586999171 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90b01a1e30 con 0x7f90b00740a0 2024-03-28T16:03:33.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.942+0000 7f90ae000640 1 -- 172.21.15.67:0/586999171 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f90a401e070 con 0x7f90b00740a0 2024-03-28T16:03:33.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.943+0000 7f90bad91740 1 Processor -- start 2024-03-28T16:03:33.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.943+0000 7f90bad91740 1 -- start start 2024-03-28T16:03:33.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.943+0000 7f90ae000640 1 --2- 172.21.15.67:0/586999171 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f90980a2800 0x7f90980a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:33.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.943+0000 7f90ae000640 1 -- 172.21.15.67:0/586999171 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f90a40ce9e0 con 0x7f90b00740a0 2024-03-28T16:03:33.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.943+0000 7f90bad91740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5567bb660050 0x5567bb660470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:33.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.943+0000 7f90bad91740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5567bb660a40 0x5567bb665540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:33.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.943+0000 7f90b6000640 1 --2- 172.21.15.67:0/586999171 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f90980a2800 0x7f90980a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:33.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.943+0000 7f90bad91740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5567bb665bb0 0x5567bb668040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:33.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.943+0000 7f90bad91740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5567bb5c5250 con 0x5567bb665bb0 2024-03-28T16:03:33.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.943+0000 7f90bad91740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5567bb56da30 con 0x5567bb660050 2024-03-28T16:03:33.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.943+0000 7f90bad91740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x5567bb5c12f0 con 0x5567bb660a40 2024-03-28T16:03:33.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.944+0000 7f90b6000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5567bb665bb0 0x5567bb668040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:33.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.944+0000 7f90b4c00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5567bb660a40 0x5567bb665540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:33.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.944+0000 7f90b6000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5567bb665bb0 0x5567bb668040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50806/0 (socket says 172.21.15.67:50806) 2024-03-28T16:03:33.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.944+0000 7f90b6000640 1 -- 172.21.15.67:0/2697565085 learned_addr learned my addr 172.21.15.67:0/2697565085 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:03:33.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.944+0000 7f90b5600640 1 --2- 172.21.15.67:0/2697565085 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5567bb660050 0x5567bb660470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:33.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.944+0000 7f90b4c00640 1 -- 172.21.15.67:0/2697565085 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5567bb660050 msgr2=0x5567bb660470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:03:33.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.944+0000 7f90b4c00640 1 --2- 172.21.15.67:0/2697565085 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5567bb660050 0x5567bb660470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.944+0000 7f90b4c00640 1 -- 172.21.15.67:0/2697565085 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5567bb665bb0 msgr2=0x5567bb668040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:03:33.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.944+0000 7f90b4c00640 1 --2- 172.21.15.67:0/2697565085 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5567bb665bb0 0x5567bb668040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.944+0000 7f90b4c00640 1 -- 172.21.15.67:0/2697565085 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5567bb558690 con 0x5567bb660a40 2024-03-28T16:03:33.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.944+0000 7f90b6000640 1 --2- 172.21.15.67:0/2697565085 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5567bb665bb0 0x5567bb668040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T16:03:33.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.945+0000 7f90b6000640 1 --2- 172.21.15.67:0/586999171 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f90980a2800 0x7f90980a4cc0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f90a8013d90 tx=0x7f90a8009f20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:33.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.945+0000 7f90b4c00640 1 --2- 172.21.15.67:0/2697565085 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5567bb660a40 0x5567bb665540 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f909c037040 tx=0x7f909c009490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=861641af260ddeab server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:33.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.945+0000 7f90a2600640 1 -- 172.21.15.67:0/2697565085 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f909c043020 con 0x5567bb660a40 2024-03-28T16:03:33.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.945+0000 7f90a2600640 1 -- 172.21.15.67:0/2697565085 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f909c0344c0 con 0x5567bb660a40 2024-03-28T16:03:33.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.945+0000 7f90a2600640 1 -- 172.21.15.67:0/2697565085 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f909c039040 con 0x5567bb660a40 2024-03-28T16:03:33.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.946+0000 7f90bad91740 1 -- 172.21.15.67:0/2697565085 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5567bb660a40 msgr2=0x5567bb665540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:33.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.946+0000 7f90bad91740 1 --2- 172.21.15.67:0/2697565085 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5567bb660a40 0x5567bb665540 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f909c037040 tx=0x7f909c009490 comp rx=0 tx=0).stop 2024-03-28T16:03:33.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.946+0000 7f90bad91740 1 -- 172.21.15.67:0/2697565085 shutdown_connections 2024-03-28T16:03:33.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.946+0000 7f90bad91740 1 --2- 172.21.15.67:0/2697565085 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5567bb665bb0 0x5567bb668040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.946+0000 7f90bad91740 1 --2- 172.21.15.67:0/2697565085 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5567bb660a40 0x5567bb665540 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.946+0000 7f90bad91740 1 --2- 172.21.15.67:0/2697565085 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5567bb660050 0x5567bb660470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.946+0000 7f90bad91740 1 -- 172.21.15.67:0/2697565085 >> 172.21.15.67:0/2697565085 conn(0x5567bb628200 msgr2=0x5567bb65b9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:33.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.946+0000 7f90bad91740 1 -- 172.21.15.67:0/2697565085 shutdown_connections 2024-03-28T16:03:33.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.946+0000 7f90bad91740 1 -- 172.21.15.67:0/2697565085 wait complete. 2024-03-28T16:03:33.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.947+0000 7f90bad91740 1 -- start start 2024-03-28T16:03:33.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.947+0000 7f90bad91740 1 Processor -- start 2024-03-28T16:03:33.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.948+0000 7f90bad91740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5567bb660050 0x5567bb66a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:33.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.948+0000 7f90bad91740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5567bb660a40 0x5567bb66a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:33.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.948+0000 7f90bad91740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5567bb665bb0 0x5567bb671530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:33.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.948+0000 7f90bad91740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5567bb668b00 con 0x5567bb665bb0 2024-03-28T16:03:33.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.948+0000 7f90bad91740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5567bb580e80 con 0x5567bb660a40 2024-03-28T16:03:33.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.948+0000 7f90bad91740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x5567bb668c80 con 0x5567bb660050 2024-03-28T16:03:33.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.948+0000 7f90bad91740 10 client.0 _ll_register_callbacks cb 0x5567bb593130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:03:33.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.948+0000 7f90b5600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5567bb660050 0x5567bb66a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:33.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.948+0000 7f90b6000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5567bb665bb0 0x5567bb671530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:33.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.948+0000 7f90b5600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5567bb660050 0x5567bb66a020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.67:34202/0 (socket says 172.21.15.67:34202) 2024-03-28T16:03:33.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.948+0000 7f90b5600640 1 -- 172.21.15.67:0/3612682871 learned_addr learned my addr 172.21.15.67:0/3612682871 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:03:33.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.948+0000 7f90b4c00640 1 --2- 172.21.15.67:0/3612682871 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5567bb660a40 0x5567bb66a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:33.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.949+0000 7f90b5600640 1 -- 172.21.15.67:0/3612682871 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5567bb660a40 msgr2=0x5567bb66a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:03:33.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.949+0000 7f90b5600640 1 --2- 172.21.15.67:0/3612682871 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5567bb660a40 0x5567bb66a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.949+0000 7f90b5600640 1 -- 172.21.15.67:0/3612682871 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5567bb665bb0 msgr2=0x5567bb671530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:03:33.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.949+0000 7f90b5600640 1 --2- 172.21.15.67:0/3612682871 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5567bb665bb0 0x5567bb671530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.949+0000 7f90b5600640 1 -- 172.21.15.67:0/3612682871 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5567bb3a37d0 con 0x5567bb660050 2024-03-28T16:03:33.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.949+0000 7f90b5600640 1 --2- 172.21.15.67:0/3612682871 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5567bb660050 0x5567bb66a020 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f90a4097db0 tx=0x7f90a409b520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=187344788d5bcf10 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:33.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.949+0000 7f9096a00640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-03-28T16:03:33.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.950+0000 7f9096a00640 1 -- 172.21.15.67:0/3612682871 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a40b4060 con 0x5567bb660050 2024-03-28T16:03:33.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.950+0000 7f9096a00640 1 -- 172.21.15.67:0/3612682871 <== mon.1 v2:172.21.15.183:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f90a40a3640 con 0x5567bb660050 2024-03-28T16:03:33.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.950+0000 7f90bad91740 1 -- 172.21.15.67:0/3612682871 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f909c0344c0 con 0x5567bb660050 2024-03-28T16:03:33.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.950+0000 7f9096a00640 1 -- 172.21.15.67:0/3612682871 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f90a40b9050 con 0x5567bb660050 2024-03-28T16:03:33.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.951+0000 7f90bad91740 10 client.25027 fetch_fsmap learned FSMap version 19 2024-03-28T16:03:33.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.951+0000 7f90bad91740 1 -- 172.21.15.67:0/3612682871 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5567bb2d93f0 con 0x5567bb660050 2024-03-28T16:03:33.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.951+0000 7f9096a00640 1 -- 172.21.15.67:0/3612682871 <== mon.1 v2:172.21.15.183:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f909c0344c0 con 0x5567bb660050 2024-03-28T16:03:33.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.951+0000 7f9096a00640 1 -- 172.21.15.67:0/3612682871 <== mon.1 v2:172.21.15.183:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f90a400b040 con 0x5567bb660050 2024-03-28T16:03:33.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.951+0000 7f90bad91740 10 client.25027 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:03:33.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.951+0000 7f90bad91740 10 client.25027 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:03:33.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.951+0000 7f90bad91740 20 client.25027 populate_metadata read hostname 'smithi067' 2024-03-28T16:03:33.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.951+0000 7f90bad91740 1 --2- 172.21.15.67:0/3612682871 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5567bb675e60 0x5567bb6782d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:33.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.951+0000 7f90bad91740 4 client.25027 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:03:33.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.951+0000 7f90bad91740 1 -- 172.21.15.67:0/3612682871 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5567bb5d2790 con 0x5567bb675e60 2024-03-28T16:03:33.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.952+0000 7f90b4c00640 1 --2- 172.21.15.67:0/3612682871 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5567bb675e60 0x5567bb6782d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:33.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.953+0000 7f90b4c00640 1 --2- 172.21.15.67:0/3612682871 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5567bb675e60 0x5567bb6782d0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=22e0610ce6861a2f server_cookie=dd4b5791859a380a in_seq=0 out_seq=0 2024-03-28T16:03:33.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.953+0000 7f9096a00640 10 client.25027 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:03:33.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.954+0000 7f9096a00640 1 -- 172.21.15.67:0/3612682871 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5567bb5d2790 con 0x5567bb675e60 2024-03-28T16:03:33.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.954+0000 7f9096a00640 10 client.25027 handle_command_reply: tid=0 2024-03-28T16:03:33.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.955+0000 7f90bad91740 1 client.25027 shutdown 2024-03-28T16:03:33.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.955+0000 7f90bad91740 1 -- 172.21.15.67:0/3612682871 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5567bb660050 msgr2=0x5567bb66a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:33.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.955+0000 7f90bad91740 1 --2- 172.21.15.67:0/3612682871 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5567bb660050 0x5567bb66a020 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f90a4097db0 tx=0x7f90a409b520 comp rx=0 tx=0).stop 2024-03-28T16:03:33.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.955+0000 7f90bad91740 1 -- 172.21.15.67:0/3612682871 shutdown_connections 2024-03-28T16:03:33.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.955+0000 7f90bad91740 1 --2- 172.21.15.67:0/3612682871 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5567bb675e60 0x5567bb6782d0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.955+0000 7f90bad91740 1 --2- 172.21.15.67:0/3612682871 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5567bb665bb0 0x5567bb671530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.955+0000 7f90bad91740 1 --2- 172.21.15.67:0/3612682871 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5567bb660a40 0x5567bb66a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.955+0000 7f9096a00640 0 client.25027 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:03:33.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.955+0000 7f90bad91740 1 --2- 172.21.15.67:0/3612682871 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5567bb660050 0x5567bb66a020 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.955+0000 7f90bad91740 1 -- 172.21.15.67:0/3612682871 >> 172.21.15.67:0/3612682871 conn(0x5567bb628200 msgr2=0x5567bb65abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:33.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.955+0000 7f90bad91740 1 -- 172.21.15.67:0/3612682871 shutdown_connections 2024-03-28T16:03:33.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.955+0000 7f90bad91740 1 -- 172.21.15.67:0/3612682871 wait complete. 2024-03-28T16:03:33.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.955+0000 7f90bad91740 20 client.25027 trim_cache size 0 max 16384 2024-03-28T16:03:33.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.973+0000 7f90bad91740 1 Processor -- start 2024-03-28T16:03:33.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.973+0000 7f90bad91740 1 -- start start 2024-03-28T16:03:33.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.974+0000 7f90bad91740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5567bb660050 0x5567bb65ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:33.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.974+0000 7f90bad91740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5567bb660a40 0x5567bb6714f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:33.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.974+0000 7f90bad91740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5567bb665bb0 0x5567bb65f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:33.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.974+0000 7f90bad91740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b01a8ad0 con 0x5567bb660a40 2024-03-28T16:03:33.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.974+0000 7f90bad91740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b01ab090 con 0x5567bb665bb0 2024-03-28T16:03:33.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.974+0000 7f90bad91740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b01a90d0 con 0x5567bb660050 2024-03-28T16:03:33.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.974+0000 7f90b4c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5567bb660a40 0x5567bb6714f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:33.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.974+0000 7f90b4c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5567bb660a40 0x5567bb6714f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50830/0 (socket says 172.21.15.67:50830) 2024-03-28T16:03:33.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.974+0000 7f90b4c00640 1 -- 172.21.15.67:0/1375994805 learned_addr learned my addr 172.21.15.67:0/1375994805 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:03:33.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.974+0000 7f90b6000640 1 --2- 172.21.15.67:0/1375994805 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5567bb665bb0 0x5567bb65f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:33.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.974+0000 7f90b5600640 1 --2- 172.21.15.67:0/1375994805 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5567bb660050 0x5567bb65ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:33.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.975+0000 7f90b4c00640 1 -- 172.21.15.67:0/1375994805 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5567bb660050 msgr2=0x5567bb65ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:03:33.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.975+0000 7f90b4c00640 1 --2- 172.21.15.67:0/1375994805 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5567bb660050 0x5567bb65ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.975+0000 7f90b4c00640 1 -- 172.21.15.67:0/1375994805 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5567bb665bb0 msgr2=0x5567bb65f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:03:33.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.975+0000 7f90b4c00640 1 --2- 172.21.15.67:0/1375994805 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5567bb665bb0 0x5567bb65f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.975+0000 7f90b4c00640 1 -- 172.21.15.67:0/1375994805 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5567bb2990f0 con 0x5567bb660a40 2024-03-28T16:03:33.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.975+0000 7f90b6000640 1 --2- 172.21.15.67:0/1375994805 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5567bb665bb0 0x5567bb65f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T16:03:33.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.975+0000 7f90b4c00640 1 --2- 172.21.15.67:0/1375994805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5567bb660a40 0x5567bb6714f0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f909c02c210 tx=0x7f909c02bc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59e544e54e88bda6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:33.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.975+0000 7f90a1c00640 1 -- 172.21.15.67:0/1375994805 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f909c043040 con 0x5567bb660a40 2024-03-28T16:03:33.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.975+0000 7f90a1c00640 1 -- 172.21.15.67:0/1375994805 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f909c034640 con 0x5567bb660a40 2024-03-28T16:03:33.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.975+0000 7f90a1c00640 1 -- 172.21.15.67:0/1375994805 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f909c039040 con 0x5567bb660a40 2024-03-28T16:03:33.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.976+0000 7f90bad91740 1 -- 172.21.15.67:0/1375994805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5567bb660a40 msgr2=0x5567bb6714f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:33.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.976+0000 7f90bad91740 1 --2- 172.21.15.67:0/1375994805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5567bb660a40 0x5567bb6714f0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f909c02c210 tx=0x7f909c02bc00 comp rx=0 tx=0).stop 2024-03-28T16:03:33.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.976+0000 7f90bad91740 1 -- 172.21.15.67:0/1375994805 shutdown_connections 2024-03-28T16:03:33.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.976+0000 7f90bad91740 1 --2- 172.21.15.67:0/1375994805 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5567bb665bb0 0x5567bb65f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.976+0000 7f90bad91740 1 --2- 172.21.15.67:0/1375994805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5567bb660a40 0x5567bb6714f0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.976+0000 7f90bad91740 1 --2- 172.21.15.67:0/1375994805 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5567bb660050 0x5567bb65ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.976+0000 7f90bad91740 1 -- 172.21.15.67:0/1375994805 >> 172.21.15.67:0/1375994805 conn(0x5567bb628200 msgr2=0x5567bb661d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:33.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.976+0000 7f90bad91740 1 -- 172.21.15.67:0/1375994805 shutdown_connections 2024-03-28T16:03:33.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.976+0000 7f90bad91740 1 -- 172.21.15.67:0/1375994805 wait complete. 2024-03-28T16:03:33.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.977+0000 7f90bad91740 1 -- start start 2024-03-28T16:03:33.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.977+0000 7f90bad91740 1 Processor -- start 2024-03-28T16:03:33.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.977+0000 7f90bad91740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5567bb660050 0x5567bb66ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:33.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.977+0000 7f90bad91740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5567bb660a40 0x5567bb66f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:33.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.977+0000 7f90bad91740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5567bb665bb0 0x5567bb6702e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:33.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.977+0000 7f90bad91740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b01a9710 con 0x5567bb665bb0 2024-03-28T16:03:33.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.977+0000 7f90bad91740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b01aaa30 con 0x5567bb660a40 2024-03-28T16:03:33.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.977+0000 7f90bad91740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b01a9d70 con 0x5567bb660050 2024-03-28T16:03:33.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.977+0000 7f90bad91740 10 client.0 _ll_register_callbacks cb 0x7f90a4009a90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:03:33.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.978+0000 7f90b6000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5567bb665bb0 0x5567bb6702e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:33.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.978+0000 7f90b6000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5567bb665bb0 0x5567bb6702e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50846/0 (socket says 172.21.15.67:50846) 2024-03-28T16:03:33.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.978+0000 7f90b6000640 1 -- 172.21.15.67:0/2989817812 learned_addr learned my addr 172.21.15.67:0/2989817812 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:03:33.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.978+0000 7f90b4c00640 1 --2- 172.21.15.67:0/2989817812 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5567bb660a40 0x5567bb66f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:33.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.978+0000 7f90b5600640 1 --2- 172.21.15.67:0/2989817812 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5567bb660050 0x5567bb66ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:33.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.978+0000 7f90b6000640 1 -- 172.21.15.67:0/2989817812 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5567bb660050 msgr2=0x5567bb66ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:03:33.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.978+0000 7f90b6000640 1 --2- 172.21.15.67:0/2989817812 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5567bb660050 0x5567bb66ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.978+0000 7f90b6000640 1 -- 172.21.15.67:0/2989817812 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5567bb660a40 msgr2=0x5567bb66f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:03:33.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.978+0000 7f90b6000640 1 --2- 172.21.15.67:0/2989817812 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5567bb660a40 0x5567bb66f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.978+0000 7f90b6000640 1 -- 172.21.15.67:0/2989817812 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5567bb6708d0 con 0x5567bb665bb0 2024-03-28T16:03:33.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.979+0000 7f90b6000640 1 --2- 172.21.15.67:0/2989817812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5567bb665bb0 0x5567bb6702e0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f90a80132c0 tx=0x7f90a8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38d52a04821f3065 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:33.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.979+0000 7f9096a00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:03:33.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.979+0000 7f9096a00640 1 -- 172.21.15.67:0/2989817812 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a8030060 con 0x5567bb665bb0 2024-03-28T16:03:33.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.979+0000 7f9096a00640 1 -- 172.21.15.67:0/2989817812 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f90a801c040 con 0x5567bb665bb0 2024-03-28T16:03:33.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.979+0000 7f90bad91740 1 -- 172.21.15.67:0/2989817812 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f909c034640 con 0x5567bb665bb0 2024-03-28T16:03:33.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.980+0000 7f9096a00640 1 -- 172.21.15.67:0/2989817812 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f90a80132f0 con 0x5567bb665bb0 2024-03-28T16:03:33.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.980+0000 7f90bad91740 10 client.15327 fetch_fsmap learned FSMap version 19 2024-03-28T16:03:33.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.980+0000 7f90bad91740 1 -- 172.21.15.67:0/2989817812 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5567bb6d18c0 con 0x5567bb665bb0 2024-03-28T16:03:33.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.980+0000 7f9096a00640 1 -- 172.21.15.67:0/2989817812 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f909c034640 con 0x5567bb665bb0 2024-03-28T16:03:33.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.980+0000 7f9096a00640 1 -- 172.21.15.67:0/2989817812 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f90a801f040 con 0x5567bb665bb0 2024-03-28T16:03:33.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.980+0000 7f90bad91740 10 client.15327 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:03:33.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.980+0000 7f90bad91740 10 client.15327 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:03:33.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.980+0000 7f90bad91740 20 client.15327 populate_metadata read hostname 'smithi067' 2024-03-28T16:03:33.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.980+0000 7f90bad91740 1 --2- 172.21.15.67:0/2989817812 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5567bb6d22d0 0x5567bb6d46f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:33.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.980+0000 7f90bad91740 4 client.15327 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T16:03:33.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.980+0000 7f90bad91740 1 -- 172.21.15.67:0/2989817812 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5567bb5ccae0 con 0x5567bb6d22d0 2024-03-28T16:03:33.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.981+0000 7f90b5600640 1 --2- 172.21.15.67:0/2989817812 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5567bb6d22d0 0x5567bb6d46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:33.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.982+0000 7f90b5600640 1 --2- 172.21.15.67:0/2989817812 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5567bb6d22d0 0x5567bb6d46f0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e8d0eaf834c7b68 server_cookie=a781eacae6b678b1 in_seq=0 out_seq=0 2024-03-28T16:03:33.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.982+0000 7f9096a00640 10 client.15327 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:03:33.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.982+0000 7f9096a00640 1 -- 172.21.15.67:0/2989817812 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x5567bb5ccae0 con 0x5567bb6d22d0 2024-03-28T16:03:33.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.982+0000 7f9096a00640 10 client.15327 handle_command_reply: tid=0 2024-03-28T16:03:33.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.982+0000 7f90bad91740 1 client.15327 shutdown 2024-03-28T16:03:33.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.982+0000 7f90bad91740 1 -- 172.21.15.67:0/2989817812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5567bb665bb0 msgr2=0x5567bb6702e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:33.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.983+0000 7f90bad91740 1 --2- 172.21.15.67:0/2989817812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5567bb665bb0 0x5567bb6702e0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f90a80132c0 tx=0x7f90a8004730 comp rx=0 tx=0).stop 2024-03-28T16:03:33.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.983+0000 7f90bad91740 1 -- 172.21.15.67:0/2989817812 shutdown_connections 2024-03-28T16:03:33.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.983+0000 7f90bad91740 1 --2- 172.21.15.67:0/2989817812 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5567bb6d22d0 0x5567bb6d46f0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.983+0000 7f90bad91740 1 --2- 172.21.15.67:0/2989817812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5567bb665bb0 0x5567bb6702e0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.983+0000 7f90bad91740 1 --2- 172.21.15.67:0/2989817812 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5567bb660a40 0x5567bb66f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.983+0000 7f90bad91740 1 --2- 172.21.15.67:0/2989817812 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5567bb660050 0x5567bb66ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.983+0000 7f90bad91740 1 -- 172.21.15.67:0/2989817812 >> 172.21.15.67:0/2989817812 conn(0x5567bb628200 msgr2=0x5567bb65a330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:33.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.983+0000 7f9096a00640 0 client.15327 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:03:33.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.983+0000 7f90bad91740 1 -- 172.21.15.67:0/2989817812 shutdown_connections 2024-03-28T16:03:33.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.983+0000 7f90bad91740 1 -- 172.21.15.67:0/2989817812 wait complete. 2024-03-28T16:03:33.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.983+0000 7f90bad91740 20 client.15327 trim_cache size 0 max 16384 2024-03-28T16:03:33.986 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-28T16:03:33.986 INFO:teuthology.orchestra.run.smithi067.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T16:03:33.986 INFO:teuthology.orchestra.run.smithi067.stdout: "scrubs": { 2024-03-28T16:03:33.986 INFO:teuthology.orchestra.run.smithi067.stdout: "c9eba687-ebd9-4c8d-bdae-70c79824dbc4": { 2024-03-28T16:03:33.986 INFO:teuthology.orchestra.run.smithi067.stdout: "path": "/", 2024-03-28T16:03:33.986 INFO:teuthology.orchestra.run.smithi067.stdout: "tag": "c9eba687-ebd9-4c8d-bdae-70c79824dbc4", 2024-03-28T16:03:33.986 INFO:teuthology.orchestra.run.smithi067.stdout: "options": "recursive,force" 2024-03-28T16:03:33.986 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:03:33.986 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:03:33.986 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-28T16:03:33.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.984+0000 7f90b6a00640 1 -- 172.21.15.67:0/586999171 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f90980a2800 msgr2=0x7f90980a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:33.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.984+0000 7f90b6a00640 1 --2- 172.21.15.67:0/586999171 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f90980a2800 0x7f90980a4cc0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f90a8013d90 tx=0x7f90a8009f20 comp rx=0 tx=0).stop 2024-03-28T16:03:33.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.984+0000 7f90b6a00640 1 -- 172.21.15.67:0/586999171 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f90b00740a0 msgr2=0x7f90b0199a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:33.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.984+0000 7f90b6a00640 1 --2- 172.21.15.67:0/586999171 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f90b00740a0 0x7f90b0199a00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f90a400efb0 tx=0x7f90a400c740 comp rx=0 tx=0).stop 2024-03-28T16:03:33.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.984+0000 7f90b6a00640 1 -- 172.21.15.67:0/586999171 shutdown_connections 2024-03-28T16:03:33.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.984+0000 7f90b6a00640 1 --2- 172.21.15.67:0/586999171 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f90980a2800 0x7f90980a4cc0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.984+0000 7f90b6a00640 1 --2- 172.21.15.67:0/586999171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90b0109430 0x7f90b01a0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.984+0000 7f90b6a00640 1 --2- 172.21.15.67:0/586999171 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90b0075680 0x7f90b0199f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.984+0000 7f90b6a00640 1 --2- 172.21.15.67:0/586999171 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f90b00740a0 0x7f90b0199a00 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:33.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.984+0000 7f90b6a00640 1 -- 172.21.15.67:0/586999171 >> 172.21.15.67:0/586999171 conn(0x7f90b00fcb10 msgr2=0x7f90b01053c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:33.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.984+0000 7f90b6a00640 1 -- 172.21.15.67:0/586999171 shutdown_connections 2024-03-28T16:03:33.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:03:33.984+0000 7f90b6a00640 1 -- 172.21.15.67:0/586999171 wait complete. 2024-03-28T16:03:34.000 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:33 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/3462454923' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:03:34.000 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:33 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/1457962225' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:03:34.000 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:33 smithi067 ceph-mon[31707]: scrub queued for path: / 2024-03-28T16:03:34.001 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:33 smithi067 ceph-mon[31707]: scrub summary: idle+waiting paths [/] 2024-03-28T16:03:34.001 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:33 smithi067 ceph-mon[31707]: scrub summary: active paths [/] 2024-03-28T16:03:34.001 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:33 smithi067 ceph-mon[31707]: scrub summary: idle+waiting paths [/] 2024-03-28T16:03:34.001 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:33 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:03:34.001 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:33 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:03:34.001 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:33 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:03:34.001 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:33 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:03:34.002 INFO:tasks.cephfs.filesystem:scrub status for tag:c9eba687-ebd9-4c8d-bdae-70c79824dbc4 - {'path': '/', 'tag': 'c9eba687-ebd9-4c8d-bdae-70c79824dbc4', 'options': 'recursive,force'} 2024-03-28T16:03:34.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:33 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/3462454923' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:03:34.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:33 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/1457962225' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:03:34.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:33 smithi183 ceph-mon[35580]: scrub queued for path: / 2024-03-28T16:03:34.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:33 smithi183 ceph-mon[35580]: scrub summary: idle+waiting paths [/] 2024-03-28T16:03:34.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:33 smithi183 ceph-mon[35580]: scrub summary: active paths [/] 2024-03-28T16:03:34.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:33 smithi183 ceph-mon[35580]: scrub summary: idle+waiting paths [/] 2024-03-28T16:03:34.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:33 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:03:34.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:33 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:03:34.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:33 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:03:34.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:33 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:03:34.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:33 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/3462454923' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:03:34.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:33 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/1457962225' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:03:34.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:33 smithi150 ceph-mon[34825]: scrub queued for path: / 2024-03-28T16:03:34.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:33 smithi150 ceph-mon[34825]: scrub summary: idle+waiting paths [/] 2024-03-28T16:03:34.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:33 smithi150 ceph-mon[34825]: scrub summary: active paths [/] 2024-03-28T16:03:34.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:33 smithi150 ceph-mon[34825]: scrub summary: idle+waiting paths [/] 2024-03-28T16:03:34.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:33 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:03:34.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:33 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:03:34.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:33 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:03:34.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:33 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:03:35.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:34 smithi150 ceph-mon[34825]: pgmap v439: 97 pgs: 97 active+clean; 643 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 682 B/s wr, 2 op/s 2024-03-28T16:03:35.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:34 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:03:35.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:34 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:03:35.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:34 smithi067 ceph-mon[31707]: pgmap v439: 97 pgs: 97 active+clean; 643 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 682 B/s wr, 2 op/s 2024-03-28T16:03:35.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:34 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:03:35.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:34 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:03:35.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:34 smithi183 ceph-mon[35580]: pgmap v439: 97 pgs: 97 active+clean; 643 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 682 B/s wr, 2 op/s 2024-03-28T16:03:35.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:34 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:03:35.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:34 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:03:37.123 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:36 smithi067 ceph-mon[31707]: pgmap v440: 97 pgs: 97 active+clean; 648 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-28T16:03:37.123 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:36 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:03:37.123 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:36 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T16:03:37.124 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:36 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:03:37.124 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:36 smithi067 ceph-mon[31707]: scrub summary: idle 2024-03-28T16:03:37.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:36 smithi150 ceph-mon[34825]: pgmap v440: 97 pgs: 97 active+clean; 648 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-28T16:03:38.200 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:36 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:03:38.200 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:36 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T16:03:38.200 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:36 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:03:38.200 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:36 smithi150 ceph-mon[34825]: scrub summary: idle 2024-03-28T16:03:38.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:36 smithi183 ceph-mon[35580]: pgmap v440: 97 pgs: 97 active+clean; 648 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-28T16:03:38.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:36 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:03:38.202 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:36 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T16:03:38.202 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:36 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:03:38.202 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:36 smithi183 ceph-mon[35580]: scrub summary: idle 2024-03-28T16:03:39.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:38 smithi150 ceph-mon[34825]: pgmap v441: 97 pgs: 97 active+clean; 648 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-28T16:03:39.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:38 smithi067 ceph-mon[31707]: pgmap v441: 97 pgs: 97 active+clean; 648 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-28T16:03:39.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:38 smithi183 ceph-mon[35580]: pgmap v441: 97 pgs: 97 active+clean; 648 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-28T16:03:41.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:40 smithi150 ceph-mon[34825]: pgmap v442: 97 pgs: 97 active+clean; 648 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 1.1 KiB/s wr, 2 op/s 2024-03-28T16:03:41.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:40 smithi067 ceph-mon[31707]: pgmap v442: 97 pgs: 97 active+clean; 648 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 1.1 KiB/s wr, 2 op/s 2024-03-28T16:03:41.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:40 smithi183 ceph-mon[35580]: pgmap v442: 97 pgs: 97 active+clean; 648 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 1.1 KiB/s wr, 2 op/s 2024-03-28T16:03:43.306 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:42 smithi150 ceph-mon[34825]: pgmap v443: 97 pgs: 97 active+clean; 648 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-28T16:03:43.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:42 smithi067 ceph-mon[31707]: pgmap v443: 97 pgs: 97 active+clean; 648 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-28T16:03:43.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:42 smithi183 ceph-mon[35580]: pgmap v443: 97 pgs: 97 active+clean; 648 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-28T16:03:45.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:44 smithi150 ceph-mon[34825]: pgmap v444: 97 pgs: 97 active+clean; 648 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 682 B/s wr, 2 op/s 2024-03-28T16:03:45.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:44 smithi067 ceph-mon[31707]: pgmap v444: 97 pgs: 97 active+clean; 648 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 682 B/s wr, 2 op/s 2024-03-28T16:03:45.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:44 smithi183 ceph-mon[35580]: pgmap v444: 97 pgs: 97 active+clean; 648 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 682 B/s wr, 2 op/s 2024-03-28T16:03:47.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:46 smithi150 ceph-mon[34825]: pgmap v445: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-28T16:03:47.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:46 smithi067 ceph-mon[31707]: pgmap v445: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-28T16:03:47.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:46 smithi183 ceph-mon[35580]: pgmap v445: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-28T16:03:49.217 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:48 smithi067 ceph-mon[31707]: pgmap v446: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-28T16:03:49.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:48 smithi150 ceph-mon[34825]: pgmap v446: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-28T16:03:49.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:48 smithi183 ceph-mon[35580]: pgmap v446: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-28T16:03:51.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:50 smithi150 ceph-mon[34825]: pgmap v447: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-28T16:03:51.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:50 smithi067 ceph-mon[31707]: pgmap v447: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-28T16:03:51.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:50 smithi183 ceph-mon[35580]: pgmap v447: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-28T16:03:53.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:52 smithi150 ceph-mon[34825]: pgmap v448: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 682 B/s wr, 2 op/s 2024-03-28T16:03:53.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:52 smithi067 ceph-mon[31707]: pgmap v448: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 682 B/s wr, 2 op/s 2024-03-28T16:03:53.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:52 smithi183 ceph-mon[35580]: pgmap v448: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 682 B/s wr, 2 op/s 2024-03-28T16:03:55.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:54 smithi150 ceph-mon[34825]: pgmap v449: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 682 B/s wr, 2 op/s 2024-03-28T16:03:55.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:54 smithi067 ceph-mon[31707]: pgmap v449: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 682 B/s wr, 2 op/s 2024-03-28T16:03:55.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:54 smithi183 ceph-mon[35580]: pgmap v449: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 682 B/s wr, 2 op/s 2024-03-28T16:03:57.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:56 smithi150 ceph-mon[34825]: pgmap v450: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 597 B/s wr, 2 op/s 2024-03-28T16:03:57.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:56 smithi067 ceph-mon[31707]: pgmap v450: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 597 B/s wr, 2 op/s 2024-03-28T16:03:57.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:56 smithi183 ceph-mon[35580]: pgmap v450: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 597 B/s wr, 2 op/s 2024-03-28T16:03:59.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:03:58 smithi150 ceph-mon[34825]: pgmap v451: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:03:59.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:03:58 smithi067 ceph-mon[31707]: pgmap v451: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:03:59.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:03:58 smithi183 ceph-mon[35580]: pgmap v451: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:04:01.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:04:00 smithi150 ceph-mon[34825]: pgmap v452: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:04:01.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:04:00 smithi067 ceph-mon[31707]: pgmap v452: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:04:01.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:04:00 smithi183 ceph-mon[35580]: pgmap v452: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:04:03.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:04:02 smithi150 ceph-mon[34825]: pgmap v453: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:04:03.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:04:02 smithi067 ceph-mon[31707]: pgmap v453: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:04:03.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:04:02 smithi183 ceph-mon[35580]: pgmap v453: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:04:04.004 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T16:04:04.004 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T16:04:04.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.242+0000 7f0fd9600640 1 -- 172.21.15.67:0/2107903285 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0fd4104ad0 msgr2=0x7f0fd4104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:04.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.242+0000 7f0fd9600640 1 --2- 172.21.15.67:0/2107903285 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0fd4104ad0 0x7f0fd4104f50 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f0fc0009900 tx=0x7f0fc002bc30 comp rx=0 tx=0).stop 2024-03-28T16:04:04.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.242+0000 7f0fd9600640 1 -- 172.21.15.67:0/2107903285 shutdown_connections 2024-03-28T16:04:04.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.242+0000 7f0fd9600640 1 --2- 172.21.15.67:0/2107903285 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0fd4105490 0x7f0fd410bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:04.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.242+0000 7f0fd9600640 1 --2- 172.21.15.67:0/2107903285 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0fd4104ad0 0x7f0fd4104f50 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:04.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.242+0000 7f0fd9600640 1 --2- 172.21.15.67:0/2107903285 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fd41038e0 0x7f0fd4103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:04.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.242+0000 7f0fd9600640 1 -- 172.21.15.67:0/2107903285 >> 172.21.15.67:0/2107903285 conn(0x7f0fd40fee50 msgr2=0x7f0fd4101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:04:04.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.242+0000 7f0fd9600640 1 -- 172.21.15.67:0/2107903285 shutdown_connections 2024-03-28T16:04:04.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.242+0000 7f0fd9600640 1 -- 172.21.15.67:0/2107903285 wait complete. 2024-03-28T16:04:04.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.243+0000 7f0fd9600640 1 Processor -- start 2024-03-28T16:04:04.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.244+0000 7f0fd9600640 1 -- start start 2024-03-28T16:04:04.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.244+0000 7f0fd9600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0fd41038e0 0x7f0fd4199b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:04.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.244+0000 7f0fd9600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0fd4104ad0 0x7f0fd419a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:04.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.244+0000 7f0fd9600640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fd4105490 0x7f0fd41a10c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:04.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.244+0000 7f0fd9600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd410f410 con 0x7f0fd41038e0 2024-03-28T16:04:04.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.244+0000 7f0fd9600640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd410f290 con 0x7f0fd4105490 2024-03-28T16:04:04.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.244+0000 7f0fd9600640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd410f590 con 0x7f0fd4104ad0 2024-03-28T16:04:04.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.244+0000 7f0fd2a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0fd41038e0 0x7f0fd4199b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:04.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.244+0000 7f0fd2a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0fd41038e0 0x7f0fd4199b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44380/0 (socket says 172.21.15.67:44380) 2024-03-28T16:04:04.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.244+0000 7f0fd2a00640 1 -- 172.21.15.67:0/2710856074 learned_addr learned my addr 172.21.15.67:0/2710856074 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:04:04.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.245+0000 7f0fd2a00640 1 -- 172.21.15.67:0/2710856074 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0fd4104ad0 msgr2=0x7f0fd419a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:04.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.245+0000 7f0fd2000640 1 --2- 172.21.15.67:0/2710856074 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0fd4104ad0 0x7f0fd419a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:04.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.245+0000 7f0fd3400640 1 --2- 172.21.15.67:0/2710856074 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fd4105490 0x7f0fd41a10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:04.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.245+0000 7f0fd2a00640 1 --2- 172.21.15.67:0/2710856074 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0fd4104ad0 0x7f0fd419a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:04.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.245+0000 7f0fd2a00640 1 -- 172.21.15.67:0/2710856074 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fd4105490 msgr2=0x7f0fd41a10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:04.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.245+0000 7f0fd2a00640 1 --2- 172.21.15.67:0/2710856074 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fd4105490 0x7f0fd41a10c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:04.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.245+0000 7f0fd2a00640 1 -- 172.21.15.67:0/2710856074 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fd41a1730 con 0x7f0fd41038e0 2024-03-28T16:04:04.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.245+0000 7f0fd2000640 1 --2- 172.21.15.67:0/2710856074 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0fd4104ad0 0x7f0fd419a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:04:04.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.245+0000 7f0fd2a00640 1 --2- 172.21.15.67:0/2710856074 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0fd41038e0 0x7f0fd4199b10 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f0fc400d770 tx=0x7f0fc400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:04:04.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.245+0000 7f0fcb400640 1 -- 172.21.15.67:0/2710856074 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc4017070 con 0x7f0fd41038e0 2024-03-28T16:04:04.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.245+0000 7f0fcb400640 1 -- 172.21.15.67:0/2710856074 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0fc4009c30 con 0x7f0fd41038e0 2024-03-28T16:04:04.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.245+0000 7f0fd9600640 1 -- 172.21.15.67:0/2710856074 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fd41a1a20 con 0x7f0fd41038e0 2024-03-28T16:04:04.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.245+0000 7f0fcb400640 1 -- 172.21.15.67:0/2710856074 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc4003d00 con 0x7f0fd41038e0 2024-03-28T16:04:04.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.245+0000 7f0fd9600640 1 -- 172.21.15.67:0/2710856074 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fd41a1fc0 con 0x7f0fd41038e0 2024-03-28T16:04:04.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.249+0000 7f0fdc266740 1 Processor -- start 2024-03-28T16:04:04.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.249+0000 7f0fcb400640 1 -- 172.21.15.67:0/2710856074 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f0fc4003ea0 con 0x7f0fd41038e0 2024-03-28T16:04:04.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.249+0000 7f0fdc266740 1 -- start start 2024-03-28T16:04:04.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.249+0000 7f0fdc266740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559dd0fafe80 0x559dd0fb02a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:04.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.249+0000 7f0fdc266740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559dd0fb0870 0x559dd0fb5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:04.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.249+0000 7f0fdc266740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559dd0fb59e0 0x559dd0fb7e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:04.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.249+0000 7f0fdc266740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x559dd0f15250 con 0x559dd0fb59e0 2024-03-28T16:04:04.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.249+0000 7f0fdc266740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x559dd0ebda30 con 0x559dd0fafe80 2024-03-28T16:04:04.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.249+0000 7f0fdc266740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x559dd0f112f0 con 0x559dd0fb0870 2024-03-28T16:04:04.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.249+0000 7f0fd2000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559dd0fb59e0 0x559dd0fb7e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:04.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.249+0000 7f0fd2000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559dd0fb59e0 0x559dd0fb7e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44388/0 (socket says 172.21.15.67:44388) 2024-03-28T16:04:04.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.249+0000 7f0fd2000640 1 -- 172.21.15.67:0/2215363836 learned_addr learned my addr 172.21.15.67:0/2215363836 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:04:04.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.250+0000 7f0fd3400640 1 --2- 172.21.15.67:0/2215363836 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559dd0fafe80 0x559dd0fb02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:04.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.250+0000 7f0fd2a00640 1 --2- 172.21.15.67:0/2215363836 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559dd0fb0870 0x559dd0fb5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:04.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.250+0000 7f0fd2000640 1 -- 172.21.15.67:0/2215363836 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559dd0fb0870 msgr2=0x559dd0fb5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:04:04.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.250+0000 7f0fd2000640 1 --2- 172.21.15.67:0/2215363836 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559dd0fb0870 0x559dd0fb5370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:04.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.250+0000 7f0fd2000640 1 -- 172.21.15.67:0/2215363836 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559dd0fafe80 msgr2=0x559dd0fb02a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:04:04.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.250+0000 7f0fd2000640 1 --2- 172.21.15.67:0/2215363836 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559dd0fafe80 0x559dd0fb02a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:04.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.250+0000 7f0fd2000640 1 -- 172.21.15.67:0/2215363836 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559dd0ea8690 con 0x559dd0fb59e0 2024-03-28T16:04:04.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.250+0000 7f0fd3400640 1 --2- 172.21.15.67:0/2215363836 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559dd0fafe80 0x559dd0fb02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T16:04:04.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.250+0000 7f0fcb400640 1 --2- 172.21.15.67:0/2710856074 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f0fb80a2880 0x7f0fb80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:04.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.250+0000 7f0fd2000640 1 --2- 172.21.15.67:0/2215363836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559dd0fb59e0 0x559dd0fb7e70 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f0fc0006010 tx=0x7f0fc0005e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ecef3ffeeb041e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:04:04.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.251+0000 7f0fcb400640 1 -- 172.21.15.67:0/2710856074 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f0fc40ce830 con 0x7f0fd41038e0 2024-03-28T16:04:04.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.251+0000 7f0fd3400640 1 --2- 172.21.15.67:0/2710856074 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f0fb80a2880 0x7f0fb80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:04.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.251+0000 7f0fbf400640 1 -- 172.21.15.67:0/2215363836 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc0043020 con 0x559dd0fb59e0 2024-03-28T16:04:04.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.251+0000 7f0fbf400640 1 -- 172.21.15.67:0/2215363836 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0fc0004040 con 0x559dd0fb59e0 2024-03-28T16:04:04.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.251+0000 7f0fbf400640 1 -- 172.21.15.67:0/2215363836 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc00046c0 con 0x559dd0fb59e0 2024-03-28T16:04:04.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.252+0000 7f0fdc266740 1 -- 172.21.15.67:0/2215363836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559dd0fb59e0 msgr2=0x559dd0fb7e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:04.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.252+0000 7f0fd3400640 1 --2- 172.21.15.67:0/2710856074 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f0fb80a2880 0x7f0fb80a4d40 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x559dd0fb10d0 tx=0x7f0fcc0094a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:04:04.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.252+0000 7f0fdc266740 1 --2- 172.21.15.67:0/2215363836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559dd0fb59e0 0x559dd0fb7e70 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f0fc0006010 tx=0x7f0fc0005e00 comp rx=0 tx=0).stop 2024-03-28T16:04:04.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.252+0000 7f0fdc266740 1 -- 172.21.15.67:0/2215363836 shutdown_connections 2024-03-28T16:04:04.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.252+0000 7f0fdc266740 1 --2- 172.21.15.67:0/2215363836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559dd0fb59e0 0x559dd0fb7e70 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:04.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.252+0000 7f0fdc266740 1 --2- 172.21.15.67:0/2215363836 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559dd0fb0870 0x559dd0fb5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:04.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.252+0000 7f0fdc266740 1 --2- 172.21.15.67:0/2215363836 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559dd0fafe80 0x559dd0fb02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:04.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.252+0000 7f0fdc266740 1 -- 172.21.15.67:0/2215363836 >> 172.21.15.67:0/2215363836 conn(0x559dd0f78200 msgr2=0x559dd0fab7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:04:04.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.252+0000 7f0fdc266740 1 -- 172.21.15.67:0/2215363836 shutdown_connections 2024-03-28T16:04:04.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.252+0000 7f0fdc266740 1 -- 172.21.15.67:0/2215363836 wait complete. 2024-03-28T16:04:04.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.252+0000 7f0fdc266740 1 -- start start 2024-03-28T16:04:04.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.253+0000 7f0fdc266740 1 Processor -- start 2024-03-28T16:04:04.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.253+0000 7f0fdc266740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559dd0fafe80 0x559dd0fb9df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:04.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.253+0000 7f0fdc266740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559dd0fb0870 0x559dd0fba330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:04.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.253+0000 7f0fdc266740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559dd0fb59e0 0x559dd0fc1300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:04.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.253+0000 7f0fdc266740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x559dd0fb8930 con 0x559dd0fb0870 2024-03-28T16:04:04.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.253+0000 7f0fdc266740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x559dd0ed0e80 con 0x559dd0fafe80 2024-03-28T16:04:04.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.253+0000 7f0fdc266740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x559dd0fb8ab0 con 0x559dd0fb59e0 2024-03-28T16:04:04.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.253+0000 7f0fdc266740 10 client.0 _ll_register_callbacks cb 0x559dd0ee3130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:04:04.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.253+0000 7f0fd2000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559dd0fb0870 0x559dd0fba330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:04.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.254+0000 7f0fd2000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559dd0fb0870 0x559dd0fba330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44400/0 (socket says 172.21.15.67:44400) 2024-03-28T16:04:04.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.254+0000 7f0fd2000640 1 -- 172.21.15.67:0/3749345224 learned_addr learned my addr 172.21.15.67:0/3749345224 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:04:04.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.254+0000 7f0fd2a00640 1 --2- 172.21.15.67:0/3749345224 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559dd0fafe80 0x559dd0fb9df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:04.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.254+0000 7f0fd3400640 1 --2- 172.21.15.67:0/3749345224 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559dd0fb59e0 0x559dd0fc1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:04.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.254+0000 7f0fd2000640 1 -- 172.21.15.67:0/3749345224 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559dd0fb59e0 msgr2=0x559dd0fc1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:04:04.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.254+0000 7f0fd2000640 1 --2- 172.21.15.67:0/3749345224 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559dd0fb59e0 0x559dd0fc1300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:04.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.254+0000 7f0fd2000640 1 -- 172.21.15.67:0/3749345224 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559dd0fafe80 msgr2=0x559dd0fb9df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:04:04.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.254+0000 7f0fd2000640 1 --2- 172.21.15.67:0/3749345224 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559dd0fafe80 0x559dd0fb9df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:04.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.254+0000 7f0fd2000640 1 -- 172.21.15.67:0/3749345224 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559dd0cf37d0 con 0x559dd0fb0870 2024-03-28T16:04:04.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.254+0000 7f0fd3400640 1 --2- 172.21.15.67:0/3749345224 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559dd0fb59e0 0x559dd0fc1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:04:04.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.254+0000 7f0fd2000640 1 --2- 172.21.15.67:0/3749345224 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559dd0fb0870 0x559dd0fba330 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f0fc002cb90 tx=0x7f0fc002cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63caf76ab548b2e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:04:04.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.254+0000 7f0fb3e00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:04:04.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.255+0000 7f0fb3e00640 1 -- 172.21.15.67:0/3749345224 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc004c070 con 0x559dd0fb0870 2024-03-28T16:04:04.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.255+0000 7f0fb3e00640 1 -- 172.21.15.67:0/3749345224 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f0fc0003a00 con 0x559dd0fb0870 2024-03-28T16:04:04.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.255+0000 7f0fdc266740 1 -- 172.21.15.67:0/3749345224 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0fc0004040 con 0x559dd0fb0870 2024-03-28T16:04:04.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.255+0000 7f0fb3e00640 1 -- 172.21.15.67:0/3749345224 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0fc0043040 con 0x559dd0fb0870 2024-03-28T16:04:04.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.255+0000 7f0fdc266740 10 client.15339 fetch_fsmap learned FSMap version 19 2024-03-28T16:04:04.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.255+0000 7f0fdc266740 1 -- 172.21.15.67:0/3749345224 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x559dd0c293f0 con 0x559dd0fb0870 2024-03-28T16:04:04.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.255+0000 7f0fb3e00640 1 -- 172.21.15.67:0/3749345224 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0fc0004040 con 0x559dd0fb0870 2024-03-28T16:04:04.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.255+0000 7f0fb3e00640 1 -- 172.21.15.67:0/3749345224 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f0fc0004d30 con 0x559dd0fb0870 2024-03-28T16:04:04.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.256+0000 7f0fdc266740 10 client.15339 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:04:04.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.256+0000 7f0fdc266740 10 client.15339 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:04:04.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.256+0000 7f0fdc266740 20 client.15339 populate_metadata read hostname 'smithi067' 2024-03-28T16:04:04.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.256+0000 7f0fdc266740 1 --2- 172.21.15.67:0/3749345224 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x559dd0fc57f0 0x559dd0fc7c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:04.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.256+0000 7f0fdc266740 4 client.15339 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:04:04.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.256+0000 7f0fdc266740 1 -- 172.21.15.67:0/3749345224 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559dd0f22790 con 0x559dd0fc57f0 2024-03-28T16:04:04.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.256+0000 7f0fd2a00640 1 --2- 172.21.15.67:0/3749345224 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x559dd0fc57f0 0x559dd0fc7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:04.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.257+0000 7f0fd2a00640 1 --2- 172.21.15.67:0/3749345224 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x559dd0fc57f0 0x559dd0fc7c20 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b03302f68945c0ae server_cookie=ff6deb7b954a5dfd in_seq=0 out_seq=0 2024-03-28T16:04:04.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.257+0000 7f0fb3e00640 10 client.15339 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:04:04.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.259+0000 7f0fb3e00640 1 -- 172.21.15.67:0/3749345224 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x559dd0f22790 con 0x559dd0fc57f0 2024-03-28T16:04:04.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.259+0000 7f0fb3e00640 10 client.15339 handle_command_reply: tid=0 2024-03-28T16:04:04.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.259+0000 7f0fdc266740 1 client.15339 shutdown 2024-03-28T16:04:04.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.259+0000 7f0fdc266740 1 -- 172.21.15.67:0/3749345224 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559dd0fb0870 msgr2=0x559dd0fba330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:04.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.259+0000 7f0fdc266740 1 --2- 172.21.15.67:0/3749345224 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559dd0fb0870 0x559dd0fba330 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f0fc002cb90 tx=0x7f0fc002cbc0 comp rx=0 tx=0).stop 2024-03-28T16:04:04.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.259+0000 7f0fdc266740 1 -- 172.21.15.67:0/3749345224 shutdown_connections 2024-03-28T16:04:04.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.259+0000 7f0fdc266740 1 --2- 172.21.15.67:0/3749345224 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x559dd0fc57f0 0x559dd0fc7c20 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:04.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.259+0000 7f0fdc266740 1 --2- 172.21.15.67:0/3749345224 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559dd0fb59e0 0x559dd0fc1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:04.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.259+0000 7f0fdc266740 1 --2- 172.21.15.67:0/3749345224 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559dd0fb0870 0x559dd0fba330 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:04.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.259+0000 7f0fb3e00640 0 client.15339 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:04:04.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.259+0000 7f0fdc266740 1 --2- 172.21.15.67:0/3749345224 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559dd0fafe80 0x559dd0fb9df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:04.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.259+0000 7f0fdc266740 1 -- 172.21.15.67:0/3749345224 >> 172.21.15.67:0/3749345224 conn(0x559dd0f78200 msgr2=0x559dd0faaa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:04:04.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.260+0000 7f0fdc266740 1 -- 172.21.15.67:0/3749345224 shutdown_connections 2024-03-28T16:04:04.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.260+0000 7f0fdc266740 1 -- 172.21.15.67:0/3749345224 wait complete. 2024-03-28T16:04:04.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.260+0000 7f0fdc266740 20 client.15339 trim_cache size 0 max 16384 2024-03-28T16:04:04.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.283+0000 7f0fdc266740 1 Processor -- start 2024-03-28T16:04:04.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.283+0000 7f0fdc266740 1 -- start start 2024-03-28T16:04:04.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.283+0000 7f0fdc266740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559dd0f78200 0x559dd0faea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:04.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.283+0000 7f0fdc266740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559dd0fafe80 0x559dd0fc12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:04.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.283+0000 7f0fdc266740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559dd0fb0870 0x559dd0faef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:04.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.283+0000 7f0fdc266740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd41a8c80 con 0x559dd0fafe80 2024-03-28T16:04:04.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.283+0000 7f0fdc266740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd41ab240 con 0x559dd0f78200 2024-03-28T16:04:04.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.283+0000 7f0fdc266740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd41a9280 con 0x559dd0fb0870 2024-03-28T16:04:04.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.284+0000 7f0fd2000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559dd0fafe80 0x559dd0fc12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:04.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.284+0000 7f0fd2000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559dd0fafe80 0x559dd0fc12c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44416/0 (socket says 172.21.15.67:44416) 2024-03-28T16:04:04.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.284+0000 7f0fd2000640 1 -- 172.21.15.67:0/3579531400 learned_addr learned my addr 172.21.15.67:0/3579531400 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:04:04.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.284+0000 7f0fd2a00640 1 --2- 172.21.15.67:0/3579531400 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559dd0f78200 0x559dd0faea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:04.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.284+0000 7f0fd3400640 1 --2- 172.21.15.67:0/3579531400 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559dd0fb0870 0x559dd0faef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:04.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.285+0000 7f0fd2000640 1 -- 172.21.15.67:0/3579531400 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559dd0fb0870 msgr2=0x559dd0faef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:04:04.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.285+0000 7f0fd2000640 1 --2- 172.21.15.67:0/3579531400 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559dd0fb0870 0x559dd0faef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:04.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.285+0000 7f0fd2000640 1 -- 172.21.15.67:0/3579531400 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559dd0f78200 msgr2=0x559dd0faea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:04:04.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.285+0000 7f0fd2000640 1 --2- 172.21.15.67:0/3579531400 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559dd0f78200 0x559dd0faea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:04.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.285+0000 7f0fd2000640 1 -- 172.21.15.67:0/3579531400 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559dd0be90f0 con 0x559dd0fafe80 2024-03-28T16:04:04.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.285+0000 7f0fd2000640 1 --2- 172.21.15.67:0/3579531400 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559dd0fafe80 0x559dd0fc12c0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f0fc00043b0 tx=0x7f0fc0005e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1babdd5c71f4b046 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:04:04.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.285+0000 7f0fbea00640 1 -- 172.21.15.67:0/3579531400 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc004c040 con 0x559dd0fafe80 2024-03-28T16:04:04.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.285+0000 7f0fbea00640 1 -- 172.21.15.67:0/3579531400 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0fc0004490 con 0x559dd0fafe80 2024-03-28T16:04:04.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.285+0000 7f0fbea00640 1 -- 172.21.15.67:0/3579531400 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc0058030 con 0x559dd0fafe80 2024-03-28T16:04:04.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.286+0000 7f0fdc266740 1 -- 172.21.15.67:0/3579531400 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559dd0fafe80 msgr2=0x559dd0fc12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:04.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.286+0000 7f0fdc266740 1 --2- 172.21.15.67:0/3579531400 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559dd0fafe80 0x559dd0fc12c0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f0fc00043b0 tx=0x7f0fc0005e00 comp rx=0 tx=0).stop 2024-03-28T16:04:04.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.286+0000 7f0fdc266740 1 -- 172.21.15.67:0/3579531400 shutdown_connections 2024-03-28T16:04:04.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.286+0000 7f0fdc266740 1 --2- 172.21.15.67:0/3579531400 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559dd0fb0870 0x559dd0faef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:04.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.286+0000 7f0fdc266740 1 --2- 172.21.15.67:0/3579531400 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559dd0fafe80 0x559dd0fc12c0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:04.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.286+0000 7f0fdc266740 1 --2- 172.21.15.67:0/3579531400 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559dd0f78200 0x559dd0faea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:04.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.286+0000 7f0fdc266740 1 -- 172.21.15.67:0/3579531400 >> 172.21.15.67:0/3579531400 conn(0x559dd0fb99b0 msgr2=0x559dd0fb1ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:04:04.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.286+0000 7f0fdc266740 1 -- 172.21.15.67:0/3579531400 shutdown_connections 2024-03-28T16:04:04.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.286+0000 7f0fdc266740 1 -- 172.21.15.67:0/3579531400 wait complete. 2024-03-28T16:04:04.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.287+0000 7f0fdc266740 1 -- start start 2024-03-28T16:04:04.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.287+0000 7f0fdc266740 1 Processor -- start 2024-03-28T16:04:04.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.287+0000 7f0fdc266740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559dd0f78200 0x559dd0fbf080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:04.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.287+0000 7f0fdc266740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559dd0fafe80 0x559dd0fbf5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:04.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.287+0000 7f0fdc266740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559dd0fb0870 0x559dd0fc05d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:04.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.287+0000 7f0fdc266740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd41a98c0 con 0x559dd0fafe80 2024-03-28T16:04:04.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.287+0000 7f0fdc266740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd41aabe0 con 0x559dd0f78200 2024-03-28T16:04:04.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.287+0000 7f0fdc266740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd41a9f20 con 0x559dd0fb0870 2024-03-28T16:04:04.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.287+0000 7f0fdc266740 10 client.0 _ll_register_callbacks cb 0x7f0fc0005d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:04:04.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.288+0000 7f0fd2000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559dd0fafe80 0x559dd0fbf5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:04.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.288+0000 7f0fd2000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559dd0fafe80 0x559dd0fbf5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44430/0 (socket says 172.21.15.67:44430) 2024-03-28T16:04:04.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.288+0000 7f0fd2000640 1 -- 172.21.15.67:0/484824190 learned_addr learned my addr 172.21.15.67:0/484824190 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:04:04.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.288+0000 7f0fd2a00640 1 --2- 172.21.15.67:0/484824190 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559dd0f78200 0x559dd0fbf080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:04.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.288+0000 7f0fd3400640 1 --2- 172.21.15.67:0/484824190 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559dd0fb0870 0x559dd0fc05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:04.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.288+0000 7f0fd2000640 1 -- 172.21.15.67:0/484824190 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559dd0fb0870 msgr2=0x559dd0fc05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:04:04.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.288+0000 7f0fd2000640 1 --2- 172.21.15.67:0/484824190 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559dd0fb0870 0x559dd0fc05d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:04.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.288+0000 7f0fd2000640 1 -- 172.21.15.67:0/484824190 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559dd0f78200 msgr2=0x559dd0fbf080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:04:04.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.288+0000 7f0fd2000640 1 --2- 172.21.15.67:0/484824190 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559dd0f78200 0x559dd0fbf080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:04.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.288+0000 7f0fd2000640 1 -- 172.21.15.67:0/484824190 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559dd0fc0bc0 con 0x559dd0fafe80 2024-03-28T16:04:04.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.288+0000 7f0fd2a00640 1 --2- 172.21.15.67:0/484824190 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559dd0f78200 0x559dd0fbf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T16:04:04.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.288+0000 7f0fd2000640 1 --2- 172.21.15.67:0/484824190 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559dd0fafe80 0x559dd0fbf5c0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f0fc0048470 tx=0x7f0fc0004c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b03366df3c86328 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:04:04.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.288+0000 7f0fb3e00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:04:04.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.289+0000 7f0fb3e00640 1 -- 172.21.15.67:0/484824190 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc0060020 con 0x559dd0fafe80 2024-03-28T16:04:04.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.289+0000 7f0fb3e00640 1 -- 172.21.15.67:0/484824190 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f0fc0048740 con 0x559dd0fafe80 2024-03-28T16:04:04.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.289+0000 7f0fdc266740 1 -- 172.21.15.67:0/484824190 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0fc0004490 con 0x559dd0fafe80 2024-03-28T16:04:04.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.289+0000 7f0fb3e00640 1 -- 172.21.15.67:0/484824190 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0fc0058070 con 0x559dd0fafe80 2024-03-28T16:04:04.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.289+0000 7f0fdc266740 10 client.15345 fetch_fsmap learned FSMap version 19 2024-03-28T16:04:04.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.289+0000 7f0fdc266740 1 -- 172.21.15.67:0/484824190 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x559dd1021700 con 0x559dd0fafe80 2024-03-28T16:04:04.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.289+0000 7f0fb3e00640 1 -- 172.21.15.67:0/484824190 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0fc0004490 con 0x559dd0fafe80 2024-03-28T16:04:04.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.290+0000 7f0fb3e00640 1 -- 172.21.15.67:0/484824190 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f0fc0003c80 con 0x559dd0fafe80 2024-03-28T16:04:04.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.290+0000 7f0fdc266740 10 client.15345 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:04:04.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.290+0000 7f0fdc266740 10 client.15345 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:04:04.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.290+0000 7f0fdc266740 20 client.15345 populate_metadata read hostname 'smithi067' 2024-03-28T16:04:04.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.290+0000 7f0fdc266740 1 --2- 172.21.15.67:0/484824190 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x559dd0fb59e0 0x559dd10240b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:04.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.290+0000 7f0fdc266740 4 client.15345 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T16:04:04.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.290+0000 7f0fdc266740 1 -- 172.21.15.67:0/484824190 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x559dd0f1cae0 con 0x559dd0fb59e0 2024-03-28T16:04:04.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.290+0000 7f0fd2a00640 1 --2- 172.21.15.67:0/484824190 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x559dd0fb59e0 0x559dd10240b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:04.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.291+0000 7f0fd2a00640 1 --2- 172.21.15.67:0/484824190 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x559dd0fb59e0 0x559dd10240b0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f6d73ce1fe630ea9 server_cookie=8b0a45a2f29ce448 in_seq=0 out_seq=0 2024-03-28T16:04:04.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.291+0000 7f0fb3e00640 10 client.15345 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:04:04.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.291+0000 7f0fb3e00640 1 -- 172.21.15.67:0/484824190 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x559dd0f1cae0 con 0x559dd0fb59e0 2024-03-28T16:04:04.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.291+0000 7f0fb3e00640 10 client.15345 handle_command_reply: tid=0 2024-03-28T16:04:04.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.291+0000 7f0fdc266740 1 client.15345 shutdown 2024-03-28T16:04:04.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.292+0000 7f0fdc266740 1 -- 172.21.15.67:0/484824190 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559dd0fafe80 msgr2=0x559dd0fbf5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:04.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.292+0000 7f0fdc266740 1 --2- 172.21.15.67:0/484824190 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559dd0fafe80 0x559dd0fbf5c0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f0fc0048470 tx=0x7f0fc0004c10 comp rx=0 tx=0).stop 2024-03-28T16:04:04.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.292+0000 7f0fdc266740 1 -- 172.21.15.67:0/484824190 shutdown_connections 2024-03-28T16:04:04.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.292+0000 7f0fdc266740 1 --2- 172.21.15.67:0/484824190 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x559dd0fb59e0 0x559dd10240b0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:04.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.292+0000 7f0fdc266740 1 --2- 172.21.15.67:0/484824190 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559dd0fb0870 0x559dd0fc05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:04.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.292+0000 7f0fdc266740 1 --2- 172.21.15.67:0/484824190 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559dd0fafe80 0x559dd0fbf5c0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:04.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.292+0000 7f0fdc266740 1 --2- 172.21.15.67:0/484824190 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559dd0f78200 0x559dd0fbf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:04.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.292+0000 7f0fb3e00640 0 client.15345 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:04:04.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.292+0000 7f0fdc266740 1 -- 172.21.15.67:0/484824190 >> 172.21.15.67:0/484824190 conn(0x559dd0fb99b0 msgr2=0x559dd0faa200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:04:04.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.292+0000 7f0fdc266740 1 -- 172.21.15.67:0/484824190 shutdown_connections 2024-03-28T16:04:04.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.292+0000 7f0fdc266740 1 -- 172.21.15.67:0/484824190 wait complete. 2024-03-28T16:04:04.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.292+0000 7f0fdc266740 20 client.15345 trim_cache size 0 max 16384 2024-03-28T16:04:04.297 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-28T16:04:04.298 INFO:teuthology.orchestra.run.smithi067.stdout: "status": "no active scrubs running", 2024-03-28T16:04:04.298 INFO:teuthology.orchestra.run.smithi067.stdout: "scrubs": {} 2024-03-28T16:04:04.298 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-28T16:04:04.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.293+0000 7f0fd9600640 1 -- 172.21.15.67:0/2710856074 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f0fb80a2880 msgr2=0x7f0fb80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:04.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.293+0000 7f0fd9600640 1 --2- 172.21.15.67:0/2710856074 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f0fb80a2880 0x7f0fb80a4d40 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x559dd0fb10d0 tx=0x7f0fcc0094a0 comp rx=0 tx=0).stop 2024-03-28T16:04:04.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.293+0000 7f0fd9600640 1 -- 172.21.15.67:0/2710856074 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0fd41038e0 msgr2=0x7f0fd4199b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:04.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.293+0000 7f0fd9600640 1 --2- 172.21.15.67:0/2710856074 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0fd41038e0 0x7f0fd4199b10 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f0fc400d770 tx=0x7f0fc400dc30 comp rx=0 tx=0).stop 2024-03-28T16:04:04.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.293+0000 7f0fd9600640 1 -- 172.21.15.67:0/2710856074 shutdown_connections 2024-03-28T16:04:04.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.293+0000 7f0fd9600640 1 --2- 172.21.15.67:0/2710856074 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f0fb80a2880 0x7f0fb80a4d40 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:04.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.293+0000 7f0fd9600640 1 --2- 172.21.15.67:0/2710856074 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fd4105490 0x7f0fd41a10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:04.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.293+0000 7f0fd9600640 1 --2- 172.21.15.67:0/2710856074 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0fd4104ad0 0x7f0fd419a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:04.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.293+0000 7f0fd9600640 1 --2- 172.21.15.67:0/2710856074 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0fd41038e0 0x7f0fd4199b10 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:04.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.293+0000 7f0fd9600640 1 -- 172.21.15.67:0/2710856074 >> 172.21.15.67:0/2710856074 conn(0x7f0fd40fee50 msgr2=0x7f0fd4107ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:04:04.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.293+0000 7f0fd9600640 1 -- 172.21.15.67:0/2710856074 shutdown_connections 2024-03-28T16:04:04.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.293+0000 7f0fd9600640 1 -- 172.21.15.67:0/2710856074 wait complete. 2024-03-28T16:04:04.312 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T16:04:04.313 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T16:04:04.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.515+0000 7fb7a3e00640 1 -- 172.21.15.67:0/3333116182 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7a40740a0 msgr2=0x7fb7a4074500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:04.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.515+0000 7fb7a3e00640 1 --2- 172.21.15.67:0/3333116182 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7a40740a0 0x7fb7a4074500 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fb790009900 tx=0x7fb79002bcd0 comp rx=0 tx=0).stop 2024-03-28T16:04:04.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.515+0000 7fb7a3e00640 1 -- 172.21.15.67:0/3333116182 shutdown_connections 2024-03-28T16:04:04.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.515+0000 7fb7a3e00640 1 --2- 172.21.15.67:0/3333116182 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7a4109390 0x7fb7a410b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:04.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.515+0000 7fb7a3e00640 1 --2- 172.21.15.67:0/3333116182 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7a40740a0 0x7fb7a4074500 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:04.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.515+0000 7fb7a3e00640 1 --2- 172.21.15.67:0/3333116182 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb7a4075680 0x7fb7a4073ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:04.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.516+0000 7fb7a3e00640 1 -- 172.21.15.67:0/3333116182 >> 172.21.15.67:0/3333116182 conn(0x7fb7a40fca70 msgr2=0x7fb7a40feeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:04:04.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.516+0000 7fb7a3e00640 1 -- 172.21.15.67:0/3333116182 shutdown_connections 2024-03-28T16:04:04.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.516+0000 7fb7a3e00640 1 -- 172.21.15.67:0/3333116182 wait complete. 2024-03-28T16:04:04.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.517+0000 7fb7a3e00640 1 Processor -- start 2024-03-28T16:04:04.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.517+0000 7fb7a3e00640 1 -- start start 2024-03-28T16:04:04.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.517+0000 7fb7a3e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7a40740a0 0x7fb7a4199980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:04.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.517+0000 7fb7a3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7a4075680 0x7fb7a4199ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:04.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.517+0000 7fb7a3e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb7a4109390 0x7fb7a41a0f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:04.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.517+0000 7fb7a3e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a410f280 con 0x7fb7a4075680 2024-03-28T16:04:04.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.517+0000 7fb7a3e00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a410f100 con 0x7fb7a40740a0 2024-03-28T16:04:04.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.517+0000 7fb7a3e00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a410f400 con 0x7fb7a4109390 2024-03-28T16:04:04.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.518+0000 7fb7a2000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7a4075680 0x7fb7a4199ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:04.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.518+0000 7fb7a2000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7a4075680 0x7fb7a4199ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44454/0 (socket says 172.21.15.67:44454) 2024-03-28T16:04:04.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.518+0000 7fb7a2000640 1 -- 172.21.15.67:0/2222891474 learned_addr learned my addr 172.21.15.67:0/2222891474 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:04:04.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.518+0000 7fb7a2a00640 1 --2- 172.21.15.67:0/2222891474 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7a40740a0 0x7fb7a4199980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:04.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.518+0000 7fb7a3400640 1 --2- 172.21.15.67:0/2222891474 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb7a4109390 0x7fb7a41a0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:04.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.518+0000 7fb7a2000640 1 -- 172.21.15.67:0/2222891474 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb7a4109390 msgr2=0x7fb7a41a0f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:04.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.518+0000 7fb7a2000640 1 --2- 172.21.15.67:0/2222891474 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb7a4109390 0x7fb7a41a0f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:04.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.518+0000 7fb7a2000640 1 -- 172.21.15.67:0/2222891474 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7a40740a0 msgr2=0x7fb7a4199980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:04.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.518+0000 7fb7a2000640 1 --2- 172.21.15.67:0/2222891474 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7a40740a0 0x7fb7a4199980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:04.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.518+0000 7fb7a2000640 1 -- 172.21.15.67:0/2222891474 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7a41a15a0 con 0x7fb7a4075680 2024-03-28T16:04:04.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.518+0000 7fb7a2a00640 1 --2- 172.21.15.67:0/2222891474 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7a40740a0 0x7fb7a4199980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T16:04:04.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.519+0000 7fb7a2000640 1 --2- 172.21.15.67:0/2222891474 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7a4075680 0x7fb7a4199ec0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fb790000c00 tx=0x7fb79002dc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:04:04.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.519+0000 7fb7a8a00640 1 -- 172.21.15.67:0/2222891474 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb790043020 con 0x7fb7a4075680 2024-03-28T16:04:04.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.519+0000 7fb7a8a00640 1 -- 172.21.15.67:0/2222891474 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb79002de30 con 0x7fb7a4075680 2024-03-28T16:04:04.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.519+0000 7fb7a8a00640 1 -- 172.21.15.67:0/2222891474 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb790003c60 con 0x7fb7a4075680 2024-03-28T16:04:04.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.519+0000 7fb7a3e00640 1 -- 172.21.15.67:0/2222891474 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7a41a1830 con 0x7fb7a4075680 2024-03-28T16:04:04.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.519+0000 7fb7a3e00640 1 -- 172.21.15.67:0/2222891474 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7a41a1d80 con 0x7fb7a4075680 2024-03-28T16:04:04.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.520+0000 7fb7a3e00640 1 -- 172.21.15.67:0/2222891474 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7a4108150 con 0x7fb7a4075680 2024-03-28T16:04:04.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.522+0000 7fb7a8a00640 1 -- 172.21.15.67:0/2222891474 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fb790046020 con 0x7fb7a4075680 2024-03-28T16:04:04.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.523+0000 7fb7a8a00640 1 --2- 172.21.15.67:0/2222891474 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb7880a2800 0x7fb7880a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:04.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.524+0000 7fb7a8a00640 1 -- 172.21.15.67:0/2222891474 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fb7900f0eb0 con 0x7fb7a4075680 2024-03-28T16:04:04.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.524+0000 7fb7a2a00640 1 --2- 172.21.15.67:0/2222891474 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb7880a2800 0x7fb7880a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:04.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.525+0000 7fb7a2a00640 1 --2- 172.21.15.67:0/2222891474 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb7880a2800 0x7fb7880a4cc0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fb7940058c0 tx=0x7fb794005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:04:04.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.529+0000 7fb7a8a00640 1 -- 172.21.15.67:0/2222891474 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb79004d020 con 0x7fb7a4075680 2024-03-28T16:04:04.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.777+0000 7fb7a3e00640 1 -- 172.21.15.67:0/2222891474 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb7a402d070 con 0x7fb7a4075680 2024-03-28T16:04:04.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.779+0000 7fb7a8a00640 1 -- 172.21.15.67:0/2222891474 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11341 (secure 0 0 0) 0x7fb7900eee20 con 0x7fb7a4075680 2024-03-28T16:04:04.781 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:04:04.781 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2008297659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2008297659},{"type":"v1","addr":"172.21.15.67:6835","nonce":2008297659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14532,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/320295745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":320295745},{"type":"v1","addr":"172.21.15.67:6837","nonce":320295745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6839/691821273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":691821273},{"type":"v1","addr":"172.21.15.67:6839","nonce":691821273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6841/3211254875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6840","nonce":3211254875},{"type":"v1","addr":"172.21.15.67:6841","nonce":3211254875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2129665423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2129665423},{"type":"v1","addr":"172.21.15.183:6833","nonce":2129665423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24496,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/1589784425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":1589784425},{"type":"v1","addr":"172.21.15.183:6835","nonce":1589784425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24508,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/2185863835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":2185863835},{"type":"v1","addr":"172.21.15.183:6837","nonce":2185863835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24520,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6839/1242814398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":1242814398},{"type":"v1","addr":"172.21.15.183:6839","nonce":1242814398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":34409,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1008335521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1008335521},{"type":"v1","addr":"172.21.15.150:6833","nonce":1008335521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":34445,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3269935695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3269935695},{"type":"v1","addr":"172.21.15.150:6835","nonce":3269935695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T16:01:07.696824+0000","modified":"2024-03-28T16:01:11.997773+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":34508},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34475":{"gid":34475,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/1112886277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":1112886277},{"type":"v1","addr":"172.21.15.150:6837","nonce":1112886277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_34508":{"gid":34508,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6839/1211011614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":1211011614},{"type":"v1","addr":"172.21.15.150:6839","nonce":1211011614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34508,"qdb_cluster":[34508]},"id":1}]} 2024-03-28T16:04:04.781 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 19 2024-03-28T16:04:04.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.782+0000 7fb7a3e00640 1 -- 172.21.15.67:0/2222891474 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb7880a2800 msgr2=0x7fb7880a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:04.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.782+0000 7fb7a3e00640 1 --2- 172.21.15.67:0/2222891474 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb7880a2800 0x7fb7880a4cc0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fb7940058c0 tx=0x7fb794005850 comp rx=0 tx=0).stop 2024-03-28T16:04:04.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.782+0000 7fb7a3e00640 1 -- 172.21.15.67:0/2222891474 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7a4075680 msgr2=0x7fb7a4199ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:04.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.782+0000 7fb7a3e00640 1 --2- 172.21.15.67:0/2222891474 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7a4075680 0x7fb7a4199ec0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fb790000c00 tx=0x7fb79002dc40 comp rx=0 tx=0).stop 2024-03-28T16:04:04.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.782+0000 7fb7a3e00640 1 -- 172.21.15.67:0/2222891474 shutdown_connections 2024-03-28T16:04:04.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.782+0000 7fb7a3e00640 1 --2- 172.21.15.67:0/2222891474 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb7880a2800 0x7fb7880a4cc0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:04.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.782+0000 7fb7a3e00640 1 --2- 172.21.15.67:0/2222891474 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb7a4109390 0x7fb7a41a0f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:04.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.782+0000 7fb7a3e00640 1 --2- 172.21.15.67:0/2222891474 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7a4075680 0x7fb7a4199ec0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:04.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.782+0000 7fb7a3e00640 1 --2- 172.21.15.67:0/2222891474 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7a40740a0 0x7fb7a4199980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:04.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.782+0000 7fb7a3e00640 1 -- 172.21.15.67:0/2222891474 >> 172.21.15.67:0/2222891474 conn(0x7fb7a40fca70 msgr2=0x7fb7a4105320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:04:04.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.782+0000 7fb7a3e00640 1 -- 172.21.15.67:0/2222891474 shutdown_connections 2024-03-28T16:04:04.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:04.782+0000 7fb7a3e00640 1 -- 172.21.15.67:0/2222891474 wait complete. 2024-03-28T16:04:04.804 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T16:04:04.804 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T16:04:05.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.033+0000 7fad36a00640 1 -- 172.21.15.67:0/2166675437 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad30102000 msgr2=0x7fad301043f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:05.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.033+0000 7fad36a00640 1 --2- 172.21.15.67:0/2166675437 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad30102000 0x7fad301043f0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fad24009900 tx=0x7fad2402bc30 comp rx=0 tx=0).stop 2024-03-28T16:04:05.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.033+0000 7fad36a00640 1 -- 172.21.15.67:0/2166675437 shutdown_connections 2024-03-28T16:04:05.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.033+0000 7fad36a00640 1 --2- 172.21.15.67:0/2166675437 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad30107260 0x7fad301096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:05.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.033+0000 7fad36a00640 1 --2- 172.21.15.67:0/2166675437 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fad30104930 0x7fad30106d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:05.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.033+0000 7fad36a00640 1 --2- 172.21.15.67:0/2166675437 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad30102000 0x7fad301043f0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:05.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.033+0000 7fad36a00640 1 -- 172.21.15.67:0/2166675437 >> 172.21.15.67:0/2166675437 conn(0x7fad300fba10 msgr2=0x7fad300fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:04:05.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.033+0000 7fad36a00640 1 -- 172.21.15.67:0/2166675437 shutdown_connections 2024-03-28T16:04:05.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.034+0000 7fad36a00640 1 -- 172.21.15.67:0/2166675437 wait complete. 2024-03-28T16:04:05.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.034+0000 7fad36a00640 1 Processor -- start 2024-03-28T16:04:05.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.035+0000 7fad36a00640 1 -- start start 2024-03-28T16:04:05.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.035+0000 7fad36a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad30102000 0x7fad3010df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:05.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.035+0000 7fad36a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad30104930 0x7fad3010e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:05.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.035+0000 7fad36a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fad30107260 0x7fad3010aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:05.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.035+0000 7fad36a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad3010f310 con 0x7fad30102000 2024-03-28T16:04:05.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.035+0000 7fad36a00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad3010f190 con 0x7fad30104930 2024-03-28T16:04:05.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.035+0000 7fad36a00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad3010f490 con 0x7fad30107260 2024-03-28T16:04:05.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.035+0000 7fad35600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad30102000 0x7fad3010df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:05.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.036+0000 7fad36000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fad30107260 0x7fad3010aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:05.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.036+0000 7fad34c00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad30104930 0x7fad3010e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:05.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.036+0000 7fad35600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad30102000 0x7fad3010df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44470/0 (socket says 172.21.15.67:44470) 2024-03-28T16:04:05.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.036+0000 7fad35600640 1 -- 172.21.15.67:0/858206308 learned_addr learned my addr 172.21.15.67:0/858206308 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:04:05.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.036+0000 7fad34c00640 1 -- 172.21.15.67:0/858206308 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fad30107260 msgr2=0x7fad3010aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:05.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.036+0000 7fad34c00640 1 --2- 172.21.15.67:0/858206308 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fad30107260 0x7fad3010aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:05.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.036+0000 7fad34c00640 1 -- 172.21.15.67:0/858206308 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad30102000 msgr2=0x7fad3010df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:05.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.036+0000 7fad34c00640 1 --2- 172.21.15.67:0/858206308 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad30102000 0x7fad3010df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:05.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.036+0000 7fad34c00640 1 -- 172.21.15.67:0/858206308 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad3010b380 con 0x7fad30104930 2024-03-28T16:04:05.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.036+0000 7fad36000640 1 --2- 172.21.15.67:0/858206308 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fad30107260 0x7fad3010aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T16:04:05.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.037+0000 7fad34c00640 1 --2- 172.21.15.67:0/858206308 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad30104930 0x7fad3010e490 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fad30069360 tx=0x7fad1c00dbb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:04:05.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.037+0000 7fad2e000640 1 -- 172.21.15.67:0/858206308 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad1c017070 con 0x7fad30104930 2024-03-28T16:04:05.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.037+0000 7fad36a00640 1 -- 172.21.15.67:0/858206308 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad301a5e60 con 0x7fad30104930 2024-03-28T16:04:05.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.037+0000 7fad2e000640 1 -- 172.21.15.67:0/858206308 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fad1c0099d0 con 0x7fad30104930 2024-03-28T16:04:05.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.037+0000 7fad2e000640 1 -- 172.21.15.67:0/858206308 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad1c0047a0 con 0x7fad30104930 2024-03-28T16:04:05.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.038+0000 7fad36a00640 1 -- 172.21.15.67:0/858206308 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad301a6210 con 0x7fad30104930 2024-03-28T16:04:05.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.041+0000 7fad3ad7e740 1 Processor -- start 2024-03-28T16:04:05.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.041+0000 7fad3ad7e740 1 -- start start 2024-03-28T16:04:05.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.041+0000 7fad3ad7e740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559539e559f0 0x559539e55e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:05.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.041+0000 7fad3ad7e740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559539e56bc0 0x559539e57040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:05.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.041+0000 7fad3ad7e740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559539e57580 0x559539e5de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:05.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.042+0000 7fad3ad7e740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x559539dba2c0 con 0x559539e559f0 2024-03-28T16:04:05.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.042+0000 7fad3ad7e740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x559539d62c90 con 0x559539e57580 2024-03-28T16:04:05.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.042+0000 7fad3ad7e740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x559539db6360 con 0x559539e56bc0 2024-03-28T16:04:05.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.042+0000 7fad2e000640 1 -- 172.21.15.67:0/858206308 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fad1c004940 con 0x7fad30104930 2024-03-28T16:04:05.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.042+0000 7fad36000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559539e559f0 0x559539e55e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:05.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.042+0000 7fad36000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559539e559f0 0x559539e55e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44478/0 (socket says 172.21.15.67:44478) 2024-03-28T16:04:05.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.042+0000 7fad36000640 1 -- 172.21.15.67:0/2366732776 learned_addr learned my addr 172.21.15.67:0/2366732776 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:04:05.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.042+0000 7fad35600640 1 --2- 172.21.15.67:0/2366732776 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559539e56bc0 0x559539e57040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:05.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.042+0000 7fad36000640 1 -- 172.21.15.67:0/2366732776 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559539e56bc0 msgr2=0x559539e57040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:04:05.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.043+0000 7fad36000640 1 --2- 172.21.15.67:0/2366732776 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559539e56bc0 0x559539e57040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:05.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.043+0000 7fad36000640 1 -- 172.21.15.67:0/2366732776 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559539e57580 msgr2=0x559539e5de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:04:05.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.043+0000 7fad34c00640 1 --2- 172.21.15.67:0/2366732776 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559539e57580 0x559539e5de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:05.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.043+0000 7fad36000640 1 --2- 172.21.15.67:0/2366732776 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559539e57580 0x559539e5de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:05.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.043+0000 7fad36000640 1 -- 172.21.15.67:0/2366732776 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559539d87080 con 0x559539e559f0 2024-03-28T16:04:05.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.043+0000 7fad36000640 1 --2- 172.21.15.67:0/2366732776 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559539e559f0 0x559539e55e10 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fad28010750 tx=0x7fad28010b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55c36c094ac775d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:04:05.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.044+0000 7fad22600640 1 -- 172.21.15.67:0/2366732776 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad2800e8d0 con 0x559539e559f0 2024-03-28T16:04:05.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.044+0000 7fad22600640 1 -- 172.21.15.67:0/2366732776 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fad28012450 con 0x559539e559f0 2024-03-28T16:04:05.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.044+0000 7fad22600640 1 -- 172.21.15.67:0/2366732776 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad28011640 con 0x559539e559f0 2024-03-28T16:04:05.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.044+0000 7fad2e000640 1 --2- 172.21.15.67:0/858206308 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fad180a2800 0x7fad180a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:05.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.044+0000 7fad2e000640 1 -- 172.21.15.67:0/858206308 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fad1c0ce750 con 0x7fad30104930 2024-03-28T16:04:05.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.044+0000 7fad35600640 1 --2- 172.21.15.67:0/858206308 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fad180a2800 0x7fad180a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:05.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.044+0000 7fad3ad7e740 1 -- 172.21.15.67:0/2366732776 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559539e559f0 msgr2=0x559539e55e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:05.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.044+0000 7fad3ad7e740 1 --2- 172.21.15.67:0/2366732776 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559539e559f0 0x559539e55e10 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fad28010750 tx=0x7fad28010b50 comp rx=0 tx=0).stop 2024-03-28T16:04:05.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.045+0000 7fad3ad7e740 1 -- 172.21.15.67:0/2366732776 shutdown_connections 2024-03-28T16:04:05.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.045+0000 7fad3ad7e740 1 --2- 172.21.15.67:0/2366732776 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559539e57580 0x559539e5de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:05.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.045+0000 7fad3ad7e740 1 --2- 172.21.15.67:0/2366732776 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559539e56bc0 0x559539e57040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:05.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.045+0000 7fad3ad7e740 1 --2- 172.21.15.67:0/2366732776 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559539e559f0 0x559539e55e10 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:05.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.045+0000 7fad3ad7e740 1 -- 172.21.15.67:0/2366732776 >> 172.21.15.67:0/2366732776 conn(0x559539e50de0 msgr2=0x559539e511e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:04:05.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.045+0000 7fad3ad7e740 1 -- 172.21.15.67:0/2366732776 shutdown_connections 2024-03-28T16:04:05.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.045+0000 7fad3ad7e740 1 -- 172.21.15.67:0/2366732776 wait complete. 2024-03-28T16:04:05.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.045+0000 7fad35600640 1 --2- 172.21.15.67:0/858206308 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fad180a2800 0x7fad180a4cc0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x559539e51d50 tx=0x7fad2402c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:04:05.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.045+0000 7fad3ad7e740 1 -- start start 2024-03-28T16:04:05.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.046+0000 7fad3ad7e740 1 Processor -- start 2024-03-28T16:04:05.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.046+0000 7fad3ad7e740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559539e559f0 0x559539e5fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:05.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.046+0000 7fad3ad7e740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559539e56bc0 0x559539e601b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:05.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.046+0000 7fad3ad7e740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559539e57580 0x559539e671f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:05.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.046+0000 7fad3ad7e740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x559539e5e590 con 0x559539e559f0 2024-03-28T16:04:05.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.046+0000 7fad3ad7e740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x559539d760d0 con 0x559539e56bc0 2024-03-28T16:04:05.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.046+0000 7fad3ad7e740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x559539e5e710 con 0x559539e57580 2024-03-28T16:04:05.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.046+0000 7fad3ad7e740 10 client.0 _ll_register_callbacks cb 0x559539d7ae80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:04:05.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.046+0000 7fad35600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559539e559f0 0x559539e5fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:05.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.046+0000 7fad35600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559539e559f0 0x559539e5fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44486/0 (socket says 172.21.15.67:44486) 2024-03-28T16:04:05.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.046+0000 7fad35600640 1 -- 172.21.15.67:0/3060047313 learned_addr learned my addr 172.21.15.67:0/3060047313 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:04:05.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.047+0000 7fad34c00640 1 --2- 172.21.15.67:0/3060047313 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559539e56bc0 0x559539e601b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:05.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.047+0000 7fad36000640 1 --2- 172.21.15.67:0/3060047313 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559539e57580 0x559539e671f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:05.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.047+0000 7fad35600640 1 -- 172.21.15.67:0/3060047313 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559539e57580 msgr2=0x559539e671f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:04:05.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.047+0000 7fad35600640 1 --2- 172.21.15.67:0/3060047313 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559539e57580 0x559539e671f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:05.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.047+0000 7fad35600640 1 -- 172.21.15.67:0/3060047313 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559539e56bc0 msgr2=0x559539e601b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:04:05.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.047+0000 7fad35600640 1 --2- 172.21.15.67:0/3060047313 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559539e56bc0 0x559539e601b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:05.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.047+0000 7fad35600640 1 -- 172.21.15.67:0/3060047313 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559539ace3f0 con 0x559539e559f0 2024-03-28T16:04:05.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.047+0000 7fad35600640 1 --2- 172.21.15.67:0/3060047313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559539e559f0 0x559539e5fc70 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fad24036040 tx=0x7fad2402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=905647eadfc06a87 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:04:05.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.047+0000 7fad16a00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:04:05.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.048+0000 7fad16a00640 1 -- 172.21.15.67:0/3060047313 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad24050030 con 0x559539e559f0 2024-03-28T16:04:05.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.048+0000 7fad16a00640 1 -- 172.21.15.67:0/3060047313 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fad2403a040 con 0x559539e559f0 2024-03-28T16:04:05.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.048+0000 7fad3ad7e740 1 -- 172.21.15.67:0/3060047313 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fad28012450 con 0x559539e559f0 2024-03-28T16:04:05.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.048+0000 7fad16a00640 1 -- 172.21.15.67:0/3060047313 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fad240048d0 con 0x559539e559f0 2024-03-28T16:04:05.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.048+0000 7fad3ad7e740 10 client.15363 fetch_fsmap learned FSMap version 19 2024-03-28T16:04:05.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.048+0000 7fad3ad7e740 1 -- 172.21.15.67:0/3060047313 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x559539b987d0 con 0x559539e559f0 2024-03-28T16:04:05.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.048+0000 7fad16a00640 1 -- 172.21.15.67:0/3060047313 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fad28012450 con 0x559539e559f0 2024-03-28T16:04:05.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.048+0000 7fad16a00640 1 -- 172.21.15.67:0/3060047313 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fad24003c80 con 0x559539e559f0 2024-03-28T16:04:05.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.049+0000 7fad3ad7e740 10 client.15363 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:04:05.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.049+0000 7fad3ad7e740 10 client.15363 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:04:05.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.049+0000 7fad3ad7e740 20 client.15363 populate_metadata read hostname 'smithi067' 2024-03-28T16:04:05.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.049+0000 7fad3ad7e740 1 --2- 172.21.15.67:0/3060047313 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x559539e6b740 0x559539e6dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:05.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.049+0000 7fad3ad7e740 4 client.15363 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:04:05.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.049+0000 7fad3ad7e740 1 -- 172.21.15.67:0/3060047313 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559539dc7800 con 0x559539e6b740 2024-03-28T16:04:05.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.049+0000 7fad34c00640 1 --2- 172.21.15.67:0/3060047313 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x559539e6b740 0x559539e6dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:05.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.050+0000 7fad34c00640 1 --2- 172.21.15.67:0/3060047313 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x559539e6b740 0x559539e6dbc0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7dbd310c28cb5c56 server_cookie=e595a0c9ee8b9606 in_seq=0 out_seq=0 2024-03-28T16:04:05.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.050+0000 7fad16a00640 10 client.15363 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:04:05.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.052+0000 7fad16a00640 1 -- 172.21.15.67:0/3060047313 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x559539dc7800 con 0x559539e6b740 2024-03-28T16:04:05.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.052+0000 7fad16a00640 10 client.15363 handle_command_reply: tid=0 2024-03-28T16:04:05.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.052+0000 7fad3ad7e740 1 client.15363 shutdown 2024-03-28T16:04:05.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.052+0000 7fad3ad7e740 1 -- 172.21.15.67:0/3060047313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559539e559f0 msgr2=0x559539e5fc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:05.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.052+0000 7fad3ad7e740 1 --2- 172.21.15.67:0/3060047313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559539e559f0 0x559539e5fc70 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fad24036040 tx=0x7fad2402bc30 comp rx=0 tx=0).stop 2024-03-28T16:04:05.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.052+0000 7fad3ad7e740 1 -- 172.21.15.67:0/3060047313 shutdown_connections 2024-03-28T16:04:05.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.052+0000 7fad3ad7e740 1 --2- 172.21.15.67:0/3060047313 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x559539e6b740 0x559539e6dbc0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:05.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.052+0000 7fad3ad7e740 1 --2- 172.21.15.67:0/3060047313 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559539e57580 0x559539e671f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:05.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.053+0000 7fad16a00640 0 client.15363 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:04:05.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.052+0000 7fad3ad7e740 1 --2- 172.21.15.67:0/3060047313 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559539e56bc0 0x559539e601b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:05.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.053+0000 7fad3ad7e740 1 --2- 172.21.15.67:0/3060047313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559539e559f0 0x559539e5fc70 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:05.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.053+0000 7fad3ad7e740 1 -- 172.21.15.67:0/3060047313 >> 172.21.15.67:0/3060047313 conn(0x559539e50de0 msgr2=0x559539e50160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:04:05.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.053+0000 7fad3ad7e740 1 -- 172.21.15.67:0/3060047313 shutdown_connections 2024-03-28T16:04:05.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.053+0000 7fad3ad7e740 1 -- 172.21.15.67:0/3060047313 wait complete. 2024-03-28T16:04:05.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.053+0000 7fad3ad7e740 20 client.15363 trim_cache size 0 max 16384 2024-03-28T16:04:05.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.066+0000 7fad3ad7e740 1 Processor -- start 2024-03-28T16:04:05.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.067+0000 7fad3ad7e740 1 -- start start 2024-03-28T16:04:05.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.067+0000 7fad3ad7e740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559539e559f0 0x559539e59c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:05.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.067+0000 7fad3ad7e740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559539e56bc0 0x559539e671b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:05.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.067+0000 7fad3ad7e740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559539e57580 0x559539e5a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:05.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.067+0000 7fad3ad7e740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad301ace50 con 0x559539e56bc0 2024-03-28T16:04:05.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.067+0000 7fad3ad7e740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad301af410 con 0x559539e559f0 2024-03-28T16:04:05.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.067+0000 7fad3ad7e740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad301ad450 con 0x559539e57580 2024-03-28T16:04:05.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.067+0000 7fad34c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559539e56bc0 0x559539e671b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:05.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.067+0000 7fad34c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559539e56bc0 0x559539e671b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44494/0 (socket says 172.21.15.67:44494) 2024-03-28T16:04:05.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.067+0000 7fad34c00640 1 -- 172.21.15.67:0/1298357280 learned_addr learned my addr 172.21.15.67:0/1298357280 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:04:05.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.067+0000 7fad35600640 1 --2- 172.21.15.67:0/1298357280 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559539e559f0 0x559539e59c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:05.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.068+0000 7fad36000640 1 --2- 172.21.15.67:0/1298357280 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559539e57580 0x559539e5a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:05.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.068+0000 7fad34c00640 1 -- 172.21.15.67:0/1298357280 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559539e57580 msgr2=0x559539e5a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:04:05.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.068+0000 7fad34c00640 1 --2- 172.21.15.67:0/1298357280 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559539e57580 0x559539e5a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:05.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.068+0000 7fad34c00640 1 -- 172.21.15.67:0/1298357280 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559539e559f0 msgr2=0x559539e59c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:04:05.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.068+0000 7fad34c00640 1 --2- 172.21.15.67:0/1298357280 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559539e559f0 0x559539e59c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:05.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.068+0000 7fad34c00640 1 -- 172.21.15.67:0/1298357280 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559539e4c8b0 con 0x559539e56bc0 2024-03-28T16:04:05.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.068+0000 7fad36000640 1 --2- 172.21.15.67:0/1298357280 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559539e57580 0x559539e5a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:04:05.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:04:04 smithi067 ceph-mon[31707]: pgmap v454: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:04:05.071 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:04:04 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/2222891474' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:04:05.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.068+0000 7fad34c00640 1 --2- 172.21.15.67:0/1298357280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559539e56bc0 0x559539e671b0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fad1c0cc200 tx=0x7fad1c09aac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa374f1e017bf476 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:04:05.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.068+0000 7fad21c00640 1 -- 172.21.15.67:0/1298357280 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad1c09f1f0 con 0x559539e56bc0 2024-03-28T16:04:05.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.068+0000 7fad21c00640 1 -- 172.21.15.67:0/1298357280 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fad1c0a3840 con 0x559539e56bc0 2024-03-28T16:04:05.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.068+0000 7fad21c00640 1 -- 172.21.15.67:0/1298357280 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad1c0978a0 con 0x559539e56bc0 2024-03-28T16:04:05.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.069+0000 7fad3ad7e740 1 -- 172.21.15.67:0/1298357280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559539e56bc0 msgr2=0x559539e671b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:05.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.069+0000 7fad3ad7e740 1 --2- 172.21.15.67:0/1298357280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559539e56bc0 0x559539e671b0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fad1c0cc200 tx=0x7fad1c09aac0 comp rx=0 tx=0).stop 2024-03-28T16:04:05.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.069+0000 7fad3ad7e740 1 -- 172.21.15.67:0/1298357280 shutdown_connections 2024-03-28T16:04:05.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.069+0000 7fad3ad7e740 1 --2- 172.21.15.67:0/1298357280 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559539e57580 0x559539e5a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:05.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.069+0000 7fad3ad7e740 1 --2- 172.21.15.67:0/1298357280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559539e56bc0 0x559539e671b0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:05.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.069+0000 7fad3ad7e740 1 --2- 172.21.15.67:0/1298357280 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559539e559f0 0x559539e59c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:05.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.069+0000 7fad3ad7e740 1 -- 172.21.15.67:0/1298357280 >> 172.21.15.67:0/1298357280 conn(0x559539e50de0 msgr2=0x559539e548d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:04:05.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.069+0000 7fad3ad7e740 1 -- 172.21.15.67:0/1298357280 shutdown_connections 2024-03-28T16:04:05.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.069+0000 7fad3ad7e740 1 -- 172.21.15.67:0/1298357280 wait complete. 2024-03-28T16:04:05.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.070+0000 7fad3ad7e740 1 -- start start 2024-03-28T16:04:05.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.070+0000 7fad3ad7e740 1 Processor -- start 2024-03-28T16:04:05.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.070+0000 7fad3ad7e740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559539e559f0 0x559539e66ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:05.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.070+0000 7fad3ad7e740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559539e56bc0 0x559539e66ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:05.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.070+0000 7fad3ad7e740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559539e57580 0x559539e635c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:05.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.070+0000 7fad3ad7e740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad301ada90 con 0x559539e559f0 2024-03-28T16:04:05.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.070+0000 7fad3ad7e740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad301aedb0 con 0x559539e57580 2024-03-28T16:04:05.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.070+0000 7fad3ad7e740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad301ae0f0 con 0x559539e56bc0 2024-03-28T16:04:05.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.070+0000 7fad3ad7e740 10 client.0 _ll_register_callbacks cb 0x7fad24044050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:04:05.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.071+0000 7fad35600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559539e559f0 0x559539e66ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:05.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.071+0000 7fad35600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559539e559f0 0x559539e66ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44510/0 (socket says 172.21.15.67:44510) 2024-03-28T16:04:05.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.071+0000 7fad35600640 1 -- 172.21.15.67:0/2411756527 learned_addr learned my addr 172.21.15.67:0/2411756527 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:04:05.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.071+0000 7fad34c00640 1 --2- 172.21.15.67:0/2411756527 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559539e56bc0 0x559539e66ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:05.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.071+0000 7fad36000640 1 --2- 172.21.15.67:0/2411756527 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559539e57580 0x559539e635c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:05.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.071+0000 7fad35600640 1 -- 172.21.15.67:0/2411756527 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559539e56bc0 msgr2=0x559539e66ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:04:05.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.071+0000 7fad35600640 1 --2- 172.21.15.67:0/2411756527 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559539e56bc0 0x559539e66ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:05.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.071+0000 7fad35600640 1 -- 172.21.15.67:0/2411756527 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559539e57580 msgr2=0x559539e635c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:04:05.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.071+0000 7fad35600640 1 --2- 172.21.15.67:0/2411756527 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559539e57580 0x559539e635c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:05.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.071+0000 7fad35600640 1 -- 172.21.15.67:0/2411756527 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559539e63d30 con 0x559539e559f0 2024-03-28T16:04:05.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.071+0000 7fad35600640 1 --2- 172.21.15.67:0/2411756527 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559539e559f0 0x559539e66ab0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fad24006010 tx=0x7fad240043e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1dae5cb4ce000e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:04:05.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.071+0000 7fad16a00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:04:05.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.072+0000 7fad16a00640 1 -- 172.21.15.67:0/2411756527 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad24064020 con 0x559539e559f0 2024-03-28T16:04:05.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.072+0000 7fad16a00640 1 -- 172.21.15.67:0/2411756527 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fad2404ccf0 con 0x559539e559f0 2024-03-28T16:04:05.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.072+0000 7fad3ad7e740 1 -- 172.21.15.67:0/2411756527 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fad1c0a3840 con 0x559539e559f0 2024-03-28T16:04:05.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.072+0000 7fad16a00640 1 -- 172.21.15.67:0/2411756527 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fad24050040 con 0x559539e559f0 2024-03-28T16:04:05.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.072+0000 7fad3ad7e740 10 client.15369 fetch_fsmap learned FSMap version 19 2024-03-28T16:04:05.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.072+0000 7fad3ad7e740 1 -- 172.21.15.67:0/2411756527 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x559539ec7ce0 con 0x559539e559f0 2024-03-28T16:04:05.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.072+0000 7fad16a00640 1 -- 172.21.15.67:0/2411756527 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fad1c0a3840 con 0x559539e559f0 2024-03-28T16:04:05.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.073+0000 7fad16a00640 1 -- 172.21.15.67:0/2411756527 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fad24003ab0 con 0x559539e559f0 2024-03-28T16:04:05.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.073+0000 7fad3ad7e740 10 client.15369 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:04:05.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.073+0000 7fad3ad7e740 10 client.15369 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:04:05.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.073+0000 7fad3ad7e740 20 client.15369 populate_metadata read hostname 'smithi067' 2024-03-28T16:04:05.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.073+0000 7fad3ad7e740 1 --2- 172.21.15.67:0/2411756527 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x559539ec85e0 0x559539ecaa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:05.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.073+0000 7fad3ad7e740 4 client.15369 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T16:04:05.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.073+0000 7fad3ad7e740 1 -- 172.21.15.67:0/2411756527 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x559539dc1b50 con 0x559539ec85e0 2024-03-28T16:04:05.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.073+0000 7fad34c00640 1 --2- 172.21.15.67:0/2411756527 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x559539ec85e0 0x559539ecaa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:05.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.074+0000 7fad34c00640 1 --2- 172.21.15.67:0/2411756527 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x559539ec85e0 0x559539ecaa00 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=83f8a81663e62b45 server_cookie=15aee471c374b6d9 in_seq=0 out_seq=0 2024-03-28T16:04:05.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.074+0000 7fad16a00640 10 client.15369 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:04:05.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.075+0000 7fad16a00640 1 -- 172.21.15.67:0/2411756527 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x559539dc1b50 con 0x559539ec85e0 2024-03-28T16:04:05.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.075+0000 7fad16a00640 10 client.15369 handle_command_reply: tid=0 2024-03-28T16:04:05.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.075+0000 7fad3ad7e740 1 client.15369 shutdown 2024-03-28T16:04:05.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.075+0000 7fad3ad7e740 1 -- 172.21.15.67:0/2411756527 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559539e559f0 msgr2=0x559539e66ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:05.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.075+0000 7fad3ad7e740 1 --2- 172.21.15.67:0/2411756527 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559539e559f0 0x559539e66ab0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fad24006010 tx=0x7fad240043e0 comp rx=0 tx=0).stop 2024-03-28T16:04:05.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.075+0000 7fad3ad7e740 1 -- 172.21.15.67:0/2411756527 shutdown_connections 2024-03-28T16:04:05.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.075+0000 7fad3ad7e740 1 --2- 172.21.15.67:0/2411756527 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x559539ec85e0 0x559539ecaa00 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:05.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.075+0000 7fad3ad7e740 1 --2- 172.21.15.67:0/2411756527 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559539e57580 0x559539e635c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:05.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.075+0000 7fad3ad7e740 1 --2- 172.21.15.67:0/2411756527 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559539e56bc0 0x559539e66ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:05.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.075+0000 7fad3ad7e740 1 --2- 172.21.15.67:0/2411756527 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559539e559f0 0x559539e66ab0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:05.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.075+0000 7fad16a00640 0 client.15369 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:04:05.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.075+0000 7fad3ad7e740 1 -- 172.21.15.67:0/2411756527 >> 172.21.15.67:0/2411756527 conn(0x559539e50de0 msgr2=0x559539e4f8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:04:05.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.075+0000 7fad3ad7e740 1 -- 172.21.15.67:0/2411756527 shutdown_connections 2024-03-28T16:04:05.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.075+0000 7fad3ad7e740 1 -- 172.21.15.67:0/2411756527 wait complete. 2024-03-28T16:04:05.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.076+0000 7fad3ad7e740 20 client.15369 trim_cache size 0 max 16384 2024-03-28T16:04:05.081 INFO:teuthology.orchestra.run.smithi067.stdout:[] 2024-03-28T16:04:05.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.076+0000 7fad36a00640 1 -- 172.21.15.67:0/858206308 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fad180a2800 msgr2=0x7fad180a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:05.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.076+0000 7fad36a00640 1 --2- 172.21.15.67:0/858206308 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fad180a2800 0x7fad180a4cc0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x559539e51d50 tx=0x7fad2402c7e0 comp rx=0 tx=0).stop 2024-03-28T16:04:05.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.076+0000 7fad36a00640 1 -- 172.21.15.67:0/858206308 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad30104930 msgr2=0x7fad3010e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:05.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.076+0000 7fad36a00640 1 --2- 172.21.15.67:0/858206308 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad30104930 0x7fad3010e490 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fad30069360 tx=0x7fad1c00dbb0 comp rx=0 tx=0).stop 2024-03-28T16:04:05.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.076+0000 7fad36a00640 1 -- 172.21.15.67:0/858206308 shutdown_connections 2024-03-28T16:04:05.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.076+0000 7fad36a00640 1 --2- 172.21.15.67:0/858206308 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fad180a2800 0x7fad180a4cc0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:05.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.076+0000 7fad36a00640 1 --2- 172.21.15.67:0/858206308 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fad30107260 0x7fad3010aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:05.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.076+0000 7fad36a00640 1 --2- 172.21.15.67:0/858206308 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad30104930 0x7fad3010e490 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:05.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.076+0000 7fad36a00640 1 --2- 172.21.15.67:0/858206308 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad30102000 0x7fad3010df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:05.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.076+0000 7fad36a00640 1 -- 172.21.15.67:0/858206308 >> 172.21.15.67:0/858206308 conn(0x7fad300fba10 msgr2=0x7fad30102ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:04:05.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.076+0000 7fad36a00640 1 -- 172.21.15.67:0/858206308 shutdown_connections 2024-03-28T16:04:05.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:05.076+0000 7fad36a00640 1 -- 172.21.15.67:0/858206308 wait complete. 2024-03-28T16:04:05.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:04:04 smithi150 ceph-mon[34825]: pgmap v454: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:04:05.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:04:04 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/2222891474' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:04:05.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:04:04 smithi183 ceph-mon[35580]: pgmap v454: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:04:05.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:04:04 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/2222891474' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:04:06.098 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T16:04:06.099 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T16:04:06.099 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T16:04:06.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.336+0000 7f063d000640 1 -- 172.21.15.67:0/3094808653 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0638107cc0 msgr2=0x7f06381080e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:06.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.336+0000 7f063d000640 1 --2- 172.21.15.67:0/3094808653 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0638107cc0 0x7f06381080e0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f0628009900 tx=0x7f062802bc30 comp rx=0 tx=0).stop 2024-03-28T16:04:06.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.336+0000 7f063d000640 1 -- 172.21.15.67:0/3094808653 shutdown_connections 2024-03-28T16:04:06.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.336+0000 7f063d000640 1 --2- 172.21.15.67:0/3094808653 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0638109870 0x7f0638110100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:06.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.336+0000 7f063d000640 1 --2- 172.21.15.67:0/3094808653 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0638108eb0 0x7f0638109330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:06.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.336+0000 7f063d000640 1 --2- 172.21.15.67:0/3094808653 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0638107cc0 0x7f06381080e0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:06.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.336+0000 7f063d000640 1 -- 172.21.15.67:0/3094808653 >> 172.21.15.67:0/3094808653 conn(0x7f0638076c60 msgr2=0x7f06380790a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:04:06.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.336+0000 7f063d000640 1 -- 172.21.15.67:0/3094808653 shutdown_connections 2024-03-28T16:04:06.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.336+0000 7f063d000640 1 -- 172.21.15.67:0/3094808653 wait complete. 2024-03-28T16:04:06.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.337+0000 7f063d000640 1 Processor -- start 2024-03-28T16:04:06.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.338+0000 7f063d000640 1 -- start start 2024-03-28T16:04:06.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.338+0000 7f063d000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0638107cc0 0x7f063819df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:06.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.338+0000 7f063d000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0638108eb0 0x7f063819e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:06.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.338+0000 7f063d000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0638109870 0x7f06381a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:06.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.338+0000 7f063d000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0638113810 con 0x7f0638108eb0 2024-03-28T16:04:06.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.338+0000 7f063d000640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0638113690 con 0x7f0638109870 2024-03-28T16:04:06.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.338+0000 7f063d000640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0638113990 con 0x7f0638107cc0 2024-03-28T16:04:06.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.338+0000 7f062d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0638108eb0 0x7f063819e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:06.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.338+0000 7f062d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0638108eb0 0x7f063819e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44524/0 (socket says 172.21.15.67:44524) 2024-03-28T16:04:06.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.338+0000 7f062d400640 1 -- 172.21.15.67:0/199296220 learned_addr learned my addr 172.21.15.67:0/199296220 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:04:06.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.339+0000 7f0635e00640 1 --2- 172.21.15.67:0/199296220 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0638107cc0 0x7f063819df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:06.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.339+0000 7f0636800640 1 --2- 172.21.15.67:0/199296220 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0638109870 0x7f06381a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:06.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.339+0000 7f062d400640 1 -- 172.21.15.67:0/199296220 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0638107cc0 msgr2=0x7f063819df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:06.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.339+0000 7f062d400640 1 --2- 172.21.15.67:0/199296220 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0638107cc0 0x7f063819df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:06.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.339+0000 7f062d400640 1 -- 172.21.15.67:0/199296220 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0638109870 msgr2=0x7f06381a5500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:06.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.339+0000 7f062d400640 1 --2- 172.21.15.67:0/199296220 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0638109870 0x7f06381a5500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:06.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.339+0000 7f062d400640 1 -- 172.21.15.67:0/199296220 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06381a5b70 con 0x7f0638108eb0 2024-03-28T16:04:06.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.339+0000 7f062d400640 1 --2- 172.21.15.67:0/199296220 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0638108eb0 0x7f063819e490 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f062000d950 tx=0x7f062000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:04:06.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.339+0000 7f062f400640 1 -- 172.21.15.67:0/199296220 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0620017070 con 0x7f0638108eb0 2024-03-28T16:04:06.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.339+0000 7f062f400640 1 -- 172.21.15.67:0/199296220 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0620009c30 con 0x7f0638108eb0 2024-03-28T16:04:06.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.340+0000 7f063d000640 1 -- 172.21.15.67:0/199296220 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06381a5e60 con 0x7f0638108eb0 2024-03-28T16:04:06.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.340+0000 7f062f400640 1 -- 172.21.15.67:0/199296220 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f062000f620 con 0x7f0638108eb0 2024-03-28T16:04:06.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.340+0000 7f063d000640 1 -- 172.21.15.67:0/199296220 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06381a6320 con 0x7f0638108eb0 2024-03-28T16:04:06.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.343+0000 7f063fc55740 1 Processor -- start 2024-03-28T16:04:06.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.343+0000 7f062f400640 1 -- 172.21.15.67:0/199296220 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f06200109a0 con 0x7f0638108eb0 2024-03-28T16:04:06.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.343+0000 7f063fc55740 1 -- start start 2024-03-28T16:04:06.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.343+0000 7f063fc55740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c862e20a0 0x563c862e24c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:06.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.343+0000 7f063fc55740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563c862e2a90 0x563c862e7590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:06.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.343+0000 7f063fc55740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c862e7c00 0x563c862ea090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:06.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.343+0000 7f063fc55740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c86247eb0 con 0x563c862e7c00 2024-03-28T16:04:06.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.343+0000 7f063fc55740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c861f0a60 con 0x563c862e2a90 2024-03-28T16:04:06.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.343+0000 7f063fc55740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c86243f50 con 0x563c862e20a0 2024-03-28T16:04:06.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.344+0000 7f062d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c862e7c00 0x563c862ea090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:06.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.344+0000 7f062d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c862e7c00 0x563c862ea090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44536/0 (socket says 172.21.15.67:44536) 2024-03-28T16:04:06.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.344+0000 7f062d400640 1 -- 172.21.15.67:0/4041183211 learned_addr learned my addr 172.21.15.67:0/4041183211 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:04:06.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.344+0000 7f0636800640 1 --2- 172.21.15.67:0/4041183211 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c862e20a0 0x563c862e24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:06.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.344+0000 7f0635e00640 1 --2- 172.21.15.67:0/4041183211 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563c862e2a90 0x563c862e7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:06.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.344+0000 7f062d400640 1 -- 172.21.15.67:0/4041183211 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c862e20a0 msgr2=0x563c862e24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:04:06.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.344+0000 7f062d400640 1 --2- 172.21.15.67:0/4041183211 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c862e20a0 0x563c862e24c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:06.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.344+0000 7f062d400640 1 -- 172.21.15.67:0/4041183211 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563c862e2a90 msgr2=0x563c862e7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:04:06.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.344+0000 7f062d400640 1 --2- 172.21.15.67:0/4041183211 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563c862e2a90 0x563c862e7590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:06.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.344+0000 7f062d400640 1 -- 172.21.15.67:0/4041183211 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563c860247e0 con 0x563c862e7c00 2024-03-28T16:04:06.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.345+0000 7f062d400640 1 --2- 172.21.15.67:0/4041183211 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c862e7c00 0x563c862ea090 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f06200d4c20 tx=0x7f06200d6670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48a77e5a3613e523 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:04:06.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.345+0000 7f062f400640 1 --2- 172.21.15.67:0/199296220 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f061c0a2800 0x7f061c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:06.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.345+0000 7f0627400640 1 -- 172.21.15.67:0/4041183211 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06200dd070 con 0x563c862e7c00 2024-03-28T16:04:06.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.345+0000 7f0627400640 1 -- 172.21.15.67:0/4041183211 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0620020c20 con 0x563c862e7c00 2024-03-28T16:04:06.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.345+0000 7f0627400640 1 -- 172.21.15.67:0/4041183211 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f062009aed0 con 0x563c862e7c00 2024-03-28T16:04:06.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.345+0000 7f0636800640 1 --2- 172.21.15.67:0/199296220 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f061c0a2800 0x7f061c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:06.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.345+0000 7f062f400640 1 -- 172.21.15.67:0/199296220 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f06200ce7b0 con 0x7f0638108eb0 2024-03-28T16:04:06.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.346+0000 7f063fc55740 1 -- 172.21.15.67:0/4041183211 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c862e7c00 msgr2=0x563c862ea090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:06.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.346+0000 7f063fc55740 1 --2- 172.21.15.67:0/4041183211 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c862e7c00 0x563c862ea090 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f06200d4c20 tx=0x7f06200d6670 comp rx=0 tx=0).stop 2024-03-28T16:04:06.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.346+0000 7f0636800640 1 --2- 172.21.15.67:0/199296220 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f061c0a2800 0x7f061c0a4cc0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x563c862e32f0 tx=0x7f0630008370 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:04:06.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.346+0000 7f063fc55740 1 -- 172.21.15.67:0/4041183211 shutdown_connections 2024-03-28T16:04:06.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.346+0000 7f063fc55740 1 --2- 172.21.15.67:0/4041183211 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c862e7c00 0x563c862ea090 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:06.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.346+0000 7f063fc55740 1 --2- 172.21.15.67:0/4041183211 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563c862e2a90 0x563c862e7590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:06.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.346+0000 7f063fc55740 1 --2- 172.21.15.67:0/4041183211 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c862e20a0 0x563c862e24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:06.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.346+0000 7f063fc55740 1 -- 172.21.15.67:0/4041183211 >> 172.21.15.67:0/4041183211 conn(0x563c862dd610 msgr2=0x563c862dda10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:04:06.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.346+0000 7f063fc55740 1 -- 172.21.15.67:0/4041183211 shutdown_connections 2024-03-28T16:04:06.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.346+0000 7f063fc55740 1 -- 172.21.15.67:0/4041183211 wait complete. 2024-03-28T16:04:06.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.347+0000 7f063fc55740 1 -- start start 2024-03-28T16:04:06.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.347+0000 7f063fc55740 1 Processor -- start 2024-03-28T16:04:06.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.347+0000 7f063fc55740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c862e20a0 0x563c862ec220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:06.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.347+0000 7f063fc55740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563c862e2a90 0x563c862ec760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:06.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.347+0000 7f063fc55740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c862e7c00 0x563c862f37a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:06.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.347+0000 7f063fc55740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c862eabb0 con 0x563c862e20a0 2024-03-28T16:04:06.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.347+0000 7f063fc55740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c86203eb0 con 0x563c862e2a90 2024-03-28T16:04:06.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.347+0000 7f063fc55740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c862ead30 con 0x563c862e7c00 2024-03-28T16:04:06.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.347+0000 7f063fc55740 10 client.0 _ll_register_callbacks cb 0x563c861f8150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:04:06.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.348+0000 7f0635e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c862e20a0 0x563c862ec220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:06.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.348+0000 7f0635e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c862e20a0 0x563c862ec220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44538/0 (socket says 172.21.15.67:44538) 2024-03-28T16:04:06.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.348+0000 7f0635e00640 1 -- 172.21.15.67:0/4000191061 learned_addr learned my addr 172.21.15.67:0/4000191061 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:04:06.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.348+0000 7f0636800640 1 --2- 172.21.15.67:0/4000191061 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c862e7c00 0x563c862f37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:06.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.348+0000 7f062d400640 1 --2- 172.21.15.67:0/4000191061 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563c862e2a90 0x563c862ec760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:06.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.348+0000 7f0635e00640 1 -- 172.21.15.67:0/4000191061 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c862e7c00 msgr2=0x563c862f37a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:04:06.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.348+0000 7f0635e00640 1 --2- 172.21.15.67:0/4000191061 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c862e7c00 0x563c862f37a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:06.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.348+0000 7f0635e00640 1 -- 172.21.15.67:0/4000191061 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563c862e2a90 msgr2=0x563c862ec760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:04:06.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.348+0000 7f0635e00640 1 --2- 172.21.15.67:0/4000191061 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563c862e2a90 0x563c862ec760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:06.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.348+0000 7f0635e00640 1 -- 172.21.15.67:0/4000191061 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563c86084f70 con 0x563c862e20a0 2024-03-28T16:04:06.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.348+0000 7f062d400640 1 --2- 172.21.15.67:0/4000191061 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563c862e2a90 0x563c862ec760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:04:06.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.349+0000 7f0635e00640 1 --2- 172.21.15.67:0/4000191061 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c862e20a0 0x563c862ec220 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f062802c840 tx=0x7f0628004250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de65147082b0eea1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:04:06.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.349+0000 7f061be00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:04:06.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.349+0000 7f061be00640 1 -- 172.21.15.67:0/4000191061 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0628005230 con 0x563c862e20a0 2024-03-28T16:04:06.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.349+0000 7f061be00640 1 -- 172.21.15.67:0/4000191061 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f06280053d0 con 0x563c862e20a0 2024-03-28T16:04:06.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.349+0000 7f063fc55740 1 -- 172.21.15.67:0/4000191061 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0620020c20 con 0x563c862e20a0 2024-03-28T16:04:06.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.349+0000 7f061be00640 1 -- 172.21.15.67:0/4000191061 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f06280055d0 con 0x563c862e20a0 2024-03-28T16:04:06.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.349+0000 7f063fc55740 10 client.15381 fetch_fsmap learned FSMap version 19 2024-03-28T16:04:06.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.349+0000 7f063fc55740 1 -- 172.21.15.67:0/4000191061 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x563c85f1c170 con 0x563c862e20a0 2024-03-28T16:04:06.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.349+0000 7f061be00640 1 -- 172.21.15.67:0/4000191061 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0620020c20 con 0x563c862e20a0 2024-03-28T16:04:06.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.350+0000 7f061be00640 1 -- 172.21.15.67:0/4000191061 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f0628003990 con 0x563c862e20a0 2024-03-28T16:04:06.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.350+0000 7f063fc55740 10 client.15381 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:04:06.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.350+0000 7f063fc55740 10 client.15381 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:04:06.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.350+0000 7f063fc55740 20 client.15381 populate_metadata read hostname 'smithi067' 2024-03-28T16:04:06.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.350+0000 7f063fc55740 1 --2- 172.21.15.67:0/4000191061 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x563c862f8100 0x563c862fa5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:06.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.350+0000 7f063fc55740 4 client.15381 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:04:06.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.350+0000 7f063fc55740 1 -- 172.21.15.67:0/4000191061 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563c862553f0 con 0x563c862f8100 2024-03-28T16:04:06.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.350+0000 7f062d400640 1 --2- 172.21.15.67:0/4000191061 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x563c862f8100 0x563c862fa5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:06.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.351+0000 7f062d400640 1 --2- 172.21.15.67:0/4000191061 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x563c862f8100 0x563c862fa5a0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2989cc53056436e6 server_cookie=f0f85baf3eb9188b in_seq=0 out_seq=0 2024-03-28T16:04:06.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.351+0000 7f061be00640 10 client.15381 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:04:06.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.354+0000 7f061be00640 1 -- 172.21.15.67:0/4000191061 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x563c862553f0 con 0x563c862f8100 2024-03-28T16:04:06.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.354+0000 7f061be00640 10 client.15381 handle_command_reply: tid=0 2024-03-28T16:04:06.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.354+0000 7f063fc55740 1 client.15381 shutdown 2024-03-28T16:04:06.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.354+0000 7f063fc55740 1 -- 172.21.15.67:0/4000191061 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c862e20a0 msgr2=0x563c862ec220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:06.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.354+0000 7f063fc55740 1 --2- 172.21.15.67:0/4000191061 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c862e20a0 0x563c862ec220 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f062802c840 tx=0x7f0628004250 comp rx=0 tx=0).stop 2024-03-28T16:04:06.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.354+0000 7f063fc55740 1 -- 172.21.15.67:0/4000191061 shutdown_connections 2024-03-28T16:04:06.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.354+0000 7f063fc55740 1 --2- 172.21.15.67:0/4000191061 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x563c862f8100 0x563c862fa5a0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:06.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.354+0000 7f063fc55740 1 --2- 172.21.15.67:0/4000191061 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c862e7c00 0x563c862f37a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:06.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.354+0000 7f063fc55740 1 --2- 172.21.15.67:0/4000191061 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563c862e2a90 0x563c862ec760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:06.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.354+0000 7f063fc55740 1 --2- 172.21.15.67:0/4000191061 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c862e20a0 0x563c862ec220 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:06.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.354+0000 7f061be00640 0 client.15381 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:04:06.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.354+0000 7f063fc55740 1 -- 172.21.15.67:0/4000191061 >> 172.21.15.67:0/4000191061 conn(0x563c862dd610 msgr2=0x563c862dc830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:04:06.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.354+0000 7f063fc55740 1 -- 172.21.15.67:0/4000191061 shutdown_connections 2024-03-28T16:04:06.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.354+0000 7f063fc55740 1 -- 172.21.15.67:0/4000191061 wait complete. 2024-03-28T16:04:06.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.354+0000 7f063fc55740 20 client.15381 trim_cache size 0 max 16384 2024-03-28T16:04:06.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.372+0000 7f063fc55740 1 Processor -- start 2024-03-28T16:04:06.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.372+0000 7f063fc55740 1 -- start start 2024-03-28T16:04:06.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.372+0000 7f063fc55740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c862e20a0 0x563c862dc6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:06.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.372+0000 7f063fc55740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563c862e2a90 0x563c862dcbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:06.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.372+0000 7f063fc55740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c862e7c00 0x563c862fee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:06.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.372+0000 7f063fc55740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06381ad050 con 0x563c862e20a0 2024-03-28T16:04:06.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.372+0000 7f063fc55740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06381af610 con 0x563c862e2a90 2024-03-28T16:04:06.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.372+0000 7f063fc55740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06381ad650 con 0x563c862e7c00 2024-03-28T16:04:06.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.373+0000 7f0635e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c862e20a0 0x563c862dc6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:06.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.373+0000 7f062d400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563c862e2a90 0x563c862dcbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:06.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.373+0000 7f0635e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c862e20a0 0x563c862dc6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44540/0 (socket says 172.21.15.67:44540) 2024-03-28T16:04:06.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.373+0000 7f0635e00640 1 -- 172.21.15.67:0/3004139914 learned_addr learned my addr 172.21.15.67:0/3004139914 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:04:06.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.373+0000 7f0636800640 1 --2- 172.21.15.67:0/3004139914 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c862e7c00 0x563c862fee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:06.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.374+0000 7f062d400640 1 -- 172.21.15.67:0/3004139914 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c862e7c00 msgr2=0x563c862fee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:04:06.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.374+0000 7f062d400640 1 --2- 172.21.15.67:0/3004139914 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c862e7c00 0x563c862fee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:06.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.374+0000 7f062d400640 1 -- 172.21.15.67:0/3004139914 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c862e20a0 msgr2=0x563c862dc6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:04:06.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.374+0000 7f062d400640 1 --2- 172.21.15.67:0/3004139914 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c862e20a0 0x563c862dc6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:06.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.374+0000 7f062d400640 1 -- 172.21.15.67:0/3004139914 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563c862e2e70 con 0x563c862e2a90 2024-03-28T16:04:06.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.374+0000 7f062d400640 1 --2- 172.21.15.67:0/3004139914 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563c862e2a90 0x563c862dcbf0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0620048d00 tx=0x7f06200d4530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8eb787c32b88654d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:04:06.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.374+0000 7f0626a00640 1 -- 172.21.15.67:0/3004139914 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f062004a4b0 con 0x563c862e2a90 2024-03-28T16:04:06.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.374+0000 7f0626a00640 1 -- 172.21.15.67:0/3004139914 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f06200bb0b0 con 0x563c862e2a90 2024-03-28T16:04:06.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.374+0000 7f0626a00640 1 -- 172.21.15.67:0/3004139914 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f062004a680 con 0x563c862e2a90 2024-03-28T16:04:06.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.375+0000 7f063fc55740 1 -- 172.21.15.67:0/3004139914 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563c862e2a90 msgr2=0x563c862dcbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:06.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.375+0000 7f063fc55740 1 --2- 172.21.15.67:0/3004139914 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563c862e2a90 0x563c862dcbf0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0620048d00 tx=0x7f06200d4530 comp rx=0 tx=0).stop 2024-03-28T16:04:06.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.375+0000 7f063fc55740 1 -- 172.21.15.67:0/3004139914 shutdown_connections 2024-03-28T16:04:06.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.375+0000 7f063fc55740 1 --2- 172.21.15.67:0/3004139914 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c862e7c00 0x563c862fee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:06.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.375+0000 7f063fc55740 1 --2- 172.21.15.67:0/3004139914 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563c862e2a90 0x563c862dcbf0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:06.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.375+0000 7f063fc55740 1 --2- 172.21.15.67:0/3004139914 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c862e20a0 0x563c862dc6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:06.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.375+0000 7f063fc55740 1 -- 172.21.15.67:0/3004139914 >> 172.21.15.67:0/3004139914 conn(0x563c862dd610 msgr2=0x563c862dbfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:04:06.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.375+0000 7f063fc55740 1 -- 172.21.15.67:0/3004139914 shutdown_connections 2024-03-28T16:04:06.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.375+0000 7f063fc55740 1 -- 172.21.15.67:0/3004139914 wait complete. 2024-03-28T16:04:06.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.376+0000 7f063fc55740 1 -- start start 2024-03-28T16:04:06.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.376+0000 7f063fc55740 1 Processor -- start 2024-03-28T16:04:06.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.376+0000 7f063fc55740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c862e20a0 0x563c862f2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:06.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.376+0000 7f063fc55740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563c862e2a90 0x563c862f1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:06.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.376+0000 7f063fc55740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c862e7c00 0x563c862f1640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:06.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.376+0000 7f063fc55740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06381adc90 con 0x563c862e7c00 2024-03-28T16:04:06.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.376+0000 7f063fc55740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06381aefb0 con 0x563c862e2a90 2024-03-28T16:04:06.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.376+0000 7f063fc55740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06381ae2f0 con 0x563c862e20a0 2024-03-28T16:04:06.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.376+0000 7f063fc55740 10 client.0 _ll_register_callbacks cb 0x7f0628009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:04:06.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.377+0000 7f0636800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c862e7c00 0x563c862f1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:06.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.377+0000 7f0636800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c862e7c00 0x563c862f1640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44548/0 (socket says 172.21.15.67:44548) 2024-03-28T16:04:06.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.377+0000 7f0636800640 1 -- 172.21.15.67:0/794812570 learned_addr learned my addr 172.21.15.67:0/794812570 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:04:06.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.377+0000 7f0635e00640 1 --2- 172.21.15.67:0/794812570 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c862e20a0 0x563c862f2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:06.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.377+0000 7f062d400640 1 --2- 172.21.15.67:0/794812570 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563c862e2a90 0x563c862f1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:06.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.377+0000 7f0636800640 1 -- 172.21.15.67:0/794812570 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c862e20a0 msgr2=0x563c862f2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:04:06.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.377+0000 7f0636800640 1 --2- 172.21.15.67:0/794812570 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c862e20a0 0x563c862f2aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:06.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.377+0000 7f0636800640 1 -- 172.21.15.67:0/794812570 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563c862e2a90 msgr2=0x563c862f1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:04:06.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.377+0000 7f0636800640 1 --2- 172.21.15.67:0/794812570 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563c862e2a90 0x563c862f1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:06.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.377+0000 7f0636800640 1 -- 172.21.15.67:0/794812570 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563c862f1b80 con 0x563c862e7c00 2024-03-28T16:04:06.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.377+0000 7f0636800640 1 --2- 172.21.15.67:0/794812570 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c862e7c00 0x563c862f1640 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f0630011f30 tx=0x7f0630004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b8629c2bc285042 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:04:06.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.377+0000 7f061be00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:04:06.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.378+0000 7f061be00640 1 -- 172.21.15.67:0/794812570 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f063001e630 con 0x563c862e7c00 2024-03-28T16:04:06.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.378+0000 7f061be00640 1 -- 172.21.15.67:0/794812570 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f063001e7d0 con 0x563c862e7c00 2024-03-28T16:04:06.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.378+0000 7f063fc55740 1 -- 172.21.15.67:0/794812570 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f06200bb0b0 con 0x563c862e7c00 2024-03-28T16:04:06.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.378+0000 7f061be00640 1 -- 172.21.15.67:0/794812570 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f063001e9d0 con 0x563c862e7c00 2024-03-28T16:04:06.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.378+0000 7f063fc55740 10 client.15387 fetch_fsmap learned FSMap version 19 2024-03-28T16:04:06.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.378+0000 7f063fc55740 1 -- 172.21.15.67:0/794812570 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x563c863547b0 con 0x563c862e7c00 2024-03-28T16:04:06.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.378+0000 7f061be00640 1 -- 172.21.15.67:0/794812570 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f06200bb0b0 con 0x563c862e7c00 2024-03-28T16:04:06.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.379+0000 7f061be00640 1 -- 172.21.15.67:0/794812570 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f0630029c60 con 0x563c862e7c00 2024-03-28T16:04:06.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.379+0000 7f063fc55740 10 client.15387 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:04:06.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.379+0000 7f063fc55740 10 client.15387 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:04:06.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.379+0000 7f063fc55740 20 client.15387 populate_metadata read hostname 'smithi067' 2024-03-28T16:04:06.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.379+0000 7f063fc55740 1 --2- 172.21.15.67:0/794812570 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x563c86355170 0x563c86357590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:06.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.379+0000 7f063fc55740 4 client.15387 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T16:04:06.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.379+0000 7f063fc55740 1 -- 172.21.15.67:0/794812570 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x563c8624f740 con 0x563c86355170 2024-03-28T16:04:06.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.379+0000 7f0635e00640 1 --2- 172.21.15.67:0/794812570 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x563c86355170 0x563c86357590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:06.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.380+0000 7f0635e00640 1 --2- 172.21.15.67:0/794812570 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x563c86355170 0x563c86357590 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17d71021f16c69ef server_cookie=e84ae19af71b4837 in_seq=0 out_seq=0 2024-03-28T16:04:06.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.380+0000 7f061be00640 10 client.15387 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:04:06.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.383+0000 7f061be00640 1 -- 172.21.15.67:0/794812570 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x563c8624f740 con 0x563c86355170 2024-03-28T16:04:06.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.383+0000 7f061be00640 10 client.15387 handle_command_reply: tid=0 2024-03-28T16:04:06.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.383+0000 7f063fc55740 1 client.15387 shutdown 2024-03-28T16:04:06.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.384+0000 7f063fc55740 1 -- 172.21.15.67:0/794812570 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c862e7c00 msgr2=0x563c862f1640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:06.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.384+0000 7f063fc55740 1 --2- 172.21.15.67:0/794812570 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c862e7c00 0x563c862f1640 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f0630011f30 tx=0x7f0630004a40 comp rx=0 tx=0).stop 2024-03-28T16:04:06.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.384+0000 7f063fc55740 1 -- 172.21.15.67:0/794812570 shutdown_connections 2024-03-28T16:04:06.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.384+0000 7f063fc55740 1 --2- 172.21.15.67:0/794812570 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x563c86355170 0x563c86357590 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:06.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.384+0000 7f063fc55740 1 --2- 172.21.15.67:0/794812570 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c862e7c00 0x563c862f1640 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:06.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.384+0000 7f063fc55740 1 --2- 172.21.15.67:0/794812570 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563c862e2a90 0x563c862f1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:06.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.384+0000 7f063fc55740 1 --2- 172.21.15.67:0/794812570 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c862e20a0 0x563c862f2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:06.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.384+0000 7f061be00640 0 client.15387 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:04:06.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.384+0000 7f063fc55740 1 -- 172.21.15.67:0/794812570 >> 172.21.15.67:0/794812570 conn(0x563c862dd610 msgr2=0x563c862dc830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:04:06.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.384+0000 7f063fc55740 1 -- 172.21.15.67:0/794812570 shutdown_connections 2024-03-28T16:04:06.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.384+0000 7f063fc55740 1 -- 172.21.15.67:0/794812570 wait complete. 2024-03-28T16:04:06.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.384+0000 7f063fc55740 20 client.15387 trim_cache size 0 max 16384 2024-03-28T16:04:06.386 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-28T16:04:06.386 INFO:teuthology.orchestra.run.smithi067.stdout: "return_code": 0, 2024-03-28T16:04:06.387 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_tag": "cf3bd0d4-6e79-4a89-8d39-f40f540b7cd5", 2024-03-28T16:04:06.387 INFO:teuthology.orchestra.run.smithi067.stdout: "mode": "asynchronous" 2024-03-28T16:04:06.387 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-28T16:04:06.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.385+0000 7f063d000640 1 -- 172.21.15.67:0/199296220 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f061c0a2800 msgr2=0x7f061c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:06.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.385+0000 7f063d000640 1 --2- 172.21.15.67:0/199296220 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f061c0a2800 0x7f061c0a4cc0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x563c862e32f0 tx=0x7f0630008370 comp rx=0 tx=0).stop 2024-03-28T16:04:06.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.385+0000 7f063d000640 1 -- 172.21.15.67:0/199296220 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0638108eb0 msgr2=0x7f063819e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:06.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.385+0000 7f063d000640 1 --2- 172.21.15.67:0/199296220 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0638108eb0 0x7f063819e490 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f062000d950 tx=0x7f062000de10 comp rx=0 tx=0).stop 2024-03-28T16:04:06.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.385+0000 7f063d000640 1 -- 172.21.15.67:0/199296220 shutdown_connections 2024-03-28T16:04:06.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.385+0000 7f063d000640 1 --2- 172.21.15.67:0/199296220 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f061c0a2800 0x7f061c0a4cc0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:06.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.385+0000 7f063d000640 1 --2- 172.21.15.67:0/199296220 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0638109870 0x7f06381a5500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:06.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.385+0000 7f063d000640 1 --2- 172.21.15.67:0/199296220 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0638108eb0 0x7f063819e490 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:06.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.385+0000 7f063d000640 1 --2- 172.21.15.67:0/199296220 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0638107cc0 0x7f063819df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:06.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.385+0000 7f063d000640 1 -- 172.21.15.67:0/199296220 >> 172.21.15.67:0/199296220 conn(0x7f0638076c60 msgr2=0x7f063810c2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:04:06.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.385+0000 7f063d000640 1 -- 172.21.15.67:0/199296220 shutdown_connections 2024-03-28T16:04:06.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.385+0000 7f063d000640 1 -- 172.21.15.67:0/199296220 wait complete. 2024-03-28T16:04:06.404 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T16:04:06.404 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T16:04:06.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.626+0000 7f9c05800640 1 -- 172.21.15.67:0/3541873154 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c00107bc0 msgr2=0x7f9c00107fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:06.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.626+0000 7f9c05800640 1 --2- 172.21.15.67:0/3541873154 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c00107bc0 0x7f9c00107fe0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f9bf0009900 tx=0x7f9bf002bc30 comp rx=0 tx=0).stop 2024-03-28T16:04:06.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.626+0000 7f9c05800640 1 -- 172.21.15.67:0/3541873154 shutdown_connections 2024-03-28T16:04:06.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.626+0000 7f9c05800640 1 --2- 172.21.15.67:0/3541873154 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c00109770 0x7f9c00110000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:06.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.626+0000 7f9c05800640 1 --2- 172.21.15.67:0/3541873154 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9c00108db0 0x7f9c00109230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:06.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.626+0000 7f9c05800640 1 --2- 172.21.15.67:0/3541873154 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c00107bc0 0x7f9c00107fe0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:06.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.626+0000 7f9c05800640 1 -- 172.21.15.67:0/3541873154 >> 172.21.15.67:0/3541873154 conn(0x7f9c00076b70 msgr2=0x7f9c00078f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:04:06.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.626+0000 7f9c05800640 1 -- 172.21.15.67:0/3541873154 shutdown_connections 2024-03-28T16:04:06.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.626+0000 7f9c05800640 1 -- 172.21.15.67:0/3541873154 wait complete. 2024-03-28T16:04:06.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.627+0000 7f9c05800640 1 Processor -- start 2024-03-28T16:04:06.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.627+0000 7f9c05800640 1 -- start start 2024-03-28T16:04:06.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.628+0000 7f9c05800640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c00107bc0 0x7f9c0019de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:06.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.628+0000 7f9c05800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9c00108db0 0x7f9c0019e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:06.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.628+0000 7f9c05800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c00109770 0x7f9c001a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:06.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.628+0000 7f9c05800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c001136e0 con 0x7f9c00109770 2024-03-28T16:04:06.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.628+0000 7f9c05800640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c00113560 con 0x7f9c00107bc0 2024-03-28T16:04:06.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.628+0000 7f9c05800640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c00113860 con 0x7f9c00108db0 2024-03-28T16:04:06.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.628+0000 7f9c04e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c00109770 0x7f9c001a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:06.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.628+0000 7f9c04e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c00109770 0x7f9c001a53e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44564/0 (socket says 172.21.15.67:44564) 2024-03-28T16:04:06.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.628+0000 7f9c04e00640 1 -- 172.21.15.67:0/3462652857 learned_addr learned my addr 172.21.15.67:0/3462652857 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:04:06.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.628+0000 7f9bff400640 1 --2- 172.21.15.67:0/3462652857 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9c00108db0 0x7f9c0019e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:06.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.628+0000 7f9c04e00640 1 -- 172.21.15.67:0/3462652857 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9c00108db0 msgr2=0x7f9c0019e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:06.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.628+0000 7f9c04e00640 1 --2- 172.21.15.67:0/3462652857 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9c00108db0 0x7f9c0019e370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:06.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.628+0000 7f9c04e00640 1 -- 172.21.15.67:0/3462652857 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c00107bc0 msgr2=0x7f9c0019de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:06.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.628+0000 7f9c04e00640 1 --2- 172.21.15.67:0/3462652857 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c00107bc0 0x7f9c0019de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:06.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.628+0000 7f9c04e00640 1 -- 172.21.15.67:0/3462652857 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c001a5a50 con 0x7f9c00109770 2024-03-28T16:04:06.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.629+0000 7f9bff400640 1 --2- 172.21.15.67:0/3462652857 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9c00108db0 0x7f9c0019e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:04:06.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.629+0000 7f9c04e00640 1 --2- 172.21.15.67:0/3462652857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c00109770 0x7f9c001a53e0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f9bf8009fd0 tx=0x7f9bf800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:04:06.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.629+0000 7f9bfcc00640 1 -- 172.21.15.67:0/3462652857 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bf8017070 con 0x7f9c00109770 2024-03-28T16:04:06.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.629+0000 7f9bfcc00640 1 -- 172.21.15.67:0/3462652857 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9bf8003e20 con 0x7f9c00109770 2024-03-28T16:04:06.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.629+0000 7f9c05800640 1 -- 172.21.15.67:0/3462652857 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c001a5d40 con 0x7f9c00109770 2024-03-28T16:04:06.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.629+0000 7f9bfcc00640 1 -- 172.21.15.67:0/3462652857 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bf8010460 con 0x7f9c00109770 2024-03-28T16:04:06.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.629+0000 7f9c05800640 1 -- 172.21.15.67:0/3462652857 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c001a61e0 con 0x7f9c00109770 2024-03-28T16:04:06.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.632+0000 7f9bfcc00640 1 -- 172.21.15.67:0/3462652857 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f9bf8003970 con 0x7f9c00109770 2024-03-28T16:04:06.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.632+0000 7f9c09a45740 1 Processor -- start 2024-03-28T16:04:06.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.633+0000 7f9c09a45740 1 -- start start 2024-03-28T16:04:06.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.633+0000 7f9c09a45740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563912dc0050 0x563912dc0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:06.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.633+0000 7f9c09a45740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563912dc0a40 0x563912dc5540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:06.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.633+0000 7f9c09a45740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563912dc5bb0 0x563912dc8040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:06.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.633+0000 7f9c09a45740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x563912d25250 con 0x563912dc0a40 2024-03-28T16:04:06.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.633+0000 7f9c09a45740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x563912ccda30 con 0x563912dc5bb0 2024-03-28T16:04:06.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.633+0000 7f9c09a45740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x563912d212f0 con 0x563912dc0050 2024-03-28T16:04:06.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.633+0000 7f9bffe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563912dc0a40 0x563912dc5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:06.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.633+0000 7f9bffe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563912dc0a40 0x563912dc5540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44580/0 (socket says 172.21.15.67:44580) 2024-03-28T16:04:06.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.633+0000 7f9bffe00640 1 -- 172.21.15.67:0/1146338733 learned_addr learned my addr 172.21.15.67:0/1146338733 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:04:06.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.634+0000 7f9bff400640 1 --2- 172.21.15.67:0/1146338733 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563912dc0050 0x563912dc0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:06.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.634+0000 7f9bfcc00640 1 --2- 172.21.15.67:0/3462652857 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f9be40a2800 0x7f9be40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:06.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.634+0000 7f9bff400640 1 --2- 172.21.15.67:0/1146338733 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563912dc5bb0 0x563912dc8040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:06.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.634+0000 7f9bffe00640 1 -- 172.21.15.67:0/1146338733 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563912dc0050 msgr2=0x563912dc0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:04:06.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.634+0000 7f9bffe00640 1 --2- 172.21.15.67:0/1146338733 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563912dc0050 0x563912dc0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:06.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.634+0000 7f9bffe00640 1 -- 172.21.15.67:0/1146338733 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563912dc5bb0 msgr2=0x563912dc8040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:04:06.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.634+0000 7f9bffe00640 1 --2- 172.21.15.67:0/1146338733 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563912dc5bb0 0x563912dc8040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:06.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.634+0000 7f9bffe00640 1 -- 172.21.15.67:0/1146338733 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563912cb8690 con 0x563912dc0a40 2024-03-28T16:04:06.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.634+0000 7f9bff400640 1 --2- 172.21.15.67:0/1146338733 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563912dc5bb0 0x563912dc8040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:04:06.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.634+0000 7f9bfcc00640 1 -- 172.21.15.67:0/3462652857 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f9bf80ce950 con 0x7f9c00109770 2024-03-28T16:04:06.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.634+0000 7f9c04e00640 1 --2- 172.21.15.67:0/3462652857 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f9be40a2800 0x7f9be40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:06.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.634+0000 7f9bffe00640 1 --2- 172.21.15.67:0/1146338733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563912dc0a40 0x563912dc5540 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f9bf0005ab0 tx=0x7f9bf00095e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3311bb23d5e43c17 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:04:06.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.635+0000 7f9bf5600640 1 -- 172.21.15.67:0/1146338733 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bf0041020 con 0x563912dc0a40 2024-03-28T16:04:06.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.635+0000 7f9bf5600640 1 -- 172.21.15.67:0/1146338733 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9bf002cba0 con 0x563912dc0a40 2024-03-28T16:04:06.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.635+0000 7f9bf5600640 1 -- 172.21.15.67:0/1146338733 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bf0004920 con 0x563912dc0a40 2024-03-28T16:04:06.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.636+0000 7f9c09a45740 1 -- 172.21.15.67:0/1146338733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563912dc0a40 msgr2=0x563912dc5540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:06.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.636+0000 7f9c04e00640 1 --2- 172.21.15.67:0/3462652857 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f9be40a2800 0x7f9be40a4cc0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f9bf80a3510 tx=0x7f9bf8004390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:04:06.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.636+0000 7f9c09a45740 1 --2- 172.21.15.67:0/1146338733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563912dc0a40 0x563912dc5540 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f9bf0005ab0 tx=0x7f9bf00095e0 comp rx=0 tx=0).stop 2024-03-28T16:04:06.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.636+0000 7f9c09a45740 1 -- 172.21.15.67:0/1146338733 shutdown_connections 2024-03-28T16:04:06.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.636+0000 7f9c09a45740 1 --2- 172.21.15.67:0/1146338733 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563912dc5bb0 0x563912dc8040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:06.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.636+0000 7f9c09a45740 1 --2- 172.21.15.67:0/1146338733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563912dc0a40 0x563912dc5540 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:06.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.636+0000 7f9c09a45740 1 --2- 172.21.15.67:0/1146338733 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563912dc0050 0x563912dc0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:06.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.636+0000 7f9c09a45740 1 -- 172.21.15.67:0/1146338733 >> 172.21.15.67:0/1146338733 conn(0x563912d88200 msgr2=0x563912dbb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:04:06.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.636+0000 7f9c09a45740 1 -- 172.21.15.67:0/1146338733 shutdown_connections 2024-03-28T16:04:06.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.636+0000 7f9c09a45740 1 -- 172.21.15.67:0/1146338733 wait complete. 2024-03-28T16:04:06.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.636+0000 7f9c09a45740 1 -- start start 2024-03-28T16:04:06.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.637+0000 7f9c09a45740 1 Processor -- start 2024-03-28T16:04:06.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.637+0000 7f9c09a45740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563912dc0050 0x563912dc9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:06.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.637+0000 7f9c09a45740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563912dc0a40 0x563912dca4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:06.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.637+0000 7f9c09a45740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563912dc5bb0 0x563912dd1520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:06.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.637+0000 7f9c09a45740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x563912dc8b00 con 0x563912dc5bb0 2024-03-28T16:04:06.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.637+0000 7f9c09a45740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x563912ce0e80 con 0x563912dc0a40 2024-03-28T16:04:06.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.637+0000 7f9c09a45740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x563912dc8c80 con 0x563912dc0050 2024-03-28T16:04:06.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.637+0000 7f9c09a45740 10 client.0 _ll_register_callbacks cb 0x563912cf3130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:04:06.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.637+0000 7f9bff400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563912dc5bb0 0x563912dd1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:06.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.638+0000 7f9bff400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563912dc5bb0 0x563912dd1520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44590/0 (socket says 172.21.15.67:44590) 2024-03-28T16:04:06.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.638+0000 7f9bff400640 1 -- 172.21.15.67:0/3062945636 learned_addr learned my addr 172.21.15.67:0/3062945636 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:04:06.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.638+0000 7f9bffe00640 1 --2- 172.21.15.67:0/3062945636 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563912dc0a40 0x563912dca4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:06.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.638+0000 7f9bff400640 1 --2- 172.21.15.67:0/3062945636 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563912dc0050 0x563912dc9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:06.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.638+0000 7f9bff400640 1 -- 172.21.15.67:0/3062945636 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563912dc0050 msgr2=0x563912dc9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:04:06.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.638+0000 7f9bff400640 1 --2- 172.21.15.67:0/3062945636 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563912dc0050 0x563912dc9fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:06.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.638+0000 7f9bff400640 1 -- 172.21.15.67:0/3062945636 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563912dc0a40 msgr2=0x563912dca4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:04:06.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.638+0000 7f9bff400640 1 --2- 172.21.15.67:0/3062945636 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563912dc0a40 0x563912dca4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:06.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.638+0000 7f9bff400640 1 -- 172.21.15.67:0/3062945636 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563912b037d0 con 0x563912dc5bb0 2024-03-28T16:04:06.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.638+0000 7f9bff400640 1 --2- 172.21.15.67:0/3062945636 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563912dc5bb0 0x563912dd1520 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f9bec00bdd0 tx=0x7f9bec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95648a85a1f43ca server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:04:06.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.638+0000 7f9bea000640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:04:06.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.638+0000 7f9bea000640 1 -- 172.21.15.67:0/3062945636 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bec01f5f0 con 0x563912dc5bb0 2024-03-28T16:04:06.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.639+0000 7f9bea000640 1 -- 172.21.15.67:0/3062945636 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f9bec01f790 con 0x563912dc5bb0 2024-03-28T16:04:06.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.639+0000 7f9c09a45740 1 -- 172.21.15.67:0/3062945636 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9bf002cba0 con 0x563912dc5bb0 2024-03-28T16:04:06.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.639+0000 7f9bea000640 1 -- 172.21.15.67:0/3062945636 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9bec01f990 con 0x563912dc5bb0 2024-03-28T16:04:06.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.639+0000 7f9c09a45740 10 client.15399 fetch_fsmap learned FSMap version 19 2024-03-28T16:04:06.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.639+0000 7f9c09a45740 1 -- 172.21.15.67:0/3062945636 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x563912a393f0 con 0x563912dc5bb0 2024-03-28T16:04:06.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.639+0000 7f9bea000640 1 -- 172.21.15.67:0/3062945636 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9bf002cba0 con 0x563912dc5bb0 2024-03-28T16:04:06.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.639+0000 7f9bea000640 1 -- 172.21.15.67:0/3062945636 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f9bec00f040 con 0x563912dc5bb0 2024-03-28T16:04:06.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.639+0000 7f9c09a45740 10 client.15399 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:04:06.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.639+0000 7f9c09a45740 10 client.15399 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:04:06.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.639+0000 7f9c09a45740 20 client.15399 populate_metadata read hostname 'smithi067' 2024-03-28T16:04:06.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.640+0000 7f9c09a45740 1 --2- 172.21.15.67:0/3062945636 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x563912dd5d50 0x563912dd81d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:06.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.640+0000 7f9c09a45740 4 client.15399 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:04:06.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.640+0000 7f9c09a45740 1 -- 172.21.15.67:0/3062945636 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563912d32790 con 0x563912dd5d50 2024-03-28T16:04:06.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.640+0000 7f9bffe00640 1 --2- 172.21.15.67:0/3062945636 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x563912dd5d50 0x563912dd81d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:06.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.641+0000 7f9bffe00640 1 --2- 172.21.15.67:0/3062945636 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x563912dd5d50 0x563912dd81d0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=81ce1304794f1d7e server_cookie=e05214fc587abd13 in_seq=0 out_seq=0 2024-03-28T16:04:06.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.641+0000 7f9bea000640 10 client.15399 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:04:06.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.643+0000 7f9bea000640 1 -- 172.21.15.67:0/3062945636 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x563912d32790 con 0x563912dd5d50 2024-03-28T16:04:06.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.643+0000 7f9bea000640 10 client.15399 handle_command_reply: tid=0 2024-03-28T16:04:06.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.643+0000 7f9c09a45740 1 client.15399 shutdown 2024-03-28T16:04:06.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.644+0000 7f9c09a45740 1 -- 172.21.15.67:0/3062945636 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563912dc5bb0 msgr2=0x563912dd1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:06.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.644+0000 7f9c09a45740 1 --2- 172.21.15.67:0/3062945636 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563912dc5bb0 0x563912dd1520 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f9bec00bdd0 tx=0x7f9bec0046c0 comp rx=0 tx=0).stop 2024-03-28T16:04:06.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.644+0000 7f9c09a45740 1 -- 172.21.15.67:0/3062945636 shutdown_connections 2024-03-28T16:04:06.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.644+0000 7f9c09a45740 1 --2- 172.21.15.67:0/3062945636 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x563912dd5d50 0x563912dd81d0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:06.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.644+0000 7f9c09a45740 1 --2- 172.21.15.67:0/3062945636 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563912dc5bb0 0x563912dd1520 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:06.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.644+0000 7f9c09a45740 1 --2- 172.21.15.67:0/3062945636 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563912dc0a40 0x563912dca4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:06.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.644+0000 7f9c09a45740 1 --2- 172.21.15.67:0/3062945636 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563912dc0050 0x563912dc9fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:06.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.644+0000 7f9bea000640 0 client.15399 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:04:06.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.644+0000 7f9c09a45740 1 -- 172.21.15.67:0/3062945636 >> 172.21.15.67:0/3062945636 conn(0x563912d88200 msgr2=0x563912dbabc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:04:06.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.644+0000 7f9c09a45740 1 -- 172.21.15.67:0/3062945636 shutdown_connections 2024-03-28T16:04:06.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.644+0000 7f9c09a45740 1 -- 172.21.15.67:0/3062945636 wait complete. 2024-03-28T16:04:06.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.644+0000 7f9c09a45740 20 client.15399 trim_cache size 0 max 16384 2024-03-28T16:04:06.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.662+0000 7f9c09a45740 1 Processor -- start 2024-03-28T16:04:06.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.662+0000 7f9c09a45740 1 -- start start 2024-03-28T16:04:06.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.662+0000 7f9c09a45740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563912dc0050 0x563912ddca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:06.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.662+0000 7f9c09a45740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563912dc0a40 0x563912dd14e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:06.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.662+0000 7f9c09a45740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563912dc5bb0 0x563912ddcfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:06.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.662+0000 7f9c09a45740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c001ace40 con 0x563912dc5bb0 2024-03-28T16:04:06.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.662+0000 7f9c09a45740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c001af400 con 0x563912dc0050 2024-03-28T16:04:06.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.662+0000 7f9c09a45740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c001ad440 con 0x563912dc0a40 2024-03-28T16:04:06.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.662+0000 7f9bff400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563912dc5bb0 0x563912ddcfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:06.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.663+0000 7f9bff400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563912dc5bb0 0x563912ddcfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44604/0 (socket says 172.21.15.67:44604) 2024-03-28T16:04:06.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.663+0000 7f9bff400640 1 -- 172.21.15.67:0/3386960657 learned_addr learned my addr 172.21.15.67:0/3386960657 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:04:06.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.663+0000 7f9bffe00640 1 --2- 172.21.15.67:0/3386960657 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563912dc0a40 0x563912dd14e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:06.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.663+0000 7f9bff400640 1 --2- 172.21.15.67:0/3386960657 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563912dc0050 0x563912ddca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:06.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.663+0000 7f9bff400640 1 -- 172.21.15.67:0/3386960657 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563912dc0a40 msgr2=0x563912dd14e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:04:06.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.663+0000 7f9bff400640 1 --2- 172.21.15.67:0/3386960657 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563912dc0a40 0x563912dd14e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:06.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.663+0000 7f9bff400640 1 -- 172.21.15.67:0/3386960657 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563912dc0050 msgr2=0x563912ddca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:04:06.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.663+0000 7f9bff400640 1 --2- 172.21.15.67:0/3386960657 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563912dc0050 0x563912ddca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:06.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.663+0000 7f9bff400640 1 -- 172.21.15.67:0/3386960657 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5639129f90f0 con 0x563912dc5bb0 2024-03-28T16:04:06.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.663+0000 7f9bff400640 1 --2- 172.21.15.67:0/3386960657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563912dc5bb0 0x563912ddcfd0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f9bec001eb0 tx=0x7f9bec000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a946a0849290eea0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:04:06.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.664+0000 7f9bf4c00640 1 -- 172.21.15.67:0/3386960657 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bec01fd30 con 0x563912dc5bb0 2024-03-28T16:04:06.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.664+0000 7f9bf4c00640 1 -- 172.21.15.67:0/3386960657 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9bec00e3e0 con 0x563912dc5bb0 2024-03-28T16:04:06.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.664+0000 7f9bf4c00640 1 -- 172.21.15.67:0/3386960657 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bec005290 con 0x563912dc5bb0 2024-03-28T16:04:06.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.664+0000 7f9c09a45740 1 -- 172.21.15.67:0/3386960657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563912dc5bb0 msgr2=0x563912ddcfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:06.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.664+0000 7f9c09a45740 1 --2- 172.21.15.67:0/3386960657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563912dc5bb0 0x563912ddcfd0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f9bec001eb0 tx=0x7f9bec000bc0 comp rx=0 tx=0).stop 2024-03-28T16:04:06.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.664+0000 7f9c09a45740 1 -- 172.21.15.67:0/3386960657 shutdown_connections 2024-03-28T16:04:06.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.664+0000 7f9c09a45740 1 --2- 172.21.15.67:0/3386960657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563912dc5bb0 0x563912ddcfd0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:06.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.664+0000 7f9c09a45740 1 --2- 172.21.15.67:0/3386960657 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563912dc0a40 0x563912dd14e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:06.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.664+0000 7f9c09a45740 1 --2- 172.21.15.67:0/3386960657 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563912dc0050 0x563912ddca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:06.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.664+0000 7f9c09a45740 1 -- 172.21.15.67:0/3386960657 >> 172.21.15.67:0/3386960657 conn(0x563912d88200 msgr2=0x563912dc1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:04:06.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.665+0000 7f9c09a45740 1 -- 172.21.15.67:0/3386960657 shutdown_connections 2024-03-28T16:04:06.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.665+0000 7f9c09a45740 1 -- 172.21.15.67:0/3386960657 wait complete. 2024-03-28T16:04:06.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.665+0000 7f9c09a45740 1 -- start start 2024-03-28T16:04:06.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.665+0000 7f9c09a45740 1 Processor -- start 2024-03-28T16:04:06.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.665+0000 7f9c09a45740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563912dc0050 0x563912dceb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:06.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.666+0000 7f9c09a45740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563912dc0a40 0x563912dcf080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:06.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.666+0000 7f9c09a45740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563912dc5bb0 0x563912dd0090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:06.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.666+0000 7f9c09a45740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c001ada80 con 0x563912dc0050 2024-03-28T16:04:06.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.666+0000 7f9c09a45740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c001aeda0 con 0x563912dc5bb0 2024-03-28T16:04:06.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.666+0000 7f9c09a45740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c001ae0e0 con 0x563912dc0a40 2024-03-28T16:04:06.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.666+0000 7f9c09a45740 10 client.0 _ll_register_callbacks cb 0x7f9bec00b710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:04:06.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.666+0000 7f9bff400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563912dc0050 0x563912dceb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:06.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.666+0000 7f9bff400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563912dc0050 0x563912dceb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44610/0 (socket says 172.21.15.67:44610) 2024-03-28T16:04:06.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.666+0000 7f9bff400640 1 -- 172.21.15.67:0/992324466 learned_addr learned my addr 172.21.15.67:0/992324466 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:04:06.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.666+0000 7f9bffe00640 1 --2- 172.21.15.67:0/992324466 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563912dc0a40 0x563912dcf080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:06.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.666+0000 7f9bff400640 1 -- 172.21.15.67:0/992324466 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563912dc0a40 msgr2=0x563912dcf080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:04:06.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.666+0000 7f9bff400640 1 --2- 172.21.15.67:0/992324466 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563912dc0a40 0x563912dcf080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:06.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.666+0000 7f9bff400640 1 -- 172.21.15.67:0/992324466 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563912dc5bb0 msgr2=0x563912dd0090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:04:06.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.666+0000 7f9bff400640 1 --2- 172.21.15.67:0/992324466 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563912dc5bb0 0x563912dd0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:06.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.666+0000 7f9bff400640 1 -- 172.21.15.67:0/992324466 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563912dd0680 con 0x563912dc0050 2024-03-28T16:04:06.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.667+0000 7f9bff400640 1 --2- 172.21.15.67:0/992324466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563912dc0050 0x563912dceb40 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f9bec02c040 tx=0x7f9bec013800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a252b64d3f3f923 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:04:06.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.667+0000 7f9bea000640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:04:06.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.667+0000 7f9bea000640 1 -- 172.21.15.67:0/992324466 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bec005e50 con 0x563912dc0050 2024-03-28T16:04:06.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.667+0000 7f9bea000640 1 -- 172.21.15.67:0/992324466 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f9bec00e590 con 0x563912dc0050 2024-03-28T16:04:06.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.667+0000 7f9c09a45740 1 -- 172.21.15.67:0/992324466 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9bec00e3e0 con 0x563912dc0050 2024-03-28T16:04:06.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.667+0000 7f9bea000640 1 -- 172.21.15.67:0/992324466 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9bec00e7b0 con 0x563912dc0050 2024-03-28T16:04:06.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.667+0000 7f9c09a45740 10 client.15405 fetch_fsmap learned FSMap version 19 2024-03-28T16:04:06.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.667+0000 7f9c09a45740 1 -- 172.21.15.67:0/992324466 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x563912e31660 con 0x563912dc0050 2024-03-28T16:04:06.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.667+0000 7f9bea000640 1 -- 172.21.15.67:0/992324466 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9bec00e3e0 con 0x563912dc0050 2024-03-28T16:04:06.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.668+0000 7f9bea000640 1 -- 172.21.15.67:0/992324466 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f9bec0049d0 con 0x563912dc0050 2024-03-28T16:04:06.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.668+0000 7f9c09a45740 10 client.15405 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:04:06.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.668+0000 7f9c09a45740 10 client.15405 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:04:06.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.668+0000 7f9c09a45740 20 client.15405 populate_metadata read hostname 'smithi067' 2024-03-28T16:04:06.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.668+0000 7f9c09a45740 1 --2- 172.21.15.67:0/992324466 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x563912e31db0 0x563912e341d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:06.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.668+0000 7f9c09a45740 4 client.15405 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T16:04:06.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.668+0000 7f9c09a45740 1 -- 172.21.15.67:0/992324466 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x563912d2cae0 con 0x563912e31db0 2024-03-28T16:04:06.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.668+0000 7f9bffe00640 1 --2- 172.21.15.67:0/992324466 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x563912e31db0 0x563912e341d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:06.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.669+0000 7f9bffe00640 1 --2- 172.21.15.67:0/992324466 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x563912e31db0 0x563912e341d0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=13cfb5f00424bd59 server_cookie=942c44c57bfed1e in_seq=0 out_seq=0 2024-03-28T16:04:06.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.669+0000 7f9bea000640 10 client.15405 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:04:06.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.670+0000 7f9bea000640 1 -- 172.21.15.67:0/992324466 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x563912d2cae0 con 0x563912e31db0 2024-03-28T16:04:06.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.670+0000 7f9bea000640 10 client.15405 handle_command_reply: tid=0 2024-03-28T16:04:06.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.670+0000 7f9c09a45740 1 client.15405 shutdown 2024-03-28T16:04:06.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.670+0000 7f9c09a45740 1 -- 172.21.15.67:0/992324466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563912dc0050 msgr2=0x563912dceb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:06.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.670+0000 7f9c09a45740 1 --2- 172.21.15.67:0/992324466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563912dc0050 0x563912dceb40 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f9bec02c040 tx=0x7f9bec013800 comp rx=0 tx=0).stop 2024-03-28T16:04:06.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.670+0000 7f9c09a45740 1 -- 172.21.15.67:0/992324466 shutdown_connections 2024-03-28T16:04:06.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.670+0000 7f9c09a45740 1 --2- 172.21.15.67:0/992324466 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x563912e31db0 0x563912e341d0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:06.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.670+0000 7f9c09a45740 1 --2- 172.21.15.67:0/992324466 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563912dc5bb0 0x563912dd0090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:06.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.670+0000 7f9c09a45740 1 --2- 172.21.15.67:0/992324466 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563912dc0a40 0x563912dcf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:06.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.670+0000 7f9bea000640 0 client.15405 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:04:06.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.670+0000 7f9c09a45740 1 --2- 172.21.15.67:0/992324466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563912dc0050 0x563912dceb40 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:06.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.670+0000 7f9c09a45740 1 -- 172.21.15.67:0/992324466 >> 172.21.15.67:0/992324466 conn(0x563912d88200 msgr2=0x563912dbec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:04:06.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.670+0000 7f9c09a45740 1 -- 172.21.15.67:0/992324466 shutdown_connections 2024-03-28T16:04:06.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.671+0000 7f9c09a45740 1 -- 172.21.15.67:0/992324466 wait complete. 2024-03-28T16:04:06.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.671+0000 7f9c09a45740 20 client.15405 trim_cache size 0 max 16384 2024-03-28T16:04:06.674 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-28T16:04:06.675 INFO:teuthology.orchestra.run.smithi067.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T16:04:06.675 INFO:teuthology.orchestra.run.smithi067.stdout: "scrubs": { 2024-03-28T16:04:06.675 INFO:teuthology.orchestra.run.smithi067.stdout: "cf3bd0d4-6e79-4a89-8d39-f40f540b7cd5": { 2024-03-28T16:04:06.675 INFO:teuthology.orchestra.run.smithi067.stdout: "path": "/", 2024-03-28T16:04:06.675 INFO:teuthology.orchestra.run.smithi067.stdout: "tag": "cf3bd0d4-6e79-4a89-8d39-f40f540b7cd5", 2024-03-28T16:04:06.675 INFO:teuthology.orchestra.run.smithi067.stdout: "options": "recursive,force" 2024-03-28T16:04:06.675 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:04:06.675 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:04:06.675 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-28T16:04:06.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.671+0000 7f9c05800640 1 -- 172.21.15.67:0/3462652857 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f9be40a2800 msgr2=0x7f9be40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:06.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.671+0000 7f9c05800640 1 --2- 172.21.15.67:0/3462652857 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f9be40a2800 0x7f9be40a4cc0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f9bf80a3510 tx=0x7f9bf8004390 comp rx=0 tx=0).stop 2024-03-28T16:04:06.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.671+0000 7f9c05800640 1 -- 172.21.15.67:0/3462652857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c00109770 msgr2=0x7f9c001a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:06.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.671+0000 7f9c05800640 1 --2- 172.21.15.67:0/3462652857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c00109770 0x7f9c001a53e0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f9bf8009fd0 tx=0x7f9bf800b260 comp rx=0 tx=0).stop 2024-03-28T16:04:06.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.671+0000 7f9c05800640 1 -- 172.21.15.67:0/3462652857 shutdown_connections 2024-03-28T16:04:06.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.671+0000 7f9c05800640 1 --2- 172.21.15.67:0/3462652857 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f9be40a2800 0x7f9be40a4cc0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:06.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.671+0000 7f9c05800640 1 --2- 172.21.15.67:0/3462652857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c00109770 0x7f9c001a53e0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:06.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.671+0000 7f9c05800640 1 --2- 172.21.15.67:0/3462652857 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9c00108db0 0x7f9c0019e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:06.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.671+0000 7f9c05800640 1 --2- 172.21.15.67:0/3462652857 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c00107bc0 0x7f9c0019de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:06.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.671+0000 7f9c05800640 1 -- 172.21.15.67:0/3462652857 >> 172.21.15.67:0/3462652857 conn(0x7f9c00076b70 msgr2=0x7f9c0010c180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:04:06.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.672+0000 7f9c05800640 1 -- 172.21.15.67:0/3462652857 shutdown_connections 2024-03-28T16:04:06.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:06.672+0000 7f9c05800640 1 -- 172.21.15.67:0/3462652857 wait complete. 2024-03-28T16:04:06.690 INFO:tasks.cephfs.filesystem:scrub status for tag:cf3bd0d4-6e79-4a89-8d39-f40f540b7cd5 - {'path': '/', 'tag': 'cf3bd0d4-6e79-4a89-8d39-f40f540b7cd5', 'options': 'recursive,force'} 2024-03-28T16:04:07.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:04:06 smithi150 ceph-mon[34825]: pgmap v455: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:04:07.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:04:06 smithi150 ceph-mon[34825]: scrub queued for path: / 2024-03-28T16:04:07.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:04:06 smithi150 ceph-mon[34825]: scrub summary: idle+waiting paths [/] 2024-03-28T16:04:07.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:04:06 smithi150 ceph-mon[34825]: scrub summary: active paths [/] 2024-03-28T16:04:07.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:04:06 smithi150 ceph-mon[34825]: scrub summary: idle+waiting paths [/] 2024-03-28T16:04:07.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:04:06 smithi150 ceph-mon[34825]: scrub summary: idle 2024-03-28T16:04:07.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:04:06 smithi067 ceph-mon[31707]: pgmap v455: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:04:07.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:04:06 smithi067 ceph-mon[31707]: scrub queued for path: / 2024-03-28T16:04:07.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:04:06 smithi067 ceph-mon[31707]: scrub summary: idle+waiting paths [/] 2024-03-28T16:04:07.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:04:06 smithi067 ceph-mon[31707]: scrub summary: active paths [/] 2024-03-28T16:04:07.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:04:06 smithi067 ceph-mon[31707]: scrub summary: idle+waiting paths [/] 2024-03-28T16:04:07.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:04:06 smithi067 ceph-mon[31707]: scrub summary: idle 2024-03-28T16:04:07.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:04:06 smithi183 ceph-mon[35580]: pgmap v455: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:04:07.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:04:06 smithi183 ceph-mon[35580]: scrub queued for path: / 2024-03-28T16:04:07.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:04:06 smithi183 ceph-mon[35580]: scrub summary: idle+waiting paths [/] 2024-03-28T16:04:07.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:04:06 smithi183 ceph-mon[35580]: scrub summary: active paths [/] 2024-03-28T16:04:07.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:04:06 smithi183 ceph-mon[35580]: scrub summary: idle+waiting paths [/] 2024-03-28T16:04:07.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:04:06 smithi183 ceph-mon[35580]: scrub summary: idle 2024-03-28T16:04:09.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:04:08 smithi150 ceph-mon[34825]: pgmap v456: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:04:09.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:04:08 smithi067 ceph-mon[31707]: pgmap v456: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:04:09.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:04:08 smithi183 ceph-mon[35580]: pgmap v456: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:04:11.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:04:10 smithi150 ceph-mon[34825]: pgmap v457: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1 op/s 2024-03-28T16:04:11.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:04:10 smithi067 ceph-mon[31707]: pgmap v457: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1 op/s 2024-03-28T16:04:11.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:04:10 smithi183 ceph-mon[35580]: pgmap v457: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1 op/s 2024-03-28T16:04:13.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:04:12 smithi150 ceph-mon[34825]: pgmap v458: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 2 op/s 2024-03-28T16:04:13.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:04:12 smithi067 ceph-mon[31707]: pgmap v458: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 2 op/s 2024-03-28T16:04:13.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:04:12 smithi183 ceph-mon[35580]: pgmap v458: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 2 op/s 2024-03-28T16:04:15.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:04:14 smithi150 ceph-mon[34825]: pgmap v459: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-03-28T16:04:15.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:04:14 smithi067 ceph-mon[31707]: pgmap v459: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-03-28T16:04:15.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:04:14 smithi183 ceph-mon[35580]: pgmap v459: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-03-28T16:04:17.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:04:16 smithi150 ceph-mon[34825]: pgmap v460: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 2 op/s 2024-03-28T16:04:17.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:04:16 smithi067 ceph-mon[31707]: pgmap v460: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 2 op/s 2024-03-28T16:04:17.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:04:16 smithi183 ceph-mon[35580]: pgmap v460: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 2 op/s 2024-03-28T16:04:18.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:04:18 smithi150 ceph-mon[34825]: pgmap v461: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-03-28T16:04:18.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:04:18 smithi067 ceph-mon[31707]: pgmap v461: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-03-28T16:04:18.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:04:18 smithi183 ceph-mon[35580]: pgmap v461: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-03-28T16:04:20.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:04:20 smithi150 ceph-mon[34825]: pgmap v462: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-03-28T16:04:20.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:04:20 smithi067 ceph-mon[31707]: pgmap v462: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-03-28T16:04:20.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:04:20 smithi183 ceph-mon[35580]: pgmap v462: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-03-28T16:04:22.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:04:22 smithi150 ceph-mon[34825]: pgmap v463: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 2 op/s 2024-03-28T16:04:22.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:04:22 smithi067 ceph-mon[31707]: pgmap v463: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 2 op/s 2024-03-28T16:04:22.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:04:22 smithi183 ceph-mon[35580]: pgmap v463: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 2 op/s 2024-03-28T16:04:24.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:04:24 smithi150 ceph-mon[34825]: pgmap v464: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:04:24.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:04:24 smithi067 ceph-mon[31707]: pgmap v464: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:04:24.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:04:24 smithi183 ceph-mon[35580]: pgmap v464: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:04:26.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:04:26 smithi150 ceph-mon[34825]: pgmap v465: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:04:26.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:04:26 smithi067 ceph-mon[31707]: pgmap v465: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:04:26.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:04:26 smithi183 ceph-mon[35580]: pgmap v465: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:04:28.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:04:28 smithi150 ceph-mon[34825]: pgmap v466: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:04:28.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:04:28 smithi067 ceph-mon[31707]: pgmap v466: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:04:28.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:04:28 smithi183 ceph-mon[35580]: pgmap v466: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:04:30.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:04:30 smithi150 ceph-mon[34825]: pgmap v467: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:04:30.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:04:30 smithi067 ceph-mon[31707]: pgmap v467: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:04:30.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:04:30 smithi183 ceph-mon[35580]: pgmap v467: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:04:32.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:04:32 smithi150 ceph-mon[34825]: pgmap v468: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:04:32.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:04:32 smithi067 ceph-mon[31707]: pgmap v468: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:04:32.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:04:32 smithi183 ceph-mon[35580]: pgmap v468: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:04:34.553 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:04:34 smithi150 ceph-mon[34825]: pgmap v469: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:04:34.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:04:34 smithi067 ceph-mon[31707]: pgmap v469: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:04:34.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:04:34 smithi183 ceph-mon[35580]: pgmap v469: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:04:36.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:04:36 smithi150 ceph-mon[34825]: pgmap v470: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:04:36.574 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:04:36 smithi067 ceph-mon[31707]: pgmap v470: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:04:36.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:04:36 smithi183 ceph-mon[35580]: pgmap v470: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:04:36.691 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T16:04:36.691 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T16:04:36.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.906+0000 7fb5dfe00640 1 -- 172.21.15.67:0/3688589000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5e0102000 msgr2=0x7fb5e01043f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:36.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.906+0000 7fb5dfe00640 1 --2- 172.21.15.67:0/3688589000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5e0102000 0x7fb5e01043f0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fb5d0009900 tx=0x7fb5d002bc50 comp rx=0 tx=0).stop 2024-03-28T16:04:36.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.906+0000 7fb5dfe00640 1 -- 172.21.15.67:0/3688589000 shutdown_connections 2024-03-28T16:04:36.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.906+0000 7fb5dfe00640 1 --2- 172.21.15.67:0/3688589000 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb5e0107260 0x7fb5e01096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:36.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.906+0000 7fb5dfe00640 1 --2- 172.21.15.67:0/3688589000 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5e0104930 0x7fb5e0106d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:36.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.906+0000 7fb5dfe00640 1 --2- 172.21.15.67:0/3688589000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5e0102000 0x7fb5e01043f0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:36.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.906+0000 7fb5dfe00640 1 -- 172.21.15.67:0/3688589000 >> 172.21.15.67:0/3688589000 conn(0x7fb5e00fba10 msgr2=0x7fb5e00fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:04:36.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.906+0000 7fb5dfe00640 1 -- 172.21.15.67:0/3688589000 shutdown_connections 2024-03-28T16:04:36.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.906+0000 7fb5dfe00640 1 -- 172.21.15.67:0/3688589000 wait complete. 2024-03-28T16:04:36.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.907+0000 7fb5dfe00640 1 Processor -- start 2024-03-28T16:04:36.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.907+0000 7fb5dfe00640 1 -- start start 2024-03-28T16:04:36.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.908+0000 7fb5dfe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5e0102000 0x7fb5e01977c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:36.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.908+0000 7fb5dfe00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5e0104930 0x7fb5e0197d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:36.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.908+0000 7fb5dfe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb5e0107260 0x7fb5e019ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:36.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.908+0000 7fb5dfe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e010d110 con 0x7fb5e0102000 2024-03-28T16:04:36.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.908+0000 7fb5dfe00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e010cf90 con 0x7fb5e0104930 2024-03-28T16:04:36.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.908+0000 7fb5dfe00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e010d290 con 0x7fb5e0107260 2024-03-28T16:04:36.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.908+0000 7fb5dea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5e0102000 0x7fb5e01977c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:36.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.908+0000 7fb5dea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5e0102000 0x7fb5e01977c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58208/0 (socket says 172.21.15.67:58208) 2024-03-28T16:04:36.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.908+0000 7fb5dea00640 1 -- 172.21.15.67:0/1027349473 learned_addr learned my addr 172.21.15.67:0/1027349473 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:04:36.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.908+0000 7fb5de000640 1 --2- 172.21.15.67:0/1027349473 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5e0104930 0x7fb5e0197d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:36.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.909+0000 7fb5df400640 1 --2- 172.21.15.67:0/1027349473 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb5e0107260 0x7fb5e019ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:36.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.909+0000 7fb5dea00640 1 -- 172.21.15.67:0/1027349473 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb5e0107260 msgr2=0x7fb5e019ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:36.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.909+0000 7fb5dea00640 1 --2- 172.21.15.67:0/1027349473 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb5e0107260 0x7fb5e019ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:36.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.909+0000 7fb5dea00640 1 -- 172.21.15.67:0/1027349473 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5e0104930 msgr2=0x7fb5e0197d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:36.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.909+0000 7fb5dea00640 1 --2- 172.21.15.67:0/1027349473 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5e0104930 0x7fb5e0197d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:36.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.909+0000 7fb5dea00640 1 -- 172.21.15.67:0/1027349473 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5e00ffe70 con 0x7fb5e0102000 2024-03-28T16:04:36.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.909+0000 7fb5de000640 1 --2- 172.21.15.67:0/1027349473 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5e0104930 0x7fb5e0197d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T16:04:36.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.910+0000 7fb5dea00640 1 --2- 172.21.15.67:0/1027349473 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5e0102000 0x7fb5e01977c0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fb5d0000c00 tx=0x7fb5d00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:04:36.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.910+0000 7fb5e4a00640 1 -- 172.21.15.67:0/1027349473 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d0043020 con 0x7fb5e0102000 2024-03-28T16:04:36.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.910+0000 7fb5e4a00640 1 -- 172.21.15.67:0/1027349473 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb5d00054c0 con 0x7fb5e0102000 2024-03-28T16:04:36.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.910+0000 7fb5e4a00640 1 -- 172.21.15.67:0/1027349473 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d0003d40 con 0x7fb5e0102000 2024-03-28T16:04:36.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.911+0000 7fb5dfe00640 1 -- 172.21.15.67:0/1027349473 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5e0100100 con 0x7fb5e0102000 2024-03-28T16:04:36.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.911+0000 7fb5dfe00640 1 -- 172.21.15.67:0/1027349473 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5e0100600 con 0x7fb5e0102000 2024-03-28T16:04:36.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.915+0000 7fb5e8b76740 1 Processor -- start 2024-03-28T16:04:36.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.915+0000 7fb5e8b76740 1 -- start start 2024-03-28T16:04:36.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.916+0000 7fb5e8b76740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55670d946050 0x55670d946470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:36.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.916+0000 7fb5e8b76740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55670d946a40 0x55670d94b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:36.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.916+0000 7fb5e8b76740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55670d94bbb0 0x55670d94e040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:36.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.916+0000 7fb5e8b76740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55670d8ab250 con 0x55670d946050 2024-03-28T16:04:36.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.916+0000 7fb5e8b76740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55670d853a30 con 0x55670d94bbb0 2024-03-28T16:04:36.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.916+0000 7fb5e8b76740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55670d8a72f0 con 0x55670d946a40 2024-03-28T16:04:36.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.916+0000 7fb5e4a00640 1 -- 172.21.15.67:0/1027349473 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fb5d0046020 con 0x7fb5e0102000 2024-03-28T16:04:36.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.916+0000 7fb5df400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55670d946050 0x55670d946470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:36.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.917+0000 7fb5df400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55670d946050 0x55670d946470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58214/0 (socket says 172.21.15.67:58214) 2024-03-28T16:04:36.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.917+0000 7fb5df400640 1 -- 172.21.15.67:0/168213979 learned_addr learned my addr 172.21.15.67:0/168213979 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:04:36.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.917+0000 7fb5de000640 1 --2- 172.21.15.67:0/168213979 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55670d94bbb0 0x55670d94e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:36.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.917+0000 7fb5dea00640 1 --2- 172.21.15.67:0/168213979 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55670d946a40 0x55670d94b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:36.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.917+0000 7fb5df400640 1 -- 172.21.15.67:0/168213979 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55670d946a40 msgr2=0x55670d94b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:04:36.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.917+0000 7fb5df400640 1 --2- 172.21.15.67:0/168213979 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55670d946a40 0x55670d94b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:36.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.917+0000 7fb5df400640 1 -- 172.21.15.67:0/168213979 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55670d94bbb0 msgr2=0x55670d94e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:04:36.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.917+0000 7fb5df400640 1 --2- 172.21.15.67:0/168213979 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55670d94bbb0 0x55670d94e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:36.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.917+0000 7fb5df400640 1 -- 172.21.15.67:0/168213979 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55670d83e690 con 0x55670d946050 2024-03-28T16:04:36.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.918+0000 7fb5df400640 1 --2- 172.21.15.67:0/168213979 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55670d946050 0x55670d946470 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fb5d8009f70 tx=0x7fb5d8010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8a4546256b24b18 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:04:36.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.918+0000 7fb5cbe00640 1 -- 172.21.15.67:0/168213979 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d8016070 con 0x55670d946050 2024-03-28T16:04:36.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.918+0000 7fb5cbe00640 1 -- 172.21.15.67:0/168213979 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb5d8004d20 con 0x55670d946050 2024-03-28T16:04:36.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.918+0000 7fb5cbe00640 1 -- 172.21.15.67:0/168213979 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d80127d0 con 0x55670d946050 2024-03-28T16:04:36.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.919+0000 7fb5e4a00640 1 --2- 172.21.15.67:0/1027349473 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb5c40a2880 0x7fb5c40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:36.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.919+0000 7fb5e4a00640 1 -- 172.21.15.67:0/1027349473 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fb5d00f0f90 con 0x7fb5e0102000 2024-03-28T16:04:36.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.919+0000 7fb5dea00640 1 --2- 172.21.15.67:0/1027349473 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb5c40a2880 0x7fb5c40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:36.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.920+0000 7fb5e8b76740 1 -- 172.21.15.67:0/168213979 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55670d946050 msgr2=0x55670d946470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:36.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.920+0000 7fb5e8b76740 1 --2- 172.21.15.67:0/168213979 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55670d946050 0x55670d946470 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fb5d8009f70 tx=0x7fb5d8010800 comp rx=0 tx=0).stop 2024-03-28T16:04:36.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.920+0000 7fb5e8b76740 1 -- 172.21.15.67:0/168213979 shutdown_connections 2024-03-28T16:04:36.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.920+0000 7fb5e8b76740 1 --2- 172.21.15.67:0/168213979 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55670d94bbb0 0x55670d94e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:36.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.920+0000 7fb5e8b76740 1 --2- 172.21.15.67:0/168213979 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55670d946a40 0x55670d94b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:36.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.920+0000 7fb5e8b76740 1 --2- 172.21.15.67:0/168213979 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55670d946050 0x55670d946470 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:36.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.920+0000 7fb5e8b76740 1 -- 172.21.15.67:0/168213979 >> 172.21.15.67:0/168213979 conn(0x55670d90e200 msgr2=0x55670d9419c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:04:36.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.920+0000 7fb5e8b76740 1 -- 172.21.15.67:0/168213979 shutdown_connections 2024-03-28T16:04:36.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.920+0000 7fb5e8b76740 1 -- 172.21.15.67:0/168213979 wait complete. 2024-03-28T16:04:36.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.921+0000 7fb5dea00640 1 --2- 172.21.15.67:0/1027349473 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb5c40a2880 0x7fb5c40a4d40 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fb5d00b9e60 tx=0x7fb5d0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:04:36.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.921+0000 7fb5e8b76740 1 -- start start 2024-03-28T16:04:36.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.921+0000 7fb5e8b76740 1 Processor -- start 2024-03-28T16:04:36.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.922+0000 7fb5e8b76740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55670d946050 0x55670d9588e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:36.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.922+0000 7fb5e8b76740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55670d946a40 0x55670d958e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:36.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.922+0000 7fb5e8b76740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55670d94bbb0 0x55670d95fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:36.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.922+0000 7fb5e8b76740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55670d94eb60 con 0x55670d94bbb0 2024-03-28T16:04:36.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.922+0000 7fb5e8b76740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55670d866e80 con 0x55670d946a40 2024-03-28T16:04:36.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.922+0000 7fb5e8b76740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55670d94ece0 con 0x55670d946050 2024-03-28T16:04:36.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.922+0000 7fb5e8b76740 10 client.0 _ll_register_callbacks cb 0x55670d879130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:04:36.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.922+0000 7fb5de000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55670d946a40 0x55670d958e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:36.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.922+0000 7fb5df400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55670d94bbb0 0x55670d95fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:36.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.922+0000 7fb5de000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55670d946a40 0x55670d958e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.67:51054/0 (socket says 172.21.15.67:51054) 2024-03-28T16:04:36.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.922+0000 7fb5de000640 1 -- 172.21.15.67:0/4294013082 learned_addr learned my addr 172.21.15.67:0/4294013082 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:04:36.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.923+0000 7fb5df400640 1 --2- 172.21.15.67:0/4294013082 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55670d946050 0x55670d9588e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:36.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.923+0000 7fb5de000640 1 -- 172.21.15.67:0/4294013082 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55670d946050 msgr2=0x55670d9588e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:04:36.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.923+0000 7fb5de000640 1 --2- 172.21.15.67:0/4294013082 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55670d946050 0x55670d9588e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:36.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.923+0000 7fb5de000640 1 -- 172.21.15.67:0/4294013082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55670d94bbb0 msgr2=0x55670d95fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:04:36.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.923+0000 7fb5de000640 1 --2- 172.21.15.67:0/4294013082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55670d94bbb0 0x55670d95fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:36.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.923+0000 7fb5de000640 1 -- 172.21.15.67:0/4294013082 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55670d6897d0 con 0x55670d946a40 2024-03-28T16:04:36.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.923+0000 7fb5df400640 1 --2- 172.21.15.67:0/4294013082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55670d94bbb0 0x55670d95fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T16:04:36.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.923+0000 7fb5de000640 1 --2- 172.21.15.67:0/4294013082 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55670d946a40 0x55670d958e20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb5cc011800 tx=0x7fb5cc011cc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fee259b301585e53 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:04:36.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.923+0000 7fb5c8c00640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-28T16:04:36.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.923+0000 7fb5c8c00640 1 -- 172.21.15.67:0/4294013082 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5cc010c60 con 0x55670d946a40 2024-03-28T16:04:36.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.924+0000 7fb5c8c00640 1 -- 172.21.15.67:0/4294013082 <== mon.2 v2:172.21.15.150:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fb5cc010e00 con 0x55670d946a40 2024-03-28T16:04:36.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.924+0000 7fb5e8b76740 1 -- 172.21.15.67:0/4294013082 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb5d8004d20 con 0x55670d946a40 2024-03-28T16:04:36.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.924+0000 7fb5c8c00640 1 -- 172.21.15.67:0/4294013082 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb5cc004020 con 0x55670d946a40 2024-03-28T16:04:36.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.924+0000 7fb5e8b76740 10 client.35054 fetch_fsmap learned FSMap version 19 2024-03-28T16:04:36.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.924+0000 7fb5e8b76740 1 -- 172.21.15.67:0/4294013082 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55670d5bf3f0 con 0x55670d946a40 2024-03-28T16:04:36.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.925+0000 7fb5c8c00640 1 -- 172.21.15.67:0/4294013082 <== mon.2 v2:172.21.15.150:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb5cc0041c0 con 0x55670d946a40 2024-03-28T16:04:36.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.925+0000 7fb5c8c00640 1 -- 172.21.15.67:0/4294013082 <== mon.2 v2:172.21.15.150:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fb5cc01a410 con 0x55670d946a40 2024-03-28T16:04:36.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.925+0000 7fb5e8b76740 10 client.35054 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:04:36.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.925+0000 7fb5e8b76740 10 client.35054 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:04:36.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.925+0000 7fb5e8b76740 20 client.35054 populate_metadata read hostname 'smithi067' 2024-03-28T16:04:36.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.925+0000 7fb5e8b76740 1 --2- 172.21.15.67:0/4294013082 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55670d964780 0x55670d966ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:36.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.925+0000 7fb5e8b76740 4 client.35054 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:04:36.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.925+0000 7fb5e8b76740 1 -- 172.21.15.67:0/4294013082 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb5d8004d20 con 0x55670d964780 2024-03-28T16:04:36.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.925+0000 7fb5df400640 1 --2- 172.21.15.67:0/4294013082 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55670d964780 0x55670d966ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:36.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.926+0000 7fb5df400640 1 --2- 172.21.15.67:0/4294013082 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55670d964780 0x55670d966ba0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b1c6aa1d0c7cc6ed server_cookie=17cdef76c3117778 in_seq=0 out_seq=0 2024-03-28T16:04:36.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.926+0000 7fb5c8c00640 10 client.35054 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:04:36.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.928+0000 7fb5c8c00640 1 -- 172.21.15.67:0/4294013082 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x7fb5d8004d20 con 0x55670d964780 2024-03-28T16:04:36.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.928+0000 7fb5c8c00640 10 client.35054 handle_command_reply: tid=0 2024-03-28T16:04:36.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.928+0000 7fb5e8b76740 1 client.35054 shutdown 2024-03-28T16:04:36.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.928+0000 7fb5e8b76740 1 -- 172.21.15.67:0/4294013082 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55670d946a40 msgr2=0x55670d958e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:36.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.928+0000 7fb5e8b76740 1 --2- 172.21.15.67:0/4294013082 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55670d946a40 0x55670d958e20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb5cc011800 tx=0x7fb5cc011cc0 comp rx=0 tx=0).stop 2024-03-28T16:04:36.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.928+0000 7fb5e8b76740 1 -- 172.21.15.67:0/4294013082 shutdown_connections 2024-03-28T16:04:36.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.928+0000 7fb5e8b76740 1 --2- 172.21.15.67:0/4294013082 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55670d964780 0x55670d966ba0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:36.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.928+0000 7fb5e8b76740 1 --2- 172.21.15.67:0/4294013082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55670d94bbb0 0x55670d95fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:36.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.928+0000 7fb5e8b76740 1 --2- 172.21.15.67:0/4294013082 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55670d946a40 0x55670d958e20 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:36.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.928+0000 7fb5c8c00640 0 client.35054 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:04:36.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.928+0000 7fb5e8b76740 1 --2- 172.21.15.67:0/4294013082 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55670d946050 0x55670d9588e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:36.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.928+0000 7fb5e8b76740 1 -- 172.21.15.67:0/4294013082 >> 172.21.15.67:0/4294013082 conn(0x55670d90e200 msgr2=0x55670d940bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:04:36.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.929+0000 7fb5e8b76740 1 -- 172.21.15.67:0/4294013082 shutdown_connections 2024-03-28T16:04:36.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.929+0000 7fb5e8b76740 1 -- 172.21.15.67:0/4294013082 wait complete. 2024-03-28T16:04:36.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.929+0000 7fb5e8b76740 20 client.35054 trim_cache size 0 max 16384 2024-03-28T16:04:36.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.946+0000 7fb5e8b76740 1 Processor -- start 2024-03-28T16:04:36.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.947+0000 7fb5e8b76740 1 -- start start 2024-03-28T16:04:36.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.947+0000 7fb5e8b76740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55670d946050 0x55670d944c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:36.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.947+0000 7fb5e8b76740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55670d946a40 0x55670d95fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:36.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.947+0000 7fb5e8b76740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55670d94bbb0 0x55670d945140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:36.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.947+0000 7fb5e8b76740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e01a46a0 con 0x55670d946a40 2024-03-28T16:04:36.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.947+0000 7fb5e8b76740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e01a6ca0 con 0x55670d946050 2024-03-28T16:04:36.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.947+0000 7fb5e8b76740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e01a4cc0 con 0x55670d94bbb0 2024-03-28T16:04:36.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.947+0000 7fb5de000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55670d946a40 0x55670d95fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:36.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.948+0000 7fb5de000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55670d946a40 0x55670d95fd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58230/0 (socket says 172.21.15.67:58230) 2024-03-28T16:04:36.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.948+0000 7fb5de000640 1 -- 172.21.15.67:0/467731 learned_addr learned my addr 172.21.15.67:0/467731 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:04:36.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.948+0000 7fb5df400640 1 --2- 172.21.15.67:0/467731 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55670d946050 0x55670d944c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:36.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.948+0000 7fb5df400640 1 --2- 172.21.15.67:0/467731 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55670d94bbb0 0x55670d945140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:36.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.948+0000 7fb5de000640 1 -- 172.21.15.67:0/467731 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55670d94bbb0 msgr2=0x55670d945140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:04:36.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.948+0000 7fb5de000640 1 --2- 172.21.15.67:0/467731 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55670d94bbb0 0x55670d945140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:36.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.948+0000 7fb5de000640 1 -- 172.21.15.67:0/467731 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55670d946050 msgr2=0x55670d944c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:04:36.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.948+0000 7fb5de000640 1 --2- 172.21.15.67:0/467731 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55670d946050 0x55670d944c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:36.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.948+0000 7fb5de000640 1 -- 172.21.15.67:0/467731 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55670d966f10 con 0x55670d946a40 2024-03-28T16:04:36.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.948+0000 7fb5df400640 1 --2- 172.21.15.67:0/467731 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55670d94bbb0 0x55670d945140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T16:04:36.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.949+0000 7fb5de000640 1 --2- 172.21.15.67:0/467731 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55670d946a40 0x55670d95fd80 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fb5cc018f50 tx=0x7fb5cc001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e15f5f01f332f43e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:04:36.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.949+0000 7fb5cb400640 1 -- 172.21.15.67:0/467731 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5cc004020 con 0x55670d946a40 2024-03-28T16:04:36.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.949+0000 7fb5cb400640 1 -- 172.21.15.67:0/467731 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb5cc010e00 con 0x55670d946a40 2024-03-28T16:04:36.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.949+0000 7fb5cb400640 1 -- 172.21.15.67:0/467731 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5cc018310 con 0x55670d946a40 2024-03-28T16:04:36.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.950+0000 7fb5e8b76740 1 -- 172.21.15.67:0/467731 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55670d946a40 msgr2=0x55670d95fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:36.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.950+0000 7fb5e8b76740 1 --2- 172.21.15.67:0/467731 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55670d946a40 0x55670d95fd80 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fb5cc018f50 tx=0x7fb5cc001e70 comp rx=0 tx=0).stop 2024-03-28T16:04:36.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.950+0000 7fb5e8b76740 1 -- 172.21.15.67:0/467731 shutdown_connections 2024-03-28T16:04:36.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.950+0000 7fb5e8b76740 1 --2- 172.21.15.67:0/467731 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55670d94bbb0 0x55670d945140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:36.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.950+0000 7fb5e8b76740 1 --2- 172.21.15.67:0/467731 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55670d946a40 0x55670d95fd80 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:36.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.950+0000 7fb5e8b76740 1 --2- 172.21.15.67:0/467731 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55670d946050 0x55670d944c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:36.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.950+0000 7fb5e8b76740 1 -- 172.21.15.67:0/467731 >> 172.21.15.67:0/467731 conn(0x55670d90e200 msgr2=0x55670d9588e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:04:36.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.950+0000 7fb5e8b76740 1 -- 172.21.15.67:0/467731 shutdown_connections 2024-03-28T16:04:36.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.950+0000 7fb5e8b76740 1 -- 172.21.15.67:0/467731 wait complete. 2024-03-28T16:04:36.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.951+0000 7fb5e8b76740 1 -- start start 2024-03-28T16:04:36.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.951+0000 7fb5e8b76740 1 Processor -- start 2024-03-28T16:04:36.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.951+0000 7fb5e8b76740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55670d946050 0x55670d95b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:36.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.951+0000 7fb5e8b76740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55670d946a40 0x55670d95be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:36.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.951+0000 7fb5e8b76740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55670d94bbb0 0x55670d95ee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:36.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.951+0000 7fb5e8b76740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e01a5320 con 0x55670d946a40 2024-03-28T16:04:36.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.951+0000 7fb5e8b76740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e01a6640 con 0x55670d94bbb0 2024-03-28T16:04:36.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.951+0000 7fb5e8b76740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e01a5980 con 0x55670d946050 2024-03-28T16:04:36.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.952+0000 7fb5e8b76740 10 client.0 _ll_register_callbacks cb 0x7fb5cc00bcf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:04:36.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.952+0000 7fb5de000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55670d946a40 0x55670d95be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:36.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.952+0000 7fb5de000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55670d946a40 0x55670d95be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58240/0 (socket says 172.21.15.67:58240) 2024-03-28T16:04:36.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.952+0000 7fb5de000640 1 -- 172.21.15.67:0/2646790152 learned_addr learned my addr 172.21.15.67:0/2646790152 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:04:36.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.952+0000 7fb5df400640 1 --2- 172.21.15.67:0/2646790152 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55670d946050 0x55670d95b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:36.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.952+0000 7fb5df400640 1 --2- 172.21.15.67:0/2646790152 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55670d94bbb0 0x55670d95ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:36.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.953+0000 7fb5de000640 1 -- 172.21.15.67:0/2646790152 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55670d946050 msgr2=0x55670d95b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:04:36.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.953+0000 7fb5de000640 1 --2- 172.21.15.67:0/2646790152 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55670d946050 0x55670d95b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:36.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.953+0000 7fb5de000640 1 -- 172.21.15.67:0/2646790152 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55670d94bbb0 msgr2=0x55670d95ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:04:36.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.953+0000 7fb5de000640 1 --2- 172.21.15.67:0/2646790152 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55670d94bbb0 0x55670d95ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:36.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.953+0000 7fb5de000640 1 -- 172.21.15.67:0/2646790152 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55670d57f0f0 con 0x55670d946a40 2024-03-28T16:04:36.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.953+0000 7fb5df400640 1 --2- 172.21.15.67:0/2646790152 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55670d94bbb0 0x55670d95ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T16:04:36.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.953+0000 7fb5de000640 1 --2- 172.21.15.67:0/2646790152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55670d946a40 0x55670d95be70 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fb5cc00c010 tx=0x7fb5cc00fee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8f121eb5a121b78 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:04:36.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.953+0000 7fb5c8c00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:04:36.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.953+0000 7fb5c8c00640 1 -- 172.21.15.67:0/2646790152 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5cc033070 con 0x55670d946a40 2024-03-28T16:04:36.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.953+0000 7fb5c8c00640 1 -- 172.21.15.67:0/2646790152 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fb5cc029040 con 0x55670d946a40 2024-03-28T16:04:36.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.953+0000 7fb5e8b76740 1 -- 172.21.15.67:0/2646790152 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb5cc010e00 con 0x55670d946a40 2024-03-28T16:04:36.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.954+0000 7fb5c8c00640 1 -- 172.21.15.67:0/2646790152 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb5cc022d20 con 0x55670d946a40 2024-03-28T16:04:36.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.954+0000 7fb5e8b76740 10 client.15423 fetch_fsmap learned FSMap version 19 2024-03-28T16:04:36.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.954+0000 7fb5e8b76740 1 -- 172.21.15.67:0/2646790152 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55670d9d1320 con 0x55670d946a40 2024-03-28T16:04:36.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.954+0000 7fb5c8c00640 1 -- 172.21.15.67:0/2646790152 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb5cc010e00 con 0x55670d946a40 2024-03-28T16:04:36.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.954+0000 7fb5c8c00640 1 -- 172.21.15.67:0/2646790152 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fb5cc01a410 con 0x55670d946a40 2024-03-28T16:04:36.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.955+0000 7fb5e8b76740 10 client.15423 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:04:36.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.955+0000 7fb5e8b76740 10 client.15423 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:04:36.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.955+0000 7fb5e8b76740 20 client.15423 populate_metadata read hostname 'smithi067' 2024-03-28T16:04:36.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.955+0000 7fb5e8b76740 1 --2- 172.21.15.67:0/2646790152 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55670d964780 0x55670d9d3b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:36.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.955+0000 7fb5e8b76740 4 client.15423 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T16:04:36.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.955+0000 7fb5e8b76740 1 -- 172.21.15.67:0/2646790152 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55670d8b8790 con 0x55670d964780 2024-03-28T16:04:36.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.955+0000 7fb5df400640 1 --2- 172.21.15.67:0/2646790152 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55670d964780 0x55670d9d3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:36.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.956+0000 7fb5df400640 1 --2- 172.21.15.67:0/2646790152 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55670d964780 0x55670d9d3b70 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76fc0ac9de623eaf server_cookie=cc0a85a2dd706be7 in_seq=0 out_seq=0 2024-03-28T16:04:36.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.956+0000 7fb5c8c00640 10 client.15423 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:04:36.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.956+0000 7fb5c8c00640 1 -- 172.21.15.67:0/2646790152 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55670d8b8790 con 0x55670d964780 2024-03-28T16:04:36.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.956+0000 7fb5c8c00640 10 client.15423 handle_command_reply: tid=0 2024-03-28T16:04:36.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.956+0000 7fb5e8b76740 1 client.15423 shutdown 2024-03-28T16:04:36.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.956+0000 7fb5e8b76740 1 -- 172.21.15.67:0/2646790152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55670d946a40 msgr2=0x55670d95be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:36.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.956+0000 7fb5e8b76740 1 --2- 172.21.15.67:0/2646790152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55670d946a40 0x55670d95be70 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fb5cc00c010 tx=0x7fb5cc00fee0 comp rx=0 tx=0).stop 2024-03-28T16:04:36.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.957+0000 7fb5e8b76740 1 -- 172.21.15.67:0/2646790152 shutdown_connections 2024-03-28T16:04:36.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.957+0000 7fb5e8b76740 1 --2- 172.21.15.67:0/2646790152 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55670d964780 0x55670d9d3b70 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:36.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.957+0000 7fb5e8b76740 1 --2- 172.21.15.67:0/2646790152 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55670d94bbb0 0x55670d95ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:36.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.957+0000 7fb5e8b76740 1 --2- 172.21.15.67:0/2646790152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55670d946a40 0x55670d95be70 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:36.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.957+0000 7fb5e8b76740 1 --2- 172.21.15.67:0/2646790152 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55670d946050 0x55670d95b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:36.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.957+0000 7fb5e8b76740 1 -- 172.21.15.67:0/2646790152 >> 172.21.15.67:0/2646790152 conn(0x55670d90e200 msgr2=0x55670d944c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:04:36.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.957+0000 7fb5c8c00640 0 client.15423 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:04:36.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.957+0000 7fb5e8b76740 1 -- 172.21.15.67:0/2646790152 shutdown_connections 2024-03-28T16:04:36.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.957+0000 7fb5e8b76740 1 -- 172.21.15.67:0/2646790152 wait complete. 2024-03-28T16:04:36.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.957+0000 7fb5e8b76740 20 client.15423 trim_cache size 0 max 16384 2024-03-28T16:04:36.959 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-28T16:04:36.959 INFO:teuthology.orchestra.run.smithi067.stdout: "status": "no active scrubs running", 2024-03-28T16:04:36.959 INFO:teuthology.orchestra.run.smithi067.stdout: "scrubs": {} 2024-03-28T16:04:36.959 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-28T16:04:36.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.958+0000 7fb5dfe00640 1 -- 172.21.15.67:0/1027349473 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb5c40a2880 msgr2=0x7fb5c40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:36.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.958+0000 7fb5dfe00640 1 --2- 172.21.15.67:0/1027349473 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb5c40a2880 0x7fb5c40a4d40 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fb5d00b9e60 tx=0x7fb5d0009260 comp rx=0 tx=0).stop 2024-03-28T16:04:36.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.958+0000 7fb5dfe00640 1 -- 172.21.15.67:0/1027349473 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5e0102000 msgr2=0x7fb5e01977c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:36.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.958+0000 7fb5dfe00640 1 --2- 172.21.15.67:0/1027349473 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5e0102000 0x7fb5e01977c0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fb5d0000c00 tx=0x7fb5d00042f0 comp rx=0 tx=0).stop 2024-03-28T16:04:36.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.958+0000 7fb5dfe00640 1 -- 172.21.15.67:0/1027349473 shutdown_connections 2024-03-28T16:04:36.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.958+0000 7fb5dfe00640 1 --2- 172.21.15.67:0/1027349473 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb5c40a2880 0x7fb5c40a4d40 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:36.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.958+0000 7fb5dfe00640 1 --2- 172.21.15.67:0/1027349473 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb5e0107260 0x7fb5e019ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:36.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.958+0000 7fb5dfe00640 1 --2- 172.21.15.67:0/1027349473 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5e0104930 0x7fb5e0197d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:36.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.958+0000 7fb5dfe00640 1 --2- 172.21.15.67:0/1027349473 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5e0102000 0x7fb5e01977c0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:36.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.958+0000 7fb5dfe00640 1 -- 172.21.15.67:0/1027349473 >> 172.21.15.67:0/1027349473 conn(0x7fb5e00fba10 msgr2=0x7fb5e0102a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:04:36.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.959+0000 7fb5dfe00640 1 -- 172.21.15.67:0/1027349473 shutdown_connections 2024-03-28T16:04:36.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:36.959+0000 7fb5dfe00640 1 -- 172.21.15.67:0/1027349473 wait complete. 2024-03-28T16:04:36.975 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T16:04:36.975 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T16:04:37.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.155+0000 7f436b200640 1 -- 172.21.15.67:0/3815744423 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f43641071d0 msgr2=0x7f4364109620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:37.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.155+0000 7f436b200640 1 --2- 172.21.15.67:0/3815744423 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f43641071d0 0x7f4364109620 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4360009ca0 tx=0x7f436002c100 comp rx=0 tx=0).stop 2024-03-28T16:04:37.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.156+0000 7f436b200640 1 -- 172.21.15.67:0/3815744423 shutdown_connections 2024-03-28T16:04:37.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.156+0000 7f436b200640 1 --2- 172.21.15.67:0/3815744423 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f43641071d0 0x7f4364109620 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:37.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.156+0000 7f436b200640 1 --2- 172.21.15.67:0/3815744423 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43641048a0 0x7f4364106c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:37.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.156+0000 7f436b200640 1 --2- 172.21.15.67:0/3815744423 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4364101f70 0x7f4364104360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:37.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.156+0000 7f436b200640 1 -- 172.21.15.67:0/3815744423 >> 172.21.15.67:0/3815744423 conn(0x7f43640fb980 msgr2=0x7f43640fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:04:37.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.156+0000 7f436b200640 1 -- 172.21.15.67:0/3815744423 shutdown_connections 2024-03-28T16:04:37.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.156+0000 7f436b200640 1 -- 172.21.15.67:0/3815744423 wait complete. 2024-03-28T16:04:37.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.157+0000 7f436b200640 1 Processor -- start 2024-03-28T16:04:37.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.157+0000 7f436b200640 1 -- start start 2024-03-28T16:04:37.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.157+0000 7f436b200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4364101f70 0x7f4364199860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:37.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.158+0000 7f436b200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f43641048a0 0x7f4364199da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:37.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.158+0000 7f436b200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43641071d0 0x7f43641a0e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:37.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.158+0000 7f436b200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f436410d030 con 0x7f43641071d0 2024-03-28T16:04:37.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.158+0000 7f436b200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f436410ceb0 con 0x7f4364101f70 2024-03-28T16:04:37.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.158+0000 7f436b200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f436410d1b0 con 0x7f43641048a0 2024-03-28T16:04:37.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.158+0000 7f4369e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4364101f70 0x7f4364199860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:37.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.158+0000 7f436a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43641071d0 0x7f43641a0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:37.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.158+0000 7f436a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43641071d0 0x7f43641a0e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58260/0 (socket says 172.21.15.67:58260) 2024-03-28T16:04:37.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.158+0000 7f436a800640 1 -- 172.21.15.67:0/1762517833 learned_addr learned my addr 172.21.15.67:0/1762517833 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:04:37.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.159+0000 7f4369400640 1 --2- 172.21.15.67:0/1762517833 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f43641048a0 0x7f4364199da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:37.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.159+0000 7f436a800640 1 -- 172.21.15.67:0/1762517833 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f43641048a0 msgr2=0x7f4364199da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:37.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.159+0000 7f436a800640 1 --2- 172.21.15.67:0/1762517833 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f43641048a0 0x7f4364199da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:37.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.159+0000 7f436a800640 1 -- 172.21.15.67:0/1762517833 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4364101f70 msgr2=0x7f4364199860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:37.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.159+0000 7f436a800640 1 --2- 172.21.15.67:0/1762517833 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4364101f70 0x7f4364199860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:37.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.159+0000 7f436a800640 1 -- 172.21.15.67:0/1762517833 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43641a1480 con 0x7f43641071d0 2024-03-28T16:04:37.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.160+0000 7f436a800640 1 --2- 172.21.15.67:0/1762517833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43641071d0 0x7f43641a0e10 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f4360032520 tx=0x7f4360035ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:04:37.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.160+0000 7f435ea00640 1 -- 172.21.15.67:0/1762517833 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4360039040 con 0x7f43641071d0 2024-03-28T16:04:37.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.160+0000 7f435ea00640 1 -- 172.21.15.67:0/1762517833 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4360044070 con 0x7f43641071d0 2024-03-28T16:04:37.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.160+0000 7f435ea00640 1 -- 172.21.15.67:0/1762517833 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4360004330 con 0x7f43641071d0 2024-03-28T16:04:37.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.160+0000 7f436b200640 1 -- 172.21.15.67:0/1762517833 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43641a1710 con 0x7f43641071d0 2024-03-28T16:04:37.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.161+0000 7f436b200640 1 -- 172.21.15.67:0/1762517833 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43641a1bb0 con 0x7f43641071d0 2024-03-28T16:04:37.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.162+0000 7f436b200640 1 -- 172.21.15.67:0/1762517833 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4364108cc0 con 0x7f43641071d0 2024-03-28T16:04:37.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.166+0000 7f435ea00640 1 -- 172.21.15.67:0/1762517833 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f43600044d0 con 0x7f43641071d0 2024-03-28T16:04:37.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.167+0000 7f435ea00640 1 --2- 172.21.15.67:0/1762517833 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f434c0a2880 0x7f434c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:37.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.167+0000 7f435ea00640 1 -- 172.21.15.67:0/1762517833 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f43600f0b10 con 0x7f43641071d0 2024-03-28T16:04:37.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.168+0000 7f4369e00640 1 --2- 172.21.15.67:0/1762517833 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f434c0a2880 0x7f434c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:37.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.169+0000 7f4369e00640 1 --2- 172.21.15.67:0/1762517833 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f434c0a2880 0x7f434c0a4d40 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f4358009b60 tx=0x7f4358005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:04:37.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.177+0000 7f435ea00640 1 -- 172.21.15.67:0/1762517833 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f43600bdbc0 con 0x7f43641071d0 2024-03-28T16:04:37.321 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:04:37 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T16:04:37.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.540+0000 7f436b200640 1 -- 172.21.15.67:0/1762517833 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f436402d070 con 0x7f43641071d0 2024-03-28T16:04:37.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.542+0000 7f435ea00640 1 -- 172.21.15.67:0/1762517833 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11341 (secure 0 0 0) 0x7f43600bd4c0 con 0x7f43641071d0 2024-03-28T16:04:37.543 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:04:37.543 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2008297659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2008297659},{"type":"v1","addr":"172.21.15.67:6835","nonce":2008297659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14532,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/320295745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":320295745},{"type":"v1","addr":"172.21.15.67:6837","nonce":320295745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6839/691821273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":691821273},{"type":"v1","addr":"172.21.15.67:6839","nonce":691821273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6841/3211254875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6840","nonce":3211254875},{"type":"v1","addr":"172.21.15.67:6841","nonce":3211254875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2129665423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2129665423},{"type":"v1","addr":"172.21.15.183:6833","nonce":2129665423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24496,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/1589784425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":1589784425},{"type":"v1","addr":"172.21.15.183:6835","nonce":1589784425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24508,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/2185863835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":2185863835},{"type":"v1","addr":"172.21.15.183:6837","nonce":2185863835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24520,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6839/1242814398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":1242814398},{"type":"v1","addr":"172.21.15.183:6839","nonce":1242814398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":34409,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1008335521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1008335521},{"type":"v1","addr":"172.21.15.150:6833","nonce":1008335521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":34445,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3269935695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3269935695},{"type":"v1","addr":"172.21.15.150:6835","nonce":3269935695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T16:01:07.696824+0000","modified":"2024-03-28T16:01:11.997773+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":34508},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34475":{"gid":34475,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/1112886277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":1112886277},{"type":"v1","addr":"172.21.15.150:6837","nonce":1112886277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_34508":{"gid":34508,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6839/1211011614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":1211011614},{"type":"v1","addr":"172.21.15.150:6839","nonce":1211011614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34508,"qdb_cluster":[34508]},"id":1}]} 2024-03-28T16:04:37.543 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 19 2024-03-28T16:04:37.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.545+0000 7f436b200640 1 -- 172.21.15.67:0/1762517833 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f434c0a2880 msgr2=0x7f434c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:37.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.545+0000 7f436b200640 1 --2- 172.21.15.67:0/1762517833 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f434c0a2880 0x7f434c0a4d40 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f4358009b60 tx=0x7f4358005f90 comp rx=0 tx=0).stop 2024-03-28T16:04:37.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.545+0000 7f436b200640 1 -- 172.21.15.67:0/1762517833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43641071d0 msgr2=0x7f43641a0e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:37.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.545+0000 7f436b200640 1 --2- 172.21.15.67:0/1762517833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43641071d0 0x7f43641a0e10 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f4360032520 tx=0x7f4360035ce0 comp rx=0 tx=0).stop 2024-03-28T16:04:37.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.546+0000 7f436b200640 1 -- 172.21.15.67:0/1762517833 shutdown_connections 2024-03-28T16:04:37.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.546+0000 7f436b200640 1 --2- 172.21.15.67:0/1762517833 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f434c0a2880 0x7f434c0a4d40 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:37.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.546+0000 7f436b200640 1 --2- 172.21.15.67:0/1762517833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43641071d0 0x7f43641a0e10 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:37.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.546+0000 7f436b200640 1 --2- 172.21.15.67:0/1762517833 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f43641048a0 0x7f4364199da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:37.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.546+0000 7f436b200640 1 --2- 172.21.15.67:0/1762517833 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4364101f70 0x7f4364199860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:37.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.546+0000 7f436b200640 1 -- 172.21.15.67:0/1762517833 >> 172.21.15.67:0/1762517833 conn(0x7f43640fb980 msgr2=0x7f43641015a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:04:37.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.546+0000 7f436b200640 1 -- 172.21.15.67:0/1762517833 shutdown_connections 2024-03-28T16:04:37.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.546+0000 7f436b200640 1 -- 172.21.15.67:0/1762517833 wait complete. 2024-03-28T16:04:37.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:04:37 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T16:04:37.572 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T16:04:37.573 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T16:04:37.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:04:37 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T16:04:37.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.772+0000 7fea37e00640 1 -- 172.21.15.67:0/2078330154 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea381015a0 msgr2=0x7fea381019c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:37.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.772+0000 7fea37e00640 1 --2- 172.21.15.67:0/2078330154 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea381015a0 0x7fea381019c0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fea28009900 tx=0x7fea2802bc30 comp rx=0 tx=0).stop 2024-03-28T16:04:37.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.772+0000 7fea37e00640 1 -- 172.21.15.67:0/2078330154 shutdown_connections 2024-03-28T16:04:37.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.772+0000 7fea37e00640 1 --2- 172.21.15.67:0/2078330154 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fea38103150 0x7fea381099e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:37.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.772+0000 7fea37e00640 1 --2- 172.21.15.67:0/2078330154 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea38102790 0x7fea38102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:37.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.772+0000 7fea37e00640 1 --2- 172.21.15.67:0/2078330154 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea381015a0 0x7fea381019c0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:37.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.772+0000 7fea37e00640 1 -- 172.21.15.67:0/2078330154 >> 172.21.15.67:0/2078330154 conn(0x7fea380fcb10 msgr2=0x7fea380fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:04:37.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.772+0000 7fea37e00640 1 -- 172.21.15.67:0/2078330154 shutdown_connections 2024-03-28T16:04:37.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.772+0000 7fea37e00640 1 -- 172.21.15.67:0/2078330154 wait complete. 2024-03-28T16:04:37.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.773+0000 7fea37e00640 1 Processor -- start 2024-03-28T16:04:37.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.773+0000 7fea37e00640 1 -- start start 2024-03-28T16:04:37.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.774+0000 7fea37e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea381015a0 0x7fea381073f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:37.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.774+0000 7fea37e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fea38102790 0x7fea38107930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:37.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.774+0000 7fea37e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea38103150 0x7fea38103f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:37.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.774+0000 7fea37e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea3810d110 con 0x7fea38103150 2024-03-28T16:04:37.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.774+0000 7fea37e00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea3810cf90 con 0x7fea381015a0 2024-03-28T16:04:37.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.774+0000 7fea37e00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea3810d290 con 0x7fea38102790 2024-03-28T16:04:37.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.774+0000 7fea37400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea38103150 0x7fea38103f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:37.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.774+0000 7fea36a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea381015a0 0x7fea381073f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:37.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.774+0000 7fea36a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea381015a0 0x7fea381073f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.67:51112/0 (socket says 172.21.15.67:51112) 2024-03-28T16:04:37.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.774+0000 7fea36a00640 1 -- 172.21.15.67:0/2854358990 learned_addr learned my addr 172.21.15.67:0/2854358990 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:04:37.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.774+0000 7fea36000640 1 --2- 172.21.15.67:0/2854358990 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fea38102790 0x7fea38107930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:37.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.775+0000 7fea37400640 1 -- 172.21.15.67:0/2854358990 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fea38102790 msgr2=0x7fea38107930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:37.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.775+0000 7fea37400640 1 --2- 172.21.15.67:0/2854358990 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fea38102790 0x7fea38107930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:37.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.775+0000 7fea37400640 1 -- 172.21.15.67:0/2854358990 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea381015a0 msgr2=0x7fea381073f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:37.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.775+0000 7fea37400640 1 --2- 172.21.15.67:0/2854358990 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea381015a0 0x7fea381073f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:37.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.775+0000 7fea37400640 1 -- 172.21.15.67:0/2854358990 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea38104820 con 0x7fea38103150 2024-03-28T16:04:37.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.775+0000 7fea36a00640 1 --2- 172.21.15.67:0/2854358990 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea381015a0 0x7fea381073f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T16:04:37.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.776+0000 7fea37400640 1 --2- 172.21.15.67:0/2854358990 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea38103150 0x7fea38103f60 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fea3000e940 tx=0x7fea3000ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:04:37.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.776+0000 7fea2f400640 1 -- 172.21.15.67:0/2854358990 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea30004d20 con 0x7fea38103150 2024-03-28T16:04:37.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.776+0000 7fea2f400640 1 -- 172.21.15.67:0/2854358990 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fea30005020 con 0x7fea38103150 2024-03-28T16:04:37.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.776+0000 7fea2f400640 1 -- 172.21.15.67:0/2854358990 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea300116d0 con 0x7fea38103150 2024-03-28T16:04:37.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.776+0000 7fea37e00640 1 -- 172.21.15.67:0/2854358990 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea381a5e20 con 0x7fea38103150 2024-03-28T16:04:37.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.776+0000 7fea37e00640 1 -- 172.21.15.67:0/2854358990 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea381a61d0 con 0x7fea38103150 2024-03-28T16:04:37.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.780+0000 7fea4096b740 1 Processor -- start 2024-03-28T16:04:37.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.782+0000 7fea2f400640 1 -- 172.21.15.67:0/2854358990 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fea3000cb20 con 0x7fea38103150 2024-03-28T16:04:37.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.782+0000 7fea4096b740 1 -- start start 2024-03-28T16:04:37.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.782+0000 7fea4096b740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f75b9ed9f0 0x55f75b9ede10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:37.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.782+0000 7fea4096b740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f75b9eebc0 0x55f75b9ef040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:37.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.783+0000 7fea4096b740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f75b9ef580 0x55f75b9f5e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:37.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.783+0000 7fea4096b740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f75b9522c0 con 0x55f75b9ed9f0 2024-03-28T16:04:37.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.783+0000 7fea4096b740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f75b8fac90 con 0x55f75b9eebc0 2024-03-28T16:04:37.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.783+0000 7fea4096b740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f75b94e360 con 0x55f75b9ef580 2024-03-28T16:04:37.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.783+0000 7fea36000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f75b9ed9f0 0x55f75b9ede10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:37.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.783+0000 7fea36000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f75b9ed9f0 0x55f75b9ede10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58282/0 (socket says 172.21.15.67:58282) 2024-03-28T16:04:37.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.783+0000 7fea36000640 1 -- 172.21.15.67:0/2115118795 learned_addr learned my addr 172.21.15.67:0/2115118795 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:04:37.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.783+0000 7fea36a00640 1 --2- 172.21.15.67:0/2115118795 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f75b9eebc0 0x55f75b9ef040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:37.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.784+0000 7fea36000640 1 --2- 172.21.15.67:0/2115118795 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f75b9ef580 0x55f75b9f5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:37.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.784+0000 7fea36a00640 1 -- 172.21.15.67:0/2115118795 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f75b9ef580 msgr2=0x55f75b9f5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:04:37.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.784+0000 7fea36a00640 1 --2- 172.21.15.67:0/2115118795 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f75b9ef580 0x55f75b9f5e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:37.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.784+0000 7fea36a00640 1 -- 172.21.15.67:0/2115118795 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f75b9ed9f0 msgr2=0x55f75b9ede10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:04:37.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.784+0000 7fea36a00640 1 --2- 172.21.15.67:0/2115118795 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f75b9ed9f0 0x55f75b9ede10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:37.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.784+0000 7fea36a00640 1 -- 172.21.15.67:0/2115118795 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f75b91f080 con 0x55f75b9eebc0 2024-03-28T16:04:37.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.784+0000 7fea36000640 1 --2- 172.21.15.67:0/2115118795 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f75b9ed9f0 0x55f75b9ede10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T16:04:37.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.785+0000 7fea2f400640 1 --2- 172.21.15.67:0/2854358990 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fea1c0a2880 0x7fea1c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:37.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.785+0000 7fea2f400640 1 -- 172.21.15.67:0/2854358990 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fea300ced60 con 0x7fea38103150 2024-03-28T16:04:37.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.785+0000 7fea36a00640 1 --2- 172.21.15.67:0/2115118795 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f75b9eebc0 0x55f75b9ef040 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fea28005ab0 tx=0x7fea28005e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55a7993c05dd5dcd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:04:37.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.786+0000 7fea36000640 1 --2- 172.21.15.67:0/2854358990 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fea1c0a2880 0x7fea1c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:37.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.786+0000 7fea23e00640 1 -- 172.21.15.67:0/2115118795 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea28045020 con 0x55f75b9eebc0 2024-03-28T16:04:37.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.786+0000 7fea23e00640 1 -- 172.21.15.67:0/2115118795 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fea28005040 con 0x55f75b9eebc0 2024-03-28T16:04:37.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.786+0000 7fea23e00640 1 -- 172.21.15.67:0/2115118795 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea28004a00 con 0x55f75b9eebc0 2024-03-28T16:04:37.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.787+0000 7fea36000640 1 --2- 172.21.15.67:0/2854358990 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fea1c0a2880 0x7fea1c0a4d40 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x55f75b9eeac0 tx=0x7fea24014040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:04:37.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.787+0000 7fea4096b740 1 -- 172.21.15.67:0/2115118795 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f75b9eebc0 msgr2=0x55f75b9ef040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:37.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.787+0000 7fea4096b740 1 --2- 172.21.15.67:0/2115118795 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f75b9eebc0 0x55f75b9ef040 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fea28005ab0 tx=0x7fea28005e00 comp rx=0 tx=0).stop 2024-03-28T16:04:37.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.787+0000 7fea4096b740 1 -- 172.21.15.67:0/2115118795 shutdown_connections 2024-03-28T16:04:37.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.787+0000 7fea4096b740 1 --2- 172.21.15.67:0/2115118795 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f75b9ef580 0x55f75b9f5e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:37.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.787+0000 7fea4096b740 1 --2- 172.21.15.67:0/2115118795 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f75b9eebc0 0x55f75b9ef040 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:37.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.787+0000 7fea4096b740 1 --2- 172.21.15.67:0/2115118795 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f75b9ed9f0 0x55f75b9ede10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:37.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.787+0000 7fea4096b740 1 -- 172.21.15.67:0/2115118795 >> 172.21.15.67:0/2115118795 conn(0x55f75b9e8de0 msgr2=0x55f75b9e91e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:04:37.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.788+0000 7fea4096b740 1 -- 172.21.15.67:0/2115118795 shutdown_connections 2024-03-28T16:04:37.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.788+0000 7fea4096b740 1 -- 172.21.15.67:0/2115118795 wait complete. 2024-03-28T16:04:37.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.788+0000 7fea4096b740 1 -- start start 2024-03-28T16:04:37.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.788+0000 7fea4096b740 1 Processor -- start 2024-03-28T16:04:37.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.789+0000 7fea4096b740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f75b9ed9f0 0x55f75b9f7b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:37.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.789+0000 7fea4096b740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f75b9eebc0 0x55f75b9f8090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:37.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.789+0000 7fea4096b740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f75b9ef580 0x55f75b9ff0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:37.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.789+0000 7fea4096b740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f75b9f6530 con 0x55f75b9ef580 2024-03-28T16:04:37.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.789+0000 7fea4096b740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f75b90e0d0 con 0x55f75b9eebc0 2024-03-28T16:04:37.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.789+0000 7fea4096b740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f75b9f66b0 con 0x55f75b9ed9f0 2024-03-28T16:04:37.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.789+0000 7fea4096b740 10 client.0 _ll_register_callbacks cb 0x55f75b912e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:04:37.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.790+0000 7fea37400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f75b9ef580 0x55f75b9ff0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:37.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.790+0000 7fea37400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f75b9ef580 0x55f75b9ff0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58284/0 (socket says 172.21.15.67:58284) 2024-03-28T16:04:37.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.790+0000 7fea36000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f75b9eebc0 0x55f75b9f8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:37.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.790+0000 7fea37400640 1 -- 172.21.15.67:0/665568477 learned_addr learned my addr 172.21.15.67:0/665568477 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:04:37.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.790+0000 7fea36a00640 1 --2- 172.21.15.67:0/665568477 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f75b9ed9f0 0x55f75b9f7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:37.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.790+0000 7fea37400640 1 -- 172.21.15.67:0/665568477 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f75b9ed9f0 msgr2=0x55f75b9f7b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:04:37.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.790+0000 7fea37400640 1 --2- 172.21.15.67:0/665568477 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f75b9ed9f0 0x55f75b9f7b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:37.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.790+0000 7fea37400640 1 -- 172.21.15.67:0/665568477 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f75b9eebc0 msgr2=0x55f75b9f8090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:04:37.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.790+0000 7fea37400640 1 --2- 172.21.15.67:0/665568477 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f75b9eebc0 0x55f75b9f8090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:37.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.790+0000 7fea37400640 1 -- 172.21.15.67:0/665568477 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f75b6663f0 con 0x55f75b9ef580 2024-03-28T16:04:37.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.791+0000 7fea37400640 1 --2- 172.21.15.67:0/665568477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f75b9ef580 0x55f75b9ff0d0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fea300a3920 tx=0x7fea3009b360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f927436d1ce8168 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:04:37.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.791+0000 7fea20c00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:04:37.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.791+0000 7fea20c00640 1 -- 172.21.15.67:0/665568477 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea3001f040 con 0x55f75b9ef580 2024-03-28T16:04:37.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.791+0000 7fea20c00640 1 -- 172.21.15.67:0/665568477 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fea300054d0 con 0x55f75b9ef580 2024-03-28T16:04:37.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.791+0000 7fea4096b740 1 -- 172.21.15.67:0/665568477 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fea28005040 con 0x55f75b9ef580 2024-03-28T16:04:37.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.791+0000 7fea20c00640 1 -- 172.21.15.67:0/665568477 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fea3009f800 con 0x55f75b9ef580 2024-03-28T16:04:37.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.791+0000 7fea4096b740 10 client.15441 fetch_fsmap learned FSMap version 19 2024-03-28T16:04:37.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.791+0000 7fea4096b740 1 -- 172.21.15.67:0/665568477 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55f75b7307d0 con 0x55f75b9ef580 2024-03-28T16:04:37.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.792+0000 7fea20c00640 1 -- 172.21.15.67:0/665568477 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fea28005040 con 0x55f75b9ef580 2024-03-28T16:04:37.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.792+0000 7fea20c00640 1 -- 172.21.15.67:0/665568477 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fea3000f040 con 0x55f75b9ef580 2024-03-28T16:04:37.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.792+0000 7fea4096b740 10 client.15441 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:04:37.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.792+0000 7fea4096b740 10 client.15441 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:04:37.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.792+0000 7fea4096b740 20 client.15441 populate_metadata read hostname 'smithi067' 2024-03-28T16:04:37.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.792+0000 7fea4096b740 1 --2- 172.21.15.67:0/665568477 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55f75ba03600 0x55f75ba05a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:37.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.792+0000 7fea4096b740 4 client.15441 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:04:37.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.792+0000 7fea4096b740 1 -- 172.21.15.67:0/665568477 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f75b95f800 con 0x55f75ba03600 2024-03-28T16:04:37.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.793+0000 7fea36a00640 1 --2- 172.21.15.67:0/665568477 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55f75ba03600 0x55f75ba05a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:37.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.794+0000 7fea20c00640 10 client.15441 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:04:37.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.794+0000 7fea36a00640 1 --2- 172.21.15.67:0/665568477 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55f75ba03600 0x55f75ba05a10 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=15a3c4378ebf7664 server_cookie=6bfcda371ac2efe9 in_seq=0 out_seq=0 2024-03-28T16:04:37.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.796+0000 7fea20c00640 1 -- 172.21.15.67:0/665568477 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f75b95f800 con 0x55f75ba03600 2024-03-28T16:04:37.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.796+0000 7fea20c00640 10 client.15441 handle_command_reply: tid=0 2024-03-28T16:04:37.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.796+0000 7fea4096b740 1 client.15441 shutdown 2024-03-28T16:04:37.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.796+0000 7fea4096b740 1 -- 172.21.15.67:0/665568477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f75b9ef580 msgr2=0x55f75b9ff0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:37.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.796+0000 7fea4096b740 1 --2- 172.21.15.67:0/665568477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f75b9ef580 0x55f75b9ff0d0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fea300a3920 tx=0x7fea3009b360 comp rx=0 tx=0).stop 2024-03-28T16:04:37.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.796+0000 7fea4096b740 1 -- 172.21.15.67:0/665568477 shutdown_connections 2024-03-28T16:04:37.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.796+0000 7fea4096b740 1 --2- 172.21.15.67:0/665568477 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55f75ba03600 0x55f75ba05a10 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:37.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.797+0000 7fea4096b740 1 --2- 172.21.15.67:0/665568477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f75b9ef580 0x55f75b9ff0d0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:37.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.797+0000 7fea4096b740 1 --2- 172.21.15.67:0/665568477 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f75b9eebc0 0x55f75b9f8090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:37.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.797+0000 7fea4096b740 1 --2- 172.21.15.67:0/665568477 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f75b9ed9f0 0x55f75b9f7b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:37.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.797+0000 7fea4096b740 1 -- 172.21.15.67:0/665568477 >> 172.21.15.67:0/665568477 conn(0x55f75b9e8de0 msgr2=0x55f75b9e8160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:04:37.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.797+0000 7fea20c00640 0 client.15441 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:04:37.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.797+0000 7fea4096b740 1 -- 172.21.15.67:0/665568477 shutdown_connections 2024-03-28T16:04:37.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.797+0000 7fea4096b740 1 -- 172.21.15.67:0/665568477 wait complete. 2024-03-28T16:04:37.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.797+0000 7fea4096b740 20 client.15441 trim_cache size 0 max 16384 2024-03-28T16:04:37.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.810+0000 7fea4096b740 1 Processor -- start 2024-03-28T16:04:37.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.810+0000 7fea4096b740 1 -- start start 2024-03-28T16:04:37.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.811+0000 7fea4096b740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f75b9ed9f0 0x55f75b9f1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:37.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.811+0000 7fea4096b740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f75b9eebc0 0x55f75b9ff090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:37.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.811+0000 7fea4096b740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f75b9ef580 0x55f75b9f2100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:37.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.811+0000 7fea4096b740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea381acdf0 con 0x55f75b9eebc0 2024-03-28T16:04:37.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.811+0000 7fea4096b740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea381af3b0 con 0x55f75b9ef580 2024-03-28T16:04:37.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.811+0000 7fea4096b740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea381ad3f0 con 0x55f75b9ed9f0 2024-03-28T16:04:37.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.811+0000 7fea36000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f75b9eebc0 0x55f75b9ff090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:37.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.811+0000 7fea36000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f75b9eebc0 0x55f75b9ff090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58288/0 (socket says 172.21.15.67:58288) 2024-03-28T16:04:37.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.811+0000 7fea36000640 1 -- 172.21.15.67:0/3615446303 learned_addr learned my addr 172.21.15.67:0/3615446303 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:04:37.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.811+0000 7fea37400640 1 --2- 172.21.15.67:0/3615446303 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f75b9ef580 0x55f75b9f2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:37.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.812+0000 7fea36a00640 1 --2- 172.21.15.67:0/3615446303 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f75b9ed9f0 0x55f75b9f1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:37.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.812+0000 7fea36000640 1 -- 172.21.15.67:0/3615446303 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f75b9ed9f0 msgr2=0x55f75b9f1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:04:37.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.812+0000 7fea36000640 1 --2- 172.21.15.67:0/3615446303 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f75b9ed9f0 0x55f75b9f1bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:37.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.812+0000 7fea36000640 1 -- 172.21.15.67:0/3615446303 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f75b9ef580 msgr2=0x55f75b9f2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:04:37.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.812+0000 7fea36000640 1 --2- 172.21.15.67:0/3615446303 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f75b9ef580 0x55f75b9f2100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:37.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.812+0000 7fea36000640 1 -- 172.21.15.67:0/3615446303 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f75b9f7940 con 0x55f75b9eebc0 2024-03-28T16:04:37.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.812+0000 7fea36000640 1 --2- 172.21.15.67:0/3615446303 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f75b9eebc0 0x55f75b9ff090 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fea24012dc0 tx=0x7fea240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a47e4afaace291d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:04:37.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.813+0000 7fea23400640 1 -- 172.21.15.67:0/3615446303 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea24018030 con 0x55f75b9eebc0 2024-03-28T16:04:37.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.813+0000 7fea23400640 1 -- 172.21.15.67:0/3615446303 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fea2401a070 con 0x55f75b9eebc0 2024-03-28T16:04:37.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.813+0000 7fea23400640 1 -- 172.21.15.67:0/3615446303 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea240208b0 con 0x55f75b9eebc0 2024-03-28T16:04:37.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.814+0000 7fea4096b740 1 -- 172.21.15.67:0/3615446303 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f75b9eebc0 msgr2=0x55f75b9ff090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:37.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.814+0000 7fea4096b740 1 --2- 172.21.15.67:0/3615446303 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f75b9eebc0 0x55f75b9ff090 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fea24012dc0 tx=0x7fea240046c0 comp rx=0 tx=0).stop 2024-03-28T16:04:37.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.814+0000 7fea4096b740 1 -- 172.21.15.67:0/3615446303 shutdown_connections 2024-03-28T16:04:37.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.814+0000 7fea4096b740 1 --2- 172.21.15.67:0/3615446303 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f75b9ef580 0x55f75b9f2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:37.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.814+0000 7fea4096b740 1 --2- 172.21.15.67:0/3615446303 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f75b9eebc0 0x55f75b9ff090 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:37.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.814+0000 7fea4096b740 1 --2- 172.21.15.67:0/3615446303 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f75b9ed9f0 0x55f75b9f1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:37.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.814+0000 7fea4096b740 1 -- 172.21.15.67:0/3615446303 >> 172.21.15.67:0/3615446303 conn(0x55f75b9e8de0 msgr2=0x55f75b9ec8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:04:37.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.814+0000 7fea4096b740 1 -- 172.21.15.67:0/3615446303 shutdown_connections 2024-03-28T16:04:37.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.814+0000 7fea4096b740 1 -- 172.21.15.67:0/3615446303 wait complete. 2024-03-28T16:04:37.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.814+0000 7fea4096b740 1 -- start start 2024-03-28T16:04:37.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.815+0000 7fea4096b740 1 Processor -- start 2024-03-28T16:04:37.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.815+0000 7fea4096b740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f75b9ed9f0 0x55f75b9fb110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:37.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.815+0000 7fea4096b740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f75b9ef580 0x55f75b9fb650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:37.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.815+0000 7fea4096b740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f75b9fe670 0x55f75b9feaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:37.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.815+0000 7fea4096b740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea381ada30 con 0x55f75b9fe670 2024-03-28T16:04:37.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.815+0000 7fea4096b740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea381aed50 con 0x55f75b9ef580 2024-03-28T16:04:37.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.815+0000 7fea4096b740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea381ae090 con 0x55f75b9ed9f0 2024-03-28T16:04:37.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.815+0000 7fea4096b740 10 client.0 _ll_register_callbacks cb 0x7fea30004a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:04:37.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.816+0000 7fea37400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f75b9fe670 0x55f75b9feaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:37.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.816+0000 7fea37400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f75b9fe670 0x55f75b9feaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58290/0 (socket says 172.21.15.67:58290) 2024-03-28T16:04:37.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.816+0000 7fea37400640 1 -- 172.21.15.67:0/135395585 learned_addr learned my addr 172.21.15.67:0/135395585 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:04:37.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.816+0000 7fea36000640 1 --2- 172.21.15.67:0/135395585 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f75b9ef580 0x55f75b9fb650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:37.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.816+0000 7fea36a00640 1 --2- 172.21.15.67:0/135395585 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f75b9ed9f0 0x55f75b9fb110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:37.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.816+0000 7fea37400640 1 -- 172.21.15.67:0/135395585 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f75b9ed9f0 msgr2=0x55f75b9fb110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:04:37.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.816+0000 7fea37400640 1 --2- 172.21.15.67:0/135395585 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f75b9ed9f0 0x55f75b9fb110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:37.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.816+0000 7fea37400640 1 -- 172.21.15.67:0/135395585 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f75b9ef580 msgr2=0x55f75b9fb650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:04:37.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.816+0000 7fea37400640 1 --2- 172.21.15.67:0/135395585 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f75b9ef580 0x55f75b9fb650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:37.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.816+0000 7fea37400640 1 -- 172.21.15.67:0/135395585 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f75b9ff030 con 0x55f75b9fe670 2024-03-28T16:04:37.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.817+0000 7fea37400640 1 --2- 172.21.15.67:0/135395585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f75b9fe670 0x55f75b9feaf0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fea30004ff0 tx=0x7fea30098030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d6f007bef5649bf server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:04:37.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.817+0000 7fea20c00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:04:37.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.817+0000 7fea20c00640 1 -- 172.21.15.67:0/135395585 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea3001f070 con 0x55f75b9fe670 2024-03-28T16:04:37.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.817+0000 7fea20c00640 1 -- 172.21.15.67:0/135395585 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fea300a3bf0 con 0x55f75b9fe670 2024-03-28T16:04:37.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.817+0000 7fea4096b740 1 -- 172.21.15.67:0/135395585 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fea2401a070 con 0x55f75b9fe670 2024-03-28T16:04:37.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.818+0000 7fea20c00640 1 -- 172.21.15.67:0/135395585 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fea300a3e50 con 0x55f75b9fe670 2024-03-28T16:04:37.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.818+0000 7fea4096b740 10 client.15447 fetch_fsmap learned FSMap version 19 2024-03-28T16:04:37.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.818+0000 7fea4096b740 1 -- 172.21.15.67:0/135395585 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55f75ba5fc20 con 0x55f75b9fe670 2024-03-28T16:04:37.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.818+0000 7fea20c00640 1 -- 172.21.15.67:0/135395585 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fea2401a070 con 0x55f75b9fe670 2024-03-28T16:04:37.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.818+0000 7fea20c00640 1 -- 172.21.15.67:0/135395585 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fea3009a640 con 0x55f75b9fe670 2024-03-28T16:04:37.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.818+0000 7fea4096b740 10 client.15447 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:04:37.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.818+0000 7fea4096b740 10 client.15447 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:04:37.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.819+0000 7fea4096b740 20 client.15447 populate_metadata read hostname 'smithi067' 2024-03-28T16:04:37.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.819+0000 7fea4096b740 1 --2- 172.21.15.67:0/135395585 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55f75ba604f0 0x55f75ba62910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:37.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.819+0000 7fea4096b740 4 client.15447 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T16:04:37.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.819+0000 7fea4096b740 1 -- 172.21.15.67:0/135395585 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55f75b959b50 con 0x55f75ba604f0 2024-03-28T16:04:37.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.819+0000 7fea36a00640 1 --2- 172.21.15.67:0/135395585 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55f75ba604f0 0x55f75ba62910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:37.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.820+0000 7fea36a00640 1 --2- 172.21.15.67:0/135395585 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55f75ba604f0 0x55f75ba62910 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9abdc75cd7e2a227 server_cookie=ca6b5bb772fc954 in_seq=0 out_seq=0 2024-03-28T16:04:37.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.820+0000 7fea20c00640 10 client.15447 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:04:37.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.821+0000 7fea20c00640 1 -- 172.21.15.67:0/135395585 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55f75b959b50 con 0x55f75ba604f0 2024-03-28T16:04:37.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.821+0000 7fea20c00640 10 client.15447 handle_command_reply: tid=0 2024-03-28T16:04:37.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.821+0000 7fea4096b740 1 client.15447 shutdown 2024-03-28T16:04:37.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.821+0000 7fea4096b740 1 -- 172.21.15.67:0/135395585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f75b9fe670 msgr2=0x55f75b9feaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:37.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.821+0000 7fea4096b740 1 --2- 172.21.15.67:0/135395585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f75b9fe670 0x55f75b9feaf0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fea30004ff0 tx=0x7fea30098030 comp rx=0 tx=0).stop 2024-03-28T16:04:37.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.821+0000 7fea4096b740 1 -- 172.21.15.67:0/135395585 shutdown_connections 2024-03-28T16:04:37.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.821+0000 7fea4096b740 1 --2- 172.21.15.67:0/135395585 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55f75ba604f0 0x55f75ba62910 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:37.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.821+0000 7fea4096b740 1 --2- 172.21.15.67:0/135395585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f75b9fe670 0x55f75b9feaf0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:37.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.821+0000 7fea4096b740 1 --2- 172.21.15.67:0/135395585 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f75b9ef580 0x55f75b9fb650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:37.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.821+0000 7fea4096b740 1 --2- 172.21.15.67:0/135395585 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f75b9ed9f0 0x55f75b9fb110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:37.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.821+0000 7fea4096b740 1 -- 172.21.15.67:0/135395585 >> 172.21.15.67:0/135395585 conn(0x55f75b9e8de0 msgr2=0x55f75b9e78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:04:37.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.821+0000 7fea20c00640 0 client.15447 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:04:37.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.821+0000 7fea4096b740 1 -- 172.21.15.67:0/135395585 shutdown_connections 2024-03-28T16:04:37.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.821+0000 7fea4096b740 1 -- 172.21.15.67:0/135395585 wait complete. 2024-03-28T16:04:37.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.822+0000 7fea4096b740 20 client.15447 trim_cache size 0 max 16384 2024-03-28T16:04:37.824 INFO:teuthology.orchestra.run.smithi067.stdout:[] 2024-03-28T16:04:37.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.822+0000 7fea37e00640 1 -- 172.21.15.67:0/2854358990 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fea1c0a2880 msgr2=0x7fea1c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:37.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.822+0000 7fea37e00640 1 --2- 172.21.15.67:0/2854358990 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fea1c0a2880 0x7fea1c0a4d40 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x55f75b9eeac0 tx=0x7fea24014040 comp rx=0 tx=0).stop 2024-03-28T16:04:37.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.822+0000 7fea37e00640 1 -- 172.21.15.67:0/2854358990 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea38103150 msgr2=0x7fea38103f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:37.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.822+0000 7fea37e00640 1 --2- 172.21.15.67:0/2854358990 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea38103150 0x7fea38103f60 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fea3000e940 tx=0x7fea3000ee00 comp rx=0 tx=0).stop 2024-03-28T16:04:37.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.823+0000 7fea37e00640 1 -- 172.21.15.67:0/2854358990 shutdown_connections 2024-03-28T16:04:37.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.823+0000 7fea37e00640 1 --2- 172.21.15.67:0/2854358990 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fea1c0a2880 0x7fea1c0a4d40 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:37.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.823+0000 7fea37e00640 1 --2- 172.21.15.67:0/2854358990 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea38103150 0x7fea38103f60 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:37.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.823+0000 7fea37e00640 1 --2- 172.21.15.67:0/2854358990 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fea38102790 0x7fea38107930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:37.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.823+0000 7fea37e00640 1 --2- 172.21.15.67:0/2854358990 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea381015a0 0x7fea381073f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:37.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.823+0000 7fea37e00640 1 -- 172.21.15.67:0/2854358990 >> 172.21.15.67:0/2854358990 conn(0x7fea380fcb10 msgr2=0x7fea380738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:04:37.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.823+0000 7fea37e00640 1 -- 172.21.15.67:0/2854358990 shutdown_connections 2024-03-28T16:04:37.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:37.823+0000 7fea37e00640 1 -- 172.21.15.67:0/2854358990 wait complete. 2024-03-28T16:04:38.230 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:04:38 smithi067 ceph-mon[31707]: pgmap v471: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T16:04:38.231 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:04:38 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/1762517833' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:04:38.231 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:04:38 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:04:38.231 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:04:38 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T16:04:38.231 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:04:38 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:04:38.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:04:38 smithi150 ceph-mon[34825]: pgmap v471: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T16:04:38.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:04:38 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/1762517833' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:04:38.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:04:38 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:04:38.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:04:38 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T16:04:38.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:04:38 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:04:38.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:04:38 smithi183 ceph-mon[35580]: pgmap v471: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T16:04:38.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:04:38 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/1762517833' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:04:38.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:04:38 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:04:38.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:04:38 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T16:04:38.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:04:38 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:04:38.840 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T16:04:38.840 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T16:04:38.840 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T16:04:39.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.088+0000 7fd146600640 1 -- 172.21.15.67:0/3111379025 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd140104930 msgr2=0x7fd140106d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:39.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.088+0000 7fd146600640 1 --2- 172.21.15.67:0/3111379025 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd140104930 0x7fd140106d20 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fd12c009900 tx=0x7fd12c02bc30 comp rx=0 tx=0).stop 2024-03-28T16:04:39.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.089+0000 7fd146600640 1 -- 172.21.15.67:0/3111379025 shutdown_connections 2024-03-28T16:04:39.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.089+0000 7fd146600640 1 --2- 172.21.15.67:0/3111379025 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd140107260 0x7fd1401096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:39.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.089+0000 7fd146600640 1 --2- 172.21.15.67:0/3111379025 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd140104930 0x7fd140106d20 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:39.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.089+0000 7fd146600640 1 --2- 172.21.15.67:0/3111379025 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd140102000 0x7fd1401043f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:39.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.089+0000 7fd146600640 1 -- 172.21.15.67:0/3111379025 >> 172.21.15.67:0/3111379025 conn(0x7fd1400fba10 msgr2=0x7fd1400fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:04:39.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.089+0000 7fd146600640 1 -- 172.21.15.67:0/3111379025 shutdown_connections 2024-03-28T16:04:39.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.089+0000 7fd146600640 1 -- 172.21.15.67:0/3111379025 wait complete. 2024-03-28T16:04:39.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.090+0000 7fd146600640 1 Processor -- start 2024-03-28T16:04:39.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.090+0000 7fd146600640 1 -- start start 2024-03-28T16:04:39.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.091+0000 7fd146600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd140102000 0x7fd140197800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:39.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.091+0000 7fd146600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd140104930 0x7fd140197d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:39.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.091+0000 7fd146600640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd140107260 0x7fd14019edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:39.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.091+0000 7fd146600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd14010d110 con 0x7fd140104930 2024-03-28T16:04:39.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.091+0000 7fd146600640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd14010cf90 con 0x7fd140107260 2024-03-28T16:04:39.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.091+0000 7fd146600640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd14010d290 con 0x7fd140102000 2024-03-28T16:04:39.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.091+0000 7fd13fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd140104930 0x7fd140197d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:39.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.091+0000 7fd13fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd140104930 0x7fd140197d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58306/0 (socket says 172.21.15.67:58306) 2024-03-28T16:04:39.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.091+0000 7fd13fe00640 1 -- 172.21.15.67:0/781139062 learned_addr learned my addr 172.21.15.67:0/781139062 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:04:39.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.091+0000 7fd145200640 1 --2- 172.21.15.67:0/781139062 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd140102000 0x7fd140197800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:39.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.092+0000 7fd145c00640 1 --2- 172.21.15.67:0/781139062 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd140107260 0x7fd14019edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:39.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.092+0000 7fd13fe00640 1 -- 172.21.15.67:0/781139062 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd140102000 msgr2=0x7fd140197800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:39.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.092+0000 7fd13fe00640 1 --2- 172.21.15.67:0/781139062 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd140102000 0x7fd140197800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:39.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.092+0000 7fd13fe00640 1 -- 172.21.15.67:0/781139062 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd140107260 msgr2=0x7fd14019edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:39.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.092+0000 7fd13fe00640 1 --2- 172.21.15.67:0/781139062 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd140107260 0x7fd14019edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:39.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.092+0000 7fd13fe00640 1 -- 172.21.15.67:0/781139062 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd14019f420 con 0x7fd140104930 2024-03-28T16:04:39.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.093+0000 7fd13fe00640 1 --2- 172.21.15.67:0/781139062 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd140104930 0x7fd140197d40 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fd12c005ab0 tx=0x7fd12c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:04:39.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.093+0000 7fd13d600640 1 -- 172.21.15.67:0/781139062 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd12c043020 con 0x7fd140104930 2024-03-28T16:04:39.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.093+0000 7fd13d600640 1 -- 172.21.15.67:0/781139062 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd12c0054c0 con 0x7fd140104930 2024-03-28T16:04:39.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.093+0000 7fd146600640 1 -- 172.21.15.67:0/781139062 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd14019f6b0 con 0x7fd140104930 2024-03-28T16:04:39.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.093+0000 7fd13d600640 1 -- 172.21.15.67:0/781139062 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd12c003d10 con 0x7fd140104930 2024-03-28T16:04:39.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.093+0000 7fd146600640 1 -- 172.21.15.67:0/781139062 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd14019fc00 con 0x7fd140104930 2024-03-28T16:04:39.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.096+0000 7fd14a7fa740 1 Processor -- start 2024-03-28T16:04:39.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.096+0000 7fd14a7fa740 1 -- start start 2024-03-28T16:04:39.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.097+0000 7fd14a7fa740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5650faff1230 0x5650faff1650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:39.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.097+0000 7fd14a7fa740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5650faff1c20 0x5650faff4710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:39.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.097+0000 7fd14a7fa740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5650faff4d80 0x5650fb001bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:39.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.097+0000 7fd14a7fa740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650faf56eb0 con 0x5650faff1230 2024-03-28T16:04:39.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.097+0000 7fd14a7fa740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650faeffa60 con 0x5650faff4d80 2024-03-28T16:04:39.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.097+0000 7fd14a7fa740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650faf52f50 con 0x5650faff1c20 2024-03-28T16:04:39.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.097+0000 7fd145c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5650faff1230 0x5650faff1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:39.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.097+0000 7fd145c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5650faff1230 0x5650faff1650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58320/0 (socket says 172.21.15.67:58320) 2024-03-28T16:04:39.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.097+0000 7fd145c00640 1 -- 172.21.15.67:0/2851330811 learned_addr learned my addr 172.21.15.67:0/2851330811 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:04:39.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.097+0000 7fd145200640 1 --2- 172.21.15.67:0/2851330811 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5650faff1c20 0x5650faff4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:39.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.098+0000 7fd145c00640 1 -- 172.21.15.67:0/2851330811 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5650faff1c20 msgr2=0x5650faff4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:04:39.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.098+0000 7fd145c00640 1 --2- 172.21.15.67:0/2851330811 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5650faff1c20 0x5650faff4710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:39.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.098+0000 7fd145c00640 1 -- 172.21.15.67:0/2851330811 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5650faff4d80 msgr2=0x5650fb001bd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T16:04:39.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.098+0000 7fd145c00640 1 --2- 172.21.15.67:0/2851330811 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5650faff4d80 0x5650fb001bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:39.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.098+0000 7fd145c00640 1 -- 172.21.15.67:0/2851330811 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5650fad337e0 con 0x5650faff1230 2024-03-28T16:04:39.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.098+0000 7fd145c00640 1 --2- 172.21.15.67:0/2851330811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5650faff1230 0x5650faff1650 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fd138010370 tx=0x7fd138010830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9095230c1c4a6c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:04:39.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.098+0000 7fd13d600640 1 -- 172.21.15.67:0/781139062 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fd12c046020 con 0x7fd140104930 2024-03-28T16:04:39.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.099+0000 7fd132000640 1 -- 172.21.15.67:0/2851330811 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd138016070 con 0x5650faff1230 2024-03-28T16:04:39.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.099+0000 7fd132000640 1 -- 172.21.15.67:0/2851330811 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd138004d20 con 0x5650faff1230 2024-03-28T16:04:39.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.099+0000 7fd132000640 1 -- 172.21.15.67:0/2851330811 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1380053e0 con 0x5650faff1230 2024-03-28T16:04:39.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.100+0000 7fd14a7fa740 1 -- 172.21.15.67:0/2851330811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5650faff1230 msgr2=0x5650faff1650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:39.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.100+0000 7fd14a7fa740 1 --2- 172.21.15.67:0/2851330811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5650faff1230 0x5650faff1650 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fd138010370 tx=0x7fd138010830 comp rx=0 tx=0).stop 2024-03-28T16:04:39.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.100+0000 7fd14a7fa740 1 -- 172.21.15.67:0/2851330811 shutdown_connections 2024-03-28T16:04:39.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.100+0000 7fd14a7fa740 1 --2- 172.21.15.67:0/2851330811 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5650faff4d80 0x5650fb001bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:39.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.100+0000 7fd14a7fa740 1 --2- 172.21.15.67:0/2851330811 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5650faff1c20 0x5650faff4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:39.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.100+0000 7fd14a7fa740 1 --2- 172.21.15.67:0/2851330811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5650faff1230 0x5650faff1650 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:39.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.100+0000 7fd14a7fa740 1 -- 172.21.15.67:0/2851330811 >> 172.21.15.67:0/2851330811 conn(0x5650fafec820 msgr2=0x5650fafecc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:04:39.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.100+0000 7fd14a7fa740 1 -- 172.21.15.67:0/2851330811 shutdown_connections 2024-03-28T16:04:39.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.100+0000 7fd14a7fa740 1 -- 172.21.15.67:0/2851330811 wait complete. 2024-03-28T16:04:39.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.100+0000 7fd13d600640 1 --2- 172.21.15.67:0/781139062 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fd1280c4720 0x7fd1280c6be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:39.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.101+0000 7fd13d600640 1 -- 172.21.15.67:0/781139062 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fd12c0f09e0 con 0x7fd140104930 2024-03-28T16:04:39.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.101+0000 7fd14a7fa740 1 -- start start 2024-03-28T16:04:39.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.101+0000 7fd14a7fa740 1 Processor -- start 2024-03-28T16:04:39.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.101+0000 7fd145200640 1 --2- 172.21.15.67:0/781139062 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fd1280c4720 0x7fd1280c6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:39.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.102+0000 7fd14a7fa740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5650faff1230 0x5650faffb460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:39.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.102+0000 7fd14a7fa740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5650faff1c20 0x5650faffb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:39.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.102+0000 7fd14a7fa740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5650faff4d80 0x5650fb0009d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:39.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.102+0000 7fd14a7fa740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650faff57d0 con 0x5650faff4d80 2024-03-28T16:04:39.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.102+0000 7fd14a7fa740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650faf12eb0 con 0x5650faff1c20 2024-03-28T16:04:39.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.102+0000 7fd14a7fa740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650faff5950 con 0x5650faff1230 2024-03-28T16:04:39.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.102+0000 7fd14a7fa740 10 client.0 _ll_register_callbacks cb 0x5650faf07150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:04:39.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.102+0000 7fd145c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5650faff4d80 0x5650fb0009d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:39.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.102+0000 7fd145c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5650faff4d80 0x5650fb0009d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58328/0 (socket says 172.21.15.67:58328) 2024-03-28T16:04:39.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.102+0000 7fd145c00640 1 -- 172.21.15.67:0/3380627116 learned_addr learned my addr 172.21.15.67:0/3380627116 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:04:39.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.102+0000 7fd13fe00640 1 --2- 172.21.15.67:0/3380627116 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5650faff1c20 0x5650faffb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:39.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.103+0000 7fd145c00640 1 -- 172.21.15.67:0/3380627116 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5650faff1230 msgr2=0x5650faffb460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:04:39.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.103+0000 7fd145c00640 1 --2- 172.21.15.67:0/3380627116 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5650faff1230 0x5650faffb460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:39.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.103+0000 7fd145c00640 1 -- 172.21.15.67:0/3380627116 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5650faff1c20 msgr2=0x5650faffb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:04:39.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.103+0000 7fd145c00640 1 --2- 172.21.15.67:0/3380627116 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5650faff1c20 0x5650faffb9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:39.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.103+0000 7fd145c00640 1 -- 172.21.15.67:0/3380627116 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5650fad93f70 con 0x5650faff4d80 2024-03-28T16:04:39.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.103+0000 7fd145200640 1 --2- 172.21.15.67:0/781139062 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fd1280c4720 0x7fd1280c6be0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x5650fafed790 tx=0x7fd13400bb00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:04:39.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.103+0000 7fd145c00640 1 --2- 172.21.15.67:0/3380627116 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5650faff4d80 0x5650fb0009d0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fd138009bf0 tx=0x7fd138003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=991903a48e1aa3a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:04:39.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.103+0000 7fd126a00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:04:39.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.104+0000 7fd126a00640 1 -- 172.21.15.67:0/3380627116 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd138016070 con 0x5650faff4d80 2024-03-28T16:04:39.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.104+0000 7fd126a00640 1 -- 172.21.15.67:0/3380627116 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fd138012920 con 0x5650faff4d80 2024-03-28T16:04:39.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.104+0000 7fd14a7fa740 1 -- 172.21.15.67:0/3380627116 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd138004d20 con 0x5650faff4d80 2024-03-28T16:04:39.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.104+0000 7fd126a00640 1 -- 172.21.15.67:0/3380627116 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd138004b80 con 0x5650faff4d80 2024-03-28T16:04:39.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.104+0000 7fd14a7fa740 10 client.15459 fetch_fsmap learned FSMap version 19 2024-03-28T16:04:39.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.104+0000 7fd14a7fa740 1 -- 172.21.15.67:0/3380627116 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5650fac2b170 con 0x5650faff4d80 2024-03-28T16:04:39.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.105+0000 7fd126a00640 1 -- 172.21.15.67:0/3380627116 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd138028070 con 0x5650faff4d80 2024-03-28T16:04:39.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.105+0000 7fd126a00640 1 -- 172.21.15.67:0/3380627116 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fd13800f040 con 0x5650faff4d80 2024-03-28T16:04:39.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.105+0000 7fd14a7fa740 10 client.15459 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:04:39.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.105+0000 7fd14a7fa740 10 client.15459 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:04:39.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.105+0000 7fd14a7fa740 20 client.15459 populate_metadata read hostname 'smithi067' 2024-03-28T16:04:39.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.105+0000 7fd14a7fa740 1 --2- 172.21.15.67:0/3380627116 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5650fb007020 0x5650fb009410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:39.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.105+0000 7fd14a7fa740 4 client.15459 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:04:39.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.105+0000 7fd14a7fa740 1 -- 172.21.15.67:0/3380627116 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd138004d20 con 0x5650fb007020 2024-03-28T16:04:39.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.106+0000 7fd145200640 1 --2- 172.21.15.67:0/3380627116 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5650fb007020 0x5650fb009410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:39.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.107+0000 7fd126a00640 10 client.15459 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:04:39.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.107+0000 7fd145200640 1 --2- 172.21.15.67:0/3380627116 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5650fb007020 0x5650fb009410 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=43c16fb49214f3c6 server_cookie=ae5d8c1145b9eff5 in_seq=0 out_seq=0 2024-03-28T16:04:39.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.109+0000 7fd126a00640 1 -- 172.21.15.67:0/3380627116 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x7fd138004d20 con 0x5650fb007020 2024-03-28T16:04:39.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.109+0000 7fd126a00640 10 client.15459 handle_command_reply: tid=0 2024-03-28T16:04:39.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.109+0000 7fd14a7fa740 1 client.15459 shutdown 2024-03-28T16:04:39.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.109+0000 7fd14a7fa740 1 -- 172.21.15.67:0/3380627116 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5650faff4d80 msgr2=0x5650fb0009d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:39.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.109+0000 7fd14a7fa740 1 --2- 172.21.15.67:0/3380627116 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5650faff4d80 0x5650fb0009d0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fd138009bf0 tx=0x7fd138003c20 comp rx=0 tx=0).stop 2024-03-28T16:04:39.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.109+0000 7fd14a7fa740 1 -- 172.21.15.67:0/3380627116 shutdown_connections 2024-03-28T16:04:39.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.109+0000 7fd14a7fa740 1 --2- 172.21.15.67:0/3380627116 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5650fb007020 0x5650fb009410 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:39.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.109+0000 7fd14a7fa740 1 --2- 172.21.15.67:0/3380627116 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5650faff4d80 0x5650fb0009d0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:39.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.109+0000 7fd14a7fa740 1 --2- 172.21.15.67:0/3380627116 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5650faff1c20 0x5650faffb9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:39.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.109+0000 7fd14a7fa740 1 --2- 172.21.15.67:0/3380627116 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5650faff1230 0x5650faffb460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:39.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.109+0000 7fd14a7fa740 1 -- 172.21.15.67:0/3380627116 >> 172.21.15.67:0/3380627116 conn(0x5650fafec820 msgr2=0x5650fafeba80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:04:39.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.109+0000 7fd126a00640 0 client.15459 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:04:39.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.109+0000 7fd14a7fa740 1 -- 172.21.15.67:0/3380627116 shutdown_connections 2024-03-28T16:04:39.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.109+0000 7fd14a7fa740 1 -- 172.21.15.67:0/3380627116 wait complete. 2024-03-28T16:04:39.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.110+0000 7fd14a7fa740 20 client.15459 trim_cache size 0 max 16384 2024-03-28T16:04:39.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.140+0000 7fd14a7fa740 1 Processor -- start 2024-03-28T16:04:39.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.141+0000 7fd14a7fa740 1 -- start start 2024-03-28T16:04:39.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.141+0000 7fd14a7fa740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5650faff1230 0x5650faff5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:39.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.141+0000 7fd14a7fa740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5650faff1c20 0x5650faff6220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:39.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.141+0000 7fd14a7fa740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5650faff4d80 0x5650faff6760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:39.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.141+0000 7fd14a7fa740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1401a4690 con 0x5650faff4d80 2024-03-28T16:04:39.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.141+0000 7fd14a7fa740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1401a6c50 con 0x5650faff1c20 2024-03-28T16:04:39.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.141+0000 7fd14a7fa740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1401a4c90 con 0x5650faff1230 2024-03-28T16:04:39.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.141+0000 7fd145c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5650faff4d80 0x5650faff6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:39.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.141+0000 7fd145c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5650faff4d80 0x5650faff6760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58340/0 (socket says 172.21.15.67:58340) 2024-03-28T16:04:39.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.141+0000 7fd145c00640 1 -- 172.21.15.67:0/2446374937 learned_addr learned my addr 172.21.15.67:0/2446374937 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:04:39.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.142+0000 7fd13fe00640 1 --2- 172.21.15.67:0/2446374937 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5650faff1c20 0x5650faff6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:39.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.142+0000 7fd145200640 1 --2- 172.21.15.67:0/2446374937 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5650faff1230 0x5650faff5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:39.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.142+0000 7fd145c00640 1 -- 172.21.15.67:0/2446374937 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5650faff1230 msgr2=0x5650faff5ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:04:39.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.142+0000 7fd145c00640 1 --2- 172.21.15.67:0/2446374937 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5650faff1230 0x5650faff5ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:39.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.142+0000 7fd145c00640 1 -- 172.21.15.67:0/2446374937 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5650faff1c20 msgr2=0x5650faff6220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:04:39.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.142+0000 7fd145c00640 1 --2- 172.21.15.67:0/2446374937 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5650faff1c20 0x5650faff6220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:39.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.142+0000 7fd145c00640 1 -- 172.21.15.67:0/2446374937 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5650faffa1a0 con 0x5650faff4d80 2024-03-28T16:04:39.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.142+0000 7fd13fe00640 1 --2- 172.21.15.67:0/2446374937 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5650faff1c20 0x5650faff6220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:04:39.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.142+0000 7fd145200640 1 --2- 172.21.15.67:0/2446374937 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5650faff1230 0x5650faff5ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:04:39.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.142+0000 7fd145c00640 1 --2- 172.21.15.67:0/2446374937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5650faff4d80 0x5650faff6760 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fd138006010 tx=0x7fd138010830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d96e6acc5d3e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:04:39.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.143+0000 7fd131600640 1 -- 172.21.15.67:0/2446374937 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd138016070 con 0x5650faff4d80 2024-03-28T16:04:39.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.143+0000 7fd131600640 1 -- 172.21.15.67:0/2446374937 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd138028040 con 0x5650faff4d80 2024-03-28T16:04:39.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.143+0000 7fd131600640 1 -- 172.21.15.67:0/2446374937 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd13802f480 con 0x5650faff4d80 2024-03-28T16:04:39.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.144+0000 7fd14a7fa740 1 -- 172.21.15.67:0/2446374937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5650faff4d80 msgr2=0x5650faff6760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:39.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.144+0000 7fd14a7fa740 1 --2- 172.21.15.67:0/2446374937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5650faff4d80 0x5650faff6760 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fd138006010 tx=0x7fd138010830 comp rx=0 tx=0).stop 2024-03-28T16:04:39.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.144+0000 7fd14a7fa740 1 -- 172.21.15.67:0/2446374937 shutdown_connections 2024-03-28T16:04:39.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.144+0000 7fd14a7fa740 1 --2- 172.21.15.67:0/2446374937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5650faff4d80 0x5650faff6760 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:39.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.144+0000 7fd14a7fa740 1 --2- 172.21.15.67:0/2446374937 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5650faff1c20 0x5650faff6220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:39.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.144+0000 7fd14a7fa740 1 --2- 172.21.15.67:0/2446374937 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5650faff1230 0x5650faff5ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:39.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.144+0000 7fd14a7fa740 1 -- 172.21.15.67:0/2446374937 >> 172.21.15.67:0/2446374937 conn(0x5650fafec820 msgr2=0x5650faff0310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:04:39.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.144+0000 7fd14a7fa740 1 -- 172.21.15.67:0/2446374937 shutdown_connections 2024-03-28T16:04:39.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.144+0000 7fd14a7fa740 1 -- 172.21.15.67:0/2446374937 wait complete. 2024-03-28T16:04:39.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.145+0000 7fd14a7fa740 1 -- start start 2024-03-28T16:04:39.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.145+0000 7fd14a7fa740 1 Processor -- start 2024-03-28T16:04:39.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.145+0000 7fd14a7fa740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5650faff1230 0x5650fb018610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:39.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.145+0000 7fd14a7fa740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5650faff1c20 0x5650fb018b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:39.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.145+0000 7fd14a7fa740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5650faff4d80 0x5650fb061d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:39.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.145+0000 7fd14a7fa740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1401a52d0 con 0x5650faff1c20 2024-03-28T16:04:39.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.145+0000 7fd14a7fa740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1401a65f0 con 0x5650faff4d80 2024-03-28T16:04:39.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.145+0000 7fd14a7fa740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1401a5930 con 0x5650faff1230 2024-03-28T16:04:39.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.146+0000 7fd14a7fa740 10 client.0 _ll_register_callbacks cb 0x7fd138009e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:04:39.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.146+0000 7fd13fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5650faff1c20 0x5650fb018b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:39.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.146+0000 7fd13fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5650faff1c20 0x5650fb018b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58350/0 (socket says 172.21.15.67:58350) 2024-03-28T16:04:39.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.146+0000 7fd13fe00640 1 -- 172.21.15.67:0/4231677666 learned_addr learned my addr 172.21.15.67:0/4231677666 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:04:39.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.146+0000 7fd145200640 1 --2- 172.21.15.67:0/4231677666 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5650faff1230 0x5650fb018610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:39.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.146+0000 7fd145c00640 1 --2- 172.21.15.67:0/4231677666 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5650faff4d80 0x5650fb061d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:39.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.146+0000 7fd13fe00640 1 -- 172.21.15.67:0/4231677666 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5650faff1230 msgr2=0x5650fb018610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:04:39.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.146+0000 7fd13fe00640 1 --2- 172.21.15.67:0/4231677666 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5650faff1230 0x5650fb018610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:39.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.146+0000 7fd13fe00640 1 -- 172.21.15.67:0/4231677666 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5650faff4d80 msgr2=0x5650fb061d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:04:39.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.146+0000 7fd13fe00640 1 --2- 172.21.15.67:0/4231677666 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5650faff4d80 0x5650fb061d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:39.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.146+0000 7fd13fe00640 1 -- 172.21.15.67:0/4231677666 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5650fb062330 con 0x5650faff1c20 2024-03-28T16:04:39.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.146+0000 7fd145c00640 1 --2- 172.21.15.67:0/4231677666 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5650faff4d80 0x5650fb061d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:04:39.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.147+0000 7fd13fe00640 1 --2- 172.21.15.67:0/4231677666 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5650faff1c20 0x5650fb018b50 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fd12c004be0 tx=0x7fd12c005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb54984750641d0c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:04:39.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.147+0000 7fd126a00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:04:39.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.147+0000 7fd126a00640 1 -- 172.21.15.67:0/4231677666 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd12c0bda90 con 0x5650faff1c20 2024-03-28T16:04:39.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.147+0000 7fd14a7fa740 1 -- 172.21.15.67:0/4231677666 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd138028040 con 0x5650faff1c20 2024-03-28T16:04:39.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.147+0000 7fd126a00640 1 -- 172.21.15.67:0/4231677666 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fd12c0d6190 con 0x5650faff1c20 2024-03-28T16:04:39.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.147+0000 7fd126a00640 1 -- 172.21.15.67:0/4231677666 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd12c0bd390 con 0x5650faff1c20 2024-03-28T16:04:39.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.147+0000 7fd14a7fa740 10 client.15465 fetch_fsmap learned FSMap version 19 2024-03-28T16:04:39.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.147+0000 7fd14a7fa740 1 -- 172.21.15.67:0/4231677666 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5650fb0636b0 con 0x5650faff1c20 2024-03-28T16:04:39.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.148+0000 7fd126a00640 1 -- 172.21.15.67:0/4231677666 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd138028040 con 0x5650faff1c20 2024-03-28T16:04:39.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.148+0000 7fd126a00640 1 -- 172.21.15.67:0/4231677666 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fd12c0c56a0 con 0x5650faff1c20 2024-03-28T16:04:39.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.148+0000 7fd14a7fa740 10 client.15465 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:04:39.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.148+0000 7fd14a7fa740 10 client.15465 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:04:39.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.148+0000 7fd14a7fa740 20 client.15465 populate_metadata read hostname 'smithi067' 2024-03-28T16:04:39.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.148+0000 7fd14a7fa740 1 --2- 172.21.15.67:0/4231677666 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5650fb063fc0 0x5650fb0663b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:39.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.148+0000 7fd14a7fa740 4 client.15465 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T16:04:39.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.148+0000 7fd14a7fa740 1 -- 172.21.15.67:0/4231677666 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5650faf643f0 con 0x5650fb063fc0 2024-03-28T16:04:39.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.149+0000 7fd145200640 1 --2- 172.21.15.67:0/4231677666 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5650fb063fc0 0x5650fb0663b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:39.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.150+0000 7fd126a00640 10 client.15465 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:04:39.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.150+0000 7fd145200640 1 --2- 172.21.15.67:0/4231677666 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5650fb063fc0 0x5650fb0663b0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e4ae1d81337146fe server_cookie=6e2d380dfcc3174e in_seq=0 out_seq=0 2024-03-28T16:04:39.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.152+0000 7fd126a00640 1 -- 172.21.15.67:0/4231677666 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5650faf643f0 con 0x5650fb063fc0 2024-03-28T16:04:39.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.152+0000 7fd126a00640 10 client.15465 handle_command_reply: tid=0 2024-03-28T16:04:39.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.152+0000 7fd14a7fa740 1 client.15465 shutdown 2024-03-28T16:04:39.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.152+0000 7fd14a7fa740 1 -- 172.21.15.67:0/4231677666 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5650faff1c20 msgr2=0x5650fb018b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:39.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.152+0000 7fd14a7fa740 1 --2- 172.21.15.67:0/4231677666 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5650faff1c20 0x5650fb018b50 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fd12c004be0 tx=0x7fd12c005fd0 comp rx=0 tx=0).stop 2024-03-28T16:04:39.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.153+0000 7fd14a7fa740 1 -- 172.21.15.67:0/4231677666 shutdown_connections 2024-03-28T16:04:39.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.153+0000 7fd14a7fa740 1 --2- 172.21.15.67:0/4231677666 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5650fb063fc0 0x5650fb0663b0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:39.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.153+0000 7fd14a7fa740 1 --2- 172.21.15.67:0/4231677666 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5650faff4d80 0x5650fb061d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:39.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.153+0000 7fd14a7fa740 1 --2- 172.21.15.67:0/4231677666 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5650faff1c20 0x5650fb018b50 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:39.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.153+0000 7fd14a7fa740 1 --2- 172.21.15.67:0/4231677666 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5650faff1230 0x5650fb018610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:39.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.153+0000 7fd14a7fa740 1 -- 172.21.15.67:0/4231677666 >> 172.21.15.67:0/4231677666 conn(0x5650fafec820 msgr2=0x5650fafeb1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:04:39.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.153+0000 7fd126a00640 0 client.15465 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:04:39.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.153+0000 7fd14a7fa740 1 -- 172.21.15.67:0/4231677666 shutdown_connections 2024-03-28T16:04:39.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.153+0000 7fd14a7fa740 1 -- 172.21.15.67:0/4231677666 wait complete. 2024-03-28T16:04:39.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.153+0000 7fd14a7fa740 20 client.15465 trim_cache size 0 max 16384 2024-03-28T16:04:39.155 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-28T16:04:39.155 INFO:teuthology.orchestra.run.smithi067.stdout: "return_code": 0, 2024-03-28T16:04:39.155 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_tag": "2dc0fdef-8e12-4aef-b9d0-f7b73e3b8ea9", 2024-03-28T16:04:39.155 INFO:teuthology.orchestra.run.smithi067.stdout: "mode": "asynchronous" 2024-03-28T16:04:39.155 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-28T16:04:39.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.154+0000 7fd146600640 1 -- 172.21.15.67:0/781139062 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fd1280c4720 msgr2=0x7fd1280c6be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:39.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.154+0000 7fd146600640 1 --2- 172.21.15.67:0/781139062 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fd1280c4720 0x7fd1280c6be0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x5650fafed790 tx=0x7fd13400bb00 comp rx=0 tx=0).stop 2024-03-28T16:04:39.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.154+0000 7fd146600640 1 -- 172.21.15.67:0/781139062 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd140104930 msgr2=0x7fd140197d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:39.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.154+0000 7fd146600640 1 --2- 172.21.15.67:0/781139062 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd140104930 0x7fd140197d40 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fd12c005ab0 tx=0x7fd12c0042f0 comp rx=0 tx=0).stop 2024-03-28T16:04:39.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.154+0000 7fd146600640 1 -- 172.21.15.67:0/781139062 shutdown_connections 2024-03-28T16:04:39.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.154+0000 7fd146600640 1 --2- 172.21.15.67:0/781139062 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fd1280c4720 0x7fd1280c6be0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:39.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.154+0000 7fd146600640 1 --2- 172.21.15.67:0/781139062 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd140107260 0x7fd14019edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:39.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.154+0000 7fd146600640 1 --2- 172.21.15.67:0/781139062 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd140104930 0x7fd140197d40 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:39.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.154+0000 7fd146600640 1 --2- 172.21.15.67:0/781139062 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd140102000 0x7fd140197800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:39.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.154+0000 7fd146600640 1 -- 172.21.15.67:0/781139062 >> 172.21.15.67:0/781139062 conn(0x7fd1400fba10 msgr2=0x7fd140102a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:04:39.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.155+0000 7fd146600640 1 -- 172.21.15.67:0/781139062 shutdown_connections 2024-03-28T16:04:39.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.155+0000 7fd146600640 1 -- 172.21.15.67:0/781139062 wait complete. 2024-03-28T16:04:39.172 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T16:04:39.172 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T16:04:39.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.354+0000 7fe3a4600640 1 -- 172.21.15.67:0/1467484323 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe39c109540 msgr2=0x7fe39c10b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:39.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.354+0000 7fe3a4600640 1 --2- 172.21.15.67:0/1467484323 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe39c109540 0x7fe39c10b9e0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fe394009ca0 tx=0x7fe39402c180 comp rx=0 tx=0).stop 2024-03-28T16:04:39.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.354+0000 7fe3a4600640 1 -- 172.21.15.67:0/1467484323 shutdown_connections 2024-03-28T16:04:39.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.354+0000 7fe3a4600640 1 --2- 172.21.15.67:0/1467484323 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe39c109540 0x7fe39c10b9e0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:39.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.354+0000 7fe3a4600640 1 --2- 172.21.15.67:0/1467484323 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe39c0740a0 0x7fe39c074500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:39.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.354+0000 7fe3a4600640 1 --2- 172.21.15.67:0/1467484323 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe39c075680 0x7fe39c073ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:39.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.354+0000 7fe3a4600640 1 -- 172.21.15.67:0/1467484323 >> 172.21.15.67:0/1467484323 conn(0x7fe39c0fcc40 msgr2=0x7fe39c0ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:04:39.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.354+0000 7fe3a4600640 1 -- 172.21.15.67:0/1467484323 shutdown_connections 2024-03-28T16:04:39.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.354+0000 7fe3a4600640 1 -- 172.21.15.67:0/1467484323 wait complete. 2024-03-28T16:04:39.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.355+0000 7fe3a4600640 1 Processor -- start 2024-03-28T16:04:39.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.355+0000 7fe3a4600640 1 -- start start 2024-03-28T16:04:39.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.355+0000 7fe3a4600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe39c0740a0 0x7fe39c199a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:39.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.355+0000 7fe3a4600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe39c075680 0x7fe39c199fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:39.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.355+0000 7fe3a4600640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe39c109540 0x7fe39c1a1030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:39.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.355+0000 7fe3a4600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe39c10f3e0 con 0x7fe39c0740a0 2024-03-28T16:04:39.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.355+0000 7fe3a4600640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe39c10f260 con 0x7fe39c109540 2024-03-28T16:04:39.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.355+0000 7fe3a4600640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe39c10f560 con 0x7fe39c075680 2024-03-28T16:04:39.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.356+0000 7fe3a1c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe39c0740a0 0x7fe39c199a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:39.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.356+0000 7fe3a1c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe39c0740a0 0x7fe39c199a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58202/0 (socket says 172.21.15.67:58202) 2024-03-28T16:04:39.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.356+0000 7fe3a1c00640 1 -- 172.21.15.67:0/2497401670 learned_addr learned my addr 172.21.15.67:0/2497401670 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:04:39.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.356+0000 7fe3a2600640 1 --2- 172.21.15.67:0/2497401670 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe39c109540 0x7fe39c1a1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:39.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.356+0000 7fe3a1200640 1 --2- 172.21.15.67:0/2497401670 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe39c075680 0x7fe39c199fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:39.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.356+0000 7fe3a1c00640 1 -- 172.21.15.67:0/2497401670 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe39c075680 msgr2=0x7fe39c199fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:39.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.357+0000 7fe3a1c00640 1 --2- 172.21.15.67:0/2497401670 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe39c075680 0x7fe39c199fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:39.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.357+0000 7fe3a1c00640 1 -- 172.21.15.67:0/2497401670 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe39c109540 msgr2=0x7fe39c1a1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:39.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.357+0000 7fe3a1c00640 1 --2- 172.21.15.67:0/2497401670 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe39c109540 0x7fe39c1a1030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:39.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.357+0000 7fe3a1c00640 1 -- 172.21.15.67:0/2497401670 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe39c1a16a0 con 0x7fe39c0740a0 2024-03-28T16:04:39.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.357+0000 7fe3a1c00640 1 --2- 172.21.15.67:0/2497401670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe39c0740a0 0x7fe39c199a80 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fe39000e9f0 tx=0x7fe39000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:04:39.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.357+0000 7fe39a000640 1 -- 172.21.15.67:0/2497401670 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe39000cd80 con 0x7fe39c0740a0 2024-03-28T16:04:39.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.357+0000 7fe39a000640 1 -- 172.21.15.67:0/2497401670 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe3900054d0 con 0x7fe39c0740a0 2024-03-28T16:04:39.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.357+0000 7fe39a000640 1 -- 172.21.15.67:0/2497401670 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe390010490 con 0x7fe39c0740a0 2024-03-28T16:04:39.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.357+0000 7fe3a4600640 1 -- 172.21.15.67:0/2497401670 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe39c1a1990 con 0x7fe39c0740a0 2024-03-28T16:04:39.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.358+0000 7fe3a4600640 1 -- 172.21.15.67:0/2497401670 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe39c1a1ee0 con 0x7fe39c0740a0 2024-03-28T16:04:39.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.361+0000 7fe3a73a1740 1 Processor -- start 2024-03-28T16:04:39.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.361+0000 7fe3a73a1740 1 -- start start 2024-03-28T16:04:39.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.361+0000 7fe3a73a1740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d2ec34b690 0x55d2ec34bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:39.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.361+0000 7fe3a73a1740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d2ec34c080 0x55d2ec350b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:39.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.361+0000 7fe3a73a1740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d2ec3511f0 0x55d2ec353680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:39.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.361+0000 7fe3a73a1740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d2ec2a8250 con 0x55d2ec34b690 2024-03-28T16:04:39.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.361+0000 7fe3a73a1740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d2ec250a30 con 0x55d2ec3511f0 2024-03-28T16:04:39.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.361+0000 7fe3a73a1740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d2ec2a42f0 con 0x55d2ec34c080 2024-03-28T16:04:39.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.362+0000 7fe3a2600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d2ec34b690 0x55d2ec34bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:39.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.362+0000 7fe3a2600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d2ec34b690 0x55d2ec34bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58216/0 (socket says 172.21.15.67:58216) 2024-03-28T16:04:39.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.362+0000 7fe3a2600640 1 -- 172.21.15.67:0/654915640 learned_addr learned my addr 172.21.15.67:0/654915640 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:04:39.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.362+0000 7fe3a1200640 1 --2- 172.21.15.67:0/654915640 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d2ec3511f0 0x55d2ec353680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:39.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.362+0000 7fe3a2600640 1 -- 172.21.15.67:0/654915640 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d2ec34c080 msgr2=0x55d2ec350b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T16:04:39.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.362+0000 7fe3a2600640 1 --2- 172.21.15.67:0/654915640 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d2ec34c080 0x55d2ec350b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:39.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.362+0000 7fe3a2600640 1 -- 172.21.15.67:0/654915640 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d2ec3511f0 msgr2=0x55d2ec353680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:04:39.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.362+0000 7fe3a2600640 1 --2- 172.21.15.67:0/654915640 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d2ec3511f0 0x55d2ec353680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:39.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.362+0000 7fe3a2600640 1 -- 172.21.15.67:0/654915640 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d2ec23b690 con 0x55d2ec34b690 2024-03-28T16:04:39.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.363+0000 7fe3a2600640 1 --2- 172.21.15.67:0/654915640 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d2ec34b690 0x55d2ec34bab0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fe394009ca0 tx=0x7fe394005f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a42e073082fe9b1e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:04:39.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.363+0000 7fe38ea00640 1 -- 172.21.15.67:0/654915640 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe394043070 con 0x55d2ec34b690 2024-03-28T16:04:39.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.363+0000 7fe39a000640 1 -- 172.21.15.67:0/2497401670 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fe390005000 con 0x7fe39c0740a0 2024-03-28T16:04:39.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.363+0000 7fe38ea00640 1 -- 172.21.15.67:0/654915640 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe394032ba0 con 0x55d2ec34b690 2024-03-28T16:04:39.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.363+0000 7fe38ea00640 1 -- 172.21.15.67:0/654915640 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe394039040 con 0x55d2ec34b690 2024-03-28T16:04:39.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.364+0000 7fe3a73a1740 1 -- 172.21.15.67:0/654915640 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d2ec34b690 msgr2=0x55d2ec34bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:39.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.364+0000 7fe3a73a1740 1 --2- 172.21.15.67:0/654915640 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d2ec34b690 0x55d2ec34bab0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fe394009ca0 tx=0x7fe394005f70 comp rx=0 tx=0).stop 2024-03-28T16:04:39.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.365+0000 7fe3a73a1740 1 -- 172.21.15.67:0/654915640 shutdown_connections 2024-03-28T16:04:39.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.365+0000 7fe3a73a1740 1 --2- 172.21.15.67:0/654915640 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d2ec3511f0 0x55d2ec353680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:39.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.365+0000 7fe3a73a1740 1 --2- 172.21.15.67:0/654915640 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d2ec34c080 0x55d2ec350b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:39.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.365+0000 7fe3a73a1740 1 --2- 172.21.15.67:0/654915640 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d2ec34b690 0x55d2ec34bab0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:39.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.365+0000 7fe3a73a1740 1 -- 172.21.15.67:0/654915640 >> 172.21.15.67:0/654915640 conn(0x55d2ec30b200 msgr2=0x55d2ec33e7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:04:39.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.365+0000 7fe39a000640 1 --2- 172.21.15.67:0/2497401670 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fe3840a2880 0x7fe3840a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:39.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.365+0000 7fe3a73a1740 1 -- 172.21.15.67:0/654915640 shutdown_connections 2024-03-28T16:04:39.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.365+0000 7fe39a000640 1 -- 172.21.15.67:0/2497401670 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fe3900ced00 con 0x7fe39c0740a0 2024-03-28T16:04:39.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.366+0000 7fe3a1200640 1 --2- 172.21.15.67:0/2497401670 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fe3840a2880 0x7fe3840a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:39.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.366+0000 7fe3a73a1740 1 -- 172.21.15.67:0/654915640 wait complete. 2024-03-28T16:04:39.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.367+0000 7fe3a73a1740 1 -- start start 2024-03-28T16:04:39.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.367+0000 7fe3a1200640 1 --2- 172.21.15.67:0/2497401670 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fe3840a2880 0x7fe3840a4d40 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x55d2ec33f740 tx=0x7fe38800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:04:39.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.367+0000 7fe3a73a1740 1 Processor -- start 2024-03-28T16:04:39.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.367+0000 7fe3a73a1740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d2ec34b690 0x55d2ec355660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:39.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.367+0000 7fe3a73a1740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d2ec34c080 0x55d2ec355ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:39.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.367+0000 7fe3a73a1740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d2ec3511f0 0x55d2ec35cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:39.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.367+0000 7fe3a73a1740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d2ec354140 con 0x55d2ec34b690 2024-03-28T16:04:39.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.367+0000 7fe3a73a1740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d2ec263e80 con 0x55d2ec3511f0 2024-03-28T16:04:39.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.367+0000 7fe3a73a1740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d2ec3542c0 con 0x55d2ec34c080 2024-03-28T16:04:39.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.368+0000 7fe3a73a1740 10 client.0 _ll_register_callbacks cb 0x55d2ec276130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:04:39.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.368+0000 7fe3a1c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d2ec34b690 0x55d2ec355660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:39.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.368+0000 7fe3a1c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d2ec34b690 0x55d2ec355660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58232/0 (socket says 172.21.15.67:58232) 2024-03-28T16:04:39.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.368+0000 7fe3a1c00640 1 -- 172.21.15.67:0/2947031526 learned_addr learned my addr 172.21.15.67:0/2947031526 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:04:39.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.368+0000 7fe3a2600640 1 --2- 172.21.15.67:0/2947031526 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d2ec3511f0 0x55d2ec35cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:39.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.368+0000 7fe3a1c00640 1 -- 172.21.15.67:0/2947031526 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d2ec34c080 msgr2=0x55d2ec355ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:04:39.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.368+0000 7fe3a1c00640 1 --2- 172.21.15.67:0/2947031526 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d2ec34c080 0x55d2ec355ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:39.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.368+0000 7fe3a1c00640 1 -- 172.21.15.67:0/2947031526 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d2ec3511f0 msgr2=0x55d2ec35cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:04:39.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.368+0000 7fe3a1c00640 1 --2- 172.21.15.67:0/2947031526 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d2ec3511f0 0x55d2ec35cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:39.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.368+0000 7fe3a1c00640 1 -- 172.21.15.67:0/2947031526 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d2ec0867d0 con 0x55d2ec34b690 2024-03-28T16:04:39.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.369+0000 7fe3a1c00640 1 --2- 172.21.15.67:0/2947031526 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d2ec34b690 0x55d2ec355660 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fe390098020 tx=0x7fe39009b070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2390865e3c8998f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:04:39.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.369+0000 7fe383400640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:04:39.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.369+0000 7fe383400640 1 -- 172.21.15.67:0/2947031526 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3900b4060 con 0x55d2ec34b690 2024-03-28T16:04:39.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.369+0000 7fe383400640 1 -- 172.21.15.67:0/2947031526 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fe39009b8b0 con 0x55d2ec34b690 2024-03-28T16:04:39.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.369+0000 7fe3a73a1740 1 -- 172.21.15.67:0/2947031526 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe394032ba0 con 0x55d2ec34b690 2024-03-28T16:04:39.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.369+0000 7fe383400640 1 -- 172.21.15.67:0/2947031526 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe39009bb10 con 0x55d2ec34b690 2024-03-28T16:04:39.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.369+0000 7fe3a73a1740 10 client.15477 fetch_fsmap learned FSMap version 19 2024-03-28T16:04:39.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.369+0000 7fe3a73a1740 1 -- 172.21.15.67:0/2947031526 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55d2ebfbc3f0 con 0x55d2ec34b690 2024-03-28T16:04:39.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.370+0000 7fe383400640 1 -- 172.21.15.67:0/2947031526 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe394032ba0 con 0x55d2ec34b690 2024-03-28T16:04:39.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.370+0000 7fe383400640 1 -- 172.21.15.67:0/2947031526 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fe3900a3960 con 0x55d2ec34b690 2024-03-28T16:04:39.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.370+0000 7fe3a73a1740 10 client.15477 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:04:39.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.370+0000 7fe3a73a1740 10 client.15477 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:04:39.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.370+0000 7fe3a73a1740 20 client.15477 populate_metadata read hostname 'smithi067' 2024-03-28T16:04:39.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.370+0000 7fe3a73a1740 1 --2- 172.21.15.67:0/2947031526 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55d2ec3614c0 0x55d2ec363900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:39.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.370+0000 7fe3a73a1740 4 client.15477 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:04:39.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.370+0000 7fe3a73a1740 1 -- 172.21.15.67:0/2947031526 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d2ec2b5790 con 0x55d2ec3614c0 2024-03-28T16:04:39.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.371+0000 7fe3a1200640 1 --2- 172.21.15.67:0/2947031526 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55d2ec3614c0 0x55d2ec363900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:39.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.372+0000 7fe3a1200640 1 --2- 172.21.15.67:0/2947031526 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55d2ec3614c0 0x55d2ec363900 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d831840fd4e7cae0 server_cookie=dd0b1f7016511b3f in_seq=0 out_seq=0 2024-03-28T16:04:39.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.372+0000 7fe383400640 10 client.15477 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:04:39.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.374+0000 7fe383400640 1 -- 172.21.15.67:0/2947031526 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55d2ec2b5790 con 0x55d2ec3614c0 2024-03-28T16:04:39.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.374+0000 7fe383400640 10 client.15477 handle_command_reply: tid=0 2024-03-28T16:04:39.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.374+0000 7fe3a73a1740 1 client.15477 shutdown 2024-03-28T16:04:39.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.374+0000 7fe3a73a1740 1 -- 172.21.15.67:0/2947031526 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d2ec34b690 msgr2=0x55d2ec355660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:39.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.374+0000 7fe3a73a1740 1 --2- 172.21.15.67:0/2947031526 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d2ec34b690 0x55d2ec355660 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fe390098020 tx=0x7fe39009b070 comp rx=0 tx=0).stop 2024-03-28T16:04:39.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.374+0000 7fe3a73a1740 1 -- 172.21.15.67:0/2947031526 shutdown_connections 2024-03-28T16:04:39.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.374+0000 7fe3a73a1740 1 --2- 172.21.15.67:0/2947031526 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55d2ec3614c0 0x55d2ec363900 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:39.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.374+0000 7fe3a73a1740 1 --2- 172.21.15.67:0/2947031526 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d2ec3511f0 0x55d2ec35cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:39.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.374+0000 7fe3a73a1740 1 --2- 172.21.15.67:0/2947031526 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d2ec34c080 0x55d2ec355ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:39.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.374+0000 7fe3a73a1740 1 --2- 172.21.15.67:0/2947031526 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d2ec34b690 0x55d2ec355660 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:39.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.374+0000 7fe3a73a1740 1 -- 172.21.15.67:0/2947031526 >> 172.21.15.67:0/2947031526 conn(0x55d2ec30b200 msgr2=0x55d2ec33da90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:04:39.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.374+0000 7fe383400640 0 client.15477 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:04:39.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.374+0000 7fe3a73a1740 1 -- 172.21.15.67:0/2947031526 shutdown_connections 2024-03-28T16:04:39.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.374+0000 7fe3a73a1740 1 -- 172.21.15.67:0/2947031526 wait complete. 2024-03-28T16:04:39.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.375+0000 7fe3a73a1740 20 client.15477 trim_cache size 0 max 16384 2024-03-28T16:04:39.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.409+0000 7fe3a73a1740 1 Processor -- start 2024-03-28T16:04:39.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.409+0000 7fe3a73a1740 1 -- start start 2024-03-28T16:04:39.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.409+0000 7fe3a73a1740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d2ec34b690 0x55d2ec355b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:39.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.409+0000 7fe3a73a1740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d2ec34c080 0x55d2ec35cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:39.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.409+0000 7fe3a73a1740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d2ec3511f0 0x55d2ec34d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:39.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.409+0000 7fe3a73a1740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe39c1a46e0 con 0x55d2ec34c080 2024-03-28T16:04:39.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.409+0000 7fe3a73a1740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe39c1a6c80 con 0x55d2ec3511f0 2024-03-28T16:04:39.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.409+0000 7fe3a73a1740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe39c1a4ce0 con 0x55d2ec34b690 2024-03-28T16:04:39.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.410+0000 7fe3a1200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d2ec34c080 0x55d2ec35cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:39.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.410+0000 7fe3a1200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d2ec34c080 0x55d2ec35cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58234/0 (socket says 172.21.15.67:58234) 2024-03-28T16:04:39.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.410+0000 7fe3a1200640 1 -- 172.21.15.67:0/1454507564 learned_addr learned my addr 172.21.15.67:0/1454507564 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:04:39.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.410+0000 7fe3a1c00640 1 --2- 172.21.15.67:0/1454507564 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d2ec34b690 0x55d2ec355b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:39.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.410+0000 7fe3a2600640 1 --2- 172.21.15.67:0/1454507564 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d2ec3511f0 0x55d2ec34d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:39.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.410+0000 7fe3a1200640 1 -- 172.21.15.67:0/1454507564 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d2ec34b690 msgr2=0x55d2ec355b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:04:39.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.410+0000 7fe3a1200640 1 --2- 172.21.15.67:0/1454507564 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d2ec34b690 0x55d2ec355b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:39.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.410+0000 7fe3a1200640 1 -- 172.21.15.67:0/1454507564 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d2ec3511f0 msgr2=0x55d2ec34d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:04:39.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.410+0000 7fe3a1200640 1 --2- 172.21.15.67:0/1454507564 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d2ec3511f0 0x55d2ec34d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:39.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.411+0000 7fe3a1200640 1 -- 172.21.15.67:0/1454507564 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d2ebf7c0f0 con 0x55d2ec34c080 2024-03-28T16:04:39.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.411+0000 7fe3a1c00640 1 --2- 172.21.15.67:0/1454507564 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d2ec34b690 0x55d2ec355b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T16:04:39.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.411+0000 7fe3a1200640 1 --2- 172.21.15.67:0/1454507564 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d2ec34c080 0x55d2ec35cb70 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fe388005fd0 tx=0x7fe388015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f77c1dcb8d238f51 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:04:39.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.411+0000 7fe38e000640 1 -- 172.21.15.67:0/1454507564 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe388027d90 con 0x55d2ec34c080 2024-03-28T16:04:39.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.412+0000 7fe38e000640 1 -- 172.21.15.67:0/1454507564 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe3880035f0 con 0x55d2ec34c080 2024-03-28T16:04:39.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.412+0000 7fe38e000640 1 -- 172.21.15.67:0/1454507564 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe388010770 con 0x55d2ec34c080 2024-03-28T16:04:39.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.413+0000 7fe3a73a1740 1 -- 172.21.15.67:0/1454507564 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d2ec34c080 msgr2=0x55d2ec35cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:39.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.413+0000 7fe3a73a1740 1 --2- 172.21.15.67:0/1454507564 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d2ec34c080 0x55d2ec35cb70 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fe388005fd0 tx=0x7fe388015000 comp rx=0 tx=0).stop 2024-03-28T16:04:39.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.413+0000 7fe3a73a1740 1 -- 172.21.15.67:0/1454507564 shutdown_connections 2024-03-28T16:04:39.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.413+0000 7fe3a73a1740 1 --2- 172.21.15.67:0/1454507564 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d2ec3511f0 0x55d2ec34d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:39.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.413+0000 7fe3a73a1740 1 --2- 172.21.15.67:0/1454507564 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d2ec34c080 0x55d2ec35cb70 secure :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fe388005fd0 tx=0x7fe388015000 comp rx=0 tx=0).stop 2024-03-28T16:04:39.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.413+0000 7fe3a73a1740 1 --2- 172.21.15.67:0/1454507564 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d2ec34b690 0x55d2ec355b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:39.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.413+0000 7fe3a73a1740 1 -- 172.21.15.67:0/1454507564 >> 172.21.15.67:0/1454507564 conn(0x55d2ec30b200 msgr2=0x55d2ec34caf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:04:39.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.413+0000 7fe3a73a1740 1 -- 172.21.15.67:0/1454507564 shutdown_connections 2024-03-28T16:04:39.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.413+0000 7fe3a73a1740 1 -- 172.21.15.67:0/1454507564 wait complete. 2024-03-28T16:04:39.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.414+0000 7fe3a73a1740 1 -- start start 2024-03-28T16:04:39.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.414+0000 7fe3a73a1740 1 Processor -- start 2024-03-28T16:04:39.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.414+0000 7fe3a73a1740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d2ec34b690 0x55d2ec35a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:39.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.414+0000 7fe3a73a1740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d2ec3511f0 0x55d2ec35a780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:39.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.414+0000 7fe3a73a1740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d2ec35b790 0x55d2ec35bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:39.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.414+0000 7fe3a73a1740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe39c1a5300 con 0x55d2ec34b690 2024-03-28T16:04:39.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.414+0000 7fe3a73a1740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe39c1a6620 con 0x55d2ec35b790 2024-03-28T16:04:39.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.414+0000 7fe3a73a1740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe39c1a5960 con 0x55d2ec3511f0 2024-03-28T16:04:39.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.414+0000 7fe3a73a1740 10 client.0 _ll_register_callbacks cb 0x7fe39000cf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:04:39.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.415+0000 7fe3a1c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d2ec34b690 0x55d2ec35a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:39.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.415+0000 7fe3a1c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d2ec34b690 0x55d2ec35a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58238/0 (socket says 172.21.15.67:58238) 2024-03-28T16:04:39.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.415+0000 7fe3a1c00640 1 -- 172.21.15.67:0/568045830 learned_addr learned my addr 172.21.15.67:0/568045830 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:04:39.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.415+0000 7fe3a2600640 1 --2- 172.21.15.67:0/568045830 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d2ec35b790 0x55d2ec35bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:39.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.415+0000 7fe3a1c00640 1 -- 172.21.15.67:0/568045830 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d2ec3511f0 msgr2=0x55d2ec35a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:04:39.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.415+0000 7fe3a1c00640 1 --2- 172.21.15.67:0/568045830 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d2ec3511f0 0x55d2ec35a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:39.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.415+0000 7fe3a1c00640 1 -- 172.21.15.67:0/568045830 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d2ec35b790 msgr2=0x55d2ec35bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:04:39.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.415+0000 7fe3a1c00640 1 --2- 172.21.15.67:0/568045830 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d2ec35b790 0x55d2ec35bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:39.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.415+0000 7fe3a1c00640 1 -- 172.21.15.67:0/568045830 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d2ec35c150 con 0x55d2ec34b690 2024-03-28T16:04:39.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.415+0000 7fe3a2600640 1 --2- 172.21.15.67:0/568045830 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d2ec35b790 0x55d2ec35bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:04:39.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.415+0000 7fe3a1c00640 1 --2- 172.21.15.67:0/568045830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d2ec34b690 0x55d2ec35a240 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fe39000c6a0 tx=0x7fe390086180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65d447a290068d2b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:04:39.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.415+0000 7fe383400640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:04:39.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.416+0000 7fe383400640 1 -- 172.21.15.67:0/568045830 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe39009b190 con 0x55d2ec34b690 2024-03-28T16:04:39.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.416+0000 7fe383400640 1 -- 172.21.15.67:0/568045830 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fe3900973a0 con 0x55d2ec34b690 2024-03-28T16:04:39.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.416+0000 7fe3a73a1740 1 -- 172.21.15.67:0/568045830 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe3880035f0 con 0x55d2ec34b690 2024-03-28T16:04:39.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.416+0000 7fe383400640 1 -- 172.21.15.67:0/568045830 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe39009b420 con 0x55d2ec34b690 2024-03-28T16:04:39.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.416+0000 7fe3a73a1740 10 client.15483 fetch_fsmap learned FSMap version 19 2024-03-28T16:04:39.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.416+0000 7fe3a73a1740 1 -- 172.21.15.67:0/568045830 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55d2ec3bd160 con 0x55d2ec34b690 2024-03-28T16:04:39.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.417+0000 7fe383400640 1 -- 172.21.15.67:0/568045830 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe3880035f0 con 0x55d2ec34b690 2024-03-28T16:04:39.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.417+0000 7fe383400640 1 -- 172.21.15.67:0/568045830 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fe3900975a0 con 0x55d2ec34b690 2024-03-28T16:04:39.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.417+0000 7fe3a73a1740 10 client.15483 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:04:39.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.417+0000 7fe3a73a1740 10 client.15483 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:04:39.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.417+0000 7fe3a73a1740 20 client.15483 populate_metadata read hostname 'smithi067' 2024-03-28T16:04:39.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.417+0000 7fe3a73a1740 1 --2- 172.21.15.67:0/568045830 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55d2ec3bda30 0x55d2ec3bfe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:04:39.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.417+0000 7fe3a73a1740 4 client.15483 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T16:04:39.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.417+0000 7fe3a73a1740 1 -- 172.21.15.67:0/568045830 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55d2ec2afae0 con 0x55d2ec3bda30 2024-03-28T16:04:39.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.418+0000 7fe3a1200640 1 --2- 172.21.15.67:0/568045830 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55d2ec3bda30 0x55d2ec3bfe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:04:39.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.419+0000 7fe3a1200640 1 --2- 172.21.15.67:0/568045830 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55d2ec3bda30 0x55d2ec3bfe50 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=23329b4b1218ec15 server_cookie=8b94a492e9312636 in_seq=0 out_seq=0 2024-03-28T16:04:39.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.419+0000 7fe383400640 10 client.15483 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:04:39.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.419+0000 7fe383400640 1 -- 172.21.15.67:0/568045830 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55d2ec2afae0 con 0x55d2ec3bda30 2024-03-28T16:04:39.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.419+0000 7fe383400640 10 client.15483 handle_command_reply: tid=0 2024-03-28T16:04:39.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.419+0000 7fe3a73a1740 1 client.15483 shutdown 2024-03-28T16:04:39.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.419+0000 7fe3a73a1740 1 -- 172.21.15.67:0/568045830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d2ec34b690 msgr2=0x55d2ec35a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:39.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.419+0000 7fe3a73a1740 1 --2- 172.21.15.67:0/568045830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d2ec34b690 0x55d2ec35a240 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fe39000c6a0 tx=0x7fe390086180 comp rx=0 tx=0).stop 2024-03-28T16:04:39.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.419+0000 7fe3a73a1740 1 -- 172.21.15.67:0/568045830 shutdown_connections 2024-03-28T16:04:39.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.419+0000 7fe3a73a1740 1 --2- 172.21.15.67:0/568045830 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55d2ec3bda30 0x55d2ec3bfe50 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:39.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.419+0000 7fe3a73a1740 1 --2- 172.21.15.67:0/568045830 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d2ec35b790 0x55d2ec35bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:39.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.419+0000 7fe3a73a1740 1 --2- 172.21.15.67:0/568045830 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d2ec3511f0 0x55d2ec35a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:39.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.419+0000 7fe3a73a1740 1 --2- 172.21.15.67:0/568045830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d2ec34b690 0x55d2ec35a240 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:39.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.419+0000 7fe3a73a1740 1 -- 172.21.15.67:0/568045830 >> 172.21.15.67:0/568045830 conn(0x55d2ec30b200 msgr2=0x55d2ec34d380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:04:39.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.420+0000 7fe383400640 0 client.15483 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:04:39.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.420+0000 7fe3a73a1740 1 -- 172.21.15.67:0/568045830 shutdown_connections 2024-03-28T16:04:39.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.420+0000 7fe3a73a1740 1 -- 172.21.15.67:0/568045830 wait complete. 2024-03-28T16:04:39.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.420+0000 7fe3a73a1740 20 client.15483 trim_cache size 0 max 16384 2024-03-28T16:04:39.436 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-28T16:04:39.436 INFO:teuthology.orchestra.run.smithi067.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T16:04:39.437 INFO:teuthology.orchestra.run.smithi067.stdout: "scrubs": { 2024-03-28T16:04:39.437 INFO:teuthology.orchestra.run.smithi067.stdout: "2dc0fdef-8e12-4aef-b9d0-f7b73e3b8ea9": { 2024-03-28T16:04:39.437 INFO:teuthology.orchestra.run.smithi067.stdout: "path": "/", 2024-03-28T16:04:39.437 INFO:teuthology.orchestra.run.smithi067.stdout: "tag": "2dc0fdef-8e12-4aef-b9d0-f7b73e3b8ea9", 2024-03-28T16:04:39.437 INFO:teuthology.orchestra.run.smithi067.stdout: "options": "recursive,force" 2024-03-28T16:04:39.437 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:04:39.437 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:04:39.437 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-28T16:04:39.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.421+0000 7fe3a4600640 1 -- 172.21.15.67:0/2497401670 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fe3840a2880 msgr2=0x7fe3840a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:39.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.421+0000 7fe3a4600640 1 --2- 172.21.15.67:0/2497401670 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fe3840a2880 0x7fe3840a4d40 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x55d2ec33f740 tx=0x7fe38800b410 comp rx=0 tx=0).stop 2024-03-28T16:04:39.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.421+0000 7fe3a4600640 1 -- 172.21.15.67:0/2497401670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe39c0740a0 msgr2=0x7fe39c199a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:04:39.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.421+0000 7fe3a4600640 1 --2- 172.21.15.67:0/2497401670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe39c0740a0 0x7fe39c199a80 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fe39000e9f0 tx=0x7fe39000edf0 comp rx=0 tx=0).stop 2024-03-28T16:04:39.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.421+0000 7fe3a4600640 1 -- 172.21.15.67:0/2497401670 shutdown_connections 2024-03-28T16:04:39.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.421+0000 7fe3a4600640 1 --2- 172.21.15.67:0/2497401670 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fe3840a2880 0x7fe3840a4d40 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:39.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.421+0000 7fe3a4600640 1 --2- 172.21.15.67:0/2497401670 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe39c109540 0x7fe39c1a1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:39.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.421+0000 7fe3a4600640 1 --2- 172.21.15.67:0/2497401670 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe39c075680 0x7fe39c199fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:39.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.421+0000 7fe3a4600640 1 --2- 172.21.15.67:0/2497401670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe39c0740a0 0x7fe39c199a80 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:04:39.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.421+0000 7fe3a4600640 1 -- 172.21.15.67:0/2497401670 >> 172.21.15.67:0/2497401670 conn(0x7fe39c0fcc40 msgr2=0x7fe39c1054d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:04:39.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.421+0000 7fe3a4600640 1 -- 172.21.15.67:0/2497401670 shutdown_connections 2024-03-28T16:04:39.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:04:39.421+0000 7fe3a4600640 1 -- 172.21.15.67:0/2497401670 wait complete. 2024-03-28T16:04:39.444 INFO:tasks.cephfs.filesystem:scrub status for tag:2dc0fdef-8e12-4aef-b9d0-f7b73e3b8ea9 - {'path': '/', 'tag': '2dc0fdef-8e12-4aef-b9d0-f7b73e3b8ea9', 'options': 'recursive,force'} 2024-03-28T16:04:40.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:04:40 smithi150 ceph-mon[34825]: pgmap v472: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T16:04:40.553 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:04:40 smithi150 ceph-mon[34825]: scrub queued for path: / 2024-03-28T16:04:40.553 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:04:40 smithi150 ceph-mon[34825]: scrub summary: idle+waiting paths [/] 2024-03-28T16:04:40.553 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:04:40 smithi150 ceph-mon[34825]: scrub summary: active paths [/] 2024-03-28T16:04:40.553 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:04:40 smithi150 ceph-mon[34825]: scrub summary: idle+waiting paths [/] 2024-03-28T16:04:40.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:04:40 smithi067 ceph-mon[31707]: pgmap v472: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T16:04:40.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:04:40 smithi067 ceph-mon[31707]: scrub queued for path: / 2024-03-28T16:04:40.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:04:40 smithi067 ceph-mon[31707]: scrub summary: idle+waiting paths [/] 2024-03-28T16:04:40.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:04:40 smithi067 ceph-mon[31707]: scrub summary: active paths [/] 2024-03-28T16:04:40.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:04:40 smithi067 ceph-mon[31707]: scrub summary: idle+waiting paths [/] 2024-03-28T16:04:40.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:04:40 smithi183 ceph-mon[35580]: pgmap v472: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T16:04:40.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:04:40 smithi183 ceph-mon[35580]: scrub queued for path: / 2024-03-28T16:04:40.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:04:40 smithi183 ceph-mon[35580]: scrub summary: idle+waiting paths [/] 2024-03-28T16:04:40.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:04:40 smithi183 ceph-mon[35580]: scrub summary: active paths [/] 2024-03-28T16:04:40.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:04:40 smithi183 ceph-mon[35580]: scrub summary: idle+waiting paths [/] 2024-03-28T16:04:42.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:04:42 smithi150 ceph-mon[34825]: pgmap v473: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 2 op/s 2024-03-28T16:04:42.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:04:42 smithi150 ceph-mon[34825]: scrub summary: idle 2024-03-28T16:04:42.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:04:42 smithi067 ceph-mon[31707]: pgmap v473: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 2 op/s 2024-03-28T16:04:42.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:04:42 smithi067 ceph-mon[31707]: scrub summary: idle 2024-03-28T16:04:42.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:04:42 smithi183 ceph-mon[35580]: pgmap v473: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 2 op/s 2024-03-28T16:04:42.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:04:42 smithi183 ceph-mon[35580]: scrub summary: idle 2024-03-28T16:04:44.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:04:44 smithi150 ceph-mon[34825]: pgmap v474: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-03-28T16:04:44.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:04:44 smithi067 ceph-mon[31707]: pgmap v474: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-03-28T16:04:44.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:04:44 smithi183 ceph-mon[35580]: pgmap v474: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-03-28T16:04:46.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:04:46 smithi150 ceph-mon[34825]: pgmap v475: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 2 op/s 2024-03-28T16:04:46.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:04:46 smithi067 ceph-mon[31707]: pgmap v475: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 2 op/s 2024-03-28T16:04:46.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:04:46 smithi183 ceph-mon[35580]: pgmap v475: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 2 op/s 2024-03-28T16:04:48.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:04:48 smithi150 ceph-mon[34825]: pgmap v476: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-03-28T16:04:48.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:04:48 smithi067 ceph-mon[31707]: pgmap v476: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-03-28T16:04:48.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:04:48 smithi183 ceph-mon[35580]: pgmap v476: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-03-28T16:04:50.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:04:50 smithi150 ceph-mon[34825]: pgmap v477: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-03-28T16:04:50.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:04:50 smithi067 ceph-mon[31707]: pgmap v477: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-03-28T16:04:50.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:04:50 smithi183 ceph-mon[35580]: pgmap v477: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-03-28T16:04:52.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:04:52 smithi150 ceph-mon[34825]: pgmap v478: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 2 op/s 2024-03-28T16:04:52.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:04:52 smithi067 ceph-mon[31707]: pgmap v478: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 2 op/s 2024-03-28T16:04:52.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:04:52 smithi183 ceph-mon[35580]: pgmap v478: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 2 op/s 2024-03-28T16:04:54.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:04:54 smithi150 ceph-mon[34825]: pgmap v479: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1 op/s 2024-03-28T16:04:54.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:04:54 smithi067 ceph-mon[31707]: pgmap v479: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1 op/s 2024-03-28T16:04:54.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:04:54 smithi183 ceph-mon[35580]: pgmap v479: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1 op/s 2024-03-28T16:04:56.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:04:56 smithi150 ceph-mon[34825]: pgmap v480: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:04:56.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:04:56 smithi067 ceph-mon[31707]: pgmap v480: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:04:56.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:04:56 smithi183 ceph-mon[35580]: pgmap v480: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:04:58.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:04:58 smithi150 ceph-mon[34825]: pgmap v481: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T16:04:58.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:04:58 smithi067 ceph-mon[31707]: pgmap v481: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T16:04:58.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:04:58 smithi183 ceph-mon[35580]: pgmap v481: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T16:04:59.544 INFO:tasks.workunit.client.0.smithi067.stderr:Updating files: 59% (7334/12271) Updating files: 60% (7363/12271) Updating files: 61% (7486/12271) Updating files: 62% (7609/12271) Updating files: 63% (7731/12271) Updating files: 64% (7854/12271) Updating files: 65% (7977/12271) Updating files: 66% (8099/12271) Updating files: 67% (8222/12271) Updating files: 68% (8345/12271) Updating files: 69% (8467/12271) Updating files: 70% (8590/12271) Updating files: 71% (8713/12271) Updating files: 72% (8836/12271) Updating files: 73% (8958/12271) Updating files: 74% (9081/12271) Updating files: 75% (9204/12271) Updating files: 76% (9326/12271) Updating files: 77% (9449/12271) Updating files: 78% (9572/12271) Updating files: 79% (9695/12271) Updating files: 80% (9817/12271) Updating files: 81% (9940/12271) Updating files: 82% (10063/12271) Updating files: 83% (10185/12271) Updating files: 84% (10308/12271) Updating files: 85% (10431/12271) Updating files: 86% (10554/12271) Updating files: 87% (10676/12271) Updating files: 88% (10799/12271) Updating files: 89% (10922/12271) Updating files: 90% (11044/12271) Updating files: 91% (11167/12271) Updating files: 92% (11290/12271) Updating files: 93% (11413/12271) Updating files: 94% (11535/12271) Updating files: 95% (11658/12271) Updating files: 96% (11781/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 99% (12225/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-03-28T16:05:00.173 INFO:tasks.workunit.client.0.smithi067.stderr:Note: switching to '3be9a5c9bc793e11e2800a8c0c696e8b46742033'. 2024-03-28T16:05:00.173 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-03-28T16:05:00.174 INFO:tasks.workunit.client.0.smithi067.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-28T16:05:00.174 INFO:tasks.workunit.client.0.smithi067.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-28T16:05:00.174 INFO:tasks.workunit.client.0.smithi067.stderr:state without impacting any branches by switching back to a branch. 2024-03-28T16:05:00.174 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-03-28T16:05:00.174 INFO:tasks.workunit.client.0.smithi067.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-28T16:05:00.174 INFO:tasks.workunit.client.0.smithi067.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-28T16:05:00.174 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-03-28T16:05:00.174 INFO:tasks.workunit.client.0.smithi067.stderr: git switch -c 2024-03-28T16:05:00.175 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-03-28T16:05:00.175 INFO:tasks.workunit.client.0.smithi067.stderr:Or undo this operation with: 2024-03-28T16:05:00.175 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-03-28T16:05:00.175 INFO:tasks.workunit.client.0.smithi067.stderr: git switch - 2024-03-28T16:05:00.175 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-03-28T16:05:00.175 INFO:tasks.workunit.client.0.smithi067.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-28T16:05:00.175 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-03-28T16:05:00.176 INFO:tasks.workunit.client.0.smithi067.stderr:HEAD is now at 3be9a5c9bc7 dummy commit #1 2024-03-28T16:05:00.184 DEBUG:teuthology.orchestra.run.smithi067:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-28T16:05:00.250 INFO:tasks.workunit.client.0.smithi067.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-28T16:05:00.258 INFO:tasks.workunit.client.0.smithi067.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-28T16:05:00.258 INFO:tasks.workunit.client.0.smithi067.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-28T16:05:00.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:05:00 smithi150 ceph-mon[34825]: pgmap v482: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T16:05:00.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:05:00 smithi067 ceph-mon[31707]: pgmap v482: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T16:05:00.602 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:05:00 smithi183 ceph-mon[35580]: pgmap v482: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T16:05:01.186 INFO:tasks.workunit.client.0.smithi067.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-28T16:05:01.309 INFO:tasks.workunit.client.0.smithi067.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-28T16:05:01.398 INFO:tasks.workunit.client.0.smithi067.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-28T16:05:01.403 INFO:tasks.workunit.client.0.smithi067.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-28T16:05:01.403 INFO:tasks.workunit.client.0.smithi067.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-28T16:05:01.483 INFO:tasks.workunit.client.0.smithi067.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-28T16:05:01.494 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-28T16:05:01.494 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-28T16:05:01.545 INFO:tasks.workunit:Running workunits matching suites/iozone.sh on client.0... 2024-03-28T16:05:01.546 INFO:tasks.workunit:Running workunit suites/iozone.sh... 2024-03-28T16:05:01.546 DEBUG:teuthology.orchestra.run.smithi067:workunit test suites/iozone.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3be9a5c9bc793e11e2800a8c0c696e8b46742033 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/iozone.sh 2024-03-28T16:05:01.644 INFO:tasks.workunit.client.0.smithi067.stderr:+ iozone -c -e -s 1024M -r 16K -t 1 -F f1 -i 0 -i 1 2024-03-28T16:05:01.754 INFO:tasks.workunit.client.0.smithi067.stdout: Iozone: Performance Test of File I/O 2024-03-28T16:05:01.754 INFO:tasks.workunit.client.0.smithi067.stdout: Version $Revision: 3.494 $ 2024-03-28T16:05:01.754 INFO:tasks.workunit.client.0.smithi067.stdout: Compiled for 64 bit mode. 2024-03-28T16:05:01.754 INFO:tasks.workunit.client.0.smithi067.stdout: Build: linux-AMD64 2024-03-28T16:05:01.754 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-03-28T16:05:01.754 INFO:tasks.workunit.client.0.smithi067.stdout: Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins 2024-03-28T16:05:01.754 INFO:tasks.workunit.client.0.smithi067.stdout: Al Slater, Scott Rhine, Mike Wisner, Ken Goss 2024-03-28T16:05:01.754 INFO:tasks.workunit.client.0.smithi067.stdout: Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, 2024-03-28T16:05:01.754 INFO:tasks.workunit.client.0.smithi067.stdout: Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, 2024-03-28T16:05:01.754 INFO:tasks.workunit.client.0.smithi067.stdout: Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone, 2024-03-28T16:05:01.754 INFO:tasks.workunit.client.0.smithi067.stdout: Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root, 2024-03-28T16:05:01.754 INFO:tasks.workunit.client.0.smithi067.stdout: Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer, 2024-03-28T16:05:01.755 INFO:tasks.workunit.client.0.smithi067.stdout: Vangel Bojaxhi, Ben England, Vikentsi Lapa, 2024-03-28T16:05:01.755 INFO:tasks.workunit.client.0.smithi067.stdout: Alexey Skidanov, Sudhir Kumar. 2024-03-28T16:05:01.755 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-03-28T16:05:01.755 INFO:tasks.workunit.client.0.smithi067.stdout: Run began: Thu Mar 28 16:05:01 2024 2024-03-28T16:05:01.755 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-03-28T16:05:01.755 INFO:tasks.workunit.client.0.smithi067.stdout: Include close in write timing 2024-03-28T16:05:01.755 INFO:tasks.workunit.client.0.smithi067.stdout: Include fsync in write timing 2024-03-28T16:05:01.755 INFO:tasks.workunit.client.0.smithi067.stdout: File size set to 1048576 kB 2024-03-28T16:05:01.755 INFO:tasks.workunit.client.0.smithi067.stdout: Record Size 16 kB 2024-03-28T16:05:01.755 INFO:tasks.workunit.client.0.smithi067.stdout: Command line used: iozone -c -e -s 1024M -r 16K -t 1 -F f1 -i 0 -i 1 2024-03-28T16:05:01.755 INFO:tasks.workunit.client.0.smithi067.stdout: Output is in kBytes/sec 2024-03-28T16:05:01.755 INFO:tasks.workunit.client.0.smithi067.stdout: Time Resolution = 0.000001 seconds. 2024-03-28T16:05:01.755 INFO:tasks.workunit.client.0.smithi067.stdout: Processor cache size set to 1024 kBytes. 2024-03-28T16:05:01.755 INFO:tasks.workunit.client.0.smithi067.stdout: Processor cache line size set to 32 bytes. 2024-03-28T16:05:01.755 INFO:tasks.workunit.client.0.smithi067.stdout: File stride size set to 17 * record size. 2024-03-28T16:05:01.756 INFO:tasks.workunit.client.0.smithi067.stdout: Throughput test with 1 process 2024-03-28T16:05:01.756 INFO:tasks.workunit.client.0.smithi067.stdout: Each process writes a 1048576 kByte file in 16 kByte records 2024-03-28T16:05:02.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:05:02 smithi150 ceph-mon[34825]: pgmap v483: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:05:02.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:05:02 smithi067 ceph-mon[31707]: pgmap v483: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:05:02.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:05:02 smithi183 ceph-mon[35580]: pgmap v483: 97 pgs: 97 active+clean; 653 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:05:04.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:05:04 smithi150 ceph-mon[34825]: pgmap v484: 97 pgs: 97 active+clean; 8.6 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 682 KiB/s wr, 1 op/s 2024-03-28T16:05:04.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:05:04 smithi067 ceph-mon[31707]: pgmap v484: 97 pgs: 97 active+clean; 8.6 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 682 KiB/s wr, 1 op/s 2024-03-28T16:05:04.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:05:04 smithi183 ceph-mon[35580]: pgmap v484: 97 pgs: 97 active+clean; 8.6 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 682 KiB/s wr, 1 op/s 2024-03-28T16:05:06.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:05:06 smithi067 ceph-mon[31707]: pgmap v485: 97 pgs: 97 active+clean; 29 MiB data, 423 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 2.3 MiB/s wr, 4 op/s 2024-03-28T16:05:06.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:05:06 smithi183 ceph-mon[35580]: pgmap v485: 97 pgs: 97 active+clean; 29 MiB data, 423 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 2.3 MiB/s wr, 4 op/s 2024-03-28T16:05:06.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:05:06 smithi150 ceph-mon[34825]: pgmap v485: 97 pgs: 97 active+clean; 29 MiB data, 423 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 2.3 MiB/s wr, 4 op/s 2024-03-28T16:05:08.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:05:08 smithi067 ceph-mon[31707]: pgmap v486: 97 pgs: 97 active+clean; 61 MiB data, 487 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5.0 MiB/s wr, 3 op/s 2024-03-28T16:05:08.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:05:08 smithi183 ceph-mon[35580]: pgmap v486: 97 pgs: 97 active+clean; 61 MiB data, 487 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5.0 MiB/s wr, 3 op/s 2024-03-28T16:05:08.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:05:08 smithi150 ceph-mon[34825]: pgmap v486: 97 pgs: 97 active+clean; 61 MiB data, 487 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5.0 MiB/s wr, 3 op/s 2024-03-28T16:05:09.445 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T16:05:09.446 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T16:05:09.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.702+0000 7f92e9800640 1 -- 172.21.15.67:0/3958063481 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92e4105110 msgr2=0x7f92e4105590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:09.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.702+0000 7f92e9800640 1 --2- 172.21.15.67:0/3958063481 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92e4105110 0x7f92e4105590 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f92dc009ca0 tx=0x7f92dc02c120 comp rx=0 tx=0).stop 2024-03-28T16:05:09.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.703+0000 7f92e9800640 1 -- 172.21.15.67:0/3958063481 shutdown_connections 2024-03-28T16:05:09.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.703+0000 7f92e9800640 1 --2- 172.21.15.67:0/3958063481 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92e4105110 0x7f92e4105590 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:09.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.703+0000 7f92e9800640 1 --2- 172.21.15.67:0/3958063481 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f92e4102760 0x7f92e4104bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:09.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.703+0000 7f92e9800640 1 --2- 172.21.15.67:0/3958063481 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92e40ffdf0 0x7f92e4102220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:09.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.703+0000 7f92e9800640 1 -- 172.21.15.67:0/3958063481 >> 172.21.15.67:0/3958063481 conn(0x7f92e40fba10 msgr2=0x7f92e40fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:05:09.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.703+0000 7f92e9800640 1 -- 172.21.15.67:0/3958063481 shutdown_connections 2024-03-28T16:05:09.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.703+0000 7f92e9800640 1 -- 172.21.15.67:0/3958063481 wait complete. 2024-03-28T16:05:09.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.704+0000 7f92e9800640 1 Processor -- start 2024-03-28T16:05:09.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.704+0000 7f92e9800640 1 -- start start 2024-03-28T16:05:09.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.705+0000 7f92e9800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f92e40ffdf0 0x7f92e419ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:09.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.705+0000 7f92e9800640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92e4102760 0x7f92e419e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:09.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.705+0000 7f92e9800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92e4105110 0x7f92e41a5330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:09.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.705+0000 7f92e9800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92e4113720 con 0x7f92e4105110 2024-03-28T16:05:09.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.705+0000 7f92e9800640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92e41135a0 con 0x7f92e4102760 2024-03-28T16:05:09.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.705+0000 7f92e9800640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92e41138a0 con 0x7f92e40ffdf0 2024-03-28T16:05:09.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.705+0000 7f92e8e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92e4105110 0x7f92e41a5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:09.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.705+0000 7f92e8e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92e4105110 0x7f92e41a5330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54946/0 (socket says 172.21.15.67:54946) 2024-03-28T16:05:09.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.705+0000 7f92e8e00640 1 -- 172.21.15.67:0/3736008710 learned_addr learned my addr 172.21.15.67:0/3736008710 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:05:09.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.705+0000 7f92e3e00640 1 --2- 172.21.15.67:0/3736008710 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f92e40ffdf0 0x7f92e419ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:09.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.706+0000 7f92e3400640 1 --2- 172.21.15.67:0/3736008710 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92e4102760 0x7f92e419e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:09.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.706+0000 7f92e8e00640 1 -- 172.21.15.67:0/3736008710 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f92e40ffdf0 msgr2=0x7f92e419ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:09.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.706+0000 7f92e8e00640 1 --2- 172.21.15.67:0/3736008710 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f92e40ffdf0 0x7f92e419ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:09.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.706+0000 7f92e8e00640 1 -- 172.21.15.67:0/3736008710 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92e4102760 msgr2=0x7f92e419e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:09.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.706+0000 7f92e8e00640 1 --2- 172.21.15.67:0/3736008710 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92e4102760 0x7f92e419e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:09.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.706+0000 7f92e8e00640 1 -- 172.21.15.67:0/3736008710 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92e41a59d0 con 0x7f92e4105110 2024-03-28T16:05:09.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.706+0000 7f92e8e00640 1 --2- 172.21.15.67:0/3736008710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92e4105110 0x7f92e41a5330 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f92dc02c730 tx=0x7f92dc034830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:05:09.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.707+0000 7f92e0c00640 1 -- 172.21.15.67:0/3736008710 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92dc039040 con 0x7f92e4105110 2024-03-28T16:05:09.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.707+0000 7f92e0c00640 1 -- 172.21.15.67:0/3736008710 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f92dc044070 con 0x7f92e4105110 2024-03-28T16:05:09.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.707+0000 7f92e9800640 1 -- 172.21.15.67:0/3736008710 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92e41a5c60 con 0x7f92e4105110 2024-03-28T16:05:09.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.707+0000 7f92e0c00640 1 -- 172.21.15.67:0/3736008710 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92dc0042a0 con 0x7f92e4105110 2024-03-28T16:05:09.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.707+0000 7f92e9800640 1 -- 172.21.15.67:0/3736008710 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92e41a6150 con 0x7f92e4105110 2024-03-28T16:05:09.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.711+0000 7f92edbd7740 1 Processor -- start 2024-03-28T16:05:09.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.711+0000 7f92e0c00640 1 -- 172.21.15.67:0/3736008710 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f92dc003970 con 0x7f92e4105110 2024-03-28T16:05:09.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.711+0000 7f92edbd7740 1 -- start start 2024-03-28T16:05:09.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.712+0000 7f92edbd7740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cd62f35050 0x55cd62f35470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:09.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.712+0000 7f92edbd7740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cd62f35a40 0x55cd62f3a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:09.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.712+0000 7f92edbd7740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cd62f3abb0 0x55cd62f3d040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:09.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.712+0000 7f92edbd7740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd62e9a250 con 0x55cd62f35050 2024-03-28T16:05:09.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.712+0000 7f92edbd7740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd62e42a30 con 0x55cd62f35a40 2024-03-28T16:05:09.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.712+0000 7f92edbd7740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd62e962f0 con 0x55cd62f3abb0 2024-03-28T16:05:09.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.713+0000 7f92e3e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cd62f35a40 0x55cd62f3a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:09.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.713+0000 7f92e3400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cd62f35050 0x55cd62f35470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:09.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.713+0000 7f92e3e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cd62f35a40 0x55cd62f3a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.67:40646/0 (socket says 172.21.15.67:40646) 2024-03-28T16:05:09.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.713+0000 7f92e3e00640 1 -- 172.21.15.67:0/3617731259 learned_addr learned my addr 172.21.15.67:0/3617731259 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:05:09.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.713+0000 7f92e0c00640 1 --2- 172.21.15.67:0/3736008710 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f92c80a2800 0x7f92c80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:09.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.713+0000 7f92e0c00640 1 -- 172.21.15.67:0/3736008710 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f92dc0f1130 con 0x7f92e4105110 2024-03-28T16:05:09.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.713+0000 7f92e3400640 1 --2- 172.21.15.67:0/3617731259 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cd62f3abb0 0x55cd62f3d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:09.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.713+0000 7f92e8e00640 1 --2- 172.21.15.67:0/3736008710 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f92c80a2800 0x7f92c80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:09.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.713+0000 7f92e3400640 1 -- 172.21.15.67:0/3617731259 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cd62f3abb0 msgr2=0x55cd62f3d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:05:09.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.713+0000 7f92e3400640 1 --2- 172.21.15.67:0/3617731259 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cd62f3abb0 0x55cd62f3d040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:09.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.713+0000 7f92e3400640 1 -- 172.21.15.67:0/3617731259 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cd62f35a40 msgr2=0x55cd62f3a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:05:09.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.713+0000 7f92e3400640 1 --2- 172.21.15.67:0/3617731259 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cd62f35a40 0x55cd62f3a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:09.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.714+0000 7f92e3400640 1 -- 172.21.15.67:0/3617731259 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cd62e2d690 con 0x55cd62f35050 2024-03-28T16:05:09.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.714+0000 7f92e3e00640 1 --2- 172.21.15.67:0/3617731259 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cd62f35a40 0x55cd62f3a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T16:05:09.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.714+0000 7f92e3400640 1 --2- 172.21.15.67:0/3617731259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cd62f35050 0x55cd62f35470 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f92cc00be20 tx=0x7f92cc00b380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38264aeb5368e778 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:05:09.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.714+0000 7f92d9600640 1 -- 172.21.15.67:0/3617731259 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92cc01c070 con 0x55cd62f35050 2024-03-28T16:05:09.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.714+0000 7f92e8e00640 1 --2- 172.21.15.67:0/3736008710 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f92c80a2800 0x7f92c80a4cc0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f92dc0c5d50 tx=0x7f92dc0095c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:05:09.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.714+0000 7f92d9600640 1 -- 172.21.15.67:0/3617731259 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f92cc010040 con 0x55cd62f35050 2024-03-28T16:05:09.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.714+0000 7f92d9600640 1 -- 172.21.15.67:0/3617731259 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92cc014610 con 0x55cd62f35050 2024-03-28T16:05:09.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.715+0000 7f92edbd7740 1 -- 172.21.15.67:0/3617731259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cd62f35050 msgr2=0x55cd62f35470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:09.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.715+0000 7f92edbd7740 1 --2- 172.21.15.67:0/3617731259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cd62f35050 0x55cd62f35470 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f92cc00be20 tx=0x7f92cc00b380 comp rx=0 tx=0).stop 2024-03-28T16:05:09.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.715+0000 7f92edbd7740 1 -- 172.21.15.67:0/3617731259 shutdown_connections 2024-03-28T16:05:09.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.715+0000 7f92edbd7740 1 --2- 172.21.15.67:0/3617731259 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cd62f3abb0 0x55cd62f3d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:09.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.715+0000 7f92edbd7740 1 --2- 172.21.15.67:0/3617731259 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cd62f35a40 0x55cd62f3a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:09.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.715+0000 7f92edbd7740 1 --2- 172.21.15.67:0/3617731259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cd62f35050 0x55cd62f35470 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:09.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.715+0000 7f92edbd7740 1 -- 172.21.15.67:0/3617731259 >> 172.21.15.67:0/3617731259 conn(0x55cd62efd200 msgr2=0x55cd62f309c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:05:09.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.716+0000 7f92edbd7740 1 -- 172.21.15.67:0/3617731259 shutdown_connections 2024-03-28T16:05:09.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.716+0000 7f92edbd7740 1 -- 172.21.15.67:0/3617731259 wait complete. 2024-03-28T16:05:09.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.716+0000 7f92edbd7740 1 -- start start 2024-03-28T16:05:09.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.716+0000 7f92edbd7740 1 Processor -- start 2024-03-28T16:05:09.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.717+0000 7f92edbd7740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cd62f35050 0x55cd62f3f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:09.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.717+0000 7f92edbd7740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cd62f35a40 0x55cd62f3f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:09.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.717+0000 7f92edbd7740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cd62f3abb0 0x55cd62f46610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:09.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.717+0000 7f92edbd7740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd62f3db60 con 0x55cd62f35050 2024-03-28T16:05:09.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.717+0000 7f92edbd7740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd62e55e80 con 0x55cd62f3abb0 2024-03-28T16:05:09.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.717+0000 7f92edbd7740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd62f3dce0 con 0x55cd62f35a40 2024-03-28T16:05:09.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.717+0000 7f92edbd7740 10 client.0 _ll_register_callbacks cb 0x55cd62e68130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:05:09.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.717+0000 7f92e3400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cd62f35050 0x55cd62f3f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:09.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.717+0000 7f92e3400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cd62f35050 0x55cd62f3f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54964/0 (socket says 172.21.15.67:54964) 2024-03-28T16:05:09.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.717+0000 7f92e3400640 1 -- 172.21.15.67:0/2181166888 learned_addr learned my addr 172.21.15.67:0/2181166888 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:05:09.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.718+0000 7f92e3e00640 1 --2- 172.21.15.67:0/2181166888 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cd62f35a40 0x55cd62f3f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:09.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.718+0000 7f92e3400640 1 -- 172.21.15.67:0/2181166888 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cd62f35a40 msgr2=0x55cd62f3f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:05:09.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.718+0000 7f92e3400640 1 --2- 172.21.15.67:0/2181166888 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cd62f35a40 0x55cd62f3f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:09.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.718+0000 7f92e3400640 1 -- 172.21.15.67:0/2181166888 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cd62f3abb0 msgr2=0x55cd62f46610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:05:09.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.718+0000 7f92e3400640 1 --2- 172.21.15.67:0/2181166888 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cd62f3abb0 0x55cd62f46610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:09.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.718+0000 7f92e3400640 1 -- 172.21.15.67:0/2181166888 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cd62c787d0 con 0x55cd62f35050 2024-03-28T16:05:09.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.718+0000 7f92e3400640 1 --2- 172.21.15.67:0/2181166888 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cd62f35050 0x55cd62f3f090 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f92cc006d70 tx=0x7f92cc00ba60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad4d5a5d2d9c74d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:05:09.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.718+0000 7f92d2000640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:05:09.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.718+0000 7f92d2000640 1 -- 172.21.15.67:0/2181166888 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92cc01c040 con 0x55cd62f35050 2024-03-28T16:05:09.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.719+0000 7f92d2000640 1 -- 172.21.15.67:0/2181166888 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f92cc023460 con 0x55cd62f35050 2024-03-28T16:05:09.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.719+0000 7f92edbd7740 1 -- 172.21.15.67:0/2181166888 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f92cc010040 con 0x55cd62f35050 2024-03-28T16:05:09.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.719+0000 7f92d2000640 1 -- 172.21.15.67:0/2181166888 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f92cc00fea0 con 0x55cd62f35050 2024-03-28T16:05:09.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.719+0000 7f92edbd7740 10 client.15495 fetch_fsmap learned FSMap version 19 2024-03-28T16:05:09.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.719+0000 7f92edbd7740 1 -- 172.21.15.67:0/2181166888 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55cd62bae3f0 con 0x55cd62f35050 2024-03-28T16:05:09.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.719+0000 7f92d2000640 1 -- 172.21.15.67:0/2181166888 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f92cc010040 con 0x55cd62f35050 2024-03-28T16:05:09.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.719+0000 7f92d2000640 1 -- 172.21.15.67:0/2181166888 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f92cc02c6f0 con 0x55cd62f35050 2024-03-28T16:05:09.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.720+0000 7f92edbd7740 10 client.15495 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:05:09.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.720+0000 7f92edbd7740 10 client.15495 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:05:09.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.720+0000 7f92edbd7740 20 client.15495 populate_metadata read hostname 'smithi067' 2024-03-28T16:05:09.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.720+0000 7f92edbd7740 1 --2- 172.21.15.67:0/2181166888 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55cd62f4abb0 0x55cd62f4cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:09.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.720+0000 7f92edbd7740 4 client.15495 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:05:09.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.720+0000 7f92edbd7740 1 -- 172.21.15.67:0/2181166888 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cd62ea7790 con 0x55cd62f4abb0 2024-03-28T16:05:09.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.720+0000 7f92e3e00640 1 --2- 172.21.15.67:0/2181166888 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55cd62f4abb0 0x55cd62f4cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:09.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.721+0000 7f92e3e00640 1 --2- 172.21.15.67:0/2181166888 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55cd62f4abb0 0x55cd62f4cfa0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c7c62c9c28201680 server_cookie=b0a1e7fe68870d70 in_seq=0 out_seq=0 2024-03-28T16:05:09.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.721+0000 7f92d2000640 10 client.15495 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:05:09.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.723+0000 7f92d2000640 1 -- 172.21.15.67:0/2181166888 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55cd62ea7790 con 0x55cd62f4abb0 2024-03-28T16:05:09.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.723+0000 7f92d2000640 10 client.15495 handle_command_reply: tid=0 2024-03-28T16:05:09.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.723+0000 7f92edbd7740 1 client.15495 shutdown 2024-03-28T16:05:09.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.723+0000 7f92edbd7740 1 -- 172.21.15.67:0/2181166888 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cd62f35050 msgr2=0x55cd62f3f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:09.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.723+0000 7f92edbd7740 1 --2- 172.21.15.67:0/2181166888 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cd62f35050 0x55cd62f3f090 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f92cc006d70 tx=0x7f92cc00ba60 comp rx=0 tx=0).stop 2024-03-28T16:05:09.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.723+0000 7f92edbd7740 1 -- 172.21.15.67:0/2181166888 shutdown_connections 2024-03-28T16:05:09.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.723+0000 7f92edbd7740 1 --2- 172.21.15.67:0/2181166888 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55cd62f4abb0 0x55cd62f4cfa0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:09.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.723+0000 7f92edbd7740 1 --2- 172.21.15.67:0/2181166888 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cd62f3abb0 0x55cd62f46610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:09.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.723+0000 7f92edbd7740 1 --2- 172.21.15.67:0/2181166888 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cd62f35a40 0x55cd62f3f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:09.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.723+0000 7f92edbd7740 1 --2- 172.21.15.67:0/2181166888 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cd62f35050 0x55cd62f3f090 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:09.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.723+0000 7f92edbd7740 1 -- 172.21.15.67:0/2181166888 >> 172.21.15.67:0/2181166888 conn(0x55cd62efd200 msgr2=0x55cd62f2fbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:05:09.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.723+0000 7f92d2000640 0 client.15495 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:05:09.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.724+0000 7f92edbd7740 1 -- 172.21.15.67:0/2181166888 shutdown_connections 2024-03-28T16:05:09.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.724+0000 7f92edbd7740 1 -- 172.21.15.67:0/2181166888 wait complete. 2024-03-28T16:05:09.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.724+0000 7f92edbd7740 20 client.15495 trim_cache size 0 max 16384 2024-03-28T16:05:09.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.744+0000 7f92edbd7740 1 Processor -- start 2024-03-28T16:05:09.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.744+0000 7f92edbd7740 1 -- start start 2024-03-28T16:05:09.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.745+0000 7f92edbd7740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cd62f35050 0x55cd62f3f530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:09.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.745+0000 7f92edbd7740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cd62f35a40 0x55cd62f465a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:09.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.745+0000 7f92edbd7740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cd62f3abb0 0x55cd62f36bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:09.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.745+0000 7f92edbd7740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92e41acd20 con 0x55cd62f3abb0 2024-03-28T16:05:09.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.745+0000 7f92edbd7740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92e41af2e0 con 0x55cd62f35050 2024-03-28T16:05:09.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.745+0000 7f92edbd7740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92e41ad320 con 0x55cd62f35a40 2024-03-28T16:05:09.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.745+0000 7f92e3400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cd62f3abb0 0x55cd62f36bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:09.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.745+0000 7f92e3400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cd62f3abb0 0x55cd62f36bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54972/0 (socket says 172.21.15.67:54972) 2024-03-28T16:05:09.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.745+0000 7f92e3400640 1 -- 172.21.15.67:0/3510590355 learned_addr learned my addr 172.21.15.67:0/3510590355 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:05:09.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.745+0000 7f92e3e00640 1 --2- 172.21.15.67:0/3510590355 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cd62f35a40 0x55cd62f465a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:09.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.746+0000 7f92e3400640 1 --2- 172.21.15.67:0/3510590355 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cd62f35050 0x55cd62f3f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:09.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.746+0000 7f92e3400640 1 -- 172.21.15.67:0/3510590355 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cd62f35a40 msgr2=0x55cd62f465a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:05:09.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.746+0000 7f92e3400640 1 --2- 172.21.15.67:0/3510590355 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cd62f35a40 0x55cd62f465a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:09.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.746+0000 7f92e3400640 1 -- 172.21.15.67:0/3510590355 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cd62f35050 msgr2=0x55cd62f3f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:05:09.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.746+0000 7f92e3400640 1 --2- 172.21.15.67:0/3510590355 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cd62f35050 0x55cd62f3f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:09.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.746+0000 7f92e3400640 1 -- 172.21.15.67:0/3510590355 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cd62b6e0f0 con 0x55cd62f3abb0 2024-03-28T16:05:09.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.746+0000 7f92e3400640 1 --2- 172.21.15.67:0/3510590355 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cd62f3abb0 0x55cd62f36bf0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f92cc006dc0 tx=0x7f92cc00f9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4cdae1d839033ad server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:05:09.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.747+0000 7f92d8c00640 1 -- 172.21.15.67:0/3510590355 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92cc01c030 con 0x55cd62f3abb0 2024-03-28T16:05:09.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.747+0000 7f92d8c00640 1 -- 172.21.15.67:0/3510590355 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f92cc028070 con 0x55cd62f3abb0 2024-03-28T16:05:09.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.747+0000 7f92d8c00640 1 -- 172.21.15.67:0/3510590355 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92cc02c820 con 0x55cd62f3abb0 2024-03-28T16:05:09.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.747+0000 7f92edbd7740 1 -- 172.21.15.67:0/3510590355 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cd62f3abb0 msgr2=0x55cd62f36bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:09.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.747+0000 7f92edbd7740 1 --2- 172.21.15.67:0/3510590355 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cd62f3abb0 0x55cd62f36bf0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f92cc006dc0 tx=0x7f92cc00f9e0 comp rx=0 tx=0).stop 2024-03-28T16:05:09.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.748+0000 7f92edbd7740 1 -- 172.21.15.67:0/3510590355 shutdown_connections 2024-03-28T16:05:09.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.748+0000 7f92edbd7740 1 --2- 172.21.15.67:0/3510590355 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cd62f3abb0 0x55cd62f36bf0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:09.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.748+0000 7f92edbd7740 1 --2- 172.21.15.67:0/3510590355 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cd62f35a40 0x55cd62f465a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:09.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.748+0000 7f92edbd7740 1 --2- 172.21.15.67:0/3510590355 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cd62f35050 0x55cd62f3f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:09.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.748+0000 7f92edbd7740 1 -- 172.21.15.67:0/3510590355 >> 172.21.15.67:0/3510590355 conn(0x55cd62efd200 msgr2=0x55cd62f364e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:05:09.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.748+0000 7f92edbd7740 1 -- 172.21.15.67:0/3510590355 shutdown_connections 2024-03-28T16:05:09.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.748+0000 7f92edbd7740 1 -- 172.21.15.67:0/3510590355 wait complete. 2024-03-28T16:05:09.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.748+0000 7f92edbd7740 1 -- start start 2024-03-28T16:05:09.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.748+0000 7f92edbd7740 1 Processor -- start 2024-03-28T16:05:09.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.749+0000 7f92edbd7740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cd62f35050 0x55cd62f436e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:09.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.749+0000 7f92edbd7740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cd62f35a40 0x55cd62f43c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:09.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.749+0000 7f92edbd7740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cd62f3abb0 0x55cd62f44b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:09.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.749+0000 7f92edbd7740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92e41ad960 con 0x55cd62f3abb0 2024-03-28T16:05:09.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.749+0000 7f92edbd7740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92e41aec80 con 0x55cd62f35a40 2024-03-28T16:05:09.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.749+0000 7f92edbd7740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92e41adfc0 con 0x55cd62f35050 2024-03-28T16:05:09.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.749+0000 7f92edbd7740 10 client.0 _ll_register_callbacks cb 0x7f92cc01a070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:05:09.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.749+0000 7f92e3400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cd62f3abb0 0x55cd62f44b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:09.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.749+0000 7f92e3400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cd62f3abb0 0x55cd62f44b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54978/0 (socket says 172.21.15.67:54978) 2024-03-28T16:05:09.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.749+0000 7f92e3400640 1 -- 172.21.15.67:0/1699699068 learned_addr learned my addr 172.21.15.67:0/1699699068 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:05:09.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.749+0000 7f92e3e00640 1 --2- 172.21.15.67:0/1699699068 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cd62f35a40 0x55cd62f43c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:09.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.749+0000 7f92e3400640 1 --2- 172.21.15.67:0/1699699068 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cd62f35050 0x55cd62f436e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:09.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.750+0000 7f92e3400640 1 -- 172.21.15.67:0/1699699068 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cd62f35050 msgr2=0x55cd62f436e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:05:09.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.750+0000 7f92e3400640 1 --2- 172.21.15.67:0/1699699068 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cd62f35050 0x55cd62f436e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:09.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.750+0000 7f92e3400640 1 -- 172.21.15.67:0/1699699068 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cd62f35a40 msgr2=0x55cd62f43c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:05:09.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.750+0000 7f92e3400640 1 --2- 172.21.15.67:0/1699699068 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cd62f35a40 0x55cd62f43c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:09.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.750+0000 7f92e3400640 1 -- 172.21.15.67:0/1699699068 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cd62f45160 con 0x55cd62f3abb0 2024-03-28T16:05:09.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.750+0000 7f92e3400640 1 --2- 172.21.15.67:0/1699699068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cd62f3abb0 0x55cd62f44b70 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f92cc02cb90 tx=0x7f92cc0046e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a6aeb79d969e934 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:05:09.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.750+0000 7f92d2000640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:05:09.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.750+0000 7f92d2000640 1 -- 172.21.15.67:0/1699699068 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92cc005970 con 0x55cd62f3abb0 2024-03-28T16:05:09.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.751+0000 7f92d2000640 1 -- 172.21.15.67:0/1699699068 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f92cc005b10 con 0x55cd62f3abb0 2024-03-28T16:05:09.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.751+0000 7f92edbd7740 1 -- 172.21.15.67:0/1699699068 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f92cc028070 con 0x55cd62f3abb0 2024-03-28T16:05:09.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.751+0000 7f92d2000640 1 -- 172.21.15.67:0/1699699068 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f92cc005d10 con 0x55cd62f3abb0 2024-03-28T16:05:09.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.751+0000 7f92edbd7740 10 client.15501 fetch_fsmap learned FSMap version 19 2024-03-28T16:05:09.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.751+0000 7f92edbd7740 1 -- 172.21.15.67:0/1699699068 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55cd62f46470 con 0x55cd62f3abb0 2024-03-28T16:05:09.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.751+0000 7f92d2000640 1 -- 172.21.15.67:0/1699699068 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f92cc028070 con 0x55cd62f3abb0 2024-03-28T16:05:09.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.751+0000 7f92d2000640 1 -- 172.21.15.67:0/1699699068 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f92cc02f480 con 0x55cd62f3abb0 2024-03-28T16:05:09.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.751+0000 7f92edbd7740 10 client.15501 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:05:09.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.751+0000 7f92edbd7740 10 client.15501 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:05:09.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.751+0000 7f92edbd7740 20 client.15501 populate_metadata read hostname 'smithi067' 2024-03-28T16:05:09.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.752+0000 7f92edbd7740 1 --2- 172.21.15.67:0/1699699068 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55cd62fa66d0 0x55cd62fa8ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:09.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.752+0000 7f92edbd7740 4 client.15501 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T16:05:09.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.752+0000 7f92edbd7740 1 -- 172.21.15.67:0/1699699068 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55cd62ea1ae0 con 0x55cd62fa66d0 2024-03-28T16:05:09.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.752+0000 7f92e3e00640 1 --2- 172.21.15.67:0/1699699068 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55cd62fa66d0 0x55cd62fa8ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:09.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.753+0000 7f92e3e00640 1 --2- 172.21.15.67:0/1699699068 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55cd62fa66d0 0x55cd62fa8ac0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b9d38e2af9061a9 server_cookie=d05fbc19b8f0d219 in_seq=0 out_seq=0 2024-03-28T16:05:09.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.753+0000 7f92d2000640 10 client.15501 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:05:09.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.754+0000 7f92d2000640 1 -- 172.21.15.67:0/1699699068 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55cd62ea1ae0 con 0x55cd62fa66d0 2024-03-28T16:05:09.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.754+0000 7f92d2000640 10 client.15501 handle_command_reply: tid=0 2024-03-28T16:05:09.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.754+0000 7f92edbd7740 1 client.15501 shutdown 2024-03-28T16:05:09.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.754+0000 7f92edbd7740 1 -- 172.21.15.67:0/1699699068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cd62f3abb0 msgr2=0x55cd62f44b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:09.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.754+0000 7f92edbd7740 1 --2- 172.21.15.67:0/1699699068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cd62f3abb0 0x55cd62f44b70 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f92cc02cb90 tx=0x7f92cc0046e0 comp rx=0 tx=0).stop 2024-03-28T16:05:09.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.754+0000 7f92edbd7740 1 -- 172.21.15.67:0/1699699068 shutdown_connections 2024-03-28T16:05:09.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.754+0000 7f92edbd7740 1 --2- 172.21.15.67:0/1699699068 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55cd62fa66d0 0x55cd62fa8ac0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:09.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.754+0000 7f92edbd7740 1 --2- 172.21.15.67:0/1699699068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cd62f3abb0 0x55cd62f44b70 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:09.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.754+0000 7f92edbd7740 1 --2- 172.21.15.67:0/1699699068 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cd62f35a40 0x55cd62f43c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:09.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.754+0000 7f92edbd7740 1 --2- 172.21.15.67:0/1699699068 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cd62f35050 0x55cd62f436e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:09.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.754+0000 7f92edbd7740 1 -- 172.21.15.67:0/1699699068 >> 172.21.15.67:0/1699699068 conn(0x55cd62efd200 msgr2=0x55cd62f34490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:05:09.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.754+0000 7f92d2000640 0 client.15501 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:05:09.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.754+0000 7f92edbd7740 1 -- 172.21.15.67:0/1699699068 shutdown_connections 2024-03-28T16:05:09.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.754+0000 7f92edbd7740 1 -- 172.21.15.67:0/1699699068 wait complete. 2024-03-28T16:05:09.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.754+0000 7f92edbd7740 20 client.15501 trim_cache size 0 max 16384 2024-03-28T16:05:09.757 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-28T16:05:09.757 INFO:teuthology.orchestra.run.smithi067.stdout: "status": "no active scrubs running", 2024-03-28T16:05:09.757 INFO:teuthology.orchestra.run.smithi067.stdout: "scrubs": {} 2024-03-28T16:05:09.757 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-28T16:05:09.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.755+0000 7f92e9800640 1 -- 172.21.15.67:0/3736008710 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f92c80a2800 msgr2=0x7f92c80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:09.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.755+0000 7f92e9800640 1 --2- 172.21.15.67:0/3736008710 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f92c80a2800 0x7f92c80a4cc0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f92dc0c5d50 tx=0x7f92dc0095c0 comp rx=0 tx=0).stop 2024-03-28T16:05:09.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.755+0000 7f92e9800640 1 -- 172.21.15.67:0/3736008710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92e4105110 msgr2=0x7f92e41a5330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:09.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.755+0000 7f92e9800640 1 --2- 172.21.15.67:0/3736008710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92e4105110 0x7f92e41a5330 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f92dc02c730 tx=0x7f92dc034830 comp rx=0 tx=0).stop 2024-03-28T16:05:09.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.755+0000 7f92e9800640 1 -- 172.21.15.67:0/3736008710 shutdown_connections 2024-03-28T16:05:09.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.755+0000 7f92e9800640 1 --2- 172.21.15.67:0/3736008710 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f92c80a2800 0x7f92c80a4cc0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:09.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.755+0000 7f92e9800640 1 --2- 172.21.15.67:0/3736008710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92e4105110 0x7f92e41a5330 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:09.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.755+0000 7f92e9800640 1 --2- 172.21.15.67:0/3736008710 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92e4102760 0x7f92e419e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:09.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.755+0000 7f92e9800640 1 --2- 172.21.15.67:0/3736008710 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f92e40ffdf0 0x7f92e419ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:09.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.755+0000 7f92e9800640 1 -- 172.21.15.67:0/3736008710 >> 172.21.15.67:0/3736008710 conn(0x7f92e40fba10 msgr2=0x7f92e4100790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:05:09.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.755+0000 7f92e9800640 1 -- 172.21.15.67:0/3736008710 shutdown_connections 2024-03-28T16:05:09.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:09.755+0000 7f92e9800640 1 -- 172.21.15.67:0/3736008710 wait complete. 2024-03-28T16:05:09.777 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T16:05:09.777 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T16:05:10.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.006+0000 7ff3b5400640 1 -- 172.21.15.67:0/3185432223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3b0101510 msgr2=0x7ff3b0101930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:10.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.006+0000 7ff3b5400640 1 --2- 172.21.15.67:0/3185432223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3b0101510 0x7ff3b0101930 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7ff3a0009ca0 tx=0x7ff3a002c100 comp rx=0 tx=0).stop 2024-03-28T16:05:10.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.007+0000 7ff3b5400640 1 -- 172.21.15.67:0/3185432223 shutdown_connections 2024-03-28T16:05:10.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.007+0000 7ff3b5400640 1 --2- 172.21.15.67:0/3185432223 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff3b01030c0 0x7ff3b0109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:10.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.007+0000 7ff3b5400640 1 --2- 172.21.15.67:0/3185432223 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3b0102700 0x7ff3b0102b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:10.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.007+0000 7ff3b5400640 1 --2- 172.21.15.67:0/3185432223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3b0101510 0x7ff3b0101930 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:10.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.007+0000 7ff3b5400640 1 -- 172.21.15.67:0/3185432223 >> 172.21.15.67:0/3185432223 conn(0x7ff3b00fca80 msgr2=0x7ff3b00feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:05:10.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.007+0000 7ff3b5400640 1 -- 172.21.15.67:0/3185432223 shutdown_connections 2024-03-28T16:05:10.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.007+0000 7ff3b5400640 1 -- 172.21.15.67:0/3185432223 wait complete. 2024-03-28T16:05:10.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.008+0000 7ff3b5400640 1 Processor -- start 2024-03-28T16:05:10.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.008+0000 7ff3b5400640 1 -- start start 2024-03-28T16:05:10.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.008+0000 7ff3b5400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3b0101510 0x7ff3b0075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:10.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.008+0000 7ff3b5400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3b0102700 0x7ff3b0073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:10.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.008+0000 7ff3b5400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff3b01030c0 0x7ff3b0073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:10.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.008+0000 7ff3b5400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b010d020 con 0x7ff3b0101510 2024-03-28T16:05:10.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.008+0000 7ff3b5400640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b010cea0 con 0x7ff3b0102700 2024-03-28T16:05:10.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.008+0000 7ff3b5400640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b010d1a0 con 0x7ff3b01030c0 2024-03-28T16:05:10.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.009+0000 7ff3afe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3b0101510 0x7ff3b0075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:10.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.009+0000 7ff3afe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3b0101510 0x7ff3b0075450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55000/0 (socket says 172.21.15.67:55000) 2024-03-28T16:05:10.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.009+0000 7ff3afe00640 1 -- 172.21.15.67:0/2327319508 learned_addr learned my addr 172.21.15.67:0/2327319508 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:05:10.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.009+0000 7ff3b4a00640 1 --2- 172.21.15.67:0/2327319508 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff3b01030c0 0x7ff3b0073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:10.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.009+0000 7ff3a7400640 1 --2- 172.21.15.67:0/2327319508 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3b0102700 0x7ff3b0073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:10.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.009+0000 7ff3afe00640 1 -- 172.21.15.67:0/2327319508 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff3b01030c0 msgr2=0x7ff3b0073fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:10.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.009+0000 7ff3afe00640 1 --2- 172.21.15.67:0/2327319508 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff3b01030c0 0x7ff3b0073fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:10.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.009+0000 7ff3afe00640 1 -- 172.21.15.67:0/2327319508 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3b0102700 msgr2=0x7ff3b0073a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:10.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.009+0000 7ff3afe00640 1 --2- 172.21.15.67:0/2327319508 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3b0102700 0x7ff3b0073a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:10.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.009+0000 7ff3afe00640 1 -- 172.21.15.67:0/2327319508 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3b0074860 con 0x7ff3b0101510 2024-03-28T16:05:10.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.010+0000 7ff3afe00640 1 --2- 172.21.15.67:0/2327319508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3b0101510 0x7ff3b0075450 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7ff3a0033000 tx=0x7ff3a0035d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:05:10.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.010+0000 7ff3ad600640 1 -- 172.21.15.67:0/2327319508 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a0039040 con 0x7ff3b0101510 2024-03-28T16:05:10.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.010+0000 7ff3ad600640 1 -- 172.21.15.67:0/2327319508 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff3a0044070 con 0x7ff3b0101510 2024-03-28T16:05:10.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.010+0000 7ff3b5400640 1 -- 172.21.15.67:0/2327319508 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3b01a17e0 con 0x7ff3b0101510 2024-03-28T16:05:10.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.010+0000 7ff3ad600640 1 -- 172.21.15.67:0/2327319508 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a00044d0 con 0x7ff3b0101510 2024-03-28T16:05:10.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.010+0000 7ff3b5400640 1 -- 172.21.15.67:0/2327319508 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3b01a1d80 con 0x7ff3b0101510 2024-03-28T16:05:10.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.012+0000 7ff3b5400640 1 -- 172.21.15.67:0/2327319508 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3b01088f0 con 0x7ff3b0101510 2024-03-28T16:05:10.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.015+0000 7ff3ad600640 1 -- 172.21.15.67:0/2327319508 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ff3a00327c0 con 0x7ff3b0101510 2024-03-28T16:05:10.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.016+0000 7ff3ad600640 1 --2- 172.21.15.67:0/2327319508 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7ff3940a2880 0x7ff3940a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:10.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.017+0000 7ff3ad600640 1 -- 172.21.15.67:0/2327319508 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7ff3a00f1c20 con 0x7ff3b0101510 2024-03-28T16:05:10.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.017+0000 7ff3a7400640 1 --2- 172.21.15.67:0/2327319508 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7ff3940a2880 0x7ff3940a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:10.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.017+0000 7ff3a7400640 1 --2- 172.21.15.67:0/2327319508 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7ff3940a2880 0x7ff3940a4d40 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7ff398005fd0 tx=0x7ff398005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:05:10.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.018+0000 7ff3ad600640 1 -- 172.21.15.67:0/2327319508 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff3a0051380 con 0x7ff3b0101510 2024-03-28T16:05:10.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.306+0000 7ff3b5400640 1 -- 172.21.15.67:0/2327319508 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff3b002d070 con 0x7ff3b0101510 2024-03-28T16:05:10.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.307+0000 7ff3ad600640 1 -- 172.21.15.67:0/2327319508 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11341 (secure 0 0 0) 0x7ff3a00bec70 con 0x7ff3b0101510 2024-03-28T16:05:10.308 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:05:10.308 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2008297659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2008297659},{"type":"v1","addr":"172.21.15.67:6835","nonce":2008297659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14532,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/320295745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":320295745},{"type":"v1","addr":"172.21.15.67:6837","nonce":320295745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6839/691821273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":691821273},{"type":"v1","addr":"172.21.15.67:6839","nonce":691821273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6841/3211254875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6840","nonce":3211254875},{"type":"v1","addr":"172.21.15.67:6841","nonce":3211254875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2129665423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2129665423},{"type":"v1","addr":"172.21.15.183:6833","nonce":2129665423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24496,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/1589784425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":1589784425},{"type":"v1","addr":"172.21.15.183:6835","nonce":1589784425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24508,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/2185863835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":2185863835},{"type":"v1","addr":"172.21.15.183:6837","nonce":2185863835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24520,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6839/1242814398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":1242814398},{"type":"v1","addr":"172.21.15.183:6839","nonce":1242814398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":34409,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1008335521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1008335521},{"type":"v1","addr":"172.21.15.150:6833","nonce":1008335521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":34445,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3269935695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3269935695},{"type":"v1","addr":"172.21.15.150:6835","nonce":3269935695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T16:01:07.696824+0000","modified":"2024-03-28T16:01:11.997773+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":34508},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34475":{"gid":34475,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/1112886277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":1112886277},{"type":"v1","addr":"172.21.15.150:6837","nonce":1112886277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_34508":{"gid":34508,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6839/1211011614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":1211011614},{"type":"v1","addr":"172.21.15.150:6839","nonce":1211011614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34508,"qdb_cluster":[34508]},"id":1}]} 2024-03-28T16:05:10.308 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 19 2024-03-28T16:05:10.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.310+0000 7ff3b5400640 1 -- 172.21.15.67:0/2327319508 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7ff3940a2880 msgr2=0x7ff3940a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:10.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.310+0000 7ff3b5400640 1 --2- 172.21.15.67:0/2327319508 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7ff3940a2880 0x7ff3940a4d40 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7ff398005fd0 tx=0x7ff398005850 comp rx=0 tx=0).stop 2024-03-28T16:05:10.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.310+0000 7ff3b5400640 1 -- 172.21.15.67:0/2327319508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3b0101510 msgr2=0x7ff3b0075450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:10.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.310+0000 7ff3b5400640 1 --2- 172.21.15.67:0/2327319508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3b0101510 0x7ff3b0075450 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7ff3a0033000 tx=0x7ff3a0035d60 comp rx=0 tx=0).stop 2024-03-28T16:05:10.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.311+0000 7ff3b5400640 1 -- 172.21.15.67:0/2327319508 shutdown_connections 2024-03-28T16:05:10.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.311+0000 7ff3b5400640 1 --2- 172.21.15.67:0/2327319508 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7ff3940a2880 0x7ff3940a4d40 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:10.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.311+0000 7ff3b5400640 1 --2- 172.21.15.67:0/2327319508 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff3b01030c0 0x7ff3b0073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:10.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.311+0000 7ff3b5400640 1 --2- 172.21.15.67:0/2327319508 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3b0102700 0x7ff3b0073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:10.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.311+0000 7ff3b5400640 1 --2- 172.21.15.67:0/2327319508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3b0101510 0x7ff3b0075450 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:10.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.311+0000 7ff3b5400640 1 -- 172.21.15.67:0/2327319508 >> 172.21.15.67:0/2327319508 conn(0x7ff3b00fca80 msgr2=0x7ff3b0105ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:05:10.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.311+0000 7ff3b5400640 1 -- 172.21.15.67:0/2327319508 shutdown_connections 2024-03-28T16:05:10.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.311+0000 7ff3b5400640 1 -- 172.21.15.67:0/2327319508 wait complete. 2024-03-28T16:05:10.315 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:05:10 smithi067 ceph-mon[31707]: pgmap v487: 97 pgs: 97 active+clean; 73 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6.0 MiB/s wr, 4 op/s 2024-03-28T16:05:10.315 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:05:10 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/2327319508' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:05:10.335 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T16:05:10.348 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T16:05:10.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.588+0000 7f8adca00640 1 -- 172.21.15.67:0/1657338983 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ad81015a0 msgr2=0x7f8ad81019c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:10.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.588+0000 7f8adca00640 1 --2- 172.21.15.67:0/1657338983 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ad81015a0 0x7f8ad81019c0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f8ac8009900 tx=0x7f8ac802bc30 comp rx=0 tx=0).stop 2024-03-28T16:05:10.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.588+0000 7f8adca00640 1 -- 172.21.15.67:0/1657338983 shutdown_connections 2024-03-28T16:05:10.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.588+0000 7f8adca00640 1 --2- 172.21.15.67:0/1657338983 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8ad8103150 0x7f8ad81099e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:10.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.588+0000 7f8adca00640 1 --2- 172.21.15.67:0/1657338983 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ad8102790 0x7f8ad8102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:10.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.588+0000 7f8adca00640 1 --2- 172.21.15.67:0/1657338983 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ad81015a0 0x7f8ad81019c0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:10.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.588+0000 7f8adca00640 1 -- 172.21.15.67:0/1657338983 >> 172.21.15.67:0/1657338983 conn(0x7f8ad80fcb10 msgr2=0x7f8ad80fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:05:10.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.588+0000 7f8adca00640 1 -- 172.21.15.67:0/1657338983 shutdown_connections 2024-03-28T16:05:10.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.588+0000 7f8adca00640 1 -- 172.21.15.67:0/1657338983 wait complete. 2024-03-28T16:05:10.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.589+0000 7f8adca00640 1 Processor -- start 2024-03-28T16:05:10.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.590+0000 7f8adca00640 1 -- start start 2024-03-28T16:05:10.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.590+0000 7f8adca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ad81015a0 0x7f8ad81977a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:10.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.590+0000 7f8adca00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ad8102790 0x7f8ad8197ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:10.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.590+0000 7f8adca00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8ad8103150 0x7f8ad819ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:10.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.590+0000 7f8adca00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad810d0c0 con 0x7f8ad81015a0 2024-03-28T16:05:10.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.590+0000 7f8adca00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad810cf40 con 0x7f8ad8102790 2024-03-28T16:05:10.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.590+0000 7f8adca00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad810d240 con 0x7f8ad8103150 2024-03-28T16:05:10.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.590+0000 7f8ad6800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ad81015a0 0x7f8ad81977a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:10.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.590+0000 7f8ad6800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ad81015a0 0x7f8ad81977a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55014/0 (socket says 172.21.15.67:55014) 2024-03-28T16:05:10.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.590+0000 7f8ad6800640 1 -- 172.21.15.67:0/1556695976 learned_addr learned my addr 172.21.15.67:0/1556695976 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:05:10.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.591+0000 7f8ad5e00640 1 --2- 172.21.15.67:0/1556695976 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ad8102790 0x7f8ad8197ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:10.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.591+0000 7f8ad7200640 1 --2- 172.21.15.67:0/1556695976 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8ad8103150 0x7f8ad819ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:10.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.591+0000 7f8ad6800640 1 -- 172.21.15.67:0/1556695976 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8ad8103150 msgr2=0x7f8ad819ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:10.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.591+0000 7f8ad6800640 1 --2- 172.21.15.67:0/1556695976 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8ad8103150 0x7f8ad819ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:10.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.591+0000 7f8ad6800640 1 -- 172.21.15.67:0/1556695976 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ad8102790 msgr2=0x7f8ad8197ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:10.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.591+0000 7f8ad6800640 1 --2- 172.21.15.67:0/1556695976 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ad8102790 0x7f8ad8197ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:10.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.591+0000 7f8ad6800640 1 -- 172.21.15.67:0/1556695976 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ad819f3c0 con 0x7f8ad81015a0 2024-03-28T16:05:10.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.591+0000 7f8ad6800640 1 --2- 172.21.15.67:0/1556695976 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ad81015a0 0x7f8ad81977a0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f8ac8000c00 tx=0x7f8ac8004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:05:10.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.591+0000 7f8acf400640 1 -- 172.21.15.67:0/1556695976 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ac8043020 con 0x7f8ad81015a0 2024-03-28T16:05:10.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.592+0000 7f8acf400640 1 -- 172.21.15.67:0/1556695976 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8ac8004aa0 con 0x7f8ad81015a0 2024-03-28T16:05:10.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.592+0000 7f8adca00640 1 -- 172.21.15.67:0/1556695976 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ad819f650 con 0x7f8ad81015a0 2024-03-28T16:05:10.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.592+0000 7f8acf400640 1 -- 172.21.15.67:0/1556695976 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ac8003d00 con 0x7f8ad81015a0 2024-03-28T16:05:10.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.592+0000 7f8adca00640 1 -- 172.21.15.67:0/1556695976 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ad819fbf0 con 0x7f8ad81015a0 2024-03-28T16:05:10.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.596+0000 7f8ae004c740 1 Processor -- start 2024-03-28T16:05:10.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.596+0000 7f8acf400640 1 -- 172.21.15.67:0/1556695976 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f8ac8046020 con 0x7f8ad81015a0 2024-03-28T16:05:10.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.596+0000 7f8ae004c740 1 -- start start 2024-03-28T16:05:10.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.596+0000 7f8ae004c740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556d962fce00 0x556d962fd220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:10.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.596+0000 7f8ae004c740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556d962fdfd0 0x556d962fe450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:10.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.596+0000 7f8ae004c740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556d962fe990 0x556d96305220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:10.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.596+0000 7f8ae004c740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d9625d2c0 con 0x556d962fe990 2024-03-28T16:05:10.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.596+0000 7f8ae004c740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d96205c90 con 0x556d962fce00 2024-03-28T16:05:10.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.596+0000 7f8ae004c740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d96259360 con 0x556d962fdfd0 2024-03-28T16:05:10.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.597+0000 7f8ad5e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556d962fe990 0x556d96305220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:10.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.597+0000 7f8ad5e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556d962fe990 0x556d96305220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55026/0 (socket says 172.21.15.67:55026) 2024-03-28T16:05:10.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.597+0000 7f8ad5e00640 1 -- 172.21.15.67:0/3172010833 learned_addr learned my addr 172.21.15.67:0/3172010833 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:05:10.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.597+0000 7f8ad7200640 1 --2- 172.21.15.67:0/3172010833 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556d962fce00 0x556d962fd220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:10.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.597+0000 7f8ad6800640 1 --2- 172.21.15.67:0/3172010833 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556d962fdfd0 0x556d962fe450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:10.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.597+0000 7f8ad5e00640 1 -- 172.21.15.67:0/3172010833 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556d962fdfd0 msgr2=0x556d962fe450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:05:10.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.597+0000 7f8ad5e00640 1 --2- 172.21.15.67:0/3172010833 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556d962fdfd0 0x556d962fe450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:10.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.597+0000 7f8ad5e00640 1 -- 172.21.15.67:0/3172010833 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556d962fce00 msgr2=0x556d962fd220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:05:10.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.597+0000 7f8ad5e00640 1 --2- 172.21.15.67:0/3172010833 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556d962fce00 0x556d962fd220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:10.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.597+0000 7f8ad5e00640 1 -- 172.21.15.67:0/3172010833 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556d9622a080 con 0x556d962fe990 2024-03-28T16:05:10.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.598+0000 7f8acf400640 1 --2- 172.21.15.67:0/1556695976 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f8abc0a2880 0x7f8abc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:10.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.598+0000 7f8ad5e00640 1 --2- 172.21.15.67:0/3172010833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556d962fe990 0x556d96305220 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f8ac400bc80 tx=0x7f8ac400f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1ed1293bf0bde7b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:05:10.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.598+0000 7f8acf400640 1 -- 172.21.15.67:0/1556695976 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f8ac80f1050 con 0x7f8ad81015a0 2024-03-28T16:05:10.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.598+0000 7f8ad7200640 1 --2- 172.21.15.67:0/1556695976 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f8abc0a2880 0x7f8abc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:10.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.598+0000 7f8ac3400640 1 -- 172.21.15.67:0/3172010833 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ac4015070 con 0x556d962fe990 2024-03-28T16:05:10.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.598+0000 7f8ac3400640 1 -- 172.21.15.67:0/3172010833 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8ac40118f0 con 0x556d962fe990 2024-03-28T16:05:10.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.598+0000 7f8ac3400640 1 -- 172.21.15.67:0/3172010833 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ac4010830 con 0x556d962fe990 2024-03-28T16:05:10.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.599+0000 7f8ae004c740 1 -- 172.21.15.67:0/3172010833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556d962fe990 msgr2=0x556d96305220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:10.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.599+0000 7f8ad7200640 1 --2- 172.21.15.67:0/1556695976 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f8abc0a2880 0x7f8abc0a4d40 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x556d962fded0 tx=0x7f8ad0009740 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:05:10.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.599+0000 7f8ae004c740 1 --2- 172.21.15.67:0/3172010833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556d962fe990 0x556d96305220 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f8ac400bc80 tx=0x7f8ac400f730 comp rx=0 tx=0).stop 2024-03-28T16:05:10.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.599+0000 7f8ae004c740 1 -- 172.21.15.67:0/3172010833 shutdown_connections 2024-03-28T16:05:10.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.599+0000 7f8ae004c740 1 --2- 172.21.15.67:0/3172010833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556d962fe990 0x556d96305220 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:10.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.599+0000 7f8ae004c740 1 --2- 172.21.15.67:0/3172010833 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556d962fdfd0 0x556d962fe450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:10.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.599+0000 7f8ae004c740 1 --2- 172.21.15.67:0/3172010833 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556d962fce00 0x556d962fd220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:10.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.599+0000 7f8ae004c740 1 -- 172.21.15.67:0/3172010833 >> 172.21.15.67:0/3172010833 conn(0x556d962f3de0 msgr2=0x556d962f41e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:05:10.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.599+0000 7f8ae004c740 1 -- 172.21.15.67:0/3172010833 shutdown_connections 2024-03-28T16:05:10.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.599+0000 7f8ae004c740 1 -- 172.21.15.67:0/3172010833 wait complete. 2024-03-28T16:05:10.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.600+0000 7f8ae004c740 1 -- start start 2024-03-28T16:05:10.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:05:10 smithi183 ceph-mon[35580]: pgmap v487: 97 pgs: 97 active+clean; 73 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6.0 MiB/s wr, 4 op/s 2024-03-28T16:05:10.602 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:05:10 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/2327319508' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:05:10.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.600+0000 7f8ae004c740 1 Processor -- start 2024-03-28T16:05:10.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.601+0000 7f8ae004c740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556d962fce00 0x556d96307130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:10.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.601+0000 7f8ae004c740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556d962fdfd0 0x556d96307670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:10.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.601+0000 7f8ae004c740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556d962fe990 0x556d9630e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:10.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.601+0000 7f8ae004c740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d963059a0 con 0x556d962fe990 2024-03-28T16:05:10.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.601+0000 7f8ae004c740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d962190d0 con 0x556d962fce00 2024-03-28T16:05:10.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.601+0000 7f8ae004c740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d96305b20 con 0x556d962fdfd0 2024-03-28T16:05:10.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.601+0000 7f8ae004c740 10 client.0 _ll_register_callbacks cb 0x556d9621de80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:05:10.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.601+0000 7f8ad7200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556d962fe990 0x556d9630e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:10.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.601+0000 7f8ad7200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556d962fe990 0x556d9630e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55028/0 (socket says 172.21.15.67:55028) 2024-03-28T16:05:10.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.601+0000 7f8ad7200640 1 -- 172.21.15.67:0/168575278 learned_addr learned my addr 172.21.15.67:0/168575278 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:05:10.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.601+0000 7f8ad6800640 1 --2- 172.21.15.67:0/168575278 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556d962fce00 0x556d96307130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:10.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.602+0000 7f8ad5e00640 1 --2- 172.21.15.67:0/168575278 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556d962fdfd0 0x556d96307670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:10.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.602+0000 7f8ad7200640 1 -- 172.21.15.67:0/168575278 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556d962fdfd0 msgr2=0x556d96307670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:05:10.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.602+0000 7f8ad7200640 1 --2- 172.21.15.67:0/168575278 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556d962fdfd0 0x556d96307670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:10.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.602+0000 7f8ad7200640 1 -- 172.21.15.67:0/168575278 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556d962fce00 msgr2=0x556d96307130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:05:10.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.602+0000 7f8ad7200640 1 --2- 172.21.15.67:0/168575278 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556d962fce00 0x556d96307130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:10.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.602+0000 7f8ad7200640 1 -- 172.21.15.67:0/168575278 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556d95f713f0 con 0x556d962fe990 2024-03-28T16:05:10.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.602+0000 7f8ad7200640 1 --2- 172.21.15.67:0/168575278 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556d962fe990 0x556d9630e5d0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f8ad0005c40 tx=0x7f8ad0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db2f6ac89f5e311 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:05:10.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.602+0000 7f8ab7e00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:05:10.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.603+0000 7f8ab7e00640 1 -- 172.21.15.67:0/168575278 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad001ed40 con 0x556d962fe990 2024-03-28T16:05:10.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.603+0000 7f8ab7e00640 1 -- 172.21.15.67:0/168575278 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f8ad001f420 con 0x556d962fe990 2024-03-28T16:05:10.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.603+0000 7f8ae004c740 1 -- 172.21.15.67:0/168575278 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8ac40118f0 con 0x556d962fe990 2024-03-28T16:05:10.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.603+0000 7f8ab7e00640 1 -- 172.21.15.67:0/168575278 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8ad001f660 con 0x556d962fe990 2024-03-28T16:05:10.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.603+0000 7f8ae004c740 10 client.15519 fetch_fsmap learned FSMap version 19 2024-03-28T16:05:10.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.603+0000 7f8ae004c740 1 -- 172.21.15.67:0/168575278 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x556d9603b7d0 con 0x556d962fe990 2024-03-28T16:05:10.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.603+0000 7f8ab7e00640 1 -- 172.21.15.67:0/168575278 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8ac40118f0 con 0x556d962fe990 2024-03-28T16:05:10.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.604+0000 7f8ab7e00640 1 -- 172.21.15.67:0/168575278 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f8ad000f040 con 0x556d962fe990 2024-03-28T16:05:10.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.604+0000 7f8ae004c740 10 client.15519 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:05:10.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.604+0000 7f8ae004c740 10 client.15519 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:05:10.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.604+0000 7f8ae004c740 20 client.15519 populate_metadata read hostname 'smithi067' 2024-03-28T16:05:10.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.604+0000 7f8ae004c740 1 --2- 172.21.15.67:0/168575278 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x556d96312bc0 0x556d96315060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:10.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.604+0000 7f8ae004c740 4 client.15519 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:05:10.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.604+0000 7f8ae004c740 1 -- 172.21.15.67:0/168575278 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556d9626a800 con 0x556d96312bc0 2024-03-28T16:05:10.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.605+0000 7f8ad6800640 1 --2- 172.21.15.67:0/168575278 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x556d96312bc0 0x556d96315060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:10.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.606+0000 7f8ad6800640 1 --2- 172.21.15.67:0/168575278 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x556d96312bc0 0x556d96315060 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bbce6fa8ecd419a server_cookie=bacb432e0224c217 in_seq=0 out_seq=0 2024-03-28T16:05:10.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.606+0000 7f8ab7e00640 10 client.15519 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:05:10.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.607+0000 7f8ab7e00640 1 -- 172.21.15.67:0/168575278 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x556d9626a800 con 0x556d96312bc0 2024-03-28T16:05:10.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.607+0000 7f8ab7e00640 10 client.15519 handle_command_reply: tid=0 2024-03-28T16:05:10.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.608+0000 7f8ae004c740 1 client.15519 shutdown 2024-03-28T16:05:10.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.608+0000 7f8ae004c740 1 -- 172.21.15.67:0/168575278 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556d962fe990 msgr2=0x556d9630e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:10.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.608+0000 7f8ae004c740 1 --2- 172.21.15.67:0/168575278 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556d962fe990 0x556d9630e5d0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f8ad0005c40 tx=0x7f8ad0004a40 comp rx=0 tx=0).stop 2024-03-28T16:05:10.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.608+0000 7f8ae004c740 1 -- 172.21.15.67:0/168575278 shutdown_connections 2024-03-28T16:05:10.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.608+0000 7f8ae004c740 1 --2- 172.21.15.67:0/168575278 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x556d96312bc0 0x556d96315060 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:10.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.608+0000 7f8ae004c740 1 --2- 172.21.15.67:0/168575278 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556d962fe990 0x556d9630e5d0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:10.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.608+0000 7f8ab7e00640 0 client.15519 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:05:10.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.608+0000 7f8ae004c740 1 --2- 172.21.15.67:0/168575278 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556d962fdfd0 0x556d96307670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:10.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.608+0000 7f8ae004c740 1 --2- 172.21.15.67:0/168575278 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556d962fce00 0x556d96307130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:10.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.608+0000 7f8ae004c740 1 -- 172.21.15.67:0/168575278 >> 172.21.15.67:0/168575278 conn(0x556d962f3de0 msgr2=0x556d962f3160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:05:10.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.608+0000 7f8ae004c740 1 -- 172.21.15.67:0/168575278 shutdown_connections 2024-03-28T16:05:10.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.608+0000 7f8ae004c740 1 -- 172.21.15.67:0/168575278 wait complete. 2024-03-28T16:05:10.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.608+0000 7f8ae004c740 20 client.15519 trim_cache size 0 max 16384 2024-03-28T16:05:10.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.627+0000 7f8ae004c740 1 Processor -- start 2024-03-28T16:05:10.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.627+0000 7f8ae004c740 1 -- start start 2024-03-28T16:05:10.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.628+0000 7f8ae004c740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556d962fce00 0x556d962f7750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:10.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.628+0000 7f8ae004c740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556d962fdfd0 0x556d9630e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:10.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.628+0000 7f8ae004c740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556d962fe990 0x556d96301110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:10.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.628+0000 7f8ae004c740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad81a45c0 con 0x556d962fe990 2024-03-28T16:05:10.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.628+0000 7f8ae004c740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad81a6b80 con 0x556d962fce00 2024-03-28T16:05:10.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.628+0000 7f8ae004c740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad81a4bc0 con 0x556d962fdfd0 2024-03-28T16:05:10.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.628+0000 7f8ad7200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556d962fe990 0x556d96301110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:10.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.628+0000 7f8ad7200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556d962fe990 0x556d96301110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55038/0 (socket says 172.21.15.67:55038) 2024-03-28T16:05:10.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.628+0000 7f8ad7200640 1 -- 172.21.15.67:0/3680089827 learned_addr learned my addr 172.21.15.67:0/3680089827 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:05:10.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.628+0000 7f8ad6800640 1 --2- 172.21.15.67:0/3680089827 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556d962fce00 0x556d962f7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:10.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.628+0000 7f8ad5e00640 1 --2- 172.21.15.67:0/3680089827 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556d962fdfd0 0x556d9630e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:10.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.629+0000 7f8ad7200640 1 -- 172.21.15.67:0/3680089827 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556d962fdfd0 msgr2=0x556d9630e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:05:10.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.629+0000 7f8ad7200640 1 --2- 172.21.15.67:0/3680089827 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556d962fdfd0 0x556d9630e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:10.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.629+0000 7f8ad7200640 1 -- 172.21.15.67:0/3680089827 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556d962fce00 msgr2=0x556d962f7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:05:10.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.629+0000 7f8ad7200640 1 --2- 172.21.15.67:0/3680089827 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556d962fce00 0x556d962f7750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:10.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.629+0000 7f8ad7200640 1 -- 172.21.15.67:0/3680089827 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556d962f4830 con 0x556d962fe990 2024-03-28T16:05:10.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.629+0000 7f8ad6800640 1 --2- 172.21.15.67:0/3680089827 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556d962fce00 0x556d962f7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T16:05:10.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.629+0000 7f8ad7200640 1 --2- 172.21.15.67:0/3680089827 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556d962fe990 0x556d96301110 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f8ad000a010 tx=0x7f8ad0005fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b67901f22afc6ff6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:05:10.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.629+0000 7f8ac2a00640 1 -- 172.21.15.67:0/3680089827 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad003f070 con 0x556d962fe990 2024-03-28T16:05:10.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.629+0000 7f8ac2a00640 1 -- 172.21.15.67:0/3680089827 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8ad001e780 con 0x556d962fe990 2024-03-28T16:05:10.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.629+0000 7f8ac2a00640 1 -- 172.21.15.67:0/3680089827 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad0036070 con 0x556d962fe990 2024-03-28T16:05:10.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.630+0000 7f8ae004c740 1 -- 172.21.15.67:0/3680089827 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556d962fe990 msgr2=0x556d96301110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:10.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.630+0000 7f8ae004c740 1 --2- 172.21.15.67:0/3680089827 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556d962fe990 0x556d96301110 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f8ad000a010 tx=0x7f8ad0005fc0 comp rx=0 tx=0).stop 2024-03-28T16:05:10.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.630+0000 7f8ae004c740 1 -- 172.21.15.67:0/3680089827 shutdown_connections 2024-03-28T16:05:10.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.631+0000 7f8ae004c740 1 --2- 172.21.15.67:0/3680089827 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556d962fe990 0x556d96301110 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:10.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.631+0000 7f8ae004c740 1 --2- 172.21.15.67:0/3680089827 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556d962fdfd0 0x556d9630e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:10.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.631+0000 7f8ae004c740 1 --2- 172.21.15.67:0/3680089827 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556d962fce00 0x556d962f7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:10.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.631+0000 7f8ae004c740 1 -- 172.21.15.67:0/3680089827 >> 172.21.15.67:0/3680089827 conn(0x556d962f3de0 msgr2=0x556d962f7040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:05:10.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.631+0000 7f8ae004c740 1 -- 172.21.15.67:0/3680089827 shutdown_connections 2024-03-28T16:05:10.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.631+0000 7f8ae004c740 1 -- 172.21.15.67:0/3680089827 wait complete. 2024-03-28T16:05:10.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.631+0000 7f8ae004c740 1 -- start start 2024-03-28T16:05:10.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.631+0000 7f8ae004c740 1 Processor -- start 2024-03-28T16:05:10.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.632+0000 7f8ae004c740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556d962fce00 0x556d9630a420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:10.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.632+0000 7f8ae004c740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556d962fdfd0 0x556d9630a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:10.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.632+0000 7f8ae004c740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556d962fe990 0x556d9630d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:10.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.632+0000 7f8ae004c740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad81a5200 con 0x556d962fce00 2024-03-28T16:05:10.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.632+0000 7f8ae004c740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad81a6520 con 0x556d962fe990 2024-03-28T16:05:10.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.632+0000 7f8ae004c740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad81a5860 con 0x556d962fdfd0 2024-03-28T16:05:10.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.632+0000 7f8ae004c740 10 client.0 _ll_register_callbacks cb 0x7f8ad00103d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:05:10.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.632+0000 7f8ad6800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556d962fce00 0x556d9630a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:10.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.632+0000 7f8ad6800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556d962fce00 0x556d9630a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55048/0 (socket says 172.21.15.67:55048) 2024-03-28T16:05:10.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.632+0000 7f8ad6800640 1 -- 172.21.15.67:0/842149705 learned_addr learned my addr 172.21.15.67:0/842149705 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:05:10.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.633+0000 7f8ad7200640 1 --2- 172.21.15.67:0/842149705 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556d962fe990 0x556d9630d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:10.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.633+0000 7f8ad5e00640 1 --2- 172.21.15.67:0/842149705 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556d962fdfd0 0x556d9630a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:10.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.633+0000 7f8ad6800640 1 -- 172.21.15.67:0/842149705 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556d962fdfd0 msgr2=0x556d9630a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:05:10.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.633+0000 7f8ad6800640 1 --2- 172.21.15.67:0/842149705 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556d962fdfd0 0x556d9630a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:10.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.633+0000 7f8ad6800640 1 -- 172.21.15.67:0/842149705 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556d962fe990 msgr2=0x556d9630d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:05:10.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.633+0000 7f8ad6800640 1 --2- 172.21.15.67:0/842149705 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556d962fe990 0x556d9630d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:10.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.633+0000 7f8ad6800640 1 -- 172.21.15.67:0/842149705 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556d9630df70 con 0x556d962fce00 2024-03-28T16:05:10.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.633+0000 7f8ad7200640 1 --2- 172.21.15.67:0/842149705 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556d962fe990 0x556d9630d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:05:10.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.633+0000 7f8ad5e00640 1 --2- 172.21.15.67:0/842149705 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556d962fdfd0 0x556d9630a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:05:10.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.633+0000 7f8ad6800640 1 --2- 172.21.15.67:0/842149705 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556d962fce00 0x556d9630a420 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f8ac80ba3d0 tx=0x7f8ac80a8040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8ba258287046014 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:05:10.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.633+0000 7f8ab7e00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:05:10.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.633+0000 7f8ab7e00640 1 -- 172.21.15.67:0/842149705 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ac80bd7d0 con 0x556d962fce00 2024-03-28T16:05:10.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.633+0000 7f8ab7e00640 1 -- 172.21.15.67:0/842149705 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f8ac80bbd00 con 0x556d962fce00 2024-03-28T16:05:10.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.633+0000 7f8ae004c740 1 -- 172.21.15.67:0/842149705 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8ad001e780 con 0x556d962fce00 2024-03-28T16:05:10.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.633+0000 7f8ab7e00640 1 -- 172.21.15.67:0/842149705 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8ac80bd360 con 0x556d962fce00 2024-03-28T16:05:10.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.633+0000 7f8ae004c740 10 client.15525 fetch_fsmap learned FSMap version 19 2024-03-28T16:05:10.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.634+0000 7f8ae004c740 1 -- 172.21.15.67:0/842149705 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x556d9636ed20 con 0x556d962fce00 2024-03-28T16:05:10.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.634+0000 7f8ab7e00640 1 -- 172.21.15.67:0/842149705 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8ad001e780 con 0x556d962fce00 2024-03-28T16:05:10.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.634+0000 7f8ab7e00640 1 -- 172.21.15.67:0/842149705 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f8ac80ba070 con 0x556d962fce00 2024-03-28T16:05:10.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.634+0000 7f8ae004c740 10 client.15525 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:05:10.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.634+0000 7f8ae004c740 10 client.15525 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:05:10.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.634+0000 7f8ae004c740 20 client.15525 populate_metadata read hostname 'smithi067' 2024-03-28T16:05:10.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.634+0000 7f8ae004c740 1 --2- 172.21.15.67:0/842149705 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x556d9636f6a0 0x556d96371ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:10.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.634+0000 7f8ae004c740 4 client.15525 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T16:05:10.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.634+0000 7f8ae004c740 1 -- 172.21.15.67:0/842149705 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x556d96264b50 con 0x556d9636f6a0 2024-03-28T16:05:10.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.635+0000 7f8ad5e00640 1 --2- 172.21.15.67:0/842149705 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x556d9636f6a0 0x556d96371ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:10.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.636+0000 7f8ad5e00640 1 --2- 172.21.15.67:0/842149705 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x556d9636f6a0 0x556d96371ac0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6b8c01fcbb7e7f0c server_cookie=3d540a9ebbbb312b in_seq=0 out_seq=0 2024-03-28T16:05:10.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.636+0000 7f8ab7e00640 10 client.15525 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:05:10.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.636+0000 7f8ab7e00640 1 -- 172.21.15.67:0/842149705 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x556d96264b50 con 0x556d9636f6a0 2024-03-28T16:05:10.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.636+0000 7f8ab7e00640 10 client.15525 handle_command_reply: tid=0 2024-03-28T16:05:10.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.636+0000 7f8ae004c740 1 client.15525 shutdown 2024-03-28T16:05:10.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.636+0000 7f8ae004c740 1 -- 172.21.15.67:0/842149705 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556d962fce00 msgr2=0x556d9630a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:10.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.636+0000 7f8ae004c740 1 --2- 172.21.15.67:0/842149705 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556d962fce00 0x556d9630a420 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f8ac80ba3d0 tx=0x7f8ac80a8040 comp rx=0 tx=0).stop 2024-03-28T16:05:10.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.637+0000 7f8ae004c740 1 -- 172.21.15.67:0/842149705 shutdown_connections 2024-03-28T16:05:10.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.637+0000 7f8ae004c740 1 --2- 172.21.15.67:0/842149705 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x556d9636f6a0 0x556d96371ac0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:10.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.637+0000 7f8ae004c740 1 --2- 172.21.15.67:0/842149705 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556d962fe990 0x556d9630d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:10.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.637+0000 7f8ae004c740 1 --2- 172.21.15.67:0/842149705 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556d962fdfd0 0x556d9630a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:10.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.637+0000 7f8ae004c740 1 --2- 172.21.15.67:0/842149705 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556d962fce00 0x556d9630a420 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:10.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.637+0000 7f8ae004c740 1 -- 172.21.15.67:0/842149705 >> 172.21.15.67:0/842149705 conn(0x556d962f3de0 msgr2=0x556d962f78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:05:10.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.637+0000 7f8ab7e00640 0 client.15525 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:05:10.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.637+0000 7f8ae004c740 1 -- 172.21.15.67:0/842149705 shutdown_connections 2024-03-28T16:05:10.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.637+0000 7f8ae004c740 1 -- 172.21.15.67:0/842149705 wait complete. 2024-03-28T16:05:10.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.637+0000 7f8ae004c740 20 client.15525 trim_cache size 0 max 16384 2024-03-28T16:05:10.640 INFO:teuthology.orchestra.run.smithi067.stdout:[] 2024-03-28T16:05:10.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.638+0000 7f8adca00640 1 -- 172.21.15.67:0/1556695976 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f8abc0a2880 msgr2=0x7f8abc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:10.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.638+0000 7f8adca00640 1 --2- 172.21.15.67:0/1556695976 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f8abc0a2880 0x7f8abc0a4d40 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x556d962fded0 tx=0x7f8ad0009740 comp rx=0 tx=0).stop 2024-03-28T16:05:10.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.638+0000 7f8adca00640 1 -- 172.21.15.67:0/1556695976 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ad81015a0 msgr2=0x7f8ad81977a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:10.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.638+0000 7f8adca00640 1 --2- 172.21.15.67:0/1556695976 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ad81015a0 0x7f8ad81977a0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f8ac8000c00 tx=0x7f8ac8004270 comp rx=0 tx=0).stop 2024-03-28T16:05:10.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.638+0000 7f8adca00640 1 -- 172.21.15.67:0/1556695976 shutdown_connections 2024-03-28T16:05:10.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.638+0000 7f8adca00640 1 --2- 172.21.15.67:0/1556695976 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f8abc0a2880 0x7f8abc0a4d40 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:10.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.638+0000 7f8adca00640 1 --2- 172.21.15.67:0/1556695976 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8ad8103150 0x7f8ad819ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:10.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.638+0000 7f8adca00640 1 --2- 172.21.15.67:0/1556695976 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ad8102790 0x7f8ad8197ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:10.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.638+0000 7f8adca00640 1 --2- 172.21.15.67:0/1556695976 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ad81015a0 0x7f8ad81977a0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:10.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.638+0000 7f8adca00640 1 -- 172.21.15.67:0/1556695976 >> 172.21.15.67:0/1556695976 conn(0x7f8ad80fcb10 msgr2=0x7f8ad8105ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:05:10.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.638+0000 7f8adca00640 1 -- 172.21.15.67:0/1556695976 shutdown_connections 2024-03-28T16:05:10.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:10.638+0000 7f8adca00640 1 -- 172.21.15.67:0/1556695976 wait complete. 2024-03-28T16:05:10.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:05:10 smithi150 ceph-mon[34825]: pgmap v487: 97 pgs: 97 active+clean; 73 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6.0 MiB/s wr, 4 op/s 2024-03-28T16:05:10.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:05:10 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/2327319508' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:05:11.657 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T16:05:11.657 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T16:05:11.657 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T16:05:11.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.906+0000 7efe65600640 1 -- 172.21.15.67:0/4090652980 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe60103260 msgr2=0x7efe60109af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:11.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.906+0000 7efe65600640 1 --2- 172.21.15.67:0/4090652980 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe60103260 0x7efe60109af0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7efe58009ca0 tx=0x7efe5802c0f0 comp rx=0 tx=0).stop 2024-03-28T16:05:11.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.906+0000 7efe65600640 1 -- 172.21.15.67:0/4090652980 shutdown_connections 2024-03-28T16:05:11.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.906+0000 7efe65600640 1 --2- 172.21.15.67:0/4090652980 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe60103260 0x7efe60109af0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:11.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.906+0000 7efe65600640 1 --2- 172.21.15.67:0/4090652980 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efe601028a0 0x7efe60102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:11.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.906+0000 7efe65600640 1 --2- 172.21.15.67:0/4090652980 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe601016b0 0x7efe60101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:11.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.906+0000 7efe65600640 1 -- 172.21.15.67:0/4090652980 >> 172.21.15.67:0/4090652980 conn(0x7efe600fcc40 msgr2=0x7efe600ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:05:11.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.906+0000 7efe65600640 1 -- 172.21.15.67:0/4090652980 shutdown_connections 2024-03-28T16:05:11.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.906+0000 7efe65600640 1 -- 172.21.15.67:0/4090652980 wait complete. 2024-03-28T16:05:11.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.908+0000 7efe65600640 1 Processor -- start 2024-03-28T16:05:11.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.908+0000 7efe65600640 1 -- start start 2024-03-28T16:05:11.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.908+0000 7efe65600640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe601016b0 0x7efe60075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:11.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.908+0000 7efe65600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efe601028a0 0x7efe60073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:11.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.908+0000 7efe65600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe60103260 0x7efe60073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:11.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.908+0000 7efe65600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe6010d1c0 con 0x7efe60103260 2024-03-28T16:05:11.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.908+0000 7efe65600640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe6010d040 con 0x7efe601016b0 2024-03-28T16:05:11.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.908+0000 7efe65600640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe6010d340 con 0x7efe601028a0 2024-03-28T16:05:11.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.909+0000 7efe5f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe60103260 0x7efe60073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:11.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.909+0000 7efe5f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe60103260 0x7efe60073fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55080/0 (socket says 172.21.15.67:55080) 2024-03-28T16:05:11.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.909+0000 7efe5f400640 1 -- 172.21.15.67:0/2512242534 learned_addr learned my addr 172.21.15.67:0/2512242534 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:05:11.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.909+0000 7efe57e00640 1 --2- 172.21.15.67:0/2512242534 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efe601028a0 0x7efe60073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:11.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.909+0000 7efe5f400640 1 -- 172.21.15.67:0/2512242534 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efe601028a0 msgr2=0x7efe60073a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:11.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.909+0000 7efe5f400640 1 --2- 172.21.15.67:0/2512242534 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efe601028a0 0x7efe60073a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:11.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.909+0000 7efe5ea00640 1 --2- 172.21.15.67:0/2512242534 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe601016b0 0x7efe60075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:11.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.909+0000 7efe5f400640 1 -- 172.21.15.67:0/2512242534 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe601016b0 msgr2=0x7efe60075450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:11.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.909+0000 7efe5f400640 1 --2- 172.21.15.67:0/2512242534 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe601016b0 0x7efe60075450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:11.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.909+0000 7efe5f400640 1 -- 172.21.15.67:0/2512242534 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe60074860 con 0x7efe60103260 2024-03-28T16:05:11.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.909+0000 7efe57e00640 1 --2- 172.21.15.67:0/2512242534 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efe601028a0 0x7efe60073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T16:05:11.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.910+0000 7efe5f400640 1 --2- 172.21.15.67:0/2512242534 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe60103260 0x7efe60073fd0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7efe5802c700 tx=0x7efe580328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:05:11.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.910+0000 7efe57400640 1 -- 172.21.15.67:0/2512242534 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe58041020 con 0x7efe60103260 2024-03-28T16:05:11.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.910+0000 7efe57400640 1 -- 172.21.15.67:0/2512242534 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7efe58003e20 con 0x7efe60103260 2024-03-28T16:05:11.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.910+0000 7efe57400640 1 -- 172.21.15.67:0/2512242534 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe58039040 con 0x7efe60103260 2024-03-28T16:05:11.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.910+0000 7efe65600640 1 -- 172.21.15.67:0/2512242534 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe601a1930 con 0x7efe60103260 2024-03-28T16:05:11.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.910+0000 7efe65600640 1 -- 172.21.15.67:0/2512242534 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe601a1e80 con 0x7efe60103260 2024-03-28T16:05:11.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.915+0000 7efe68221740 1 Processor -- start 2024-03-28T16:05:11.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.915+0000 7efe57400640 1 -- 172.21.15.67:0/2512242534 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7efe58044020 con 0x7efe60103260 2024-03-28T16:05:11.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.915+0000 7efe68221740 1 -- start start 2024-03-28T16:05:11.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.916+0000 7efe68221740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5640bc8268b0 0x5640bc826cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:11.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.916+0000 7efe68221740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5640bc8272a0 0x5640bc82bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:11.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.916+0000 7efe68221740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5640bc82c410 0x5640bc82e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:11.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.916+0000 7efe68221740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640bc783eb0 con 0x5640bc82c410 2024-03-28T16:05:11.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.916+0000 7efe68221740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640bc72ca60 con 0x5640bc8268b0 2024-03-28T16:05:11.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.916+0000 7efe68221740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640bc77ff50 con 0x5640bc8272a0 2024-03-28T16:05:11.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.916+0000 7efe57e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5640bc82c410 0x5640bc82e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:11.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.916+0000 7efe57e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5640bc82c410 0x5640bc82e8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55086/0 (socket says 172.21.15.67:55086) 2024-03-28T16:05:11.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.916+0000 7efe5ea00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5640bc8272a0 0x5640bc82bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:11.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.916+0000 7efe57e00640 1 -- 172.21.15.67:0/1443344281 learned_addr learned my addr 172.21.15.67:0/1443344281 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:05:11.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.916+0000 7efe57e00640 1 --2- 172.21.15.67:0/1443344281 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5640bc8268b0 0x5640bc826cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:11.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.917+0000 7efe57e00640 1 -- 172.21.15.67:0/1443344281 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5640bc8272a0 msgr2=0x5640bc82bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:05:11.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.917+0000 7efe57e00640 1 --2- 172.21.15.67:0/1443344281 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5640bc8272a0 0x5640bc82bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:11.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.917+0000 7efe57e00640 1 -- 172.21.15.67:0/1443344281 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5640bc8268b0 msgr2=0x5640bc826cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:05:11.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.917+0000 7efe57e00640 1 --2- 172.21.15.67:0/1443344281 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5640bc8268b0 0x5640bc826cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:11.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.917+0000 7efe57e00640 1 -- 172.21.15.67:0/1443344281 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5640bc5607e0 con 0x5640bc82c410 2024-03-28T16:05:11.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.917+0000 7efe57400640 1 --2- 172.21.15.67:0/2512242534 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7efe440a2800 0x7efe440a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:11.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.917+0000 7efe57400640 1 -- 172.21.15.67:0/2512242534 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7efe580f0e50 con 0x7efe60103260 2024-03-28T16:05:11.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.917+0000 7efe57e00640 1 --2- 172.21.15.67:0/1443344281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5640bc82c410 0x5640bc82e8a0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x5640bc827b00 tx=0x7efe4800c570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97ba65fc3e1417ab server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:05:11.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.917+0000 7efe5ea00640 1 --2- 172.21.15.67:0/2512242534 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7efe440a2800 0x7efe440a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:11.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.918+0000 7efe4f400640 1 -- 172.21.15.67:0/1443344281 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe480142d0 con 0x5640bc82c410 2024-03-28T16:05:11.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.918+0000 7efe4f400640 1 -- 172.21.15.67:0/1443344281 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7efe4800f820 con 0x5640bc82c410 2024-03-28T16:05:11.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.918+0000 7efe4f400640 1 -- 172.21.15.67:0/1443344281 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe48003850 con 0x5640bc82c410 2024-03-28T16:05:11.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.919+0000 7efe68221740 1 -- 172.21.15.67:0/1443344281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5640bc82c410 msgr2=0x5640bc82e8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:11.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.919+0000 7efe68221740 1 --2- 172.21.15.67:0/1443344281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5640bc82c410 0x5640bc82e8a0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x5640bc827b00 tx=0x7efe4800c570 comp rx=0 tx=0).stop 2024-03-28T16:05:11.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.919+0000 7efe5ea00640 1 --2- 172.21.15.67:0/2512242534 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7efe440a2800 0x7efe440a4cc0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x5640bc8276a0 tx=0x7efe50005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:05:11.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.919+0000 7efe68221740 1 -- 172.21.15.67:0/1443344281 shutdown_connections 2024-03-28T16:05:11.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.919+0000 7efe68221740 1 --2- 172.21.15.67:0/1443344281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5640bc82c410 0x5640bc82e8a0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:11.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.919+0000 7efe68221740 1 --2- 172.21.15.67:0/1443344281 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5640bc8272a0 0x5640bc82bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:11.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.919+0000 7efe68221740 1 --2- 172.21.15.67:0/1443344281 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5640bc8268b0 0x5640bc826cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:11.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.919+0000 7efe68221740 1 -- 172.21.15.67:0/1443344281 >> 172.21.15.67:0/1443344281 conn(0x5640bc821e20 msgr2=0x5640bc822220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:05:11.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.919+0000 7efe68221740 1 -- 172.21.15.67:0/1443344281 shutdown_connections 2024-03-28T16:05:11.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.919+0000 7efe68221740 1 -- 172.21.15.67:0/1443344281 wait complete. 2024-03-28T16:05:11.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.920+0000 7efe68221740 1 -- start start 2024-03-28T16:05:11.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.920+0000 7efe68221740 1 Processor -- start 2024-03-28T16:05:11.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.920+0000 7efe68221740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5640bc8268b0 0x5640bc830aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:11.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.920+0000 7efe68221740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5640bc8272a0 0x5640bc830fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:11.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.920+0000 7efe68221740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5640bc82c410 0x5640bc838020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:11.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.920+0000 7efe68221740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640bc82f360 con 0x5640bc8272a0 2024-03-28T16:05:11.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.920+0000 7efe68221740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640bc73feb0 con 0x5640bc8268b0 2024-03-28T16:05:11.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.920+0000 7efe68221740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640bc82f4e0 con 0x5640bc82c410 2024-03-28T16:05:11.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.921+0000 7efe68221740 10 client.0 _ll_register_callbacks cb 0x5640bc734150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:05:11.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.921+0000 7efe57e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5640bc8272a0 0x5640bc830fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:11.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.921+0000 7efe57e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5640bc8272a0 0x5640bc830fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55090/0 (socket says 172.21.15.67:55090) 2024-03-28T16:05:11.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.921+0000 7efe57e00640 1 -- 172.21.15.67:0/1341580229 learned_addr learned my addr 172.21.15.67:0/1341580229 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:05:11.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.921+0000 7efe5ea00640 1 --2- 172.21.15.67:0/1341580229 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5640bc8268b0 0x5640bc830aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:11.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.921+0000 7efe5f400640 1 --2- 172.21.15.67:0/1341580229 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5640bc82c410 0x5640bc838020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:11.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.921+0000 7efe57e00640 1 -- 172.21.15.67:0/1341580229 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5640bc82c410 msgr2=0x5640bc838020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:05:11.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.921+0000 7efe57e00640 1 --2- 172.21.15.67:0/1341580229 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5640bc82c410 0x5640bc838020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:11.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.921+0000 7efe57e00640 1 -- 172.21.15.67:0/1341580229 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5640bc8268b0 msgr2=0x5640bc830aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:05:11.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.921+0000 7efe57e00640 1 --2- 172.21.15.67:0/1341580229 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5640bc8268b0 0x5640bc830aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:11.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.922+0000 7efe57e00640 1 -- 172.21.15.67:0/1341580229 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5640bc5c0f70 con 0x5640bc8272a0 2024-03-28T16:05:11.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.922+0000 7efe57e00640 1 --2- 172.21.15.67:0/1341580229 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5640bc8272a0 0x5640bc830fe0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7efe48014300 tx=0x7efe4800ffa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89b4965f37286d41 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:05:11.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.922+0000 7efe43e00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:05:11.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.922+0000 7efe43e00640 1 -- 172.21.15.67:0/1341580229 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe480146d0 con 0x5640bc8272a0 2024-03-28T16:05:11.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.922+0000 7efe43e00640 1 -- 172.21.15.67:0/1341580229 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7efe48004030 con 0x5640bc8272a0 2024-03-28T16:05:11.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.922+0000 7efe68221740 1 -- 172.21.15.67:0/1341580229 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efe4800f820 con 0x5640bc8272a0 2024-03-28T16:05:11.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.923+0000 7efe43e00640 1 -- 172.21.15.67:0/1341580229 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7efe48004270 con 0x5640bc8272a0 2024-03-28T16:05:11.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.923+0000 7efe68221740 10 client.15537 fetch_fsmap learned FSMap version 19 2024-03-28T16:05:11.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.923+0000 7efe68221740 1 -- 172.21.15.67:0/1341580229 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5640bc458170 con 0x5640bc8272a0 2024-03-28T16:05:11.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.923+0000 7efe43e00640 1 -- 172.21.15.67:0/1341580229 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7efe4800f820 con 0x5640bc8272a0 2024-03-28T16:05:11.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.923+0000 7efe43e00640 1 -- 172.21.15.67:0/1341580229 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7efe48012450 con 0x5640bc8272a0 2024-03-28T16:05:11.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.923+0000 7efe68221740 10 client.15537 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:05:11.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.923+0000 7efe68221740 10 client.15537 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:05:11.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.923+0000 7efe68221740 20 client.15537 populate_metadata read hostname 'smithi067' 2024-03-28T16:05:11.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.923+0000 7efe68221740 1 --2- 172.21.15.67:0/1341580229 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5640bc83c570 0x5640bc83e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:11.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.924+0000 7efe68221740 4 client.15537 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:05:11.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.924+0000 7efe68221740 1 -- 172.21.15.67:0/1341580229 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5640bc7913f0 con 0x5640bc83c570 2024-03-28T16:05:11.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.924+0000 7efe5ea00640 1 --2- 172.21.15.67:0/1341580229 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5640bc83c570 0x5640bc83e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:11.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.926+0000 7efe5ea00640 1 --2- 172.21.15.67:0/1341580229 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5640bc83c570 0x5640bc83e960 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4a366254c80afca5 server_cookie=fa178a88d6b5f945 in_seq=0 out_seq=0 2024-03-28T16:05:11.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.926+0000 7efe43e00640 10 client.15537 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:05:11.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.928+0000 7efe43e00640 1 -- 172.21.15.67:0/1341580229 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5640bc7913f0 con 0x5640bc83c570 2024-03-28T16:05:11.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.928+0000 7efe43e00640 10 client.15537 handle_command_reply: tid=0 2024-03-28T16:05:11.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.928+0000 7efe68221740 1 client.15537 shutdown 2024-03-28T16:05:11.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.929+0000 7efe68221740 1 -- 172.21.15.67:0/1341580229 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5640bc8272a0 msgr2=0x5640bc830fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:11.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.929+0000 7efe68221740 1 --2- 172.21.15.67:0/1341580229 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5640bc8272a0 0x5640bc830fe0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7efe48014300 tx=0x7efe4800ffa0 comp rx=0 tx=0).stop 2024-03-28T16:05:11.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.929+0000 7efe68221740 1 -- 172.21.15.67:0/1341580229 shutdown_connections 2024-03-28T16:05:11.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.929+0000 7efe68221740 1 --2- 172.21.15.67:0/1341580229 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5640bc83c570 0x5640bc83e960 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:11.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.929+0000 7efe68221740 1 --2- 172.21.15.67:0/1341580229 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5640bc82c410 0x5640bc838020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:11.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.929+0000 7efe68221740 1 --2- 172.21.15.67:0/1341580229 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5640bc8272a0 0x5640bc830fe0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:11.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.929+0000 7efe68221740 1 --2- 172.21.15.67:0/1341580229 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5640bc8268b0 0x5640bc830aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:11.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.929+0000 7efe43e00640 0 client.15537 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:05:11.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.929+0000 7efe68221740 1 -- 172.21.15.67:0/1341580229 >> 172.21.15.67:0/1341580229 conn(0x5640bc821e20 msgr2=0x5640bc818830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:05:11.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.930+0000 7efe68221740 1 -- 172.21.15.67:0/1341580229 shutdown_connections 2024-03-28T16:05:11.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.930+0000 7efe68221740 1 -- 172.21.15.67:0/1341580229 wait complete. 2024-03-28T16:05:11.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.930+0000 7efe68221740 20 client.15537 trim_cache size 0 max 16384 2024-03-28T16:05:11.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.957+0000 7efe68221740 1 Processor -- start 2024-03-28T16:05:11.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.957+0000 7efe68221740 1 -- start start 2024-03-28T16:05:11.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.958+0000 7efe68221740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5640bc8268b0 0x5640bc827d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:11.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.958+0000 7efe68221740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5640bc82c410 0x5640bc837fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:11.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.958+0000 7efe68221740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5640bc828280 0x5640bc828700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:11.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.958+0000 7efe68221740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe601a4790 con 0x5640bc8268b0 2024-03-28T16:05:11.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.958+0000 7efe68221740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe601a6d30 con 0x5640bc82c410 2024-03-28T16:05:11.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.958+0000 7efe68221740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe601a4d90 con 0x5640bc828280 2024-03-28T16:05:11.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.958+0000 7efe5ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5640bc8268b0 0x5640bc827d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:11.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.958+0000 7efe5ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5640bc8268b0 0x5640bc827d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55098/0 (socket says 172.21.15.67:55098) 2024-03-28T16:05:11.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.958+0000 7efe5ea00640 1 -- 172.21.15.67:0/1348573006 learned_addr learned my addr 172.21.15.67:0/1348573006 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:05:11.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.958+0000 7efe5f400640 1 --2- 172.21.15.67:0/1348573006 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5640bc828280 0x5640bc828700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:11.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.959+0000 7efe57e00640 1 --2- 172.21.15.67:0/1348573006 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5640bc82c410 0x5640bc837fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:11.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.959+0000 7efe5ea00640 1 -- 172.21.15.67:0/1348573006 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5640bc828280 msgr2=0x5640bc828700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:05:11.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.959+0000 7efe5ea00640 1 --2- 172.21.15.67:0/1348573006 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5640bc828280 0x5640bc828700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:11.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.959+0000 7efe5ea00640 1 -- 172.21.15.67:0/1348573006 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5640bc82c410 msgr2=0x5640bc837fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:05:11.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.959+0000 7efe5ea00640 1 --2- 172.21.15.67:0/1348573006 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5640bc82c410 0x5640bc837fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:11.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.959+0000 7efe5ea00640 1 -- 172.21.15.67:0/1348573006 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5640bc82b750 con 0x5640bc8268b0 2024-03-28T16:05:11.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.959+0000 7efe5ea00640 1 --2- 172.21.15.67:0/1348573006 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5640bc8268b0 0x5640bc827d40 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x5640bc822d90 tx=0x7efe50016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e815b7b83350524 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:05:11.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.959+0000 7efe4ea00640 1 -- 172.21.15.67:0/1348573006 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe5002ae40 con 0x5640bc8268b0 2024-03-28T16:05:11.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.960+0000 7efe4ea00640 1 -- 172.21.15.67:0/1348573006 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7efe5002a030 con 0x5640bc8268b0 2024-03-28T16:05:11.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.960+0000 7efe4ea00640 1 -- 172.21.15.67:0/1348573006 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe50029980 con 0x5640bc8268b0 2024-03-28T16:05:11.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.960+0000 7efe68221740 1 -- 172.21.15.67:0/1348573006 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5640bc8268b0 msgr2=0x5640bc827d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:11.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.960+0000 7efe68221740 1 --2- 172.21.15.67:0/1348573006 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5640bc8268b0 0x5640bc827d40 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x5640bc822d90 tx=0x7efe50016000 comp rx=0 tx=0).stop 2024-03-28T16:05:11.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.960+0000 7efe68221740 1 -- 172.21.15.67:0/1348573006 shutdown_connections 2024-03-28T16:05:11.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.960+0000 7efe68221740 1 --2- 172.21.15.67:0/1348573006 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5640bc828280 0x5640bc828700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:11.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.960+0000 7efe68221740 1 --2- 172.21.15.67:0/1348573006 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5640bc82c410 0x5640bc837fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:11.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.960+0000 7efe68221740 1 --2- 172.21.15.67:0/1348573006 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5640bc8268b0 0x5640bc827d40 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:11.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.960+0000 7efe68221740 1 -- 172.21.15.67:0/1348573006 >> 172.21.15.67:0/1348573006 conn(0x5640bc821e20 msgr2=0x5640bc825910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:05:11.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.961+0000 7efe68221740 1 -- 172.21.15.67:0/1348573006 shutdown_connections 2024-03-28T16:05:11.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.961+0000 7efe68221740 1 -- 172.21.15.67:0/1348573006 wait complete. 2024-03-28T16:05:11.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.961+0000 7efe68221740 1 -- start start 2024-03-28T16:05:11.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.961+0000 7efe68221740 1 Processor -- start 2024-03-28T16:05:11.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.962+0000 7efe68221740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5640bc8268b0 0x5640bc8369e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:11.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.962+0000 7efe68221740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5640bc828280 0x5640bc836f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:11.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.962+0000 7efe68221740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5640bc82c410 0x5640bc837460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:11.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.962+0000 7efe68221740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe601a53b0 con 0x5640bc8268b0 2024-03-28T16:05:11.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.962+0000 7efe68221740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe601a66d0 con 0x5640bc82c410 2024-03-28T16:05:11.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.962+0000 7efe68221740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe601a5a10 con 0x5640bc828280 2024-03-28T16:05:11.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.962+0000 7efe68221740 10 client.0 _ll_register_callbacks cb 0x7efe48014470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:05:11.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.962+0000 7efe5ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5640bc8268b0 0x5640bc8369e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:11.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.962+0000 7efe5ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5640bc8268b0 0x5640bc8369e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55112/0 (socket says 172.21.15.67:55112) 2024-03-28T16:05:11.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.962+0000 7efe5ea00640 1 -- 172.21.15.67:0/313750071 learned_addr learned my addr 172.21.15.67:0/313750071 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:05:11.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.963+0000 7efe5f400640 1 --2- 172.21.15.67:0/313750071 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5640bc82c410 0x5640bc837460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:11.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.963+0000 7efe57e00640 1 --2- 172.21.15.67:0/313750071 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5640bc828280 0x5640bc836f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:11.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.963+0000 7efe5ea00640 1 -- 172.21.15.67:0/313750071 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5640bc828280 msgr2=0x5640bc836f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:05:11.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.963+0000 7efe5ea00640 1 --2- 172.21.15.67:0/313750071 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5640bc828280 0x5640bc836f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:11.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.963+0000 7efe5ea00640 1 -- 172.21.15.67:0/313750071 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5640bc82c410 msgr2=0x5640bc837460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:05:11.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.963+0000 7efe5ea00640 1 --2- 172.21.15.67:0/313750071 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5640bc82c410 0x5640bc837460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:11.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.963+0000 7efe5ea00640 1 -- 172.21.15.67:0/313750071 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5640bc897f60 con 0x5640bc8268b0 2024-03-28T16:05:11.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.963+0000 7efe5ea00640 1 --2- 172.21.15.67:0/313750071 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5640bc8268b0 0x5640bc8369e0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7efe5002d040 tx=0x7efe5002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e331a0479160a659 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:05:11.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.963+0000 7efe43e00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:05:11.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.964+0000 7efe43e00640 1 -- 172.21.15.67:0/313750071 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe50039020 con 0x5640bc8268b0 2024-03-28T16:05:11.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.964+0000 7efe43e00640 1 -- 172.21.15.67:0/313750071 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7efe500209b0 con 0x5640bc8268b0 2024-03-28T16:05:11.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.964+0000 7efe68221740 1 -- 172.21.15.67:0/313750071 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efe5002a030 con 0x5640bc8268b0 2024-03-28T16:05:11.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.964+0000 7efe43e00640 1 -- 172.21.15.67:0/313750071 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7efe5002f040 con 0x5640bc8268b0 2024-03-28T16:05:11.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.964+0000 7efe68221740 10 client.15543 fetch_fsmap learned FSMap version 19 2024-03-28T16:05:11.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.964+0000 7efe68221740 1 -- 172.21.15.67:0/313750071 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5640bc8992a0 con 0x5640bc8268b0 2024-03-28T16:05:11.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.964+0000 7efe43e00640 1 -- 172.21.15.67:0/313750071 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7efe5002a030 con 0x5640bc8268b0 2024-03-28T16:05:11.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.965+0000 7efe43e00640 1 -- 172.21.15.67:0/313750071 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7efe50032070 con 0x5640bc8268b0 2024-03-28T16:05:11.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.965+0000 7efe68221740 10 client.15543 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:05:11.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.965+0000 7efe68221740 10 client.15543 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:05:11.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.965+0000 7efe68221740 20 client.15543 populate_metadata read hostname 'smithi067' 2024-03-28T16:05:11.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.965+0000 7efe68221740 1 --2- 172.21.15.67:0/313750071 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5640bc899c50 0x5640bc89c070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:11.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.965+0000 7efe68221740 4 client.15543 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T16:05:11.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.965+0000 7efe68221740 1 -- 172.21.15.67:0/313750071 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5640bc78b740 con 0x5640bc899c50 2024-03-28T16:05:11.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.965+0000 7efe57e00640 1 --2- 172.21.15.67:0/313750071 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5640bc899c50 0x5640bc89c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:11.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.967+0000 7efe57e00640 1 --2- 172.21.15.67:0/313750071 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5640bc899c50 0x5640bc89c070 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c16cb3f10983ad8f server_cookie=87e238120a17c221 in_seq=0 out_seq=0 2024-03-28T16:05:11.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.967+0000 7efe43e00640 10 client.15543 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:05:11.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.970+0000 7efe43e00640 1 -- 172.21.15.67:0/313750071 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5640bc78b740 con 0x5640bc899c50 2024-03-28T16:05:11.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.970+0000 7efe43e00640 10 client.15543 handle_command_reply: tid=0 2024-03-28T16:05:11.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.970+0000 7efe68221740 1 client.15543 shutdown 2024-03-28T16:05:11.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.971+0000 7efe68221740 1 -- 172.21.15.67:0/313750071 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5640bc8268b0 msgr2=0x5640bc8369e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:11.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.971+0000 7efe68221740 1 --2- 172.21.15.67:0/313750071 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5640bc8268b0 0x5640bc8369e0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7efe5002d040 tx=0x7efe5002a1e0 comp rx=0 tx=0).stop 2024-03-28T16:05:11.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.971+0000 7efe68221740 1 -- 172.21.15.67:0/313750071 shutdown_connections 2024-03-28T16:05:11.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.971+0000 7efe68221740 1 --2- 172.21.15.67:0/313750071 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5640bc899c50 0x5640bc89c070 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:11.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.971+0000 7efe68221740 1 --2- 172.21.15.67:0/313750071 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5640bc82c410 0x5640bc837460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:11.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.971+0000 7efe68221740 1 --2- 172.21.15.67:0/313750071 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5640bc828280 0x5640bc836f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:11.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.971+0000 7efe68221740 1 --2- 172.21.15.67:0/313750071 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5640bc8268b0 0x5640bc8369e0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:11.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.971+0000 7efe68221740 1 -- 172.21.15.67:0/313750071 >> 172.21.15.67:0/313750071 conn(0x5640bc821e20 msgr2=0x5640bc825910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:05:11.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.971+0000 7efe43e00640 0 client.15543 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:05:11.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.971+0000 7efe68221740 1 -- 172.21.15.67:0/313750071 shutdown_connections 2024-03-28T16:05:11.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.971+0000 7efe68221740 1 -- 172.21.15.67:0/313750071 wait complete. 2024-03-28T16:05:11.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.971+0000 7efe68221740 20 client.15543 trim_cache size 0 max 16384 2024-03-28T16:05:11.972 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-28T16:05:11.973 INFO:teuthology.orchestra.run.smithi067.stdout: "return_code": 0, 2024-03-28T16:05:11.973 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_tag": "dc17b6e3-222c-4ce1-bbf1-c1205cfd01ef", 2024-03-28T16:05:11.973 INFO:teuthology.orchestra.run.smithi067.stdout: "mode": "asynchronous" 2024-03-28T16:05:11.973 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-28T16:05:11.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.972+0000 7efe65600640 1 -- 172.21.15.67:0/2512242534 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7efe440a2800 msgr2=0x7efe440a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:11.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.972+0000 7efe65600640 1 --2- 172.21.15.67:0/2512242534 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7efe440a2800 0x7efe440a4cc0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x5640bc8276a0 tx=0x7efe50005f90 comp rx=0 tx=0).stop 2024-03-28T16:05:11.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.972+0000 7efe65600640 1 -- 172.21.15.67:0/2512242534 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe60103260 msgr2=0x7efe60073fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:11.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.972+0000 7efe65600640 1 --2- 172.21.15.67:0/2512242534 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe60103260 0x7efe60073fd0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7efe5802c700 tx=0x7efe580328a0 comp rx=0 tx=0).stop 2024-03-28T16:05:11.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.972+0000 7efe65600640 1 -- 172.21.15.67:0/2512242534 shutdown_connections 2024-03-28T16:05:11.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.972+0000 7efe65600640 1 --2- 172.21.15.67:0/2512242534 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7efe440a2800 0x7efe440a4cc0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:11.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.972+0000 7efe65600640 1 --2- 172.21.15.67:0/2512242534 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe60103260 0x7efe60073fd0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:11.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.972+0000 7efe65600640 1 --2- 172.21.15.67:0/2512242534 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efe601028a0 0x7efe60073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:11.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.972+0000 7efe65600640 1 --2- 172.21.15.67:0/2512242534 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe601016b0 0x7efe60075450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:11.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.972+0000 7efe65600640 1 -- 172.21.15.67:0/2512242534 >> 172.21.15.67:0/2512242534 conn(0x7efe600fcc40 msgr2=0x7efe60105cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:05:11.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.972+0000 7efe65600640 1 -- 172.21.15.67:0/2512242534 shutdown_connections 2024-03-28T16:05:11.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:11.972+0000 7efe65600640 1 -- 172.21.15.67:0/2512242534 wait complete. 2024-03-28T16:05:11.992 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T16:05:11.992 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T16:05:12.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.213+0000 7f7ab6000640 1 -- 172.21.15.67:0/825098318 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ab0104a60 msgr2=0x7f7ab0106e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:12.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.213+0000 7f7ab6000640 1 --2- 172.21.15.67:0/825098318 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ab0104a60 0x7f7ab0106e50 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f7a9c009900 tx=0x7f7a9c02bc30 comp rx=0 tx=0).stop 2024-03-28T16:05:12.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.213+0000 7f7ab6000640 1 -- 172.21.15.67:0/825098318 shutdown_connections 2024-03-28T16:05:12.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.213+0000 7f7ab6000640 1 --2- 172.21.15.67:0/825098318 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7ab0107390 0x7f7ab01097e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:12.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.213+0000 7f7ab6000640 1 --2- 172.21.15.67:0/825098318 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ab0104a60 0x7f7ab0106e50 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:12.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.213+0000 7f7ab6000640 1 --2- 172.21.15.67:0/825098318 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ab0102130 0x7f7ab0104520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:12.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.213+0000 7f7ab6000640 1 -- 172.21.15.67:0/825098318 >> 172.21.15.67:0/825098318 conn(0x7f7ab00fbb40 msgr2=0x7f7ab00fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:05:12.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.213+0000 7f7ab6000640 1 -- 172.21.15.67:0/825098318 shutdown_connections 2024-03-28T16:05:12.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.213+0000 7f7ab6000640 1 -- 172.21.15.67:0/825098318 wait complete. 2024-03-28T16:05:12.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.214+0000 7f7ab6000640 1 Processor -- start 2024-03-28T16:05:12.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.214+0000 7f7ab6000640 1 -- start start 2024-03-28T16:05:12.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.215+0000 7f7ab6000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ab0102130 0x7f7ab0199b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:12.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.215+0000 7f7ab6000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7ab0104a60 0x7f7ab019a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:12.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.215+0000 7f7ab6000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ab0107390 0x7f7ab01a10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:12.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.215+0000 7f7ab6000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ab010d1d0 con 0x7f7ab0102130 2024-03-28T16:05:12.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.215+0000 7f7ab6000640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ab010d050 con 0x7f7ab0107390 2024-03-28T16:05:12.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.215+0000 7f7ab6000640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ab010d350 con 0x7f7ab0104a60 2024-03-28T16:05:12.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.215+0000 7f7aaf400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ab0102130 0x7f7ab0199b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:12.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.215+0000 7f7aaf400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ab0102130 0x7f7ab0199b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55128/0 (socket says 172.21.15.67:55128) 2024-03-28T16:05:12.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.215+0000 7f7aaf400640 1 -- 172.21.15.67:0/1892659058 learned_addr learned my addr 172.21.15.67:0/1892659058 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:05:12.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.215+0000 7f7aaea00640 1 --2- 172.21.15.67:0/1892659058 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7ab0104a60 0x7f7ab019a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:12.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.215+0000 7f7aafe00640 1 --2- 172.21.15.67:0/1892659058 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ab0107390 0x7f7ab01a10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:12.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.215+0000 7f7aaf400640 1 -- 172.21.15.67:0/1892659058 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7ab0104a60 msgr2=0x7f7ab019a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:12.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.215+0000 7f7aaf400640 1 --2- 172.21.15.67:0/1892659058 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7ab0104a60 0x7f7ab019a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:12.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.215+0000 7f7aaf400640 1 -- 172.21.15.67:0/1892659058 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ab0107390 msgr2=0x7f7ab01a10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:12.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.215+0000 7f7aaf400640 1 --2- 172.21.15.67:0/1892659058 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ab0107390 0x7f7ab01a10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:12.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.215+0000 7f7aaf400640 1 -- 172.21.15.67:0/1892659058 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ab01a1740 con 0x7f7ab0102130 2024-03-28T16:05:12.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.216+0000 7f7aaf400640 1 --2- 172.21.15.67:0/1892659058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ab0102130 0x7f7ab0199b20 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f7aa000e9f0 tx=0x7f7aa000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:05:12.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.216+0000 7f7aa7e00640 1 -- 172.21.15.67:0/1892659058 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa000cd80 con 0x7f7ab0102130 2024-03-28T16:05:12.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.216+0000 7f7aa7e00640 1 -- 172.21.15.67:0/1892659058 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7aa00108e0 con 0x7f7ab0102130 2024-03-28T16:05:12.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.216+0000 7f7aa7e00640 1 -- 172.21.15.67:0/1892659058 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa0018920 con 0x7f7ab0102130 2024-03-28T16:05:12.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.216+0000 7f7ab6000640 1 -- 172.21.15.67:0/1892659058 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ab01a1a30 con 0x7f7ab0102130 2024-03-28T16:05:12.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.216+0000 7f7ab6000640 1 -- 172.21.15.67:0/1892659058 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ab01a1f00 con 0x7f7ab0102130 2024-03-28T16:05:12.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.220+0000 7f7ab8dfc740 1 Processor -- start 2024-03-28T16:05:12.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.220+0000 7f7ab8dfc740 1 -- start start 2024-03-28T16:05:12.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.220+0000 7f7aa7e00640 1 -- 172.21.15.67:0/1892659058 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f7aa0010a90 con 0x7f7ab0102130 2024-03-28T16:05:12.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.221+0000 7f7ab8dfc740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56019ffede80 0x56019ffee2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:12.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.221+0000 7f7ab8dfc740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56019ffee870 0x56019fff3370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:12.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.221+0000 7f7ab8dfc740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56019fff39e0 0x56019fff5e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:12.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.221+0000 7f7ab8dfc740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x56019ff53250 con 0x56019fff39e0 2024-03-28T16:05:12.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.221+0000 7f7ab8dfc740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x56019fefba30 con 0x56019ffede80 2024-03-28T16:05:12.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.221+0000 7f7ab8dfc740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x56019ff4f2f0 con 0x56019ffee870 2024-03-28T16:05:12.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.221+0000 7f7aaea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56019fff39e0 0x56019fff5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:12.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.221+0000 7f7aaea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56019fff39e0 0x56019fff5e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55138/0 (socket says 172.21.15.67:55138) 2024-03-28T16:05:12.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.221+0000 7f7aaea00640 1 -- 172.21.15.67:0/3233840797 learned_addr learned my addr 172.21.15.67:0/3233840797 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:05:12.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.221+0000 7f7aafe00640 1 --2- 172.21.15.67:0/3233840797 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56019ffede80 0x56019ffee2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:12.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.221+0000 7f7aaf400640 1 --2- 172.21.15.67:0/3233840797 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56019ffee870 0x56019fff3370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:12.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.222+0000 7f7aaea00640 1 -- 172.21.15.67:0/3233840797 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56019ffee870 msgr2=0x56019fff3370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:05:12.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.222+0000 7f7aaea00640 1 --2- 172.21.15.67:0/3233840797 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56019ffee870 0x56019fff3370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:12.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.222+0000 7f7aaea00640 1 -- 172.21.15.67:0/3233840797 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56019ffede80 msgr2=0x56019ffee2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:05:12.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.222+0000 7f7aaea00640 1 --2- 172.21.15.67:0/3233840797 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56019ffede80 0x56019ffee2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:12.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.222+0000 7f7aaea00640 1 -- 172.21.15.67:0/3233840797 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56019fee6690 con 0x56019fff39e0 2024-03-28T16:05:12.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.222+0000 7f7aa7e00640 1 --2- 172.21.15.67:0/1892659058 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7a940a2880 0x7f7a940a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:12.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.222+0000 7f7aa7e00640 1 -- 172.21.15.67:0/1892659058 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f7aa00ceea0 con 0x7f7ab0102130 2024-03-28T16:05:12.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.222+0000 7f7aaea00640 1 --2- 172.21.15.67:0/3233840797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56019fff39e0 0x56019fff5e70 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f7a9c005ab0 tx=0x7f7a9c0095e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49fb0bbb5993a92f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:05:12.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.222+0000 7f7aafe00640 1 --2- 172.21.15.67:0/1892659058 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7a940a2880 0x7f7a940a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:12.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.223+0000 7f7a9be00640 1 -- 172.21.15.67:0/3233840797 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a9c044020 con 0x56019fff39e0 2024-03-28T16:05:12.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.223+0000 7f7a9be00640 1 -- 172.21.15.67:0/3233840797 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7a9c02cba0 con 0x56019fff39e0 2024-03-28T16:05:12.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.223+0000 7f7a9be00640 1 -- 172.21.15.67:0/3233840797 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a9c0046c0 con 0x56019fff39e0 2024-03-28T16:05:12.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.224+0000 7f7ab8dfc740 1 -- 172.21.15.67:0/3233840797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56019fff39e0 msgr2=0x56019fff5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:12.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.224+0000 7f7ab8dfc740 1 --2- 172.21.15.67:0/3233840797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56019fff39e0 0x56019fff5e70 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f7a9c005ab0 tx=0x7f7a9c0095e0 comp rx=0 tx=0).stop 2024-03-28T16:05:12.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.224+0000 7f7aafe00640 1 --2- 172.21.15.67:0/1892659058 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7a940a2880 0x7f7a940a4d40 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x56019ffef0d0 tx=0x7f7aa8009740 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:05:12.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.224+0000 7f7ab8dfc740 1 -- 172.21.15.67:0/3233840797 shutdown_connections 2024-03-28T16:05:12.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.224+0000 7f7ab8dfc740 1 --2- 172.21.15.67:0/3233840797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56019fff39e0 0x56019fff5e70 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:12.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.224+0000 7f7ab8dfc740 1 --2- 172.21.15.67:0/3233840797 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56019ffee870 0x56019fff3370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:12.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.224+0000 7f7ab8dfc740 1 --2- 172.21.15.67:0/3233840797 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56019ffede80 0x56019ffee2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:12.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.224+0000 7f7ab8dfc740 1 -- 172.21.15.67:0/3233840797 >> 172.21.15.67:0/3233840797 conn(0x56019ffb6200 msgr2=0x56019ffe97f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:05:12.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.224+0000 7f7ab8dfc740 1 -- 172.21.15.67:0/3233840797 shutdown_connections 2024-03-28T16:05:12.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.224+0000 7f7ab8dfc740 1 -- 172.21.15.67:0/3233840797 wait complete. 2024-03-28T16:05:12.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.224+0000 7f7ab8dfc740 1 -- start start 2024-03-28T16:05:12.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.225+0000 7f7ab8dfc740 1 Processor -- start 2024-03-28T16:05:12.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.225+0000 7f7ab8dfc740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56019ffede80 0x56019fff7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:12.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.225+0000 7f7ab8dfc740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56019ffee870 0x56019fff8330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:12.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.225+0000 7f7ab8dfc740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56019fff39e0 0x56019ffff370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:12.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.225+0000 7f7ab8dfc740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x56019fff6930 con 0x56019fff39e0 2024-03-28T16:05:12.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.225+0000 7f7ab8dfc740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x56019ff0ee80 con 0x56019ffede80 2024-03-28T16:05:12.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.225+0000 7f7ab8dfc740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x56019fff6ab0 con 0x56019ffee870 2024-03-28T16:05:12.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.225+0000 7f7ab8dfc740 10 client.0 _ll_register_callbacks cb 0x56019ff21130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:05:12.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.226+0000 7f7aafe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56019fff39e0 0x56019ffff370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:12.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.226+0000 7f7aafe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56019fff39e0 0x56019ffff370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55146/0 (socket says 172.21.15.67:55146) 2024-03-28T16:05:12.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.226+0000 7f7aafe00640 1 -- 172.21.15.67:0/691933737 learned_addr learned my addr 172.21.15.67:0/691933737 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:05:12.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.226+0000 7f7aaf400640 1 --2- 172.21.15.67:0/691933737 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56019ffede80 0x56019fff7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:12.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.226+0000 7f7aaea00640 1 --2- 172.21.15.67:0/691933737 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56019ffee870 0x56019fff8330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:12.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.226+0000 7f7aafe00640 1 -- 172.21.15.67:0/691933737 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56019ffee870 msgr2=0x56019fff8330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:05:12.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.226+0000 7f7aafe00640 1 --2- 172.21.15.67:0/691933737 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56019ffee870 0x56019fff8330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:12.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.226+0000 7f7aafe00640 1 -- 172.21.15.67:0/691933737 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56019ffede80 msgr2=0x56019fff7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:05:12.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.226+0000 7f7aafe00640 1 --2- 172.21.15.67:0/691933737 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56019ffede80 0x56019fff7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:12.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.226+0000 7f7aafe00640 1 -- 172.21.15.67:0/691933737 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56019fd317d0 con 0x56019fff39e0 2024-03-28T16:05:12.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.227+0000 7f7aafe00640 1 --2- 172.21.15.67:0/691933737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56019fff39e0 0x56019ffff370 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f7aa8009fd0 tx=0x7f7aa8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e9445b77689d108 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:05:12.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.227+0000 7f7a98c00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:05:12.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.227+0000 7f7a98c00640 1 -- 172.21.15.67:0/691933737 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa8010d30 con 0x56019fff39e0 2024-03-28T16:05:12.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.227+0000 7f7a98c00640 1 -- 172.21.15.67:0/691933737 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f7aa8028470 con 0x56019fff39e0 2024-03-28T16:05:12.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.227+0000 7f7ab8dfc740 1 -- 172.21.15.67:0/691933737 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7a9c02cba0 con 0x56019fff39e0 2024-03-28T16:05:12.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.227+0000 7f7a98c00640 1 -- 172.21.15.67:0/691933737 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7aa8028670 con 0x56019fff39e0 2024-03-28T16:05:12.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.227+0000 7f7ab8dfc740 10 client.15555 fetch_fsmap learned FSMap version 19 2024-03-28T16:05:12.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.227+0000 7f7ab8dfc740 1 -- 172.21.15.67:0/691933737 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x56019fc673f0 con 0x56019fff39e0 2024-03-28T16:05:12.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.228+0000 7f7a98c00640 1 -- 172.21.15.67:0/691933737 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7a9c02cba0 con 0x56019fff39e0 2024-03-28T16:05:12.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.228+0000 7f7a98c00640 1 -- 172.21.15.67:0/691933737 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f7aa800f040 con 0x56019fff39e0 2024-03-28T16:05:12.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.228+0000 7f7ab8dfc740 10 client.15555 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:05:12.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.228+0000 7f7ab8dfc740 10 client.15555 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:05:12.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.228+0000 7f7ab8dfc740 20 client.15555 populate_metadata read hostname 'smithi067' 2024-03-28T16:05:12.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.228+0000 7f7ab8dfc740 1 --2- 172.21.15.67:0/691933737 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5601a0003ce0 0x5601a00061a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:12.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.228+0000 7f7ab8dfc740 4 client.15555 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:05:12.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.228+0000 7f7ab8dfc740 1 -- 172.21.15.67:0/691933737 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56019ff60790 con 0x5601a0003ce0 2024-03-28T16:05:12.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.229+0000 7f7aaf400640 1 --2- 172.21.15.67:0/691933737 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5601a0003ce0 0x5601a00061a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:12.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.230+0000 7f7a98c00640 10 client.15555 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:05:12.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.230+0000 7f7aaf400640 1 --2- 172.21.15.67:0/691933737 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5601a0003ce0 0x5601a00061a0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=633f8015aee7b4c3 server_cookie=f17a4e9863968b6a in_seq=0 out_seq=0 2024-03-28T16:05:12.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.231+0000 7f7a98c00640 1 -- 172.21.15.67:0/691933737 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56019ff60790 con 0x5601a0003ce0 2024-03-28T16:05:12.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.231+0000 7f7a98c00640 10 client.15555 handle_command_reply: tid=0 2024-03-28T16:05:12.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.232+0000 7f7ab8dfc740 1 client.15555 shutdown 2024-03-28T16:05:12.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.232+0000 7f7ab8dfc740 1 -- 172.21.15.67:0/691933737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56019fff39e0 msgr2=0x56019ffff370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:12.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.232+0000 7f7ab8dfc740 1 --2- 172.21.15.67:0/691933737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56019fff39e0 0x56019ffff370 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f7aa8009fd0 tx=0x7f7aa8004a40 comp rx=0 tx=0).stop 2024-03-28T16:05:12.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.232+0000 7f7ab8dfc740 1 -- 172.21.15.67:0/691933737 shutdown_connections 2024-03-28T16:05:12.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.232+0000 7f7ab8dfc740 1 --2- 172.21.15.67:0/691933737 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5601a0003ce0 0x5601a00061a0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:12.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.232+0000 7f7ab8dfc740 1 --2- 172.21.15.67:0/691933737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56019fff39e0 0x56019ffff370 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:12.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.232+0000 7f7ab8dfc740 1 --2- 172.21.15.67:0/691933737 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56019ffee870 0x56019fff8330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:12.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.232+0000 7f7ab8dfc740 1 --2- 172.21.15.67:0/691933737 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56019ffede80 0x56019fff7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:12.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.232+0000 7f7ab8dfc740 1 -- 172.21.15.67:0/691933737 >> 172.21.15.67:0/691933737 conn(0x56019ffb6200 msgr2=0x56019ffe8a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:05:12.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.232+0000 7f7a98c00640 0 client.15555 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:05:12.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.232+0000 7f7ab8dfc740 1 -- 172.21.15.67:0/691933737 shutdown_connections 2024-03-28T16:05:12.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.232+0000 7f7ab8dfc740 1 -- 172.21.15.67:0/691933737 wait complete. 2024-03-28T16:05:12.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.232+0000 7f7ab8dfc740 20 client.15555 trim_cache size 0 max 16384 2024-03-28T16:05:12.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.256+0000 7f7ab8dfc740 1 Processor -- start 2024-03-28T16:05:12.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.256+0000 7f7ab8dfc740 1 -- start start 2024-03-28T16:05:12.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.256+0000 7f7ab8dfc740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56019ffede80 0x56019fff2e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:12.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.256+0000 7f7ab8dfc740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56019ffee870 0x56019ffff330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:12.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.256+0000 7f7ab8dfc740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56019fff39e0 0x56019ffeca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:12.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.256+0000 7f7ab8dfc740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ab01a8c70 con 0x56019ffede80 2024-03-28T16:05:12.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.256+0000 7f7ab8dfc740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ab01ab230 con 0x56019ffee870 2024-03-28T16:05:12.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.256+0000 7f7ab8dfc740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ab01a9270 con 0x56019fff39e0 2024-03-28T16:05:12.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.256+0000 7f7aaf400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56019ffede80 0x56019fff2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:12.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.256+0000 7f7aaf400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56019ffede80 0x56019fff2e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55154/0 (socket says 172.21.15.67:55154) 2024-03-28T16:05:12.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.256+0000 7f7aaf400640 1 -- 172.21.15.67:0/3757809599 learned_addr learned my addr 172.21.15.67:0/3757809599 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:05:12.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.257+0000 7f7aaea00640 1 --2- 172.21.15.67:0/3757809599 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56019ffee870 0x56019ffff330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:12.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.257+0000 7f7aafe00640 1 --2- 172.21.15.67:0/3757809599 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56019fff39e0 0x56019ffeca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:12.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.257+0000 7f7aaf400640 1 -- 172.21.15.67:0/3757809599 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56019fff39e0 msgr2=0x56019ffeca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:05:12.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.257+0000 7f7aaf400640 1 --2- 172.21.15.67:0/3757809599 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56019fff39e0 0x56019ffeca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:12.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.257+0000 7f7aaf400640 1 -- 172.21.15.67:0/3757809599 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56019ffee870 msgr2=0x56019ffff330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:05:12.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.257+0000 7f7aaf400640 1 --2- 172.21.15.67:0/3757809599 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56019ffee870 0x56019ffff330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:12.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.257+0000 7f7aaf400640 1 -- 172.21.15.67:0/3757809599 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56019fc270f0 con 0x56019ffede80 2024-03-28T16:05:12.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.257+0000 7f7aafe00640 1 --2- 172.21.15.67:0/3757809599 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56019fff39e0 0x56019ffeca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:05:12.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.257+0000 7f7aaf400640 1 --2- 172.21.15.67:0/3757809599 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56019ffede80 0x56019fff2e70 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f7aa000e9c0 tx=0x7f7aa00d12f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92fbb75dcb9a7d7c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:05:12.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.258+0000 7f7a9b400640 1 -- 172.21.15.67:0/3757809599 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa009b7f0 con 0x56019ffede80 2024-03-28T16:05:12.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.258+0000 7f7a9b400640 1 -- 172.21.15.67:0/3757809599 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7aa009f940 con 0x56019ffede80 2024-03-28T16:05:12.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.258+0000 7f7a9b400640 1 -- 172.21.15.67:0/3757809599 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa0097e60 con 0x56019ffede80 2024-03-28T16:05:12.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.258+0000 7f7ab8dfc740 1 -- 172.21.15.67:0/3757809599 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56019ffede80 msgr2=0x56019fff2e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:12.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.258+0000 7f7ab8dfc740 1 --2- 172.21.15.67:0/3757809599 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56019ffede80 0x56019fff2e70 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f7aa000e9c0 tx=0x7f7aa00d12f0 comp rx=0 tx=0).stop 2024-03-28T16:05:12.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.259+0000 7f7ab8dfc740 1 -- 172.21.15.67:0/3757809599 shutdown_connections 2024-03-28T16:05:12.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.259+0000 7f7ab8dfc740 1 --2- 172.21.15.67:0/3757809599 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56019fff39e0 0x56019ffeca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:12.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.259+0000 7f7ab8dfc740 1 --2- 172.21.15.67:0/3757809599 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56019ffee870 0x56019ffff330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:12.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.259+0000 7f7ab8dfc740 1 --2- 172.21.15.67:0/3757809599 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56019ffede80 0x56019fff2e70 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:12.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.259+0000 7f7ab8dfc740 1 -- 172.21.15.67:0/3757809599 >> 172.21.15.67:0/3757809599 conn(0x56019ffb6200 msgr2=0x56019ffefba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:05:12.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.259+0000 7f7ab8dfc740 1 -- 172.21.15.67:0/3757809599 shutdown_connections 2024-03-28T16:05:12.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.259+0000 7f7ab8dfc740 1 -- 172.21.15.67:0/3757809599 wait complete. 2024-03-28T16:05:12.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.259+0000 7f7ab8dfc740 1 -- start start 2024-03-28T16:05:12.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.259+0000 7f7ab8dfc740 1 Processor -- start 2024-03-28T16:05:12.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.260+0000 7f7ab8dfc740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56019ffede80 0x56019fffd060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:12.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.260+0000 7f7ab8dfc740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56019ffee870 0x56019fffd5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:12.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.260+0000 7f7ab8dfc740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56019fff39e0 0x56019fffe5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:12.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.260+0000 7f7ab8dfc740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ab01a98b0 con 0x56019fff39e0 2024-03-28T16:05:12.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.260+0000 7f7ab8dfc740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ab01aabd0 con 0x56019ffede80 2024-03-28T16:05:12.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.260+0000 7f7ab8dfc740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ab01a9f10 con 0x56019ffee870 2024-03-28T16:05:12.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.260+0000 7f7ab8dfc740 10 client.0 _ll_register_callbacks cb 0x7f7aa8009a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:05:12.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.260+0000 7f7aafe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56019fff39e0 0x56019fffe5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:12.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.260+0000 7f7aafe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56019fff39e0 0x56019fffe5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55170/0 (socket says 172.21.15.67:55170) 2024-03-28T16:05:12.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.260+0000 7f7aafe00640 1 -- 172.21.15.67:0/3720987325 learned_addr learned my addr 172.21.15.67:0/3720987325 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:05:12.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.261+0000 7f7aaf400640 1 --2- 172.21.15.67:0/3720987325 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56019ffede80 0x56019fffd060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:12.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.261+0000 7f7aaea00640 1 --2- 172.21.15.67:0/3720987325 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56019ffee870 0x56019fffd5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:12.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.261+0000 7f7aafe00640 1 -- 172.21.15.67:0/3720987325 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56019ffee870 msgr2=0x56019fffd5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:05:12.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.261+0000 7f7aafe00640 1 --2- 172.21.15.67:0/3720987325 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56019ffee870 0x56019fffd5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:12.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.261+0000 7f7aafe00640 1 -- 172.21.15.67:0/3720987325 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56019ffede80 msgr2=0x56019fffd060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:05:12.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.261+0000 7f7aafe00640 1 --2- 172.21.15.67:0/3720987325 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56019ffede80 0x56019fffd060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:12.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.261+0000 7f7aafe00640 1 -- 172.21.15.67:0/3720987325 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56019fffeba0 con 0x56019fff39e0 2024-03-28T16:05:12.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.261+0000 7f7aafe00640 1 --2- 172.21.15.67:0/3720987325 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56019fff39e0 0x56019fffe5b0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f7aa801fdd0 tx=0x7f7aa801f750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53c5e4de7b531ec server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:05:12.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.261+0000 7f7a98c00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:05:12.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.262+0000 7f7a98c00640 1 -- 172.21.15.67:0/3720987325 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa8012ca0 con 0x56019fff39e0 2024-03-28T16:05:12.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.262+0000 7f7a98c00640 1 -- 172.21.15.67:0/3720987325 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f7aa8010d40 con 0x56019fff39e0 2024-03-28T16:05:12.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.262+0000 7f7ab8dfc740 1 -- 172.21.15.67:0/3720987325 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7aa009f940 con 0x56019fff39e0 2024-03-28T16:05:12.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.262+0000 7f7a98c00640 1 -- 172.21.15.67:0/3720987325 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7aa8004ab0 con 0x56019fff39e0 2024-03-28T16:05:12.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.262+0000 7f7ab8dfc740 10 client.15561 fetch_fsmap learned FSMap version 19 2024-03-28T16:05:12.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.262+0000 7f7ab8dfc740 1 -- 172.21.15.67:0/3720987325 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5601a005faf0 con 0x56019fff39e0 2024-03-28T16:05:12.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.262+0000 7f7a98c00640 1 -- 172.21.15.67:0/3720987325 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7aa009f940 con 0x56019fff39e0 2024-03-28T16:05:12.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.263+0000 7f7a98c00640 1 -- 172.21.15.67:0/3720987325 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f7aa8045030 con 0x56019fff39e0 2024-03-28T16:05:12.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.263+0000 7f7ab8dfc740 10 client.15561 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:05:12.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.263+0000 7f7ab8dfc740 10 client.15561 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:05:12.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.263+0000 7f7ab8dfc740 20 client.15561 populate_metadata read hostname 'smithi067' 2024-03-28T16:05:12.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.263+0000 7f7ab8dfc740 1 --2- 172.21.15.67:0/3720987325 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5601a00604c0 0x5601a00628e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:12.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.263+0000 7f7ab8dfc740 4 client.15561 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T16:05:12.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.263+0000 7f7ab8dfc740 1 -- 172.21.15.67:0/3720987325 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x56019ff5aae0 con 0x5601a00604c0 2024-03-28T16:05:12.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.263+0000 7f7aaf400640 1 --2- 172.21.15.67:0/3720987325 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5601a00604c0 0x5601a00628e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:12.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.264+0000 7f7aaf400640 1 --2- 172.21.15.67:0/3720987325 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5601a00604c0 0x5601a00628e0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c562786a7239bdf server_cookie=6e709d7b8d26786 in_seq=0 out_seq=0 2024-03-28T16:05:12.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.264+0000 7f7a98c00640 10 client.15561 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:05:12.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.265+0000 7f7a98c00640 1 -- 172.21.15.67:0/3720987325 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x56019ff5aae0 con 0x5601a00604c0 2024-03-28T16:05:12.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.265+0000 7f7a98c00640 10 client.15561 handle_command_reply: tid=0 2024-03-28T16:05:12.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.265+0000 7f7ab8dfc740 1 client.15561 shutdown 2024-03-28T16:05:12.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.265+0000 7f7ab8dfc740 1 -- 172.21.15.67:0/3720987325 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56019fff39e0 msgr2=0x56019fffe5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:12.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.265+0000 7f7ab8dfc740 1 --2- 172.21.15.67:0/3720987325 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56019fff39e0 0x56019fffe5b0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f7aa801fdd0 tx=0x7f7aa801f750 comp rx=0 tx=0).stop 2024-03-28T16:05:12.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.265+0000 7f7ab8dfc740 1 -- 172.21.15.67:0/3720987325 shutdown_connections 2024-03-28T16:05:12.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.265+0000 7f7ab8dfc740 1 --2- 172.21.15.67:0/3720987325 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5601a00604c0 0x5601a00628e0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:12.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.265+0000 7f7ab8dfc740 1 --2- 172.21.15.67:0/3720987325 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56019fff39e0 0x56019fffe5b0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:12.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.265+0000 7f7ab8dfc740 1 --2- 172.21.15.67:0/3720987325 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56019ffee870 0x56019fffd5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:12.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.265+0000 7f7ab8dfc740 1 --2- 172.21.15.67:0/3720987325 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56019ffede80 0x56019fffd060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:12.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.265+0000 7f7ab8dfc740 1 -- 172.21.15.67:0/3720987325 >> 172.21.15.67:0/3720987325 conn(0x56019ffb6200 msgr2=0x56019ffefb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:05:12.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.265+0000 7f7a98c00640 0 client.15561 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:05:12.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.265+0000 7f7ab8dfc740 1 -- 172.21.15.67:0/3720987325 shutdown_connections 2024-03-28T16:05:12.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.265+0000 7f7ab8dfc740 1 -- 172.21.15.67:0/3720987325 wait complete. 2024-03-28T16:05:12.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.266+0000 7f7ab8dfc740 20 client.15561 trim_cache size 0 max 16384 2024-03-28T16:05:12.268 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-28T16:05:12.268 INFO:teuthology.orchestra.run.smithi067.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T16:05:12.268 INFO:teuthology.orchestra.run.smithi067.stdout: "scrubs": { 2024-03-28T16:05:12.268 INFO:teuthology.orchestra.run.smithi067.stdout: "dc17b6e3-222c-4ce1-bbf1-c1205cfd01ef": { 2024-03-28T16:05:12.268 INFO:teuthology.orchestra.run.smithi067.stdout: "path": "/", 2024-03-28T16:05:12.268 INFO:teuthology.orchestra.run.smithi067.stdout: "tag": "dc17b6e3-222c-4ce1-bbf1-c1205cfd01ef", 2024-03-28T16:05:12.268 INFO:teuthology.orchestra.run.smithi067.stdout: "options": "recursive,force" 2024-03-28T16:05:12.268 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:05:12.268 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:05:12.269 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-28T16:05:12.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.266+0000 7f7ab6000640 1 -- 172.21.15.67:0/1892659058 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7a940a2880 msgr2=0x7f7a940a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:12.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.266+0000 7f7ab6000640 1 --2- 172.21.15.67:0/1892659058 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7a940a2880 0x7f7a940a4d40 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x56019ffef0d0 tx=0x7f7aa8009740 comp rx=0 tx=0).stop 2024-03-28T16:05:12.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.266+0000 7f7ab6000640 1 -- 172.21.15.67:0/1892659058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ab0102130 msgr2=0x7f7ab0199b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:12.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.266+0000 7f7ab6000640 1 --2- 172.21.15.67:0/1892659058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ab0102130 0x7f7ab0199b20 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f7aa000e9f0 tx=0x7f7aa000eeb0 comp rx=0 tx=0).stop 2024-03-28T16:05:12.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.266+0000 7f7ab6000640 1 -- 172.21.15.67:0/1892659058 shutdown_connections 2024-03-28T16:05:12.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.267+0000 7f7ab6000640 1 --2- 172.21.15.67:0/1892659058 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7a940a2880 0x7f7a940a4d40 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:12.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.267+0000 7f7ab6000640 1 --2- 172.21.15.67:0/1892659058 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ab0107390 0x7f7ab01a10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:12.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.267+0000 7f7ab6000640 1 --2- 172.21.15.67:0/1892659058 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7ab0104a60 0x7f7ab019a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:12.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.267+0000 7f7ab6000640 1 --2- 172.21.15.67:0/1892659058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ab0102130 0x7f7ab0199b20 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:12.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.267+0000 7f7ab6000640 1 -- 172.21.15.67:0/1892659058 >> 172.21.15.67:0/1892659058 conn(0x7f7ab00fbb40 msgr2=0x7f7ab0101810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:05:12.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.267+0000 7f7ab6000640 1 -- 172.21.15.67:0/1892659058 shutdown_connections 2024-03-28T16:05:12.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:12.267+0000 7f7ab6000640 1 -- 172.21.15.67:0/1892659058 wait complete. 2024-03-28T16:05:12.289 INFO:tasks.cephfs.filesystem:scrub status for tag:dc17b6e3-222c-4ce1-bbf1-c1205cfd01ef - {'path': '/', 'tag': 'dc17b6e3-222c-4ce1-bbf1-c1205cfd01ef', 'options': 'recursive,force'} 2024-03-28T16:05:12.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:05:12 smithi183 ceph-mon[35580]: pgmap v488: 97 pgs: 97 active+clean; 129 MiB data, 711 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 11 MiB/s wr, 6 op/s 2024-03-28T16:05:12.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:05:12 smithi150 ceph-mon[34825]: pgmap v488: 97 pgs: 97 active+clean; 129 MiB data, 711 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 11 MiB/s wr, 6 op/s 2024-03-28T16:05:12.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:05:12 smithi067 ceph-mon[31707]: pgmap v488: 97 pgs: 97 active+clean; 129 MiB data, 711 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 11 MiB/s wr, 6 op/s 2024-03-28T16:05:13.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:05:13 smithi183 ceph-mon[35580]: scrub queued for path: / 2024-03-28T16:05:13.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:05:13 smithi183 ceph-mon[35580]: scrub summary: idle+waiting paths [/] 2024-03-28T16:05:13.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:05:13 smithi183 ceph-mon[35580]: scrub summary: active paths [/] 2024-03-28T16:05:13.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:05:13 smithi183 ceph-mon[35580]: scrub summary: idle+waiting paths [/] 2024-03-28T16:05:13.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:05:13 smithi150 ceph-mon[34825]: scrub queued for path: / 2024-03-28T16:05:13.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:05:13 smithi150 ceph-mon[34825]: scrub summary: idle+waiting paths [/] 2024-03-28T16:05:13.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:05:13 smithi150 ceph-mon[34825]: scrub summary: active paths [/] 2024-03-28T16:05:13.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:05:13 smithi150 ceph-mon[34825]: scrub summary: idle+waiting paths [/] 2024-03-28T16:05:13.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:05:13 smithi067 ceph-mon[31707]: scrub queued for path: / 2024-03-28T16:05:13.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:05:13 smithi067 ceph-mon[31707]: scrub summary: idle+waiting paths [/] 2024-03-28T16:05:13.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:05:13 smithi067 ceph-mon[31707]: scrub summary: active paths [/] 2024-03-28T16:05:13.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:05:13 smithi067 ceph-mon[31707]: scrub summary: idle+waiting paths [/] 2024-03-28T16:05:14.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:05:14 smithi183 ceph-mon[35580]: pgmap v489: 97 pgs: 97 active+clean; 165 MiB data, 823 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 14 MiB/s wr, 6 op/s 2024-03-28T16:05:14.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:05:14 smithi150 ceph-mon[34825]: pgmap v489: 97 pgs: 97 active+clean; 165 MiB data, 823 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 14 MiB/s wr, 6 op/s 2024-03-28T16:05:14.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:05:14 smithi067 ceph-mon[31707]: pgmap v489: 97 pgs: 97 active+clean; 165 MiB data, 823 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 14 MiB/s wr, 6 op/s 2024-03-28T16:05:16.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:05:16 smithi183 ceph-mon[35580]: pgmap v490: 97 pgs: 97 active+clean; 217 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 17 MiB/s wr, 8 op/s 2024-03-28T16:05:16.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:05:16 smithi150 ceph-mon[34825]: pgmap v490: 97 pgs: 97 active+clean; 217 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 17 MiB/s wr, 8 op/s 2024-03-28T16:05:16.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:05:16 smithi067 ceph-mon[31707]: pgmap v490: 97 pgs: 97 active+clean; 217 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 17 MiB/s wr, 8 op/s 2024-03-28T16:05:17.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:05:17 smithi150 ceph-mon[34825]: scrub summary: idle 2024-03-28T16:05:17.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:05:17 smithi067 ceph-mon[31707]: scrub summary: idle 2024-03-28T16:05:17.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:05:17 smithi183 ceph-mon[35580]: scrub summary: idle 2024-03-28T16:05:18.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:05:18 smithi150 ceph-mon[34825]: pgmap v491: 97 pgs: 97 active+clean; 233 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 17 MiB/s wr, 6 op/s 2024-03-28T16:05:18.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:05:18 smithi067 ceph-mon[31707]: pgmap v491: 97 pgs: 97 active+clean; 233 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 17 MiB/s wr, 6 op/s 2024-03-28T16:05:18.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:05:18 smithi183 ceph-mon[35580]: pgmap v491: 97 pgs: 97 active+clean; 233 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 17 MiB/s wr, 6 op/s 2024-03-28T16:05:20.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:05:20 smithi150 ceph-mon[34825]: pgmap v492: 97 pgs: 97 active+clean; 261 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 17 MiB/s wr, 6 op/s 2024-03-28T16:05:20.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:05:20 smithi067 ceph-mon[31707]: pgmap v492: 97 pgs: 97 active+clean; 261 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 17 MiB/s wr, 6 op/s 2024-03-28T16:05:20.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:05:20 smithi183 ceph-mon[35580]: pgmap v492: 97 pgs: 97 active+clean; 261 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 17 MiB/s wr, 6 op/s 2024-03-28T16:05:22.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:05:22 smithi150 ceph-mon[34825]: pgmap v493: 97 pgs: 97 active+clean; 325 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 21 MiB/s wr, 8 op/s 2024-03-28T16:05:22.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:05:22 smithi067 ceph-mon[31707]: pgmap v493: 97 pgs: 97 active+clean; 325 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 21 MiB/s wr, 8 op/s 2024-03-28T16:05:22.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:05:22 smithi183 ceph-mon[35580]: pgmap v493: 97 pgs: 97 active+clean; 325 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 21 MiB/s wr, 8 op/s 2024-03-28T16:05:24.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:05:24 smithi150 ceph-mon[34825]: pgmap v494: 97 pgs: 97 active+clean; 369 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 20 MiB/s wr, 7 op/s 2024-03-28T16:05:24.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:05:24 smithi067 ceph-mon[31707]: pgmap v494: 97 pgs: 97 active+clean; 369 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 20 MiB/s wr, 7 op/s 2024-03-28T16:05:24.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:05:24 smithi183 ceph-mon[35580]: pgmap v494: 97 pgs: 97 active+clean; 369 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 20 MiB/s wr, 7 op/s 2024-03-28T16:05:26.653 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:05:26 smithi183 ceph-mon[35580]: pgmap v495: 97 pgs: 97 active+clean; 409 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 20 MiB/s wr, 8 op/s 2024-03-28T16:05:26.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:05:26 smithi150 ceph-mon[34825]: pgmap v495: 97 pgs: 97 active+clean; 409 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 20 MiB/s wr, 8 op/s 2024-03-28T16:05:26.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:05:26 smithi067 ceph-mon[31707]: pgmap v495: 97 pgs: 97 active+clean; 409 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 20 MiB/s wr, 8 op/s 2024-03-28T16:05:28.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:05:28 smithi150 ceph-mon[34825]: pgmap v496: 97 pgs: 97 active+clean; 425 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 17 MiB/s wr, 6 op/s 2024-03-28T16:05:28.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:05:28 smithi067 ceph-mon[31707]: pgmap v496: 97 pgs: 97 active+clean; 425 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 17 MiB/s wr, 6 op/s 2024-03-28T16:05:28.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:05:28 smithi183 ceph-mon[35580]: pgmap v496: 97 pgs: 97 active+clean; 425 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 17 MiB/s wr, 6 op/s 2024-03-28T16:05:30.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:05:30 smithi150 ceph-mon[34825]: pgmap v497: 97 pgs: 97 active+clean; 453 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 18 MiB/s wr, 6 op/s 2024-03-28T16:05:30.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:05:30 smithi067 ceph-mon[31707]: pgmap v497: 97 pgs: 97 active+clean; 453 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 18 MiB/s wr, 6 op/s 2024-03-28T16:05:30.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:05:30 smithi183 ceph-mon[35580]: pgmap v497: 97 pgs: 97 active+clean; 453 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 18 MiB/s wr, 6 op/s 2024-03-28T16:05:32.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:05:32 smithi150 ceph-mon[34825]: pgmap v498: 97 pgs: 97 active+clean; 493 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 19 MiB/s wr, 7 op/s 2024-03-28T16:05:32.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:05:32 smithi067 ceph-mon[31707]: pgmap v498: 97 pgs: 97 active+clean; 493 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 19 MiB/s wr, 7 op/s 2024-03-28T16:05:32.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:05:32 smithi183 ceph-mon[35580]: pgmap v498: 97 pgs: 97 active+clean; 493 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 19 MiB/s wr, 7 op/s 2024-03-28T16:05:34.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:05:34 smithi150 ceph-mon[34825]: pgmap v499: 97 pgs: 97 active+clean; 537 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 18 MiB/s wr, 6 op/s 2024-03-28T16:05:34.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:05:34 smithi067 ceph-mon[31707]: pgmap v499: 97 pgs: 97 active+clean; 537 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 18 MiB/s wr, 6 op/s 2024-03-28T16:05:34.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:05:34 smithi183 ceph-mon[35580]: pgmap v499: 97 pgs: 97 active+clean; 537 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 18 MiB/s wr, 6 op/s 2024-03-28T16:05:36.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:05:36 smithi150 ceph-mon[34825]: pgmap v500: 97 pgs: 97 active+clean; 577 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 17 MiB/s wr, 7 op/s 2024-03-28T16:05:36.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:05:36 smithi067 ceph-mon[31707]: pgmap v500: 97 pgs: 97 active+clean; 577 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 17 MiB/s wr, 7 op/s 2024-03-28T16:05:36.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:05:36 smithi183 ceph-mon[35580]: pgmap v500: 97 pgs: 97 active+clean; 577 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 17 MiB/s wr, 7 op/s 2024-03-28T16:05:38.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:05:38 smithi150 ceph-mon[34825]: pgmap v501: 97 pgs: 97 active+clean; 597 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 16 MiB/s wr, 5 op/s 2024-03-28T16:05:38.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:05:38 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T16:05:38.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:05:38 smithi067 ceph-mon[31707]: pgmap v501: 97 pgs: 97 active+clean; 597 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 16 MiB/s wr, 5 op/s 2024-03-28T16:05:38.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:05:38 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T16:05:38.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:05:38 smithi183 ceph-mon[35580]: pgmap v501: 97 pgs: 97 active+clean; 597 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 16 MiB/s wr, 5 op/s 2024-03-28T16:05:38.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:05:38 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T16:05:40.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:05:40 smithi150 ceph-mon[34825]: pgmap v502: 97 pgs: 97 active+clean; 641 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 18 MiB/s wr, 6 op/s 2024-03-28T16:05:40.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:05:40 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:05:40.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:05:40 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T16:05:40.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:05:40 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:05:40.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:05:40 smithi067 ceph-mon[31707]: pgmap v502: 97 pgs: 97 active+clean; 641 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 18 MiB/s wr, 6 op/s 2024-03-28T16:05:40.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:05:40 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:05:40.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:05:40 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T16:05:40.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:05:40 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:05:40.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:05:40 smithi183 ceph-mon[35580]: pgmap v502: 97 pgs: 97 active+clean; 641 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 18 MiB/s wr, 6 op/s 2024-03-28T16:05:40.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:05:40 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:05:40.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:05:40 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T16:05:40.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:05:40 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:05:42.290 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T16:05:42.291 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T16:05:42.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.540+0000 7fd98ec00640 1 -- 172.21.15.67:0/1017902434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd988107260 msgr2=0x7fd9881096b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:42.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.540+0000 7fd98ec00640 1 --2- 172.21.15.67:0/1017902434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd988107260 0x7fd9881096b0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fd980009be0 tx=0x7fd98002bef0 comp rx=0 tx=0).stop 2024-03-28T16:05:42.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.540+0000 7fd98ec00640 1 -- 172.21.15.67:0/1017902434 shutdown_connections 2024-03-28T16:05:42.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.540+0000 7fd98ec00640 1 --2- 172.21.15.67:0/1017902434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd988107260 0x7fd9881096b0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:42.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.540+0000 7fd98ec00640 1 --2- 172.21.15.67:0/1017902434 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd988104930 0x7fd988106d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:42.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.540+0000 7fd98ec00640 1 --2- 172.21.15.67:0/1017902434 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd988102000 0x7fd9881043f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:42.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.540+0000 7fd98ec00640 1 -- 172.21.15.67:0/1017902434 >> 172.21.15.67:0/1017902434 conn(0x7fd9880fba10 msgr2=0x7fd9880fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:05:42.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.540+0000 7fd98ec00640 1 -- 172.21.15.67:0/1017902434 shutdown_connections 2024-03-28T16:05:42.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.540+0000 7fd98ec00640 1 -- 172.21.15.67:0/1017902434 wait complete. 2024-03-28T16:05:42.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.541+0000 7fd98ec00640 1 Processor -- start 2024-03-28T16:05:42.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.541+0000 7fd98ec00640 1 -- start start 2024-03-28T16:05:42.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.542+0000 7fd98ec00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd988102000 0x7fd9881999a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:42.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.542+0000 7fd98ec00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd988104930 0x7fd988199ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:42.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.542+0000 7fd98ec00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd988107260 0x7fd9881a0f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:42.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.542+0000 7fd98ec00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd98810d110 con 0x7fd988107260 2024-03-28T16:05:42.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.542+0000 7fd98ec00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd98810cf90 con 0x7fd988102000 2024-03-28T16:05:42.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.542+0000 7fd98ec00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd98810d290 con 0x7fd988104930 2024-03-28T16:05:42.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.542+0000 7fd98e200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd988107260 0x7fd9881a0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:42.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.542+0000 7fd98e200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd988107260 0x7fd9881a0f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50896/0 (socket says 172.21.15.67:50896) 2024-03-28T16:05:42.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.542+0000 7fd98e200640 1 -- 172.21.15.67:0/4269052000 learned_addr learned my addr 172.21.15.67:0/4269052000 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:05:42.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.542+0000 7fd98ce00640 1 --2- 172.21.15.67:0/4269052000 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd988104930 0x7fd988199ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:42.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.542+0000 7fd98d800640 1 --2- 172.21.15.67:0/4269052000 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd988102000 0x7fd9881999a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:42.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.543+0000 7fd98e200640 1 -- 172.21.15.67:0/4269052000 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd988104930 msgr2=0x7fd988199ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:42.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.543+0000 7fd98e200640 1 --2- 172.21.15.67:0/4269052000 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd988104930 0x7fd988199ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:42.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.543+0000 7fd98e200640 1 -- 172.21.15.67:0/4269052000 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd988102000 msgr2=0x7fd9881999a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:42.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.543+0000 7fd98e200640 1 --2- 172.21.15.67:0/4269052000 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd988102000 0x7fd9881999a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:42.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.543+0000 7fd98e200640 1 -- 172.21.15.67:0/4269052000 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9881a15c0 con 0x7fd988107260 2024-03-28T16:05:42.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.543+0000 7fd98e200640 1 --2- 172.21.15.67:0/4269052000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd988107260 0x7fd9881a0f50 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fd980034fd0 tx=0x7fd980033c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:05:42.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.543+0000 7fd986000640 1 -- 172.21.15.67:0/4269052000 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd980043020 con 0x7fd988107260 2024-03-28T16:05:42.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.544+0000 7fd986000640 1 -- 172.21.15.67:0/4269052000 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd980003e20 con 0x7fd988107260 2024-03-28T16:05:42.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.544+0000 7fd98ec00640 1 -- 172.21.15.67:0/4269052000 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9881a1850 con 0x7fd988107260 2024-03-28T16:05:42.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.544+0000 7fd986000640 1 -- 172.21.15.67:0/4269052000 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd980034430 con 0x7fd988107260 2024-03-28T16:05:42.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.544+0000 7fd98ec00640 1 -- 172.21.15.67:0/4269052000 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9881a1da0 con 0x7fd988107260 2024-03-28T16:05:42.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.547+0000 7fd992f9d740 1 Processor -- start 2024-03-28T16:05:42.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.548+0000 7fd986000640 1 -- 172.21.15.67:0/4269052000 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fd980046020 con 0x7fd988107260 2024-03-28T16:05:42.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.548+0000 7fd992f9d740 1 -- start start 2024-03-28T16:05:42.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.548+0000 7fd992f9d740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55811e279050 0x55811e279470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:42.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.548+0000 7fd992f9d740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55811e279a40 0x55811e27e540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:42.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.548+0000 7fd992f9d740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55811e27ebb0 0x55811e281040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:42.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.548+0000 7fd992f9d740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55811e1de250 con 0x55811e279050 2024-03-28T16:05:42.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.548+0000 7fd992f9d740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55811e186a30 con 0x55811e279a40 2024-03-28T16:05:42.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.548+0000 7fd992f9d740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55811e1da2f0 con 0x55811e27ebb0 2024-03-28T16:05:42.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.548+0000 7fd98ce00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55811e279050 0x55811e279470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:42.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.548+0000 7fd98ce00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55811e279050 0x55811e279470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50902/0 (socket says 172.21.15.67:50902) 2024-03-28T16:05:42.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.548+0000 7fd98ce00640 1 -- 172.21.15.67:0/2931172344 learned_addr learned my addr 172.21.15.67:0/2931172344 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:05:42.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.549+0000 7fd98d800640 1 --2- 172.21.15.67:0/2931172344 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55811e279a40 0x55811e27e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:42.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.549+0000 7fd98ce00640 1 --2- 172.21.15.67:0/2931172344 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55811e27ebb0 0x55811e281040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:42.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.549+0000 7fd98ce00640 1 -- 172.21.15.67:0/2931172344 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55811e27ebb0 msgr2=0x55811e281040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:05:42.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.549+0000 7fd98ce00640 1 --2- 172.21.15.67:0/2931172344 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55811e27ebb0 0x55811e281040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:42.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.549+0000 7fd98ce00640 1 -- 172.21.15.67:0/2931172344 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55811e279a40 msgr2=0x55811e27e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:05:42.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.549+0000 7fd98ce00640 1 --2- 172.21.15.67:0/2931172344 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55811e279a40 0x55811e27e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:42.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.549+0000 7fd98ce00640 1 -- 172.21.15.67:0/2931172344 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55811e171690 con 0x55811e279050 2024-03-28T16:05:42.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.549+0000 7fd98ce00640 1 --2- 172.21.15.67:0/2931172344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55811e279050 0x55811e279470 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fd974009e20 tx=0x7fd974009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=283fe96b56ebe683 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:05:42.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.550+0000 7fd986000640 1 --2- 172.21.15.67:0/4269052000 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fd9700a2800 0x7fd9700a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:42.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.550+0000 7fd97aa00640 1 -- 172.21.15.67:0/2931172344 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd974004030 con 0x55811e279050 2024-03-28T16:05:42.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.550+0000 7fd97aa00640 1 -- 172.21.15.67:0/2931172344 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd9740035f0 con 0x55811e279050 2024-03-28T16:05:42.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.550+0000 7fd97aa00640 1 -- 172.21.15.67:0/2931172344 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd974011670 con 0x55811e279050 2024-03-28T16:05:42.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.550+0000 7fd986000640 1 -- 172.21.15.67:0/4269052000 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fd9800f1250 con 0x7fd988107260 2024-03-28T16:05:42.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.550+0000 7fd98d800640 1 --2- 172.21.15.67:0/4269052000 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fd9700a2800 0x7fd9700a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:42.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.551+0000 7fd992f9d740 1 -- 172.21.15.67:0/2931172344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55811e279050 msgr2=0x55811e279470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:42.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.551+0000 7fd992f9d740 1 --2- 172.21.15.67:0/2931172344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55811e279050 0x55811e279470 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fd974009e20 tx=0x7fd974009750 comp rx=0 tx=0).stop 2024-03-28T16:05:42.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.551+0000 7fd992f9d740 1 -- 172.21.15.67:0/2931172344 shutdown_connections 2024-03-28T16:05:42.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.551+0000 7fd992f9d740 1 --2- 172.21.15.67:0/2931172344 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55811e27ebb0 0x55811e281040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:42.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.551+0000 7fd992f9d740 1 --2- 172.21.15.67:0/2931172344 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55811e279a40 0x55811e27e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:42.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.551+0000 7fd992f9d740 1 --2- 172.21.15.67:0/2931172344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55811e279050 0x55811e279470 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:42.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.551+0000 7fd992f9d740 1 -- 172.21.15.67:0/2931172344 >> 172.21.15.67:0/2931172344 conn(0x55811e241200 msgr2=0x55811e2749c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:05:42.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.551+0000 7fd98d800640 1 --2- 172.21.15.67:0/4269052000 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fd9700a2800 0x7fd9700a4cc0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x55811e279e40 tx=0x7fd97c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:05:42.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.551+0000 7fd992f9d740 1 -- 172.21.15.67:0/2931172344 shutdown_connections 2024-03-28T16:05:42.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.551+0000 7fd992f9d740 1 -- 172.21.15.67:0/2931172344 wait complete. 2024-03-28T16:05:42.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.552+0000 7fd992f9d740 1 -- start start 2024-03-28T16:05:42.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.552+0000 7fd992f9d740 1 Processor -- start 2024-03-28T16:05:42.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.552+0000 7fd992f9d740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55811e279050 0x55811e283080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:42.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.552+0000 7fd992f9d740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55811e279a40 0x55811e2835c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:42.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.552+0000 7fd992f9d740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55811e27ebb0 0x55811e28a600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:42.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.553+0000 7fd992f9d740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55811e281b60 con 0x55811e27ebb0 2024-03-28T16:05:42.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.553+0000 7fd992f9d740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55811e199e80 con 0x55811e279a40 2024-03-28T16:05:42.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.553+0000 7fd992f9d740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55811e281ce0 con 0x55811e279050 2024-03-28T16:05:42.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.553+0000 7fd992f9d740 10 client.0 _ll_register_callbacks cb 0x55811e1ac130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:05:42.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.553+0000 7fd98e200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55811e27ebb0 0x55811e28a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:42.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.553+0000 7fd98e200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55811e27ebb0 0x55811e28a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50916/0 (socket says 172.21.15.67:50916) 2024-03-28T16:05:42.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.553+0000 7fd98e200640 1 -- 172.21.15.67:0/2203958336 learned_addr learned my addr 172.21.15.67:0/2203958336 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:05:42.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.553+0000 7fd98d800640 1 --2- 172.21.15.67:0/2203958336 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55811e279050 0x55811e283080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:42.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.553+0000 7fd98ce00640 1 --2- 172.21.15.67:0/2203958336 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55811e279a40 0x55811e2835c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:42.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.553+0000 7fd98e200640 1 -- 172.21.15.67:0/2203958336 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55811e279050 msgr2=0x55811e283080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:05:42.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.553+0000 7fd98e200640 1 --2- 172.21.15.67:0/2203958336 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55811e279050 0x55811e283080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:42.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.553+0000 7fd98e200640 1 -- 172.21.15.67:0/2203958336 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55811e279a40 msgr2=0x55811e2835c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:05:42.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.553+0000 7fd98e200640 1 --2- 172.21.15.67:0/2203958336 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55811e279a40 0x55811e2835c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:42.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.553+0000 7fd98e200640 1 -- 172.21.15.67:0/2203958336 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55811dfbc7d0 con 0x55811e27ebb0 2024-03-28T16:05:42.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.553+0000 7fd98d800640 1 --2- 172.21.15.67:0/2203958336 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55811e279050 0x55811e283080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T16:05:42.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.554+0000 7fd98e200640 1 --2- 172.21.15.67:0/2203958336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55811e27ebb0 0x55811e28a600 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fd9800c5e10 tx=0x7fd9800bd500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98c40b707a884843 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:05:42.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.554+0000 7fd96f400640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:05:42.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.554+0000 7fd96f400640 1 -- 172.21.15.67:0/2203958336 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9800d6060 con 0x55811e27ebb0 2024-03-28T16:05:42.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.554+0000 7fd96f400640 1 -- 172.21.15.67:0/2203958336 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fd980003970 con 0x55811e27ebb0 2024-03-28T16:05:42.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.554+0000 7fd992f9d740 1 -- 172.21.15.67:0/2203958336 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd9740035f0 con 0x55811e27ebb0 2024-03-28T16:05:42.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.554+0000 7fd96f400640 1 -- 172.21.15.67:0/2203958336 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd9800be2a0 con 0x55811e27ebb0 2024-03-28T16:05:42.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.554+0000 7fd992f9d740 10 client.15573 fetch_fsmap learned FSMap version 19 2024-03-28T16:05:42.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.554+0000 7fd992f9d740 1 -- 172.21.15.67:0/2203958336 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55811def23f0 con 0x55811e27ebb0 2024-03-28T16:05:42.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.555+0000 7fd96f400640 1 -- 172.21.15.67:0/2203958336 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd9740035f0 con 0x55811e27ebb0 2024-03-28T16:05:42.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.555+0000 7fd96f400640 1 -- 172.21.15.67:0/2203958336 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fd980004b40 con 0x55811e27ebb0 2024-03-28T16:05:42.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.555+0000 7fd992f9d740 10 client.15573 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:05:42.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.555+0000 7fd992f9d740 10 client.15573 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:05:42.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.555+0000 7fd992f9d740 20 client.15573 populate_metadata read hostname 'smithi067' 2024-03-28T16:05:42.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.555+0000 7fd992f9d740 1 --2- 172.21.15.67:0/2203958336 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55811e28ee80 0x55811e2912a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:42.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.555+0000 7fd992f9d740 4 client.15573 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:05:42.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.555+0000 7fd992f9d740 1 -- 172.21.15.67:0/2203958336 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55811e1eb790 con 0x55811e28ee80 2024-03-28T16:05:42.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.556+0000 7fd98d800640 1 --2- 172.21.15.67:0/2203958336 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55811e28ee80 0x55811e2912a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:42.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.557+0000 7fd98d800640 1 --2- 172.21.15.67:0/2203958336 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55811e28ee80 0x55811e2912a0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=510cb4ebee4269f4 server_cookie=6313a7193231c619 in_seq=0 out_seq=0 2024-03-28T16:05:42.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.557+0000 7fd96f400640 10 client.15573 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:05:42.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.559+0000 7fd96f400640 1 -- 172.21.15.67:0/2203958336 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55811e1eb790 con 0x55811e28ee80 2024-03-28T16:05:42.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.559+0000 7fd96f400640 10 client.15573 handle_command_reply: tid=0 2024-03-28T16:05:42.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.559+0000 7fd992f9d740 1 client.15573 shutdown 2024-03-28T16:05:42.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.559+0000 7fd992f9d740 1 -- 172.21.15.67:0/2203958336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55811e27ebb0 msgr2=0x55811e28a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:42.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.559+0000 7fd992f9d740 1 --2- 172.21.15.67:0/2203958336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55811e27ebb0 0x55811e28a600 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fd9800c5e10 tx=0x7fd9800bd500 comp rx=0 tx=0).stop 2024-03-28T16:05:42.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.559+0000 7fd992f9d740 1 -- 172.21.15.67:0/2203958336 shutdown_connections 2024-03-28T16:05:42.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.559+0000 7fd992f9d740 1 --2- 172.21.15.67:0/2203958336 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55811e28ee80 0x55811e2912a0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:42.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.559+0000 7fd992f9d740 1 --2- 172.21.15.67:0/2203958336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55811e27ebb0 0x55811e28a600 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:42.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.559+0000 7fd992f9d740 1 --2- 172.21.15.67:0/2203958336 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55811e279a40 0x55811e2835c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:42.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.559+0000 7fd992f9d740 1 --2- 172.21.15.67:0/2203958336 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55811e279050 0x55811e283080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:42.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.559+0000 7fd992f9d740 1 -- 172.21.15.67:0/2203958336 >> 172.21.15.67:0/2203958336 conn(0x55811e241200 msgr2=0x55811e273bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:05:42.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.559+0000 7fd96f400640 0 client.15573 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:05:42.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.559+0000 7fd992f9d740 1 -- 172.21.15.67:0/2203958336 shutdown_connections 2024-03-28T16:05:42.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.559+0000 7fd992f9d740 1 -- 172.21.15.67:0/2203958336 wait complete. 2024-03-28T16:05:42.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.559+0000 7fd992f9d740 20 client.15573 trim_cache size 0 max 16384 2024-03-28T16:05:42.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.579+0000 7fd992f9d740 1 Processor -- start 2024-03-28T16:05:42.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.579+0000 7fd992f9d740 1 -- start start 2024-03-28T16:05:42.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.579+0000 7fd992f9d740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55811e279050 0x55811e27e040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:42.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.579+0000 7fd992f9d740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55811e279a40 0x55811e28a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:42.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.579+0000 7fd992f9d740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55811e27ebb0 0x55811e27abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:42.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.579+0000 7fd992f9d740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9881a8920 con 0x55811e279050 2024-03-28T16:05:42.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.579+0000 7fd992f9d740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9881aaee0 con 0x55811e279a40 2024-03-28T16:05:42.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.579+0000 7fd992f9d740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9881a8f20 con 0x55811e27ebb0 2024-03-28T16:05:42.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.580+0000 7fd98d800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55811e279050 0x55811e27e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:42.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.580+0000 7fd98d800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55811e279050 0x55811e27e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50924/0 (socket says 172.21.15.67:50924) 2024-03-28T16:05:42.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.580+0000 7fd98d800640 1 -- 172.21.15.67:0/2242139071 learned_addr learned my addr 172.21.15.67:0/2242139071 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:05:42.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.580+0000 7fd98e200640 1 --2- 172.21.15.67:0/2242139071 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55811e27ebb0 0x55811e27abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:42.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.580+0000 7fd98d800640 1 -- 172.21.15.67:0/2242139071 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55811e27ebb0 msgr2=0x55811e27abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:05:42.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.580+0000 7fd98d800640 1 --2- 172.21.15.67:0/2242139071 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55811e27ebb0 0x55811e27abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:42.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.580+0000 7fd98d800640 1 -- 172.21.15.67:0/2242139071 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55811e279a40 msgr2=0x55811e28a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:05:42.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.580+0000 7fd98d800640 1 --2- 172.21.15.67:0/2242139071 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55811e279a40 0x55811e28a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:42.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.580+0000 7fd98d800640 1 -- 172.21.15.67:0/2242139071 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55811deb20f0 con 0x55811e279050 2024-03-28T16:05:42.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.580+0000 7fd98e200640 1 --2- 172.21.15.67:0/2242139071 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55811e27ebb0 0x55811e27abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:05:42.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.580+0000 7fd98d800640 1 --2- 172.21.15.67:0/2242139071 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55811e279050 0x55811e27e040 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fd97c011550 tx=0x7fd97c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b5324a331c901ea server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:05:42.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.581+0000 7fd97a000640 1 -- 172.21.15.67:0/2242139071 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd97c005020 con 0x55811e279050 2024-03-28T16:05:42.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.581+0000 7fd97a000640 1 -- 172.21.15.67:0/2242139071 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd97c0047a0 con 0x55811e279050 2024-03-28T16:05:42.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.581+0000 7fd97a000640 1 -- 172.21.15.67:0/2242139071 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd97c011850 con 0x55811e279050 2024-03-28T16:05:42.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.582+0000 7fd992f9d740 1 -- 172.21.15.67:0/2242139071 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55811e279050 msgr2=0x55811e27e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:42.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.582+0000 7fd992f9d740 1 --2- 172.21.15.67:0/2242139071 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55811e279050 0x55811e27e040 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fd97c011550 tx=0x7fd97c016000 comp rx=0 tx=0).stop 2024-03-28T16:05:42.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.582+0000 7fd992f9d740 1 -- 172.21.15.67:0/2242139071 shutdown_connections 2024-03-28T16:05:42.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.582+0000 7fd992f9d740 1 --2- 172.21.15.67:0/2242139071 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55811e27ebb0 0x55811e27abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:42.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.582+0000 7fd992f9d740 1 --2- 172.21.15.67:0/2242139071 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55811e279a40 0x55811e28a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:42.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.582+0000 7fd992f9d740 1 --2- 172.21.15.67:0/2242139071 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55811e279050 0x55811e27e040 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:42.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.582+0000 7fd992f9d740 1 -- 172.21.15.67:0/2242139071 >> 172.21.15.67:0/2242139071 conn(0x55811e241200 msgr2=0x55811e27a4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:05:42.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.582+0000 7fd992f9d740 1 -- 172.21.15.67:0/2242139071 shutdown_connections 2024-03-28T16:05:42.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.582+0000 7fd992f9d740 1 -- 172.21.15.67:0/2242139071 wait complete. 2024-03-28T16:05:42.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.582+0000 7fd992f9d740 1 -- start start 2024-03-28T16:05:42.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.582+0000 7fd992f9d740 1 Processor -- start 2024-03-28T16:05:42.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.583+0000 7fd992f9d740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55811e279050 0x55811e2881b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:42.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.583+0000 7fd992f9d740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55811e279a40 0x55811e2886f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:42.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.583+0000 7fd992f9d740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55811e27ebb0 0x55811e289700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:42.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.583+0000 7fd992f9d740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9881a9560 con 0x55811e279a40 2024-03-28T16:05:42.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.583+0000 7fd992f9d740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9881aa880 con 0x55811e279050 2024-03-28T16:05:42.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.583+0000 7fd992f9d740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9881a9bc0 con 0x55811e27ebb0 2024-03-28T16:05:42.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.583+0000 7fd992f9d740 10 client.0 _ll_register_callbacks cb 0x7fd9800bd970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:05:42.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.583+0000 7fd98ce00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55811e279a40 0x55811e2886f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:42.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.583+0000 7fd98ce00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55811e279a40 0x55811e2886f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50928/0 (socket says 172.21.15.67:50928) 2024-03-28T16:05:42.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.583+0000 7fd98ce00640 1 -- 172.21.15.67:0/3517184877 learned_addr learned my addr 172.21.15.67:0/3517184877 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:05:42.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.583+0000 7fd98d800640 1 --2- 172.21.15.67:0/3517184877 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55811e279050 0x55811e2881b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:42.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.584+0000 7fd98e200640 1 --2- 172.21.15.67:0/3517184877 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55811e27ebb0 0x55811e289700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:42.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.584+0000 7fd98ce00640 1 -- 172.21.15.67:0/3517184877 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55811e27ebb0 msgr2=0x55811e289700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:05:42.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.584+0000 7fd98ce00640 1 --2- 172.21.15.67:0/3517184877 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55811e27ebb0 0x55811e289700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:42.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.584+0000 7fd98ce00640 1 -- 172.21.15.67:0/3517184877 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55811e279050 msgr2=0x55811e2881b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:05:42.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.584+0000 7fd98ce00640 1 --2- 172.21.15.67:0/3517184877 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55811e279050 0x55811e2881b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:42.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.584+0000 7fd98ce00640 1 -- 172.21.15.67:0/3517184877 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55811e289cf0 con 0x55811e279a40 2024-03-28T16:05:42.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.584+0000 7fd98d800640 1 --2- 172.21.15.67:0/3517184877 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55811e279050 0x55811e2881b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:05:42.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.584+0000 7fd98e200640 1 --2- 172.21.15.67:0/3517184877 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55811e27ebb0 0x55811e289700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:05:42.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.584+0000 7fd98ce00640 1 --2- 172.21.15.67:0/3517184877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55811e279a40 0x55811e2886f0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fd974009e20 tx=0x7fd974009460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=700c7736c49d5614 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:05:42.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.584+0000 7fd96f400640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:05:42.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.584+0000 7fd96f400640 1 -- 172.21.15.67:0/3517184877 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd974003a00 con 0x55811e279a40 2024-03-28T16:05:42.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.584+0000 7fd96f400640 1 -- 172.21.15.67:0/3517184877 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fd974003ba0 con 0x55811e279a40 2024-03-28T16:05:42.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.584+0000 7fd992f9d740 1 -- 172.21.15.67:0/3517184877 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd97c0047a0 con 0x55811e279a40 2024-03-28T16:05:42.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.585+0000 7fd96f400640 1 -- 172.21.15.67:0/3517184877 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd9740099a0 con 0x55811e279a40 2024-03-28T16:05:42.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.585+0000 7fd992f9d740 10 client.15579 fetch_fsmap learned FSMap version 19 2024-03-28T16:05:42.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.585+0000 7fd992f9d740 1 -- 172.21.15.67:0/3517184877 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55811e2eac10 con 0x55811e279a40 2024-03-28T16:05:42.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.585+0000 7fd96f400640 1 -- 172.21.15.67:0/3517184877 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd97c0047a0 con 0x55811e279a40 2024-03-28T16:05:42.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.585+0000 7fd96f400640 1 -- 172.21.15.67:0/3517184877 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fd97401d410 con 0x55811e279a40 2024-03-28T16:05:42.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.585+0000 7fd992f9d740 10 client.15579 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:05:42.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.585+0000 7fd992f9d740 10 client.15579 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:05:42.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.585+0000 7fd992f9d740 20 client.15579 populate_metadata read hostname 'smithi067' 2024-03-28T16:05:42.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.585+0000 7fd992f9d740 1 --2- 172.21.15.67:0/3517184877 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55811e2eb410 0x55811e2ed830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:42.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.585+0000 7fd992f9d740 4 client.15579 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T16:05:42.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.585+0000 7fd992f9d740 1 -- 172.21.15.67:0/3517184877 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55811e1e5ae0 con 0x55811e2eb410 2024-03-28T16:05:42.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.586+0000 7fd98d800640 1 --2- 172.21.15.67:0/3517184877 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55811e2eb410 0x55811e2ed830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:42.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.587+0000 7fd98d800640 1 --2- 172.21.15.67:0/3517184877 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55811e2eb410 0x55811e2ed830 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b48391e181b15609 server_cookie=ff7489807a1058d0 in_seq=0 out_seq=0 2024-03-28T16:05:42.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.587+0000 7fd96f400640 10 client.15579 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:05:42.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.587+0000 7fd96f400640 1 -- 172.21.15.67:0/3517184877 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55811e1e5ae0 con 0x55811e2eb410 2024-03-28T16:05:42.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.587+0000 7fd96f400640 10 client.15579 handle_command_reply: tid=0 2024-03-28T16:05:42.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.587+0000 7fd992f9d740 1 client.15579 shutdown 2024-03-28T16:05:42.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.588+0000 7fd992f9d740 1 -- 172.21.15.67:0/3517184877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55811e279a40 msgr2=0x55811e2886f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:42.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.588+0000 7fd992f9d740 1 --2- 172.21.15.67:0/3517184877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55811e279a40 0x55811e2886f0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fd974009e20 tx=0x7fd974009460 comp rx=0 tx=0).stop 2024-03-28T16:05:42.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.588+0000 7fd992f9d740 1 -- 172.21.15.67:0/3517184877 shutdown_connections 2024-03-28T16:05:42.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.588+0000 7fd992f9d740 1 --2- 172.21.15.67:0/3517184877 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55811e2eb410 0x55811e2ed830 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:42.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.588+0000 7fd992f9d740 1 --2- 172.21.15.67:0/3517184877 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55811e27ebb0 0x55811e289700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:42.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.588+0000 7fd992f9d740 1 --2- 172.21.15.67:0/3517184877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55811e279a40 0x55811e2886f0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:42.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.588+0000 7fd992f9d740 1 --2- 172.21.15.67:0/3517184877 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55811e279050 0x55811e2881b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:42.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.588+0000 7fd992f9d740 1 -- 172.21.15.67:0/3517184877 >> 172.21.15.67:0/3517184877 conn(0x55811e241200 msgr2=0x55811e277c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:05:42.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.588+0000 7fd96f400640 0 client.15579 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:05:42.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.588+0000 7fd992f9d740 1 -- 172.21.15.67:0/3517184877 shutdown_connections 2024-03-28T16:05:42.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.588+0000 7fd992f9d740 1 -- 172.21.15.67:0/3517184877 wait complete. 2024-03-28T16:05:42.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.588+0000 7fd992f9d740 20 client.15579 trim_cache size 0 max 16384 2024-03-28T16:05:42.592 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-28T16:05:42.592 INFO:teuthology.orchestra.run.smithi067.stdout: "status": "no active scrubs running", 2024-03-28T16:05:42.592 INFO:teuthology.orchestra.run.smithi067.stdout: "scrubs": {} 2024-03-28T16:05:42.592 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-28T16:05:42.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.589+0000 7fd98ec00640 1 -- 172.21.15.67:0/4269052000 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fd9700a2800 msgr2=0x7fd9700a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:42.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.589+0000 7fd98ec00640 1 --2- 172.21.15.67:0/4269052000 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fd9700a2800 0x7fd9700a4cc0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x55811e279e40 tx=0x7fd97c005f90 comp rx=0 tx=0).stop 2024-03-28T16:05:42.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.589+0000 7fd98ec00640 1 -- 172.21.15.67:0/4269052000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd988107260 msgr2=0x7fd9881a0f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:42.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.589+0000 7fd98ec00640 1 --2- 172.21.15.67:0/4269052000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd988107260 0x7fd9881a0f50 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fd980034fd0 tx=0x7fd980033c40 comp rx=0 tx=0).stop 2024-03-28T16:05:42.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.589+0000 7fd98ec00640 1 -- 172.21.15.67:0/4269052000 shutdown_connections 2024-03-28T16:05:42.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.589+0000 7fd98ec00640 1 --2- 172.21.15.67:0/4269052000 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fd9700a2800 0x7fd9700a4cc0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:42.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.589+0000 7fd98ec00640 1 --2- 172.21.15.67:0/4269052000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd988107260 0x7fd9881a0f50 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:42.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.589+0000 7fd98ec00640 1 --2- 172.21.15.67:0/4269052000 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd988104930 0x7fd988199ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:42.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.589+0000 7fd98ec00640 1 --2- 172.21.15.67:0/4269052000 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd988102000 0x7fd9881999a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:42.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.589+0000 7fd98ec00640 1 -- 172.21.15.67:0/4269052000 >> 172.21.15.67:0/4269052000 conn(0x7fd9880fba10 msgr2=0x7fd988101680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:05:42.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.589+0000 7fd98ec00640 1 -- 172.21.15.67:0/4269052000 shutdown_connections 2024-03-28T16:05:42.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.589+0000 7fd98ec00640 1 -- 172.21.15.67:0/4269052000 wait complete. 2024-03-28T16:05:42.608 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T16:05:42.608 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T16:05:42.686 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:05:42 smithi067 ceph-mon[31707]: pgmap v503: 97 pgs: 97 active+clean; 697 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 20 MiB/s wr, 7 op/s 2024-03-28T16:05:42.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:05:42 smithi150 ceph-mon[34825]: pgmap v503: 97 pgs: 97 active+clean; 697 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 20 MiB/s wr, 7 op/s 2024-03-28T16:05:42.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.847+0000 7f46bee00640 1 -- 172.21.15.67:0/3954635792 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46b81016c0 msgr2=0x7f46b8101ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:42.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.847+0000 7f46bee00640 1 --2- 172.21.15.67:0/3954635792 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46b81016c0 0x7f46b8101ae0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f46a8009900 tx=0x7f46a802bc30 comp rx=0 tx=0).stop 2024-03-28T16:05:42.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.847+0000 7f46bee00640 1 -- 172.21.15.67:0/3954635792 shutdown_connections 2024-03-28T16:05:42.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.847+0000 7f46bee00640 1 --2- 172.21.15.67:0/3954635792 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f46b8103270 0x7f46b8109b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:42.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.847+0000 7f46bee00640 1 --2- 172.21.15.67:0/3954635792 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46b81028b0 0x7f46b8102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:42.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.847+0000 7f46bee00640 1 --2- 172.21.15.67:0/3954635792 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46b81016c0 0x7f46b8101ae0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:42.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.847+0000 7f46bee00640 1 -- 172.21.15.67:0/3954635792 >> 172.21.15.67:0/3954635792 conn(0x7f46b80fcc50 msgr2=0x7f46b80ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:05:42.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.847+0000 7f46bee00640 1 -- 172.21.15.67:0/3954635792 shutdown_connections 2024-03-28T16:05:42.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.847+0000 7f46bee00640 1 -- 172.21.15.67:0/3954635792 wait complete. 2024-03-28T16:05:42.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.848+0000 7f46bee00640 1 Processor -- start 2024-03-28T16:05:42.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.849+0000 7f46bee00640 1 -- start start 2024-03-28T16:05:42.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.849+0000 7f46bee00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46b81016c0 0x7f46b8199af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:42.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.849+0000 7f46bee00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f46b81028b0 0x7f46b819a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:42.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.849+0000 7f46bee00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46b8103270 0x7f46b81a10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:42.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.849+0000 7f46bee00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46b810d210 con 0x7f46b81016c0 2024-03-28T16:05:42.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.849+0000 7f46bee00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46b810d090 con 0x7f46b8103270 2024-03-28T16:05:42.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.849+0000 7f46bee00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46b810d390 con 0x7f46b81028b0 2024-03-28T16:05:42.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.849+0000 7f46b7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46b81016c0 0x7f46b8199af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:42.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.849+0000 7f46b7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46b81016c0 0x7f46b8199af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50956/0 (socket says 172.21.15.67:50956) 2024-03-28T16:05:42.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.850+0000 7f46b7e00640 1 -- 172.21.15.67:0/953677247 learned_addr learned my addr 172.21.15.67:0/953677247 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:05:42.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.850+0000 7f46b7400640 1 --2- 172.21.15.67:0/953677247 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f46b81028b0 0x7f46b819a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:42.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.850+0000 7f46bce00640 1 --2- 172.21.15.67:0/953677247 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46b8103270 0x7f46b81a10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:42.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.850+0000 7f46b7e00640 1 -- 172.21.15.67:0/953677247 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f46b81028b0 msgr2=0x7f46b819a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:42.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.850+0000 7f46b7e00640 1 --2- 172.21.15.67:0/953677247 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f46b81028b0 0x7f46b819a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:42.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.850+0000 7f46b7e00640 1 -- 172.21.15.67:0/953677247 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46b8103270 msgr2=0x7f46b81a10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:42.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.850+0000 7f46b7e00640 1 --2- 172.21.15.67:0/953677247 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46b8103270 0x7f46b81a10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:42.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.850+0000 7f46b7e00640 1 -- 172.21.15.67:0/953677247 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46b81a1710 con 0x7f46b81016c0 2024-03-28T16:05:42.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.850+0000 7f46bce00640 1 --2- 172.21.15.67:0/953677247 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46b8103270 0x7f46b81a10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:05:42.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.850+0000 7f46b7e00640 1 --2- 172.21.15.67:0/953677247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46b81016c0 0x7f46b8199af0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f46a8005ab0 tx=0x7f46a80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:05:42.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.851+0000 7f46b4c00640 1 -- 172.21.15.67:0/953677247 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46a8043020 con 0x7f46b81016c0 2024-03-28T16:05:42.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.851+0000 7f46bee00640 1 -- 172.21.15.67:0/953677247 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46b81a19a0 con 0x7f46b81016c0 2024-03-28T16:05:42.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.851+0000 7f46b4c00640 1 -- 172.21.15.67:0/953677247 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f46a80054c0 con 0x7f46b81016c0 2024-03-28T16:05:42.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.851+0000 7f46b4c00640 1 -- 172.21.15.67:0/953677247 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46a8003d10 con 0x7f46b81016c0 2024-03-28T16:05:42.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.851+0000 7f46bee00640 1 -- 172.21.15.67:0/953677247 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46b81a1f40 con 0x7f46b81016c0 2024-03-28T16:05:42.854 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:05:42 smithi183 ceph-mon[35580]: pgmap v503: 97 pgs: 97 active+clean; 697 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 20 MiB/s wr, 7 op/s 2024-03-28T16:05:42.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.852+0000 7f46bee00640 1 -- 172.21.15.67:0/953677247 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46b81a2110 con 0x7f46b81016c0 2024-03-28T16:05:42.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.854+0000 7f46b4c00640 1 -- 172.21.15.67:0/953677247 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f46a8046020 con 0x7f46b81016c0 2024-03-28T16:05:42.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.856+0000 7f46b4c00640 1 --2- 172.21.15.67:0/953677247 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f469c0a2880 0x7f469c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:42.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.856+0000 7f46b4c00640 1 -- 172.21.15.67:0/953677247 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f46a80f0e60 con 0x7f46b81016c0 2024-03-28T16:05:42.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.856+0000 7f46b7400640 1 --2- 172.21.15.67:0/953677247 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f469c0a2880 0x7f469c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:42.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.857+0000 7f46b7400640 1 --2- 172.21.15.67:0/953677247 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f469c0a2880 0x7f469c0a4d40 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f46a4005fd0 tx=0x7f46a4005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:05:42.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:42.862+0000 7f46b4c00640 1 -- 172.21.15.67:0/953677247 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f46a804d020 con 0x7f46b81016c0 2024-03-28T16:05:43.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.159+0000 7f46bee00640 1 -- 172.21.15.67:0/953677247 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f46b81a22e0 con 0x7f46b81016c0 2024-03-28T16:05:43.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.161+0000 7f46b4c00640 1 -- 172.21.15.67:0/953677247 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11341 (secure 0 0 0) 0x7f46a80bdfd0 con 0x7f46b81016c0 2024-03-28T16:05:43.163 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:05:43.163 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2008297659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2008297659},{"type":"v1","addr":"172.21.15.67:6835","nonce":2008297659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14532,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/320295745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":320295745},{"type":"v1","addr":"172.21.15.67:6837","nonce":320295745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6839/691821273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":691821273},{"type":"v1","addr":"172.21.15.67:6839","nonce":691821273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6841/3211254875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6840","nonce":3211254875},{"type":"v1","addr":"172.21.15.67:6841","nonce":3211254875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2129665423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2129665423},{"type":"v1","addr":"172.21.15.183:6833","nonce":2129665423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24496,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/1589784425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":1589784425},{"type":"v1","addr":"172.21.15.183:6835","nonce":1589784425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24508,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/2185863835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":2185863835},{"type":"v1","addr":"172.21.15.183:6837","nonce":2185863835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24520,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6839/1242814398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":1242814398},{"type":"v1","addr":"172.21.15.183:6839","nonce":1242814398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":34409,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1008335521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1008335521},{"type":"v1","addr":"172.21.15.150:6833","nonce":1008335521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":34445,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3269935695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3269935695},{"type":"v1","addr":"172.21.15.150:6835","nonce":3269935695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T16:01:07.696824+0000","modified":"2024-03-28T16:01:11.997773+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":34508},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34475":{"gid":34475,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/1112886277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":1112886277},{"type":"v1","addr":"172.21.15.150:6837","nonce":1112886277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_34508":{"gid":34508,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6839/1211011614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":1211011614},{"type":"v1","addr":"172.21.15.150:6839","nonce":1211011614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34508,"qdb_cluster":[34508]},"id":1}]} 2024-03-28T16:05:43.163 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 19 2024-03-28T16:05:43.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.164+0000 7f46bee00640 1 -- 172.21.15.67:0/953677247 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f469c0a2880 msgr2=0x7f469c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:43.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.164+0000 7f46bee00640 1 --2- 172.21.15.67:0/953677247 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f469c0a2880 0x7f469c0a4d40 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f46a4005fd0 tx=0x7f46a4005850 comp rx=0 tx=0).stop 2024-03-28T16:05:43.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.164+0000 7f46bee00640 1 -- 172.21.15.67:0/953677247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46b81016c0 msgr2=0x7f46b8199af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:43.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.164+0000 7f46bee00640 1 --2- 172.21.15.67:0/953677247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46b81016c0 0x7f46b8199af0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f46a8005ab0 tx=0x7f46a80042f0 comp rx=0 tx=0).stop 2024-03-28T16:05:43.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.165+0000 7f46bee00640 1 -- 172.21.15.67:0/953677247 shutdown_connections 2024-03-28T16:05:43.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.165+0000 7f46bee00640 1 --2- 172.21.15.67:0/953677247 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f469c0a2880 0x7f469c0a4d40 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:43.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.165+0000 7f46bee00640 1 --2- 172.21.15.67:0/953677247 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46b8103270 0x7f46b81a10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:43.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.165+0000 7f46bee00640 1 --2- 172.21.15.67:0/953677247 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f46b81028b0 0x7f46b819a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:43.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.165+0000 7f46bee00640 1 --2- 172.21.15.67:0/953677247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46b81016c0 0x7f46b8199af0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:43.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.165+0000 7f46bee00640 1 -- 172.21.15.67:0/953677247 >> 172.21.15.67:0/953677247 conn(0x7f46b80fcc50 msgr2=0x7f46b80738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:05:43.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.165+0000 7f46bee00640 1 -- 172.21.15.67:0/953677247 shutdown_connections 2024-03-28T16:05:43.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.165+0000 7f46bee00640 1 -- 172.21.15.67:0/953677247 wait complete. 2024-03-28T16:05:43.190 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T16:05:43.190 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T16:05:43.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.413+0000 7fb6c1e00640 1 -- 172.21.15.67:0/2792078341 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6bc101250 msgr2=0x7fb6bc1016d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:43.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.413+0000 7fb6c1e00640 1 --2- 172.21.15.67:0/2792078341 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6bc101250 0x7fb6bc1016d0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fb6b4009ca0 tx=0x7fb6b402c140 comp rx=0 tx=0).stop 2024-03-28T16:05:43.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.413+0000 7fb6c1e00640 1 -- 172.21.15.67:0/2792078341 shutdown_connections 2024-03-28T16:05:43.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.413+0000 7fb6c1e00640 1 --2- 172.21.15.67:0/2792078341 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6bc101250 0x7fb6bc1016d0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:43.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.413+0000 7fb6c1e00640 1 --2- 172.21.15.67:0/2792078341 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb6bc1008b0 0x7fb6bc100d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:43.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.413+0000 7fb6c1e00640 1 --2- 172.21.15.67:0/2792078341 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6bc0fff50 0x7fb6bc100370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:43.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.414+0000 7fb6c1e00640 1 -- 172.21.15.67:0/2792078341 >> 172.21.15.67:0/2792078341 conn(0x7fb6bc0fbb30 msgr2=0x7fb6bc0fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:05:43.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.414+0000 7fb6c1e00640 1 -- 172.21.15.67:0/2792078341 shutdown_connections 2024-03-28T16:05:43.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.414+0000 7fb6c1e00640 1 -- 172.21.15.67:0/2792078341 wait complete. 2024-03-28T16:05:43.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.415+0000 7fb6c1e00640 1 Processor -- start 2024-03-28T16:05:43.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.415+0000 7fb6c1e00640 1 -- start start 2024-03-28T16:05:43.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.415+0000 7fb6c1e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6bc0fff50 0x7fb6bc199ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:43.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.415+0000 7fb6c1e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb6bc1008b0 0x7fb6bc19a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:43.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.416+0000 7fb6c1e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6bc101250 0x7fb6bc1a1070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:43.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.416+0000 7fb6c1e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6bc10f3b0 con 0x7fb6bc0fff50 2024-03-28T16:05:43.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.416+0000 7fb6c1e00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6bc10f230 con 0x7fb6bc101250 2024-03-28T16:05:43.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.416+0000 7fb6c1e00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6bc10f530 con 0x7fb6bc1008b0 2024-03-28T16:05:43.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.416+0000 7fb6baa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6bc0fff50 0x7fb6bc199ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:43.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.416+0000 7fb6baa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6bc0fff50 0x7fb6bc199ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50972/0 (socket says 172.21.15.67:50972) 2024-03-28T16:05:43.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.416+0000 7fb6baa00640 1 -- 172.21.15.67:0/4152950361 learned_addr learned my addr 172.21.15.67:0/4152950361 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:05:43.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.416+0000 7fb6ba000640 1 --2- 172.21.15.67:0/4152950361 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb6bc1008b0 0x7fb6bc19a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:43.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.416+0000 7fb6bb400640 1 --2- 172.21.15.67:0/4152950361 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6bc101250 0x7fb6bc1a1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:43.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.416+0000 7fb6baa00640 1 -- 172.21.15.67:0/4152950361 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb6bc1008b0 msgr2=0x7fb6bc19a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:43.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.416+0000 7fb6baa00640 1 --2- 172.21.15.67:0/4152950361 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb6bc1008b0 0x7fb6bc19a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:43.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.416+0000 7fb6baa00640 1 -- 172.21.15.67:0/4152950361 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6bc101250 msgr2=0x7fb6bc1a1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:43.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.416+0000 7fb6baa00640 1 --2- 172.21.15.67:0/4152950361 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6bc101250 0x7fb6bc1a1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:43.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.416+0000 7fb6baa00640 1 -- 172.21.15.67:0/4152950361 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6bc1a16e0 con 0x7fb6bc0fff50 2024-03-28T16:05:43.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.417+0000 7fb6baa00640 1 --2- 172.21.15.67:0/4152950361 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6bc0fff50 0x7fb6bc199ac0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac00d950 tx=0x7fb6ac00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:05:43.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.417+0000 7fb6c0a00640 1 -- 172.21.15.67:0/4152950361 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6ac017070 con 0x7fb6bc0fff50 2024-03-28T16:05:43.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.417+0000 7fb6c0a00640 1 -- 172.21.15.67:0/4152950361 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb6ac009c30 con 0x7fb6bc0fff50 2024-03-28T16:05:43.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.417+0000 7fb6c1e00640 1 -- 172.21.15.67:0/4152950361 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6bc1a19d0 con 0x7fb6bc0fff50 2024-03-28T16:05:43.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.417+0000 7fb6c0a00640 1 -- 172.21.15.67:0/4152950361 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6ac00f620 con 0x7fb6bc0fff50 2024-03-28T16:05:43.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.417+0000 7fb6c1e00640 1 -- 172.21.15.67:0/4152950361 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6bc1a1f20 con 0x7fb6bc0fff50 2024-03-28T16:05:43.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.422+0000 7fb6c4bb3740 1 Processor -- start 2024-03-28T16:05:43.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.422+0000 7fb6c4bb3740 1 -- start start 2024-03-28T16:05:43.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.422+0000 7fb6c0a00640 1 -- 172.21.15.67:0/4152950361 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fb6ac0109a0 con 0x7fb6bc0fff50 2024-03-28T16:05:43.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.422+0000 7fb6c4bb3740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55768b4ad810 0x55768b4adc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:43.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.422+0000 7fb6c4bb3740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55768b4ae9e0 0x55768b4aee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:43.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.423+0000 7fb6c4bb3740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55768b4af3a0 0x55768b4b5c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:43.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.423+0000 7fb6c4bb3740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55768b4122c0 con 0x55768b4ad810 2024-03-28T16:05:43.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.423+0000 7fb6c4bb3740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55768b3bac90 con 0x55768b4ae9e0 2024-03-28T16:05:43.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.423+0000 7fb6c4bb3740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55768b40e360 con 0x55768b4af3a0 2024-03-28T16:05:43.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.423+0000 7fb6bb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55768b4ad810 0x55768b4adc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:43.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.423+0000 7fb6bb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55768b4ad810 0x55768b4adc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50980/0 (socket says 172.21.15.67:50980) 2024-03-28T16:05:43.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.423+0000 7fb6bb400640 1 -- 172.21.15.67:0/66691885 learned_addr learned my addr 172.21.15.67:0/66691885 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:05:43.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.423+0000 7fb6baa00640 1 --2- 172.21.15.67:0/66691885 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55768b4ae9e0 0x55768b4aee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:43.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.423+0000 7fb6ba000640 1 --2- 172.21.15.67:0/66691885 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55768b4af3a0 0x55768b4b5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:43.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.423+0000 7fb6bb400640 1 -- 172.21.15.67:0/66691885 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55768b4af3a0 msgr2=0x55768b4b5c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:05:43.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.423+0000 7fb6bb400640 1 --2- 172.21.15.67:0/66691885 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55768b4af3a0 0x55768b4b5c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:43.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.424+0000 7fb6bb400640 1 -- 172.21.15.67:0/66691885 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55768b4ae9e0 msgr2=0x55768b4aee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:05:43.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.424+0000 7fb6bb400640 1 --2- 172.21.15.67:0/66691885 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55768b4ae9e0 0x55768b4aee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:43.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.424+0000 7fb6bb400640 1 -- 172.21.15.67:0/66691885 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55768b3df080 con 0x55768b4ad810 2024-03-28T16:05:43.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.424+0000 7fb6baa00640 1 --2- 172.21.15.67:0/66691885 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55768b4ae9e0 0x55768b4aee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T16:05:43.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.424+0000 7fb6bb400640 1 --2- 172.21.15.67:0/66691885 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55768b4ad810 0x55768b4adc30 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fb6b4032550 tx=0x7fb6b40093e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1918b2d764a85430 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:05:43.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.424+0000 7fb6c0a00640 1 --2- 172.21.15.67:0/4152950361 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb6a00a2800 0x7fb6a00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:43.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.424+0000 7fb6c0a00640 1 -- 172.21.15.67:0/4152950361 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fb6ac0ce070 con 0x7fb6bc0fff50 2024-03-28T16:05:43.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.424+0000 7fb6a7e00640 1 -- 172.21.15.67:0/66691885 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6b4043020 con 0x55768b4ad810 2024-03-28T16:05:43.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.425+0000 7fb6a7e00640 1 -- 172.21.15.67:0/66691885 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb6b4032df0 con 0x55768b4ad810 2024-03-28T16:05:43.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.425+0000 7fb6a7e00640 1 -- 172.21.15.67:0/66691885 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6b4039040 con 0x55768b4ad810 2024-03-28T16:05:43.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.425+0000 7fb6baa00640 1 --2- 172.21.15.67:0/4152950361 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb6a00a2800 0x7fb6a00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:43.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.425+0000 7fb6c4bb3740 1 -- 172.21.15.67:0/66691885 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55768b4ad810 msgr2=0x55768b4adc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:43.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.425+0000 7fb6c4bb3740 1 --2- 172.21.15.67:0/66691885 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55768b4ad810 0x55768b4adc30 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fb6b4032550 tx=0x7fb6b40093e0 comp rx=0 tx=0).stop 2024-03-28T16:05:43.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.425+0000 7fb6baa00640 1 --2- 172.21.15.67:0/4152950361 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb6a00a2800 0x7fb6a00a4cc0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac0058c0 tx=0x7fb6ac00bf60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:05:43.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.425+0000 7fb6c4bb3740 1 -- 172.21.15.67:0/66691885 shutdown_connections 2024-03-28T16:05:43.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.425+0000 7fb6c4bb3740 1 --2- 172.21.15.67:0/66691885 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55768b4af3a0 0x55768b4b5c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:43.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.425+0000 7fb6c4bb3740 1 --2- 172.21.15.67:0/66691885 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55768b4ae9e0 0x55768b4aee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:43.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.425+0000 7fb6c4bb3740 1 --2- 172.21.15.67:0/66691885 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55768b4ad810 0x55768b4adc30 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:43.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.426+0000 7fb6c4bb3740 1 -- 172.21.15.67:0/66691885 >> 172.21.15.67:0/66691885 conn(0x55768b4a8c00 msgr2=0x55768b4a9000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:05:43.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.426+0000 7fb6c4bb3740 1 -- 172.21.15.67:0/66691885 shutdown_connections 2024-03-28T16:05:43.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.426+0000 7fb6c4bb3740 1 -- 172.21.15.67:0/66691885 wait complete. 2024-03-28T16:05:43.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.426+0000 7fb6c4bb3740 1 -- start start 2024-03-28T16:05:43.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.426+0000 7fb6c4bb3740 1 Processor -- start 2024-03-28T16:05:43.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.427+0000 7fb6c4bb3740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55768b4ad810 0x55768b4b7a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:43.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.427+0000 7fb6c4bb3740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55768b4ae9e0 0x55768b4b7f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:43.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.427+0000 7fb6c4bb3740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55768b4af3a0 0x55768b4befa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:43.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.427+0000 7fb6c4bb3740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55768b4b6350 con 0x55768b4ad810 2024-03-28T16:05:43.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.427+0000 7fb6c4bb3740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55768b3ce0d0 con 0x55768b4af3a0 2024-03-28T16:05:43.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.427+0000 7fb6c4bb3740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55768b4b64d0 con 0x55768b4ae9e0 2024-03-28T16:05:43.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.427+0000 7fb6c4bb3740 10 client.0 _ll_register_callbacks cb 0x55768b3d2e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:05:43.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.427+0000 7fb6bb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55768b4ad810 0x55768b4b7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:43.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.427+0000 7fb6bb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55768b4ad810 0x55768b4b7a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50994/0 (socket says 172.21.15.67:50994) 2024-03-28T16:05:43.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.427+0000 7fb6bb400640 1 -- 172.21.15.67:0/3596486637 learned_addr learned my addr 172.21.15.67:0/3596486637 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:05:43.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.428+0000 7fb6ba000640 1 --2- 172.21.15.67:0/3596486637 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55768b4ae9e0 0x55768b4b7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:43.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.428+0000 7fb6bb400640 1 -- 172.21.15.67:0/3596486637 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55768b4ae9e0 msgr2=0x55768b4b7f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:05:43.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.428+0000 7fb6bb400640 1 --2- 172.21.15.67:0/3596486637 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55768b4ae9e0 0x55768b4b7f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:43.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.428+0000 7fb6bb400640 1 -- 172.21.15.67:0/3596486637 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55768b4af3a0 msgr2=0x55768b4befa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:05:43.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.428+0000 7fb6bb400640 1 --2- 172.21.15.67:0/3596486637 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55768b4af3a0 0x55768b4befa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:43.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.428+0000 7fb6bb400640 1 -- 172.21.15.67:0/3596486637 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55768b1263f0 con 0x55768b4ad810 2024-03-28T16:05:43.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.428+0000 7fb6ba000640 1 --2- 172.21.15.67:0/3596486637 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55768b4ae9e0 0x55768b4b7f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:05:43.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.428+0000 7fb6bb400640 1 --2- 172.21.15.67:0/3596486637 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55768b4ad810 0x55768b4b7a20 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fb6b4009bf0 tx=0x7fb6b4032bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cc719ad73b2fce3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:05:43.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.428+0000 7fb6a4c00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:05:43.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.428+0000 7fb6a4c00640 1 -- 172.21.15.67:0/3596486637 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6b404c070 con 0x55768b4ad810 2024-03-28T16:05:43.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.428+0000 7fb6a4c00640 1 -- 172.21.15.67:0/3596486637 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fb6b40040b0 con 0x55768b4ad810 2024-03-28T16:05:43.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.428+0000 7fb6c4bb3740 1 -- 172.21.15.67:0/3596486637 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb6b4032df0 con 0x55768b4ad810 2024-03-28T16:05:43.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.429+0000 7fb6a4c00640 1 -- 172.21.15.67:0/3596486637 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb6b4058050 con 0x55768b4ad810 2024-03-28T16:05:43.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.429+0000 7fb6c4bb3740 10 client.15597 fetch_fsmap learned FSMap version 19 2024-03-28T16:05:43.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.429+0000 7fb6c4bb3740 1 -- 172.21.15.67:0/3596486637 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55768b1f07d0 con 0x55768b4ad810 2024-03-28T16:05:43.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.429+0000 7fb6a4c00640 1 -- 172.21.15.67:0/3596486637 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb6b4032df0 con 0x55768b4ad810 2024-03-28T16:05:43.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.429+0000 7fb6a4c00640 1 -- 172.21.15.67:0/3596486637 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fb6b403f9d0 con 0x55768b4ad810 2024-03-28T16:05:43.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.429+0000 7fb6c4bb3740 10 client.15597 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:05:43.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.429+0000 7fb6c4bb3740 10 client.15597 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:05:43.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.429+0000 7fb6c4bb3740 20 client.15597 populate_metadata read hostname 'smithi067' 2024-03-28T16:05:43.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.429+0000 7fb6c4bb3740 1 --2- 172.21.15.67:0/3596486637 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55768b4c3100 0x55768b4c54f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:43.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.429+0000 7fb6c4bb3740 4 client.15597 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:05:43.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.429+0000 7fb6c4bb3740 1 -- 172.21.15.67:0/3596486637 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55768b41f800 con 0x55768b4c3100 2024-03-28T16:05:43.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.430+0000 7fb6ba000640 1 --2- 172.21.15.67:0/3596486637 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55768b4c3100 0x55768b4c54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:43.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.431+0000 7fb6ba000640 1 --2- 172.21.15.67:0/3596486637 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55768b4c3100 0x55768b4c54f0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e4654335d6d2fc server_cookie=a5fe934940f7a58f in_seq=0 out_seq=0 2024-03-28T16:05:43.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.431+0000 7fb6a4c00640 10 client.15597 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:05:43.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.433+0000 7fb6a4c00640 1 -- 172.21.15.67:0/3596486637 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55768b41f800 con 0x55768b4c3100 2024-03-28T16:05:43.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.433+0000 7fb6a4c00640 10 client.15597 handle_command_reply: tid=0 2024-03-28T16:05:43.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.433+0000 7fb6c4bb3740 1 client.15597 shutdown 2024-03-28T16:05:43.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.433+0000 7fb6c4bb3740 1 -- 172.21.15.67:0/3596486637 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55768b4ad810 msgr2=0x55768b4b7a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:43.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.433+0000 7fb6c4bb3740 1 --2- 172.21.15.67:0/3596486637 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55768b4ad810 0x55768b4b7a20 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fb6b4009bf0 tx=0x7fb6b4032bf0 comp rx=0 tx=0).stop 2024-03-28T16:05:43.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.433+0000 7fb6c4bb3740 1 -- 172.21.15.67:0/3596486637 shutdown_connections 2024-03-28T16:05:43.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.433+0000 7fb6c4bb3740 1 --2- 172.21.15.67:0/3596486637 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55768b4c3100 0x55768b4c54f0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:43.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.433+0000 7fb6c4bb3740 1 --2- 172.21.15.67:0/3596486637 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55768b4af3a0 0x55768b4befa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:43.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.433+0000 7fb6c4bb3740 1 --2- 172.21.15.67:0/3596486637 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55768b4ae9e0 0x55768b4b7f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:43.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.433+0000 7fb6c4bb3740 1 --2- 172.21.15.67:0/3596486637 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55768b4ad810 0x55768b4b7a20 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:43.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.433+0000 7fb6a4c00640 0 client.15597 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:05:43.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.433+0000 7fb6c4bb3740 1 -- 172.21.15.67:0/3596486637 >> 172.21.15.67:0/3596486637 conn(0x55768b4a8c00 msgr2=0x55768b4a7eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:05:43.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.433+0000 7fb6c4bb3740 1 -- 172.21.15.67:0/3596486637 shutdown_connections 2024-03-28T16:05:43.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.433+0000 7fb6c4bb3740 1 -- 172.21.15.67:0/3596486637 wait complete. 2024-03-28T16:05:43.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.433+0000 7fb6c4bb3740 20 client.15597 trim_cache size 0 max 16384 2024-03-28T16:05:43.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:05:43 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/953677247' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:05:43.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.449+0000 7fb6c4bb3740 1 Processor -- start 2024-03-28T16:05:43.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.449+0000 7fb6c4bb3740 1 -- start start 2024-03-28T16:05:43.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.449+0000 7fb6c4bb3740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55768b4ad810 0x55768b4c9d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:43.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.449+0000 7fb6c4bb3740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55768b4ae9e0 0x55768b4bef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:43.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.449+0000 7fb6c4bb3740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55768b4af3a0 0x55768b4ca250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:43.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.449+0000 7fb6c4bb3740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6bc1a8c20 con 0x55768b4ae9e0 2024-03-28T16:05:43.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.449+0000 7fb6c4bb3740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6bc1ab1e0 con 0x55768b4ad810 2024-03-28T16:05:43.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.449+0000 7fb6c4bb3740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6bc1a9220 con 0x55768b4af3a0 2024-03-28T16:05:43.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.450+0000 7fb6ba000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55768b4ae9e0 0x55768b4bef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:43.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.450+0000 7fb6ba000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55768b4ae9e0 0x55768b4bef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50998/0 (socket says 172.21.15.67:50998) 2024-03-28T16:05:43.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.450+0000 7fb6ba000640 1 -- 172.21.15.67:0/1737009259 learned_addr learned my addr 172.21.15.67:0/1737009259 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:05:43.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.450+0000 7fb6bb400640 1 --2- 172.21.15.67:0/1737009259 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55768b4ad810 0x55768b4c9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:43.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.450+0000 7fb6bb400640 1 --2- 172.21.15.67:0/1737009259 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55768b4af3a0 0x55768b4ca250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:43.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.450+0000 7fb6ba000640 1 -- 172.21.15.67:0/1737009259 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55768b4af3a0 msgr2=0x55768b4ca250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:05:43.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.450+0000 7fb6ba000640 1 --2- 172.21.15.67:0/1737009259 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55768b4af3a0 0x55768b4ca250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:43.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.450+0000 7fb6ba000640 1 -- 172.21.15.67:0/1737009259 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55768b4ad810 msgr2=0x55768b4c9d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:05:43.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.450+0000 7fb6ba000640 1 --2- 172.21.15.67:0/1737009259 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55768b4ad810 0x55768b4c9d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:43.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.450+0000 7fb6ba000640 1 -- 172.21.15.67:0/1737009259 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55768b4b5f30 con 0x55768b4ae9e0 2024-03-28T16:05:43.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.450+0000 7fb6bb400640 1 --2- 172.21.15.67:0/1737009259 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55768b4ad810 0x55768b4c9d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T16:05:43.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.451+0000 7fb6ba000640 1 --2- 172.21.15.67:0/1737009259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55768b4ae9e0 0x55768b4bef60 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fb6a800bb40 tx=0x7fb6a8005c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42e1b586d5239794 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:05:43.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.453+0000 7fb6a7400640 1 -- 172.21.15.67:0/1737009259 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6a8004230 con 0x55768b4ae9e0 2024-03-28T16:05:43.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.454+0000 7fb6a7400640 1 -- 172.21.15.67:0/1737009259 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb6a80043d0 con 0x55768b4ae9e0 2024-03-28T16:05:43.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.454+0000 7fb6a7400640 1 -- 172.21.15.67:0/1737009259 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6a8012920 con 0x55768b4ae9e0 2024-03-28T16:05:43.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.454+0000 7fb6c4bb3740 1 -- 172.21.15.67:0/1737009259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55768b4ae9e0 msgr2=0x55768b4bef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:43.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.454+0000 7fb6c4bb3740 1 --2- 172.21.15.67:0/1737009259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55768b4ae9e0 0x55768b4bef60 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fb6a800bb40 tx=0x7fb6a8005c90 comp rx=0 tx=0).stop 2024-03-28T16:05:43.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.454+0000 7fb6c4bb3740 1 -- 172.21.15.67:0/1737009259 shutdown_connections 2024-03-28T16:05:43.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.454+0000 7fb6c4bb3740 1 --2- 172.21.15.67:0/1737009259 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55768b4af3a0 0x55768b4ca250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:43.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.454+0000 7fb6c4bb3740 1 --2- 172.21.15.67:0/1737009259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55768b4ae9e0 0x55768b4bef60 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:43.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.454+0000 7fb6c4bb3740 1 --2- 172.21.15.67:0/1737009259 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55768b4ad810 0x55768b4c9d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:43.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.454+0000 7fb6c4bb3740 1 -- 172.21.15.67:0/1737009259 >> 172.21.15.67:0/1737009259 conn(0x55768b4a8c00 msgr2=0x55768b4ac6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:05:43.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.455+0000 7fb6c4bb3740 1 -- 172.21.15.67:0/1737009259 shutdown_connections 2024-03-28T16:05:43.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.455+0000 7fb6c4bb3740 1 -- 172.21.15.67:0/1737009259 wait complete. 2024-03-28T16:05:43.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.455+0000 7fb6c4bb3740 1 -- start start 2024-03-28T16:05:43.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.456+0000 7fb6c4bb3740 1 Processor -- start 2024-03-28T16:05:43.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.456+0000 7fb6c4bb3740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55768b4ad810 0x55768b4bce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:43.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.456+0000 7fb6c4bb3740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55768b4ae9e0 0x55768b4bd390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:43.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.456+0000 7fb6c4bb3740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55768b4af3a0 0x55768b4be3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:43.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.456+0000 7fb6c4bb3740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6bc1a9860 con 0x55768b4af3a0 2024-03-28T16:05:43.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.456+0000 7fb6c4bb3740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6bc1aab80 con 0x55768b4ad810 2024-03-28T16:05:43.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.456+0000 7fb6c4bb3740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6bc1a9ec0 con 0x55768b4ae9e0 2024-03-28T16:05:43.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.456+0000 7fb6c4bb3740 10 client.0 _ll_register_callbacks cb 0x7fb6b4009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:05:43.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.457+0000 7fb6bb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55768b4af3a0 0x55768b4be3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:43.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.457+0000 7fb6bb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55768b4af3a0 0x55768b4be3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51004/0 (socket says 172.21.15.67:51004) 2024-03-28T16:05:43.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.457+0000 7fb6bb400640 1 -- 172.21.15.67:0/577284383 learned_addr learned my addr 172.21.15.67:0/577284383 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:05:43.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.457+0000 7fb6bb400640 1 -- 172.21.15.67:0/577284383 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55768b4ae9e0 msgr2=0x55768b4bd390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:05:43.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.457+0000 7fb6bb400640 1 --2- 172.21.15.67:0/577284383 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55768b4ae9e0 0x55768b4bd390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:43.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.457+0000 7fb6bb400640 1 -- 172.21.15.67:0/577284383 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55768b4ad810 msgr2=0x55768b4bce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:05:43.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.457+0000 7fb6bb400640 1 --2- 172.21.15.67:0/577284383 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55768b4ad810 0x55768b4bce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:43.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.457+0000 7fb6bb400640 1 -- 172.21.15.67:0/577284383 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55768b4be990 con 0x55768b4af3a0 2024-03-28T16:05:43.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.458+0000 7fb6bb400640 1 --2- 172.21.15.67:0/577284383 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55768b4af3a0 0x55768b4be3a0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fb6b4009a60 tx=0x7fb6b40344b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcde2db4499f6299 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:05:43.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.458+0000 7fb6a4c00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:05:43.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.458+0000 7fb6a4c00640 1 -- 172.21.15.67:0/577284383 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6b404c040 con 0x55768b4af3a0 2024-03-28T16:05:43.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.458+0000 7fb6a4c00640 1 -- 172.21.15.67:0/577284383 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fb6b403f470 con 0x55768b4af3a0 2024-03-28T16:05:43.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.458+0000 7fb6c4bb3740 1 -- 172.21.15.67:0/577284383 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb6a80043d0 con 0x55768b4af3a0 2024-03-28T16:05:43.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.458+0000 7fb6a4c00640 1 -- 172.21.15.67:0/577284383 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb6b4043030 con 0x55768b4af3a0 2024-03-28T16:05:43.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.458+0000 7fb6c4bb3740 10 client.15603 fetch_fsmap learned FSMap version 19 2024-03-28T16:05:43.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.460+0000 7fb6c4bb3740 1 -- 172.21.15.67:0/577284383 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55768b51f150 con 0x55768b4af3a0 2024-03-28T16:05:43.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.460+0000 7fb6a4c00640 1 -- 172.21.15.67:0/577284383 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb6a80043d0 con 0x55768b4af3a0 2024-03-28T16:05:43.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.460+0000 7fb6a4c00640 1 -- 172.21.15.67:0/577284383 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fb6b4034950 con 0x55768b4af3a0 2024-03-28T16:05:43.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.460+0000 7fb6c4bb3740 10 client.15603 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:05:43.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.460+0000 7fb6c4bb3740 10 client.15603 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:05:43.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.460+0000 7fb6c4bb3740 20 client.15603 populate_metadata read hostname 'smithi067' 2024-03-28T16:05:43.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.460+0000 7fb6c4bb3740 1 --2- 172.21.15.67:0/577284383 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55768b51fa40 0x55768b521e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:43.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.460+0000 7fb6c4bb3740 4 client.15603 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T16:05:43.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.460+0000 7fb6c4bb3740 1 -- 172.21.15.67:0/577284383 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55768b419b50 con 0x55768b51fa40 2024-03-28T16:05:43.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.460+0000 7fb6ba000640 1 --2- 172.21.15.67:0/577284383 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55768b51fa40 0x55768b521e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:43.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.461+0000 7fb6ba000640 1 --2- 172.21.15.67:0/577284383 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55768b51fa40 0x55768b521e60 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fe6757d5ebd5bcf3 server_cookie=44bf8fc865f434ed in_seq=0 out_seq=0 2024-03-28T16:05:43.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.461+0000 7fb6a4c00640 10 client.15603 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:05:43.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.461+0000 7fb6a4c00640 1 -- 172.21.15.67:0/577284383 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55768b419b50 con 0x55768b51fa40 2024-03-28T16:05:43.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.461+0000 7fb6a4c00640 10 client.15603 handle_command_reply: tid=0 2024-03-28T16:05:43.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.462+0000 7fb6c4bb3740 1 client.15603 shutdown 2024-03-28T16:05:43.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.462+0000 7fb6c4bb3740 1 -- 172.21.15.67:0/577284383 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55768b4af3a0 msgr2=0x55768b4be3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:43.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.462+0000 7fb6c4bb3740 1 --2- 172.21.15.67:0/577284383 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55768b4af3a0 0x55768b4be3a0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fb6b4009a60 tx=0x7fb6b40344b0 comp rx=0 tx=0).stop 2024-03-28T16:05:43.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.462+0000 7fb6c4bb3740 1 -- 172.21.15.67:0/577284383 shutdown_connections 2024-03-28T16:05:43.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.462+0000 7fb6c4bb3740 1 --2- 172.21.15.67:0/577284383 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55768b51fa40 0x55768b521e60 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:43.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.462+0000 7fb6c4bb3740 1 --2- 172.21.15.67:0/577284383 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55768b4af3a0 0x55768b4be3a0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:43.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.462+0000 7fb6c4bb3740 1 --2- 172.21.15.67:0/577284383 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55768b4ae9e0 0x55768b4bd390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:43.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.462+0000 7fb6c4bb3740 1 --2- 172.21.15.67:0/577284383 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55768b4ad810 0x55768b4bce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:43.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.462+0000 7fb6a4c00640 0 client.15603 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:05:43.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.462+0000 7fb6c4bb3740 1 -- 172.21.15.67:0/577284383 >> 172.21.15.67:0/577284383 conn(0x55768b4a8c00 msgr2=0x55768b4b1c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:05:43.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.462+0000 7fb6c4bb3740 1 -- 172.21.15.67:0/577284383 shutdown_connections 2024-03-28T16:05:43.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.462+0000 7fb6c4bb3740 1 -- 172.21.15.67:0/577284383 wait complete. 2024-03-28T16:05:43.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.462+0000 7fb6c4bb3740 20 client.15603 trim_cache size 0 max 16384 2024-03-28T16:05:43.466 INFO:teuthology.orchestra.run.smithi067.stdout:[] 2024-03-28T16:05:43.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.463+0000 7fb6c1e00640 1 -- 172.21.15.67:0/4152950361 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb6a00a2800 msgr2=0x7fb6a00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:43.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.463+0000 7fb6c1e00640 1 --2- 172.21.15.67:0/4152950361 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb6a00a2800 0x7fb6a00a4cc0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac0058c0 tx=0x7fb6ac00bf60 comp rx=0 tx=0).stop 2024-03-28T16:05:43.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.463+0000 7fb6c1e00640 1 -- 172.21.15.67:0/4152950361 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6bc0fff50 msgr2=0x7fb6bc199ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:43.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.463+0000 7fb6c1e00640 1 --2- 172.21.15.67:0/4152950361 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6bc0fff50 0x7fb6bc199ac0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac00d950 tx=0x7fb6ac00de10 comp rx=0 tx=0).stop 2024-03-28T16:05:43.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.463+0000 7fb6c1e00640 1 -- 172.21.15.67:0/4152950361 shutdown_connections 2024-03-28T16:05:43.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.463+0000 7fb6c1e00640 1 --2- 172.21.15.67:0/4152950361 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb6a00a2800 0x7fb6a00a4cc0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:43.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.463+0000 7fb6c1e00640 1 --2- 172.21.15.67:0/4152950361 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6bc101250 0x7fb6bc1a1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:43.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.463+0000 7fb6c1e00640 1 --2- 172.21.15.67:0/4152950361 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb6bc1008b0 0x7fb6bc19a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:43.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.463+0000 7fb6c1e00640 1 --2- 172.21.15.67:0/4152950361 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6bc0fff50 0x7fb6bc199ac0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:43.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.463+0000 7fb6c1e00640 1 -- 172.21.15.67:0/4152950361 >> 172.21.15.67:0/4152950361 conn(0x7fb6bc0fbb30 msgr2=0x7fb6bc0fd320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:05:43.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.464+0000 7fb6c1e00640 1 -- 172.21.15.67:0/4152950361 shutdown_connections 2024-03-28T16:05:43.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:43.464+0000 7fb6c1e00640 1 -- 172.21.15.67:0/4152950361 wait complete. 2024-03-28T16:05:43.807 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:05:43 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/953677247' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:05:43.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:05:43 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/953677247' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:05:44.486 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T16:05:44.486 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T16:05:44.487 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T16:05:44.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.734+0000 7f571e400640 1 -- 172.21.15.67:0/3602007059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5718103150 msgr2=0x7f57181099e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:44.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.734+0000 7f571e400640 1 --2- 172.21.15.67:0/3602007059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5718103150 0x7f57181099e0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f5710009ca0 tx=0x7f571002c100 comp rx=0 tx=0).stop 2024-03-28T16:05:44.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.734+0000 7f571e400640 1 -- 172.21.15.67:0/3602007059 shutdown_connections 2024-03-28T16:05:44.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.734+0000 7f571e400640 1 --2- 172.21.15.67:0/3602007059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5718103150 0x7f57181099e0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:44.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.734+0000 7f571e400640 1 --2- 172.21.15.67:0/3602007059 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5718102790 0x7f5718102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:44.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.734+0000 7f571e400640 1 --2- 172.21.15.67:0/3602007059 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57181015a0 0x7f57181019c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:44.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.734+0000 7f571e400640 1 -- 172.21.15.67:0/3602007059 >> 172.21.15.67:0/3602007059 conn(0x7f57180fcb10 msgr2=0x7f57180fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:05:44.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.734+0000 7f571e400640 1 -- 172.21.15.67:0/3602007059 shutdown_connections 2024-03-28T16:05:44.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.734+0000 7f571e400640 1 -- 172.21.15.67:0/3602007059 wait complete. 2024-03-28T16:05:44.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.735+0000 7f571e400640 1 Processor -- start 2024-03-28T16:05:44.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.735+0000 7f571e400640 1 -- start start 2024-03-28T16:05:44.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.736+0000 7f571e400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f57181015a0 0x7f571819ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:44.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.736+0000 7f571e400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5718102790 0x7f571819e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:44.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.736+0000 7f571e400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5718103150 0x7f57181a5320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:44.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.736+0000 7f571e400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f571810d0e0 con 0x7f5718103150 2024-03-28T16:05:44.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.736+0000 7f571e400640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f571810cf60 con 0x7f5718102790 2024-03-28T16:05:44.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.736+0000 7f571e400640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f571810d260 con 0x7f57181015a0 2024-03-28T16:05:44.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.736+0000 7f571da00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5718103150 0x7f57181a5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:44.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.736+0000 7f571da00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5718103150 0x7f57181a5320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51030/0 (socket says 172.21.15.67:51030) 2024-03-28T16:05:44.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.736+0000 7f571da00640 1 -- 172.21.15.67:0/2785748420 learned_addr learned my addr 172.21.15.67:0/2785748420 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:05:44.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.736+0000 7f571d000640 1 --2- 172.21.15.67:0/2785748420 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f57181015a0 0x7f571819ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:44.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.736+0000 7f5717e00640 1 --2- 172.21.15.67:0/2785748420 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5718102790 0x7f571819e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:44.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.736+0000 7f571da00640 1 -- 172.21.15.67:0/2785748420 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f57181015a0 msgr2=0x7f571819ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:44.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.736+0000 7f571da00640 1 --2- 172.21.15.67:0/2785748420 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f57181015a0 0x7f571819ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:44.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.737+0000 7f571da00640 1 -- 172.21.15.67:0/2785748420 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5718102790 msgr2=0x7f571819e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:44.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.737+0000 7f571da00640 1 --2- 172.21.15.67:0/2785748420 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5718102790 0x7f571819e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:44.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.737+0000 7f571da00640 1 -- 172.21.15.67:0/2785748420 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57181a59c0 con 0x7f5718103150 2024-03-28T16:05:44.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.737+0000 7f571da00640 1 --2- 172.21.15.67:0/2785748420 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5718103150 0x7f57181a5320 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f5710032520 tx=0x7f5710034880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:05:44.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.737+0000 7f5715600640 1 -- 172.21.15.67:0/2785748420 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5710039040 con 0x7f5718103150 2024-03-28T16:05:44.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.737+0000 7f5715600640 1 -- 172.21.15.67:0/2785748420 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5710044070 con 0x7f5718103150 2024-03-28T16:05:44.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.737+0000 7f571e400640 1 -- 172.21.15.67:0/2785748420 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57181a5c50 con 0x7f5718103150 2024-03-28T16:05:44.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.737+0000 7f5715600640 1 -- 172.21.15.67:0/2785748420 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5710004430 con 0x7f5718103150 2024-03-28T16:05:44.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.738+0000 7f571e400640 1 -- 172.21.15.67:0/2785748420 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57181a6140 con 0x7f5718103150 2024-03-28T16:05:44.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.741+0000 7f5715600640 1 -- 172.21.15.67:0/2785748420 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f5710003bc0 con 0x7f5718103150 2024-03-28T16:05:44.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.742+0000 7f5722678740 1 Processor -- start 2024-03-28T16:05:44.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.742+0000 7f5722678740 1 -- start start 2024-03-28T16:05:44.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.742+0000 7f5722678740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad48dce2b0 0x55ad48dce6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:44.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.742+0000 7f5722678740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ad48dceca0 0x55ad48dd37a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:44.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.742+0000 7f5722678740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ad48dd3e10 0x55ad48dd62a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:44.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.742+0000 7f5722678740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad48d33eb0 con 0x55ad48dce2b0 2024-03-28T16:05:44.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.742+0000 7f5722678740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad48cdca60 con 0x55ad48dceca0 2024-03-28T16:05:44.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.742+0000 7f5722678740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad48d2ff50 con 0x55ad48dd3e10 2024-03-28T16:05:44.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.743+0000 7f5717e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad48dce2b0 0x55ad48dce6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:44.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.743+0000 7f5717e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad48dce2b0 0x55ad48dce6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51032/0 (socket says 172.21.15.67:51032) 2024-03-28T16:05:44.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.743+0000 7f5717e00640 1 -- 172.21.15.67:0/3177174330 learned_addr learned my addr 172.21.15.67:0/3177174330 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:05:44.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.743+0000 7f571d000640 1 --2- 172.21.15.67:0/3177174330 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ad48dceca0 0x55ad48dd37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:44.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.743+0000 7f5717e00640 1 -- 172.21.15.67:0/3177174330 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ad48dd3e10 msgr2=0x55ad48dd62a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:05:44.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.743+0000 7f5717e00640 1 --2- 172.21.15.67:0/3177174330 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ad48dd3e10 0x55ad48dd62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:44.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.743+0000 7f5717e00640 1 -- 172.21.15.67:0/3177174330 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ad48dceca0 msgr2=0x55ad48dd37a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:05:44.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.743+0000 7f5717e00640 1 --2- 172.21.15.67:0/3177174330 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ad48dceca0 0x55ad48dd37a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:44.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.743+0000 7f5717e00640 1 -- 172.21.15.67:0/3177174330 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ad48b107e0 con 0x55ad48dce2b0 2024-03-28T16:05:44.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.743+0000 7f5715600640 1 --2- 172.21.15.67:0/2785748420 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f57000a2800 0x7f57000a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:44.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.744+0000 7f5717e00640 1 --2- 172.21.15.67:0/3177174330 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad48dce2b0 0x55ad48dce6d0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f570400bda0 tx=0x7f5704011750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96586d94b5a551d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:05:44.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.744+0000 7f5715600640 1 -- 172.21.15.67:0/2785748420 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f57100f12c0 con 0x7f5718103150 2024-03-28T16:05:44.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.744+0000 7f571d000640 1 --2- 172.21.15.67:0/2785748420 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f57000a2800 0x7f57000a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:44.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.744+0000 7f570a000640 1 -- 172.21.15.67:0/3177174330 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f570401a600 con 0x55ad48dce2b0 2024-03-28T16:05:44.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.744+0000 7f570a000640 1 -- 172.21.15.67:0/3177174330 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5704011e60 con 0x55ad48dce2b0 2024-03-28T16:05:44.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.744+0000 7f570a000640 1 -- 172.21.15.67:0/3177174330 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5704012a70 con 0x55ad48dce2b0 2024-03-28T16:05:44.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.745+0000 7f5722678740 1 -- 172.21.15.67:0/3177174330 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad48dce2b0 msgr2=0x55ad48dce6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:44.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.745+0000 7f571d000640 1 --2- 172.21.15.67:0/2785748420 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f57000a2800 0x7f57000a4cc0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f570c005fd0 tx=0x7f570c005c50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:05:44.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.745+0000 7f5722678740 1 --2- 172.21.15.67:0/3177174330 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad48dce2b0 0x55ad48dce6d0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f570400bda0 tx=0x7f5704011750 comp rx=0 tx=0).stop 2024-03-28T16:05:44.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.745+0000 7f5722678740 1 -- 172.21.15.67:0/3177174330 shutdown_connections 2024-03-28T16:05:44.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.745+0000 7f5722678740 1 --2- 172.21.15.67:0/3177174330 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ad48dd3e10 0x55ad48dd62a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:44.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.745+0000 7f5722678740 1 --2- 172.21.15.67:0/3177174330 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ad48dceca0 0x55ad48dd37a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:44.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.745+0000 7f5722678740 1 --2- 172.21.15.67:0/3177174330 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad48dce2b0 0x55ad48dce6d0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:44.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.745+0000 7f5722678740 1 -- 172.21.15.67:0/3177174330 >> 172.21.15.67:0/3177174330 conn(0x55ad48dc9820 msgr2=0x55ad48dc9c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:05:44.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.745+0000 7f5722678740 1 -- 172.21.15.67:0/3177174330 shutdown_connections 2024-03-28T16:05:44.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.745+0000 7f5722678740 1 -- 172.21.15.67:0/3177174330 wait complete. 2024-03-28T16:05:44.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.746+0000 7f5722678740 1 -- start start 2024-03-28T16:05:44.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.746+0000 7f5722678740 1 Processor -- start 2024-03-28T16:05:44.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.747+0000 7f5722678740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad48dce2b0 0x55ad48dd84f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:44.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.747+0000 7f5722678740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ad48dceca0 0x55ad48dd8a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:44.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.747+0000 7f5722678740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ad48dd3e10 0x55ad48ddfa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:44.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.747+0000 7f5722678740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad48dd6dc0 con 0x55ad48dce2b0 2024-03-28T16:05:44.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.747+0000 7f5722678740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad48cefeb0 con 0x55ad48dceca0 2024-03-28T16:05:44.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.747+0000 7f5722678740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad48dd6f40 con 0x55ad48dd3e10 2024-03-28T16:05:44.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.747+0000 7f5722678740 10 client.0 _ll_register_callbacks cb 0x55ad48ce4150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:05:44.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.747+0000 7f571d000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad48dce2b0 0x55ad48dd84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:44.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.747+0000 7f571d000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad48dce2b0 0x55ad48dd84f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51044/0 (socket says 172.21.15.67:51044) 2024-03-28T16:05:44.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.747+0000 7f571d000640 1 -- 172.21.15.67:0/2720899438 learned_addr learned my addr 172.21.15.67:0/2720899438 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:05:44.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.747+0000 7f571da00640 1 --2- 172.21.15.67:0/2720899438 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ad48dd3e10 0x55ad48ddfa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:44.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.748+0000 7f5717e00640 1 --2- 172.21.15.67:0/2720899438 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ad48dceca0 0x55ad48dd8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:44.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.748+0000 7f571d000640 1 -- 172.21.15.67:0/2720899438 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ad48dd3e10 msgr2=0x55ad48ddfa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:05:44.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.748+0000 7f571d000640 1 --2- 172.21.15.67:0/2720899438 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ad48dd3e10 0x55ad48ddfa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:44.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.748+0000 7f571d000640 1 -- 172.21.15.67:0/2720899438 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ad48dceca0 msgr2=0x55ad48dd8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:05:44.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.748+0000 7f571d000640 1 --2- 172.21.15.67:0/2720899438 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ad48dceca0 0x55ad48dd8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:44.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.748+0000 7f571d000640 1 -- 172.21.15.67:0/2720899438 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ad48b70f70 con 0x55ad48dce2b0 2024-03-28T16:05:44.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.748+0000 7f571d000640 1 --2- 172.21.15.67:0/2720899438 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad48dce2b0 0x55ad48dd84f0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f57181025f0 tx=0x7f570c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e82d82de672ba4e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:05:44.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.748+0000 7f56fea00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:05:44.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.749+0000 7f56fea00640 1 -- 172.21.15.67:0/2720899438 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f570c01f8f0 con 0x55ad48dce2b0 2024-03-28T16:05:44.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.749+0000 7f56fea00640 1 -- 172.21.15.67:0/2720899438 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f570c01fa90 con 0x55ad48dce2b0 2024-03-28T16:05:44.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.749+0000 7f5722678740 1 -- 172.21.15.67:0/2720899438 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5704011e60 con 0x55ad48dce2b0 2024-03-28T16:05:44.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.749+0000 7f56fea00640 1 -- 172.21.15.67:0/2720899438 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f570c01fc90 con 0x55ad48dce2b0 2024-03-28T16:05:44.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.749+0000 7f5722678740 10 client.15615 fetch_fsmap learned FSMap version 19 2024-03-28T16:05:44.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.749+0000 7f5722678740 1 -- 172.21.15.67:0/2720899438 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55ad48a08170 con 0x55ad48dce2b0 2024-03-28T16:05:44.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.749+0000 7f56fea00640 1 -- 172.21.15.67:0/2720899438 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5704011e60 con 0x55ad48dce2b0 2024-03-28T16:05:44.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.750+0000 7f56fea00640 1 -- 172.21.15.67:0/2720899438 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f570c0323f0 con 0x55ad48dce2b0 2024-03-28T16:05:44.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.750+0000 7f5722678740 10 client.15615 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:05:44.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.750+0000 7f5722678740 10 client.15615 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:05:44.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.750+0000 7f5722678740 20 client.15615 populate_metadata read hostname 'smithi067' 2024-03-28T16:05:44.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.750+0000 7f5722678740 1 --2- 172.21.15.67:0/2720899438 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55ad48de4380 0x55ad48de6800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:44.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.750+0000 7f5722678740 4 client.15615 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:05:44.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.750+0000 7f5722678740 1 -- 172.21.15.67:0/2720899438 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ad48d413f0 con 0x55ad48de4380 2024-03-28T16:05:44.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.750+0000 7f5717e00640 1 --2- 172.21.15.67:0/2720899438 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55ad48de4380 0x55ad48de6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:44.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.751+0000 7f5717e00640 1 --2- 172.21.15.67:0/2720899438 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55ad48de4380 0x55ad48de6800 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1434ebe96c050460 server_cookie=6b5ae2d90a5c5417 in_seq=0 out_seq=0 2024-03-28T16:05:44.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.751+0000 7f56fea00640 10 client.15615 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:05:44.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.753+0000 7f56fea00640 1 -- 172.21.15.67:0/2720899438 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55ad48d413f0 con 0x55ad48de4380 2024-03-28T16:05:44.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.753+0000 7f56fea00640 10 client.15615 handle_command_reply: tid=0 2024-03-28T16:05:44.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.753+0000 7f5722678740 1 client.15615 shutdown 2024-03-28T16:05:44.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.753+0000 7f5722678740 1 -- 172.21.15.67:0/2720899438 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad48dce2b0 msgr2=0x55ad48dd84f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:44.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.753+0000 7f5722678740 1 --2- 172.21.15.67:0/2720899438 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad48dce2b0 0x55ad48dd84f0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f57181025f0 tx=0x7f570c0046c0 comp rx=0 tx=0).stop 2024-03-28T16:05:44.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.753+0000 7f5722678740 1 -- 172.21.15.67:0/2720899438 shutdown_connections 2024-03-28T16:05:44.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.753+0000 7f5722678740 1 --2- 172.21.15.67:0/2720899438 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55ad48de4380 0x55ad48de6800 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:44.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.753+0000 7f5722678740 1 --2- 172.21.15.67:0/2720899438 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ad48dd3e10 0x55ad48ddfa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:44.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.753+0000 7f5722678740 1 --2- 172.21.15.67:0/2720899438 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ad48dceca0 0x55ad48dd8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:44.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.753+0000 7f56fea00640 0 client.15615 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:05:44.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.753+0000 7f5722678740 1 --2- 172.21.15.67:0/2720899438 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad48dce2b0 0x55ad48dd84f0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:44.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.753+0000 7f5722678740 1 -- 172.21.15.67:0/2720899438 >> 172.21.15.67:0/2720899438 conn(0x55ad48dc9820 msgr2=0x55ad48dc8a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:05:44.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.754+0000 7f5722678740 1 -- 172.21.15.67:0/2720899438 shutdown_connections 2024-03-28T16:05:44.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.754+0000 7f5722678740 1 -- 172.21.15.67:0/2720899438 wait complete. 2024-03-28T16:05:44.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.754+0000 7f5722678740 20 client.15615 trim_cache size 0 max 16384 2024-03-28T16:05:44.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.775+0000 7f5722678740 1 Processor -- start 2024-03-28T16:05:44.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.776+0000 7f5722678740 1 -- start start 2024-03-28T16:05:44.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.776+0000 7f5722678740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ad48dce2b0 0x55ad48dc8900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:44.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.776+0000 7f5722678740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ad48dceca0 0x55ad48ddfa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:44.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.776+0000 7f5722678740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad48dd3e10 0x55ad48deb0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:44.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.776+0000 7f5722678740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57181ace00 con 0x55ad48dd3e10 2024-03-28T16:05:44.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.776+0000 7f5722678740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57181af3c0 con 0x55ad48dceca0 2024-03-28T16:05:44.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.776+0000 7f5722678740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57181ad400 con 0x55ad48dce2b0 2024-03-28T16:05:44.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.777+0000 7f571da00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad48dd3e10 0x55ad48deb0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:44.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.777+0000 7f571da00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad48dd3e10 0x55ad48deb0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51050/0 (socket says 172.21.15.67:51050) 2024-03-28T16:05:44.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.777+0000 7f571da00640 1 -- 172.21.15.67:0/1800383237 learned_addr learned my addr 172.21.15.67:0/1800383237 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:05:44.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.777+0000 7f571d000640 1 --2- 172.21.15.67:0/1800383237 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ad48dce2b0 0x55ad48dc8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:44.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.777+0000 7f571da00640 1 -- 172.21.15.67:0/1800383237 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ad48dce2b0 msgr2=0x55ad48dc8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:05:44.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.777+0000 7f571da00640 1 --2- 172.21.15.67:0/1800383237 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ad48dce2b0 0x55ad48dc8900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:44.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.777+0000 7f571da00640 1 -- 172.21.15.67:0/1800383237 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ad48dceca0 msgr2=0x55ad48ddfa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:05:44.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.777+0000 7f5717e00640 1 --2- 172.21.15.67:0/1800383237 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ad48dceca0 0x55ad48ddfa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:44.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.777+0000 7f571da00640 1 --2- 172.21.15.67:0/1800383237 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ad48dceca0 0x55ad48ddfa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:44.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.777+0000 7f571da00640 1 -- 172.21.15.67:0/1800383237 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ad48dd3170 con 0x55ad48dd3e10 2024-03-28T16:05:44.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.777+0000 7f5717e00640 1 --2- 172.21.15.67:0/1800383237 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ad48dceca0 0x55ad48ddfa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:05:44.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.778+0000 7f571da00640 1 --2- 172.21.15.67:0/1800383237 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad48dd3e10 0x55ad48deb0c0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f57100eed70 tx=0x7f57100bd810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0065a708220675b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:05:44.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.778+0000 7f5709600640 1 -- 172.21.15.67:0/1800383237 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57100be310 con 0x55ad48dd3e10 2024-03-28T16:05:44.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.778+0000 7f5709600640 1 -- 172.21.15.67:0/1800383237 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f57100bdc10 con 0x55ad48dd3e10 2024-03-28T16:05:44.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.778+0000 7f5709600640 1 -- 172.21.15.67:0/1800383237 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57100333a0 con 0x55ad48dd3e10 2024-03-28T16:05:44.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.779+0000 7f5722678740 1 -- 172.21.15.67:0/1800383237 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad48dd3e10 msgr2=0x55ad48deb0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:44.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.779+0000 7f5722678740 1 --2- 172.21.15.67:0/1800383237 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad48dd3e10 0x55ad48deb0c0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f57100eed70 tx=0x7f57100bd810 comp rx=0 tx=0).stop 2024-03-28T16:05:44.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.780+0000 7f5722678740 1 -- 172.21.15.67:0/1800383237 shutdown_connections 2024-03-28T16:05:44.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.780+0000 7f5722678740 1 --2- 172.21.15.67:0/1800383237 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad48dd3e10 0x55ad48deb0c0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:44.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.780+0000 7f5722678740 1 --2- 172.21.15.67:0/1800383237 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ad48dceca0 0x55ad48ddfa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:44.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.780+0000 7f5722678740 1 --2- 172.21.15.67:0/1800383237 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ad48dce2b0 0x55ad48dc8900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:44.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.780+0000 7f5722678740 1 -- 172.21.15.67:0/1800383237 >> 172.21.15.67:0/1800383237 conn(0x55ad48dc9820 msgr2=0x55ad48dc81f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:05:44.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.780+0000 7f5722678740 1 -- 172.21.15.67:0/1800383237 shutdown_connections 2024-03-28T16:05:44.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.780+0000 7f5722678740 1 -- 172.21.15.67:0/1800383237 wait complete. 2024-03-28T16:05:44.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.780+0000 7f5722678740 1 -- start start 2024-03-28T16:05:44.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.781+0000 7f5722678740 1 Processor -- start 2024-03-28T16:05:44.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.781+0000 7f5722678740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ad48dce2b0 0x55ad48ddf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:44.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.781+0000 7f5722678740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad48dceca0 0x55ad48ddde50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:44.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.781+0000 7f5722678740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ad48dd3e10 0x55ad48dde390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:44.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.781+0000 7f5722678740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57181ada40 con 0x55ad48dceca0 2024-03-28T16:05:44.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.781+0000 7f5722678740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57181aed60 con 0x55ad48dd3e10 2024-03-28T16:05:44.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.781+0000 7f5722678740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57181ae0a0 con 0x55ad48dce2b0 2024-03-28T16:05:44.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.781+0000 7f5722678740 10 client.0 _ll_register_callbacks cb 0x7f570c00b3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:05:44.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.781+0000 7f5717e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad48dceca0 0x55ad48ddde50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:44.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.782+0000 7f5717e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad48dceca0 0x55ad48ddde50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51056/0 (socket says 172.21.15.67:51056) 2024-03-28T16:05:44.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.782+0000 7f5717e00640 1 -- 172.21.15.67:0/2769067126 learned_addr learned my addr 172.21.15.67:0/2769067126 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:05:44.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.782+0000 7f571da00640 1 --2- 172.21.15.67:0/2769067126 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ad48dd3e10 0x55ad48dde390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:44.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.782+0000 7f5717e00640 1 -- 172.21.15.67:0/2769067126 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ad48dce2b0 msgr2=0x55ad48ddf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:05:44.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.782+0000 7f5717e00640 1 --2- 172.21.15.67:0/2769067126 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ad48dce2b0 0x55ad48ddf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:44.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.782+0000 7f5717e00640 1 -- 172.21.15.67:0/2769067126 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ad48dd3e10 msgr2=0x55ad48dde390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:05:44.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.782+0000 7f5717e00640 1 --2- 172.21.15.67:0/2769067126 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ad48dd3e10 0x55ad48dde390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:44.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.782+0000 7f5717e00640 1 -- 172.21.15.67:0/2769067126 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ad48ddea00 con 0x55ad48dceca0 2024-03-28T16:05:44.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.782+0000 7f5717e00640 1 --2- 172.21.15.67:0/2769067126 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad48dceca0 0x55ad48ddde50 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f570400bd40 tx=0x7f570400ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe263b6807777c8a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:05:44.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.782+0000 7f56fea00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:05:44.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.783+0000 7f56fea00640 1 -- 172.21.15.67:0/2769067126 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5704005340 con 0x55ad48dceca0 2024-03-28T16:05:44.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.783+0000 7f56fea00640 1 -- 172.21.15.67:0/2769067126 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f57040054e0 con 0x55ad48dceca0 2024-03-28T16:05:44.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.783+0000 7f5722678740 1 -- 172.21.15.67:0/2769067126 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f57100bdc10 con 0x55ad48dceca0 2024-03-28T16:05:44.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.783+0000 7f56fea00640 1 -- 172.21.15.67:0/2769067126 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f570403a050 con 0x55ad48dceca0 2024-03-28T16:05:44.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.783+0000 7f5722678740 10 client.15621 fetch_fsmap learned FSMap version 19 2024-03-28T16:05:44.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.783+0000 7f5722678740 1 -- 172.21.15.67:0/2769067126 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55ad48e40b80 con 0x55ad48dceca0 2024-03-28T16:05:44.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.783+0000 7f56fea00640 1 -- 172.21.15.67:0/2769067126 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f57100bdc10 con 0x55ad48dceca0 2024-03-28T16:05:44.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.784+0000 7f56fea00640 1 -- 172.21.15.67:0/2769067126 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f5704025030 con 0x55ad48dceca0 2024-03-28T16:05:44.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.784+0000 7f5722678740 10 client.15621 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:05:44.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.784+0000 7f5722678740 10 client.15621 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:05:44.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.784+0000 7f5722678740 20 client.15621 populate_metadata read hostname 'smithi067' 2024-03-28T16:05:44.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.784+0000 7f5722678740 1 --2- 172.21.15.67:0/2769067126 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55ad48e41280 0x55ad48e436a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:44.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.784+0000 7f5722678740 4 client.15621 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T16:05:44.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.784+0000 7f5722678740 1 -- 172.21.15.67:0/2769067126 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55ad48d3b740 con 0x55ad48e41280 2024-03-28T16:05:44.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.784+0000 7f571d000640 1 --2- 172.21.15.67:0/2769067126 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55ad48e41280 0x55ad48e436a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:44.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.786+0000 7f56fea00640 10 client.15621 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:05:44.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.786+0000 7f571d000640 1 --2- 172.21.15.67:0/2769067126 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55ad48e41280 0x55ad48e436a0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2e6b3d524b8aedb server_cookie=5c7c7d1a70021ea0 in_seq=0 out_seq=0 2024-03-28T16:05:44.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.788+0000 7f56fea00640 1 -- 172.21.15.67:0/2769067126 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55ad48d3b740 con 0x55ad48e41280 2024-03-28T16:05:44.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.788+0000 7f56fea00640 10 client.15621 handle_command_reply: tid=0 2024-03-28T16:05:44.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.788+0000 7f5722678740 1 client.15621 shutdown 2024-03-28T16:05:44.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.788+0000 7f5722678740 1 -- 172.21.15.67:0/2769067126 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad48dceca0 msgr2=0x55ad48ddde50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:44.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.788+0000 7f5722678740 1 --2- 172.21.15.67:0/2769067126 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad48dceca0 0x55ad48ddde50 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f570400bd40 tx=0x7f570400ba90 comp rx=0 tx=0).stop 2024-03-28T16:05:44.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.788+0000 7f5722678740 1 -- 172.21.15.67:0/2769067126 shutdown_connections 2024-03-28T16:05:44.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.788+0000 7f5722678740 1 --2- 172.21.15.67:0/2769067126 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55ad48e41280 0x55ad48e436a0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:44.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.788+0000 7f5722678740 1 --2- 172.21.15.67:0/2769067126 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ad48dd3e10 0x55ad48dde390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:44.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.788+0000 7f56fea00640 0 client.15621 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:05:44.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.788+0000 7f5722678740 1 --2- 172.21.15.67:0/2769067126 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad48dceca0 0x55ad48ddde50 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:44.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.788+0000 7f5722678740 1 --2- 172.21.15.67:0/2769067126 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ad48dce2b0 0x55ad48ddf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:44.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.788+0000 7f5722678740 1 -- 172.21.15.67:0/2769067126 >> 172.21.15.67:0/2769067126 conn(0x55ad48dc9820 msgr2=0x55ad48dc8a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:05:44.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.789+0000 7f5722678740 1 -- 172.21.15.67:0/2769067126 shutdown_connections 2024-03-28T16:05:44.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.789+0000 7f5722678740 1 -- 172.21.15.67:0/2769067126 wait complete. 2024-03-28T16:05:44.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.789+0000 7f5722678740 20 client.15621 trim_cache size 0 max 16384 2024-03-28T16:05:44.792 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-28T16:05:44.792 INFO:teuthology.orchestra.run.smithi067.stdout: "return_code": 0, 2024-03-28T16:05:44.792 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_tag": "2599a8ca-4d10-4079-92c0-ac1b71729586", 2024-03-28T16:05:44.792 INFO:teuthology.orchestra.run.smithi067.stdout: "mode": "asynchronous" 2024-03-28T16:05:44.792 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-28T16:05:44.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.790+0000 7f571e400640 1 -- 172.21.15.67:0/2785748420 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f57000a2800 msgr2=0x7f57000a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:44.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.790+0000 7f571e400640 1 --2- 172.21.15.67:0/2785748420 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f57000a2800 0x7f57000a4cc0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f570c005fd0 tx=0x7f570c005c50 comp rx=0 tx=0).stop 2024-03-28T16:05:44.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.790+0000 7f571e400640 1 -- 172.21.15.67:0/2785748420 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5718103150 msgr2=0x7f57181a5320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:44.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.790+0000 7f571e400640 1 --2- 172.21.15.67:0/2785748420 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5718103150 0x7f57181a5320 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f5710032520 tx=0x7f5710034880 comp rx=0 tx=0).stop 2024-03-28T16:05:44.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.790+0000 7f571e400640 1 -- 172.21.15.67:0/2785748420 shutdown_connections 2024-03-28T16:05:44.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.790+0000 7f571e400640 1 --2- 172.21.15.67:0/2785748420 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f57000a2800 0x7f57000a4cc0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:44.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.790+0000 7f571e400640 1 --2- 172.21.15.67:0/2785748420 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5718103150 0x7f57181a5320 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:44.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.790+0000 7f571e400640 1 --2- 172.21.15.67:0/2785748420 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5718102790 0x7f571819e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:44.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.790+0000 7f571e400640 1 --2- 172.21.15.67:0/2785748420 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f57181015a0 0x7f571819ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:44.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.790+0000 7f571e400640 1 -- 172.21.15.67:0/2785748420 >> 172.21.15.67:0/2785748420 conn(0x7f57180fcb10 msgr2=0x7f5718103b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:05:44.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.790+0000 7f571e400640 1 -- 172.21.15.67:0/2785748420 shutdown_connections 2024-03-28T16:05:44.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:44.790+0000 7f571e400640 1 -- 172.21.15.67:0/2785748420 wait complete. 2024-03-28T16:05:44.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:05:44 smithi150 ceph-mon[34825]: pgmap v504: 97 pgs: 97 active+clean; 761 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 22 MiB/s wr, 7 op/s 2024-03-28T16:05:44.806 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:05:44 smithi067 ceph-mon[31707]: pgmap v504: 97 pgs: 97 active+clean; 761 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 22 MiB/s wr, 7 op/s 2024-03-28T16:05:44.811 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T16:05:44.811 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T16:05:44.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:05:44 smithi183 ceph-mon[35580]: pgmap v504: 97 pgs: 97 active+clean; 761 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 22 MiB/s wr, 7 op/s 2024-03-28T16:05:45.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.055+0000 7fd2ce600640 1 -- 172.21.15.67:0/559520419 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2c8101580 msgr2=0x7fd2c81019a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:45.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.055+0000 7fd2ce600640 1 --2- 172.21.15.67:0/559520419 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2c8101580 0x7fd2c81019a0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc009900 tx=0x7fd2bc02bc40 comp rx=0 tx=0).stop 2024-03-28T16:05:45.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.055+0000 7fd2ce600640 1 -- 172.21.15.67:0/559520419 shutdown_connections 2024-03-28T16:05:45.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.055+0000 7fd2ce600640 1 --2- 172.21.15.67:0/559520419 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd2c8103130 0x7fd2c81099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:45.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.055+0000 7fd2ce600640 1 --2- 172.21.15.67:0/559520419 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2c8102770 0x7fd2c8102bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:45.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.055+0000 7fd2ce600640 1 --2- 172.21.15.67:0/559520419 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2c8101580 0x7fd2c81019a0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:45.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.055+0000 7fd2ce600640 1 -- 172.21.15.67:0/559520419 >> 172.21.15.67:0/559520419 conn(0x7fd2c80fcb10 msgr2=0x7fd2c80fef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:05:45.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.055+0000 7fd2ce600640 1 -- 172.21.15.67:0/559520419 shutdown_connections 2024-03-28T16:05:45.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.056+0000 7fd2ce600640 1 -- 172.21.15.67:0/559520419 wait complete. 2024-03-28T16:05:45.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.056+0000 7fd2ce600640 1 Processor -- start 2024-03-28T16:05:45.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.057+0000 7fd2ce600640 1 -- start start 2024-03-28T16:05:45.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.057+0000 7fd2ce600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2c8101580 0x7fd2c8197760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:45.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.057+0000 7fd2ce600640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2c8102770 0x7fd2c8197ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:45.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.057+0000 7fd2ce600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd2c8103130 0x7fd2c819ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:45.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.057+0000 7fd2ce600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c810d090 con 0x7fd2c8101580 2024-03-28T16:05:45.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.057+0000 7fd2ce600640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c810cf10 con 0x7fd2c8102770 2024-03-28T16:05:45.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.057+0000 7fd2ce600640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c810d210 con 0x7fd2c8103130 2024-03-28T16:05:45.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.057+0000 7fd2cd200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2c8101580 0x7fd2c8197760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:45.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.057+0000 7fd2cd200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2c8101580 0x7fd2c8197760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51078/0 (socket says 172.21.15.67:51078) 2024-03-28T16:05:45.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.057+0000 7fd2cd200640 1 -- 172.21.15.67:0/2853960374 learned_addr learned my addr 172.21.15.67:0/2853960374 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:05:45.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.058+0000 7fd2c7e00640 1 --2- 172.21.15.67:0/2853960374 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2c8102770 0x7fd2c8197ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:45.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.058+0000 7fd2cdc00640 1 --2- 172.21.15.67:0/2853960374 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd2c8103130 0x7fd2c819ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:45.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.058+0000 7fd2cd200640 1 -- 172.21.15.67:0/2853960374 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd2c8103130 msgr2=0x7fd2c819ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:45.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.058+0000 7fd2cd200640 1 --2- 172.21.15.67:0/2853960374 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd2c8103130 0x7fd2c819ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:45.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.058+0000 7fd2cd200640 1 -- 172.21.15.67:0/2853960374 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2c8102770 msgr2=0x7fd2c8197ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:45.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.058+0000 7fd2cd200640 1 --2- 172.21.15.67:0/2853960374 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2c8102770 0x7fd2c8197ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:45.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.058+0000 7fd2cd200640 1 -- 172.21.15.67:0/2853960374 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2c819f380 con 0x7fd2c8101580 2024-03-28T16:05:45.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.058+0000 7fd2cdc00640 1 --2- 172.21.15.67:0/2853960374 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd2c8103130 0x7fd2c819ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:05:45.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.058+0000 7fd2cd200640 1 --2- 172.21.15.67:0/2853960374 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2c8101580 0x7fd2c8197760 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc000c00 tx=0x7fd2bc004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:05:45.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.058+0000 7fd2c5600640 1 -- 172.21.15.67:0/2853960374 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2bc043020 con 0x7fd2c8101580 2024-03-28T16:05:45.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.059+0000 7fd2c5600640 1 -- 172.21.15.67:0/2853960374 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd2bc004aa0 con 0x7fd2c8101580 2024-03-28T16:05:45.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.059+0000 7fd2ce600640 1 -- 172.21.15.67:0/2853960374 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2c819f610 con 0x7fd2c8101580 2024-03-28T16:05:45.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.059+0000 7fd2c5600640 1 -- 172.21.15.67:0/2853960374 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2bc003d00 con 0x7fd2c8101580 2024-03-28T16:05:45.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.059+0000 7fd2ce600640 1 -- 172.21.15.67:0/2853960374 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2c819fb60 con 0x7fd2c8101580 2024-03-28T16:05:45.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.062+0000 7fd2d2872740 1 Processor -- start 2024-03-28T16:05:45.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.063+0000 7fd2d2872740 1 -- start start 2024-03-28T16:05:45.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.063+0000 7fd2d2872740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5610156fc460 0x5610156fc880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:45.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.063+0000 7fd2d2872740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5610156fce50 0x561015701950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:45.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.063+0000 7fd2c5600640 1 -- 172.21.15.67:0/2853960374 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fd2bc046020 con 0x7fd2c8101580 2024-03-28T16:05:45.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.063+0000 7fd2d2872740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x561015701fc0 0x561015704450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:45.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.063+0000 7fd2d2872740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x56101565d250 con 0x5610156fc460 2024-03-28T16:05:45.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.063+0000 7fd2d2872740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x561015605a30 con 0x5610156fce50 2024-03-28T16:05:45.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.063+0000 7fd2d2872740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610156592f0 con 0x561015701fc0 2024-03-28T16:05:45.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.063+0000 7fd2cdc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5610156fc460 0x5610156fc880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:45.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.063+0000 7fd2cdc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5610156fc460 0x5610156fc880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51088/0 (socket says 172.21.15.67:51088) 2024-03-28T16:05:45.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.063+0000 7fd2cdc00640 1 -- 172.21.15.67:0/2882464906 learned_addr learned my addr 172.21.15.67:0/2882464906 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:05:45.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.064+0000 7fd2c7e00640 1 --2- 172.21.15.67:0/2882464906 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x561015701fc0 0x561015704450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:45.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.064+0000 7fd2cdc00640 1 -- 172.21.15.67:0/2882464906 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x561015701fc0 msgr2=0x561015704450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:05:45.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.064+0000 7fd2cdc00640 1 --2- 172.21.15.67:0/2882464906 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x561015701fc0 0x561015704450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:45.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.064+0000 7fd2cdc00640 1 -- 172.21.15.67:0/2882464906 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5610156fce50 msgr2=0x561015701950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:05:45.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.064+0000 7fd2cdc00640 1 --2- 172.21.15.67:0/2882464906 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5610156fce50 0x561015701950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:45.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.064+0000 7fd2cdc00640 1 -- 172.21.15.67:0/2882464906 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5610155f0690 con 0x5610156fc460 2024-03-28T16:05:45.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.064+0000 7fd2c7e00640 1 --2- 172.21.15.67:0/2882464906 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x561015701fc0 0x561015704450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:05:45.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.065+0000 7fd2cdc00640 1 --2- 172.21.15.67:0/2882464906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5610156fc460 0x5610156fc880 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fd2c0008e60 tx=0x7fd2c000d710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=144d3a8d6349abe1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:05:45.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.065+0000 7fd2c5600640 1 --2- 172.21.15.67:0/2853960374 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fd2b00a2800 0x7fd2b00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:45.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.065+0000 7fd2ba000640 1 -- 172.21.15.67:0/2882464906 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2c000f830 con 0x5610156fc460 2024-03-28T16:05:45.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.065+0000 7fd2ba000640 1 -- 172.21.15.67:0/2882464906 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd2c0018480 con 0x5610156fc460 2024-03-28T16:05:45.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.065+0000 7fd2ba000640 1 -- 172.21.15.67:0/2882464906 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2c000e5d0 con 0x5610156fc460 2024-03-28T16:05:45.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.065+0000 7fd2c5600640 1 -- 172.21.15.67:0/2853960374 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fd2bc0f1180 con 0x7fd2c8101580 2024-03-28T16:05:45.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.065+0000 7fd2cd200640 1 --2- 172.21.15.67:0/2853960374 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fd2b00a2800 0x7fd2b00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:45.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.066+0000 7fd2d2872740 1 -- 172.21.15.67:0/2882464906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5610156fc460 msgr2=0x5610156fc880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:45.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.066+0000 7fd2d2872740 1 --2- 172.21.15.67:0/2882464906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5610156fc460 0x5610156fc880 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fd2c0008e60 tx=0x7fd2c000d710 comp rx=0 tx=0).stop 2024-03-28T16:05:45.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.066+0000 7fd2d2872740 1 -- 172.21.15.67:0/2882464906 shutdown_connections 2024-03-28T16:05:45.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.066+0000 7fd2d2872740 1 --2- 172.21.15.67:0/2882464906 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x561015701fc0 0x561015704450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:45.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.066+0000 7fd2d2872740 1 --2- 172.21.15.67:0/2882464906 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5610156fce50 0x561015701950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:45.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.066+0000 7fd2d2872740 1 --2- 172.21.15.67:0/2882464906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5610156fc460 0x5610156fc880 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:45.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.066+0000 7fd2d2872740 1 -- 172.21.15.67:0/2882464906 >> 172.21.15.67:0/2882464906 conn(0x5610156c0200 msgr2=0x5610156f39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:05:45.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.066+0000 7fd2cd200640 1 --2- 172.21.15.67:0/2853960374 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fd2b00a2800 0x7fd2b00a4cc0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc0ba4a0 tx=0x7fd2bc02c7f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:05:45.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.066+0000 7fd2d2872740 1 -- 172.21.15.67:0/2882464906 shutdown_connections 2024-03-28T16:05:45.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.066+0000 7fd2d2872740 1 -- 172.21.15.67:0/2882464906 wait complete. 2024-03-28T16:05:45.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.067+0000 7fd2d2872740 1 -- start start 2024-03-28T16:05:45.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.067+0000 7fd2d2872740 1 Processor -- start 2024-03-28T16:05:45.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.067+0000 7fd2d2872740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5610156fc460 0x561015706470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:45.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.067+0000 7fd2d2872740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5610156fce50 0x5610157069b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:45.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.067+0000 7fd2d2872740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561015701fc0 0x56101570d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:45.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.067+0000 7fd2d2872740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x561015704f70 con 0x561015701fc0 2024-03-28T16:05:45.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.067+0000 7fd2d2872740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x561015618e80 con 0x5610156fce50 2024-03-28T16:05:45.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.067+0000 7fd2d2872740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610157050f0 con 0x5610156fc460 2024-03-28T16:05:45.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.067+0000 7fd2d2872740 10 client.0 _ll_register_callbacks cb 0x56101562b130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:05:45.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.068+0000 7fd2cdc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561015701fc0 0x56101570d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:45.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.068+0000 7fd2cdc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561015701fc0 0x56101570d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51090/0 (socket says 172.21.15.67:51090) 2024-03-28T16:05:45.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.068+0000 7fd2cdc00640 1 -- 172.21.15.67:0/1460130693 learned_addr learned my addr 172.21.15.67:0/1460130693 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:05:45.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.068+0000 7fd2c7e00640 1 --2- 172.21.15.67:0/1460130693 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5610156fce50 0x5610157069b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:45.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.068+0000 7fd2cdc00640 1 --2- 172.21.15.67:0/1460130693 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5610156fc460 0x561015706470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:45.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.068+0000 7fd2cdc00640 1 -- 172.21.15.67:0/1460130693 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5610156fc460 msgr2=0x561015706470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:05:45.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.068+0000 7fd2cdc00640 1 --2- 172.21.15.67:0/1460130693 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5610156fc460 0x561015706470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:45.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.068+0000 7fd2cdc00640 1 -- 172.21.15.67:0/1460130693 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5610156fce50 msgr2=0x5610157069b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:05:45.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.068+0000 7fd2cdc00640 1 --2- 172.21.15.67:0/1460130693 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5610156fce50 0x5610157069b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:45.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.068+0000 7fd2cdc00640 1 -- 172.21.15.67:0/1460130693 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56101543b7d0 con 0x561015701fc0 2024-03-28T16:05:45.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.069+0000 7fd2cdc00640 1 --2- 172.21.15.67:0/1460130693 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561015701fc0 0x56101570d9f0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fd2c00085d0 tx=0x7fd2c0018f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e69c3a2d1af196b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:05:45.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.069+0000 7fd2aea00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:05:45.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.069+0000 7fd2aea00640 1 -- 172.21.15.67:0/1460130693 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2c0021040 con 0x561015701fc0 2024-03-28T16:05:45.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.069+0000 7fd2d2872740 1 -- 172.21.15.67:0/1460130693 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd2c0018480 con 0x561015701fc0 2024-03-28T16:05:45.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.069+0000 7fd2aea00640 1 -- 172.21.15.67:0/1460130693 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fd2c000ec60 con 0x561015701fc0 2024-03-28T16:05:45.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.070+0000 7fd2aea00640 1 -- 172.21.15.67:0/1460130693 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd2c000ee60 con 0x561015701fc0 2024-03-28T16:05:45.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.070+0000 7fd2d2872740 10 client.15633 fetch_fsmap learned FSMap version 19 2024-03-28T16:05:45.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.070+0000 7fd2d2872740 1 -- 172.21.15.67:0/1460130693 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5610153713f0 con 0x561015701fc0 2024-03-28T16:05:45.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.070+0000 7fd2aea00640 1 -- 172.21.15.67:0/1460130693 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd2c0018480 con 0x561015701fc0 2024-03-28T16:05:45.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.070+0000 7fd2aea00640 1 -- 172.21.15.67:0/1460130693 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fd2c00285c0 con 0x561015701fc0 2024-03-28T16:05:45.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.070+0000 7fd2d2872740 10 client.15633 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:05:45.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.070+0000 7fd2d2872740 10 client.15633 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:05:45.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.070+0000 7fd2d2872740 20 client.15633 populate_metadata read hostname 'smithi067' 2024-03-28T16:05:45.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.070+0000 7fd2d2872740 1 --2- 172.21.15.67:0/1460130693 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x561015711f60 0x561015714350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:45.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.070+0000 7fd2d2872740 4 client.15633 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:05:45.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.070+0000 7fd2d2872740 1 -- 172.21.15.67:0/1460130693 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56101566a790 con 0x561015711f60 2024-03-28T16:05:45.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.071+0000 7fd2c7e00640 1 --2- 172.21.15.67:0/1460130693 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x561015711f60 0x561015714350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:45.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.072+0000 7fd2c7e00640 1 --2- 172.21.15.67:0/1460130693 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x561015711f60 0x561015714350 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab87728ce4c800e5 server_cookie=eed6c7d5f25c0cfc in_seq=0 out_seq=0 2024-03-28T16:05:45.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.072+0000 7fd2aea00640 10 client.15633 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:05:45.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.074+0000 7fd2aea00640 1 -- 172.21.15.67:0/1460130693 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56101566a790 con 0x561015711f60 2024-03-28T16:05:45.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.074+0000 7fd2aea00640 10 client.15633 handle_command_reply: tid=0 2024-03-28T16:05:45.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.074+0000 7fd2d2872740 1 client.15633 shutdown 2024-03-28T16:05:45.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.074+0000 7fd2d2872740 1 -- 172.21.15.67:0/1460130693 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561015701fc0 msgr2=0x56101570d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:45.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.074+0000 7fd2d2872740 1 --2- 172.21.15.67:0/1460130693 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561015701fc0 0x56101570d9f0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fd2c00085d0 tx=0x7fd2c0018f90 comp rx=0 tx=0).stop 2024-03-28T16:05:45.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.074+0000 7fd2d2872740 1 -- 172.21.15.67:0/1460130693 shutdown_connections 2024-03-28T16:05:45.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.074+0000 7fd2d2872740 1 --2- 172.21.15.67:0/1460130693 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x561015711f60 0x561015714350 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:45.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.074+0000 7fd2d2872740 1 --2- 172.21.15.67:0/1460130693 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561015701fc0 0x56101570d9f0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:45.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.074+0000 7fd2d2872740 1 --2- 172.21.15.67:0/1460130693 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5610156fce50 0x5610157069b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:45.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.074+0000 7fd2d2872740 1 --2- 172.21.15.67:0/1460130693 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5610156fc460 0x561015706470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:45.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.074+0000 7fd2aea00640 0 client.15633 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:05:45.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.074+0000 7fd2d2872740 1 -- 172.21.15.67:0/1460130693 >> 172.21.15.67:0/1460130693 conn(0x5610156c0200 msgr2=0x5610156f2bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:05:45.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.074+0000 7fd2d2872740 1 -- 172.21.15.67:0/1460130693 shutdown_connections 2024-03-28T16:05:45.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.074+0000 7fd2d2872740 1 -- 172.21.15.67:0/1460130693 wait complete. 2024-03-28T16:05:45.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.074+0000 7fd2d2872740 20 client.15633 trim_cache size 0 max 16384 2024-03-28T16:05:45.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.094+0000 7fd2d2872740 1 Processor -- start 2024-03-28T16:05:45.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.094+0000 7fd2d2872740 1 -- start start 2024-03-28T16:05:45.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.094+0000 7fd2d2872740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5610156fc460 0x5610156fe3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:45.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.094+0000 7fd2d2872740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5610156fce50 0x56101570d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:45.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.094+0000 7fd2d2872740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x561015701fc0 0x5610156f2330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:45.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.094+0000 7fd2d2872740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c81a45e0 con 0x5610156fce50 2024-03-28T16:05:45.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.094+0000 7fd2d2872740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c81a6ba0 con 0x5610156fc460 2024-03-28T16:05:45.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.094+0000 7fd2d2872740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c81a4be0 con 0x561015701fc0 2024-03-28T16:05:45.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.095+0000 7fd2c7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5610156fce50 0x56101570d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:45.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.095+0000 7fd2c7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5610156fce50 0x56101570d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51106/0 (socket says 172.21.15.67:51106) 2024-03-28T16:05:45.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.095+0000 7fd2c7e00640 1 -- 172.21.15.67:0/327808317 learned_addr learned my addr 172.21.15.67:0/327808317 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:05:45.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.095+0000 7fd2cdc00640 1 --2- 172.21.15.67:0/327808317 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5610156fc460 0x5610156fe3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:45.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.095+0000 7fd2cdc00640 1 --2- 172.21.15.67:0/327808317 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x561015701fc0 0x5610156f2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:45.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.095+0000 7fd2c7e00640 1 -- 172.21.15.67:0/327808317 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x561015701fc0 msgr2=0x5610156f2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:05:45.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.095+0000 7fd2c7e00640 1 --2- 172.21.15.67:0/327808317 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x561015701fc0 0x5610156f2330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:45.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.095+0000 7fd2c7e00640 1 -- 172.21.15.67:0/327808317 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5610156fc460 msgr2=0x5610156fe3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:05:45.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.095+0000 7fd2c7e00640 1 --2- 172.21.15.67:0/327808317 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5610156fc460 0x5610156fe3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:45.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.095+0000 7fd2c7e00640 1 -- 172.21.15.67:0/327808317 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5610153310f0 con 0x5610156fce50 2024-03-28T16:05:45.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.096+0000 7fd2c7e00640 1 --2- 172.21.15.67:0/327808317 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5610156fce50 0x56101570d980 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x561015714bc0 tx=0x7fd2b400b4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1991b38e73002e92 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:05:45.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.096+0000 7fd2b9600640 1 -- 172.21.15.67:0/327808317 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b401f020 con 0x5610156fce50 2024-03-28T16:05:45.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.096+0000 7fd2b9600640 1 -- 172.21.15.67:0/327808317 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd2b401b2c0 con 0x5610156fce50 2024-03-28T16:05:45.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.096+0000 7fd2b9600640 1 -- 172.21.15.67:0/327808317 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b401b7f0 con 0x5610156fce50 2024-03-28T16:05:45.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.097+0000 7fd2d2872740 1 -- 172.21.15.67:0/327808317 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5610156fce50 msgr2=0x56101570d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:45.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.097+0000 7fd2d2872740 1 --2- 172.21.15.67:0/327808317 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5610156fce50 0x56101570d980 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x561015714bc0 tx=0x7fd2b400b4d0 comp rx=0 tx=0).stop 2024-03-28T16:05:45.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.097+0000 7fd2d2872740 1 -- 172.21.15.67:0/327808317 shutdown_connections 2024-03-28T16:05:45.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.097+0000 7fd2d2872740 1 --2- 172.21.15.67:0/327808317 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x561015701fc0 0x5610156f2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:45.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.097+0000 7fd2d2872740 1 --2- 172.21.15.67:0/327808317 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5610156fce50 0x56101570d980 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:45.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.097+0000 7fd2d2872740 1 --2- 172.21.15.67:0/327808317 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5610156fc460 0x5610156fe3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:45.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.097+0000 7fd2d2872740 1 -- 172.21.15.67:0/327808317 >> 172.21.15.67:0/327808317 conn(0x5610156c0200 msgr2=0x5610156f7490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:05:45.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.097+0000 7fd2d2872740 1 -- 172.21.15.67:0/327808317 shutdown_connections 2024-03-28T16:05:45.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.097+0000 7fd2d2872740 1 -- 172.21.15.67:0/327808317 wait complete. 2024-03-28T16:05:45.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.098+0000 7fd2d2872740 1 -- start start 2024-03-28T16:05:45.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.098+0000 7fd2d2872740 1 Processor -- start 2024-03-28T16:05:45.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.099+0000 7fd2d2872740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5610156fc460 0x56101570b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:45.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.099+0000 7fd2d2872740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5610156fce50 0x56101570bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:45.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.099+0000 7fd2d2872740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x561015701fc0 0x56101570cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:45.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.099+0000 7fd2d2872740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c81a5220 con 0x5610156fce50 2024-03-28T16:05:45.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.099+0000 7fd2d2872740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c81a6540 con 0x5610156fc460 2024-03-28T16:05:45.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.099+0000 7fd2d2872740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c81a5880 con 0x561015701fc0 2024-03-28T16:05:45.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.099+0000 7fd2d2872740 10 client.0 _ll_register_callbacks cb 0x7fd2c0008cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:05:45.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.099+0000 7fd2c7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5610156fce50 0x56101570bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:45.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.099+0000 7fd2c7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5610156fce50 0x56101570bdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51112/0 (socket says 172.21.15.67:51112) 2024-03-28T16:05:45.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.099+0000 7fd2c7e00640 1 -- 172.21.15.67:0/1732067941 learned_addr learned my addr 172.21.15.67:0/1732067941 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:05:45.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.099+0000 7fd2cdc00640 1 --2- 172.21.15.67:0/1732067941 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5610156fc460 0x56101570b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:45.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.099+0000 7fd2c7e00640 1 -- 172.21.15.67:0/1732067941 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x561015701fc0 msgr2=0x56101570cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:05:45.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.099+0000 7fd2c7e00640 1 --2- 172.21.15.67:0/1732067941 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x561015701fc0 0x56101570cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:45.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.099+0000 7fd2c7e00640 1 -- 172.21.15.67:0/1732067941 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5610156fc460 msgr2=0x56101570b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:05:45.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.099+0000 7fd2c7e00640 1 --2- 172.21.15.67:0/1732067941 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5610156fc460 0x56101570b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:45.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.099+0000 7fd2c7e00640 1 -- 172.21.15.67:0/1732067941 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56101570d350 con 0x5610156fce50 2024-03-28T16:05:45.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.100+0000 7fd2c7e00640 1 --2- 172.21.15.67:0/1732067941 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5610156fce50 0x56101570bdc0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fd2b4003920 tx=0x7fd2b4003950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2baf4b865452587d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:05:45.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.100+0000 7fd2aea00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:05:45.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.100+0000 7fd2aea00640 1 -- 172.21.15.67:0/1732067941 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b401f030 con 0x5610156fce50 2024-03-28T16:05:45.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.100+0000 7fd2aea00640 1 -- 172.21.15.67:0/1732067941 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fd2b4004340 con 0x5610156fce50 2024-03-28T16:05:45.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.100+0000 7fd2d2872740 1 -- 172.21.15.67:0/1732067941 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd2b401b2c0 con 0x5610156fce50 2024-03-28T16:05:45.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.101+0000 7fd2aea00640 1 -- 172.21.15.67:0/1732067941 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd2b402d480 con 0x5610156fce50 2024-03-28T16:05:45.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.101+0000 7fd2d2872740 10 client.15639 fetch_fsmap learned FSMap version 19 2024-03-28T16:05:45.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.101+0000 7fd2d2872740 1 -- 172.21.15.67:0/1732067941 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x56101576df40 con 0x5610156fce50 2024-03-28T16:05:45.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.101+0000 7fd2aea00640 1 -- 172.21.15.67:0/1732067941 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd2b401b2c0 con 0x5610156fce50 2024-03-28T16:05:45.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.101+0000 7fd2aea00640 1 -- 172.21.15.67:0/1732067941 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fd2b400f040 con 0x5610156fce50 2024-03-28T16:05:45.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.101+0000 7fd2d2872740 10 client.15639 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:05:45.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.101+0000 7fd2d2872740 10 client.15639 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:05:45.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.101+0000 7fd2d2872740 20 client.15639 populate_metadata read hostname 'smithi067' 2024-03-28T16:05:45.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.101+0000 7fd2d2872740 1 --2- 172.21.15.67:0/1732067941 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x56101576e810 0x561015770c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:05:45.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.101+0000 7fd2d2872740 4 client.15639 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T16:05:45.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.101+0000 7fd2d2872740 1 -- 172.21.15.67:0/1732067941 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x561015664ae0 con 0x56101576e810 2024-03-28T16:05:45.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.102+0000 7fd2cdc00640 1 --2- 172.21.15.67:0/1732067941 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x56101576e810 0x561015770c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:05:45.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.103+0000 7fd2cdc00640 1 --2- 172.21.15.67:0/1732067941 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x56101576e810 0x561015770c30 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4949feaa0cb5ecea server_cookie=e4b73285dbd2a7d4 in_seq=0 out_seq=0 2024-03-28T16:05:45.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.103+0000 7fd2aea00640 10 client.15639 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:05:45.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.105+0000 7fd2aea00640 1 -- 172.21.15.67:0/1732067941 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x561015664ae0 con 0x56101576e810 2024-03-28T16:05:45.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.105+0000 7fd2aea00640 10 client.15639 handle_command_reply: tid=0 2024-03-28T16:05:45.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.105+0000 7fd2d2872740 1 client.15639 shutdown 2024-03-28T16:05:45.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.105+0000 7fd2d2872740 1 -- 172.21.15.67:0/1732067941 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5610156fce50 msgr2=0x56101570bdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:45.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.105+0000 7fd2d2872740 1 --2- 172.21.15.67:0/1732067941 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5610156fce50 0x56101570bdc0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fd2b4003920 tx=0x7fd2b4003950 comp rx=0 tx=0).stop 2024-03-28T16:05:45.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.105+0000 7fd2d2872740 1 -- 172.21.15.67:0/1732067941 shutdown_connections 2024-03-28T16:05:45.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.105+0000 7fd2d2872740 1 --2- 172.21.15.67:0/1732067941 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x56101576e810 0x561015770c30 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:45.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.105+0000 7fd2d2872740 1 --2- 172.21.15.67:0/1732067941 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x561015701fc0 0x56101570cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:45.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.105+0000 7fd2d2872740 1 --2- 172.21.15.67:0/1732067941 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5610156fce50 0x56101570bdc0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:45.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.105+0000 7fd2d2872740 1 --2- 172.21.15.67:0/1732067941 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5610156fc460 0x56101570b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:45.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.105+0000 7fd2d2872740 1 -- 172.21.15.67:0/1732067941 >> 172.21.15.67:0/1732067941 conn(0x5610156c0200 msgr2=0x561015718c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:05:45.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.105+0000 7fd2aea00640 0 client.15639 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:05:45.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.105+0000 7fd2d2872740 1 -- 172.21.15.67:0/1732067941 shutdown_connections 2024-03-28T16:05:45.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.106+0000 7fd2d2872740 1 -- 172.21.15.67:0/1732067941 wait complete. 2024-03-28T16:05:45.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.106+0000 7fd2d2872740 20 client.15639 trim_cache size 0 max 16384 2024-03-28T16:05:45.117 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-28T16:05:45.118 INFO:teuthology.orchestra.run.smithi067.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T16:05:45.118 INFO:teuthology.orchestra.run.smithi067.stdout: "scrubs": { 2024-03-28T16:05:45.118 INFO:teuthology.orchestra.run.smithi067.stdout: "2599a8ca-4d10-4079-92c0-ac1b71729586": { 2024-03-28T16:05:45.118 INFO:teuthology.orchestra.run.smithi067.stdout: "path": "/", 2024-03-28T16:05:45.118 INFO:teuthology.orchestra.run.smithi067.stdout: "tag": "2599a8ca-4d10-4079-92c0-ac1b71729586", 2024-03-28T16:05:45.118 INFO:teuthology.orchestra.run.smithi067.stdout: "options": "recursive,force" 2024-03-28T16:05:45.118 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:05:45.119 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:05:45.119 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-28T16:05:45.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.106+0000 7fd2ce600640 1 -- 172.21.15.67:0/2853960374 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fd2b00a2800 msgr2=0x7fd2b00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:45.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.106+0000 7fd2ce600640 1 --2- 172.21.15.67:0/2853960374 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fd2b00a2800 0x7fd2b00a4cc0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc0ba4a0 tx=0x7fd2bc02c7f0 comp rx=0 tx=0).stop 2024-03-28T16:05:45.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.106+0000 7fd2ce600640 1 -- 172.21.15.67:0/2853960374 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2c8101580 msgr2=0x7fd2c8197760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:05:45.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.106+0000 7fd2ce600640 1 --2- 172.21.15.67:0/2853960374 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2c8101580 0x7fd2c8197760 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc000c00 tx=0x7fd2bc004270 comp rx=0 tx=0).stop 2024-03-28T16:05:45.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.107+0000 7fd2ce600640 1 -- 172.21.15.67:0/2853960374 shutdown_connections 2024-03-28T16:05:45.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.107+0000 7fd2ce600640 1 --2- 172.21.15.67:0/2853960374 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fd2b00a2800 0x7fd2b00a4cc0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:45.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.107+0000 7fd2ce600640 1 --2- 172.21.15.67:0/2853960374 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd2c8103130 0x7fd2c819ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:45.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.107+0000 7fd2ce600640 1 --2- 172.21.15.67:0/2853960374 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2c8102770 0x7fd2c8197ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:45.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.107+0000 7fd2ce600640 1 --2- 172.21.15.67:0/2853960374 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2c8101580 0x7fd2c8197760 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:05:45.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.107+0000 7fd2ce600640 1 -- 172.21.15.67:0/2853960374 >> 172.21.15.67:0/2853960374 conn(0x7fd2c80fcb10 msgr2=0x7fd2c8105af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:05:45.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.107+0000 7fd2ce600640 1 -- 172.21.15.67:0/2853960374 shutdown_connections 2024-03-28T16:05:45.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:05:45.107+0000 7fd2ce600640 1 -- 172.21.15.67:0/2853960374 wait complete. 2024-03-28T16:05:45.125 INFO:tasks.cephfs.filesystem:scrub status for tag:2599a8ca-4d10-4079-92c0-ac1b71729586 - {'path': '/', 'tag': '2599a8ca-4d10-4079-92c0-ac1b71729586', 'options': 'recursive,force'} 2024-03-28T16:05:46.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:05:46 smithi150 ceph-mon[34825]: scrub queued for path: / 2024-03-28T16:05:46.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:05:46 smithi150 ceph-mon[34825]: scrub summary: idle+waiting paths [/] 2024-03-28T16:05:46.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:05:46 smithi150 ceph-mon[34825]: scrub summary: active paths [/] 2024-03-28T16:05:46.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:05:46 smithi150 ceph-mon[34825]: scrub summary: idle+waiting paths [/] 2024-03-28T16:05:46.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:05:46 smithi150 ceph-mon[34825]: pgmap v505: 97 pgs: 97 active+clean; 797 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 22 MiB/s wr, 8 op/s 2024-03-28T16:05:46.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:05:46 smithi067 ceph-mon[31707]: scrub queued for path: / 2024-03-28T16:05:46.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:05:46 smithi067 ceph-mon[31707]: scrub summary: idle+waiting paths [/] 2024-03-28T16:05:46.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:05:46 smithi067 ceph-mon[31707]: scrub summary: active paths [/] 2024-03-28T16:05:46.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:05:46 smithi067 ceph-mon[31707]: scrub summary: idle+waiting paths [/] 2024-03-28T16:05:46.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:05:46 smithi067 ceph-mon[31707]: pgmap v505: 97 pgs: 97 active+clean; 797 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 22 MiB/s wr, 8 op/s 2024-03-28T16:05:46.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:05:46 smithi183 ceph-mon[35580]: scrub queued for path: / 2024-03-28T16:05:46.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:05:46 smithi183 ceph-mon[35580]: scrub summary: idle+waiting paths [/] 2024-03-28T16:05:46.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:05:46 smithi183 ceph-mon[35580]: scrub summary: active paths [/] 2024-03-28T16:05:46.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:05:46 smithi183 ceph-mon[35580]: scrub summary: idle+waiting paths [/] 2024-03-28T16:05:46.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:05:46 smithi183 ceph-mon[35580]: pgmap v505: 97 pgs: 97 active+clean; 797 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 22 MiB/s wr, 8 op/s 2024-03-28T16:05:47.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:05:47 smithi150 ceph-mon[34825]: scrub summary: idle 2024-03-28T16:05:47.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:05:47 smithi067 ceph-mon[31707]: scrub summary: idle 2024-03-28T16:05:47.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:05:47 smithi183 ceph-mon[35580]: scrub summary: idle 2024-03-28T16:05:48.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:05:48 smithi150 ceph-mon[34825]: pgmap v506: 97 pgs: 97 active+clean; 829 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 21 MiB/s wr, 7 op/s 2024-03-28T16:05:48.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:05:48 smithi067 ceph-mon[31707]: pgmap v506: 97 pgs: 97 active+clean; 829 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 21 MiB/s wr, 7 op/s 2024-03-28T16:05:48.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:05:48 smithi183 ceph-mon[35580]: pgmap v506: 97 pgs: 97 active+clean; 829 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 21 MiB/s wr, 7 op/s 2024-03-28T16:05:50.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:05:50 smithi150 ceph-mon[34825]: pgmap v507: 97 pgs: 97 active+clean; 873 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 23 MiB/s wr, 7 op/s 2024-03-28T16:05:50.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:05:50 smithi067 ceph-mon[31707]: pgmap v507: 97 pgs: 97 active+clean; 873 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 23 MiB/s wr, 7 op/s 2024-03-28T16:05:50.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:05:50 smithi183 ceph-mon[35580]: pgmap v507: 97 pgs: 97 active+clean; 873 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 23 MiB/s wr, 7 op/s 2024-03-28T16:05:52.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:05:52 smithi150 ceph-mon[34825]: pgmap v508: 97 pgs: 97 active+clean; 929 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 24 MiB/s wr, 8 op/s 2024-03-28T16:05:52.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:05:52 smithi067 ceph-mon[31707]: pgmap v508: 97 pgs: 97 active+clean; 929 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 24 MiB/s wr, 8 op/s 2024-03-28T16:05:52.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:05:52 smithi183 ceph-mon[35580]: pgmap v508: 97 pgs: 97 active+clean; 929 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 24 MiB/s wr, 8 op/s 2024-03-28T16:05:54.304 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-03-28T16:05:54.304 INFO:tasks.workunit.client.0.smithi067.stdout: Children see throughput for 1 initial writers = 19957.24 kB/sec 2024-03-28T16:05:54.304 INFO:tasks.workunit.client.0.smithi067.stdout: Parent sees throughput for 1 initial writers = 19956.64 kB/sec 2024-03-28T16:05:54.304 INFO:tasks.workunit.client.0.smithi067.stdout: Min throughput per process = 19957.24 kB/sec 2024-03-28T16:05:54.305 INFO:tasks.workunit.client.0.smithi067.stdout: Max throughput per process = 19957.24 kB/sec 2024-03-28T16:05:54.305 INFO:tasks.workunit.client.0.smithi067.stdout: Avg throughput per process = 19957.24 kB/sec 2024-03-28T16:05:54.305 INFO:tasks.workunit.client.0.smithi067.stdout: Min xfer = 1048576.00 kB 2024-03-28T16:05:54.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:05:54 smithi150 ceph-mon[34825]: pgmap v509: 97 pgs: 97 active+clean; 969 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 23 MiB/s wr, 7 op/s 2024-03-28T16:05:54.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:05:54 smithi067 ceph-mon[31707]: pgmap v509: 97 pgs: 97 active+clean; 969 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 23 MiB/s wr, 7 op/s 2024-03-28T16:05:54.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:05:54 smithi183 ceph-mon[35580]: pgmap v509: 97 pgs: 97 active+clean; 969 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 23 MiB/s wr, 7 op/s 2024-03-28T16:05:56.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:05:56 smithi150 ceph-mon[34825]: pgmap v510: 97 pgs: 97 active+clean; 1009 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 21 MiB/s wr, 8 op/s 2024-03-28T16:05:56.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:05:56 smithi067 ceph-mon[31707]: pgmap v510: 97 pgs: 97 active+clean; 1009 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 21 MiB/s wr, 8 op/s 2024-03-28T16:05:56.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:05:56 smithi183 ceph-mon[35580]: pgmap v510: 97 pgs: 97 active+clean; 1009 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 21 MiB/s wr, 8 op/s 2024-03-28T16:05:58.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:05:58 smithi150 ceph-mon[34825]: pgmap v511: 97 pgs: 97 active+clean; 1013 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 18 MiB/s wr, 6 op/s 2024-03-28T16:05:58.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:05:58 smithi067 ceph-mon[31707]: pgmap v511: 97 pgs: 97 active+clean; 1013 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 18 MiB/s wr, 6 op/s 2024-03-28T16:05:58.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:05:58 smithi183 ceph-mon[35580]: pgmap v511: 97 pgs: 97 active+clean; 1013 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 18 MiB/s wr, 6 op/s 2024-03-28T16:06:00.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:06:00 smithi150 ceph-mon[34825]: pgmap v512: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 17 MiB/s wr, 6 op/s 2024-03-28T16:06:00.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:06:00 smithi067 ceph-mon[31707]: pgmap v512: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 17 MiB/s wr, 6 op/s 2024-03-28T16:06:00.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:06:00 smithi183 ceph-mon[35580]: pgmap v512: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 17 MiB/s wr, 6 op/s 2024-03-28T16:06:02.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:06:02 smithi150 ceph-mon[34825]: pgmap v513: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 17 MiB/s wr, 7 op/s 2024-03-28T16:06:02.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:06:02 smithi067 ceph-mon[31707]: pgmap v513: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 17 MiB/s wr, 7 op/s 2024-03-28T16:06:02.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:06:02 smithi183 ceph-mon[35580]: pgmap v513: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 17 MiB/s wr, 7 op/s 2024-03-28T16:06:04.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:06:04 smithi150 ceph-mon[34825]: pgmap v514: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 15 MiB/s wr, 5 op/s 2024-03-28T16:06:04.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:06:04 smithi067 ceph-mon[31707]: pgmap v514: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 15 MiB/s wr, 5 op/s 2024-03-28T16:06:04.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:06:04 smithi183 ceph-mon[35580]: pgmap v514: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 15 MiB/s wr, 5 op/s 2024-03-28T16:06:06.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:06:06 smithi150 ceph-mon[34825]: pgmap v515: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 17 MiB/s wr, 7 op/s 2024-03-28T16:06:06.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:06:06 smithi067 ceph-mon[31707]: pgmap v515: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 17 MiB/s wr, 7 op/s 2024-03-28T16:06:06.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:06:06 smithi183 ceph-mon[35580]: pgmap v515: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 17 MiB/s wr, 7 op/s 2024-03-28T16:06:08.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:06:08 smithi067 ceph-mon[31707]: pgmap v516: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 14 MiB/s wr, 5 op/s 2024-03-28T16:06:08.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:06:08 smithi183 ceph-mon[35580]: pgmap v516: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 14 MiB/s wr, 5 op/s 2024-03-28T16:06:09.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:06:08 smithi150 ceph-mon[34825]: pgmap v516: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 14 MiB/s wr, 5 op/s 2024-03-28T16:06:10.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:06:10 smithi067 ceph-mon[31707]: pgmap v517: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 17 MiB/s wr, 5 op/s 2024-03-28T16:06:10.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:06:10 smithi183 ceph-mon[35580]: pgmap v517: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 17 MiB/s wr, 5 op/s 2024-03-28T16:06:11.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:06:10 smithi150 ceph-mon[34825]: pgmap v517: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 17 MiB/s wr, 5 op/s 2024-03-28T16:06:12.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:06:12 smithi183 ceph-mon[35580]: pgmap v518: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 20 MiB/s wr, 7 op/s 2024-03-28T16:06:13.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:06:12 smithi150 ceph-mon[34825]: pgmap v518: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 20 MiB/s wr, 7 op/s 2024-03-28T16:06:13.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:06:12 smithi067 ceph-mon[31707]: pgmap v518: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 20 MiB/s wr, 7 op/s 2024-03-28T16:06:14.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:06:14 smithi183 ceph-mon[35580]: pgmap v519: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 20 MiB/s wr, 6 op/s 2024-03-28T16:06:15.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:06:14 smithi150 ceph-mon[34825]: pgmap v519: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 20 MiB/s wr, 6 op/s 2024-03-28T16:06:15.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:06:14 smithi067 ceph-mon[31707]: pgmap v519: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 20 MiB/s wr, 6 op/s 2024-03-28T16:06:15.126 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T16:06:15.126 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T16:06:15.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.370+0000 7f85c9400640 1 -- 172.21.15.67:0/2502089946 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85c4105350 msgr2=0x7f85c410bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:15.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.370+0000 7f85c9400640 1 --2- 172.21.15.67:0/2502089946 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85c4105350 0x7f85c410bbe0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f85bc009ca0 tx=0x7f85bc02c100 comp rx=0 tx=0).stop 2024-03-28T16:06:15.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.371+0000 7f85c9400640 1 -- 172.21.15.67:0/2502089946 shutdown_connections 2024-03-28T16:06:15.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.371+0000 7f85c9400640 1 --2- 172.21.15.67:0/2502089946 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85c4105350 0x7f85c410bbe0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:15.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.371+0000 7f85c9400640 1 --2- 172.21.15.67:0/2502089946 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f85c4104990 0x7f85c4104e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:15.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.371+0000 7f85c9400640 1 --2- 172.21.15.67:0/2502089946 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85c41037a0 0x7f85c4103bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:15.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.371+0000 7f85c9400640 1 -- 172.21.15.67:0/2502089946 >> 172.21.15.67:0/2502089946 conn(0x7f85c40fed10 msgr2=0x7f85c4101150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:06:15.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.371+0000 7f85c9400640 1 -- 172.21.15.67:0/2502089946 shutdown_connections 2024-03-28T16:06:15.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.371+0000 7f85c9400640 1 -- 172.21.15.67:0/2502089946 wait complete. 2024-03-28T16:06:15.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.372+0000 7f85c9400640 1 Processor -- start 2024-03-28T16:06:15.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.372+0000 7f85c9400640 1 -- start start 2024-03-28T16:06:15.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.372+0000 7f85c9400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f85c41037a0 0x7f85c4199930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:15.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.372+0000 7f85c9400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85c4104990 0x7f85c4199e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:15.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.373+0000 7f85c9400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85c4105350 0x7f85c41a0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:15.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.373+0000 7f85c9400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85c410f2b0 con 0x7f85c4105350 2024-03-28T16:06:15.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.373+0000 7f85c9400640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85c410f130 con 0x7f85c4104990 2024-03-28T16:06:15.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.373+0000 7f85c9400640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85c410f430 con 0x7f85c41037a0 2024-03-28T16:06:15.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.373+0000 7f85c8a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85c4105350 0x7f85c41a0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:15.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.373+0000 7f85c8a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85c4105350 0x7f85c41a0ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48754/0 (socket says 172.21.15.67:48754) 2024-03-28T16:06:15.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.373+0000 7f85c8a00640 1 -- 172.21.15.67:0/637958647 learned_addr learned my addr 172.21.15.67:0/637958647 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:06:15.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.373+0000 7f85c3e00640 1 --2- 172.21.15.67:0/637958647 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f85c41037a0 0x7f85c4199930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:15.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.373+0000 7f85c3400640 1 --2- 172.21.15.67:0/637958647 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85c4104990 0x7f85c4199e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:15.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.373+0000 7f85c8a00640 1 -- 172.21.15.67:0/637958647 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f85c41037a0 msgr2=0x7f85c4199930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:15.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.373+0000 7f85c8a00640 1 --2- 172.21.15.67:0/637958647 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f85c41037a0 0x7f85c4199930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:15.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.373+0000 7f85c8a00640 1 -- 172.21.15.67:0/637958647 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85c4104990 msgr2=0x7f85c4199e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:15.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.373+0000 7f85c8a00640 1 --2- 172.21.15.67:0/637958647 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85c4104990 0x7f85c4199e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:15.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.373+0000 7f85c8a00640 1 -- 172.21.15.67:0/637958647 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85c41a1550 con 0x7f85c4105350 2024-03-28T16:06:15.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.374+0000 7f85c3400640 1 --2- 172.21.15.67:0/637958647 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85c4104990 0x7f85c4199e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T16:06:15.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.374+0000 7f85c8a00640 1 --2- 172.21.15.67:0/637958647 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85c4105350 0x7f85c41a0ee0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f85bc032520 tx=0x7f85bc034880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:06:15.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.374+0000 7f85c0c00640 1 -- 172.21.15.67:0/637958647 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85bc039040 con 0x7f85c4105350 2024-03-28T16:06:15.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.374+0000 7f85c0c00640 1 -- 172.21.15.67:0/637958647 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f85bc044070 con 0x7f85c4105350 2024-03-28T16:06:15.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.374+0000 7f85c9400640 1 -- 172.21.15.67:0/637958647 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85c41a17e0 con 0x7f85c4105350 2024-03-28T16:06:15.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.374+0000 7f85c0c00640 1 -- 172.21.15.67:0/637958647 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85bc0043c0 con 0x7f85c4105350 2024-03-28T16:06:15.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.374+0000 7f85c9400640 1 -- 172.21.15.67:0/637958647 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85c41a1cd0 con 0x7f85c4105350 2024-03-28T16:06:15.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.378+0000 7f85c0c00640 1 -- 172.21.15.67:0/637958647 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f85bc004560 con 0x7f85c4105350 2024-03-28T16:06:15.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.378+0000 7f85cd781740 1 Processor -- start 2024-03-28T16:06:15.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.379+0000 7f85cd781740 1 -- start start 2024-03-28T16:06:15.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.379+0000 7f85cd781740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b42bc13050 0x55b42bc13470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:15.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.379+0000 7f85cd781740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b42bc13a40 0x55b42bc18540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:15.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.379+0000 7f85cd781740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b42bc18bb0 0x55b42bc1b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:15.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.379+0000 7f85cd781740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b42bb78250 con 0x55b42bc13a40 2024-03-28T16:06:15.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.379+0000 7f85cd781740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b42bb20a30 con 0x55b42bc13050 2024-03-28T16:06:15.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.379+0000 7f85cd781740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b42bb742f0 con 0x55b42bc18bb0 2024-03-28T16:06:15.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.379+0000 7f85c3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b42bc13a40 0x55b42bc18540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:15.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.379+0000 7f85c3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b42bc13a40 0x55b42bc18540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48766/0 (socket says 172.21.15.67:48766) 2024-03-28T16:06:15.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.379+0000 7f85c3e00640 1 -- 172.21.15.67:0/3575055550 learned_addr learned my addr 172.21.15.67:0/3575055550 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:06:15.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.379+0000 7f85c3400640 1 --2- 172.21.15.67:0/3575055550 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b42bc13050 0x55b42bc13470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:15.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.380+0000 7f85c3400640 1 --2- 172.21.15.67:0/3575055550 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b42bc18bb0 0x55b42bc1b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:15.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.380+0000 7f85c0c00640 1 --2- 172.21.15.67:0/637958647 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f85a80a2800 0x7f85a80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:15.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.380+0000 7f85c3e00640 1 -- 172.21.15.67:0/3575055550 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b42bc18bb0 msgr2=0x55b42bc1b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:06:15.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.380+0000 7f85c3e00640 1 --2- 172.21.15.67:0/3575055550 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b42bc18bb0 0x55b42bc1b040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:15.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.380+0000 7f85c3e00640 1 -- 172.21.15.67:0/3575055550 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b42bc13050 msgr2=0x55b42bc13470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:06:15.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.380+0000 7f85c3e00640 1 --2- 172.21.15.67:0/3575055550 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b42bc13050 0x55b42bc13470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:15.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.380+0000 7f85c3e00640 1 -- 172.21.15.67:0/3575055550 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b42bb0b690 con 0x55b42bc13a40 2024-03-28T16:06:15.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.380+0000 7f85c0c00640 1 -- 172.21.15.67:0/637958647 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f85bc0f1190 con 0x7f85c4105350 2024-03-28T16:06:15.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.380+0000 7f85c8a00640 1 --2- 172.21.15.67:0/637958647 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f85a80a2800 0x7f85a80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:15.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.380+0000 7f85c3e00640 1 --2- 172.21.15.67:0/3575055550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b42bc13a40 0x55b42bc18540 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f85b400b7d0 tx=0x7f85b4004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=297921d879563fec server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:06:15.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.381+0000 7f85b9200640 1 -- 172.21.15.67:0/3575055550 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b4003dd0 con 0x55b42bc13a40 2024-03-28T16:06:15.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.381+0000 7f85b9200640 1 -- 172.21.15.67:0/3575055550 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f85b4015070 con 0x55b42bc13a40 2024-03-28T16:06:15.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.381+0000 7f85b9200640 1 -- 172.21.15.67:0/3575055550 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b40116a0 con 0x55b42bc13a40 2024-03-28T16:06:15.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.382+0000 7f85cd781740 1 -- 172.21.15.67:0/3575055550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b42bc13a40 msgr2=0x55b42bc18540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:15.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.382+0000 7f85cd781740 1 --2- 172.21.15.67:0/3575055550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b42bc13a40 0x55b42bc18540 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f85b400b7d0 tx=0x7f85b4004610 comp rx=0 tx=0).stop 2024-03-28T16:06:15.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.382+0000 7f85c8a00640 1 --2- 172.21.15.67:0/637958647 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f85a80a2800 0x7f85a80a4cc0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f85bc0324f0 tx=0x7f85bc02c100 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:06:15.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.382+0000 7f85cd781740 1 -- 172.21.15.67:0/3575055550 shutdown_connections 2024-03-28T16:06:15.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.382+0000 7f85cd781740 1 --2- 172.21.15.67:0/3575055550 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b42bc18bb0 0x55b42bc1b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:15.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.382+0000 7f85cd781740 1 --2- 172.21.15.67:0/3575055550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b42bc13a40 0x55b42bc18540 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:15.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.382+0000 7f85cd781740 1 --2- 172.21.15.67:0/3575055550 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b42bc13050 0x55b42bc13470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:15.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.382+0000 7f85cd781740 1 -- 172.21.15.67:0/3575055550 >> 172.21.15.67:0/3575055550 conn(0x55b42bbdb200 msgr2=0x55b42bc0e9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:06:15.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.382+0000 7f85cd781740 1 -- 172.21.15.67:0/3575055550 shutdown_connections 2024-03-28T16:06:15.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.382+0000 7f85cd781740 1 -- 172.21.15.67:0/3575055550 wait complete. 2024-03-28T16:06:15.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.383+0000 7f85cd781740 1 -- start start 2024-03-28T16:06:15.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.383+0000 7f85cd781740 1 Processor -- start 2024-03-28T16:06:15.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.383+0000 7f85cd781740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b42bc13050 0x55b42bc1cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:15.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.383+0000 7f85cd781740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b42bc13a40 0x55b42bc1d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:15.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.383+0000 7f85cd781740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b42bc18bb0 0x55b42bc24550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:15.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.383+0000 7f85cd781740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b42bc1bb60 con 0x55b42bc13a40 2024-03-28T16:06:15.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.383+0000 7f85cd781740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b42bb33e80 con 0x55b42bc18bb0 2024-03-28T16:06:15.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.383+0000 7f85cd781740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b42bc1bce0 con 0x55b42bc13050 2024-03-28T16:06:15.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.383+0000 7f85cd781740 10 client.0 _ll_register_callbacks cb 0x55b42bb46130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:06:15.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.384+0000 7f85c3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b42bc13a40 0x55b42bc1d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:15.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.384+0000 7f85c3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b42bc13a40 0x55b42bc1d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48780/0 (socket says 172.21.15.67:48780) 2024-03-28T16:06:15.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.384+0000 7f85c3e00640 1 -- 172.21.15.67:0/931265884 learned_addr learned my addr 172.21.15.67:0/931265884 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:06:15.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.384+0000 7f85c3400640 1 --2- 172.21.15.67:0/931265884 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b42bc13050 0x55b42bc1cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:15.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.384+0000 7f85c3400640 1 --2- 172.21.15.67:0/931265884 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b42bc18bb0 0x55b42bc24550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:15.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.385+0000 7f85c3400640 1 -- 172.21.15.67:0/931265884 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b42bc13050 msgr2=0x55b42bc1cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:06:15.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.385+0000 7f85c3400640 1 --2- 172.21.15.67:0/931265884 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b42bc13050 0x55b42bc1cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:15.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.385+0000 7f85c3400640 1 -- 172.21.15.67:0/931265884 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b42bc13a40 msgr2=0x55b42bc1d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:06:15.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.385+0000 7f85c3400640 1 --2- 172.21.15.67:0/931265884 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b42bc13a40 0x55b42bc1d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:15.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.385+0000 7f85c3400640 1 -- 172.21.15.67:0/931265884 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b42b9567d0 con 0x55b42bc18bb0 2024-03-28T16:06:15.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.385+0000 7f85c3400640 1 --2- 172.21.15.67:0/931265884 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b42bc18bb0 0x55b42bc24550 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x55b42bc142a0 tx=0x7f85b000c470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb4be7bd5b8ab8c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:06:15.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.385+0000 7f85ad600640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-28T16:06:15.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.385+0000 7f85ad600640 1 -- 172.21.15.67:0/931265884 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b0004040 con 0x55b42bc18bb0 2024-03-28T16:06:15.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.386+0000 7f85cd781740 1 -- 172.21.15.67:0/931265884 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f85b4015070 con 0x55b42bc18bb0 2024-03-28T16:06:15.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.386+0000 7f85ad600640 1 -- 172.21.15.67:0/931265884 <== mon.2 v2:172.21.15.150:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f85b00041e0 con 0x55b42bc18bb0 2024-03-28T16:06:15.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.386+0000 7f85ad600640 1 -- 172.21.15.67:0/931265884 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f85b0028050 con 0x55b42bc18bb0 2024-03-28T16:06:15.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.386+0000 7f85cd781740 10 client.35240 fetch_fsmap learned FSMap version 19 2024-03-28T16:06:15.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.386+0000 7f85cd781740 1 -- 172.21.15.67:0/931265884 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55b42b88c3f0 con 0x55b42bc18bb0 2024-03-28T16:06:15.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.386+0000 7f85ad600640 1 -- 172.21.15.67:0/931265884 <== mon.2 v2:172.21.15.150:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f85b4015070 con 0x55b42bc18bb0 2024-03-28T16:06:15.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.387+0000 7f85ad600640 1 -- 172.21.15.67:0/931265884 <== mon.2 v2:172.21.15.150:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f85b0020020 con 0x55b42bc18bb0 2024-03-28T16:06:15.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.387+0000 7f85cd781740 10 client.35240 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:06:15.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.387+0000 7f85cd781740 10 client.35240 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:06:15.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.387+0000 7f85cd781740 20 client.35240 populate_metadata read hostname 'smithi067' 2024-03-28T16:06:15.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.387+0000 7f85cd781740 1 --2- 172.21.15.67:0/931265884 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55b42bc28e00 0x55b42bc2b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:15.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.387+0000 7f85cd781740 4 client.35240 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:06:15.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.387+0000 7f85cd781740 1 -- 172.21.15.67:0/931265884 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b42bb85790 con 0x55b42bc28e00 2024-03-28T16:06:15.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.388+0000 7f85c3e00640 1 --2- 172.21.15.67:0/931265884 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55b42bc28e00 0x55b42bc2b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:15.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.388+0000 7f85c3e00640 1 --2- 172.21.15.67:0/931265884 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55b42bc28e00 0x55b42bc2b2a0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd1753f7d8b9a0f6 server_cookie=a36dd47bf17d5306 in_seq=0 out_seq=0 2024-03-28T16:06:15.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.388+0000 7f85ad600640 10 client.35240 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:06:15.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.391+0000 7f85ad600640 1 -- 172.21.15.67:0/931265884 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55b42bb85790 con 0x55b42bc28e00 2024-03-28T16:06:15.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.391+0000 7f85ad600640 10 client.35240 handle_command_reply: tid=0 2024-03-28T16:06:15.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.391+0000 7f85cd781740 1 client.35240 shutdown 2024-03-28T16:06:15.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.391+0000 7f85cd781740 1 -- 172.21.15.67:0/931265884 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b42bc18bb0 msgr2=0x55b42bc24550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:15.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.391+0000 7f85cd781740 1 --2- 172.21.15.67:0/931265884 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b42bc18bb0 0x55b42bc24550 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x55b42bc142a0 tx=0x7f85b000c470 comp rx=0 tx=0).stop 2024-03-28T16:06:15.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.391+0000 7f85cd781740 1 -- 172.21.15.67:0/931265884 shutdown_connections 2024-03-28T16:06:15.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.391+0000 7f85cd781740 1 --2- 172.21.15.67:0/931265884 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55b42bc28e00 0x55b42bc2b2a0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:15.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.391+0000 7f85cd781740 1 --2- 172.21.15.67:0/931265884 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b42bc18bb0 0x55b42bc24550 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:15.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.391+0000 7f85cd781740 1 --2- 172.21.15.67:0/931265884 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b42bc13a40 0x55b42bc1d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:15.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.391+0000 7f85cd781740 1 --2- 172.21.15.67:0/931265884 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b42bc13050 0x55b42bc1cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:15.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.391+0000 7f85ad600640 0 client.35240 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:06:15.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.391+0000 7f85cd781740 1 -- 172.21.15.67:0/931265884 >> 172.21.15.67:0/931265884 conn(0x55b42bbdb200 msgr2=0x55b42bc0dbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:06:15.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.391+0000 7f85cd781740 1 -- 172.21.15.67:0/931265884 shutdown_connections 2024-03-28T16:06:15.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.391+0000 7f85cd781740 1 -- 172.21.15.67:0/931265884 wait complete. 2024-03-28T16:06:15.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.391+0000 7f85cd781740 20 client.35240 trim_cache size 0 max 16384 2024-03-28T16:06:15.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.415+0000 7f85cd781740 1 Processor -- start 2024-03-28T16:06:15.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.415+0000 7f85cd781740 1 -- start start 2024-03-28T16:06:15.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.416+0000 7f85cd781740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b42bc13050 0x55b42bc149e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:15.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.416+0000 7f85cd781740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b42bc13a40 0x55b42bc14f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:15.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.416+0000 7f85cd781740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b42bc18bb0 0x55b42bc244e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:15.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.416+0000 7f85cd781740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85c41a8980 con 0x55b42bc18bb0 2024-03-28T16:06:15.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.416+0000 7f85cd781740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85c41aaf40 con 0x55b42bc13050 2024-03-28T16:06:15.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.416+0000 7f85cd781740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85c41a8f80 con 0x55b42bc13a40 2024-03-28T16:06:15.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.416+0000 7f85c3400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b42bc18bb0 0x55b42bc244e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:15.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.416+0000 7f85c3400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b42bc18bb0 0x55b42bc244e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48788/0 (socket says 172.21.15.67:48788) 2024-03-28T16:06:15.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.416+0000 7f85c3400640 1 -- 172.21.15.67:0/4267499416 learned_addr learned my addr 172.21.15.67:0/4267499416 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:06:15.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.416+0000 7f85c3e00640 1 --2- 172.21.15.67:0/4267499416 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b42bc13a40 0x55b42bc14f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:15.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.416+0000 7f85c3400640 1 --2- 172.21.15.67:0/4267499416 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b42bc13050 0x55b42bc149e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:15.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.417+0000 7f85c3400640 1 -- 172.21.15.67:0/4267499416 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b42bc13a40 msgr2=0x55b42bc14f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:06:15.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.417+0000 7f85c3400640 1 --2- 172.21.15.67:0/4267499416 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b42bc13a40 0x55b42bc14f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:15.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.417+0000 7f85c3400640 1 -- 172.21.15.67:0/4267499416 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b42bc13050 msgr2=0x55b42bc149e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:06:15.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.417+0000 7f85c3400640 1 --2- 172.21.15.67:0/4267499416 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b42bc13050 0x55b42bc149e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:15.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.417+0000 7f85c3400640 1 -- 172.21.15.67:0/4267499416 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b42b84c0f0 con 0x55b42bc18bb0 2024-03-28T16:06:15.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.417+0000 7f85c3400640 1 --2- 172.21.15.67:0/4267499416 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b42bc18bb0 0x55b42bc244e0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f85b0012dd0 tx=0x7f85b000c5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64597db94d35b1a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:06:15.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.417+0000 7f85afe00640 1 -- 172.21.15.67:0/4267499416 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b000c770 con 0x55b42bc18bb0 2024-03-28T16:06:15.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.418+0000 7f85afe00640 1 -- 172.21.15.67:0/4267499416 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f85b00213d0 con 0x55b42bc18bb0 2024-03-28T16:06:15.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.418+0000 7f85afe00640 1 -- 172.21.15.67:0/4267499416 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b0005490 con 0x55b42bc18bb0 2024-03-28T16:06:15.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.419+0000 7f85cd781740 1 -- 172.21.15.67:0/4267499416 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b42bc18bb0 msgr2=0x55b42bc244e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:15.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.419+0000 7f85cd781740 1 --2- 172.21.15.67:0/4267499416 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b42bc18bb0 0x55b42bc244e0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f85b0012dd0 tx=0x7f85b000c5c0 comp rx=0 tx=0).stop 2024-03-28T16:06:15.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.419+0000 7f85cd781740 1 -- 172.21.15.67:0/4267499416 shutdown_connections 2024-03-28T16:06:15.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.419+0000 7f85cd781740 1 --2- 172.21.15.67:0/4267499416 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b42bc18bb0 0x55b42bc244e0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:15.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.419+0000 7f85cd781740 1 --2- 172.21.15.67:0/4267499416 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b42bc13a40 0x55b42bc14f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:15.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.419+0000 7f85cd781740 1 --2- 172.21.15.67:0/4267499416 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b42bc13050 0x55b42bc149e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:15.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.419+0000 7f85cd781740 1 -- 172.21.15.67:0/4267499416 >> 172.21.15.67:0/4267499416 conn(0x55b42bbdb200 msgr2=0x55b42bc142d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:06:15.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.419+0000 7f85cd781740 1 -- 172.21.15.67:0/4267499416 shutdown_connections 2024-03-28T16:06:15.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.419+0000 7f85cd781740 1 -- 172.21.15.67:0/4267499416 wait complete. 2024-03-28T16:06:15.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.419+0000 7f85cd781740 1 -- start start 2024-03-28T16:06:15.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.420+0000 7f85cd781740 1 Processor -- start 2024-03-28T16:06:15.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.420+0000 7f85cd781740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b42bc13050 0x55b42bc223f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:15.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.420+0000 7f85cd781740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b42bc13a40 0x55b42bc22930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:15.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.420+0000 7f85cd781740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b42bc18bb0 0x55b42bc23940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:15.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.420+0000 7f85cd781740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85c41a95c0 con 0x55b42bc18bb0 2024-03-28T16:06:15.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.420+0000 7f85cd781740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85c41aa8e0 con 0x55b42bc13050 2024-03-28T16:06:15.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.420+0000 7f85cd781740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85c41a9c20 con 0x55b42bc13a40 2024-03-28T16:06:15.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.420+0000 7f85cd781740 10 client.0 _ll_register_callbacks cb 0x7f85b000ced0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:06:15.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.420+0000 7f85c3400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b42bc18bb0 0x55b42bc23940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:15.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.420+0000 7f85c3400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b42bc18bb0 0x55b42bc23940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48800/0 (socket says 172.21.15.67:48800) 2024-03-28T16:06:15.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.420+0000 7f85c3400640 1 -- 172.21.15.67:0/4041859710 learned_addr learned my addr 172.21.15.67:0/4041859710 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:06:15.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.421+0000 7f85c3400640 1 --2- 172.21.15.67:0/4041859710 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b42bc13050 0x55b42bc223f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:15.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.421+0000 7f85c3e00640 1 --2- 172.21.15.67:0/4041859710 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b42bc13a40 0x55b42bc22930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:15.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.421+0000 7f85c3400640 1 -- 172.21.15.67:0/4041859710 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b42bc13a40 msgr2=0x55b42bc22930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:06:15.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.421+0000 7f85c3400640 1 --2- 172.21.15.67:0/4041859710 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b42bc13a40 0x55b42bc22930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:15.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.421+0000 7f85c3400640 1 -- 172.21.15.67:0/4041859710 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b42bc13050 msgr2=0x55b42bc223f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:06:15.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.421+0000 7f85c3400640 1 --2- 172.21.15.67:0/4041859710 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b42bc13050 0x55b42bc223f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:15.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.421+0000 7f85c3400640 1 -- 172.21.15.67:0/4041859710 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b42bc23f30 con 0x55b42bc18bb0 2024-03-28T16:06:15.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.421+0000 7f85c3400640 1 --2- 172.21.15.67:0/4041859710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b42bc18bb0 0x55b42bc23940 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f85b0003e40 tx=0x7f85b000cfa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2559d8b345dce65a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:06:15.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.421+0000 7f85ad600640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:06:15.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.422+0000 7f85ad600640 1 -- 172.21.15.67:0/4041859710 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b0021580 con 0x55b42bc18bb0 2024-03-28T16:06:15.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.422+0000 7f85ad600640 1 -- 172.21.15.67:0/4041859710 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f85b0005420 con 0x55b42bc18bb0 2024-03-28T16:06:15.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.422+0000 7f85cd781740 1 -- 172.21.15.67:0/4041859710 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f85b00213d0 con 0x55b42bc18bb0 2024-03-28T16:06:15.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.422+0000 7f85ad600640 1 -- 172.21.15.67:0/4041859710 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f85b0004040 con 0x55b42bc18bb0 2024-03-28T16:06:15.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.422+0000 7f85cd781740 10 client.15657 fetch_fsmap learned FSMap version 19 2024-03-28T16:06:15.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.422+0000 7f85cd781740 1 -- 172.21.15.67:0/4041859710 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55b42bc84e40 con 0x55b42bc18bb0 2024-03-28T16:06:15.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.422+0000 7f85ad600640 1 -- 172.21.15.67:0/4041859710 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f85b00213d0 con 0x55b42bc18bb0 2024-03-28T16:06:15.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.422+0000 7f85ad600640 1 -- 172.21.15.67:0/4041859710 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f85b0020050 con 0x55b42bc18bb0 2024-03-28T16:06:15.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.423+0000 7f85cd781740 10 client.15657 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:06:15.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.423+0000 7f85cd781740 10 client.15657 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:06:15.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.423+0000 7f85cd781740 20 client.15657 populate_metadata read hostname 'smithi067' 2024-03-28T16:06:15.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.423+0000 7f85cd781740 1 --2- 172.21.15.67:0/4041859710 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55b42bc85770 0x55b42bc87b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:15.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.423+0000 7f85cd781740 4 client.15657 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T16:06:15.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.423+0000 7f85cd781740 1 -- 172.21.15.67:0/4041859710 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55b42bb7fae0 con 0x55b42bc85770 2024-03-28T16:06:15.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.423+0000 7f85c3e00640 1 --2- 172.21.15.67:0/4041859710 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55b42bc85770 0x55b42bc87b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:15.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.424+0000 7f85c3e00640 1 --2- 172.21.15.67:0/4041859710 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55b42bc85770 0x55b42bc87b90 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec1600fd80e1c614 server_cookie=c5802d242e29d37b in_seq=0 out_seq=0 2024-03-28T16:06:15.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.424+0000 7f85ad600640 10 client.15657 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:06:15.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.425+0000 7f85ad600640 1 -- 172.21.15.67:0/4041859710 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55b42bb7fae0 con 0x55b42bc85770 2024-03-28T16:06:15.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.425+0000 7f85ad600640 10 client.15657 handle_command_reply: tid=0 2024-03-28T16:06:15.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.425+0000 7f85cd781740 1 client.15657 shutdown 2024-03-28T16:06:15.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.425+0000 7f85cd781740 1 -- 172.21.15.67:0/4041859710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b42bc18bb0 msgr2=0x55b42bc23940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:15.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.425+0000 7f85cd781740 1 --2- 172.21.15.67:0/4041859710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b42bc18bb0 0x55b42bc23940 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f85b0003e40 tx=0x7f85b000cfa0 comp rx=0 tx=0).stop 2024-03-28T16:06:15.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.426+0000 7f85cd781740 1 -- 172.21.15.67:0/4041859710 shutdown_connections 2024-03-28T16:06:15.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.426+0000 7f85cd781740 1 --2- 172.21.15.67:0/4041859710 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55b42bc85770 0x55b42bc87b90 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:15.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.426+0000 7f85cd781740 1 --2- 172.21.15.67:0/4041859710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b42bc18bb0 0x55b42bc23940 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:15.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.426+0000 7f85cd781740 1 --2- 172.21.15.67:0/4041859710 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b42bc13a40 0x55b42bc22930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:15.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.426+0000 7f85cd781740 1 --2- 172.21.15.67:0/4041859710 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b42bc13050 0x55b42bc223f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:15.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.426+0000 7f85ad600640 0 client.15657 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:06:15.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.426+0000 7f85cd781740 1 -- 172.21.15.67:0/4041859710 >> 172.21.15.67:0/4041859710 conn(0x55b42bbdb200 msgr2=0x55b42bc11c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:06:15.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.426+0000 7f85cd781740 1 -- 172.21.15.67:0/4041859710 shutdown_connections 2024-03-28T16:06:15.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.426+0000 7f85cd781740 1 -- 172.21.15.67:0/4041859710 wait complete. 2024-03-28T16:06:15.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.426+0000 7f85cd781740 20 client.15657 trim_cache size 0 max 16384 2024-03-28T16:06:15.429 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-28T16:06:15.429 INFO:teuthology.orchestra.run.smithi067.stdout: "status": "no active scrubs running", 2024-03-28T16:06:15.429 INFO:teuthology.orchestra.run.smithi067.stdout: "scrubs": {} 2024-03-28T16:06:15.429 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-28T16:06:15.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.427+0000 7f85c9400640 1 -- 172.21.15.67:0/637958647 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f85a80a2800 msgr2=0x7f85a80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:15.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.427+0000 7f85c9400640 1 --2- 172.21.15.67:0/637958647 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f85a80a2800 0x7f85a80a4cc0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f85bc0324f0 tx=0x7f85bc02c100 comp rx=0 tx=0).stop 2024-03-28T16:06:15.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.427+0000 7f85c9400640 1 -- 172.21.15.67:0/637958647 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85c4105350 msgr2=0x7f85c41a0ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:15.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.427+0000 7f85c9400640 1 --2- 172.21.15.67:0/637958647 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85c4105350 0x7f85c41a0ee0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f85bc032520 tx=0x7f85bc034880 comp rx=0 tx=0).stop 2024-03-28T16:06:15.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.427+0000 7f85c9400640 1 -- 172.21.15.67:0/637958647 shutdown_connections 2024-03-28T16:06:15.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.427+0000 7f85c9400640 1 --2- 172.21.15.67:0/637958647 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f85a80a2800 0x7f85a80a4cc0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:15.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.427+0000 7f85c9400640 1 --2- 172.21.15.67:0/637958647 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85c4105350 0x7f85c41a0ee0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:15.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.427+0000 7f85c9400640 1 --2- 172.21.15.67:0/637958647 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85c4104990 0x7f85c4199e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:15.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.427+0000 7f85c9400640 1 --2- 172.21.15.67:0/637958647 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f85c41037a0 0x7f85c4199930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:15.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.427+0000 7f85c9400640 1 -- 172.21.15.67:0/637958647 >> 172.21.15.67:0/637958647 conn(0x7f85c40fed10 msgr2=0x7f85c4107c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:06:15.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.427+0000 7f85c9400640 1 -- 172.21.15.67:0/637958647 shutdown_connections 2024-03-28T16:06:15.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.427+0000 7f85c9400640 1 -- 172.21.15.67:0/637958647 wait complete. 2024-03-28T16:06:15.448 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T16:06:15.448 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T16:06:15.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.682+0000 7f375e200640 1 -- 172.21.15.67:0/1799832343 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37581030c0 msgr2=0x7f3758109950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:15.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.682+0000 7f375e200640 1 --2- 172.21.15.67:0/1799832343 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37581030c0 0x7f3758109950 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f3750009ca0 tx=0x7f375002c100 comp rx=0 tx=0).stop 2024-03-28T16:06:15.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.682+0000 7f375e200640 1 -- 172.21.15.67:0/1799832343 shutdown_connections 2024-03-28T16:06:15.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.682+0000 7f375e200640 1 --2- 172.21.15.67:0/1799832343 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37581030c0 0x7f3758109950 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:15.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.682+0000 7f375e200640 1 --2- 172.21.15.67:0/1799832343 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3758102700 0x7f3758102b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:15.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.682+0000 7f375e200640 1 --2- 172.21.15.67:0/1799832343 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3758101510 0x7f3758101930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:15.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.682+0000 7f375e200640 1 -- 172.21.15.67:0/1799832343 >> 172.21.15.67:0/1799832343 conn(0x7f37580fca80 msgr2=0x7f37580feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:06:15.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.683+0000 7f375e200640 1 -- 172.21.15.67:0/1799832343 shutdown_connections 2024-03-28T16:06:15.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.683+0000 7f375e200640 1 -- 172.21.15.67:0/1799832343 wait complete. 2024-03-28T16:06:15.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.683+0000 7f375e200640 1 Processor -- start 2024-03-28T16:06:15.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.684+0000 7f375e200640 1 -- start start 2024-03-28T16:06:15.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.684+0000 7f375e200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3758101510 0x7f37581998b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:15.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.684+0000 7f375e200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3758102700 0x7f3758199df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:15.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.684+0000 7f375e200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f37581030c0 0x7f37581a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:15.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.684+0000 7f375e200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f375810d050 con 0x7f3758101510 2024-03-28T16:06:15.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.684+0000 7f375e200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f375810ced0 con 0x7f3758102700 2024-03-28T16:06:15.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.684+0000 7f375e200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f375810d1d0 con 0x7f37581030c0 2024-03-28T16:06:15.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.684+0000 7f375ce00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3758101510 0x7f37581998b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:15.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.684+0000 7f375ce00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3758101510 0x7f37581998b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48822/0 (socket says 172.21.15.67:48822) 2024-03-28T16:06:15.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.684+0000 7f375ce00640 1 -- 172.21.15.67:0/2352518359 learned_addr learned my addr 172.21.15.67:0/2352518359 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:06:15.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.685+0000 7f3757e00640 1 --2- 172.21.15.67:0/2352518359 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3758102700 0x7f3758199df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:15.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.685+0000 7f375d800640 1 --2- 172.21.15.67:0/2352518359 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f37581030c0 0x7f37581a0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:15.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.685+0000 7f375ce00640 1 -- 172.21.15.67:0/2352518359 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f37581030c0 msgr2=0x7f37581a0e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:15.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.685+0000 7f375ce00640 1 --2- 172.21.15.67:0/2352518359 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f37581030c0 0x7f37581a0e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:15.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.685+0000 7f375ce00640 1 -- 172.21.15.67:0/2352518359 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3758102700 msgr2=0x7f3758199df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:15.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.685+0000 7f375ce00640 1 --2- 172.21.15.67:0/2352518359 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3758102700 0x7f3758199df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:15.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.685+0000 7f375ce00640 1 -- 172.21.15.67:0/2352518359 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37581a14d0 con 0x7f3758101510 2024-03-28T16:06:15.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.685+0000 7f375d800640 1 --2- 172.21.15.67:0/2352518359 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f37581030c0 0x7f37581a0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:06:15.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.685+0000 7f375ce00640 1 --2- 172.21.15.67:0/2352518359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3758101510 0x7f37581998b0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f374c00d950 tx=0x7f374c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:06:15.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.685+0000 7f3755600640 1 -- 172.21.15.67:0/2352518359 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f374c017020 con 0x7f3758101510 2024-03-28T16:06:15.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.686+0000 7f3755600640 1 -- 172.21.15.67:0/2352518359 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f374c009c30 con 0x7f3758101510 2024-03-28T16:06:15.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.686+0000 7f375e200640 1 -- 172.21.15.67:0/2352518359 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37581a17c0 con 0x7f3758101510 2024-03-28T16:06:15.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.686+0000 7f3755600640 1 -- 172.21.15.67:0/2352518359 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f374c00f6b0 con 0x7f3758101510 2024-03-28T16:06:15.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.686+0000 7f375e200640 1 -- 172.21.15.67:0/2352518359 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37581a1d10 con 0x7f3758101510 2024-03-28T16:06:15.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.687+0000 7f375e200640 1 -- 172.21.15.67:0/2352518359 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37581a1fd0 con 0x7f3758101510 2024-03-28T16:06:15.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.690+0000 7f3755600640 1 -- 172.21.15.67:0/2352518359 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f374c01a020 con 0x7f3758101510 2024-03-28T16:06:15.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.691+0000 7f3755600640 1 --2- 172.21.15.67:0/2352518359 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f37400a2800 0x7f37400a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:15.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.691+0000 7f3755600640 1 -- 172.21.15.67:0/2352518359 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f374c0cea60 con 0x7f3758101510 2024-03-28T16:06:15.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.691+0000 7f3757e00640 1 --2- 172.21.15.67:0/2352518359 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f37400a2800 0x7f37400a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:15.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.692+0000 7f3757e00640 1 --2- 172.21.15.67:0/2352518359 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f37400a2800 0x7f37400a4cc0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f3744005fd0 tx=0x7f374400b510 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:06:15.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.696+0000 7f3755600640 1 -- 172.21.15.67:0/2352518359 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f374c021020 con 0x7f3758101510 2024-03-28T16:06:15.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.988+0000 7f375e200640 1 -- 172.21.15.67:0/2352518359 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f37581a21a0 con 0x7f3758101510 2024-03-28T16:06:15.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.990+0000 7f3755600640 1 -- 172.21.15.67:0/2352518359 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11341 (secure 0 0 0) 0x7f374c009de0 con 0x7f3758101510 2024-03-28T16:06:15.991 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:06:15.992 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2008297659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2008297659},{"type":"v1","addr":"172.21.15.67:6835","nonce":2008297659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14532,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/320295745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":320295745},{"type":"v1","addr":"172.21.15.67:6837","nonce":320295745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6839/691821273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":691821273},{"type":"v1","addr":"172.21.15.67:6839","nonce":691821273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6841/3211254875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6840","nonce":3211254875},{"type":"v1","addr":"172.21.15.67:6841","nonce":3211254875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2129665423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2129665423},{"type":"v1","addr":"172.21.15.183:6833","nonce":2129665423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24496,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/1589784425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":1589784425},{"type":"v1","addr":"172.21.15.183:6835","nonce":1589784425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24508,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/2185863835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":2185863835},{"type":"v1","addr":"172.21.15.183:6837","nonce":2185863835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24520,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6839/1242814398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":1242814398},{"type":"v1","addr":"172.21.15.183:6839","nonce":1242814398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":34409,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1008335521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1008335521},{"type":"v1","addr":"172.21.15.150:6833","nonce":1008335521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":34445,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3269935695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3269935695},{"type":"v1","addr":"172.21.15.150:6835","nonce":3269935695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T16:01:07.696824+0000","modified":"2024-03-28T16:01:11.997773+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":34508},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34475":{"gid":34475,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/1112886277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":1112886277},{"type":"v1","addr":"172.21.15.150:6837","nonce":1112886277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_34508":{"gid":34508,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6839/1211011614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":1211011614},{"type":"v1","addr":"172.21.15.150:6839","nonce":1211011614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34508,"qdb_cluster":[34508]},"id":1}]} 2024-03-28T16:06:15.992 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 19 2024-03-28T16:06:15.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.993+0000 7f375e200640 1 -- 172.21.15.67:0/2352518359 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f37400a2800 msgr2=0x7f37400a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:15.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.993+0000 7f375e200640 1 --2- 172.21.15.67:0/2352518359 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f37400a2800 0x7f37400a4cc0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f3744005fd0 tx=0x7f374400b510 comp rx=0 tx=0).stop 2024-03-28T16:06:15.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.993+0000 7f375e200640 1 -- 172.21.15.67:0/2352518359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3758101510 msgr2=0x7f37581998b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:15.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.993+0000 7f375e200640 1 --2- 172.21.15.67:0/2352518359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3758101510 0x7f37581998b0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f374c00d950 tx=0x7f374c00de10 comp rx=0 tx=0).stop 2024-03-28T16:06:15.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.993+0000 7f375e200640 1 -- 172.21.15.67:0/2352518359 shutdown_connections 2024-03-28T16:06:15.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.993+0000 7f375e200640 1 --2- 172.21.15.67:0/2352518359 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f37400a2800 0x7f37400a4cc0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:15.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.993+0000 7f375e200640 1 --2- 172.21.15.67:0/2352518359 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f37581030c0 0x7f37581a0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:15.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.993+0000 7f375e200640 1 --2- 172.21.15.67:0/2352518359 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3758102700 0x7f3758199df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:15.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.993+0000 7f375e200640 1 --2- 172.21.15.67:0/2352518359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3758101510 0x7f37581998b0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:15.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.993+0000 7f375e200640 1 -- 172.21.15.67:0/2352518359 >> 172.21.15.67:0/2352518359 conn(0x7f37580fca80 msgr2=0x7f37580738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:06:15.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.993+0000 7f375e200640 1 -- 172.21.15.67:0/2352518359 shutdown_connections 2024-03-28T16:06:15.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:15.994+0000 7f375e200640 1 -- 172.21.15.67:0/2352518359 wait complete. 2024-03-28T16:06:16.013 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T16:06:16.013 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T16:06:16.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.234+0000 7f5e9fe00640 1 -- 172.21.15.67:0/3051288025 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ea01037c0 msgr2=0x7f5ea0103be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:16.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.234+0000 7f5e9fe00640 1 --2- 172.21.15.67:0/3051288025 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ea01037c0 0x7f5ea0103be0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f5e90009ca0 tx=0x7f5e9002c0f0 comp rx=0 tx=0).stop 2024-03-28T16:06:16.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.234+0000 7f5e9fe00640 1 -- 172.21.15.67:0/3051288025 shutdown_connections 2024-03-28T16:06:16.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.234+0000 7f5e9fe00640 1 --2- 172.21.15.67:0/3051288025 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ea0105370 0x7f5ea010bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:16.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.234+0000 7f5e9fe00640 1 --2- 172.21.15.67:0/3051288025 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5ea01049b0 0x7f5ea0104e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:16.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.234+0000 7f5e9fe00640 1 --2- 172.21.15.67:0/3051288025 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ea01037c0 0x7f5ea0103be0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:16.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.234+0000 7f5e9fe00640 1 -- 172.21.15.67:0/3051288025 >> 172.21.15.67:0/3051288025 conn(0x7f5ea00fed50 msgr2=0x7f5ea0101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:06:16.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.234+0000 7f5e9fe00640 1 -- 172.21.15.67:0/3051288025 shutdown_connections 2024-03-28T16:06:16.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.234+0000 7f5e9fe00640 1 -- 172.21.15.67:0/3051288025 wait complete. 2024-03-28T16:06:16.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.235+0000 7f5e9fe00640 1 Processor -- start 2024-03-28T16:06:16.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.236+0000 7f5e9fe00640 1 -- start start 2024-03-28T16:06:16.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.236+0000 7f5e9fe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5ea01037c0 0x7f5ea01999d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:16.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.236+0000 7f5e9fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ea01049b0 0x7f5ea0199f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:16.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.236+0000 7f5e9fe00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ea0105370 0x7f5ea01a0f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:16.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.236+0000 7f5e9fe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea010f2c0 con 0x7f5ea01049b0 2024-03-28T16:06:16.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.236+0000 7f5e9fe00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea010f140 con 0x7f5ea0105370 2024-03-28T16:06:16.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.236+0000 7f5e9fe00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea010f440 con 0x7f5ea01037c0 2024-03-28T16:06:16.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.236+0000 7f5e9e000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ea01049b0 0x7f5ea0199f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:16.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.236+0000 7f5e9e000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ea01049b0 0x7f5ea0199f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48830/0 (socket says 172.21.15.67:48830) 2024-03-28T16:06:16.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.236+0000 7f5e9e000640 1 -- 172.21.15.67:0/4092871419 learned_addr learned my addr 172.21.15.67:0/4092871419 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:06:16.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.237+0000 7f5e9ea00640 1 --2- 172.21.15.67:0/4092871419 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5ea01037c0 0x7f5ea01999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:16.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.237+0000 7f5e9f400640 1 --2- 172.21.15.67:0/4092871419 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ea0105370 0x7f5ea01a0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:16.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.237+0000 7f5e9e000640 1 -- 172.21.15.67:0/4092871419 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5ea01037c0 msgr2=0x7f5ea01999d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:16.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.237+0000 7f5e9e000640 1 --2- 172.21.15.67:0/4092871419 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5ea01037c0 0x7f5ea01999d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:16.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.237+0000 7f5e9e000640 1 -- 172.21.15.67:0/4092871419 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ea0105370 msgr2=0x7f5ea01a0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:16.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.237+0000 7f5e9e000640 1 --2- 172.21.15.67:0/4092871419 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ea0105370 0x7f5ea01a0f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:16.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.237+0000 7f5e9e000640 1 -- 172.21.15.67:0/4092871419 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ea01a15f0 con 0x7f5ea01049b0 2024-03-28T16:06:16.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.237+0000 7f5e9f400640 1 --2- 172.21.15.67:0/4092871419 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ea0105370 0x7f5ea01a0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:06:16.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.237+0000 7f5e9e000640 1 --2- 172.21.15.67:0/4092871419 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ea01049b0 0x7f5ea0199f10 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f5e8800c890 tx=0x7f5e8800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:06:16.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.238+0000 7f5e97400640 1 -- 172.21.15.67:0/4092871419 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e88004220 con 0x7f5ea01049b0 2024-03-28T16:06:16.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.238+0000 7f5e97400640 1 -- 172.21.15.67:0/4092871419 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5e880043c0 con 0x7f5ea01049b0 2024-03-28T16:06:16.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.238+0000 7f5e9fe00640 1 -- 172.21.15.67:0/4092871419 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ea01a18e0 con 0x7f5ea01049b0 2024-03-28T16:06:16.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.238+0000 7f5e97400640 1 -- 172.21.15.67:0/4092871419 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e88004a00 con 0x7f5ea01049b0 2024-03-28T16:06:16.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.238+0000 7f5e9fe00640 1 -- 172.21.15.67:0/4092871419 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ea01a1e30 con 0x7f5ea01049b0 2024-03-28T16:06:16.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.241+0000 7f5ea87a5740 1 Processor -- start 2024-03-28T16:06:16.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.241+0000 7f5ea87a5740 1 -- start start 2024-03-28T16:06:16.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.242+0000 7f5e97400640 1 -- 172.21.15.67:0/4092871419 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f5e88004ba0 con 0x7f5ea01049b0 2024-03-28T16:06:16.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.242+0000 7f5ea87a5740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556c2d10d9f0 0x556c2d10de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:16.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.242+0000 7f5ea87a5740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556c2d10ebc0 0x556c2d10f040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:16.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.242+0000 7f5ea87a5740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556c2d10f580 0x556c2d115e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:16.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.242+0000 7f5ea87a5740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x556c2d0722c0 con 0x556c2d10f580 2024-03-28T16:06:16.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.242+0000 7f5ea87a5740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x556c2d01ac90 con 0x556c2d10ebc0 2024-03-28T16:06:16.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.242+0000 7f5ea87a5740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x556c2d06e360 con 0x556c2d10d9f0 2024-03-28T16:06:16.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.242+0000 7f5e9e000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556c2d10f580 0x556c2d115e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:16.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.242+0000 7f5e9e000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556c2d10f580 0x556c2d115e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48844/0 (socket says 172.21.15.67:48844) 2024-03-28T16:06:16.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.242+0000 7f5e9e000640 1 -- 172.21.15.67:0/1359038593 learned_addr learned my addr 172.21.15.67:0/1359038593 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:06:16.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.243+0000 7f5e9ea00640 1 --2- 172.21.15.67:0/1359038593 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556c2d10ebc0 0x556c2d10f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:16.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.243+0000 7f5e9f400640 1 --2- 172.21.15.67:0/1359038593 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556c2d10d9f0 0x556c2d10de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:16.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.243+0000 7f5e9e000640 1 -- 172.21.15.67:0/1359038593 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556c2d10d9f0 msgr2=0x556c2d10de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:06:16.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.243+0000 7f5e9e000640 1 --2- 172.21.15.67:0/1359038593 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556c2d10d9f0 0x556c2d10de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:16.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.243+0000 7f5e9e000640 1 -- 172.21.15.67:0/1359038593 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556c2d10ebc0 msgr2=0x556c2d10f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:06:16.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.243+0000 7f5e9e000640 1 --2- 172.21.15.67:0/1359038593 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556c2d10ebc0 0x556c2d10f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:16.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.243+0000 7f5e9e000640 1 -- 172.21.15.67:0/1359038593 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556c2d03f080 con 0x556c2d10f580 2024-03-28T16:06:16.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.243+0000 7f5e9ea00640 1 --2- 172.21.15.67:0/1359038593 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556c2d10ebc0 0x556c2d10f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T16:06:16.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.243+0000 7f5e97400640 1 --2- 172.21.15.67:0/4092871419 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f5e840a2800 0x7f5e840a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:16.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.244+0000 7f5e97400640 1 -- 172.21.15.67:0/4092871419 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f5e880ce830 con 0x7f5ea01049b0 2024-03-28T16:06:16.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.244+0000 7f5e9f400640 1 --2- 172.21.15.67:0/4092871419 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f5e840a2800 0x7f5e840a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:16.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.244+0000 7f5e9e000640 1 --2- 172.21.15.67:0/1359038593 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556c2d10f580 0x556c2d115e10 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f5e880d4c20 tx=0x7f5e880d6670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d34c1aad05a9a63c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:06:16.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.244+0000 7f5e8fe00640 1 -- 172.21.15.67:0/1359038593 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e8801d070 con 0x556c2d10f580 2024-03-28T16:06:16.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.244+0000 7f5e8fe00640 1 -- 172.21.15.67:0/1359038593 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5e880bb110 con 0x556c2d10f580 2024-03-28T16:06:16.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.244+0000 7f5e8fe00640 1 -- 172.21.15.67:0/1359038593 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e8809af50 con 0x556c2d10f580 2024-03-28T16:06:16.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.245+0000 7f5ea87a5740 1 -- 172.21.15.67:0/1359038593 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556c2d10f580 msgr2=0x556c2d115e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:16.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.245+0000 7f5ea87a5740 1 --2- 172.21.15.67:0/1359038593 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556c2d10f580 0x556c2d115e10 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f5e880d4c20 tx=0x7f5e880d6670 comp rx=0 tx=0).stop 2024-03-28T16:06:16.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.245+0000 7f5e9f400640 1 --2- 172.21.15.67:0/4092871419 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f5e840a2800 0x7f5e840a4cc0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f5e98005fd0 tx=0x7f5e98009420 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:06:16.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.245+0000 7f5ea87a5740 1 -- 172.21.15.67:0/1359038593 shutdown_connections 2024-03-28T16:06:16.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.245+0000 7f5ea87a5740 1 --2- 172.21.15.67:0/1359038593 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556c2d10f580 0x556c2d115e10 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:16.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.245+0000 7f5ea87a5740 1 --2- 172.21.15.67:0/1359038593 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556c2d10ebc0 0x556c2d10f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:16.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.245+0000 7f5ea87a5740 1 --2- 172.21.15.67:0/1359038593 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556c2d10d9f0 0x556c2d10de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:16.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.245+0000 7f5ea87a5740 1 -- 172.21.15.67:0/1359038593 >> 172.21.15.67:0/1359038593 conn(0x556c2d108de0 msgr2=0x556c2d1091e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:06:16.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.245+0000 7f5ea87a5740 1 -- 172.21.15.67:0/1359038593 shutdown_connections 2024-03-28T16:06:16.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.246+0000 7f5ea87a5740 1 -- 172.21.15.67:0/1359038593 wait complete. 2024-03-28T16:06:16.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.246+0000 7f5ea87a5740 1 -- start start 2024-03-28T16:06:16.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.246+0000 7f5ea87a5740 1 Processor -- start 2024-03-28T16:06:16.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.247+0000 7f5ea87a5740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556c2d10d9f0 0x556c2d117c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:16.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.247+0000 7f5ea87a5740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556c2d10ebc0 0x556c2d1181a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:16.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.247+0000 7f5ea87a5740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556c2d10f580 0x556c2d11f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:16.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.247+0000 7f5ea87a5740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x556c2d116590 con 0x556c2d10d9f0 2024-03-28T16:06:16.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.247+0000 7f5ea87a5740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x556c2d02e0d0 con 0x556c2d10f580 2024-03-28T16:06:16.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.247+0000 7f5ea87a5740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x556c2d116710 con 0x556c2d10ebc0 2024-03-28T16:06:16.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.247+0000 7f5ea87a5740 10 client.0 _ll_register_callbacks cb 0x556c2d032e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:06:16.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.247+0000 7f5e9ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556c2d10d9f0 0x556c2d117c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:16.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.247+0000 7f5e9ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556c2d10d9f0 0x556c2d117c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48858/0 (socket says 172.21.15.67:48858) 2024-03-28T16:06:16.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.247+0000 7f5e9ea00640 1 -- 172.21.15.67:0/3591523951 learned_addr learned my addr 172.21.15.67:0/3591523951 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:06:16.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.247+0000 7f5e9f400640 1 --2- 172.21.15.67:0/3591523951 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556c2d10f580 0x556c2d11f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:16.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.248+0000 7f5e9e000640 1 --2- 172.21.15.67:0/3591523951 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556c2d10ebc0 0x556c2d1181a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:16.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.248+0000 7f5e9ea00640 1 -- 172.21.15.67:0/3591523951 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556c2d10ebc0 msgr2=0x556c2d1181a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:06:16.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.248+0000 7f5e9ea00640 1 --2- 172.21.15.67:0/3591523951 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556c2d10ebc0 0x556c2d1181a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:16.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.248+0000 7f5e9ea00640 1 -- 172.21.15.67:0/3591523951 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556c2d10f580 msgr2=0x556c2d11f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:06:16.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.248+0000 7f5e9ea00640 1 --2- 172.21.15.67:0/3591523951 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556c2d10f580 0x556c2d11f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:16.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.248+0000 7f5e9ea00640 1 -- 172.21.15.67:0/3591523951 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556c2cd863f0 con 0x556c2d10d9f0 2024-03-28T16:06:16.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.248+0000 7f5e9ea00640 1 --2- 172.21.15.67:0/3591523951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556c2d10d9f0 0x556c2d117c60 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f5e90000f80 tx=0x7f5e90009820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d9c8be3aad78287 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:06:16.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.248+0000 7f5e8cc00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:06:16.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.248+0000 7f5e8cc00640 1 -- 172.21.15.67:0/3591523951 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e90043070 con 0x556c2d10d9f0 2024-03-28T16:06:16.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.249+0000 7f5e8cc00640 1 -- 172.21.15.67:0/3591523951 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f5e90003970 con 0x556c2d10d9f0 2024-03-28T16:06:16.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.249+0000 7f5ea87a5740 1 -- 172.21.15.67:0/3591523951 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5e880bb110 con 0x556c2d10d9f0 2024-03-28T16:06:16.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.249+0000 7f5e8cc00640 1 -- 172.21.15.67:0/3591523951 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5e9004f050 con 0x556c2d10d9f0 2024-03-28T16:06:16.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.249+0000 7f5ea87a5740 10 client.15675 fetch_fsmap learned FSMap version 19 2024-03-28T16:06:16.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.249+0000 7f5ea87a5740 1 -- 172.21.15.67:0/3591523951 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x556c2ce507d0 con 0x556c2d10d9f0 2024-03-28T16:06:16.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.249+0000 7f5e8cc00640 1 -- 172.21.15.67:0/3591523951 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5e880bb110 con 0x556c2d10d9f0 2024-03-28T16:06:16.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.249+0000 7f5e8cc00640 1 -- 172.21.15.67:0/3591523951 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f5e90034440 con 0x556c2d10d9f0 2024-03-28T16:06:16.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.250+0000 7f5ea87a5740 10 client.15675 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:06:16.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.250+0000 7f5ea87a5740 10 client.15675 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:06:16.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.250+0000 7f5ea87a5740 20 client.15675 populate_metadata read hostname 'smithi067' 2024-03-28T16:06:16.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.250+0000 7f5ea87a5740 1 --2- 172.21.15.67:0/3591523951 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x556c2d123680 0x556c2d125af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:16.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.250+0000 7f5ea87a5740 4 client.15675 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:06:16.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.250+0000 7f5ea87a5740 1 -- 172.21.15.67:0/3591523951 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556c2d07f800 con 0x556c2d123680 2024-03-28T16:06:16.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.250+0000 7f5e9e000640 1 --2- 172.21.15.67:0/3591523951 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x556c2d123680 0x556c2d125af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:16.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.251+0000 7f5e9e000640 1 --2- 172.21.15.67:0/3591523951 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x556c2d123680 0x556c2d125af0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=485f4ca0afc398ba server_cookie=85885d90614e1023 in_seq=0 out_seq=0 2024-03-28T16:06:16.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.251+0000 7f5e8cc00640 10 client.15675 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:06:16.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.253+0000 7f5e8cc00640 1 -- 172.21.15.67:0/3591523951 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x556c2d07f800 con 0x556c2d123680 2024-03-28T16:06:16.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.253+0000 7f5e8cc00640 10 client.15675 handle_command_reply: tid=0 2024-03-28T16:06:16.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.253+0000 7f5ea87a5740 1 client.15675 shutdown 2024-03-28T16:06:16.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.253+0000 7f5ea87a5740 1 -- 172.21.15.67:0/3591523951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556c2d10d9f0 msgr2=0x556c2d117c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:16.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.253+0000 7f5ea87a5740 1 --2- 172.21.15.67:0/3591523951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556c2d10d9f0 0x556c2d117c60 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f5e90000f80 tx=0x7f5e90009820 comp rx=0 tx=0).stop 2024-03-28T16:06:16.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.253+0000 7f5ea87a5740 1 -- 172.21.15.67:0/3591523951 shutdown_connections 2024-03-28T16:06:16.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.253+0000 7f5ea87a5740 1 --2- 172.21.15.67:0/3591523951 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x556c2d123680 0x556c2d125af0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:16.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.254+0000 7f5ea87a5740 1 --2- 172.21.15.67:0/3591523951 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556c2d10f580 0x556c2d11f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:16.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.254+0000 7f5ea87a5740 1 --2- 172.21.15.67:0/3591523951 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556c2d10ebc0 0x556c2d1181a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:16.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.254+0000 7f5ea87a5740 1 --2- 172.21.15.67:0/3591523951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556c2d10d9f0 0x556c2d117c60 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:16.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.254+0000 7f5ea87a5740 1 -- 172.21.15.67:0/3591523951 >> 172.21.15.67:0/3591523951 conn(0x556c2d108de0 msgr2=0x556c2d108160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:06:16.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.254+0000 7f5e8cc00640 0 client.15675 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:06:16.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.254+0000 7f5ea87a5740 1 -- 172.21.15.67:0/3591523951 shutdown_connections 2024-03-28T16:06:16.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.254+0000 7f5ea87a5740 1 -- 172.21.15.67:0/3591523951 wait complete. 2024-03-28T16:06:16.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.254+0000 7f5ea87a5740 20 client.15675 trim_cache size 0 max 16384 2024-03-28T16:06:16.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.271+0000 7f5ea87a5740 1 Processor -- start 2024-03-28T16:06:16.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.271+0000 7f5ea87a5740 1 -- start start 2024-03-28T16:06:16.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.271+0000 7f5ea87a5740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556c2d10d9f0 0x556c2d10c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:16.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.271+0000 7f5ea87a5740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556c2d10ebc0 0x556c2d12a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:16.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.271+0000 7f5ea87a5740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556c2d10f580 0x556c2d12a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:16.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.271+0000 7f5ea87a5740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea01a8ab0 con 0x556c2d10d9f0 2024-03-28T16:06:16.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.271+0000 7f5ea87a5740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea01ab070 con 0x556c2d10f580 2024-03-28T16:06:16.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.271+0000 7f5ea87a5740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea01a90b0 con 0x556c2d10ebc0 2024-03-28T16:06:16.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.272+0000 7f5e9ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556c2d10d9f0 0x556c2d10c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:16.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.272+0000 7f5e9ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556c2d10d9f0 0x556c2d10c040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48866/0 (socket says 172.21.15.67:48866) 2024-03-28T16:06:16.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.272+0000 7f5e9ea00640 1 -- 172.21.15.67:0/3983314142 learned_addr learned my addr 172.21.15.67:0/3983314142 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:06:16.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.272+0000 7f5e9e000640 1 --2- 172.21.15.67:0/3983314142 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556c2d10ebc0 0x556c2d12a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:16.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.272+0000 7f5e9f400640 1 --2- 172.21.15.67:0/3983314142 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556c2d10f580 0x556c2d12a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:16.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.272+0000 7f5e9ea00640 1 -- 172.21.15.67:0/3983314142 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556c2d10ebc0 msgr2=0x556c2d12a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:06:16.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.272+0000 7f5e9ea00640 1 --2- 172.21.15.67:0/3983314142 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556c2d10ebc0 0x556c2d12a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:16.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.272+0000 7f5e9ea00640 1 -- 172.21.15.67:0/3983314142 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556c2d10f580 msgr2=0x556c2d12a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:06:16.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.272+0000 7f5e9ea00640 1 --2- 172.21.15.67:0/3983314142 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556c2d10f580 0x556c2d12a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:16.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.272+0000 7f5e9ea00640 1 -- 172.21.15.67:0/3983314142 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556c2d10d5f0 con 0x556c2d10d9f0 2024-03-28T16:06:16.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.272+0000 7f5e9f400640 1 --2- 172.21.15.67:0/3983314142 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556c2d10f580 0x556c2d12a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T16:06:16.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.272+0000 7f5e9e000640 1 --2- 172.21.15.67:0/3983314142 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556c2d10ebc0 0x556c2d12a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T16:06:16.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.273+0000 7f5e9ea00640 1 --2- 172.21.15.67:0/3983314142 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556c2d10d9f0 0x556c2d10c040 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f5e90009bf0 tx=0x7f5e9002c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b1d29f582e78166 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:06:16.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.273+0000 7f5e8f400640 1 -- 172.21.15.67:0/3983314142 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e9004c020 con 0x556c2d10d9f0 2024-03-28T16:06:16.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.273+0000 7f5e8f400640 1 -- 172.21.15.67:0/3983314142 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5e90003e20 con 0x556c2d10d9f0 2024-03-28T16:06:16.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.273+0000 7f5e8f400640 1 -- 172.21.15.67:0/3983314142 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e90043070 con 0x556c2d10d9f0 2024-03-28T16:06:16.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.274+0000 7f5ea87a5740 1 -- 172.21.15.67:0/3983314142 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556c2d10d9f0 msgr2=0x556c2d10c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:16.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.274+0000 7f5ea87a5740 1 --2- 172.21.15.67:0/3983314142 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556c2d10d9f0 0x556c2d10c040 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f5e90009bf0 tx=0x7f5e9002c0f0 comp rx=0 tx=0).stop 2024-03-28T16:06:16.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.274+0000 7f5ea87a5740 1 -- 172.21.15.67:0/3983314142 shutdown_connections 2024-03-28T16:06:16.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.274+0000 7f5ea87a5740 1 --2- 172.21.15.67:0/3983314142 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556c2d10f580 0x556c2d12a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:16.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.274+0000 7f5ea87a5740 1 --2- 172.21.15.67:0/3983314142 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556c2d10ebc0 0x556c2d12a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:16.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.274+0000 7f5ea87a5740 1 --2- 172.21.15.67:0/3983314142 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556c2d10d9f0 0x556c2d10c040 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:16.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.274+0000 7f5ea87a5740 1 -- 172.21.15.67:0/3983314142 >> 172.21.15.67:0/3983314142 conn(0x556c2d108de0 msgr2=0x556c2d108160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:06:16.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.274+0000 7f5ea87a5740 1 -- 172.21.15.67:0/3983314142 shutdown_connections 2024-03-28T16:06:16.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.274+0000 7f5ea87a5740 1 -- 172.21.15.67:0/3983314142 wait complete. 2024-03-28T16:06:16.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.274+0000 7f5ea87a5740 1 -- start start 2024-03-28T16:06:16.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.274+0000 7f5ea87a5740 1 Processor -- start 2024-03-28T16:06:16.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.275+0000 7f5ea87a5740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556c2d10d9f0 0x556c2d11c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:16.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.275+0000 7f5ea87a5740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556c2d10ebc0 0x556c2d11cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:16.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.275+0000 7f5ea87a5740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556c2d10f580 0x556c2d11dde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:16.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.275+0000 7f5ea87a5740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea01a96f0 con 0x556c2d10f580 2024-03-28T16:06:16.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.275+0000 7f5ea87a5740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea01aaa10 con 0x556c2d10d9f0 2024-03-28T16:06:16.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.275+0000 7f5ea87a5740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea01a9d50 con 0x556c2d10ebc0 2024-03-28T16:06:16.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.275+0000 7f5ea87a5740 10 client.0 _ll_register_callbacks cb 0x7f5e90009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:06:16.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.275+0000 7f5e9f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556c2d10f580 0x556c2d11dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:16.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.275+0000 7f5e9f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556c2d10f580 0x556c2d11dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48880/0 (socket says 172.21.15.67:48880) 2024-03-28T16:06:16.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.275+0000 7f5e9f400640 1 -- 172.21.15.67:0/1246092810 learned_addr learned my addr 172.21.15.67:0/1246092810 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:06:16.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.275+0000 7f5e9ea00640 1 --2- 172.21.15.67:0/1246092810 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556c2d10d9f0 0x556c2d11c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:16.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.276+0000 7f5e9e000640 1 --2- 172.21.15.67:0/1246092810 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556c2d10ebc0 0x556c2d11cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:16.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.276+0000 7f5e9f400640 1 -- 172.21.15.67:0/1246092810 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556c2d10ebc0 msgr2=0x556c2d11cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:06:16.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.276+0000 7f5e9f400640 1 --2- 172.21.15.67:0/1246092810 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556c2d10ebc0 0x556c2d11cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:16.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.276+0000 7f5e9f400640 1 -- 172.21.15.67:0/1246092810 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556c2d10d9f0 msgr2=0x556c2d11c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:06:16.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.276+0000 7f5e9f400640 1 --2- 172.21.15.67:0/1246092810 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556c2d10d9f0 0x556c2d11c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:16.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.276+0000 7f5e9f400640 1 -- 172.21.15.67:0/1246092810 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556c2d11e3d0 con 0x556c2d10f580 2024-03-28T16:06:16.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.276+0000 7f5e9f400640 1 --2- 172.21.15.67:0/1246092810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556c2d10f580 0x556c2d11dde0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f5e98009790 tx=0x7f5e980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43a5a6f052505d5c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:06:16.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.276+0000 7f5e8cc00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:06:16.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.277+0000 7f5e8cc00640 1 -- 172.21.15.67:0/1246092810 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e98021990 con 0x556c2d10f580 2024-03-28T16:06:16.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.277+0000 7f5ea87a5740 1 -- 172.21.15.67:0/1246092810 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5e90003e20 con 0x556c2d10f580 2024-03-28T16:06:16.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.277+0000 7f5e8cc00640 1 -- 172.21.15.67:0/1246092810 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f5e98021b30 con 0x556c2d10f580 2024-03-28T16:06:16.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.277+0000 7f5e8cc00640 1 -- 172.21.15.67:0/1246092810 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5e98021d30 con 0x556c2d10f580 2024-03-28T16:06:16.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.277+0000 7f5ea87a5740 10 client.15681 fetch_fsmap learned FSMap version 19 2024-03-28T16:06:16.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.277+0000 7f5ea87a5740 1 -- 172.21.15.67:0/1246092810 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x556c2d17f830 con 0x556c2d10f580 2024-03-28T16:06:16.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.277+0000 7f5e8cc00640 1 -- 172.21.15.67:0/1246092810 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5e90003e20 con 0x556c2d10f580 2024-03-28T16:06:16.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.278+0000 7f5e8cc00640 1 -- 172.21.15.67:0/1246092810 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f5e98028070 con 0x556c2d10f580 2024-03-28T16:06:16.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.278+0000 7f5ea87a5740 10 client.15681 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:06:16.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.278+0000 7f5ea87a5740 10 client.15681 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:06:16.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.278+0000 7f5ea87a5740 20 client.15681 populate_metadata read hostname 'smithi067' 2024-03-28T16:06:16.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.278+0000 7f5ea87a5740 1 --2- 172.21.15.67:0/1246092810 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x556c2d180240 0x556c2d182660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:16.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.278+0000 7f5ea87a5740 4 client.15681 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T16:06:16.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.278+0000 7f5ea87a5740 1 -- 172.21.15.67:0/1246092810 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x556c2d079b50 con 0x556c2d180240 2024-03-28T16:06:16.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.278+0000 7f5e9ea00640 1 --2- 172.21.15.67:0/1246092810 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x556c2d180240 0x556c2d182660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:16.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.279+0000 7f5e9ea00640 1 --2- 172.21.15.67:0/1246092810 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x556c2d180240 0x556c2d182660 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ee0b2c0806c6989f server_cookie=a8cccf7919619e50 in_seq=0 out_seq=0 2024-03-28T16:06:16.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.279+0000 7f5e8cc00640 10 client.15681 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:06:16.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.280+0000 7f5e8cc00640 1 -- 172.21.15.67:0/1246092810 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x556c2d079b50 con 0x556c2d180240 2024-03-28T16:06:16.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.280+0000 7f5e8cc00640 10 client.15681 handle_command_reply: tid=0 2024-03-28T16:06:16.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.280+0000 7f5ea87a5740 1 client.15681 shutdown 2024-03-28T16:06:16.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.280+0000 7f5ea87a5740 1 -- 172.21.15.67:0/1246092810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556c2d10f580 msgr2=0x556c2d11dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:16.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.280+0000 7f5ea87a5740 1 --2- 172.21.15.67:0/1246092810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556c2d10f580 0x556c2d11dde0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f5e98009790 tx=0x7f5e980046c0 comp rx=0 tx=0).stop 2024-03-28T16:06:16.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.280+0000 7f5ea87a5740 1 -- 172.21.15.67:0/1246092810 shutdown_connections 2024-03-28T16:06:16.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.280+0000 7f5ea87a5740 1 --2- 172.21.15.67:0/1246092810 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x556c2d180240 0x556c2d182660 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:16.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.280+0000 7f5ea87a5740 1 --2- 172.21.15.67:0/1246092810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556c2d10f580 0x556c2d11dde0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:16.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.280+0000 7f5ea87a5740 1 --2- 172.21.15.67:0/1246092810 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556c2d10ebc0 0x556c2d11cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:16.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.280+0000 7f5e8cc00640 0 client.15681 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:06:16.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.280+0000 7f5ea87a5740 1 --2- 172.21.15.67:0/1246092810 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556c2d10d9f0 0x556c2d11c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:16.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.280+0000 7f5ea87a5740 1 -- 172.21.15.67:0/1246092810 >> 172.21.15.67:0/1246092810 conn(0x556c2d108de0 msgr2=0x556c2d111df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:06:16.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.280+0000 7f5ea87a5740 1 -- 172.21.15.67:0/1246092810 shutdown_connections 2024-03-28T16:06:16.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.280+0000 7f5ea87a5740 1 -- 172.21.15.67:0/1246092810 wait complete. 2024-03-28T16:06:16.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.281+0000 7f5ea87a5740 20 client.15681 trim_cache size 0 max 16384 2024-03-28T16:06:16.284 INFO:teuthology.orchestra.run.smithi067.stdout:[] 2024-03-28T16:06:16.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.281+0000 7f5e9fe00640 1 -- 172.21.15.67:0/4092871419 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f5e840a2800 msgr2=0x7f5e840a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:16.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.281+0000 7f5e9fe00640 1 --2- 172.21.15.67:0/4092871419 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f5e840a2800 0x7f5e840a4cc0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f5e98005fd0 tx=0x7f5e98009420 comp rx=0 tx=0).stop 2024-03-28T16:06:16.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.281+0000 7f5e9fe00640 1 -- 172.21.15.67:0/4092871419 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ea01049b0 msgr2=0x7f5ea0199f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:16.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.281+0000 7f5e9fe00640 1 --2- 172.21.15.67:0/4092871419 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ea01049b0 0x7f5ea0199f10 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f5e8800c890 tx=0x7f5e8800cd50 comp rx=0 tx=0).stop 2024-03-28T16:06:16.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.281+0000 7f5e9fe00640 1 -- 172.21.15.67:0/4092871419 shutdown_connections 2024-03-28T16:06:16.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.281+0000 7f5e9fe00640 1 --2- 172.21.15.67:0/4092871419 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f5e840a2800 0x7f5e840a4cc0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:16.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.281+0000 7f5e9fe00640 1 --2- 172.21.15.67:0/4092871419 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ea0105370 0x7f5ea01a0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:16.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.281+0000 7f5e9fe00640 1 --2- 172.21.15.67:0/4092871419 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ea01049b0 0x7f5ea0199f10 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:16.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.281+0000 7f5e9fe00640 1 --2- 172.21.15.67:0/4092871419 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5ea01037c0 0x7f5ea01999d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:16.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.281+0000 7f5e9fe00640 1 -- 172.21.15.67:0/4092871419 >> 172.21.15.67:0/4092871419 conn(0x7f5ea00fed50 msgr2=0x7f5ea0107da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:06:16.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.282+0000 7f5e9fe00640 1 -- 172.21.15.67:0/4092871419 shutdown_connections 2024-03-28T16:06:16.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:16.282+0000 7f5e9fe00640 1 -- 172.21.15.67:0/4092871419 wait complete. 2024-03-28T16:06:16.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:06:16 smithi183 ceph-mon[35580]: pgmap v520: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 22 MiB/s wr, 7 op/s 2024-03-28T16:06:16.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:06:16 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/2352518359' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:06:17.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:06:16 smithi150 ceph-mon[34825]: pgmap v520: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 22 MiB/s wr, 7 op/s 2024-03-28T16:06:17.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:06:16 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/2352518359' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:06:17.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:06:16 smithi067 ceph-mon[31707]: pgmap v520: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 22 MiB/s wr, 7 op/s 2024-03-28T16:06:17.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:06:16 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/2352518359' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:06:17.302 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T16:06:17.302 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T16:06:17.302 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T16:06:17.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.556+0000 7f7077000640 1 -- 172.21.15.67:0/3157003471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7070103150 msgr2=0x7f70701099e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:17.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.556+0000 7f7077000640 1 --2- 172.21.15.67:0/3157003471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7070103150 0x7f70701099e0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f7068009ca0 tx=0x7f706802c0f0 comp rx=0 tx=0).stop 2024-03-28T16:06:17.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.556+0000 7f7077000640 1 -- 172.21.15.67:0/3157003471 shutdown_connections 2024-03-28T16:06:17.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.556+0000 7f7077000640 1 --2- 172.21.15.67:0/3157003471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7070103150 0x7f70701099e0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:17.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.556+0000 7f7077000640 1 --2- 172.21.15.67:0/3157003471 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7070102790 0x7f7070102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:17.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.556+0000 7f7077000640 1 --2- 172.21.15.67:0/3157003471 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f70701015a0 0x7f70701019c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:17.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.556+0000 7f7077000640 1 -- 172.21.15.67:0/3157003471 >> 172.21.15.67:0/3157003471 conn(0x7f70700fcb10 msgr2=0x7f70700fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:06:17.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.556+0000 7f7077000640 1 -- 172.21.15.67:0/3157003471 shutdown_connections 2024-03-28T16:06:17.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.556+0000 7f7077000640 1 -- 172.21.15.67:0/3157003471 wait complete. 2024-03-28T16:06:17.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.557+0000 7f7077000640 1 Processor -- start 2024-03-28T16:06:17.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.558+0000 7f7077000640 1 -- start start 2024-03-28T16:06:17.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.558+0000 7f7077000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f70701015a0 0x7f7070199980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:17.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.558+0000 7f7077000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7070102790 0x7f7070199ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:17.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.558+0000 7f7077000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7070103150 0x7f70701a0f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:17.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.558+0000 7f7077000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f707010d0e0 con 0x7f7070103150 2024-03-28T16:06:17.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.558+0000 7f7077000640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f707010cf60 con 0x7f7070102790 2024-03-28T16:06:17.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.558+0000 7f7077000640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f707010d260 con 0x7f70701015a0 2024-03-28T16:06:17.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.558+0000 7f7076600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7070103150 0x7f70701a0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:17.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.559+0000 7f7076600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7070103150 0x7f70701a0f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48892/0 (socket says 172.21.15.67:48892) 2024-03-28T16:06:17.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.559+0000 7f7076600640 1 -- 172.21.15.67:0/782935171 learned_addr learned my addr 172.21.15.67:0/782935171 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:06:17.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.559+0000 7f7075c00640 1 --2- 172.21.15.67:0/782935171 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f70701015a0 0x7f7070199980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:17.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.559+0000 7f7075200640 1 --2- 172.21.15.67:0/782935171 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7070102790 0x7f7070199ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:17.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.559+0000 7f7076600640 1 -- 172.21.15.67:0/782935171 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f70701015a0 msgr2=0x7f7070199980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:17.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.559+0000 7f7076600640 1 --2- 172.21.15.67:0/782935171 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f70701015a0 0x7f7070199980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:17.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.559+0000 7f7076600640 1 -- 172.21.15.67:0/782935171 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7070102790 msgr2=0x7f7070199ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:17.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.559+0000 7f7076600640 1 --2- 172.21.15.67:0/782935171 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7070102790 0x7f7070199ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:17.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.559+0000 7f7076600640 1 -- 172.21.15.67:0/782935171 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70701a15a0 con 0x7f7070103150 2024-03-28T16:06:17.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.560+0000 7f7076600640 1 --2- 172.21.15.67:0/782935171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7070103150 0x7f70701a0f30 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f706802c700 tx=0x7f70680328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:06:17.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.560+0000 7f706e000640 1 -- 172.21.15.67:0/782935171 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7068041020 con 0x7f7070103150 2024-03-28T16:06:17.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.560+0000 7f7077000640 1 -- 172.21.15.67:0/782935171 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70701a1830 con 0x7f7070103150 2024-03-28T16:06:17.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.560+0000 7f706e000640 1 -- 172.21.15.67:0/782935171 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7068003e20 con 0x7f7070103150 2024-03-28T16:06:17.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.560+0000 7f706e000640 1 -- 172.21.15.67:0/782935171 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7068039040 con 0x7f7070103150 2024-03-28T16:06:17.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.560+0000 7f7077000640 1 -- 172.21.15.67:0/782935171 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70701a1d20 con 0x7f7070103150 2024-03-28T16:06:17.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.564+0000 7f707b3ba740 1 Processor -- start 2024-03-28T16:06:17.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.566+0000 7f706e000640 1 -- 172.21.15.67:0/782935171 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f7068044020 con 0x7f7070103150 2024-03-28T16:06:17.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.566+0000 7f707b3ba740 1 -- start start 2024-03-28T16:06:17.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.566+0000 7f707b3ba740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c148ead2b0 0x55c148ead6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:17.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.566+0000 7f707b3ba740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c148eadca0 0x55c148eb27a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:17.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.566+0000 7f707b3ba740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c148eb2e10 0x55c148eb52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:17.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.566+0000 7f707b3ba740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c148e12eb0 con 0x55c148eadca0 2024-03-28T16:06:17.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.566+0000 7f707b3ba740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c148dbba60 con 0x55c148ead2b0 2024-03-28T16:06:17.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.566+0000 7f707b3ba740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c148e0ef50 con 0x55c148eb2e10 2024-03-28T16:06:17.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.567+0000 7f7075c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c148eadca0 0x55c148eb27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:17.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.567+0000 7f7075c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c148eadca0 0x55c148eb27a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48894/0 (socket says 172.21.15.67:48894) 2024-03-28T16:06:17.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.567+0000 7f7075c00640 1 -- 172.21.15.67:0/935946862 learned_addr learned my addr 172.21.15.67:0/935946862 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:06:17.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.567+0000 7f7075200640 1 --2- 172.21.15.67:0/935946862 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c148eb2e10 0x55c148eb52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:17.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.567+0000 7f7075c00640 1 -- 172.21.15.67:0/935946862 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c148eb2e10 msgr2=0x55c148eb52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:06:17.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.567+0000 7f7075c00640 1 --2- 172.21.15.67:0/935946862 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c148eb2e10 0x55c148eb52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:17.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.567+0000 7f7075c00640 1 -- 172.21.15.67:0/935946862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c148ead2b0 msgr2=0x55c148ead6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:06:17.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.567+0000 7f7075c00640 1 --2- 172.21.15.67:0/935946862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c148ead2b0 0x55c148ead6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:17.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.567+0000 7f7075c00640 1 -- 172.21.15.67:0/935946862 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c148bef7e0 con 0x55c148eadca0 2024-03-28T16:06:17.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.567+0000 7f7075200640 1 --2- 172.21.15.67:0/935946862 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c148eb2e10 0x55c148eb52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:06:17.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.567+0000 7f706e000640 1 --2- 172.21.15.67:0/782935171 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f70580a2800 0x7f70580a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:17.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.568+0000 7f7075c00640 1 --2- 172.21.15.67:0/935946862 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c148eadca0 0x55c148eb27a0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f706400bcb0 tx=0x7f70640107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4518ef05bb863590 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:06:17.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.568+0000 7f706e000640 1 -- 172.21.15.67:0/782935171 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f70680f0f40 con 0x7f7070103150 2024-03-28T16:06:17.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.568+0000 7f7075200640 1 --2- 172.21.15.67:0/782935171 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f70580a2800 0x7f70580a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:17.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.568+0000 7f7062a00640 1 -- 172.21.15.67:0/935946862 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7064004020 con 0x55c148eadca0 2024-03-28T16:06:17.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.568+0000 7f7062a00640 1 -- 172.21.15.67:0/935946862 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f70640054d0 con 0x55c148eadca0 2024-03-28T16:06:17.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.568+0000 7f7062a00640 1 -- 172.21.15.67:0/935946862 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7064004c10 con 0x55c148eadca0 2024-03-28T16:06:17.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.569+0000 7f707b3ba740 1 -- 172.21.15.67:0/935946862 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c148eadca0 msgr2=0x55c148eb27a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:17.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.569+0000 7f7075200640 1 --2- 172.21.15.67:0/782935171 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f70580a2800 0x7f70580a4cc0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x55c148eae500 tx=0x7f705c00c380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:06:17.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.569+0000 7f707b3ba740 1 --2- 172.21.15.67:0/935946862 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c148eadca0 0x55c148eb27a0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f706400bcb0 tx=0x7f70640107a0 comp rx=0 tx=0).stop 2024-03-28T16:06:17.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.569+0000 7f707b3ba740 1 -- 172.21.15.67:0/935946862 shutdown_connections 2024-03-28T16:06:17.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.569+0000 7f707b3ba740 1 --2- 172.21.15.67:0/935946862 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c148eb2e10 0x55c148eb52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:17.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.569+0000 7f707b3ba740 1 --2- 172.21.15.67:0/935946862 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c148eadca0 0x55c148eb27a0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:17.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.569+0000 7f707b3ba740 1 --2- 172.21.15.67:0/935946862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c148ead2b0 0x55c148ead6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:17.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.569+0000 7f707b3ba740 1 -- 172.21.15.67:0/935946862 >> 172.21.15.67:0/935946862 conn(0x55c148ea8820 msgr2=0x55c148ea8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:06:17.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.569+0000 7f707b3ba740 1 -- 172.21.15.67:0/935946862 shutdown_connections 2024-03-28T16:06:17.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.570+0000 7f707b3ba740 1 -- 172.21.15.67:0/935946862 wait complete. 2024-03-28T16:06:17.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.570+0000 7f707b3ba740 1 -- start start 2024-03-28T16:06:17.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.570+0000 7f707b3ba740 1 Processor -- start 2024-03-28T16:06:17.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.571+0000 7f707b3ba740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c148ead2b0 0x55c148eb7490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:17.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.571+0000 7f707b3ba740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c148eadca0 0x55c148eb79d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:17.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.571+0000 7f707b3ba740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c148eb2e10 0x55c148ebea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:17.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.571+0000 7f707b3ba740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c148eb5dc0 con 0x55c148eb2e10 2024-03-28T16:06:17.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.571+0000 7f707b3ba740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c148dceeb0 con 0x55c148ead2b0 2024-03-28T16:06:17.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.571+0000 7f707b3ba740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c148eb5f40 con 0x55c148eadca0 2024-03-28T16:06:17.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.571+0000 7f707b3ba740 10 client.0 _ll_register_callbacks cb 0x55c148dc3150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:06:17.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.571+0000 7f7076600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c148eb2e10 0x55c148ebea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:17.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.571+0000 7f7076600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c148eb2e10 0x55c148ebea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48908/0 (socket says 172.21.15.67:48908) 2024-03-28T16:06:17.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.571+0000 7f7076600640 1 -- 172.21.15.67:0/87660763 learned_addr learned my addr 172.21.15.67:0/87660763 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:06:17.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.571+0000 7f7075c00640 1 --2- 172.21.15.67:0/87660763 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c148ead2b0 0x55c148eb7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:17.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.572+0000 7f7075200640 1 --2- 172.21.15.67:0/87660763 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c148eadca0 0x55c148eb79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:17.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.572+0000 7f7076600640 1 -- 172.21.15.67:0/87660763 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c148eadca0 msgr2=0x55c148eb79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:06:17.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.572+0000 7f7076600640 1 --2- 172.21.15.67:0/87660763 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c148eadca0 0x55c148eb79d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:17.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.572+0000 7f7076600640 1 -- 172.21.15.67:0/87660763 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c148ead2b0 msgr2=0x55c148eb7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:06:17.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.572+0000 7f7076600640 1 --2- 172.21.15.67:0/87660763 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c148ead2b0 0x55c148eb7490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:17.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.572+0000 7f7076600640 1 -- 172.21.15.67:0/87660763 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c148c4ff70 con 0x55c148eb2e10 2024-03-28T16:06:17.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.572+0000 7f7075200640 1 --2- 172.21.15.67:0/87660763 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c148eadca0 0x55c148eb79d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:06:17.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.572+0000 7f7075c00640 1 --2- 172.21.15.67:0/87660763 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c148ead2b0 0x55c148eb7490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T16:06:17.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.572+0000 7f7076600640 1 --2- 172.21.15.67:0/87660763 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c148eb2e10 0x55c148ebea10 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f706802c6d0 tx=0x7f70680bd450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1eb8069c0c945fe7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:06:17.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.572+0000 7f7057400640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:06:17.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.573+0000 7f7057400640 1 -- 172.21.15.67:0/87660763 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70680bd890 con 0x55c148eb2e10 2024-03-28T16:06:17.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.573+0000 7f7057400640 1 -- 172.21.15.67:0/87660763 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f7068004650 con 0x55c148eb2e10 2024-03-28T16:06:17.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.573+0000 7f707b3ba740 1 -- 172.21.15.67:0/87660763 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f70640054d0 con 0x55c148eb2e10 2024-03-28T16:06:17.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.573+0000 7f7057400640 1 -- 172.21.15.67:0/87660763 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7068004850 con 0x55c148eb2e10 2024-03-28T16:06:17.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.573+0000 7f707b3ba740 10 client.15693 fetch_fsmap learned FSMap version 19 2024-03-28T16:06:17.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.573+0000 7f707b3ba740 1 -- 172.21.15.67:0/87660763 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55c148ae7170 con 0x55c148eb2e10 2024-03-28T16:06:17.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.573+0000 7f7057400640 1 -- 172.21.15.67:0/87660763 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f70640054d0 con 0x55c148eb2e10 2024-03-28T16:06:17.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.573+0000 7f7057400640 1 -- 172.21.15.67:0/87660763 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f706803c030 con 0x55c148eb2e10 2024-03-28T16:06:17.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.574+0000 7f707b3ba740 10 client.15693 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:06:17.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.574+0000 7f707b3ba740 10 client.15693 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:06:17.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.574+0000 7f707b3ba740 20 client.15693 populate_metadata read hostname 'smithi067' 2024-03-28T16:06:17.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.574+0000 7f707b3ba740 1 --2- 172.21.15.67:0/87660763 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55c148ec3180 0x55c148ec5620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:17.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.574+0000 7f707b3ba740 4 client.15693 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:06:17.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.574+0000 7f707b3ba740 1 -- 172.21.15.67:0/87660763 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c148e203f0 con 0x55c148ec3180 2024-03-28T16:06:17.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.574+0000 7f7075c00640 1 --2- 172.21.15.67:0/87660763 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55c148ec3180 0x55c148ec5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:17.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.575+0000 7f7057400640 10 client.15693 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:06:17.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.575+0000 7f7075c00640 1 --2- 172.21.15.67:0/87660763 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55c148ec3180 0x55c148ec5620 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=354e0194535c2c67 server_cookie=e2eab14eb0962716 in_seq=0 out_seq=0 2024-03-28T16:06:17.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.577+0000 7f7057400640 1 -- 172.21.15.67:0/87660763 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c148e203f0 con 0x55c148ec3180 2024-03-28T16:06:17.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.577+0000 7f7057400640 10 client.15693 handle_command_reply: tid=0 2024-03-28T16:06:17.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.577+0000 7f707b3ba740 1 client.15693 shutdown 2024-03-28T16:06:17.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.578+0000 7f707b3ba740 1 -- 172.21.15.67:0/87660763 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c148eb2e10 msgr2=0x55c148ebea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:17.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.578+0000 7f707b3ba740 1 --2- 172.21.15.67:0/87660763 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c148eb2e10 0x55c148ebea10 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f706802c6d0 tx=0x7f70680bd450 comp rx=0 tx=0).stop 2024-03-28T16:06:17.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.578+0000 7f707b3ba740 1 -- 172.21.15.67:0/87660763 shutdown_connections 2024-03-28T16:06:17.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.578+0000 7f707b3ba740 1 --2- 172.21.15.67:0/87660763 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55c148ec3180 0x55c148ec5620 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:17.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.578+0000 7f707b3ba740 1 --2- 172.21.15.67:0/87660763 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c148eb2e10 0x55c148ebea10 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:17.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.578+0000 7f707b3ba740 1 --2- 172.21.15.67:0/87660763 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c148eadca0 0x55c148eb79d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:17.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.578+0000 7f707b3ba740 1 --2- 172.21.15.67:0/87660763 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c148ead2b0 0x55c148eb7490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:17.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.578+0000 7f707b3ba740 1 -- 172.21.15.67:0/87660763 >> 172.21.15.67:0/87660763 conn(0x55c148ea8820 msgr2=0x55c148ea7a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:06:17.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.578+0000 7f7057400640 0 client.15693 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:06:17.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.578+0000 7f707b3ba740 1 -- 172.21.15.67:0/87660763 shutdown_connections 2024-03-28T16:06:17.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.578+0000 7f707b3ba740 1 -- 172.21.15.67:0/87660763 wait complete. 2024-03-28T16:06:17.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.578+0000 7f707b3ba740 20 client.15693 trim_cache size 0 max 16384 2024-03-28T16:06:17.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.597+0000 7f707b3ba740 1 Processor -- start 2024-03-28T16:06:17.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.597+0000 7f707b3ba740 1 -- start start 2024-03-28T16:06:17.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.597+0000 7f707b3ba740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c148ead2b0 0x55c148eae740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:17.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.597+0000 7f707b3ba740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c148eadca0 0x55c148ebe9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:17.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.597+0000 7f707b3ba740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c148eb2e10 0x55c148eaec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:17.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.597+0000 7f707b3ba740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70701a8b10 con 0x55c148eb2e10 2024-03-28T16:06:17.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.597+0000 7f707b3ba740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70701ab0d0 con 0x55c148ead2b0 2024-03-28T16:06:17.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.597+0000 7f707b3ba740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70701a9110 con 0x55c148eadca0 2024-03-28T16:06:17.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.597+0000 7f7076600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c148eb2e10 0x55c148eaec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:17.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.597+0000 7f7076600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c148eb2e10 0x55c148eaec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48924/0 (socket says 172.21.15.67:48924) 2024-03-28T16:06:17.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.597+0000 7f7076600640 1 -- 172.21.15.67:0/628033398 learned_addr learned my addr 172.21.15.67:0/628033398 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:06:17.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.598+0000 7f7075c00640 1 --2- 172.21.15.67:0/628033398 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c148ead2b0 0x55c148eae740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:17.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.598+0000 7f7075200640 1 --2- 172.21.15.67:0/628033398 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c148eadca0 0x55c148ebe9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:17.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.598+0000 7f7076600640 1 -- 172.21.15.67:0/628033398 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c148eadca0 msgr2=0x55c148ebe9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:06:17.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.598+0000 7f7076600640 1 --2- 172.21.15.67:0/628033398 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c148eadca0 0x55c148ebe9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:17.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.598+0000 7f7076600640 1 -- 172.21.15.67:0/628033398 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c148ead2b0 msgr2=0x55c148eae740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:06:17.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.598+0000 7f7076600640 1 --2- 172.21.15.67:0/628033398 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c148ead2b0 0x55c148eae740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:17.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.598+0000 7f7076600640 1 -- 172.21.15.67:0/628033398 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c148ea9590 con 0x55c148eb2e10 2024-03-28T16:06:17.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.598+0000 7f7075200640 1 --2- 172.21.15.67:0/628033398 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c148eadca0 0x55c148ebe9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:06:17.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.598+0000 7f7075c00640 1 --2- 172.21.15.67:0/628033398 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c148ead2b0 0x55c148eae740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T16:06:17.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.598+0000 7f7076600640 1 --2- 172.21.15.67:0/628033398 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c148eb2e10 0x55c148eaec80 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f70680ba260 tx=0x7f706802c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1bee4288ff27750 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:06:17.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.600+0000 7f7062000640 1 -- 172.21.15.67:0/628033398 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70680b9b30 con 0x55c148eb2e10 2024-03-28T16:06:17.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.600+0000 7f7062000640 1 -- 172.21.15.67:0/628033398 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f70680c5cc0 con 0x55c148eb2e10 2024-03-28T16:06:17.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.600+0000 7f7062000640 1 -- 172.21.15.67:0/628033398 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70680bebc0 con 0x55c148eb2e10 2024-03-28T16:06:17.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.601+0000 7f707b3ba740 1 -- 172.21.15.67:0/628033398 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c148eb2e10 msgr2=0x55c148eaec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:17.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.601+0000 7f707b3ba740 1 --2- 172.21.15.67:0/628033398 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c148eb2e10 0x55c148eaec80 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f70680ba260 tx=0x7f706802c0f0 comp rx=0 tx=0).stop 2024-03-28T16:06:17.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.601+0000 7f707b3ba740 1 -- 172.21.15.67:0/628033398 shutdown_connections 2024-03-28T16:06:17.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.601+0000 7f707b3ba740 1 --2- 172.21.15.67:0/628033398 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c148eb2e10 0x55c148eaec80 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:17.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.601+0000 7f707b3ba740 1 --2- 172.21.15.67:0/628033398 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c148eadca0 0x55c148ebe9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:17.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.601+0000 7f707b3ba740 1 --2- 172.21.15.67:0/628033398 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c148ead2b0 0x55c148eae740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:17.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.601+0000 7f707b3ba740 1 -- 172.21.15.67:0/628033398 >> 172.21.15.67:0/628033398 conn(0x55c148ea8820 msgr2=0x55c148eac310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:06:17.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.603+0000 7f707b3ba740 1 -- 172.21.15.67:0/628033398 shutdown_connections 2024-03-28T16:06:17.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.603+0000 7f707b3ba740 1 -- 172.21.15.67:0/628033398 wait complete. 2024-03-28T16:06:17.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.603+0000 7f707b3ba740 1 -- start start 2024-03-28T16:06:17.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.604+0000 7f707b3ba740 1 Processor -- start 2024-03-28T16:06:17.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.604+0000 7f707b3ba740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c148ead2b0 0x55c148ebca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:17.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.604+0000 7f707b3ba740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c148eadca0 0x55c148ebcf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:17.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.604+0000 7f707b3ba740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c148eb2e10 0x55c148ebdf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:17.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.604+0000 7f707b3ba740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70701a9750 con 0x55c148eadca0 2024-03-28T16:06:17.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.604+0000 7f707b3ba740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70701aaa70 con 0x55c148eb2e10 2024-03-28T16:06:17.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.604+0000 7f707b3ba740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70701a9db0 con 0x55c148ead2b0 2024-03-28T16:06:17.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.604+0000 7f707b3ba740 10 client.0 _ll_register_callbacks cb 0x7f7068034790 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:06:17.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.604+0000 7f7075200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c148eadca0 0x55c148ebcf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:17.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.605+0000 7f7075200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c148eadca0 0x55c148ebcf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48928/0 (socket says 172.21.15.67:48928) 2024-03-28T16:06:17.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.605+0000 7f7075200640 1 -- 172.21.15.67:0/3565106820 learned_addr learned my addr 172.21.15.67:0/3565106820 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:06:17.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.605+0000 7f7075c00640 1 --2- 172.21.15.67:0/3565106820 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c148ead2b0 0x55c148ebca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:17.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.605+0000 7f7076600640 1 --2- 172.21.15.67:0/3565106820 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c148eb2e10 0x55c148ebdf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:17.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.605+0000 7f7075200640 1 -- 172.21.15.67:0/3565106820 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c148ead2b0 msgr2=0x55c148ebca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:06:17.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.605+0000 7f7075200640 1 --2- 172.21.15.67:0/3565106820 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c148ead2b0 0x55c148ebca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:17.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.605+0000 7f7075200640 1 -- 172.21.15.67:0/3565106820 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c148eb2e10 msgr2=0x55c148ebdf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:06:17.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.605+0000 7f7075200640 1 --2- 172.21.15.67:0/3565106820 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c148eb2e10 0x55c148ebdf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:17.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.605+0000 7f7075200640 1 -- 172.21.15.67:0/3565106820 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c148ebe540 con 0x55c148eadca0 2024-03-28T16:06:17.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.605+0000 7f7075200640 1 --2- 172.21.15.67:0/3565106820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c148eadca0 0x55c148ebcf90 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f705c0118a0 tx=0x7f705c005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=120e0fc16a20e69b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:06:17.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.605+0000 7f7057400640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:06:17.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.606+0000 7f7057400640 1 -- 172.21.15.67:0/3565106820 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f705c01d8e0 con 0x55c148eadca0 2024-03-28T16:06:17.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.606+0000 7f7057400640 1 -- 172.21.15.67:0/3565106820 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f705c01da80 con 0x55c148eadca0 2024-03-28T16:06:17.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.606+0000 7f707b3ba740 1 -- 172.21.15.67:0/3565106820 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f70680c5cc0 con 0x55c148eadca0 2024-03-28T16:06:17.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.606+0000 7f7057400640 1 -- 172.21.15.67:0/3565106820 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f705c01dc80 con 0x55c148eadca0 2024-03-28T16:06:17.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.606+0000 7f707b3ba740 10 client.15699 fetch_fsmap learned FSMap version 19 2024-03-28T16:06:17.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.606+0000 7f707b3ba740 1 -- 172.21.15.67:0/3565106820 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55c148f27c30 con 0x55c148eadca0 2024-03-28T16:06:17.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.606+0000 7f7057400640 1 -- 172.21.15.67:0/3565106820 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f70680c5cc0 con 0x55c148eadca0 2024-03-28T16:06:17.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.607+0000 7f7057400640 1 -- 172.21.15.67:0/3565106820 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f705c025020 con 0x55c148eadca0 2024-03-28T16:06:17.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.607+0000 7f707b3ba740 10 client.15699 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:06:17.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.607+0000 7f707b3ba740 10 client.15699 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:06:17.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.607+0000 7f707b3ba740 20 client.15699 populate_metadata read hostname 'smithi067' 2024-03-28T16:06:17.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.607+0000 7f707b3ba740 1 --2- 172.21.15.67:0/3565106820 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55c148f28610 0x55c148f2aa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:17.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.607+0000 7f707b3ba740 4 client.15699 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T16:06:17.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.607+0000 7f707b3ba740 1 -- 172.21.15.67:0/3565106820 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55c148e1a740 con 0x55c148f28610 2024-03-28T16:06:17.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.607+0000 7f7075c00640 1 --2- 172.21.15.67:0/3565106820 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55c148f28610 0x55c148f2aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:17.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.608+0000 7f7057400640 10 client.15699 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:06:17.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.608+0000 7f7075c00640 1 --2- 172.21.15.67:0/3565106820 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55c148f28610 0x55c148f2aa30 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3255486a30b5e391 server_cookie=b69663243c371e88 in_seq=0 out_seq=0 2024-03-28T16:06:17.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.612+0000 7f7057400640 1 -- 172.21.15.67:0/3565106820 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55c148e1a740 con 0x55c148f28610 2024-03-28T16:06:17.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.612+0000 7f7057400640 10 client.15699 handle_command_reply: tid=0 2024-03-28T16:06:17.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.612+0000 7f707b3ba740 1 client.15699 shutdown 2024-03-28T16:06:17.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.612+0000 7f707b3ba740 1 -- 172.21.15.67:0/3565106820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c148eadca0 msgr2=0x55c148ebcf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:17.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.612+0000 7f707b3ba740 1 --2- 172.21.15.67:0/3565106820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c148eadca0 0x55c148ebcf90 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f705c0118a0 tx=0x7f705c005850 comp rx=0 tx=0).stop 2024-03-28T16:06:17.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.612+0000 7f707b3ba740 1 -- 172.21.15.67:0/3565106820 shutdown_connections 2024-03-28T16:06:17.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.612+0000 7f707b3ba740 1 --2- 172.21.15.67:0/3565106820 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55c148f28610 0x55c148f2aa30 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:17.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.612+0000 7f707b3ba740 1 --2- 172.21.15.67:0/3565106820 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c148eb2e10 0x55c148ebdf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:17.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.612+0000 7f707b3ba740 1 --2- 172.21.15.67:0/3565106820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c148eadca0 0x55c148ebcf90 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:17.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.612+0000 7f707b3ba740 1 --2- 172.21.15.67:0/3565106820 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c148ead2b0 0x55c148ebca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:17.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.612+0000 7f7057400640 0 client.15699 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:06:17.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.612+0000 7f707b3ba740 1 -- 172.21.15.67:0/3565106820 >> 172.21.15.67:0/3565106820 conn(0x55c148ea8820 msgr2=0x55c148ea71f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:06:17.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.612+0000 7f707b3ba740 1 -- 172.21.15.67:0/3565106820 shutdown_connections 2024-03-28T16:06:17.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.612+0000 7f707b3ba740 1 -- 172.21.15.67:0/3565106820 wait complete. 2024-03-28T16:06:17.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.612+0000 7f707b3ba740 20 client.15699 trim_cache size 0 max 16384 2024-03-28T16:06:17.615 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-28T16:06:17.615 INFO:teuthology.orchestra.run.smithi067.stdout: "return_code": 0, 2024-03-28T16:06:17.615 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_tag": "cca7ef74-652c-43dc-b177-b6598dee6793", 2024-03-28T16:06:17.615 INFO:teuthology.orchestra.run.smithi067.stdout: "mode": "asynchronous" 2024-03-28T16:06:17.615 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-28T16:06:17.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.613+0000 7f7077000640 1 -- 172.21.15.67:0/782935171 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f70580a2800 msgr2=0x7f70580a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:17.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.613+0000 7f7077000640 1 --2- 172.21.15.67:0/782935171 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f70580a2800 0x7f70580a4cc0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x55c148eae500 tx=0x7f705c00c380 comp rx=0 tx=0).stop 2024-03-28T16:06:17.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.613+0000 7f7077000640 1 -- 172.21.15.67:0/782935171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7070103150 msgr2=0x7f70701a0f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:17.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.613+0000 7f7077000640 1 --2- 172.21.15.67:0/782935171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7070103150 0x7f70701a0f30 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f706802c700 tx=0x7f70680328a0 comp rx=0 tx=0).stop 2024-03-28T16:06:17.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.614+0000 7f7077000640 1 -- 172.21.15.67:0/782935171 shutdown_connections 2024-03-28T16:06:17.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.614+0000 7f7077000640 1 --2- 172.21.15.67:0/782935171 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f70580a2800 0x7f70580a4cc0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:17.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.614+0000 7f7077000640 1 --2- 172.21.15.67:0/782935171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7070103150 0x7f70701a0f30 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:17.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.614+0000 7f7077000640 1 --2- 172.21.15.67:0/782935171 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7070102790 0x7f7070199ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:17.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.614+0000 7f7077000640 1 --2- 172.21.15.67:0/782935171 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f70701015a0 0x7f7070199980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:17.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.614+0000 7f7077000640 1 -- 172.21.15.67:0/782935171 >> 172.21.15.67:0/782935171 conn(0x7f70700fcb10 msgr2=0x7f70700738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:06:17.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.614+0000 7f7077000640 1 -- 172.21.15.67:0/782935171 shutdown_connections 2024-03-28T16:06:17.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.614+0000 7f7077000640 1 -- 172.21.15.67:0/782935171 wait complete. 2024-03-28T16:06:17.632 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T16:06:17.632 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T16:06:17.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.857+0000 7f2fd4600640 1 -- 172.21.15.67:0/2209102053 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fcc1038e0 msgr2=0x7f2fcc103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:17.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.857+0000 7f2fd4600640 1 --2- 172.21.15.67:0/2209102053 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fcc1038e0 0x7f2fcc103d00 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f2fc0009900 tx=0x7f2fc002bc30 comp rx=0 tx=0).stop 2024-03-28T16:06:17.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.857+0000 7f2fd4600640 1 -- 172.21.15.67:0/2209102053 shutdown_connections 2024-03-28T16:06:17.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.857+0000 7f2fd4600640 1 --2- 172.21.15.67:0/2209102053 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2fcc105490 0x7f2fcc10bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:17.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.857+0000 7f2fd4600640 1 --2- 172.21.15.67:0/2209102053 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2fcc104ad0 0x7f2fcc104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:17.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.857+0000 7f2fd4600640 1 --2- 172.21.15.67:0/2209102053 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fcc1038e0 0x7f2fcc103d00 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:17.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.857+0000 7f2fd4600640 1 -- 172.21.15.67:0/2209102053 >> 172.21.15.67:0/2209102053 conn(0x7f2fcc0fee50 msgr2=0x7f2fcc101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:06:17.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.857+0000 7f2fd4600640 1 -- 172.21.15.67:0/2209102053 shutdown_connections 2024-03-28T16:06:17.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.858+0000 7f2fd4600640 1 -- 172.21.15.67:0/2209102053 wait complete. 2024-03-28T16:06:17.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.859+0000 7f2fd4600640 1 Processor -- start 2024-03-28T16:06:17.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.859+0000 7f2fd4600640 1 -- start start 2024-03-28T16:06:17.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.859+0000 7f2fd4600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2fcc1038e0 0x7f2fcc199b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:17.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.859+0000 7f2fd4600640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2fcc104ad0 0x7f2fcc19a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:17.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.859+0000 7f2fd4600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fcc105490 0x7f2fcc1a1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:17.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.859+0000 7f2fd4600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc10f450 con 0x7f2fcc105490 2024-03-28T16:06:17.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.859+0000 7f2fd4600640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc10f2d0 con 0x7f2fcc104ad0 2024-03-28T16:06:17.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.859+0000 7f2fd4600640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc10f5d0 con 0x7f2fcc1038e0 2024-03-28T16:06:17.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.860+0000 7f2fd2600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fcc105490 0x7f2fcc1a1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:17.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.860+0000 7f2fd2600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fcc105490 0x7f2fcc1a1120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48948/0 (socket says 172.21.15.67:48948) 2024-03-28T16:06:17.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.860+0000 7f2fd2600640 1 -- 172.21.15.67:0/2640410851 learned_addr learned my addr 172.21.15.67:0/2640410851 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:06:17.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.860+0000 7f2fd1200640 1 --2- 172.21.15.67:0/2640410851 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2fcc104ad0 0x7f2fcc19a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:17.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.860+0000 7f2fd1c00640 1 --2- 172.21.15.67:0/2640410851 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2fcc1038e0 0x7f2fcc199b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:17.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.860+0000 7f2fd2600640 1 -- 172.21.15.67:0/2640410851 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2fcc1038e0 msgr2=0x7f2fcc199b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:17.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.860+0000 7f2fd2600640 1 --2- 172.21.15.67:0/2640410851 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2fcc1038e0 0x7f2fcc199b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:17.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.860+0000 7f2fd2600640 1 -- 172.21.15.67:0/2640410851 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2fcc104ad0 msgr2=0x7f2fcc19a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:17.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.860+0000 7f2fd2600640 1 --2- 172.21.15.67:0/2640410851 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2fcc104ad0 0x7f2fcc19a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:17.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.860+0000 7f2fd2600640 1 -- 172.21.15.67:0/2640410851 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fcc1a1790 con 0x7f2fcc105490 2024-03-28T16:06:17.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.860+0000 7f2fd1200640 1 --2- 172.21.15.67:0/2640410851 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2fcc104ad0 0x7f2fcc19a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T16:06:17.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.861+0000 7f2fd2600640 1 --2- 172.21.15.67:0/2640410851 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fcc105490 0x7f2fcc1a1120 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f2fc400eb10 tx=0x7f2fc400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:06:17.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.861+0000 7f2fca000640 1 -- 172.21.15.67:0/2640410851 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc4004b20 con 0x7f2fcc105490 2024-03-28T16:06:17.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.861+0000 7f2fca000640 1 -- 172.21.15.67:0/2640410851 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f2fc4004cc0 con 0x7f2fcc105490 2024-03-28T16:06:17.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.861+0000 7f2fca000640 1 -- 172.21.15.67:0/2640410851 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc4010690 con 0x7f2fcc105490 2024-03-28T16:06:17.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.861+0000 7f2fd4600640 1 -- 172.21.15.67:0/2640410851 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fcc1a1a80 con 0x7f2fcc105490 2024-03-28T16:06:17.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.861+0000 7f2fd4600640 1 -- 172.21.15.67:0/2640410851 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fcc1a1f80 con 0x7f2fcc105490 2024-03-28T16:06:17.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.865+0000 7f2fca000640 1 -- 172.21.15.67:0/2640410851 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f2fc4005020 con 0x7f2fcc105490 2024-03-28T16:06:17.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.865+0000 7f2fd7408740 1 Processor -- start 2024-03-28T16:06:17.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.865+0000 7f2fd7408740 1 -- start start 2024-03-28T16:06:17.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.866+0000 7f2fd7408740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c3725a3e80 0x55c3725a42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:17.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.866+0000 7f2fd7408740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c3725a4870 0x55c3725a9370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:17.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.866+0000 7f2fd7408740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c3725a99e0 0x55c3725abe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:17.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.866+0000 7f2fd7408740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c372509250 con 0x55c3725a4870 2024-03-28T16:06:17.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.866+0000 7f2fd7408740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3724b1a30 con 0x55c3725a99e0 2024-03-28T16:06:17.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.866+0000 7f2fd7408740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3725052f0 con 0x55c3725a3e80 2024-03-28T16:06:17.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.866+0000 7f2fd1c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c3725a4870 0x55c3725a9370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:17.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.866+0000 7f2fd1c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c3725a4870 0x55c3725a9370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48950/0 (socket says 172.21.15.67:48950) 2024-03-28T16:06:17.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.866+0000 7f2fd1c00640 1 -- 172.21.15.67:0/1016812546 learned_addr learned my addr 172.21.15.67:0/1016812546 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:06:17.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.867+0000 7f2fd1200640 1 --2- 172.21.15.67:0/1016812546 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c3725a3e80 0x55c3725a42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:17.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.867+0000 7f2fd1200640 1 --2- 172.21.15.67:0/1016812546 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c3725a99e0 0x55c3725abe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:17.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.867+0000 7f2fd1c00640 1 -- 172.21.15.67:0/1016812546 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c3725a3e80 msgr2=0x55c3725a42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:06:17.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.867+0000 7f2fd1c00640 1 --2- 172.21.15.67:0/1016812546 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c3725a3e80 0x55c3725a42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:17.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.867+0000 7f2fd1c00640 1 -- 172.21.15.67:0/1016812546 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c3725a99e0 msgr2=0x55c3725abe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:06:17.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.867+0000 7f2fd1c00640 1 --2- 172.21.15.67:0/1016812546 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c3725a99e0 0x55c3725abe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:17.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.867+0000 7f2fd1c00640 1 -- 172.21.15.67:0/1016812546 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c37249c690 con 0x55c3725a4870 2024-03-28T16:06:17.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.867+0000 7f2fd1200640 1 --2- 172.21.15.67:0/1016812546 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c3725a3e80 0x55c3725a42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T16:06:17.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.867+0000 7f2fca000640 1 --2- 172.21.15.67:0/2640410851 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f2fb40a2880 0x7f2fb40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:17.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.867+0000 7f2fd1c00640 1 --2- 172.21.15.67:0/1016812546 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c3725a4870 0x55c3725a9370 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f2fc0005ab0 tx=0x7f2fc0009670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b79065f7f3f65ff server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:06:17.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.868+0000 7f2fca000640 1 -- 172.21.15.67:0/2640410851 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f2fc40cf130 con 0x7f2fcc105490 2024-03-28T16:06:17.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.868+0000 7f2fd1200640 1 --2- 172.21.15.67:0/2640410851 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f2fb40a2880 0x7f2fb40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:17.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.868+0000 7f2fbea00640 1 -- 172.21.15.67:0/1016812546 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc0044020 con 0x55c3725a4870 2024-03-28T16:06:17.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.868+0000 7f2fbea00640 1 -- 172.21.15.67:0/1016812546 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f2fc002cae0 con 0x55c3725a4870 2024-03-28T16:06:17.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.868+0000 7f2fbea00640 1 -- 172.21.15.67:0/1016812546 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc0004920 con 0x55c3725a4870 2024-03-28T16:06:17.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.869+0000 7f2fd7408740 1 -- 172.21.15.67:0/1016812546 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c3725a4870 msgr2=0x55c3725a9370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:17.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.869+0000 7f2fd1200640 1 --2- 172.21.15.67:0/2640410851 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f2fb40a2880 0x7f2fb40a4d40 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x55c3725a50d0 tx=0x7f2fb8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:06:17.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.869+0000 7f2fd7408740 1 --2- 172.21.15.67:0/1016812546 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c3725a4870 0x55c3725a9370 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f2fc0005ab0 tx=0x7f2fc0009670 comp rx=0 tx=0).stop 2024-03-28T16:06:17.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.869+0000 7f2fd7408740 1 -- 172.21.15.67:0/1016812546 shutdown_connections 2024-03-28T16:06:17.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.869+0000 7f2fd7408740 1 --2- 172.21.15.67:0/1016812546 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c3725a99e0 0x55c3725abe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:17.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.869+0000 7f2fd7408740 1 --2- 172.21.15.67:0/1016812546 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c3725a4870 0x55c3725a9370 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:17.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.869+0000 7f2fd7408740 1 --2- 172.21.15.67:0/1016812546 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c3725a3e80 0x55c3725a42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:17.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.869+0000 7f2fd7408740 1 -- 172.21.15.67:0/1016812546 >> 172.21.15.67:0/1016812546 conn(0x55c37256c200 msgr2=0x55c37259f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:06:17.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.869+0000 7f2fd7408740 1 -- 172.21.15.67:0/1016812546 shutdown_connections 2024-03-28T16:06:17.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.869+0000 7f2fd7408740 1 -- 172.21.15.67:0/1016812546 wait complete. 2024-03-28T16:06:17.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.870+0000 7f2fd7408740 1 -- start start 2024-03-28T16:06:17.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.870+0000 7f2fd7408740 1 Processor -- start 2024-03-28T16:06:17.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.870+0000 7f2fd7408740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c3725a3e80 0x55c3725ade20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:17.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.870+0000 7f2fd7408740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c3725a4870 0x55c3725ae360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:17.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.870+0000 7f2fd7408740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c3725a99e0 0x55c3725b53a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:17.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.870+0000 7f2fd7408740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3725ac930 con 0x55c3725a99e0 2024-03-28T16:06:17.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.870+0000 7f2fd7408740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3724c4e80 con 0x55c3725a4870 2024-03-28T16:06:17.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.870+0000 7f2fd7408740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3725acab0 con 0x55c3725a3e80 2024-03-28T16:06:17.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.870+0000 7f2fd7408740 10 client.0 _ll_register_callbacks cb 0x55c3724d7130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:06:17.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.871+0000 7f2fd2600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c3725a99e0 0x55c3725b53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:17.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.871+0000 7f2fd2600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c3725a99e0 0x55c3725b53a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48960/0 (socket says 172.21.15.67:48960) 2024-03-28T16:06:17.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.871+0000 7f2fd2600640 1 -- 172.21.15.67:0/1936709874 learned_addr learned my addr 172.21.15.67:0/1936709874 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:06:17.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.871+0000 7f2fd1200640 1 --2- 172.21.15.67:0/1936709874 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c3725a4870 0x55c3725ae360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:17.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.871+0000 7f2fd1c00640 1 --2- 172.21.15.67:0/1936709874 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c3725a3e80 0x55c3725ade20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:17.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.871+0000 7f2fd2600640 1 -- 172.21.15.67:0/1936709874 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c3725a3e80 msgr2=0x55c3725ade20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:06:17.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.871+0000 7f2fd2600640 1 --2- 172.21.15.67:0/1936709874 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c3725a3e80 0x55c3725ade20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:17.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.871+0000 7f2fd2600640 1 -- 172.21.15.67:0/1936709874 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c3725a4870 msgr2=0x55c3725ae360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:06:17.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.871+0000 7f2fd2600640 1 --2- 172.21.15.67:0/1936709874 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c3725a4870 0x55c3725ae360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:17.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.871+0000 7f2fd2600640 1 -- 172.21.15.67:0/1936709874 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c3722e77d0 con 0x55c3725a99e0 2024-03-28T16:06:17.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.872+0000 7f2fd2600640 1 --2- 172.21.15.67:0/1936709874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c3725a99e0 0x55c3725b53a0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f2fc4005850 tx=0x7f2fc409b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a87571ee4b4272c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:06:17.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.872+0000 7f2fb3400640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:06:17.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.872+0000 7f2fb3400640 1 -- 172.21.15.67:0/1936709874 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc40b3020 con 0x55c3725a99e0 2024-03-28T16:06:17.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.872+0000 7f2fb3400640 1 -- 172.21.15.67:0/1936709874 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f2fc40a3d90 con 0x55c3725a99e0 2024-03-28T16:06:17.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.872+0000 7f2fd7408740 1 -- 172.21.15.67:0/1936709874 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2fc002cae0 con 0x55c3725a99e0 2024-03-28T16:06:17.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.872+0000 7f2fb3400640 1 -- 172.21.15.67:0/1936709874 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2fc40afe60 con 0x55c3725a99e0 2024-03-28T16:06:17.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.872+0000 7f2fd7408740 10 client.15711 fetch_fsmap learned FSMap version 19 2024-03-28T16:06:17.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.872+0000 7f2fd7408740 1 -- 172.21.15.67:0/1936709874 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55c37221d3f0 con 0x55c3725a99e0 2024-03-28T16:06:17.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.873+0000 7f2fb3400640 1 -- 172.21.15.67:0/1936709874 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f2fc002cae0 con 0x55c3725a99e0 2024-03-28T16:06:17.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.873+0000 7f2fb3400640 1 -- 172.21.15.67:0/1936709874 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f2fc4005310 con 0x55c3725a99e0 2024-03-28T16:06:17.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.873+0000 7f2fd7408740 10 client.15711 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:06:17.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.873+0000 7f2fd7408740 10 client.15711 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:06:17.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.873+0000 7f2fd7408740 20 client.15711 populate_metadata read hostname 'smithi067' 2024-03-28T16:06:17.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.873+0000 7f2fd7408740 1 --2- 172.21.15.67:0/1936709874 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55c3725b9ce0 0x55c3725bc140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:17.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.873+0000 7f2fd7408740 4 client.15711 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:06:17.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.873+0000 7f2fd7408740 1 -- 172.21.15.67:0/1936709874 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c372516790 con 0x55c3725b9ce0 2024-03-28T16:06:17.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.874+0000 7f2fd1c00640 1 --2- 172.21.15.67:0/1936709874 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55c3725b9ce0 0x55c3725bc140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:17.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.874+0000 7f2fd1c00640 1 --2- 172.21.15.67:0/1936709874 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55c3725b9ce0 0x55c3725bc140 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a5e68b64ee5159 server_cookie=97479e10e098caa2 in_seq=0 out_seq=0 2024-03-28T16:06:17.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.875+0000 7f2fb3400640 10 client.15711 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:06:17.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.876+0000 7f2fb3400640 1 -- 172.21.15.67:0/1936709874 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c372516790 con 0x55c3725b9ce0 2024-03-28T16:06:17.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.876+0000 7f2fb3400640 10 client.15711 handle_command_reply: tid=0 2024-03-28T16:06:17.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.876+0000 7f2fd7408740 1 client.15711 shutdown 2024-03-28T16:06:17.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.877+0000 7f2fd7408740 1 -- 172.21.15.67:0/1936709874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c3725a99e0 msgr2=0x55c3725b53a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:17.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.877+0000 7f2fd7408740 1 --2- 172.21.15.67:0/1936709874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c3725a99e0 0x55c3725b53a0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f2fc4005850 tx=0x7f2fc409b4a0 comp rx=0 tx=0).stop 2024-03-28T16:06:17.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.877+0000 7f2fd7408740 1 -- 172.21.15.67:0/1936709874 shutdown_connections 2024-03-28T16:06:17.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.877+0000 7f2fd7408740 1 --2- 172.21.15.67:0/1936709874 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55c3725b9ce0 0x55c3725bc140 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:17.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.877+0000 7f2fd7408740 1 --2- 172.21.15.67:0/1936709874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c3725a99e0 0x55c3725b53a0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:17.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.877+0000 7f2fd7408740 1 --2- 172.21.15.67:0/1936709874 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c3725a4870 0x55c3725ae360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:17.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.877+0000 7f2fd7408740 1 --2- 172.21.15.67:0/1936709874 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c3725a3e80 0x55c3725ade20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:17.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.877+0000 7f2fb3400640 0 client.15711 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:06:17.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.877+0000 7f2fd7408740 1 -- 172.21.15.67:0/1936709874 >> 172.21.15.67:0/1936709874 conn(0x55c37256c200 msgr2=0x55c37259ea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:06:17.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.877+0000 7f2fd7408740 1 -- 172.21.15.67:0/1936709874 shutdown_connections 2024-03-28T16:06:17.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.877+0000 7f2fd7408740 1 -- 172.21.15.67:0/1936709874 wait complete. 2024-03-28T16:06:17.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.877+0000 7f2fd7408740 20 client.15711 trim_cache size 0 max 16384 2024-03-28T16:06:17.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.897+0000 7f2fd7408740 1 Processor -- start 2024-03-28T16:06:17.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.897+0000 7f2fd7408740 1 -- start start 2024-03-28T16:06:17.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.897+0000 7f2fd7408740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c3725a3e80 0x55c3725a8e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:17.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.897+0000 7f2fd7408740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c3725a4870 0x55c3725b5360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:17.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.897+0000 7f2fd7408740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c3725a99e0 0x55c3725a2a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:17.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.897+0000 7f2fd7408740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc1a8ce0 con 0x55c3725a3e80 2024-03-28T16:06:17.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.897+0000 7f2fd7408740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc1ab2a0 con 0x55c3725a4870 2024-03-28T16:06:17.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.897+0000 7f2fd7408740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc1a92e0 con 0x55c3725a99e0 2024-03-28T16:06:17.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.898+0000 7f2fd1c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c3725a3e80 0x55c3725a8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:17.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.898+0000 7f2fd1c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c3725a3e80 0x55c3725a8e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48962/0 (socket says 172.21.15.67:48962) 2024-03-28T16:06:17.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.898+0000 7f2fd1c00640 1 -- 172.21.15.67:0/2309095194 learned_addr learned my addr 172.21.15.67:0/2309095194 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:06:17.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.898+0000 7f2fd1200640 1 --2- 172.21.15.67:0/2309095194 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c3725a4870 0x55c3725b5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:17.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.898+0000 7f2fd2600640 1 --2- 172.21.15.67:0/2309095194 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c3725a99e0 0x55c3725a2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:17.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.898+0000 7f2fd1c00640 1 -- 172.21.15.67:0/2309095194 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c3725a99e0 msgr2=0x55c3725a2a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:06:17.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.898+0000 7f2fd1c00640 1 --2- 172.21.15.67:0/2309095194 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c3725a99e0 0x55c3725a2a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:17.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.898+0000 7f2fd1c00640 1 -- 172.21.15.67:0/2309095194 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c3725a4870 msgr2=0x55c3725b5360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:06:17.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.898+0000 7f2fd1c00640 1 --2- 172.21.15.67:0/2309095194 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c3725a4870 0x55c3725b5360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:17.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.898+0000 7f2fd1c00640 1 -- 172.21.15.67:0/2309095194 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c3721dd0f0 con 0x55c3725a3e80 2024-03-28T16:06:17.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.898+0000 7f2fd1200640 1 --2- 172.21.15.67:0/2309095194 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c3725a4870 0x55c3725b5360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T16:06:17.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.899+0000 7f2fd1c00640 1 --2- 172.21.15.67:0/2309095194 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c3725a3e80 0x55c3725a8e70 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f2fc002c170 tx=0x7f2fc002c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2d09b43efc37244 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:06:17.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.899+0000 7f2fbe000640 1 -- 172.21.15.67:0/2309095194 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc004f070 con 0x55c3725a3e80 2024-03-28T16:06:17.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.899+0000 7f2fbe000640 1 -- 172.21.15.67:0/2309095194 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f2fc002cae0 con 0x55c3725a3e80 2024-03-28T16:06:17.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.899+0000 7f2fbe000640 1 -- 172.21.15.67:0/2309095194 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc0044040 con 0x55c3725a3e80 2024-03-28T16:06:17.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.900+0000 7f2fd7408740 1 -- 172.21.15.67:0/2309095194 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c3725a3e80 msgr2=0x55c3725a8e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:17.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.900+0000 7f2fd7408740 1 --2- 172.21.15.67:0/2309095194 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c3725a3e80 0x55c3725a8e70 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f2fc002c170 tx=0x7f2fc002c7e0 comp rx=0 tx=0).stop 2024-03-28T16:06:17.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.900+0000 7f2fd7408740 1 -- 172.21.15.67:0/2309095194 shutdown_connections 2024-03-28T16:06:17.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.900+0000 7f2fd7408740 1 --2- 172.21.15.67:0/2309095194 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c3725a99e0 0x55c3725a2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:17.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.900+0000 7f2fd7408740 1 --2- 172.21.15.67:0/2309095194 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c3725a4870 0x55c3725b5360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:17.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.900+0000 7f2fd7408740 1 --2- 172.21.15.67:0/2309095194 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c3725a3e80 0x55c3725a8e70 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:17.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.900+0000 7f2fd7408740 1 -- 172.21.15.67:0/2309095194 >> 172.21.15.67:0/2309095194 conn(0x55c37256c200 msgr2=0x55c3725a5ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:06:17.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.900+0000 7f2fd7408740 1 -- 172.21.15.67:0/2309095194 shutdown_connections 2024-03-28T16:06:17.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.900+0000 7f2fd7408740 1 -- 172.21.15.67:0/2309095194 wait complete. 2024-03-28T16:06:17.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.900+0000 7f2fd7408740 1 -- start start 2024-03-28T16:06:17.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.900+0000 7f2fd7408740 1 Processor -- start 2024-03-28T16:06:17.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.901+0000 7f2fd7408740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c3725a3e80 0x55c3725b2f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:17.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.901+0000 7f2fd7408740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c3725a4870 0x55c3725b3450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:17.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.901+0000 7f2fd7408740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c3725a99e0 0x55c3725b4410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:17.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.901+0000 7f2fd7408740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc1a9920 con 0x55c3725a99e0 2024-03-28T16:06:17.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.901+0000 7f2fd7408740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc1aac40 con 0x55c3725a4870 2024-03-28T16:06:17.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.901+0000 7f2fd7408740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc1a9f80 con 0x55c3725a3e80 2024-03-28T16:06:17.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.901+0000 7f2fd7408740 10 client.0 _ll_register_callbacks cb 0x7f2fc40057a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:06:17.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.901+0000 7f2fd2600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c3725a99e0 0x55c3725b4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:17.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.901+0000 7f2fd2600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c3725a99e0 0x55c3725b4410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48978/0 (socket says 172.21.15.67:48978) 2024-03-28T16:06:17.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.901+0000 7f2fd2600640 1 -- 172.21.15.67:0/1090365116 learned_addr learned my addr 172.21.15.67:0/1090365116 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:06:17.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.901+0000 7f2fd1200640 1 --2- 172.21.15.67:0/1090365116 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c3725a4870 0x55c3725b3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:17.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.901+0000 7f2fd1c00640 1 --2- 172.21.15.67:0/1090365116 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c3725a3e80 0x55c3725b2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:17.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.902+0000 7f2fd2600640 1 -- 172.21.15.67:0/1090365116 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c3725a3e80 msgr2=0x55c3725b2f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:06:17.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.902+0000 7f2fd2600640 1 --2- 172.21.15.67:0/1090365116 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c3725a3e80 0x55c3725b2f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:17.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.902+0000 7f2fd2600640 1 -- 172.21.15.67:0/1090365116 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c3725a4870 msgr2=0x55c3725b3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:06:17.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.902+0000 7f2fd2600640 1 --2- 172.21.15.67:0/1090365116 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c3725a4870 0x55c3725b3450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:17.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.902+0000 7f2fd2600640 1 -- 172.21.15.67:0/1090365116 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c3725b4a00 con 0x55c3725a99e0 2024-03-28T16:06:17.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.902+0000 7f2fd2600640 1 --2- 172.21.15.67:0/1090365116 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c3725a99e0 0x55c3725b4410 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f2fc40049d0 tx=0x7f2fc4005660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e757cf66ce872690 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:06:17.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.902+0000 7f2fb3400640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:06:17.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.902+0000 7f2fb3400640 1 -- 172.21.15.67:0/1090365116 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc40b3070 con 0x55c3725a99e0 2024-03-28T16:06:17.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.902+0000 7f2fd7408740 1 -- 172.21.15.67:0/1090365116 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2fc002cae0 con 0x55c3725a99e0 2024-03-28T16:06:17.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.902+0000 7f2fb3400640 1 -- 172.21.15.67:0/1090365116 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f2fc40b4b40 con 0x55c3725a99e0 2024-03-28T16:06:17.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.902+0000 7f2fb3400640 1 -- 172.21.15.67:0/1090365116 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2fc409b850 con 0x55c3725a99e0 2024-03-28T16:06:17.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.902+0000 7f2fd7408740 10 client.15717 fetch_fsmap learned FSMap version 19 2024-03-28T16:06:17.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.902+0000 7f2fd7408740 1 -- 172.21.15.67:0/1090365116 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55c372615a40 con 0x55c3725a99e0 2024-03-28T16:06:17.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.903+0000 7f2fb3400640 1 -- 172.21.15.67:0/1090365116 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f2fc002cae0 con 0x55c3725a99e0 2024-03-28T16:06:17.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.903+0000 7f2fb3400640 1 -- 172.21.15.67:0/1090365116 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f2fc4081020 con 0x55c3725a99e0 2024-03-28T16:06:17.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.903+0000 7f2fd7408740 10 client.15717 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:06:17.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.903+0000 7f2fd7408740 10 client.15717 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:06:17.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.903+0000 7f2fd7408740 20 client.15717 populate_metadata read hostname 'smithi067' 2024-03-28T16:06:17.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.903+0000 7f2fd7408740 1 --2- 172.21.15.67:0/1090365116 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55c372616390 0x55c3726187b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:17.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.903+0000 7f2fd7408740 4 client.15717 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T16:06:17.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.903+0000 7f2fd7408740 1 -- 172.21.15.67:0/1090365116 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55c372510ae0 con 0x55c372616390 2024-03-28T16:06:17.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.904+0000 7f2fd1c00640 1 --2- 172.21.15.67:0/1090365116 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55c372616390 0x55c3726187b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:17.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.905+0000 7f2fd1c00640 1 --2- 172.21.15.67:0/1090365116 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55c372616390 0x55c3726187b0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=24fb9944f029462e server_cookie=bce11097c48f62df in_seq=0 out_seq=0 2024-03-28T16:06:17.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.905+0000 7f2fb3400640 10 client.15717 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:06:17.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.905+0000 7f2fb3400640 1 -- 172.21.15.67:0/1090365116 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55c372510ae0 con 0x55c372616390 2024-03-28T16:06:17.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.905+0000 7f2fb3400640 10 client.15717 handle_command_reply: tid=0 2024-03-28T16:06:17.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.905+0000 7f2fd7408740 1 client.15717 shutdown 2024-03-28T16:06:17.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.905+0000 7f2fd7408740 1 -- 172.21.15.67:0/1090365116 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c3725a99e0 msgr2=0x55c3725b4410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:17.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.905+0000 7f2fd7408740 1 --2- 172.21.15.67:0/1090365116 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c3725a99e0 0x55c3725b4410 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f2fc40049d0 tx=0x7f2fc4005660 comp rx=0 tx=0).stop 2024-03-28T16:06:17.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.905+0000 7f2fd7408740 1 -- 172.21.15.67:0/1090365116 shutdown_connections 2024-03-28T16:06:17.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.905+0000 7f2fd7408740 1 --2- 172.21.15.67:0/1090365116 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55c372616390 0x55c3726187b0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:17.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.905+0000 7f2fd7408740 1 --2- 172.21.15.67:0/1090365116 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c3725a99e0 0x55c3725b4410 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:17.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.905+0000 7f2fd7408740 1 --2- 172.21.15.67:0/1090365116 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c3725a4870 0x55c3725b3450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:17.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.905+0000 7f2fd7408740 1 --2- 172.21.15.67:0/1090365116 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c3725a3e80 0x55c3725b2f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:17.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.905+0000 7f2fb3400640 0 client.15717 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:06:17.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.905+0000 7f2fd7408740 1 -- 172.21.15.67:0/1090365116 >> 172.21.15.67:0/1090365116 conn(0x55c37256c200 msgr2=0x55c37259e200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:06:17.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.906+0000 7f2fd7408740 1 -- 172.21.15.67:0/1090365116 shutdown_connections 2024-03-28T16:06:17.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.906+0000 7f2fd7408740 1 -- 172.21.15.67:0/1090365116 wait complete. 2024-03-28T16:06:17.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.906+0000 7f2fd7408740 20 client.15717 trim_cache size 0 max 16384 2024-03-28T16:06:17.910 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-28T16:06:17.910 INFO:teuthology.orchestra.run.smithi067.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T16:06:17.910 INFO:teuthology.orchestra.run.smithi067.stdout: "scrubs": { 2024-03-28T16:06:17.910 INFO:teuthology.orchestra.run.smithi067.stdout: "cca7ef74-652c-43dc-b177-b6598dee6793": { 2024-03-28T16:06:17.910 INFO:teuthology.orchestra.run.smithi067.stdout: "path": "/", 2024-03-28T16:06:17.910 INFO:teuthology.orchestra.run.smithi067.stdout: "tag": "cca7ef74-652c-43dc-b177-b6598dee6793", 2024-03-28T16:06:17.910 INFO:teuthology.orchestra.run.smithi067.stdout: "options": "recursive,force" 2024-03-28T16:06:17.910 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:06:17.910 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:06:17.910 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-28T16:06:17.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.907+0000 7f2fd4600640 1 -- 172.21.15.67:0/2640410851 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f2fb40a2880 msgr2=0x7f2fb40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:17.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.907+0000 7f2fd4600640 1 --2- 172.21.15.67:0/2640410851 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f2fb40a2880 0x7f2fb40a4d40 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x55c3725a50d0 tx=0x7f2fb8005f90 comp rx=0 tx=0).stop 2024-03-28T16:06:17.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.907+0000 7f2fd4600640 1 -- 172.21.15.67:0/2640410851 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fcc105490 msgr2=0x7f2fcc1a1120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:17.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.907+0000 7f2fd4600640 1 --2- 172.21.15.67:0/2640410851 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fcc105490 0x7f2fcc1a1120 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f2fc400eb10 tx=0x7f2fc400c2d0 comp rx=0 tx=0).stop 2024-03-28T16:06:17.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.907+0000 7f2fd4600640 1 -- 172.21.15.67:0/2640410851 shutdown_connections 2024-03-28T16:06:17.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.907+0000 7f2fd4600640 1 --2- 172.21.15.67:0/2640410851 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f2fb40a2880 0x7f2fb40a4d40 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:17.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.907+0000 7f2fd4600640 1 --2- 172.21.15.67:0/2640410851 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fcc105490 0x7f2fcc1a1120 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:17.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.907+0000 7f2fd4600640 1 --2- 172.21.15.67:0/2640410851 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2fcc104ad0 0x7f2fcc19a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:17.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.907+0000 7f2fd4600640 1 --2- 172.21.15.67:0/2640410851 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2fcc1038e0 0x7f2fcc199b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:17.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.907+0000 7f2fd4600640 1 -- 172.21.15.67:0/2640410851 >> 172.21.15.67:0/2640410851 conn(0x7f2fcc0fee50 msgr2=0x7f2fcc107ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:06:17.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.907+0000 7f2fd4600640 1 -- 172.21.15.67:0/2640410851 shutdown_connections 2024-03-28T16:06:17.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:17.907+0000 7f2fd4600640 1 -- 172.21.15.67:0/2640410851 wait complete. 2024-03-28T16:06:17.930 INFO:tasks.cephfs.filesystem:scrub status for tag:cca7ef74-652c-43dc-b177-b6598dee6793 - {'path': '/', 'tag': 'cca7ef74-652c-43dc-b177-b6598dee6793', 'options': 'recursive,force'} 2024-03-28T16:06:19.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:06:18 smithi150 ceph-mon[34825]: pgmap v521: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 20 MiB/s wr, 6 op/s 2024-03-28T16:06:19.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:06:18 smithi150 ceph-mon[34825]: scrub queued for path: / 2024-03-28T16:06:19.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:06:18 smithi150 ceph-mon[34825]: scrub summary: idle+waiting paths [/] 2024-03-28T16:06:19.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:06:18 smithi150 ceph-mon[34825]: scrub summary: active paths [/] 2024-03-28T16:06:19.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:06:18 smithi150 ceph-mon[34825]: scrub summary: idle+waiting paths [/] 2024-03-28T16:06:19.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:06:18 smithi067 ceph-mon[31707]: pgmap v521: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 20 MiB/s wr, 6 op/s 2024-03-28T16:06:19.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:06:18 smithi067 ceph-mon[31707]: scrub queued for path: / 2024-03-28T16:06:19.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:06:18 smithi067 ceph-mon[31707]: scrub summary: idle+waiting paths [/] 2024-03-28T16:06:19.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:06:18 smithi067 ceph-mon[31707]: scrub summary: active paths [/] 2024-03-28T16:06:19.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:06:18 smithi067 ceph-mon[31707]: scrub summary: idle+waiting paths [/] 2024-03-28T16:06:19.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:06:18 smithi183 ceph-mon[35580]: pgmap v521: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 20 MiB/s wr, 6 op/s 2024-03-28T16:06:19.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:06:18 smithi183 ceph-mon[35580]: scrub queued for path: / 2024-03-28T16:06:19.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:06:18 smithi183 ceph-mon[35580]: scrub summary: idle+waiting paths [/] 2024-03-28T16:06:19.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:06:18 smithi183 ceph-mon[35580]: scrub summary: active paths [/] 2024-03-28T16:06:19.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:06:18 smithi183 ceph-mon[35580]: scrub summary: idle+waiting paths [/] 2024-03-28T16:06:21.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:06:20 smithi150 ceph-mon[34825]: pgmap v522: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 23 MiB/s wr, 7 op/s 2024-03-28T16:06:21.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:06:20 smithi067 ceph-mon[31707]: pgmap v522: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 23 MiB/s wr, 7 op/s 2024-03-28T16:06:21.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:06:20 smithi183 ceph-mon[35580]: pgmap v522: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 23 MiB/s wr, 7 op/s 2024-03-28T16:06:23.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:06:22 smithi150 ceph-mon[34825]: pgmap v523: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 26 MiB/s wr, 9 op/s 2024-03-28T16:06:23.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:06:22 smithi150 ceph-mon[34825]: scrub summary: idle 2024-03-28T16:06:23.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:06:22 smithi067 ceph-mon[31707]: pgmap v523: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 26 MiB/s wr, 9 op/s 2024-03-28T16:06:23.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:06:22 smithi067 ceph-mon[31707]: scrub summary: idle 2024-03-28T16:06:23.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:06:22 smithi183 ceph-mon[35580]: pgmap v523: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 26 MiB/s wr, 9 op/s 2024-03-28T16:06:23.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:06:22 smithi183 ceph-mon[35580]: scrub summary: idle 2024-03-28T16:06:25.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:06:24 smithi150 ceph-mon[34825]: pgmap v524: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 27 MiB/s wr, 8 op/s 2024-03-28T16:06:25.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:06:24 smithi067 ceph-mon[31707]: pgmap v524: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 27 MiB/s wr, 8 op/s 2024-03-28T16:06:25.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:06:24 smithi183 ceph-mon[35580]: pgmap v524: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 27 MiB/s wr, 8 op/s 2024-03-28T16:06:26.902 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:06:26 smithi183 ceph-mon[35580]: pgmap v525: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 27 MiB/s wr, 9 op/s 2024-03-28T16:06:27.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:06:26 smithi150 ceph-mon[34825]: pgmap v525: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 27 MiB/s wr, 9 op/s 2024-03-28T16:06:27.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:06:26 smithi067 ceph-mon[31707]: pgmap v525: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 27 MiB/s wr, 9 op/s 2024-03-28T16:06:29.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:06:28 smithi150 ceph-mon[34825]: pgmap v526: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 24 MiB/s wr, 7 op/s 2024-03-28T16:06:29.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:06:28 smithi067 ceph-mon[31707]: pgmap v526: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 24 MiB/s wr, 7 op/s 2024-03-28T16:06:29.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:06:28 smithi183 ceph-mon[35580]: pgmap v526: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 24 MiB/s wr, 7 op/s 2024-03-28T16:06:31.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:06:30 smithi150 ceph-mon[34825]: pgmap v527: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 25 MiB/s wr, 8 op/s 2024-03-28T16:06:31.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:06:30 smithi067 ceph-mon[31707]: pgmap v527: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 25 MiB/s wr, 8 op/s 2024-03-28T16:06:31.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:06:30 smithi183 ceph-mon[35580]: pgmap v527: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 25 MiB/s wr, 8 op/s 2024-03-28T16:06:33.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:06:32 smithi150 ceph-mon[34825]: pgmap v528: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 24 MiB/s wr, 8 op/s 2024-03-28T16:06:33.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:06:32 smithi067 ceph-mon[31707]: pgmap v528: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 24 MiB/s wr, 8 op/s 2024-03-28T16:06:33.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:06:32 smithi183 ceph-mon[35580]: pgmap v528: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 24 MiB/s wr, 8 op/s 2024-03-28T16:06:35.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:06:34 smithi150 ceph-mon[34825]: pgmap v529: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 22 MiB/s wr, 7 op/s 2024-03-28T16:06:35.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:06:34 smithi067 ceph-mon[31707]: pgmap v529: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 22 MiB/s wr, 7 op/s 2024-03-28T16:06:35.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:06:34 smithi183 ceph-mon[35580]: pgmap v529: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 22 MiB/s wr, 7 op/s 2024-03-28T16:06:37.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:06:36 smithi150 ceph-mon[34825]: pgmap v530: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 19 MiB/s wr, 7 op/s 2024-03-28T16:06:37.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:06:36 smithi067 ceph-mon[31707]: pgmap v530: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 19 MiB/s wr, 7 op/s 2024-03-28T16:06:37.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:06:36 smithi183 ceph-mon[35580]: pgmap v530: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 19 MiB/s wr, 7 op/s 2024-03-28T16:06:39.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:06:38 smithi150 ceph-mon[34825]: pgmap v531: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 18 MiB/s wr, 6 op/s 2024-03-28T16:06:39.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:06:38 smithi067 ceph-mon[31707]: pgmap v531: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 18 MiB/s wr, 6 op/s 2024-03-28T16:06:39.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:06:38 smithi183 ceph-mon[35580]: pgmap v531: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 18 MiB/s wr, 6 op/s 2024-03-28T16:06:41.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:06:40 smithi150 ceph-mon[34825]: pgmap v532: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 21 MiB/s wr, 6 op/s 2024-03-28T16:06:41.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:06:40 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T16:06:41.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:06:40 smithi067 ceph-mon[31707]: pgmap v532: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 21 MiB/s wr, 6 op/s 2024-03-28T16:06:41.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:06:40 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T16:06:41.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:06:40 smithi183 ceph-mon[35580]: pgmap v532: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 21 MiB/s wr, 6 op/s 2024-03-28T16:06:41.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:06:40 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T16:06:41.350 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-03-28T16:06:41.350 INFO:tasks.workunit.client.0.smithi067.stdout: Children see throughput for 1 rewriters = 23482.05 kB/sec 2024-03-28T16:06:41.350 INFO:tasks.workunit.client.0.smithi067.stdout: Parent sees throughput for 1 rewriters = 23481.48 kB/sec 2024-03-28T16:06:41.351 INFO:tasks.workunit.client.0.smithi067.stdout: Min throughput per process = 23482.05 kB/sec 2024-03-28T16:06:41.351 INFO:tasks.workunit.client.0.smithi067.stdout: Max throughput per process = 23482.05 kB/sec 2024-03-28T16:06:41.351 INFO:tasks.workunit.client.0.smithi067.stdout: Avg throughput per process = 23482.05 kB/sec 2024-03-28T16:06:41.351 INFO:tasks.workunit.client.0.smithi067.stdout: Min xfer = 1048576.00 kB 2024-03-28T16:06:42.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:06:41 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:06:42.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:06:41 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T16:06:42.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:06:41 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:06:42.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:06:41 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:06:42.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:06:41 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T16:06:42.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:06:41 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:06:42.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:06:41 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:06:42.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:06:41 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T16:06:42.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:06:41 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:06:43.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:06:42 smithi150 ceph-mon[34825]: pgmap v533: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 22 MiB/s wr, 7 op/s 2024-03-28T16:06:43.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:06:42 smithi067 ceph-mon[31707]: pgmap v533: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 22 MiB/s wr, 7 op/s 2024-03-28T16:06:43.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:06:42 smithi183 ceph-mon[35580]: pgmap v533: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 22 MiB/s wr, 7 op/s 2024-03-28T16:06:44.166 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-03-28T16:06:44.166 INFO:tasks.workunit.client.0.smithi067.stdout: Children see throughput for 1 readers = 4686122.00 kB/sec 2024-03-28T16:06:44.166 INFO:tasks.workunit.client.0.smithi067.stdout: Parent sees throughput for 1 readers = 4674010.34 kB/sec 2024-03-28T16:06:44.166 INFO:tasks.workunit.client.0.smithi067.stdout: Min throughput per process = 4686122.00 kB/sec 2024-03-28T16:06:44.166 INFO:tasks.workunit.client.0.smithi067.stdout: Max throughput per process = 4686122.00 kB/sec 2024-03-28T16:06:44.166 INFO:tasks.workunit.client.0.smithi067.stdout: Avg throughput per process = 4686122.00 kB/sec 2024-03-28T16:06:44.166 INFO:tasks.workunit.client.0.smithi067.stdout: Min xfer = 1048576.00 kB 2024-03-28T16:06:45.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:06:44 smithi150 ceph-mon[34825]: pgmap v534: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 22 MiB/s wr, 7 op/s 2024-03-28T16:06:45.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:06:44 smithi067 ceph-mon[31707]: pgmap v534: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 22 MiB/s wr, 7 op/s 2024-03-28T16:06:45.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:06:44 smithi183 ceph-mon[35580]: pgmap v534: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 22 MiB/s wr, 7 op/s 2024-03-28T16:06:46.581 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-03-28T16:06:46.581 INFO:tasks.workunit.client.0.smithi067.stdout: Children see throughput for 1 re-readers = 3336980.75 kB/sec 2024-03-28T16:06:46.581 INFO:tasks.workunit.client.0.smithi067.stdout: Parent sees throughput for 1 re-readers = 3329679.53 kB/sec 2024-03-28T16:06:46.581 INFO:tasks.workunit.client.0.smithi067.stdout: Min throughput per process = 3336980.75 kB/sec 2024-03-28T16:06:46.581 INFO:tasks.workunit.client.0.smithi067.stdout: Max throughput per process = 3336980.75 kB/sec 2024-03-28T16:06:46.581 INFO:tasks.workunit.client.0.smithi067.stdout: Avg throughput per process = 3336980.75 kB/sec 2024-03-28T16:06:46.581 INFO:tasks.workunit.client.0.smithi067.stdout: Min xfer = 1048576.00 kB 2024-03-28T16:06:47.054 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:06:46 smithi150 ceph-mon[34825]: pgmap v535: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 21 MiB/s wr, 7 op/s 2024-03-28T16:06:47.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:06:46 smithi067 ceph-mon[31707]: pgmap v535: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 21 MiB/s wr, 7 op/s 2024-03-28T16:06:47.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:06:46 smithi183 ceph-mon[35580]: pgmap v535: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 21 MiB/s wr, 7 op/s 2024-03-28T16:06:47.930 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T16:06:47.930 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T16:06:48.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.168+0000 7f0d39800640 1 -- 172.21.15.67:0/1415826602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d341037c0 msgr2=0x7f0d34103be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:48.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.168+0000 7f0d39800640 1 --2- 172.21.15.67:0/1415826602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d341037c0 0x7f0d34103be0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f0d24009900 tx=0x7f0d2402bc30 comp rx=0 tx=0).stop 2024-03-28T16:06:48.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.168+0000 7f0d39800640 1 -- 172.21.15.67:0/1415826602 shutdown_connections 2024-03-28T16:06:48.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.168+0000 7f0d39800640 1 --2- 172.21.15.67:0/1415826602 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0d34105370 0x7f0d3410bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:48.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.168+0000 7f0d39800640 1 --2- 172.21.15.67:0/1415826602 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d341049b0 0x7f0d34104e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:48.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.168+0000 7f0d39800640 1 --2- 172.21.15.67:0/1415826602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d341037c0 0x7f0d34103be0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:48.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.168+0000 7f0d39800640 1 -- 172.21.15.67:0/1415826602 >> 172.21.15.67:0/1415826602 conn(0x7f0d340fed50 msgr2=0x7f0d34101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:06:48.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.168+0000 7f0d39800640 1 -- 172.21.15.67:0/1415826602 shutdown_connections 2024-03-28T16:06:48.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.169+0000 7f0d39800640 1 -- 172.21.15.67:0/1415826602 wait complete. 2024-03-28T16:06:48.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.170+0000 7f0d39800640 1 Processor -- start 2024-03-28T16:06:48.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.170+0000 7f0d39800640 1 -- start start 2024-03-28T16:06:48.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.170+0000 7f0d39800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0d341037c0 0x7f0d341999b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:48.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.170+0000 7f0d39800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d341049b0 0x7f0d34199ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:48.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.170+0000 7f0d39800640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d34105370 0x7f0d341a0f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:48.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.170+0000 7f0d39800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d3410f340 con 0x7f0d341049b0 2024-03-28T16:06:48.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.170+0000 7f0d39800640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d3410f1c0 con 0x7f0d34105370 2024-03-28T16:06:48.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.170+0000 7f0d39800640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d3410f4c0 con 0x7f0d341037c0 2024-03-28T16:06:48.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.170+0000 7f0d2be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d341049b0 0x7f0d34199ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:48.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.170+0000 7f0d2be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d341049b0 0x7f0d34199ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45728/0 (socket says 172.21.15.67:45728) 2024-03-28T16:06:48.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.170+0000 7f0d2be00640 1 -- 172.21.15.67:0/2620909002 learned_addr learned my addr 172.21.15.67:0/2620909002 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:06:48.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.171+0000 7f0d33e00640 1 --2- 172.21.15.67:0/2620909002 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0d341037c0 0x7f0d341999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:48.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.171+0000 7f0d38e00640 1 --2- 172.21.15.67:0/2620909002 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d34105370 0x7f0d341a0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:48.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.171+0000 7f0d2be00640 1 -- 172.21.15.67:0/2620909002 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0d341037c0 msgr2=0x7f0d341999b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:48.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.171+0000 7f0d2be00640 1 --2- 172.21.15.67:0/2620909002 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0d341037c0 0x7f0d341999b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:48.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.171+0000 7f0d2be00640 1 -- 172.21.15.67:0/2620909002 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d34105370 msgr2=0x7f0d341a0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:48.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.171+0000 7f0d2be00640 1 --2- 172.21.15.67:0/2620909002 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d34105370 0x7f0d341a0f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:48.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.171+0000 7f0d2be00640 1 -- 172.21.15.67:0/2620909002 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d341a15d0 con 0x7f0d341049b0 2024-03-28T16:06:48.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.171+0000 7f0d38e00640 1 --2- 172.21.15.67:0/2620909002 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d34105370 0x7f0d341a0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:06:48.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.171+0000 7f0d2be00640 1 --2- 172.21.15.67:0/2620909002 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d341049b0 0x7f0d34199ef0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f0d1c00d950 tx=0x7f0d1c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:06:48.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.171+0000 7f0d31600640 1 -- 172.21.15.67:0/2620909002 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d1c017070 con 0x7f0d341049b0 2024-03-28T16:06:48.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.172+0000 7f0d31600640 1 -- 172.21.15.67:0/2620909002 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0d1c009c30 con 0x7f0d341049b0 2024-03-28T16:06:48.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.172+0000 7f0d39800640 1 -- 172.21.15.67:0/2620909002 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d341a18c0 con 0x7f0d341049b0 2024-03-28T16:06:48.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.172+0000 7f0d31600640 1 -- 172.21.15.67:0/2620909002 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d1c00f620 con 0x7f0d341049b0 2024-03-28T16:06:48.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.172+0000 7f0d39800640 1 -- 172.21.15.67:0/2620909002 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d341a1e10 con 0x7f0d341049b0 2024-03-28T16:06:48.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.175+0000 7f0d3db8a740 1 Processor -- start 2024-03-28T16:06:48.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.175+0000 7f0d31600640 1 -- 172.21.15.67:0/2620909002 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f0d1c0109a0 con 0x7f0d341049b0 2024-03-28T16:06:48.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.175+0000 7f0d3db8a740 1 -- start start 2024-03-28T16:06:48.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.175+0000 7f0d3db8a740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555d37148050 0x555d37148470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:48.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.175+0000 7f0d3db8a740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555d37148a40 0x555d3714d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:48.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.175+0000 7f0d3db8a740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d3714dbb0 0x555d37150040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:48.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.175+0000 7f0d3db8a740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d370ad250 con 0x555d37148050 2024-03-28T16:06:48.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.175+0000 7f0d3db8a740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d37055a30 con 0x555d3714dbb0 2024-03-28T16:06:48.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.175+0000 7f0d3db8a740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d370a92f0 con 0x555d37148a40 2024-03-28T16:06:48.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.176+0000 7f0d38e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555d37148050 0x555d37148470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:48.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.176+0000 7f0d38e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555d37148050 0x555d37148470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45738/0 (socket says 172.21.15.67:45738) 2024-03-28T16:06:48.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.176+0000 7f0d38e00640 1 -- 172.21.15.67:0/3265684731 learned_addr learned my addr 172.21.15.67:0/3265684731 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:06:48.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.176+0000 7f0d2be00640 1 --2- 172.21.15.67:0/3265684731 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d3714dbb0 0x555d37150040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:48.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.176+0000 7f0d33e00640 1 --2- 172.21.15.67:0/3265684731 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555d37148a40 0x555d3714d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:48.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.176+0000 7f0d38e00640 1 -- 172.21.15.67:0/3265684731 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555d37148a40 msgr2=0x555d3714d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:06:48.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.176+0000 7f0d38e00640 1 --2- 172.21.15.67:0/3265684731 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555d37148a40 0x555d3714d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:48.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.176+0000 7f0d38e00640 1 -- 172.21.15.67:0/3265684731 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d3714dbb0 msgr2=0x555d37150040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:06:48.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.176+0000 7f0d38e00640 1 --2- 172.21.15.67:0/3265684731 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d3714dbb0 0x555d37150040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:48.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.176+0000 7f0d38e00640 1 -- 172.21.15.67:0/3265684731 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555d37040690 con 0x555d37148050 2024-03-28T16:06:48.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.176+0000 7f0d31600640 1 --2- 172.21.15.67:0/2620909002 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f0d180a2800 0x7f0d180a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:48.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.177+0000 7f0d31600640 1 -- 172.21.15.67:0/2620909002 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f0d1c0ce7b0 con 0x7f0d341049b0 2024-03-28T16:06:48.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.177+0000 7f0d38e00640 1 --2- 172.21.15.67:0/3265684731 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555d37148050 0x555d37148470 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c009ea0 tx=0x7f0d2c009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b8a09ac98dab0e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:06:48.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.177+0000 7f0d33e00640 1 --2- 172.21.15.67:0/2620909002 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f0d180a2800 0x7f0d180a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:48.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.177+0000 7f0d29600640 1 -- 172.21.15.67:0/3265684731 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d2c004d20 con 0x555d37148050 2024-03-28T16:06:48.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.177+0000 7f0d29600640 1 -- 172.21.15.67:0/3265684731 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0d2c0054d0 con 0x555d37148050 2024-03-28T16:06:48.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.177+0000 7f0d29600640 1 -- 172.21.15.67:0/3265684731 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d2c0118a0 con 0x555d37148050 2024-03-28T16:06:48.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.178+0000 7f0d3db8a740 1 -- 172.21.15.67:0/3265684731 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555d37148050 msgr2=0x555d37148470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:48.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.178+0000 7f0d3db8a740 1 --2- 172.21.15.67:0/3265684731 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555d37148050 0x555d37148470 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c009ea0 tx=0x7f0d2c009f90 comp rx=0 tx=0).stop 2024-03-28T16:06:48.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.178+0000 7f0d33e00640 1 --2- 172.21.15.67:0/2620909002 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f0d180a2800 0x7f0d180a4cc0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f0d2402c240 tx=0x7f0d24005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:06:48.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.178+0000 7f0d3db8a740 1 -- 172.21.15.67:0/3265684731 shutdown_connections 2024-03-28T16:06:48.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.178+0000 7f0d3db8a740 1 --2- 172.21.15.67:0/3265684731 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d3714dbb0 0x555d37150040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:48.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.178+0000 7f0d3db8a740 1 --2- 172.21.15.67:0/3265684731 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555d37148a40 0x555d3714d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:48.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.178+0000 7f0d3db8a740 1 --2- 172.21.15.67:0/3265684731 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555d37148050 0x555d37148470 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:48.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.178+0000 7f0d3db8a740 1 -- 172.21.15.67:0/3265684731 >> 172.21.15.67:0/3265684731 conn(0x555d37110200 msgr2=0x555d371439c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:06:48.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.178+0000 7f0d3db8a740 1 -- 172.21.15.67:0/3265684731 shutdown_connections 2024-03-28T16:06:48.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.178+0000 7f0d3db8a740 1 -- 172.21.15.67:0/3265684731 wait complete. 2024-03-28T16:06:48.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.179+0000 7f0d3db8a740 1 -- start start 2024-03-28T16:06:48.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.179+0000 7f0d3db8a740 1 Processor -- start 2024-03-28T16:06:48.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.179+0000 7f0d3db8a740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d37148050 0x555d37152050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:48.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.179+0000 7f0d3db8a740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555d37148a40 0x555d37152590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:48.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.179+0000 7f0d3db8a740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555d3714dbb0 0x555d371595d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:48.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.179+0000 7f0d3db8a740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d37150b60 con 0x555d37148a40 2024-03-28T16:06:48.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.179+0000 7f0d3db8a740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d37068e80 con 0x555d37148050 2024-03-28T16:06:48.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.179+0000 7f0d3db8a740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d37150ce0 con 0x555d3714dbb0 2024-03-28T16:06:48.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.179+0000 7f0d3db8a740 10 client.0 _ll_register_callbacks cb 0x555d3707b130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:06:48.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.180+0000 7f0d2be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555d37148a40 0x555d37152590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:48.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.180+0000 7f0d2be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555d37148a40 0x555d37152590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45744/0 (socket says 172.21.15.67:45744) 2024-03-28T16:06:48.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.180+0000 7f0d2be00640 1 -- 172.21.15.67:0/92335077 learned_addr learned my addr 172.21.15.67:0/92335077 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:06:48.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.180+0000 7f0d38e00640 1 --2- 172.21.15.67:0/92335077 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555d3714dbb0 0x555d371595d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:48.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.180+0000 7f0d33e00640 1 --2- 172.21.15.67:0/92335077 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d37148050 0x555d37152050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:48.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.180+0000 7f0d2be00640 1 -- 172.21.15.67:0/92335077 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555d3714dbb0 msgr2=0x555d371595d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:06:48.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.180+0000 7f0d2be00640 1 --2- 172.21.15.67:0/92335077 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555d3714dbb0 0x555d371595d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:48.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.180+0000 7f0d2be00640 1 -- 172.21.15.67:0/92335077 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d37148050 msgr2=0x555d37152050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:06:48.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.180+0000 7f0d2be00640 1 --2- 172.21.15.67:0/92335077 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d37148050 0x555d37152050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:48.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.180+0000 7f0d2be00640 1 -- 172.21.15.67:0/92335077 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555d36e8b7d0 con 0x555d37148a40 2024-03-28T16:06:48.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.180+0000 7f0d33e00640 1 --2- 172.21.15.67:0/92335077 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d37148050 0x555d37152050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:06:48.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.180+0000 7f0d2be00640 1 --2- 172.21.15.67:0/92335077 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555d37148a40 0x555d37152590 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f0d1c097560 tx=0x7f0d1c09ab20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18a549f4349fd91c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:06:48.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.180+0000 7f0d22000640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:06:48.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.181+0000 7f0d22000640 1 -- 172.21.15.67:0/92335077 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d1c020050 con 0x555d37148a40 2024-03-28T16:06:48.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.181+0000 7f0d22000640 1 -- 172.21.15.67:0/92335077 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f0d1c0ceba0 con 0x555d37148a40 2024-03-28T16:06:48.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.181+0000 7f0d3db8a740 1 -- 172.21.15.67:0/92335077 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0d2c0054d0 con 0x555d37148a40 2024-03-28T16:06:48.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.181+0000 7f0d22000640 1 -- 172.21.15.67:0/92335077 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0d1c09b100 con 0x555d37148a40 2024-03-28T16:06:48.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.181+0000 7f0d3db8a740 10 client.15729 fetch_fsmap learned FSMap version 19 2024-03-28T16:06:48.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.181+0000 7f0d3db8a740 1 -- 172.21.15.67:0/92335077 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x555d36dc13f0 con 0x555d37148a40 2024-03-28T16:06:48.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.181+0000 7f0d22000640 1 -- 172.21.15.67:0/92335077 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0d2c0054d0 con 0x555d37148a40 2024-03-28T16:06:48.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.181+0000 7f0d22000640 1 -- 172.21.15.67:0/92335077 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f0d1c096e50 con 0x555d37148a40 2024-03-28T16:06:48.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.181+0000 7f0d3db8a740 10 client.15729 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:06:48.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.181+0000 7f0d3db8a740 10 client.15729 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:06:48.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.182+0000 7f0d3db8a740 20 client.15729 populate_metadata read hostname 'smithi067' 2024-03-28T16:06:48.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.182+0000 7f0d3db8a740 1 --2- 172.21.15.67:0/92335077 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x555d3715dd70 0x555d371601c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:48.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.182+0000 7f0d3db8a740 4 client.15729 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:06:48.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.182+0000 7f0d3db8a740 1 -- 172.21.15.67:0/92335077 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555d370ba790 con 0x555d3715dd70 2024-03-28T16:06:48.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.182+0000 7f0d33e00640 1 --2- 172.21.15.67:0/92335077 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x555d3715dd70 0x555d371601c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:48.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.183+0000 7f0d33e00640 1 --2- 172.21.15.67:0/92335077 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x555d3715dd70 0x555d371601c0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a89ad2ddf6c5f3c server_cookie=2bb9866c93748f in_seq=0 out_seq=0 2024-03-28T16:06:48.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.183+0000 7f0d22000640 10 client.15729 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:06:48.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.185+0000 7f0d22000640 1 -- 172.21.15.67:0/92335077 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x555d370ba790 con 0x555d3715dd70 2024-03-28T16:06:48.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.185+0000 7f0d22000640 10 client.15729 handle_command_reply: tid=0 2024-03-28T16:06:48.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.185+0000 7f0d3db8a740 1 client.15729 shutdown 2024-03-28T16:06:48.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.185+0000 7f0d3db8a740 1 -- 172.21.15.67:0/92335077 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555d37148a40 msgr2=0x555d37152590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:48.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.185+0000 7f0d3db8a740 1 --2- 172.21.15.67:0/92335077 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555d37148a40 0x555d37152590 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f0d1c097560 tx=0x7f0d1c09ab20 comp rx=0 tx=0).stop 2024-03-28T16:06:48.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.185+0000 7f0d3db8a740 1 -- 172.21.15.67:0/92335077 shutdown_connections 2024-03-28T16:06:48.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.185+0000 7f0d3db8a740 1 --2- 172.21.15.67:0/92335077 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x555d3715dd70 0x555d371601c0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:48.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.185+0000 7f0d3db8a740 1 --2- 172.21.15.67:0/92335077 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555d3714dbb0 0x555d371595d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:48.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.185+0000 7f0d3db8a740 1 --2- 172.21.15.67:0/92335077 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555d37148a40 0x555d37152590 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:48.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.185+0000 7f0d22000640 0 client.15729 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:06:48.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.185+0000 7f0d3db8a740 1 --2- 172.21.15.67:0/92335077 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d37148050 0x555d37152050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:48.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.185+0000 7f0d3db8a740 1 -- 172.21.15.67:0/92335077 >> 172.21.15.67:0/92335077 conn(0x555d37110200 msgr2=0x555d37142bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:06:48.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.186+0000 7f0d3db8a740 1 -- 172.21.15.67:0/92335077 shutdown_connections 2024-03-28T16:06:48.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.186+0000 7f0d3db8a740 1 -- 172.21.15.67:0/92335077 wait complete. 2024-03-28T16:06:48.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.186+0000 7f0d3db8a740 20 client.15729 trim_cache size 0 max 16384 2024-03-28T16:06:48.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.203+0000 7f0d3db8a740 1 Processor -- start 2024-03-28T16:06:48.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.203+0000 7f0d3db8a740 1 -- start start 2024-03-28T16:06:48.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.203+0000 7f0d3db8a740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555d37148050 0x555d37142a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:48.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.203+0000 7f0d3db8a740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d37148a40 0x555d37142f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:48.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.203+0000 7f0d3db8a740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555d3714dbb0 0x555d37159590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:48.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.204+0000 7f0d3db8a740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d341a8a10 con 0x555d37148050 2024-03-28T16:06:48.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.204+0000 7f0d3db8a740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d341aafd0 con 0x555d37148a40 2024-03-28T16:06:48.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.204+0000 7f0d3db8a740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d341a9010 con 0x555d3714dbb0 2024-03-28T16:06:48.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.204+0000 7f0d33e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555d37148050 0x555d37142a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:48.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.204+0000 7f0d33e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555d37148050 0x555d37142a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45758/0 (socket says 172.21.15.67:45758) 2024-03-28T16:06:48.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.204+0000 7f0d33e00640 1 -- 172.21.15.67:0/204607002 learned_addr learned my addr 172.21.15.67:0/204607002 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:06:48.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.204+0000 7f0d2be00640 1 --2- 172.21.15.67:0/204607002 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d37148a40 0x555d37142f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:48.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.204+0000 7f0d33e00640 1 -- 172.21.15.67:0/204607002 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555d3714dbb0 msgr2=0x555d37159590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:06:48.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.204+0000 7f0d38e00640 1 --2- 172.21.15.67:0/204607002 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555d3714dbb0 0x555d37159590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:48.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.204+0000 7f0d33e00640 1 --2- 172.21.15.67:0/204607002 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555d3714dbb0 0x555d37159590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:48.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.204+0000 7f0d33e00640 1 -- 172.21.15.67:0/204607002 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d37148a40 msgr2=0x555d37142f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:06:48.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.204+0000 7f0d33e00640 1 --2- 172.21.15.67:0/204607002 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d37148a40 0x555d37142f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:48.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.204+0000 7f0d33e00640 1 -- 172.21.15.67:0/204607002 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555d36d810f0 con 0x555d37148050 2024-03-28T16:06:48.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.204+0000 7f0d38e00640 1 --2- 172.21.15.67:0/204607002 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555d3714dbb0 0x555d37159590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:06:48.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.205+0000 7f0d33e00640 1 --2- 172.21.15.67:0/204607002 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555d37148050 0x555d37142a40 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f0d2402c840 tx=0x7f0d24043000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f10248a555d979b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:06:48.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.205+0000 7f0d28c00640 1 -- 172.21.15.67:0/204607002 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d24005d50 con 0x555d37148050 2024-03-28T16:06:48.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.205+0000 7f0d28c00640 1 -- 172.21.15.67:0/204607002 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0d24057070 con 0x555d37148050 2024-03-28T16:06:48.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.205+0000 7f0d28c00640 1 -- 172.21.15.67:0/204607002 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d24051310 con 0x555d37148050 2024-03-28T16:06:48.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.206+0000 7f0d3db8a740 1 -- 172.21.15.67:0/204607002 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555d37148050 msgr2=0x555d37142a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:48.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.206+0000 7f0d3db8a740 1 --2- 172.21.15.67:0/204607002 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555d37148050 0x555d37142a40 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f0d2402c840 tx=0x7f0d24043000 comp rx=0 tx=0).stop 2024-03-28T16:06:48.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.206+0000 7f0d3db8a740 1 -- 172.21.15.67:0/204607002 shutdown_connections 2024-03-28T16:06:48.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.206+0000 7f0d3db8a740 1 --2- 172.21.15.67:0/204607002 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555d3714dbb0 0x555d37159590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:48.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.206+0000 7f0d3db8a740 1 --2- 172.21.15.67:0/204607002 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d37148a40 0x555d37142f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:48.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.206+0000 7f0d3db8a740 1 --2- 172.21.15.67:0/204607002 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555d37148050 0x555d37142a40 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:48.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.206+0000 7f0d3db8a740 1 -- 172.21.15.67:0/204607002 >> 172.21.15.67:0/204607002 conn(0x555d37110200 msgr2=0x555d37142330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:06:48.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.206+0000 7f0d3db8a740 1 -- 172.21.15.67:0/204607002 shutdown_connections 2024-03-28T16:06:48.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.206+0000 7f0d3db8a740 1 -- 172.21.15.67:0/204607002 wait complete. 2024-03-28T16:06:48.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.206+0000 7f0d3db8a740 1 -- start start 2024-03-28T16:06:48.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.206+0000 7f0d3db8a740 1 Processor -- start 2024-03-28T16:06:48.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.207+0000 7f0d3db8a740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d37148050 0x555d37158ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:48.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.207+0000 7f0d3db8a740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555d37148a40 0x555d37159410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:48.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.207+0000 7f0d3db8a740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555d3714dbb0 0x555d371574e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:48.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.207+0000 7f0d3db8a740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d341a9650 con 0x555d37148a40 2024-03-28T16:06:48.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.207+0000 7f0d3db8a740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d341aa970 con 0x555d37148050 2024-03-28T16:06:48.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.207+0000 7f0d3db8a740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d341a9cb0 con 0x555d3714dbb0 2024-03-28T16:06:48.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.207+0000 7f0d3db8a740 10 client.0 _ll_register_callbacks cb 0x7f0d1c09afa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:06:48.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.207+0000 7f0d2be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555d37148a40 0x555d37159410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:48.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.207+0000 7f0d2be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555d37148a40 0x555d37159410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45760/0 (socket says 172.21.15.67:45760) 2024-03-28T16:06:48.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.207+0000 7f0d2be00640 1 -- 172.21.15.67:0/1914574447 learned_addr learned my addr 172.21.15.67:0/1914574447 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:06:48.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.207+0000 7f0d33e00640 1 --2- 172.21.15.67:0/1914574447 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d37148050 0x555d37158ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:48.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.207+0000 7f0d38e00640 1 --2- 172.21.15.67:0/1914574447 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555d3714dbb0 0x555d371574e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:48.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.207+0000 7f0d2be00640 1 -- 172.21.15.67:0/1914574447 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555d3714dbb0 msgr2=0x555d371574e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:06:48.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.207+0000 7f0d2be00640 1 --2- 172.21.15.67:0/1914574447 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555d3714dbb0 0x555d371574e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:48.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.207+0000 7f0d2be00640 1 -- 172.21.15.67:0/1914574447 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d37148050 msgr2=0x555d37158ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:06:48.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.207+0000 7f0d2be00640 1 --2- 172.21.15.67:0/1914574447 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d37148050 0x555d37158ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:48.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.208+0000 7f0d2be00640 1 -- 172.21.15.67:0/1914574447 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555d37157bd0 con 0x555d37148a40 2024-03-28T16:06:48.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.208+0000 7f0d33e00640 1 --2- 172.21.15.67:0/1914574447 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d37148050 0x555d37158ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:06:48.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.208+0000 7f0d2be00640 1 --2- 172.21.15.67:0/1914574447 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555d37148a40 0x555d37159410 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f0d1c0b4130 tx=0x7f0d1c009f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca7d25631251d7cf server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:06:48.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.208+0000 7f0d22000640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:06:48.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.208+0000 7f0d22000640 1 -- 172.21.15.67:0/1914574447 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d1c020070 con 0x555d37148a40 2024-03-28T16:06:48.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.208+0000 7f0d22000640 1 -- 172.21.15.67:0/1914574447 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f0d1c09a750 con 0x555d37148a40 2024-03-28T16:06:48.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.208+0000 7f0d3db8a740 1 -- 172.21.15.67:0/1914574447 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0d24057070 con 0x555d37148a40 2024-03-28T16:06:48.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.208+0000 7f0d22000640 1 -- 172.21.15.67:0/1914574447 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0d1c09f250 con 0x555d37148a40 2024-03-28T16:06:48.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.208+0000 7f0d3db8a740 10 client.15735 fetch_fsmap learned FSMap version 19 2024-03-28T16:06:48.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.208+0000 7f0d3db8a740 1 -- 172.21.15.67:0/1914574447 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x555d371b9fe0 con 0x555d37148a40 2024-03-28T16:06:48.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.209+0000 7f0d22000640 1 -- 172.21.15.67:0/1914574447 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0d24057070 con 0x555d37148a40 2024-03-28T16:06:48.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.209+0000 7f0d22000640 1 -- 172.21.15.67:0/1914574447 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f0d1c09a150 con 0x555d37148a40 2024-03-28T16:06:48.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.209+0000 7f0d3db8a740 10 client.15735 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:06:48.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.209+0000 7f0d3db8a740 10 client.15735 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:06:48.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.209+0000 7f0d3db8a740 20 client.15735 populate_metadata read hostname 'smithi067' 2024-03-28T16:06:48.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.209+0000 7f0d3db8a740 1 --2- 172.21.15.67:0/1914574447 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x555d371ba760 0x555d371bcb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:48.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.209+0000 7f0d3db8a740 4 client.15735 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T16:06:48.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.209+0000 7f0d3db8a740 1 -- 172.21.15.67:0/1914574447 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x555d370b4ae0 con 0x555d371ba760 2024-03-28T16:06:48.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.209+0000 7f0d33e00640 1 --2- 172.21.15.67:0/1914574447 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x555d371ba760 0x555d371bcb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:48.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.210+0000 7f0d33e00640 1 --2- 172.21.15.67:0/1914574447 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x555d371ba760 0x555d371bcb80 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=56e9bcb93b24d9df server_cookie=54fd2ae31e21ec64 in_seq=0 out_seq=0 2024-03-28T16:06:48.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.210+0000 7f0d22000640 10 client.15735 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:06:48.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.211+0000 7f0d22000640 1 -- 172.21.15.67:0/1914574447 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x555d370b4ae0 con 0x555d371ba760 2024-03-28T16:06:48.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.211+0000 7f0d22000640 10 client.15735 handle_command_reply: tid=0 2024-03-28T16:06:48.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.211+0000 7f0d3db8a740 1 client.15735 shutdown 2024-03-28T16:06:48.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.211+0000 7f0d3db8a740 1 -- 172.21.15.67:0/1914574447 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555d37148a40 msgr2=0x555d37159410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:48.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.211+0000 7f0d3db8a740 1 --2- 172.21.15.67:0/1914574447 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555d37148a40 0x555d37159410 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f0d1c0b4130 tx=0x7f0d1c009f20 comp rx=0 tx=0).stop 2024-03-28T16:06:48.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.211+0000 7f0d3db8a740 1 -- 172.21.15.67:0/1914574447 shutdown_connections 2024-03-28T16:06:48.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.211+0000 7f0d3db8a740 1 --2- 172.21.15.67:0/1914574447 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x555d371ba760 0x555d371bcb80 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:48.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.211+0000 7f0d3db8a740 1 --2- 172.21.15.67:0/1914574447 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555d3714dbb0 0x555d371574e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:48.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.211+0000 7f0d3db8a740 1 --2- 172.21.15.67:0/1914574447 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555d37148a40 0x555d37159410 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:48.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.211+0000 7f0d3db8a740 1 --2- 172.21.15.67:0/1914574447 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d37148050 0x555d37158ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:48.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.211+0000 7f0d22000640 0 client.15735 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:06:48.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.211+0000 7f0d3db8a740 1 -- 172.21.15.67:0/1914574447 >> 172.21.15.67:0/1914574447 conn(0x555d37110200 msgr2=0x555d37142bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:06:48.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.212+0000 7f0d3db8a740 1 -- 172.21.15.67:0/1914574447 shutdown_connections 2024-03-28T16:06:48.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.212+0000 7f0d3db8a740 1 -- 172.21.15.67:0/1914574447 wait complete. 2024-03-28T16:06:48.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.212+0000 7f0d3db8a740 20 client.15735 trim_cache size 0 max 16384 2024-03-28T16:06:48.215 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-28T16:06:48.215 INFO:teuthology.orchestra.run.smithi067.stdout: "status": "no active scrubs running", 2024-03-28T16:06:48.215 INFO:teuthology.orchestra.run.smithi067.stdout: "scrubs": {} 2024-03-28T16:06:48.215 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-28T16:06:48.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.212+0000 7f0d39800640 1 -- 172.21.15.67:0/2620909002 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f0d180a2800 msgr2=0x7f0d180a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:48.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.212+0000 7f0d39800640 1 --2- 172.21.15.67:0/2620909002 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f0d180a2800 0x7f0d180a4cc0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f0d2402c240 tx=0x7f0d24005fd0 comp rx=0 tx=0).stop 2024-03-28T16:06:48.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.212+0000 7f0d39800640 1 -- 172.21.15.67:0/2620909002 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d341049b0 msgr2=0x7f0d34199ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:48.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.212+0000 7f0d39800640 1 --2- 172.21.15.67:0/2620909002 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d341049b0 0x7f0d34199ef0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f0d1c00d950 tx=0x7f0d1c00de10 comp rx=0 tx=0).stop 2024-03-28T16:06:48.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.212+0000 7f0d39800640 1 -- 172.21.15.67:0/2620909002 shutdown_connections 2024-03-28T16:06:48.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.212+0000 7f0d39800640 1 --2- 172.21.15.67:0/2620909002 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f0d180a2800 0x7f0d180a4cc0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:48.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.212+0000 7f0d39800640 1 --2- 172.21.15.67:0/2620909002 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d34105370 0x7f0d341a0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:48.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.212+0000 7f0d39800640 1 --2- 172.21.15.67:0/2620909002 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d341049b0 0x7f0d34199ef0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:48.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.212+0000 7f0d39800640 1 --2- 172.21.15.67:0/2620909002 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0d341037c0 0x7f0d341999b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:48.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.212+0000 7f0d39800640 1 -- 172.21.15.67:0/2620909002 >> 172.21.15.67:0/2620909002 conn(0x7f0d340fed50 msgr2=0x7f0d34107d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:06:48.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.213+0000 7f0d39800640 1 -- 172.21.15.67:0/2620909002 shutdown_connections 2024-03-28T16:06:48.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.213+0000 7f0d39800640 1 -- 172.21.15.67:0/2620909002 wait complete. 2024-03-28T16:06:48.232 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T16:06:48.233 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T16:06:48.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.459+0000 7fd87b800640 1 -- 172.21.15.67:0/3119254867 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd874104ad0 msgr2=0x7fd874104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:48.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.459+0000 7fd87b800640 1 --2- 172.21.15.67:0/3119254867 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd874104ad0 0x7fd874104f50 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fd85c009ca0 tx=0x7fd85c02c0f0 comp rx=0 tx=0).stop 2024-03-28T16:06:48.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.459+0000 7fd87b800640 1 -- 172.21.15.67:0/3119254867 shutdown_connections 2024-03-28T16:06:48.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.459+0000 7fd87b800640 1 --2- 172.21.15.67:0/3119254867 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd874105490 0x7fd87410bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:48.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.459+0000 7fd87b800640 1 --2- 172.21.15.67:0/3119254867 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd874104ad0 0x7fd874104f50 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:48.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.459+0000 7fd87b800640 1 --2- 172.21.15.67:0/3119254867 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd8741038e0 0x7fd874103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:48.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.459+0000 7fd87b800640 1 -- 172.21.15.67:0/3119254867 >> 172.21.15.67:0/3119254867 conn(0x7fd8740fee50 msgr2=0x7fd874101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:06:48.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.459+0000 7fd87b800640 1 -- 172.21.15.67:0/3119254867 shutdown_connections 2024-03-28T16:06:48.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.459+0000 7fd87b800640 1 -- 172.21.15.67:0/3119254867 wait complete. 2024-03-28T16:06:48.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.460+0000 7fd87b800640 1 Processor -- start 2024-03-28T16:06:48.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.460+0000 7fd87b800640 1 -- start start 2024-03-28T16:06:48.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.461+0000 7fd87b800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd8741038e0 0x7fd874199b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:48.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.461+0000 7fd87b800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd874104ad0 0x7fd87419a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:48.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.461+0000 7fd87b800640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd874105490 0x7fd8741a1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:48.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.461+0000 7fd87b800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd87410f410 con 0x7fd874104ad0 2024-03-28T16:06:48.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.461+0000 7fd87b800640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd87410f290 con 0x7fd874105490 2024-03-28T16:06:48.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.461+0000 7fd87b800640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd87410f590 con 0x7fd8741038e0 2024-03-28T16:06:48.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.461+0000 7fd873e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd874104ad0 0x7fd87419a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:48.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.461+0000 7fd873e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd874104ad0 0x7fd87419a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45776/0 (socket says 172.21.15.67:45776) 2024-03-28T16:06:48.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.461+0000 7fd873e00640 1 -- 172.21.15.67:0/1741427290 learned_addr learned my addr 172.21.15.67:0/1741427290 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:06:48.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.461+0000 7fd878e00640 1 --2- 172.21.15.67:0/1741427290 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd8741038e0 0x7fd874199b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:48.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.461+0000 7fd879800640 1 --2- 172.21.15.67:0/1741427290 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd874105490 0x7fd8741a1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:48.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.462+0000 7fd873e00640 1 -- 172.21.15.67:0/1741427290 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd8741038e0 msgr2=0x7fd874199b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:48.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.462+0000 7fd873e00640 1 --2- 172.21.15.67:0/1741427290 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd8741038e0 0x7fd874199b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:48.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.462+0000 7fd873e00640 1 -- 172.21.15.67:0/1741427290 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd874105490 msgr2=0x7fd8741a1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:48.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.462+0000 7fd873e00640 1 --2- 172.21.15.67:0/1741427290 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd874105490 0x7fd8741a1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:48.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.462+0000 7fd873e00640 1 -- 172.21.15.67:0/1741427290 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8741a1770 con 0x7fd874104ad0 2024-03-28T16:06:48.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.462+0000 7fd878e00640 1 --2- 172.21.15.67:0/1741427290 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd8741038e0 0x7fd874199b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T16:06:48.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.462+0000 7fd873e00640 1 --2- 172.21.15.67:0/1741427290 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd874104ad0 0x7fd87419a090 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fd85c000f80 tx=0x7fd85c0328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:06:48.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.462+0000 7fd871600640 1 -- 172.21.15.67:0/1741427290 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd85c041020 con 0x7fd874104ad0 2024-03-28T16:06:48.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.462+0000 7fd871600640 1 -- 172.21.15.67:0/1741427290 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd85c034720 con 0x7fd874104ad0 2024-03-28T16:06:48.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.462+0000 7fd87b800640 1 -- 172.21.15.67:0/1741427290 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8741a1a00 con 0x7fd874104ad0 2024-03-28T16:06:48.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.462+0000 7fd871600640 1 -- 172.21.15.67:0/1741427290 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd85c039040 con 0x7fd874104ad0 2024-03-28T16:06:48.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.463+0000 7fd87b800640 1 -- 172.21.15.67:0/1741427290 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8741a1f40 con 0x7fd874104ad0 2024-03-28T16:06:48.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.464+0000 7fd87b800640 1 -- 172.21.15.67:0/1741427290 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd87410ac70 con 0x7fd874104ad0 2024-03-28T16:06:48.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.466+0000 7fd871600640 1 -- 172.21.15.67:0/1741427290 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fd85c044020 con 0x7fd874104ad0 2024-03-28T16:06:48.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.467+0000 7fd871600640 1 --2- 172.21.15.67:0/1741427290 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fd8540a2880 0x7fd8540a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:48.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.467+0000 7fd871600640 1 -- 172.21.15.67:0/1741427290 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fd85c0f0c70 con 0x7fd874104ad0 2024-03-28T16:06:48.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.467+0000 7fd878e00640 1 --2- 172.21.15.67:0/1741427290 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fd8540a2880 0x7fd8540a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:48.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.468+0000 7fd878e00640 1 --2- 172.21.15.67:0/1741427290 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fd8540a2880 0x7fd8540a4d40 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fd874104930 tx=0x7fd864005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:06:48.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.472+0000 7fd871600640 1 -- 172.21.15.67:0/1741427290 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd85c04b020 con 0x7fd874104ad0 2024-03-28T16:06:48.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.717+0000 7fd87b800640 1 -- 172.21.15.67:0/1741427290 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd8740622a0 con 0x7fd874104ad0 2024-03-28T16:06:48.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.719+0000 7fd871600640 1 -- 172.21.15.67:0/1741427290 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11341 (secure 0 0 0) 0x7fd85c0bdcc0 con 0x7fd874104ad0 2024-03-28T16:06:48.719 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:06:48.720 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2008297659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2008297659},{"type":"v1","addr":"172.21.15.67:6835","nonce":2008297659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14532,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/320295745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":320295745},{"type":"v1","addr":"172.21.15.67:6837","nonce":320295745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6839/691821273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":691821273},{"type":"v1","addr":"172.21.15.67:6839","nonce":691821273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6841/3211254875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6840","nonce":3211254875},{"type":"v1","addr":"172.21.15.67:6841","nonce":3211254875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2129665423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2129665423},{"type":"v1","addr":"172.21.15.183:6833","nonce":2129665423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24496,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/1589784425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":1589784425},{"type":"v1","addr":"172.21.15.183:6835","nonce":1589784425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24508,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/2185863835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":2185863835},{"type":"v1","addr":"172.21.15.183:6837","nonce":2185863835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24520,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6839/1242814398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":1242814398},{"type":"v1","addr":"172.21.15.183:6839","nonce":1242814398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":34409,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1008335521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1008335521},{"type":"v1","addr":"172.21.15.150:6833","nonce":1008335521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":34445,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3269935695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3269935695},{"type":"v1","addr":"172.21.15.150:6835","nonce":3269935695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T16:01:07.696824+0000","modified":"2024-03-28T16:01:11.997773+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":34508},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34475":{"gid":34475,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/1112886277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":1112886277},{"type":"v1","addr":"172.21.15.150:6837","nonce":1112886277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_34508":{"gid":34508,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6839/1211011614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":1211011614},{"type":"v1","addr":"172.21.15.150:6839","nonce":1211011614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34508,"qdb_cluster":[34508]},"id":1}]} 2024-03-28T16:06:48.720 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 19 2024-03-28T16:06:48.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.722+0000 7fd87b800640 1 -- 172.21.15.67:0/1741427290 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fd8540a2880 msgr2=0x7fd8540a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:48.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.722+0000 7fd87b800640 1 --2- 172.21.15.67:0/1741427290 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fd8540a2880 0x7fd8540a4d40 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fd874104930 tx=0x7fd864005f90 comp rx=0 tx=0).stop 2024-03-28T16:06:48.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.722+0000 7fd87b800640 1 -- 172.21.15.67:0/1741427290 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd874104ad0 msgr2=0x7fd87419a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:48.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.722+0000 7fd87b800640 1 --2- 172.21.15.67:0/1741427290 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd874104ad0 0x7fd87419a090 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fd85c000f80 tx=0x7fd85c0328a0 comp rx=0 tx=0).stop 2024-03-28T16:06:48.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.722+0000 7fd87b800640 1 -- 172.21.15.67:0/1741427290 shutdown_connections 2024-03-28T16:06:48.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.722+0000 7fd87b800640 1 --2- 172.21.15.67:0/1741427290 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fd8540a2880 0x7fd8540a4d40 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:48.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.722+0000 7fd87b800640 1 --2- 172.21.15.67:0/1741427290 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd874105490 0x7fd8741a1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:48.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.722+0000 7fd87b800640 1 --2- 172.21.15.67:0/1741427290 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd874104ad0 0x7fd87419a090 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:48.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.722+0000 7fd87b800640 1 --2- 172.21.15.67:0/1741427290 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd8741038e0 0x7fd874199b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:48.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.722+0000 7fd87b800640 1 -- 172.21.15.67:0/1741427290 >> 172.21.15.67:0/1741427290 conn(0x7fd8740fee50 msgr2=0x7fd874107e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:06:48.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.722+0000 7fd87b800640 1 -- 172.21.15.67:0/1741427290 shutdown_connections 2024-03-28T16:06:48.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.722+0000 7fd87b800640 1 -- 172.21.15.67:0/1741427290 wait complete. 2024-03-28T16:06:48.743 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T16:06:48.743 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T16:06:48.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.968+0000 7f3c0b200640 1 -- 172.21.15.67:0/874262318 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c0c10db70 msgr2=0x7f3c0c10ff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:48.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.968+0000 7f3c0b200640 1 --2- 172.21.15.67:0/874262318 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c0c10db70 0x7f3c0c10ff60 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f3c04009930 tx=0x7f3c0402bc30 comp rx=0 tx=0).stop 2024-03-28T16:06:48.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.968+0000 7f3c0b200640 1 -- 172.21.15.67:0/874262318 shutdown_connections 2024-03-28T16:06:48.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.968+0000 7f3c0b200640 1 --2- 172.21.15.67:0/874262318 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c0c10db70 0x7f3c0c10ff60 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:48.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.968+0000 7f3c0b200640 1 --2- 172.21.15.67:0/874262318 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c0c074100 0x7f3c0c074580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:48.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.968+0000 7f3c0b200640 1 --2- 172.21.15.67:0/874262318 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3c0c0756e0 0x7f3c0c073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:48.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.968+0000 7f3c0b200640 1 -- 172.21.15.67:0/874262318 >> 172.21.15.67:0/874262318 conn(0x7f3c0c0fcb10 msgr2=0x7f3c0c0fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:06:48.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.968+0000 7f3c0b200640 1 -- 172.21.15.67:0/874262318 shutdown_connections 2024-03-28T16:06:48.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.969+0000 7f3c0b200640 1 -- 172.21.15.67:0/874262318 wait complete. 2024-03-28T16:06:48.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.970+0000 7f3c0b200640 1 Processor -- start 2024-03-28T16:06:48.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.970+0000 7f3c0b200640 1 -- start start 2024-03-28T16:06:48.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.970+0000 7f3c0b200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c0c074100 0x7f3c0c19de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:48.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.970+0000 7f3c0b200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c0c0756e0 0x7f3c0c19e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:48.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.970+0000 7f3c0b200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3c0c10db70 0x7f3c0c1a5320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:48.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.970+0000 7f3c0b200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0c113710 con 0x7f3c0c0756e0 2024-03-28T16:06:48.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.970+0000 7f3c0b200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0c113590 con 0x7f3c0c074100 2024-03-28T16:06:48.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.970+0000 7f3c0b200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0c113890 con 0x7f3c0c10db70 2024-03-28T16:06:48.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.970+0000 7f3c01400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c0c0756e0 0x7f3c0c19e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:48.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.970+0000 7f3c01400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c0c0756e0 0x7f3c0c19e340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45806/0 (socket says 172.21.15.67:45806) 2024-03-28T16:06:48.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.970+0000 7f3c01400640 1 -- 172.21.15.67:0/1497526226 learned_addr learned my addr 172.21.15.67:0/1497526226 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:06:48.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.971+0000 7f3c09e00640 1 --2- 172.21.15.67:0/1497526226 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c0c074100 0x7f3c0c19de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:48.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.971+0000 7f3c0a800640 1 --2- 172.21.15.67:0/1497526226 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3c0c10db70 0x7f3c0c1a5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:48.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.971+0000 7f3c01400640 1 -- 172.21.15.67:0/1497526226 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3c0c10db70 msgr2=0x7f3c0c1a5320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:48.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.971+0000 7f3c01400640 1 --2- 172.21.15.67:0/1497526226 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3c0c10db70 0x7f3c0c1a5320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:48.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.971+0000 7f3c01400640 1 -- 172.21.15.67:0/1497526226 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c0c074100 msgr2=0x7f3c0c19de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:48.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.971+0000 7f3c01400640 1 --2- 172.21.15.67:0/1497526226 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c0c074100 0x7f3c0c19de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:48.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.971+0000 7f3c01400640 1 -- 172.21.15.67:0/1497526226 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c0c1a59c0 con 0x7f3c0c0756e0 2024-03-28T16:06:48.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.971+0000 7f3c09e00640 1 --2- 172.21.15.67:0/1497526226 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c0c074100 0x7f3c0c19de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T16:06:48.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.971+0000 7f3c01400640 1 --2- 172.21.15.67:0/1497526226 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c0c0756e0 0x7f3c0c19e340 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f3bf400d950 tx=0x7f3bf400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:06:48.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.972+0000 7f3c03400640 1 -- 172.21.15.67:0/1497526226 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf4017070 con 0x7f3c0c0756e0 2024-03-28T16:06:48.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.972+0000 7f3c03400640 1 -- 172.21.15.67:0/1497526226 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3bf4009c30 con 0x7f3c0c0756e0 2024-03-28T16:06:48.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.972+0000 7f3c0b200640 1 -- 172.21.15.67:0/1497526226 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c0c1a5cb0 con 0x7f3c0c0756e0 2024-03-28T16:06:48.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.972+0000 7f3c03400640 1 -- 172.21.15.67:0/1497526226 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf400f620 con 0x7f3c0c0756e0 2024-03-28T16:06:48.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.972+0000 7f3c0b200640 1 -- 172.21.15.67:0/1497526226 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c0c1a61a0 con 0x7f3c0c0756e0 2024-03-28T16:06:48.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.976+0000 7f3c13a6c740 1 Processor -- start 2024-03-28T16:06:48.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.976+0000 7f3c13a6c740 1 -- start start 2024-03-28T16:06:48.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.977+0000 7f3c03400640 1 -- 172.21.15.67:0/1497526226 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f3bf40109a0 con 0x7f3c0c0756e0 2024-03-28T16:06:48.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.977+0000 7f3c13a6c740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557c5e1209f0 0x557c5e120e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:48.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.977+0000 7f3c13a6c740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557c5e121bc0 0x557c5e122040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:48.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.977+0000 7f3c13a6c740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557c5e122580 0x557c5e128e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:48.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.977+0000 7f3c13a6c740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c5e0852c0 con 0x557c5e121bc0 2024-03-28T16:06:48.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.977+0000 7f3c13a6c740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c5e02dc90 con 0x557c5e1209f0 2024-03-28T16:06:48.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.977+0000 7f3c13a6c740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c5e081360 con 0x557c5e122580 2024-03-28T16:06:48.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.977+0000 7f3c09e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557c5e121bc0 0x557c5e122040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:48.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.977+0000 7f3c09e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557c5e121bc0 0x557c5e122040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45820/0 (socket says 172.21.15.67:45820) 2024-03-28T16:06:48.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.977+0000 7f3c09e00640 1 -- 172.21.15.67:0/4180363510 learned_addr learned my addr 172.21.15.67:0/4180363510 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:06:48.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.977+0000 7f3c0a800640 1 --2- 172.21.15.67:0/4180363510 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557c5e1209f0 0x557c5e120e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:48.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.978+0000 7f3c01400640 1 --2- 172.21.15.67:0/4180363510 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557c5e122580 0x557c5e128e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:48.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.978+0000 7f3c09e00640 1 -- 172.21.15.67:0/4180363510 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557c5e122580 msgr2=0x557c5e128e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:06:48.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.978+0000 7f3c09e00640 1 --2- 172.21.15.67:0/4180363510 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557c5e122580 0x557c5e128e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:48.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.978+0000 7f3c09e00640 1 -- 172.21.15.67:0/4180363510 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557c5e1209f0 msgr2=0x557c5e120e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:06:48.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.978+0000 7f3c09e00640 1 --2- 172.21.15.67:0/4180363510 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557c5e1209f0 0x557c5e120e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:48.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.978+0000 7f3c09e00640 1 -- 172.21.15.67:0/4180363510 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557c5e052080 con 0x557c5e121bc0 2024-03-28T16:06:48.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.978+0000 7f3c09e00640 1 --2- 172.21.15.67:0/4180363510 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557c5e121bc0 0x557c5e122040 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc0103a0 tx=0x7f3bfc010860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5988dba9e9f8df94 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:06:48.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.978+0000 7f3bfb400640 1 -- 172.21.15.67:0/4180363510 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bfc016070 con 0x557c5e121bc0 2024-03-28T16:06:48.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.978+0000 7f3bfb400640 1 -- 172.21.15.67:0/4180363510 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3bfc004d20 con 0x557c5e121bc0 2024-03-28T16:06:48.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.979+0000 7f3bfb400640 1 -- 172.21.15.67:0/4180363510 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bfc005360 con 0x557c5e121bc0 2024-03-28T16:06:48.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.979+0000 7f3c03400640 1 --2- 172.21.15.67:0/1497526226 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f3bf00a2880 0x7f3bf00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:48.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.979+0000 7f3c03400640 1 -- 172.21.15.67:0/1497526226 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f3bf40ce7c0 con 0x7f3c0c0756e0 2024-03-28T16:06:48.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.979+0000 7f3c0a800640 1 --2- 172.21.15.67:0/1497526226 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f3bf00a2880 0x7f3bf00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:48.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.979+0000 7f3c13a6c740 1 -- 172.21.15.67:0/4180363510 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557c5e121bc0 msgr2=0x557c5e122040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:48.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.979+0000 7f3c13a6c740 1 --2- 172.21.15.67:0/4180363510 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557c5e121bc0 0x557c5e122040 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc0103a0 tx=0x7f3bfc010860 comp rx=0 tx=0).stop 2024-03-28T16:06:48.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.979+0000 7f3c13a6c740 1 -- 172.21.15.67:0/4180363510 shutdown_connections 2024-03-28T16:06:48.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.979+0000 7f3c13a6c740 1 --2- 172.21.15.67:0/4180363510 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557c5e122580 0x557c5e128e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:48.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.979+0000 7f3c13a6c740 1 --2- 172.21.15.67:0/4180363510 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557c5e121bc0 0x557c5e122040 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:48.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.979+0000 7f3c13a6c740 1 --2- 172.21.15.67:0/4180363510 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557c5e1209f0 0x557c5e120e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:48.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.979+0000 7f3c13a6c740 1 -- 172.21.15.67:0/4180363510 >> 172.21.15.67:0/4180363510 conn(0x557c5e11bde0 msgr2=0x557c5e11c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:06:48.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.980+0000 7f3c13a6c740 1 -- 172.21.15.67:0/4180363510 shutdown_connections 2024-03-28T16:06:48.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.980+0000 7f3c13a6c740 1 -- 172.21.15.67:0/4180363510 wait complete. 2024-03-28T16:06:48.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.980+0000 7f3c0a800640 1 --2- 172.21.15.67:0/1497526226 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f3bf00a2880 0x7f3bf00a4d40 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f3c04036040 tx=0x7f3c0402c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:06:48.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.980+0000 7f3c13a6c740 1 -- start start 2024-03-28T16:06:48.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.980+0000 7f3c13a6c740 1 Processor -- start 2024-03-28T16:06:48.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.981+0000 7f3c13a6c740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557c5e1209f0 0x557c5e12ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:48.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.981+0000 7f3c13a6c740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557c5e121bc0 0x557c5e12b170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:48.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.981+0000 7f3c13a6c740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557c5e122580 0x557c5e1321b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:48.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.981+0000 7f3c13a6c740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c5e1295e0 con 0x557c5e122580 2024-03-28T16:06:48.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.981+0000 7f3c13a6c740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c5e0410d0 con 0x557c5e1209f0 2024-03-28T16:06:48.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.981+0000 7f3c13a6c740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c5e129760 con 0x557c5e121bc0 2024-03-28T16:06:48.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.981+0000 7f3c13a6c740 10 client.0 _ll_register_callbacks cb 0x557c5e045e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:06:48.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.981+0000 7f3c0a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557c5e122580 0x557c5e1321b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:48.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.981+0000 7f3c0a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557c5e122580 0x557c5e1321b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45836/0 (socket says 172.21.15.67:45836) 2024-03-28T16:06:48.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.981+0000 7f3c0a800640 1 -- 172.21.15.67:0/3958861877 learned_addr learned my addr 172.21.15.67:0/3958861877 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:06:48.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.981+0000 7f3c09e00640 1 --2- 172.21.15.67:0/3958861877 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557c5e1209f0 0x557c5e12ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:48.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.982+0000 7f3c01400640 1 --2- 172.21.15.67:0/3958861877 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557c5e121bc0 0x557c5e12b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:48.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.982+0000 7f3c0a800640 1 -- 172.21.15.67:0/3958861877 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557c5e121bc0 msgr2=0x557c5e12b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:06:48.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.982+0000 7f3c0a800640 1 --2- 172.21.15.67:0/3958861877 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557c5e121bc0 0x557c5e12b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:48.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.982+0000 7f3c0a800640 1 -- 172.21.15.67:0/3958861877 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557c5e1209f0 msgr2=0x557c5e12ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:06:48.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.982+0000 7f3c0a800640 1 --2- 172.21.15.67:0/3958861877 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557c5e1209f0 0x557c5e12ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:48.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.982+0000 7f3c0a800640 1 -- 172.21.15.67:0/3958861877 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557c5dd993f0 con 0x557c5e122580 2024-03-28T16:06:48.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.982+0000 7f3c0a800640 1 --2- 172.21.15.67:0/3958861877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557c5e122580 0x557c5e1321b0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f3c04009930 tx=0x7f3c0402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=167d0fa0ff264042 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:06:48.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.982+0000 7f3befe00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:06:48.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.982+0000 7f3befe00640 1 -- 172.21.15.67:0/3958861877 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c0404e030 con 0x557c5e122580 2024-03-28T16:06:48.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.982+0000 7f3befe00640 1 -- 172.21.15.67:0/3958861877 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f3c040046c0 con 0x557c5e122580 2024-03-28T16:06:48.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.982+0000 7f3c13a6c740 1 -- 172.21.15.67:0/3958861877 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3bfc004d20 con 0x557c5e122580 2024-03-28T16:06:48.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.983+0000 7f3befe00640 1 -- 172.21.15.67:0/3958861877 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3c040048c0 con 0x557c5e122580 2024-03-28T16:06:48.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.983+0000 7f3c13a6c740 10 client.15753 fetch_fsmap learned FSMap version 19 2024-03-28T16:06:48.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.983+0000 7f3c13a6c740 1 -- 172.21.15.67:0/3958861877 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x557c5de637d0 con 0x557c5e122580 2024-03-28T16:06:48.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.983+0000 7f3befe00640 1 -- 172.21.15.67:0/3958861877 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3bfc004d20 con 0x557c5e122580 2024-03-28T16:06:48.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.983+0000 7f3befe00640 1 -- 172.21.15.67:0/3958861877 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f3c0404a450 con 0x557c5e122580 2024-03-28T16:06:48.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.983+0000 7f3c13a6c740 10 client.15753 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:06:48.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.983+0000 7f3c13a6c740 10 client.15753 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:06:48.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.983+0000 7f3c13a6c740 20 client.15753 populate_metadata read hostname 'smithi067' 2024-03-28T16:06:48.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.983+0000 7f3c13a6c740 1 --2- 172.21.15.67:0/3958861877 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x557c5e1367f0 0x557c5e138c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:48.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.983+0000 7f3c13a6c740 4 client.15753 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:06:48.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.983+0000 7f3c13a6c740 1 -- 172.21.15.67:0/3958861877 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557c5e092800 con 0x557c5e1367f0 2024-03-28T16:06:48.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.984+0000 7f3c09e00640 1 --2- 172.21.15.67:0/3958861877 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x557c5e1367f0 0x557c5e138c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:48.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.985+0000 7f3c09e00640 1 --2- 172.21.15.67:0/3958861877 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x557c5e1367f0 0x557c5e138c60 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3826fccaaea1b094 server_cookie=6d560cfda387322c in_seq=0 out_seq=0 2024-03-28T16:06:48.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.985+0000 7f3befe00640 10 client.15753 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:06:48.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.987+0000 7f3befe00640 1 -- 172.21.15.67:0/3958861877 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x557c5e092800 con 0x557c5e1367f0 2024-03-28T16:06:48.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.987+0000 7f3befe00640 10 client.15753 handle_command_reply: tid=0 2024-03-28T16:06:48.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.987+0000 7f3c13a6c740 1 client.15753 shutdown 2024-03-28T16:06:48.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.987+0000 7f3c13a6c740 1 -- 172.21.15.67:0/3958861877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557c5e122580 msgr2=0x557c5e1321b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:48.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.987+0000 7f3c13a6c740 1 --2- 172.21.15.67:0/3958861877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557c5e122580 0x557c5e1321b0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f3c04009930 tx=0x7f3c0402bc30 comp rx=0 tx=0).stop 2024-03-28T16:06:48.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.987+0000 7f3c13a6c740 1 -- 172.21.15.67:0/3958861877 shutdown_connections 2024-03-28T16:06:48.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.987+0000 7f3c13a6c740 1 --2- 172.21.15.67:0/3958861877 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x557c5e1367f0 0x557c5e138c60 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:48.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.987+0000 7f3c13a6c740 1 --2- 172.21.15.67:0/3958861877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557c5e122580 0x557c5e1321b0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:48.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.987+0000 7f3c13a6c740 1 --2- 172.21.15.67:0/3958861877 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557c5e121bc0 0x557c5e12b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:48.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.987+0000 7f3c13a6c740 1 --2- 172.21.15.67:0/3958861877 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557c5e1209f0 0x557c5e12ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:48.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.987+0000 7f3befe00640 0 client.15753 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:06:48.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.987+0000 7f3c13a6c740 1 -- 172.21.15.67:0/3958861877 >> 172.21.15.67:0/3958861877 conn(0x557c5e11bde0 msgr2=0x557c5e11b160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:06:48.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.987+0000 7f3c13a6c740 1 -- 172.21.15.67:0/3958861877 shutdown_connections 2024-03-28T16:06:48.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.987+0000 7f3c13a6c740 1 -- 172.21.15.67:0/3958861877 wait complete. 2024-03-28T16:06:48.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:48.988+0000 7f3c13a6c740 20 client.15753 trim_cache size 0 max 16384 2024-03-28T16:06:49.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.001+0000 7f3c13a6c740 1 Processor -- start 2024-03-28T16:06:49.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.001+0000 7f3c13a6c740 1 -- start start 2024-03-28T16:06:49.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.002+0000 7f3c13a6c740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557c5e1209f0 0x557c5e124df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:49.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.002+0000 7f3c13a6c740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557c5e121bc0 0x557c5e132170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:49.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.002+0000 7f3c13a6c740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557c5e122580 0x557c5e125330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:49.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.002+0000 7f3c13a6c740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0c1acd40 con 0x557c5e122580 2024-03-28T16:06:49.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.002+0000 7f3c13a6c740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0c1af300 con 0x557c5e121bc0 2024-03-28T16:06:49.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.002+0000 7f3c13a6c740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0c1ad340 con 0x557c5e1209f0 2024-03-28T16:06:49.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.002+0000 7f3c0a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557c5e122580 0x557c5e125330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:49.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.002+0000 7f3c0a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557c5e122580 0x557c5e125330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45848/0 (socket says 172.21.15.67:45848) 2024-03-28T16:06:49.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.002+0000 7f3c0a800640 1 -- 172.21.15.67:0/3928613113 learned_addr learned my addr 172.21.15.67:0/3928613113 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:06:49.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.002+0000 7f3c01400640 1 --2- 172.21.15.67:0/3928613113 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557c5e121bc0 0x557c5e132170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:49.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.002+0000 7f3c09e00640 1 --2- 172.21.15.67:0/3928613113 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557c5e1209f0 0x557c5e124df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:49.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.003+0000 7f3c0a800640 1 -- 172.21.15.67:0/3928613113 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557c5e1209f0 msgr2=0x557c5e124df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:06:49.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.003+0000 7f3c0a800640 1 --2- 172.21.15.67:0/3928613113 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557c5e1209f0 0x557c5e124df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:49.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.003+0000 7f3c0a800640 1 -- 172.21.15.67:0/3928613113 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557c5e121bc0 msgr2=0x557c5e132170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:06:49.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.003+0000 7f3c0a800640 1 --2- 172.21.15.67:0/3928613113 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557c5e121bc0 0x557c5e132170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:49.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.003+0000 7f3c0a800640 1 -- 172.21.15.67:0/3928613113 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557c5e11bb50 con 0x557c5e122580 2024-03-28T16:06:49.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.003+0000 7f3c0a800640 1 --2- 172.21.15.67:0/3928613113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557c5e122580 0x557c5e125330 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f3c0402c170 tx=0x7f3c04004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1db94ce369e90467 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:06:49.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.003+0000 7f3bfaa00640 1 -- 172.21.15.67:0/3928613113 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c04063070 con 0x557c5e122580 2024-03-28T16:06:49.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.003+0000 7f3bfaa00640 1 -- 172.21.15.67:0/3928613113 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3c0402c8e0 con 0x557c5e122580 2024-03-28T16:06:49.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.003+0000 7f3bfaa00640 1 -- 172.21.15.67:0/3928613113 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c0404e040 con 0x557c5e122580 2024-03-28T16:06:49.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.004+0000 7f3c13a6c740 1 -- 172.21.15.67:0/3928613113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557c5e122580 msgr2=0x557c5e125330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:49.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.004+0000 7f3c13a6c740 1 --2- 172.21.15.67:0/3928613113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557c5e122580 0x557c5e125330 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f3c0402c170 tx=0x7f3c04004610 comp rx=0 tx=0).stop 2024-03-28T16:06:49.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.004+0000 7f3c13a6c740 1 -- 172.21.15.67:0/3928613113 shutdown_connections 2024-03-28T16:06:49.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.004+0000 7f3c13a6c740 1 --2- 172.21.15.67:0/3928613113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557c5e122580 0x557c5e125330 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:49.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.004+0000 7f3c13a6c740 1 --2- 172.21.15.67:0/3928613113 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557c5e121bc0 0x557c5e132170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:49.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.004+0000 7f3c13a6c740 1 --2- 172.21.15.67:0/3928613113 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557c5e1209f0 0x557c5e124df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:49.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.004+0000 7f3c13a6c740 1 -- 172.21.15.67:0/3928613113 >> 172.21.15.67:0/3928613113 conn(0x557c5e11bde0 msgr2=0x557c5e11f8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:06:49.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.004+0000 7f3c13a6c740 1 -- 172.21.15.67:0/3928613113 shutdown_connections 2024-03-28T16:06:49.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.004+0000 7f3c13a6c740 1 -- 172.21.15.67:0/3928613113 wait complete. 2024-03-28T16:06:49.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.004+0000 7f3c13a6c740 1 -- start start 2024-03-28T16:06:49.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.005+0000 7f3c13a6c740 1 Processor -- start 2024-03-28T16:06:49.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.005+0000 7f3c13a6c740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557c5e1209f0 0x557c5e131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:49.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.005+0000 7f3c13a6c740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557c5e121bc0 0x557c5e1304e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:49.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.005+0000 7f3c13a6c740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557c5e122580 0x557c5e130a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:49.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.005+0000 7f3c13a6c740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0c1ad980 con 0x557c5e122580 2024-03-28T16:06:49.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.005+0000 7f3c13a6c740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0c1aeca0 con 0x557c5e121bc0 2024-03-28T16:06:49.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.005+0000 7f3c13a6c740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0c1adfe0 con 0x557c5e1209f0 2024-03-28T16:06:49.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.005+0000 7f3c13a6c740 10 client.0 _ll_register_callbacks cb 0x7f3c04042050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:06:49.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.005+0000 7f3c0a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557c5e122580 0x557c5e130a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:49.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.005+0000 7f3c0a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557c5e122580 0x557c5e130a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45858/0 (socket says 172.21.15.67:45858) 2024-03-28T16:06:49.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.005+0000 7f3c0a800640 1 -- 172.21.15.67:0/1538326156 learned_addr learned my addr 172.21.15.67:0/1538326156 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:06:49.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.006+0000 7f3c01400640 1 --2- 172.21.15.67:0/1538326156 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557c5e121bc0 0x557c5e1304e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:49.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.006+0000 7f3c09e00640 1 --2- 172.21.15.67:0/1538326156 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557c5e1209f0 0x557c5e131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:49.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.006+0000 7f3c0a800640 1 -- 172.21.15.67:0/1538326156 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557c5e1209f0 msgr2=0x557c5e131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:06:49.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.006+0000 7f3c0a800640 1 --2- 172.21.15.67:0/1538326156 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557c5e1209f0 0x557c5e131e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:49.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.006+0000 7f3c0a800640 1 -- 172.21.15.67:0/1538326156 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557c5e121bc0 msgr2=0x557c5e1304e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:06:49.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.006+0000 7f3c0a800640 1 --2- 172.21.15.67:0/1538326156 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557c5e121bc0 0x557c5e1304e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:49.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.006+0000 7f3c0a800640 1 -- 172.21.15.67:0/1538326156 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557c5e131060 con 0x557c5e122580 2024-03-28T16:06:49.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.006+0000 7f3c0a800640 1 --2- 172.21.15.67:0/1538326156 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557c5e122580 0x557c5e130a20 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f3c0402bc70 tx=0x7f3c0404a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88185a6b4d513b12 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:06:49.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.006+0000 7f3befe00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:06:49.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.006+0000 7f3befe00640 1 -- 172.21.15.67:0/1538326156 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c0406c070 con 0x557c5e122580 2024-03-28T16:06:49.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.006+0000 7f3befe00640 1 -- 172.21.15.67:0/1538326156 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f3c04004c60 con 0x557c5e122580 2024-03-28T16:06:49.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.006+0000 7f3c13a6c740 1 -- 172.21.15.67:0/1538326156 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3c0402c8e0 con 0x557c5e122580 2024-03-28T16:06:49.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.007+0000 7f3befe00640 1 -- 172.21.15.67:0/1538326156 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3c04078050 con 0x557c5e122580 2024-03-28T16:06:49.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.007+0000 7f3c13a6c740 10 client.15759 fetch_fsmap learned FSMap version 19 2024-03-28T16:06:49.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.007+0000 7f3c13a6c740 1 -- 172.21.15.67:0/1538326156 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x557c5e192f60 con 0x557c5e122580 2024-03-28T16:06:49.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.007+0000 7f3befe00640 1 -- 172.21.15.67:0/1538326156 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3c0402c8e0 con 0x557c5e122580 2024-03-28T16:06:49.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.007+0000 7f3befe00640 1 -- 172.21.15.67:0/1538326156 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f3c04068610 con 0x557c5e122580 2024-03-28T16:06:49.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.007+0000 7f3c13a6c740 10 client.15759 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:06:49.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.007+0000 7f3c13a6c740 10 client.15759 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:06:49.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.007+0000 7f3c13a6c740 20 client.15759 populate_metadata read hostname 'smithi067' 2024-03-28T16:06:49.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.007+0000 7f3c13a6c740 1 --2- 172.21.15.67:0/1538326156 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x557c5e1937b0 0x557c5e195bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:49.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.007+0000 7f3c13a6c740 4 client.15759 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T16:06:49.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.007+0000 7f3c13a6c740 1 -- 172.21.15.67:0/1538326156 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x557c5e08cb50 con 0x557c5e1937b0 2024-03-28T16:06:49.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.008+0000 7f3c09e00640 1 --2- 172.21.15.67:0/1538326156 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x557c5e1937b0 0x557c5e195bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:49.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.009+0000 7f3c09e00640 1 --2- 172.21.15.67:0/1538326156 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x557c5e1937b0 0x557c5e195bd0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b272a1503db98a2 server_cookie=5c4a73694695be43 in_seq=0 out_seq=0 2024-03-28T16:06:49.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.009+0000 7f3befe00640 10 client.15759 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:06:49.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.010+0000 7f3befe00640 1 -- 172.21.15.67:0/1538326156 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x557c5e08cb50 con 0x557c5e1937b0 2024-03-28T16:06:49.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.010+0000 7f3befe00640 10 client.15759 handle_command_reply: tid=0 2024-03-28T16:06:49.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.010+0000 7f3c13a6c740 1 client.15759 shutdown 2024-03-28T16:06:49.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.010+0000 7f3c13a6c740 1 -- 172.21.15.67:0/1538326156 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557c5e122580 msgr2=0x557c5e130a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:49.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.010+0000 7f3c13a6c740 1 --2- 172.21.15.67:0/1538326156 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557c5e122580 0x557c5e130a20 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f3c0402bc70 tx=0x7f3c0404a680 comp rx=0 tx=0).stop 2024-03-28T16:06:49.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.010+0000 7f3c13a6c740 1 -- 172.21.15.67:0/1538326156 shutdown_connections 2024-03-28T16:06:49.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.010+0000 7f3c13a6c740 1 --2- 172.21.15.67:0/1538326156 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x557c5e1937b0 0x557c5e195bd0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:49.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.010+0000 7f3c13a6c740 1 --2- 172.21.15.67:0/1538326156 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557c5e122580 0x557c5e130a20 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:49.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.010+0000 7f3c13a6c740 1 --2- 172.21.15.67:0/1538326156 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557c5e121bc0 0x557c5e1304e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:49.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.010+0000 7f3befe00640 0 client.15759 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:06:49.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.010+0000 7f3c13a6c740 1 --2- 172.21.15.67:0/1538326156 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557c5e1209f0 0x557c5e131e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:49.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.010+0000 7f3c13a6c740 1 -- 172.21.15.67:0/1538326156 >> 172.21.15.67:0/1538326156 conn(0x557c5e11bde0 msgr2=0x557c5e11a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:06:49.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.010+0000 7f3c13a6c740 1 -- 172.21.15.67:0/1538326156 shutdown_connections 2024-03-28T16:06:49.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.010+0000 7f3c13a6c740 1 -- 172.21.15.67:0/1538326156 wait complete. 2024-03-28T16:06:49.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.010+0000 7f3c13a6c740 20 client.15759 trim_cache size 0 max 16384 2024-03-28T16:06:49.013 INFO:teuthology.orchestra.run.smithi067.stdout:[] 2024-03-28T16:06:49.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.011+0000 7f3c0b200640 1 -- 172.21.15.67:0/1497526226 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f3bf00a2880 msgr2=0x7f3bf00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:49.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.011+0000 7f3c0b200640 1 --2- 172.21.15.67:0/1497526226 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f3bf00a2880 0x7f3bf00a4d40 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f3c04036040 tx=0x7f3c0402c7e0 comp rx=0 tx=0).stop 2024-03-28T16:06:49.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.011+0000 7f3c0b200640 1 -- 172.21.15.67:0/1497526226 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c0c0756e0 msgr2=0x7f3c0c19e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:49.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.011+0000 7f3c0b200640 1 --2- 172.21.15.67:0/1497526226 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c0c0756e0 0x7f3c0c19e340 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f3bf400d950 tx=0x7f3bf400de10 comp rx=0 tx=0).stop 2024-03-28T16:06:49.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.011+0000 7f3c0b200640 1 -- 172.21.15.67:0/1497526226 shutdown_connections 2024-03-28T16:06:49.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.011+0000 7f3c0b200640 1 --2- 172.21.15.67:0/1497526226 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f3bf00a2880 0x7f3bf00a4d40 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:49.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.011+0000 7f3c0b200640 1 --2- 172.21.15.67:0/1497526226 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3c0c10db70 0x7f3c0c1a5320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:49.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.011+0000 7f3c0b200640 1 --2- 172.21.15.67:0/1497526226 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c0c0756e0 0x7f3c0c19e340 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:49.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.011+0000 7f3c0b200640 1 --2- 172.21.15.67:0/1497526226 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c0c074100 0x7f3c0c19de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:49.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.011+0000 7f3c0b200640 1 -- 172.21.15.67:0/1497526226 >> 172.21.15.67:0/1497526226 conn(0x7f3c0c0fcb10 msgr2=0x7f3c0c100e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:06:49.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.011+0000 7f3c0b200640 1 -- 172.21.15.67:0/1497526226 shutdown_connections 2024-03-28T16:06:49.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:49.011+0000 7f3c0b200640 1 -- 172.21.15.67:0/1497526226 wait complete. 2024-03-28T16:06:49.026 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:06:48 smithi067 ceph-mon[31707]: pgmap v536: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 18 MiB/s wr, 6 op/s 2024-03-28T16:06:49.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:06:48 smithi150 ceph-mon[34825]: pgmap v536: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 18 MiB/s wr, 6 op/s 2024-03-28T16:06:49.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:06:48 smithi183 ceph-mon[35580]: pgmap v536: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 18 MiB/s wr, 6 op/s 2024-03-28T16:06:50.030 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T16:06:50.030 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T16:06:50.030 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T16:06:50.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:06:49 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/1741427290' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:06:50.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:06:49 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/1741427290' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:06:50.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:06:49 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/1741427290' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:06:50.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.267+0000 7f1a52600640 1 -- 172.21.15.67:0/3409401722 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a4c103260 msgr2=0x7f1a4c109af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:50.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.267+0000 7f1a52600640 1 --2- 172.21.15.67:0/3409401722 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a4c103260 0x7f1a4c109af0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f1a44009900 tx=0x7f1a4402bc30 comp rx=0 tx=0).stop 2024-03-28T16:06:50.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.267+0000 7f1a52600640 1 -- 172.21.15.67:0/3409401722 shutdown_connections 2024-03-28T16:06:50.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.267+0000 7f1a52600640 1 --2- 172.21.15.67:0/3409401722 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a4c103260 0x7f1a4c109af0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:50.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.267+0000 7f1a52600640 1 --2- 172.21.15.67:0/3409401722 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1a4c1028a0 0x7f1a4c102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:50.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.267+0000 7f1a52600640 1 --2- 172.21.15.67:0/3409401722 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a4c1016b0 0x7f1a4c101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:50.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.267+0000 7f1a52600640 1 -- 172.21.15.67:0/3409401722 >> 172.21.15.67:0/3409401722 conn(0x7f1a4c0fcc40 msgr2=0x7f1a4c0ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:06:50.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.267+0000 7f1a52600640 1 -- 172.21.15.67:0/3409401722 shutdown_connections 2024-03-28T16:06:50.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.267+0000 7f1a52600640 1 -- 172.21.15.67:0/3409401722 wait complete. 2024-03-28T16:06:50.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.268+0000 7f1a52600640 1 Processor -- start 2024-03-28T16:06:50.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.268+0000 7f1a52600640 1 -- start start 2024-03-28T16:06:50.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.269+0000 7f1a52600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a4c1016b0 0x7f1a4c197800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:50.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.269+0000 7f1a52600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1a4c1028a0 0x7f1a4c197d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:50.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.269+0000 7f1a52600640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a4c103260 0x7f1a4c19edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:50.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.269+0000 7f1a52600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a4c10d210 con 0x7f1a4c1016b0 2024-03-28T16:06:50.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.269+0000 7f1a52600640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a4c10d090 con 0x7f1a4c103260 2024-03-28T16:06:50.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.269+0000 7f1a52600640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a4c10d390 con 0x7f1a4c1028a0 2024-03-28T16:06:50.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.269+0000 7f1a4b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a4c1016b0 0x7f1a4c197800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:50.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.269+0000 7f1a4b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a4c1016b0 0x7f1a4c197800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36278/0 (socket says 172.21.15.67:36278) 2024-03-28T16:06:50.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.269+0000 7f1a4b400640 1 -- 172.21.15.67:0/762085801 learned_addr learned my addr 172.21.15.67:0/762085801 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:06:50.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.269+0000 7f1a4aa00640 1 --2- 172.21.15.67:0/762085801 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1a4c1028a0 0x7f1a4c197d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:50.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.269+0000 7f1a4be00640 1 --2- 172.21.15.67:0/762085801 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a4c103260 0x7f1a4c19edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:50.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.269+0000 7f1a4b400640 1 -- 172.21.15.67:0/762085801 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1a4c1028a0 msgr2=0x7f1a4c197d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:50.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.269+0000 7f1a4b400640 1 --2- 172.21.15.67:0/762085801 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1a4c1028a0 0x7f1a4c197d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:50.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.269+0000 7f1a4b400640 1 -- 172.21.15.67:0/762085801 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a4c103260 msgr2=0x7f1a4c19edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:50.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.269+0000 7f1a4b400640 1 --2- 172.21.15.67:0/762085801 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a4c103260 0x7f1a4c19edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:50.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.269+0000 7f1a4b400640 1 -- 172.21.15.67:0/762085801 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a4c19f420 con 0x7f1a4c1016b0 2024-03-28T16:06:50.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.269+0000 7f1a4be00640 1 --2- 172.21.15.67:0/762085801 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a4c103260 0x7f1a4c19edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:06:50.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.270+0000 7f1a4b400640 1 --2- 172.21.15.67:0/762085801 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a4c1016b0 0x7f1a4c197800 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c00ea80 tx=0x7f1a3c00ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:06:50.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.270+0000 7f1a43e00640 1 -- 172.21.15.67:0/762085801 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a3c00ce40 con 0x7f1a4c1016b0 2024-03-28T16:06:50.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.270+0000 7f1a52600640 1 -- 172.21.15.67:0/762085801 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a4c19f710 con 0x7f1a4c1016b0 2024-03-28T16:06:50.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.270+0000 7f1a43e00640 1 -- 172.21.15.67:0/762085801 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1a3c011920 con 0x7f1a4c1016b0 2024-03-28T16:06:50.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.270+0000 7f1a43e00640 1 -- 172.21.15.67:0/762085801 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a3c01aa40 con 0x7f1a4c1016b0 2024-03-28T16:06:50.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.270+0000 7f1a52600640 1 -- 172.21.15.67:0/762085801 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a4c19fc60 con 0x7f1a4c1016b0 2024-03-28T16:06:50.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.273+0000 7f1a5536d740 1 Processor -- start 2024-03-28T16:06:50.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.274+0000 7f1a43e00640 1 -- 172.21.15.67:0/762085801 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f1a3c011ad0 con 0x7f1a4c1016b0 2024-03-28T16:06:50.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.274+0000 7f1a5536d740 1 -- start start 2024-03-28T16:06:50.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.274+0000 7f1a5536d740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55da805504b0 0x55da805508d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:50.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.274+0000 7f1a5536d740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55da80550ea0 0x55da805559a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:50.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.274+0000 7f1a5536d740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55da80556010 0x55da805584a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:50.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.274+0000 7f1a5536d740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da804b1eb0 con 0x55da80550ea0 2024-03-28T16:06:50.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.274+0000 7f1a5536d740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da8045aa60 con 0x55da805504b0 2024-03-28T16:06:50.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.274+0000 7f1a5536d740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da804adf50 con 0x55da80556010 2024-03-28T16:06:50.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.274+0000 7f1a4b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55da80550ea0 0x55da805559a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:50.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.275+0000 7f1a4b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55da80550ea0 0x55da805559a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36282/0 (socket says 172.21.15.67:36282) 2024-03-28T16:06:50.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.275+0000 7f1a4b400640 1 -- 172.21.15.67:0/4158286576 learned_addr learned my addr 172.21.15.67:0/4158286576 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:06:50.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.275+0000 7f1a4aa00640 1 --2- 172.21.15.67:0/4158286576 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55da80556010 0x55da805584a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:50.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.275+0000 7f1a4be00640 1 --2- 172.21.15.67:0/4158286576 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55da805504b0 0x55da805508d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:50.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.275+0000 7f1a4b400640 1 -- 172.21.15.67:0/4158286576 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55da80556010 msgr2=0x55da805584a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:06:50.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.275+0000 7f1a4b400640 1 --2- 172.21.15.67:0/4158286576 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55da80556010 0x55da805584a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:50.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.275+0000 7f1a4b400640 1 -- 172.21.15.67:0/4158286576 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55da805504b0 msgr2=0x55da805508d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:06:50.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.275+0000 7f1a4b400640 1 --2- 172.21.15.67:0/4158286576 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55da805504b0 0x55da805508d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:50.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.275+0000 7f1a4b400640 1 -- 172.21.15.67:0/4158286576 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55da8028e7e0 con 0x55da80550ea0 2024-03-28T16:06:50.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.275+0000 7f1a4b400640 1 --2- 172.21.15.67:0/4158286576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55da80550ea0 0x55da805559a0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c0d7290 tx=0x7f1a3c0d7750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dd73406bfd605e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:06:50.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.275+0000 7f1a43e00640 1 --2- 172.21.15.67:0/762085801 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f1a300a2800 0x7f1a300a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:50.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.276+0000 7f1a43e00640 1 -- 172.21.15.67:0/762085801 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f1a3c0cf110 con 0x7f1a4c1016b0 2024-03-28T16:06:50.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.276+0000 7f1a40c00640 1 -- 172.21.15.67:0/4158286576 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a3c0e2070 con 0x55da80550ea0 2024-03-28T16:06:50.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.276+0000 7f1a40c00640 1 -- 172.21.15.67:0/4158286576 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1a3c011e00 con 0x55da80550ea0 2024-03-28T16:06:50.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.276+0000 7f1a4be00640 1 --2- 172.21.15.67:0/762085801 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f1a300a2800 0x7f1a300a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:50.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.276+0000 7f1a40c00640 1 -- 172.21.15.67:0/4158286576 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a3c09b3c0 con 0x55da80550ea0 2024-03-28T16:06:50.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.277+0000 7f1a5536d740 1 -- 172.21.15.67:0/4158286576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55da80550ea0 msgr2=0x55da805559a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:50.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.277+0000 7f1a5536d740 1 --2- 172.21.15.67:0/4158286576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55da80550ea0 0x55da805559a0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c0d7290 tx=0x7f1a3c0d7750 comp rx=0 tx=0).stop 2024-03-28T16:06:50.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.277+0000 7f1a4be00640 1 --2- 172.21.15.67:0/762085801 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f1a300a2800 0x7f1a300a4cc0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f1a44006010 tx=0x7f1a44005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:06:50.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.277+0000 7f1a5536d740 1 -- 172.21.15.67:0/4158286576 shutdown_connections 2024-03-28T16:06:50.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.277+0000 7f1a5536d740 1 --2- 172.21.15.67:0/4158286576 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55da80556010 0x55da805584a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:50.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.277+0000 7f1a5536d740 1 --2- 172.21.15.67:0/4158286576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55da80550ea0 0x55da805559a0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:50.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.277+0000 7f1a5536d740 1 --2- 172.21.15.67:0/4158286576 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55da805504b0 0x55da805508d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:50.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.277+0000 7f1a5536d740 1 -- 172.21.15.67:0/4158286576 >> 172.21.15.67:0/4158286576 conn(0x55da80547610 msgr2=0x55da80547a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:06:50.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.277+0000 7f1a5536d740 1 -- 172.21.15.67:0/4158286576 shutdown_connections 2024-03-28T16:06:50.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.277+0000 7f1a5536d740 1 -- 172.21.15.67:0/4158286576 wait complete. 2024-03-28T16:06:50.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.277+0000 7f1a5536d740 1 -- start start 2024-03-28T16:06:50.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.278+0000 7f1a5536d740 1 Processor -- start 2024-03-28T16:06:50.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.278+0000 7f1a5536d740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55da805504b0 0x55da8055a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:50.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.278+0000 7f1a5536d740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55da80550ea0 0x55da8055ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:50.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.278+0000 7f1a5536d740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55da80556010 0x55da80561bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:50.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.278+0000 7f1a5536d740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da80558fc0 con 0x55da80556010 2024-03-28T16:06:50.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.278+0000 7f1a5536d740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da8046deb0 con 0x55da80550ea0 2024-03-28T16:06:50.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.278+0000 7f1a5536d740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da80559140 con 0x55da805504b0 2024-03-28T16:06:50.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.278+0000 7f1a5536d740 10 client.0 _ll_register_callbacks cb 0x55da80462150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:06:50.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.278+0000 7f1a4be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55da80556010 0x55da80561bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:50.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.278+0000 7f1a4be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55da80556010 0x55da80561bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36284/0 (socket says 172.21.15.67:36284) 2024-03-28T16:06:50.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.278+0000 7f1a4be00640 1 -- 172.21.15.67:0/2035978509 learned_addr learned my addr 172.21.15.67:0/2035978509 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:06:50.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.279+0000 7f1a4b400640 1 --2- 172.21.15.67:0/2035978509 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55da805504b0 0x55da8055a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:50.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.279+0000 7f1a4aa00640 1 --2- 172.21.15.67:0/2035978509 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55da80550ea0 0x55da8055ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:50.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.279+0000 7f1a4be00640 1 -- 172.21.15.67:0/2035978509 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55da805504b0 msgr2=0x55da8055a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:06:50.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.279+0000 7f1a4be00640 1 --2- 172.21.15.67:0/2035978509 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55da805504b0 0x55da8055a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:50.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.279+0000 7f1a4be00640 1 -- 172.21.15.67:0/2035978509 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55da80550ea0 msgr2=0x55da8055ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:06:50.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.279+0000 7f1a4be00640 1 --2- 172.21.15.67:0/2035978509 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55da80550ea0 0x55da8055ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:50.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.279+0000 7f1a4be00640 1 -- 172.21.15.67:0/2035978509 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55da802eef70 con 0x55da80556010 2024-03-28T16:06:50.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.279+0000 7f1a4be00640 1 --2- 172.21.15.67:0/2035978509 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55da80556010 0x55da80561bc0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f1a44009900 tx=0x7f1a4402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=145fed4afa2a9e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:06:50.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.279+0000 7f1a39600640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:06:50.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.279+0000 7f1a39600640 1 -- 172.21.15.67:0/2035978509 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a4404f020 con 0x55da80556010 2024-03-28T16:06:50.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.280+0000 7f1a39600640 1 -- 172.21.15.67:0/2035978509 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f1a440048c0 con 0x55da80556010 2024-03-28T16:06:50.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.280+0000 7f1a5536d740 1 -- 172.21.15.67:0/2035978509 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55da804bf3f0 con 0x55da80556010 2024-03-28T16:06:50.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.280+0000 7f1a39600640 1 -- 172.21.15.67:0/2035978509 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1a44004ac0 con 0x55da80556010 2024-03-28T16:06:50.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.280+0000 7f1a5536d740 10 client.15771 fetch_fsmap learned FSMap version 19 2024-03-28T16:06:50.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.280+0000 7f1a5536d740 1 -- 172.21.15.67:0/2035978509 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55da80186170 con 0x55da80556010 2024-03-28T16:06:50.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.280+0000 7f1a39600640 1 -- 172.21.15.67:0/2035978509 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55da804bf3f0 con 0x55da80556010 2024-03-28T16:06:50.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.280+0000 7f1a39600640 1 -- 172.21.15.67:0/2035978509 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f1a44034440 con 0x55da80556010 2024-03-28T16:06:50.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.280+0000 7f1a5536d740 10 client.15771 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:06:50.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.280+0000 7f1a5536d740 10 client.15771 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:06:50.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.280+0000 7f1a5536d740 20 client.15771 populate_metadata read hostname 'smithi067' 2024-03-28T16:06:50.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.280+0000 7f1a5536d740 1 --2- 172.21.15.67:0/2035978509 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55da80566560 0x55da805689a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:50.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.281+0000 7f1a5536d740 4 client.15771 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:06:50.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.281+0000 7f1a5536d740 1 -- 172.21.15.67:0/2035978509 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55da804b9740 con 0x55da80566560 2024-03-28T16:06:50.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.281+0000 7f1a4b400640 1 --2- 172.21.15.67:0/2035978509 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55da80566560 0x55da805689a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:50.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.282+0000 7f1a4b400640 1 --2- 172.21.15.67:0/2035978509 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55da80566560 0x55da805689a0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b080f255016a9d9 server_cookie=183cd5c122f828f3 in_seq=0 out_seq=0 2024-03-28T16:06:50.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.282+0000 7f1a39600640 10 client.15771 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:06:50.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.284+0000 7f1a39600640 1 -- 172.21.15.67:0/2035978509 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55da804b9740 con 0x55da80566560 2024-03-28T16:06:50.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.284+0000 7f1a39600640 10 client.15771 handle_command_reply: tid=0 2024-03-28T16:06:50.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.284+0000 7f1a5536d740 1 client.15771 shutdown 2024-03-28T16:06:50.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.284+0000 7f1a5536d740 1 -- 172.21.15.67:0/2035978509 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55da80556010 msgr2=0x55da80561bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:50.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.284+0000 7f1a5536d740 1 --2- 172.21.15.67:0/2035978509 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55da80556010 0x55da80561bc0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f1a44009900 tx=0x7f1a4402bc30 comp rx=0 tx=0).stop 2024-03-28T16:06:50.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.284+0000 7f1a5536d740 1 -- 172.21.15.67:0/2035978509 shutdown_connections 2024-03-28T16:06:50.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.284+0000 7f1a5536d740 1 --2- 172.21.15.67:0/2035978509 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55da80566560 0x55da805689a0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:50.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.285+0000 7f1a5536d740 1 --2- 172.21.15.67:0/2035978509 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55da80556010 0x55da80561bc0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:50.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.285+0000 7f1a5536d740 1 --2- 172.21.15.67:0/2035978509 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55da80550ea0 0x55da8055ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:50.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.285+0000 7f1a5536d740 1 --2- 172.21.15.67:0/2035978509 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55da805504b0 0x55da8055a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:50.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.285+0000 7f1a39600640 0 client.15771 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:06:50.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.285+0000 7f1a5536d740 1 -- 172.21.15.67:0/2035978509 >> 172.21.15.67:0/2035978509 conn(0x55da80547610 msgr2=0x55da80546830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:06:50.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.285+0000 7f1a5536d740 1 -- 172.21.15.67:0/2035978509 shutdown_connections 2024-03-28T16:06:50.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.285+0000 7f1a5536d740 1 -- 172.21.15.67:0/2035978509 wait complete. 2024-03-28T16:06:50.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.285+0000 7f1a5536d740 20 client.15771 trim_cache size 0 max 16384 2024-03-28T16:06:50.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.303+0000 7f1a5536d740 1 Processor -- start 2024-03-28T16:06:50.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.303+0000 7f1a5536d740 1 -- start start 2024-03-28T16:06:50.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.303+0000 7f1a5536d740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55da805504b0 0x55da80551940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:50.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.303+0000 7f1a5536d740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55da80550ea0 0x55da80561b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:50.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.303+0000 7f1a5536d740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55da80556010 0x55da80551e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:50.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.303+0000 7f1a5536d740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a4c1a4630 con 0x55da80556010 2024-03-28T16:06:50.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.303+0000 7f1a5536d740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a4c1a6bf0 con 0x55da805504b0 2024-03-28T16:06:50.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.303+0000 7f1a5536d740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a4c1a4c30 con 0x55da80550ea0 2024-03-28T16:06:50.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.304+0000 7f1a4be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55da80556010 0x55da80551e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:50.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.304+0000 7f1a4be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55da80556010 0x55da80551e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36286/0 (socket says 172.21.15.67:36286) 2024-03-28T16:06:50.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.304+0000 7f1a4be00640 1 -- 172.21.15.67:0/1496009804 learned_addr learned my addr 172.21.15.67:0/1496009804 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:06:50.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.304+0000 7f1a4aa00640 1 --2- 172.21.15.67:0/1496009804 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55da80550ea0 0x55da80561b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:50.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.304+0000 7f1a4b400640 1 --2- 172.21.15.67:0/1496009804 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55da805504b0 0x55da80551940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:50.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.304+0000 7f1a4be00640 1 -- 172.21.15.67:0/1496009804 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55da80550ea0 msgr2=0x55da80561b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:06:50.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.304+0000 7f1a4be00640 1 --2- 172.21.15.67:0/1496009804 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55da80550ea0 0x55da80561b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:50.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.304+0000 7f1a4be00640 1 -- 172.21.15.67:0/1496009804 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55da805504b0 msgr2=0x55da80551940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:06:50.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.304+0000 7f1a4be00640 1 --2- 172.21.15.67:0/1496009804 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55da805504b0 0x55da80551940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:50.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.304+0000 7f1a4be00640 1 -- 172.21.15.67:0/1496009804 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55da80559700 con 0x55da80556010 2024-03-28T16:06:50.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.304+0000 7f1a4b400640 1 --2- 172.21.15.67:0/1496009804 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55da805504b0 0x55da80551940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:06:50.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.304+0000 7f1a4be00640 1 --2- 172.21.15.67:0/1496009804 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55da80556010 0x55da80551e80 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f1a4402c170 tx=0x7f1a44004510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ef5d99ac16b9937 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:06:50.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.305+0000 7f1a3be00640 1 -- 172.21.15.67:0/1496009804 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a44063070 con 0x55da80556010 2024-03-28T16:06:50.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.305+0000 7f1a3be00640 1 -- 172.21.15.67:0/1496009804 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1a44034440 con 0x55da80556010 2024-03-28T16:06:50.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.305+0000 7f1a3be00640 1 -- 172.21.15.67:0/1496009804 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a4404f040 con 0x55da80556010 2024-03-28T16:06:50.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.305+0000 7f1a5536d740 1 -- 172.21.15.67:0/1496009804 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55da80556010 msgr2=0x55da80551e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:50.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.305+0000 7f1a5536d740 1 --2- 172.21.15.67:0/1496009804 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55da80556010 0x55da80551e80 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f1a4402c170 tx=0x7f1a44004510 comp rx=0 tx=0).stop 2024-03-28T16:06:50.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.305+0000 7f1a5536d740 1 -- 172.21.15.67:0/1496009804 shutdown_connections 2024-03-28T16:06:50.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.305+0000 7f1a5536d740 1 --2- 172.21.15.67:0/1496009804 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55da80556010 0x55da80551e80 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:50.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.305+0000 7f1a5536d740 1 --2- 172.21.15.67:0/1496009804 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55da80550ea0 0x55da80561b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:50.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.305+0000 7f1a5536d740 1 --2- 172.21.15.67:0/1496009804 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55da805504b0 0x55da80551940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:50.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.305+0000 7f1a5536d740 1 -- 172.21.15.67:0/1496009804 >> 172.21.15.67:0/1496009804 conn(0x55da80547610 msgr2=0x55da8054b100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:06:50.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.306+0000 7f1a5536d740 1 -- 172.21.15.67:0/1496009804 shutdown_connections 2024-03-28T16:06:50.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.306+0000 7f1a5536d740 1 -- 172.21.15.67:0/1496009804 wait complete. 2024-03-28T16:06:50.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.306+0000 7f1a5536d740 1 -- start start 2024-03-28T16:06:50.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.306+0000 7f1a5536d740 1 Processor -- start 2024-03-28T16:06:50.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.307+0000 7f1a5536d740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55da805504b0 0x55da80561550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:50.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.307+0000 7f1a5536d740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55da80550ea0 0x55da80561a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:50.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.307+0000 7f1a5536d740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55da80556010 0x55da8055fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:50.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.307+0000 7f1a5536d740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a4c1a5270 con 0x55da805504b0 2024-03-28T16:06:50.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.307+0000 7f1a5536d740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a4c1a6590 con 0x55da80556010 2024-03-28T16:06:50.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.307+0000 7f1a5536d740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a4c1a58d0 con 0x55da80550ea0 2024-03-28T16:06:50.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.307+0000 7f1a5536d740 10 client.0 _ll_register_callbacks cb 0x7f1a44009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:06:50.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.307+0000 7f1a4b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55da805504b0 0x55da80561550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:50.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.307+0000 7f1a4b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55da805504b0 0x55da80561550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36292/0 (socket says 172.21.15.67:36292) 2024-03-28T16:06:50.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.307+0000 7f1a4b400640 1 -- 172.21.15.67:0/3536239784 learned_addr learned my addr 172.21.15.67:0/3536239784 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:06:50.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.307+0000 7f1a4aa00640 1 --2- 172.21.15.67:0/3536239784 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55da80550ea0 0x55da80561a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:50.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.307+0000 7f1a4b400640 1 -- 172.21.15.67:0/3536239784 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55da80550ea0 msgr2=0x55da80561a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:06:50.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.307+0000 7f1a4b400640 1 --2- 172.21.15.67:0/3536239784 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55da80550ea0 0x55da80561a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:50.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.307+0000 7f1a4b400640 1 -- 172.21.15.67:0/3536239784 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55da80556010 msgr2=0x55da8055fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:06:50.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.307+0000 7f1a4b400640 1 --2- 172.21.15.67:0/3536239784 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55da80556010 0x55da8055fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:50.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.307+0000 7f1a4b400640 1 -- 172.21.15.67:0/3536239784 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55da80560390 con 0x55da805504b0 2024-03-28T16:06:50.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.308+0000 7f1a4b400640 1 --2- 172.21.15.67:0/3536239784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55da805504b0 0x55da80561550 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c098430 tx=0x7f1a3c0d2840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=324282468fe4b47 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:06:50.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.308+0000 7f1a39600640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:06:50.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.308+0000 7f1a39600640 1 -- 172.21.15.67:0/3536239784 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a3c0d2c40 con 0x55da805504b0 2024-03-28T16:06:50.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.308+0000 7f1a39600640 1 -- 172.21.15.67:0/3536239784 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f1a3c099d60 con 0x55da805504b0 2024-03-28T16:06:50.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.308+0000 7f1a5536d740 1 -- 172.21.15.67:0/3536239784 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1a44034440 con 0x55da805504b0 2024-03-28T16:06:50.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.308+0000 7f1a39600640 1 -- 172.21.15.67:0/3536239784 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1a3c09b3c0 con 0x55da805504b0 2024-03-28T16:06:50.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.308+0000 7f1a5536d740 10 client.15777 fetch_fsmap learned FSMap version 19 2024-03-28T16:06:50.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.308+0000 7f1a5536d740 1 -- 172.21.15.67:0/3536239784 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55da805c29b0 con 0x55da805504b0 2024-03-28T16:06:50.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.308+0000 7f1a39600640 1 -- 172.21.15.67:0/3536239784 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1a44034440 con 0x55da805504b0 2024-03-28T16:06:50.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.308+0000 7f1a39600640 1 -- 172.21.15.67:0/3536239784 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f1a3c022030 con 0x55da805504b0 2024-03-28T16:06:50.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.309+0000 7f1a5536d740 10 client.15777 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:06:50.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.309+0000 7f1a5536d740 10 client.15777 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:06:50.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.309+0000 7f1a5536d740 20 client.15777 populate_metadata read hostname 'smithi067' 2024-03-28T16:06:50.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.309+0000 7f1a5536d740 1 --2- 172.21.15.67:0/3536239784 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55da805c2f00 0x55da805c5320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:50.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.309+0000 7f1a5536d740 4 client.15777 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T16:06:50.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.309+0000 7f1a5536d740 1 -- 172.21.15.67:0/3536239784 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55da80437a40 con 0x55da805c2f00 2024-03-28T16:06:50.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.309+0000 7f1a4aa00640 1 --2- 172.21.15.67:0/3536239784 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55da805c2f00 0x55da805c5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:50.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.310+0000 7f1a4aa00640 1 --2- 172.21.15.67:0/3536239784 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55da805c2f00 0x55da805c5320 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b5eb65e28b3122c4 server_cookie=cb2df171bdd84035 in_seq=0 out_seq=0 2024-03-28T16:06:50.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.310+0000 7f1a39600640 10 client.15777 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:06:50.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.314+0000 7f1a39600640 1 -- 172.21.15.67:0/3536239784 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55da80437a40 con 0x55da805c2f00 2024-03-28T16:06:50.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.314+0000 7f1a39600640 10 client.15777 handle_command_reply: tid=0 2024-03-28T16:06:50.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.314+0000 7f1a5536d740 1 client.15777 shutdown 2024-03-28T16:06:50.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.314+0000 7f1a5536d740 1 -- 172.21.15.67:0/3536239784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55da805504b0 msgr2=0x55da80561550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:50.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.314+0000 7f1a5536d740 1 --2- 172.21.15.67:0/3536239784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55da805504b0 0x55da80561550 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c098430 tx=0x7f1a3c0d2840 comp rx=0 tx=0).stop 2024-03-28T16:06:50.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.314+0000 7f1a5536d740 1 -- 172.21.15.67:0/3536239784 shutdown_connections 2024-03-28T16:06:50.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.314+0000 7f1a5536d740 1 --2- 172.21.15.67:0/3536239784 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55da805c2f00 0x55da805c5320 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:50.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.314+0000 7f1a5536d740 1 --2- 172.21.15.67:0/3536239784 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55da80556010 0x55da8055fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:50.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.314+0000 7f1a5536d740 1 --2- 172.21.15.67:0/3536239784 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55da80550ea0 0x55da80561a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:50.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.314+0000 7f1a5536d740 1 --2- 172.21.15.67:0/3536239784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55da805504b0 0x55da80561550 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:50.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.314+0000 7f1a5536d740 1 -- 172.21.15.67:0/3536239784 >> 172.21.15.67:0/3536239784 conn(0x55da80547610 msgr2=0x55da80545fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:06:50.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.314+0000 7f1a39600640 0 client.15777 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:06:50.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.314+0000 7f1a5536d740 1 -- 172.21.15.67:0/3536239784 shutdown_connections 2024-03-28T16:06:50.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.314+0000 7f1a5536d740 1 -- 172.21.15.67:0/3536239784 wait complete. 2024-03-28T16:06:50.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.315+0000 7f1a5536d740 20 client.15777 trim_cache size 0 max 16384 2024-03-28T16:06:50.316 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-28T16:06:50.316 INFO:teuthology.orchestra.run.smithi067.stdout: "return_code": 0, 2024-03-28T16:06:50.316 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_tag": "c40b92db-8aef-4c20-9b6f-37849bc69d92", 2024-03-28T16:06:50.316 INFO:teuthology.orchestra.run.smithi067.stdout: "mode": "asynchronous" 2024-03-28T16:06:50.317 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-28T16:06:50.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.315+0000 7f1a52600640 1 -- 172.21.15.67:0/762085801 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f1a300a2800 msgr2=0x7f1a300a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:50.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.315+0000 7f1a52600640 1 --2- 172.21.15.67:0/762085801 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f1a300a2800 0x7f1a300a4cc0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f1a44006010 tx=0x7f1a44005a20 comp rx=0 tx=0).stop 2024-03-28T16:06:50.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.315+0000 7f1a52600640 1 -- 172.21.15.67:0/762085801 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a4c1016b0 msgr2=0x7f1a4c197800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:50.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.315+0000 7f1a52600640 1 --2- 172.21.15.67:0/762085801 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a4c1016b0 0x7f1a4c197800 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c00ea80 tx=0x7f1a3c00ef40 comp rx=0 tx=0).stop 2024-03-28T16:06:50.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.315+0000 7f1a52600640 1 -- 172.21.15.67:0/762085801 shutdown_connections 2024-03-28T16:06:50.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.315+0000 7f1a52600640 1 --2- 172.21.15.67:0/762085801 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f1a300a2800 0x7f1a300a4cc0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:50.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.315+0000 7f1a52600640 1 --2- 172.21.15.67:0/762085801 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a4c103260 0x7f1a4c19edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:50.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.315+0000 7f1a52600640 1 --2- 172.21.15.67:0/762085801 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1a4c1028a0 0x7f1a4c197d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:50.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.315+0000 7f1a52600640 1 --2- 172.21.15.67:0/762085801 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a4c1016b0 0x7f1a4c197800 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:50.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.315+0000 7f1a52600640 1 -- 172.21.15.67:0/762085801 >> 172.21.15.67:0/762085801 conn(0x7f1a4c0fcc40 msgr2=0x7f1a4c105b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:06:50.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.315+0000 7f1a52600640 1 -- 172.21.15.67:0/762085801 shutdown_connections 2024-03-28T16:06:50.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.316+0000 7f1a52600640 1 -- 172.21.15.67:0/762085801 wait complete. 2024-03-28T16:06:50.335 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T16:06:50.335 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T16:06:50.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.605+0000 7f8d35400640 1 -- 172.21.15.67:0/1652920319 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d30105470 msgr2=0x7f8d3010bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:50.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.605+0000 7f8d35400640 1 --2- 172.21.15.67:0/1652920319 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d30105470 0x7f8d3010bd00 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f8d28009ca0 tx=0x7f8d2802c0f0 comp rx=0 tx=0).stop 2024-03-28T16:06:50.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.606+0000 7f8d35400640 1 -- 172.21.15.67:0/1652920319 shutdown_connections 2024-03-28T16:06:50.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.606+0000 7f8d35400640 1 --2- 172.21.15.67:0/1652920319 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d30105470 0x7f8d3010bd00 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:50.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.606+0000 7f8d35400640 1 --2- 172.21.15.67:0/1652920319 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8d30104ab0 0x7f8d30104f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:50.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.606+0000 7f8d35400640 1 --2- 172.21.15.67:0/1652920319 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d301038c0 0x7f8d30103ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:50.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.606+0000 7f8d35400640 1 -- 172.21.15.67:0/1652920319 >> 172.21.15.67:0/1652920319 conn(0x7f8d300fee50 msgr2=0x7f8d30101270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:06:50.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.606+0000 7f8d35400640 1 -- 172.21.15.67:0/1652920319 shutdown_connections 2024-03-28T16:06:50.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.606+0000 7f8d35400640 1 -- 172.21.15.67:0/1652920319 wait complete. 2024-03-28T16:06:50.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.607+0000 7f8d35400640 1 Processor -- start 2024-03-28T16:06:50.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.607+0000 7f8d35400640 1 -- start start 2024-03-28T16:06:50.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.607+0000 7f8d35400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d301038c0 0x7f8d30199a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:50.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.607+0000 7f8d35400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d30104ab0 0x7f8d30199fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:50.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.607+0000 7f8d35400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8d30105470 0x7f8d301a1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:50.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.607+0000 7f8d35400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d3010f3f0 con 0x7f8d30104ab0 2024-03-28T16:06:50.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.607+0000 7f8d35400640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d3010f270 con 0x7f8d301038c0 2024-03-28T16:06:50.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.607+0000 7f8d35400640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d3010f570 con 0x7f8d30105470 2024-03-28T16:06:50.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.607+0000 7f8d27e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d30104ab0 0x7f8d30199fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:50.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.607+0000 7f8d27e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d30104ab0 0x7f8d30199fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36312/0 (socket says 172.21.15.67:36312) 2024-03-28T16:06:50.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.607+0000 7f8d27e00640 1 -- 172.21.15.67:0/4056711640 learned_addr learned my addr 172.21.15.67:0/4056711640 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:06:50.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.608+0000 7f8d2f200640 1 --2- 172.21.15.67:0/4056711640 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8d30105470 0x7f8d301a1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:50.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.608+0000 7f8d27e00640 1 -- 172.21.15.67:0/4056711640 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8d30105470 msgr2=0x7f8d301a1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:50.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.608+0000 7f8d27e00640 1 --2- 172.21.15.67:0/4056711640 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8d30105470 0x7f8d301a1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:50.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.608+0000 7f8d27e00640 1 -- 172.21.15.67:0/4056711640 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d301038c0 msgr2=0x7f8d30199a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:50.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.608+0000 7f8d2e800640 1 --2- 172.21.15.67:0/4056711640 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d301038c0 0x7f8d30199a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:50.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.608+0000 7f8d27e00640 1 --2- 172.21.15.67:0/4056711640 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d301038c0 0x7f8d30199a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:50.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.608+0000 7f8d27e00640 1 -- 172.21.15.67:0/4056711640 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d301a16b0 con 0x7f8d30104ab0 2024-03-28T16:06:50.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.608+0000 7f8d2f200640 1 --2- 172.21.15.67:0/4056711640 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8d30105470 0x7f8d301a1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:06:50.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.608+0000 7f8d2e800640 1 --2- 172.21.15.67:0/4056711640 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d301038c0 0x7f8d30199a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:06:50.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.608+0000 7f8d27e00640 1 --2- 172.21.15.67:0/4056711640 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d30104ab0 0x7f8d30199fd0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f8d1800d770 tx=0x7f8d1800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:06:50.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.609+0000 7f8d27400640 1 -- 172.21.15.67:0/4056711640 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d18017070 con 0x7f8d30104ab0 2024-03-28T16:06:50.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.609+0000 7f8d27400640 1 -- 172.21.15.67:0/4056711640 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8d180054d0 con 0x7f8d30104ab0 2024-03-28T16:06:50.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.609+0000 7f8d35400640 1 -- 172.21.15.67:0/4056711640 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d301a19a0 con 0x7f8d30104ab0 2024-03-28T16:06:50.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.609+0000 7f8d27400640 1 -- 172.21.15.67:0/4056711640 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d18004cf0 con 0x7f8d30104ab0 2024-03-28T16:06:50.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.609+0000 7f8d35400640 1 -- 172.21.15.67:0/4056711640 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d301a1ea0 con 0x7f8d30104ab0 2024-03-28T16:06:50.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.612+0000 7f8d27400640 1 -- 172.21.15.67:0/4056711640 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f8d18004e90 con 0x7f8d30104ab0 2024-03-28T16:06:50.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.612+0000 7f8d381ec740 1 Processor -- start 2024-03-28T16:06:50.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.612+0000 7f8d381ec740 1 -- start start 2024-03-28T16:06:50.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.613+0000 7f8d381ec740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556221973e80 0x5562219742a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:50.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.613+0000 7f8d381ec740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556221974870 0x556221979370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:50.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.613+0000 7f8d381ec740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5562219799e0 0x55622197be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:50.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.613+0000 7f8d381ec740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562218d9250 con 0x5562219799e0 2024-03-28T16:06:50.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.613+0000 7f8d381ec740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x556221881a30 con 0x556221973e80 2024-03-28T16:06:50.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.613+0000 7f8d381ec740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562218d52f0 con 0x556221974870 2024-03-28T16:06:50.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.613+0000 7f8d27e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5562219799e0 0x55622197be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:50.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.613+0000 7f8d27e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5562219799e0 0x55622197be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36318/0 (socket says 172.21.15.67:36318) 2024-03-28T16:06:50.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.613+0000 7f8d27e00640 1 -- 172.21.15.67:0/3217050490 learned_addr learned my addr 172.21.15.67:0/3217050490 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:06:50.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.613+0000 7f8d2f200640 1 --2- 172.21.15.67:0/3217050490 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556221973e80 0x5562219742a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:50.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.614+0000 7f8d2e800640 1 --2- 172.21.15.67:0/3217050490 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556221974870 0x556221979370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:50.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.614+0000 7f8d27e00640 1 -- 172.21.15.67:0/3217050490 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556221974870 msgr2=0x556221979370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:06:50.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.614+0000 7f8d27e00640 1 --2- 172.21.15.67:0/3217050490 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556221974870 0x556221979370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:50.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.614+0000 7f8d27e00640 1 -- 172.21.15.67:0/3217050490 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556221973e80 msgr2=0x5562219742a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:06:50.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.614+0000 7f8d27e00640 1 --2- 172.21.15.67:0/3217050490 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556221973e80 0x5562219742a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:50.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.614+0000 7f8d27e00640 1 -- 172.21.15.67:0/3217050490 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55622186c690 con 0x5562219799e0 2024-03-28T16:06:50.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.614+0000 7f8d27e00640 1 --2- 172.21.15.67:0/3217050490 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5562219799e0 0x55622197be70 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f8d180d47c0 tx=0x7f8d180d4c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2266bb230375a7b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:06:50.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.614+0000 7f8d27400640 1 --2- 172.21.15.67:0/4056711640 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f8d140a2800 0x7f8d140a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:50.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.615+0000 7f8d2f200640 1 --2- 172.21.15.67:0/4056711640 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f8d140a2800 0x7f8d140a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:50.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.615+0000 7f8d27400640 1 -- 172.21.15.67:0/4056711640 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f8d180ce560 con 0x7f8d30104ab0 2024-03-28T16:06:50.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.615+0000 7f8d1f400640 1 -- 172.21.15.67:0/3217050490 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d180e1070 con 0x5562219799e0 2024-03-28T16:06:50.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.615+0000 7f8d1f400640 1 -- 172.21.15.67:0/3217050490 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8d18009c30 con 0x5562219799e0 2024-03-28T16:06:50.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.615+0000 7f8d1f400640 1 -- 172.21.15.67:0/3217050490 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d180971b0 con 0x5562219799e0 2024-03-28T16:06:50.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.615+0000 7f8d381ec740 1 -- 172.21.15.67:0/3217050490 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5562219799e0 msgr2=0x55622197be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:50.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.615+0000 7f8d381ec740 1 --2- 172.21.15.67:0/3217050490 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5562219799e0 0x55622197be70 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f8d180d47c0 tx=0x7f8d180d4c80 comp rx=0 tx=0).stop 2024-03-28T16:06:50.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.615+0000 7f8d2f200640 1 --2- 172.21.15.67:0/4056711640 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f8d140a2800 0x7f8d140a4cc0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f8d28032420 tx=0x7f8d28005f80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:06:50.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.616+0000 7f8d381ec740 1 -- 172.21.15.67:0/3217050490 shutdown_connections 2024-03-28T16:06:50.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.616+0000 7f8d381ec740 1 --2- 172.21.15.67:0/3217050490 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5562219799e0 0x55622197be70 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:50.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.616+0000 7f8d381ec740 1 --2- 172.21.15.67:0/3217050490 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556221974870 0x556221979370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:50.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.616+0000 7f8d381ec740 1 --2- 172.21.15.67:0/3217050490 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556221973e80 0x5562219742a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:50.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.616+0000 7f8d381ec740 1 -- 172.21.15.67:0/3217050490 >> 172.21.15.67:0/3217050490 conn(0x55622193c200 msgr2=0x55622196f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:06:50.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.616+0000 7f8d381ec740 1 -- 172.21.15.67:0/3217050490 shutdown_connections 2024-03-28T16:06:50.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.616+0000 7f8d381ec740 1 -- 172.21.15.67:0/3217050490 wait complete. 2024-03-28T16:06:50.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.616+0000 7f8d381ec740 1 -- start start 2024-03-28T16:06:50.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.616+0000 7f8d381ec740 1 Processor -- start 2024-03-28T16:06:50.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.617+0000 7f8d381ec740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556221973e80 0x55622197de30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:50.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.617+0000 7f8d381ec740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556221974870 0x55622197e370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:50.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.617+0000 7f8d381ec740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5562219799e0 0x5562219853b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:50.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.617+0000 7f8d381ec740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55622197c990 con 0x556221974870 2024-03-28T16:06:50.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.617+0000 7f8d381ec740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x556221894e80 con 0x5562219799e0 2024-03-28T16:06:50.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.617+0000 7f8d381ec740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55622197cb10 con 0x556221973e80 2024-03-28T16:06:50.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.617+0000 7f8d381ec740 10 client.0 _ll_register_callbacks cb 0x5562218a7130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:06:50.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.617+0000 7f8d27e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556221974870 0x55622197e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:50.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.617+0000 7f8d27e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556221974870 0x55622197e370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36326/0 (socket says 172.21.15.67:36326) 2024-03-28T16:06:50.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.617+0000 7f8d27e00640 1 -- 172.21.15.67:0/576207679 learned_addr learned my addr 172.21.15.67:0/576207679 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:06:50.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.618+0000 7f8d2e800640 1 --2- 172.21.15.67:0/576207679 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556221973e80 0x55622197de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:50.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.618+0000 7f8d27e00640 1 -- 172.21.15.67:0/576207679 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556221973e80 msgr2=0x55622197de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:06:50.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.618+0000 7f8d27e00640 1 --2- 172.21.15.67:0/576207679 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556221973e80 0x55622197de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:50.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.618+0000 7f8d27e00640 1 -- 172.21.15.67:0/576207679 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5562219799e0 msgr2=0x5562219853b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:06:50.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.618+0000 7f8d2f200640 1 --2- 172.21.15.67:0/576207679 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5562219799e0 0x5562219853b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:50.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.618+0000 7f8d27e00640 1 --2- 172.21.15.67:0/576207679 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5562219799e0 0x5562219853b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:50.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.618+0000 7f8d27e00640 1 -- 172.21.15.67:0/576207679 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5562216b77d0 con 0x556221974870 2024-03-28T16:06:50.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.618+0000 7f8d2e800640 1 --2- 172.21.15.67:0/576207679 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556221973e80 0x55622197de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:06:50.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.618+0000 7f8d27e00640 1 --2- 172.21.15.67:0/576207679 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556221974870 0x55622197e370 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f8d1809aea0 tx=0x7f8d180ce860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5ff2ca7af9fa870 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:06:50.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.618+0000 7f8d13e00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:06:50.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.618+0000 7f8d13e00640 1 -- 172.21.15.67:0/576207679 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d18020050 con 0x556221974870 2024-03-28T16:06:50.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.618+0000 7f8d13e00640 1 -- 172.21.15.67:0/576207679 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f8d180d42e0 con 0x556221974870 2024-03-28T16:06:50.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.618+0000 7f8d381ec740 1 -- 172.21.15.67:0/576207679 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5562218e6790 con 0x556221974870 2024-03-28T16:06:50.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.619+0000 7f8d13e00640 1 -- 172.21.15.67:0/576207679 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8d18005180 con 0x556221974870 2024-03-28T16:06:50.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.619+0000 7f8d381ec740 10 client.15789 fetch_fsmap learned FSMap version 19 2024-03-28T16:06:50.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.619+0000 7f8d381ec740 1 -- 172.21.15.67:0/576207679 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5562215ed3f0 con 0x556221974870 2024-03-28T16:06:50.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.619+0000 7f8d13e00640 1 -- 172.21.15.67:0/576207679 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5562218e6790 con 0x556221974870 2024-03-28T16:06:50.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.619+0000 7f8d13e00640 1 -- 172.21.15.67:0/576207679 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f8d180b0020 con 0x556221974870 2024-03-28T16:06:50.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.619+0000 7f8d381ec740 10 client.15789 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:06:50.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.619+0000 7f8d381ec740 10 client.15789 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:06:50.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.619+0000 7f8d381ec740 20 client.15789 populate_metadata read hostname 'smithi067' 2024-03-28T16:06:50.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.619+0000 7f8d381ec740 1 --2- 172.21.15.67:0/576207679 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5562219898a0 0x55622198bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:50.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.619+0000 7f8d381ec740 4 client.15789 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:06:50.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.619+0000 7f8d381ec740 1 -- 172.21.15.67:0/576207679 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5562218e0ae0 con 0x5562219898a0 2024-03-28T16:06:50.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.620+0000 7f8d2e800640 1 --2- 172.21.15.67:0/576207679 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5562219898a0 0x55622198bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:50.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.621+0000 7f8d2e800640 1 --2- 172.21.15.67:0/576207679 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5562219898a0 0x55622198bc90 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba41f71898860479 server_cookie=3d68ce98620483df in_seq=0 out_seq=0 2024-03-28T16:06:50.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.621+0000 7f8d13e00640 10 client.15789 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:06:50.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.624+0000 7f8d13e00640 1 -- 172.21.15.67:0/576207679 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5562218e0ae0 con 0x5562219898a0 2024-03-28T16:06:50.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.624+0000 7f8d13e00640 10 client.15789 handle_command_reply: tid=0 2024-03-28T16:06:50.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.624+0000 7f8d381ec740 1 client.15789 shutdown 2024-03-28T16:06:50.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.625+0000 7f8d381ec740 1 -- 172.21.15.67:0/576207679 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556221974870 msgr2=0x55622197e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:50.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.625+0000 7f8d381ec740 1 --2- 172.21.15.67:0/576207679 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556221974870 0x55622197e370 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f8d1809aea0 tx=0x7f8d180ce860 comp rx=0 tx=0).stop 2024-03-28T16:06:50.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.625+0000 7f8d381ec740 1 -- 172.21.15.67:0/576207679 shutdown_connections 2024-03-28T16:06:50.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.625+0000 7f8d381ec740 1 --2- 172.21.15.67:0/576207679 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5562219898a0 0x55622198bc90 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:50.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.625+0000 7f8d381ec740 1 --2- 172.21.15.67:0/576207679 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5562219799e0 0x5562219853b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:50.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.625+0000 7f8d381ec740 1 --2- 172.21.15.67:0/576207679 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556221974870 0x55622197e370 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:50.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.625+0000 7f8d381ec740 1 --2- 172.21.15.67:0/576207679 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556221973e80 0x55622197de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:50.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.625+0000 7f8d13e00640 0 client.15789 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:06:50.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.625+0000 7f8d381ec740 1 -- 172.21.15.67:0/576207679 >> 172.21.15.67:0/576207679 conn(0x55622193c200 msgr2=0x55622196ea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:06:50.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.625+0000 7f8d381ec740 1 -- 172.21.15.67:0/576207679 shutdown_connections 2024-03-28T16:06:50.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.625+0000 7f8d381ec740 1 -- 172.21.15.67:0/576207679 wait complete. 2024-03-28T16:06:50.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.625+0000 7f8d381ec740 20 client.15789 trim_cache size 0 max 16384 2024-03-28T16:06:50.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.643+0000 7f8d381ec740 1 Processor -- start 2024-03-28T16:06:50.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.643+0000 7f8d381ec740 1 -- start start 2024-03-28T16:06:50.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.643+0000 7f8d381ec740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556221973e80 0x556221972a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:50.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.643+0000 7f8d381ec740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556221974870 0x556221985370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:50.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.643+0000 7f8d381ec740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5562219799e0 0x556221972f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:50.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.643+0000 7f8d381ec740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d301a8c10 con 0x556221974870 2024-03-28T16:06:50.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.643+0000 7f8d381ec740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d301ab1d0 con 0x5562219799e0 2024-03-28T16:06:50.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.643+0000 7f8d381ec740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d301a9210 con 0x556221973e80 2024-03-28T16:06:50.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.643+0000 7f8d27e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556221974870 0x556221985370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:50.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.643+0000 7f8d27e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556221974870 0x556221985370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36340/0 (socket says 172.21.15.67:36340) 2024-03-28T16:06:50.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.643+0000 7f8d27e00640 1 -- 172.21.15.67:0/877065294 learned_addr learned my addr 172.21.15.67:0/877065294 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:06:50.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.644+0000 7f8d2e800640 1 --2- 172.21.15.67:0/877065294 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556221973e80 0x556221972a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:50.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.644+0000 7f8d2f200640 1 --2- 172.21.15.67:0/877065294 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5562219799e0 0x556221972f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:50.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.644+0000 7f8d27e00640 1 -- 172.21.15.67:0/877065294 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556221973e80 msgr2=0x556221972a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:06:50.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.644+0000 7f8d27e00640 1 --2- 172.21.15.67:0/877065294 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556221973e80 0x556221972a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:50.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.644+0000 7f8d27e00640 1 -- 172.21.15.67:0/877065294 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5562219799e0 msgr2=0x556221972f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:06:50.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.644+0000 7f8d27e00640 1 --2- 172.21.15.67:0/877065294 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5562219799e0 0x556221972f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:50.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.644+0000 7f8d27e00640 1 -- 172.21.15.67:0/877065294 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5562215ad0f0 con 0x556221974870 2024-03-28T16:06:50.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.644+0000 7f8d2f200640 1 --2- 172.21.15.67:0/877065294 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5562219799e0 0x556221972f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:06:50.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.644+0000 7f8d27e00640 1 --2- 172.21.15.67:0/877065294 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556221974870 0x556221985370 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f8d1800d740 tx=0x7f8d180ce8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff46ef2716e62fe8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:06:50.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.644+0000 7f8d1ea00640 1 -- 172.21.15.67:0/877065294 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d18020070 con 0x556221974870 2024-03-28T16:06:50.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.644+0000 7f8d1ea00640 1 -- 172.21.15.67:0/877065294 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8d1809b610 con 0x556221974870 2024-03-28T16:06:50.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.644+0000 7f8d1ea00640 1 -- 172.21.15.67:0/877065294 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d1809a870 con 0x556221974870 2024-03-28T16:06:50.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.645+0000 7f8d381ec740 1 -- 172.21.15.67:0/877065294 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556221974870 msgr2=0x556221985370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:50.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.645+0000 7f8d381ec740 1 --2- 172.21.15.67:0/877065294 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556221974870 0x556221985370 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f8d1800d740 tx=0x7f8d180ce8d0 comp rx=0 tx=0).stop 2024-03-28T16:06:50.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.645+0000 7f8d381ec740 1 -- 172.21.15.67:0/877065294 shutdown_connections 2024-03-28T16:06:50.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.645+0000 7f8d381ec740 1 --2- 172.21.15.67:0/877065294 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5562219799e0 0x556221972f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:50.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.645+0000 7f8d381ec740 1 --2- 172.21.15.67:0/877065294 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556221974870 0x556221985370 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:50.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.645+0000 7f8d381ec740 1 --2- 172.21.15.67:0/877065294 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556221973e80 0x556221972a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:50.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.645+0000 7f8d381ec740 1 -- 172.21.15.67:0/877065294 >> 172.21.15.67:0/877065294 conn(0x55622193c200 msgr2=0x556221975310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:06:50.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.645+0000 7f8d381ec740 1 -- 172.21.15.67:0/877065294 shutdown_connections 2024-03-28T16:06:50.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.645+0000 7f8d381ec740 1 -- 172.21.15.67:0/877065294 wait complete. 2024-03-28T16:06:50.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.646+0000 7f8d381ec740 1 -- start start 2024-03-28T16:06:50.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.646+0000 7f8d381ec740 1 Processor -- start 2024-03-28T16:06:50.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.646+0000 7f8d381ec740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556221973e80 0x556221982f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:50.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.646+0000 7f8d381ec740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556221974870 0x5562219834c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:50.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.646+0000 7f8d381ec740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5562219799e0 0x556221984480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:50.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.646+0000 7f8d381ec740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d301a9850 con 0x5562219799e0 2024-03-28T16:06:50.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.646+0000 7f8d381ec740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d301aab70 con 0x556221974870 2024-03-28T16:06:50.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.646+0000 7f8d381ec740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d301a9eb0 con 0x556221973e80 2024-03-28T16:06:50.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.646+0000 7f8d381ec740 10 client.0 _ll_register_callbacks cb 0x7f8d180df070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:06:50.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.647+0000 7f8d2f200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5562219799e0 0x556221984480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:50.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.647+0000 7f8d2f200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5562219799e0 0x556221984480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36352/0 (socket says 172.21.15.67:36352) 2024-03-28T16:06:50.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.647+0000 7f8d2f200640 1 -- 172.21.15.67:0/2654466644 learned_addr learned my addr 172.21.15.67:0/2654466644 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:06:50.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.647+0000 7f8d2e800640 1 --2- 172.21.15.67:0/2654466644 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556221973e80 0x556221982f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:50.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.647+0000 7f8d27e00640 1 --2- 172.21.15.67:0/2654466644 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556221974870 0x5562219834c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:50.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.647+0000 7f8d2f200640 1 -- 172.21.15.67:0/2654466644 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556221973e80 msgr2=0x556221982f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:06:50.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.647+0000 7f8d2f200640 1 --2- 172.21.15.67:0/2654466644 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556221973e80 0x556221982f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:50.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.647+0000 7f8d2f200640 1 -- 172.21.15.67:0/2654466644 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556221974870 msgr2=0x5562219834c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:06:50.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.647+0000 7f8d2f200640 1 --2- 172.21.15.67:0/2654466644 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556221974870 0x5562219834c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:50.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.647+0000 7f8d2f200640 1 -- 172.21.15.67:0/2654466644 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556221984a70 con 0x5562219799e0 2024-03-28T16:06:50.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.647+0000 7f8d2e800640 1 --2- 172.21.15.67:0/2654466644 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556221973e80 0x556221982f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T16:06:50.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.648+0000 7f8d2f200640 1 --2- 172.21.15.67:0/2654466644 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5562219799e0 0x556221984480 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f8d28036000 tx=0x7f8d2802c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2eea16335940a87 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:06:50.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.648+0000 7f8d13e00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:06:50.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.648+0000 7f8d13e00640 1 -- 172.21.15.67:0/2654466644 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d28050030 con 0x5562219799e0 2024-03-28T16:06:50.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.648+0000 7f8d13e00640 1 -- 172.21.15.67:0/2654466644 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f8d28003fb0 con 0x5562219799e0 2024-03-28T16:06:50.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.648+0000 7f8d381ec740 1 -- 172.21.15.67:0/2654466644 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8d1809b610 con 0x5562219799e0 2024-03-28T16:06:50.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.648+0000 7f8d13e00640 1 -- 172.21.15.67:0/2654466644 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8d28039040 con 0x5562219799e0 2024-03-28T16:06:50.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.649+0000 7f8d381ec740 10 client.15795 fetch_fsmap learned FSMap version 19 2024-03-28T16:06:50.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.649+0000 7f8d381ec740 1 -- 172.21.15.67:0/2654466644 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5562219e5520 con 0x5562219799e0 2024-03-28T16:06:50.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.649+0000 7f8d13e00640 1 -- 172.21.15.67:0/2654466644 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8d1809b610 con 0x5562219799e0 2024-03-28T16:06:50.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.649+0000 7f8d13e00640 1 -- 172.21.15.67:0/2654466644 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f8d2804a040 con 0x5562219799e0 2024-03-28T16:06:50.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.649+0000 7f8d381ec740 10 client.15795 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:06:50.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.649+0000 7f8d381ec740 10 client.15795 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:06:50.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.649+0000 7f8d381ec740 20 client.15795 populate_metadata read hostname 'smithi067' 2024-03-28T16:06:50.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.649+0000 7f8d381ec740 1 --2- 172.21.15.67:0/2654466644 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5562219e5f80 0x5562219e83a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:06:50.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.649+0000 7f8d381ec740 4 client.15795 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T16:06:50.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.649+0000 7f8d381ec740 1 -- 172.21.15.67:0/2654466644 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55622185e3a0 con 0x5562219e5f80 2024-03-28T16:06:50.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.650+0000 7f8d2e800640 1 --2- 172.21.15.67:0/2654466644 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5562219e5f80 0x5562219e83a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:06:50.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.651+0000 7f8d2e800640 1 --2- 172.21.15.67:0/2654466644 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5562219e5f80 0x5562219e83a0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=669b2bef105b071 server_cookie=a6c77bd04a07f263 in_seq=0 out_seq=0 2024-03-28T16:06:50.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.651+0000 7f8d13e00640 10 client.15795 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:06:50.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.655+0000 7f8d13e00640 1 -- 172.21.15.67:0/2654466644 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55622185e3a0 con 0x5562219e5f80 2024-03-28T16:06:50.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.655+0000 7f8d13e00640 10 client.15795 handle_command_reply: tid=0 2024-03-28T16:06:50.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.655+0000 7f8d381ec740 1 client.15795 shutdown 2024-03-28T16:06:50.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.655+0000 7f8d381ec740 1 -- 172.21.15.67:0/2654466644 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5562219799e0 msgr2=0x556221984480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:50.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.655+0000 7f8d381ec740 1 --2- 172.21.15.67:0/2654466644 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5562219799e0 0x556221984480 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f8d28036000 tx=0x7f8d2802c0f0 comp rx=0 tx=0).stop 2024-03-28T16:06:50.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.655+0000 7f8d381ec740 1 -- 172.21.15.67:0/2654466644 shutdown_connections 2024-03-28T16:06:50.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.655+0000 7f8d381ec740 1 --2- 172.21.15.67:0/2654466644 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5562219e5f80 0x5562219e83a0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:50.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.655+0000 7f8d381ec740 1 --2- 172.21.15.67:0/2654466644 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5562219799e0 0x556221984480 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:50.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.655+0000 7f8d381ec740 1 --2- 172.21.15.67:0/2654466644 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556221974870 0x5562219834c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:50.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.655+0000 7f8d13e00640 0 client.15795 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:06:50.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.655+0000 7f8d381ec740 1 --2- 172.21.15.67:0/2654466644 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556221973e80 0x556221982f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:50.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.655+0000 7f8d381ec740 1 -- 172.21.15.67:0/2654466644 >> 172.21.15.67:0/2654466644 conn(0x55622193c200 msgr2=0x55622196e200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:06:50.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.655+0000 7f8d381ec740 1 -- 172.21.15.67:0/2654466644 shutdown_connections 2024-03-28T16:06:50.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.655+0000 7f8d381ec740 1 -- 172.21.15.67:0/2654466644 wait complete. 2024-03-28T16:06:50.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.656+0000 7f8d381ec740 20 client.15795 trim_cache size 0 max 16384 2024-03-28T16:06:50.657 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-28T16:06:50.657 INFO:teuthology.orchestra.run.smithi067.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T16:06:50.657 INFO:teuthology.orchestra.run.smithi067.stdout: "scrubs": { 2024-03-28T16:06:50.657 INFO:teuthology.orchestra.run.smithi067.stdout: "c40b92db-8aef-4c20-9b6f-37849bc69d92": { 2024-03-28T16:06:50.657 INFO:teuthology.orchestra.run.smithi067.stdout: "path": "/", 2024-03-28T16:06:50.657 INFO:teuthology.orchestra.run.smithi067.stdout: "tag": "c40b92db-8aef-4c20-9b6f-37849bc69d92", 2024-03-28T16:06:50.657 INFO:teuthology.orchestra.run.smithi067.stdout: "options": "recursive,force" 2024-03-28T16:06:50.657 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:06:50.657 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:06:50.657 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-28T16:06:50.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.656+0000 7f8d35400640 1 -- 172.21.15.67:0/4056711640 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f8d140a2800 msgr2=0x7f8d140a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:50.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.656+0000 7f8d35400640 1 --2- 172.21.15.67:0/4056711640 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f8d140a2800 0x7f8d140a4cc0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f8d28032420 tx=0x7f8d28005f80 comp rx=0 tx=0).stop 2024-03-28T16:06:50.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.656+0000 7f8d35400640 1 -- 172.21.15.67:0/4056711640 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d30104ab0 msgr2=0x7f8d30199fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:06:50.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.656+0000 7f8d35400640 1 --2- 172.21.15.67:0/4056711640 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d30104ab0 0x7f8d30199fd0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f8d1800d770 tx=0x7f8d1800dc30 comp rx=0 tx=0).stop 2024-03-28T16:06:50.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.656+0000 7f8d35400640 1 -- 172.21.15.67:0/4056711640 shutdown_connections 2024-03-28T16:06:50.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.656+0000 7f8d35400640 1 --2- 172.21.15.67:0/4056711640 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f8d140a2800 0x7f8d140a4cc0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:50.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.656+0000 7f8d35400640 1 --2- 172.21.15.67:0/4056711640 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8d30105470 0x7f8d301a1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:50.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.656+0000 7f8d35400640 1 --2- 172.21.15.67:0/4056711640 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d30104ab0 0x7f8d30199fd0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:50.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.656+0000 7f8d35400640 1 --2- 172.21.15.67:0/4056711640 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d301038c0 0x7f8d30199a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:06:50.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.656+0000 7f8d35400640 1 -- 172.21.15.67:0/4056711640 >> 172.21.15.67:0/4056711640 conn(0x7f8d300fee50 msgr2=0x7f8d30107e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:06:50.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.657+0000 7f8d35400640 1 -- 172.21.15.67:0/4056711640 shutdown_connections 2024-03-28T16:06:50.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:06:50.657+0000 7f8d35400640 1 -- 172.21.15.67:0/4056711640 wait complete. 2024-03-28T16:06:50.675 INFO:tasks.cephfs.filesystem:scrub status for tag:c40b92db-8aef-4c20-9b6f-37849bc69d92 - {'path': '/', 'tag': 'c40b92db-8aef-4c20-9b6f-37849bc69d92', 'options': 'recursive,force'} 2024-03-28T16:06:51.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:06:50 smithi150 ceph-mon[34825]: pgmap v537: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 15 MiB/s wr, 5 op/s 2024-03-28T16:06:51.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:06:50 smithi067 ceph-mon[31707]: pgmap v537: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 15 MiB/s wr, 5 op/s 2024-03-28T16:06:51.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:06:50 smithi183 ceph-mon[35580]: pgmap v537: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 15 MiB/s wr, 5 op/s 2024-03-28T16:06:52.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:06:51 smithi150 ceph-mon[34825]: scrub queued for path: / 2024-03-28T16:06:52.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:06:51 smithi150 ceph-mon[34825]: scrub summary: idle+waiting paths [/] 2024-03-28T16:06:52.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:06:51 smithi150 ceph-mon[34825]: scrub summary: active paths [/] 2024-03-28T16:06:52.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:06:51 smithi150 ceph-mon[34825]: scrub summary: idle+waiting paths [/] 2024-03-28T16:06:52.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:06:51 smithi067 ceph-mon[31707]: scrub queued for path: / 2024-03-28T16:06:52.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:06:51 smithi067 ceph-mon[31707]: scrub summary: idle+waiting paths [/] 2024-03-28T16:06:52.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:06:51 smithi067 ceph-mon[31707]: scrub summary: active paths [/] 2024-03-28T16:06:52.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:06:51 smithi067 ceph-mon[31707]: scrub summary: idle+waiting paths [/] 2024-03-28T16:06:52.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:06:51 smithi183 ceph-mon[35580]: scrub queued for path: / 2024-03-28T16:06:52.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:06:51 smithi183 ceph-mon[35580]: scrub summary: idle+waiting paths [/] 2024-03-28T16:06:52.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:06:51 smithi183 ceph-mon[35580]: scrub summary: active paths [/] 2024-03-28T16:06:52.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:06:51 smithi183 ceph-mon[35580]: scrub summary: idle+waiting paths [/] 2024-03-28T16:06:53.024 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-03-28T16:06:53.024 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-03-28T16:06:53.025 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-03-28T16:06:53.025 INFO:tasks.workunit.client.0.smithi067.stdout:iozone test complete. 2024-03-28T16:06:53.028 INFO:tasks.workunit.client.0.smithi067.stderr:+ iozone -c -e -s 1024M -r 1M -t 1 -F f2 -i 0 -i 1 2024-03-28T16:06:53.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:06:52 smithi150 ceph-mon[34825]: pgmap v538: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 11 MiB/s wr, 5 op/s 2024-03-28T16:06:53.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:06:52 smithi150 ceph-mon[34825]: scrub summary: idle 2024-03-28T16:06:53.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:06:52 smithi067 ceph-mon[31707]: pgmap v538: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 11 MiB/s wr, 5 op/s 2024-03-28T16:06:53.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:06:52 smithi067 ceph-mon[31707]: scrub summary: idle 2024-03-28T16:06:53.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:06:52 smithi183 ceph-mon[35580]: pgmap v538: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 11 MiB/s wr, 5 op/s 2024-03-28T16:06:53.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:06:52 smithi183 ceph-mon[35580]: scrub summary: idle 2024-03-28T16:06:53.119 INFO:tasks.workunit.client.0.smithi067.stdout: Iozone: Performance Test of File I/O 2024-03-28T16:06:53.119 INFO:tasks.workunit.client.0.smithi067.stdout: Version $Revision: 3.494 $ 2024-03-28T16:06:53.119 INFO:tasks.workunit.client.0.smithi067.stdout: Compiled for 64 bit mode. 2024-03-28T16:06:53.119 INFO:tasks.workunit.client.0.smithi067.stdout: Build: linux-AMD64 2024-03-28T16:06:53.119 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-03-28T16:06:53.119 INFO:tasks.workunit.client.0.smithi067.stdout: Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins 2024-03-28T16:06:53.119 INFO:tasks.workunit.client.0.smithi067.stdout: Al Slater, Scott Rhine, Mike Wisner, Ken Goss 2024-03-28T16:06:53.119 INFO:tasks.workunit.client.0.smithi067.stdout: Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, 2024-03-28T16:06:53.119 INFO:tasks.workunit.client.0.smithi067.stdout: Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, 2024-03-28T16:06:53.119 INFO:tasks.workunit.client.0.smithi067.stdout: Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone, 2024-03-28T16:06:53.119 INFO:tasks.workunit.client.0.smithi067.stdout: Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root, 2024-03-28T16:06:53.119 INFO:tasks.workunit.client.0.smithi067.stdout: Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer, 2024-03-28T16:06:53.119 INFO:tasks.workunit.client.0.smithi067.stdout: Vangel Bojaxhi, Ben England, Vikentsi Lapa, 2024-03-28T16:06:53.120 INFO:tasks.workunit.client.0.smithi067.stdout: Alexey Skidanov, Sudhir Kumar. 2024-03-28T16:06:53.120 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-03-28T16:06:53.120 INFO:tasks.workunit.client.0.smithi067.stdout: Run began: Thu Mar 28 16:06:53 2024 2024-03-28T16:06:53.120 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-03-28T16:06:53.120 INFO:tasks.workunit.client.0.smithi067.stdout: Include close in write timing 2024-03-28T16:06:53.120 INFO:tasks.workunit.client.0.smithi067.stdout: Include fsync in write timing 2024-03-28T16:06:53.120 INFO:tasks.workunit.client.0.smithi067.stdout: File size set to 1048576 kB 2024-03-28T16:06:53.120 INFO:tasks.workunit.client.0.smithi067.stdout: Record Size 1024 kB 2024-03-28T16:06:53.120 INFO:tasks.workunit.client.0.smithi067.stdout: Command line used: iozone -c -e -s 1024M -r 1M -t 1 -F f2 -i 0 -i 1 2024-03-28T16:06:53.120 INFO:tasks.workunit.client.0.smithi067.stdout: Output is in kBytes/sec 2024-03-28T16:06:53.120 INFO:tasks.workunit.client.0.smithi067.stdout: Time Resolution = 0.000001 seconds. 2024-03-28T16:06:53.120 INFO:tasks.workunit.client.0.smithi067.stdout: Processor cache size set to 1024 kBytes. 2024-03-28T16:06:53.120 INFO:tasks.workunit.client.0.smithi067.stdout: Processor cache line size set to 32 bytes. 2024-03-28T16:06:53.120 INFO:tasks.workunit.client.0.smithi067.stdout: File stride size set to 17 * record size. 2024-03-28T16:06:53.120 INFO:tasks.workunit.client.0.smithi067.stdout: Throughput test with 1 process 2024-03-28T16:06:53.121 INFO:tasks.workunit.client.0.smithi067.stdout: Each process writes a 1048576 kByte file in 1024 kByte records 2024-03-28T16:06:55.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:06:54 smithi150 ceph-mon[34825]: pgmap v539: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 6.0 MiB/s wr, 3 op/s 2024-03-28T16:06:55.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:06:54 smithi067 ceph-mon[31707]: pgmap v539: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 6.0 MiB/s wr, 3 op/s 2024-03-28T16:06:55.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:06:54 smithi183 ceph-mon[35580]: pgmap v539: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 6.0 MiB/s wr, 3 op/s 2024-03-28T16:06:57.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:06:56 smithi150 ceph-mon[34825]: pgmap v540: 97 pgs: 97 active+clean; 709 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 11 MiB/s wr, 16 op/s 2024-03-28T16:06:57.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:06:56 smithi067 ceph-mon[31707]: pgmap v540: 97 pgs: 97 active+clean; 709 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 11 MiB/s wr, 16 op/s 2024-03-28T16:06:57.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:06:56 smithi183 ceph-mon[35580]: pgmap v540: 97 pgs: 97 active+clean; 709 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 11 MiB/s wr, 16 op/s 2024-03-28T16:06:59.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:06:58 smithi150 ceph-mon[34825]: pgmap v541: 97 pgs: 97 active+clean; 601 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 19 MiB/s wr, 21 op/s 2024-03-28T16:06:59.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:06:58 smithi067 ceph-mon[31707]: pgmap v541: 97 pgs: 97 active+clean; 601 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 19 MiB/s wr, 21 op/s 2024-03-28T16:06:59.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:06:58 smithi183 ceph-mon[35580]: pgmap v541: 97 pgs: 97 active+clean; 601 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 19 MiB/s wr, 21 op/s 2024-03-28T16:07:00.103 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-03-28T16:07:00.103 INFO:tasks.workunit.client.0.smithi067.stdout: Children see throughput for 1 initial writers = 151006.39 kB/sec 2024-03-28T16:07:00.103 INFO:tasks.workunit.client.0.smithi067.stdout: Parent sees throughput for 1 initial writers = 150969.54 kB/sec 2024-03-28T16:07:00.103 INFO:tasks.workunit.client.0.smithi067.stdout: Min throughput per process = 151006.39 kB/sec 2024-03-28T16:07:00.103 INFO:tasks.workunit.client.0.smithi067.stdout: Max throughput per process = 151006.39 kB/sec 2024-03-28T16:07:00.103 INFO:tasks.workunit.client.0.smithi067.stdout: Avg throughput per process = 151006.39 kB/sec 2024-03-28T16:07:00.103 INFO:tasks.workunit.client.0.smithi067.stdout: Min xfer = 1048576.00 kB 2024-03-28T16:07:01.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:07:00 smithi150 ceph-mon[34825]: pgmap v542: 97 pgs: 97 active+clean; 469 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 39 MiB/s wr, 35 op/s 2024-03-28T16:07:01.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:07:00 smithi067 ceph-mon[31707]: pgmap v542: 97 pgs: 97 active+clean; 469 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 39 MiB/s wr, 35 op/s 2024-03-28T16:07:01.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:07:00 smithi183 ceph-mon[35580]: pgmap v542: 97 pgs: 97 active+clean; 469 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 39 MiB/s wr, 35 op/s 2024-03-28T16:07:03.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:07:02 smithi150 ceph-mon[34825]: pgmap v543: 97 pgs: 97 active+clean; 845 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 70 MiB/s wr, 44 op/s 2024-03-28T16:07:03.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:07:02 smithi067 ceph-mon[31707]: pgmap v543: 97 pgs: 97 active+clean; 845 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 70 MiB/s wr, 44 op/s 2024-03-28T16:07:03.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:07:02 smithi183 ceph-mon[35580]: pgmap v543: 97 pgs: 97 active+clean; 845 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 70 MiB/s wr, 44 op/s 2024-03-28T16:07:04.830 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-03-28T16:07:04.830 INFO:tasks.workunit.client.0.smithi067.stdout: Children see throughput for 1 rewriters = 437750.06 kB/sec 2024-03-28T16:07:04.831 INFO:tasks.workunit.client.0.smithi067.stdout: Parent sees throughput for 1 rewriters = 437533.25 kB/sec 2024-03-28T16:07:04.831 INFO:tasks.workunit.client.0.smithi067.stdout: Min throughput per process = 437750.06 kB/sec 2024-03-28T16:07:04.831 INFO:tasks.workunit.client.0.smithi067.stdout: Max throughput per process = 437750.06 kB/sec 2024-03-28T16:07:04.831 INFO:tasks.workunit.client.0.smithi067.stdout: Avg throughput per process = 437750.06 kB/sec 2024-03-28T16:07:04.831 INFO:tasks.workunit.client.0.smithi067.stdout: Min xfer = 1048576.00 kB 2024-03-28T16:07:05.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:07:04 smithi150 ceph-mon[34825]: pgmap v544: 97 pgs: 97 active+clean; 1005 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 87 MiB/s wr, 47 op/s 2024-03-28T16:07:05.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:07:04 smithi067 ceph-mon[31707]: pgmap v544: 97 pgs: 97 active+clean; 1005 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 87 MiB/s wr, 47 op/s 2024-03-28T16:07:05.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:07:04 smithi183 ceph-mon[35580]: pgmap v544: 97 pgs: 97 active+clean; 1005 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 87 MiB/s wr, 47 op/s 2024-03-28T16:07:07.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:07:06 smithi150 ceph-mon[34825]: pgmap v545: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 122 MiB/s wr, 57 op/s 2024-03-28T16:07:07.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:07:06 smithi067 ceph-mon[31707]: pgmap v545: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 122 MiB/s wr, 57 op/s 2024-03-28T16:07:07.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:07:06 smithi183 ceph-mon[35580]: pgmap v545: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 122 MiB/s wr, 57 op/s 2024-03-28T16:07:07.313 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-03-28T16:07:07.313 INFO:tasks.workunit.client.0.smithi067.stdout: Children see throughput for 1 readers = 4983487.50 kB/sec 2024-03-28T16:07:07.313 INFO:tasks.workunit.client.0.smithi067.stdout: Parent sees throughput for 1 readers = 4961298.76 kB/sec 2024-03-28T16:07:07.313 INFO:tasks.workunit.client.0.smithi067.stdout: Min throughput per process = 4983487.50 kB/sec 2024-03-28T16:07:07.314 INFO:tasks.workunit.client.0.smithi067.stdout: Max throughput per process = 4983487.50 kB/sec 2024-03-28T16:07:07.314 INFO:tasks.workunit.client.0.smithi067.stdout: Avg throughput per process = 4983487.50 kB/sec 2024-03-28T16:07:07.314 INFO:tasks.workunit.client.0.smithi067.stdout: Min xfer = 1048576.00 kB 2024-03-28T16:07:09.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:07:08 smithi067 ceph-mon[31707]: pgmap v546: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 134 MiB/s wr, 49 op/s 2024-03-28T16:07:09.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:07:08 smithi183 ceph-mon[35580]: pgmap v546: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 134 MiB/s wr, 49 op/s 2024-03-28T16:07:09.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:07:08 smithi150 ceph-mon[34825]: pgmap v546: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 134 MiB/s wr, 49 op/s 2024-03-28T16:07:09.795 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-03-28T16:07:09.795 INFO:tasks.workunit.client.0.smithi067.stdout: Children see throughput for 1 re-readers = 3583809.50 kB/sec 2024-03-28T16:07:09.795 INFO:tasks.workunit.client.0.smithi067.stdout: Parent sees throughput for 1 re-readers = 3571446.60 kB/sec 2024-03-28T16:07:09.795 INFO:tasks.workunit.client.0.smithi067.stdout: Min throughput per process = 3583809.50 kB/sec 2024-03-28T16:07:09.795 INFO:tasks.workunit.client.0.smithi067.stdout: Max throughput per process = 3583809.50 kB/sec 2024-03-28T16:07:09.795 INFO:tasks.workunit.client.0.smithi067.stdout: Avg throughput per process = 3583809.50 kB/sec 2024-03-28T16:07:09.795 INFO:tasks.workunit.client.0.smithi067.stdout: Min xfer = 1048576.00 kB 2024-03-28T16:07:11.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:07:10 smithi067 ceph-mon[31707]: pgmap v547: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 150 MiB/s wr, 48 op/s 2024-03-28T16:07:11.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:07:10 smithi183 ceph-mon[35580]: pgmap v547: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 150 MiB/s wr, 48 op/s 2024-03-28T16:07:11.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:07:10 smithi150 ceph-mon[34825]: pgmap v547: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 150 MiB/s wr, 48 op/s 2024-03-28T16:07:13.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:07:12 smithi067 ceph-mon[31707]: pgmap v548: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 132 MiB/s wr, 36 op/s 2024-03-28T16:07:13.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:07:12 smithi183 ceph-mon[35580]: pgmap v548: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 132 MiB/s wr, 36 op/s 2024-03-28T16:07:13.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:07:12 smithi150 ceph-mon[34825]: pgmap v548: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 132 MiB/s wr, 36 op/s 2024-03-28T16:07:15.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:07:14 smithi183 ceph-mon[35580]: pgmap v549: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 100 MiB/s wr, 27 op/s 2024-03-28T16:07:15.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:07:14 smithi150 ceph-mon[34825]: pgmap v549: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 100 MiB/s wr, 27 op/s 2024-03-28T16:07:15.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:07:14 smithi067 ceph-mon[31707]: pgmap v549: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 100 MiB/s wr, 27 op/s 2024-03-28T16:07:16.539 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-03-28T16:07:16.539 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-03-28T16:07:16.539 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-03-28T16:07:16.539 INFO:tasks.workunit.client.0.smithi067.stdout:iozone test complete. 2024-03-28T16:07:16.546 INFO:tasks.workunit.client.0.smithi067.stderr:+ iozone -c -e -s 10240M -r 1M -t 1 -F f3 -i 0 -i 1 2024-03-28T16:07:16.637 INFO:tasks.workunit.client.0.smithi067.stdout: Iozone: Performance Test of File I/O 2024-03-28T16:07:16.637 INFO:tasks.workunit.client.0.smithi067.stdout: Version $Revision: 3.494 $ 2024-03-28T16:07:16.637 INFO:tasks.workunit.client.0.smithi067.stdout: Compiled for 64 bit mode. 2024-03-28T16:07:16.637 INFO:tasks.workunit.client.0.smithi067.stdout: Build: linux-AMD64 2024-03-28T16:07:16.637 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-03-28T16:07:16.637 INFO:tasks.workunit.client.0.smithi067.stdout: Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins 2024-03-28T16:07:16.637 INFO:tasks.workunit.client.0.smithi067.stdout: Al Slater, Scott Rhine, Mike Wisner, Ken Goss 2024-03-28T16:07:16.637 INFO:tasks.workunit.client.0.smithi067.stdout: Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, 2024-03-28T16:07:16.637 INFO:tasks.workunit.client.0.smithi067.stdout: Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, 2024-03-28T16:07:16.638 INFO:tasks.workunit.client.0.smithi067.stdout: Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone, 2024-03-28T16:07:16.638 INFO:tasks.workunit.client.0.smithi067.stdout: Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root, 2024-03-28T16:07:16.638 INFO:tasks.workunit.client.0.smithi067.stdout: Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer, 2024-03-28T16:07:16.638 INFO:tasks.workunit.client.0.smithi067.stdout: Vangel Bojaxhi, Ben England, Vikentsi Lapa, 2024-03-28T16:07:16.638 INFO:tasks.workunit.client.0.smithi067.stdout: Alexey Skidanov, Sudhir Kumar. 2024-03-28T16:07:16.638 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-03-28T16:07:16.638 INFO:tasks.workunit.client.0.smithi067.stdout: Run began: Thu Mar 28 16:07:16 2024 2024-03-28T16:07:16.638 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-03-28T16:07:16.638 INFO:tasks.workunit.client.0.smithi067.stdout: Include close in write timing 2024-03-28T16:07:16.638 INFO:tasks.workunit.client.0.smithi067.stdout: Include fsync in write timing 2024-03-28T16:07:16.638 INFO:tasks.workunit.client.0.smithi067.stdout: File size set to 10485760 kB 2024-03-28T16:07:16.638 INFO:tasks.workunit.client.0.smithi067.stdout: Record Size 1024 kB 2024-03-28T16:07:16.638 INFO:tasks.workunit.client.0.smithi067.stdout: Command line used: iozone -c -e -s 10240M -r 1M -t 1 -F f3 -i 0 -i 1 2024-03-28T16:07:16.638 INFO:tasks.workunit.client.0.smithi067.stdout: Output is in kBytes/sec 2024-03-28T16:07:16.639 INFO:tasks.workunit.client.0.smithi067.stdout: Time Resolution = 0.000001 seconds. 2024-03-28T16:07:16.639 INFO:tasks.workunit.client.0.smithi067.stdout: Processor cache size set to 1024 kBytes. 2024-03-28T16:07:16.639 INFO:tasks.workunit.client.0.smithi067.stdout: Processor cache line size set to 32 bytes. 2024-03-28T16:07:16.639 INFO:tasks.workunit.client.0.smithi067.stdout: File stride size set to 17 * record size. 2024-03-28T16:07:16.639 INFO:tasks.workunit.client.0.smithi067.stdout: Throughput test with 1 process 2024-03-28T16:07:16.639 INFO:tasks.workunit.client.0.smithi067.stdout: Each process writes a 10485760 kByte file in 1024 kByte records 2024-03-28T16:07:17.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:07:16 smithi183 ceph-mon[35580]: pgmap v550: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 84 MiB/s wr, 23 op/s 2024-03-28T16:07:17.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:07:16 smithi150 ceph-mon[34825]: pgmap v550: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 84 MiB/s wr, 23 op/s 2024-03-28T16:07:17.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:07:16 smithi067 ceph-mon[31707]: pgmap v550: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 84 MiB/s wr, 23 op/s 2024-03-28T16:07:19.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:07:18 smithi183 ceph-mon[35580]: pgmap v551: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 49 MiB/s wr, 14 op/s 2024-03-28T16:07:19.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:07:18 smithi150 ceph-mon[34825]: pgmap v551: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 49 MiB/s wr, 14 op/s 2024-03-28T16:07:19.322 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:07:18 smithi067 ceph-mon[31707]: pgmap v551: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 49 MiB/s wr, 14 op/s 2024-03-28T16:07:20.676 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T16:07:20.677 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T16:07:20.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.946+0000 7f9d99e00640 1 -- 172.21.15.67:0/2166934702 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d94104a70 msgr2=0x7f9d94106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:20.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.946+0000 7f9d99e00640 1 --2- 172.21.15.67:0/2166934702 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d94104a70 0x7f9d94106e60 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f9d7c009900 tx=0x7f9d7c02bc30 comp rx=0 tx=0).stop 2024-03-28T16:07:20.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.946+0000 7f9d99e00640 1 -- 172.21.15.67:0/2166934702 shutdown_connections 2024-03-28T16:07:20.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.946+0000 7f9d99e00640 1 --2- 172.21.15.67:0/2166934702 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9d941073a0 0x7f9d941097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:20.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.946+0000 7f9d99e00640 1 --2- 172.21.15.67:0/2166934702 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d94104a70 0x7f9d94106e60 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:20.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.946+0000 7f9d99e00640 1 --2- 172.21.15.67:0/2166934702 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d94102140 0x7f9d94104530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:20.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.946+0000 7f9d99e00640 1 -- 172.21.15.67:0/2166934702 >> 172.21.15.67:0/2166934702 conn(0x7f9d940fbb30 msgr2=0x7f9d940fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:20.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.946+0000 7f9d99e00640 1 -- 172.21.15.67:0/2166934702 shutdown_connections 2024-03-28T16:07:20.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.946+0000 7f9d99e00640 1 -- 172.21.15.67:0/2166934702 wait complete. 2024-03-28T16:07:20.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.947+0000 7f9d99e00640 1 Processor -- start 2024-03-28T16:07:20.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.947+0000 7f9d99e00640 1 -- start start 2024-03-28T16:07:20.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.948+0000 7f9d99e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d94102140 0x7f9d94101ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:20.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.948+0000 7f9d99e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9d94104a70 0x7f9d941000f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:20.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.948+0000 7f9d99e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d941073a0 0x7f9d94100630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:20.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.948+0000 7f9d99e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9410d230 con 0x7f9d941073a0 2024-03-28T16:07:20.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.948+0000 7f9d99e00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9410d0b0 con 0x7f9d94102140 2024-03-28T16:07:20.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.948+0000 7f9d99e00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9410d3b0 con 0x7f9d94104a70 2024-03-28T16:07:20.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.948+0000 7f9d93400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d941073a0 0x7f9d94100630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:20.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.948+0000 7f9d93400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d941073a0 0x7f9d94100630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58656/0 (socket says 172.21.15.67:58656) 2024-03-28T16:07:20.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.948+0000 7f9d93400640 1 -- 172.21.15.67:0/2660303722 learned_addr learned my addr 172.21.15.67:0/2660303722 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:07:20.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.948+0000 7f9d8a000640 1 --2- 172.21.15.67:0/2660303722 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9d94104a70 0x7f9d941000f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:20.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.948+0000 7f9d92a00640 1 --2- 172.21.15.67:0/2660303722 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d94102140 0x7f9d94101ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:20.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.949+0000 7f9d93400640 1 -- 172.21.15.67:0/2660303722 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9d94104a70 msgr2=0x7f9d941000f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:20.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.949+0000 7f9d93400640 1 --2- 172.21.15.67:0/2660303722 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9d94104a70 0x7f9d941000f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:20.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.949+0000 7f9d93400640 1 -- 172.21.15.67:0/2660303722 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d94102140 msgr2=0x7f9d94101ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:20.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.949+0000 7f9d93400640 1 --2- 172.21.15.67:0/2660303722 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d94102140 0x7f9d94101ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:20.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.949+0000 7f9d93400640 1 -- 172.21.15.67:0/2660303722 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d94100ec0 con 0x7f9d941073a0 2024-03-28T16:07:20.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.949+0000 7f9d8a000640 1 --2- 172.21.15.67:0/2660303722 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9d94104a70 0x7f9d941000f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T16:07:20.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.949+0000 7f9d93400640 1 --2- 172.21.15.67:0/2660303722 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d941073a0 0x7f9d94100630 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c00d750 tx=0x7f9d8c00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:20.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.949+0000 7f9d98a00640 1 -- 172.21.15.67:0/2660303722 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d8c017070 con 0x7f9d941073a0 2024-03-28T16:07:20.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.949+0000 7f9d98a00640 1 -- 172.21.15.67:0/2660303722 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9d8c0054d0 con 0x7f9d941073a0 2024-03-28T16:07:20.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.949+0000 7f9d98a00640 1 -- 172.21.15.67:0/2660303722 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d8c004c60 con 0x7f9d941073a0 2024-03-28T16:07:20.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.949+0000 7f9d99e00640 1 -- 172.21.15.67:0/2660303722 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d941a1ab0 con 0x7f9d941073a0 2024-03-28T16:07:20.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.950+0000 7f9d99e00640 1 -- 172.21.15.67:0/2660303722 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d941a1f80 con 0x7f9d941073a0 2024-03-28T16:07:20.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.955+0000 7f9d98a00640 1 -- 172.21.15.67:0/2660303722 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f9d8c004e00 con 0x7f9d941073a0 2024-03-28T16:07:20.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.955+0000 7f9d9cb8d740 1 Processor -- start 2024-03-28T16:07:20.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.955+0000 7f9d9cb8d740 1 -- start start 2024-03-28T16:07:20.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.956+0000 7f9d9cb8d740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564c67217e80 0x564c672182a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:20.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.956+0000 7f9d9cb8d740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564c67218870 0x564c6721d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:20.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.956+0000 7f9d9cb8d740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564c6721d9e0 0x564c6721fe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:20.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.956+0000 7f9d9cb8d740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c6717d250 con 0x564c6721d9e0 2024-03-28T16:07:20.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.956+0000 7f9d9cb8d740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c67125a30 con 0x564c67218870 2024-03-28T16:07:20.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.956+0000 7f9d9cb8d740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c671792f0 con 0x564c67217e80 2024-03-28T16:07:20.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.956+0000 7f9d8a000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564c6721d9e0 0x564c6721fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:20.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.956+0000 7f9d8a000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564c6721d9e0 0x564c6721fe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58660/0 (socket says 172.21.15.67:58660) 2024-03-28T16:07:20.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.957+0000 7f9d8a000640 1 -- 172.21.15.67:0/1506415412 learned_addr learned my addr 172.21.15.67:0/1506415412 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:07:20.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.957+0000 7f9d8a000640 1 --2- 172.21.15.67:0/1506415412 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564c67217e80 0x564c672182a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:20.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.957+0000 7f9d92a00640 1 --2- 172.21.15.67:0/1506415412 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564c67218870 0x564c6721d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:20.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.957+0000 7f9d8a000640 1 -- 172.21.15.67:0/1506415412 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564c67217e80 msgr2=0x564c672182a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:20.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.957+0000 7f9d8a000640 1 --2- 172.21.15.67:0/1506415412 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564c67217e80 0x564c672182a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:20.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.957+0000 7f9d8a000640 1 -- 172.21.15.67:0/1506415412 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564c67218870 msgr2=0x564c6721d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:20.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.957+0000 7f9d8a000640 1 --2- 172.21.15.67:0/1506415412 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564c67218870 0x564c6721d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:20.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.957+0000 7f9d8a000640 1 -- 172.21.15.67:0/1506415412 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564c67110690 con 0x564c6721d9e0 2024-03-28T16:07:20.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.958+0000 7f9d8a000640 1 --2- 172.21.15.67:0/1506415412 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564c6721d9e0 0x564c6721fe70 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f9d7c02c710 tx=0x7f9d7c005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7184b5e63973032b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:20.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.958+0000 7f9d98a00640 1 --2- 172.21.15.67:0/2660303722 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f9d780a2800 0x7f9d780a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:20.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.958+0000 7f9d83e00640 1 -- 172.21.15.67:0/1506415412 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d7c049070 con 0x564c6721d9e0 2024-03-28T16:07:20.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.958+0000 7f9d83e00640 1 -- 172.21.15.67:0/1506415412 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9d7c004380 con 0x564c6721d9e0 2024-03-28T16:07:20.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.958+0000 7f9d83e00640 1 -- 172.21.15.67:0/1506415412 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d7c004a20 con 0x564c6721d9e0 2024-03-28T16:07:20.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.958+0000 7f9d98a00640 1 -- 172.21.15.67:0/2660303722 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f9d8c0ce1e0 con 0x7f9d941073a0 2024-03-28T16:07:20.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.959+0000 7f9d92a00640 1 --2- 172.21.15.67:0/2660303722 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f9d780a2800 0x7f9d780a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:20.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.960+0000 7f9d9cb8d740 1 -- 172.21.15.67:0/1506415412 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564c6721d9e0 msgr2=0x564c6721fe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:20.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.960+0000 7f9d92a00640 1 --2- 172.21.15.67:0/2660303722 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f9d780a2800 0x7f9d780a4cc0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x564c67218c70 tx=0x7f9d84009580 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:20.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.960+0000 7f9d9cb8d740 1 --2- 172.21.15.67:0/1506415412 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564c6721d9e0 0x564c6721fe70 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f9d7c02c710 tx=0x7f9d7c005fd0 comp rx=0 tx=0).stop 2024-03-28T16:07:20.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.960+0000 7f9d9cb8d740 1 -- 172.21.15.67:0/1506415412 shutdown_connections 2024-03-28T16:07:20.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.960+0000 7f9d9cb8d740 1 --2- 172.21.15.67:0/1506415412 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564c6721d9e0 0x564c6721fe70 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:20.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.960+0000 7f9d9cb8d740 1 --2- 172.21.15.67:0/1506415412 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564c67218870 0x564c6721d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:20.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.960+0000 7f9d9cb8d740 1 --2- 172.21.15.67:0/1506415412 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564c67217e80 0x564c672182a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:20.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.960+0000 7f9d9cb8d740 1 -- 172.21.15.67:0/1506415412 >> 172.21.15.67:0/1506415412 conn(0x564c671e0200 msgr2=0x564c672137f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:20.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.960+0000 7f9d9cb8d740 1 -- 172.21.15.67:0/1506415412 shutdown_connections 2024-03-28T16:07:20.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.960+0000 7f9d9cb8d740 1 -- 172.21.15.67:0/1506415412 wait complete. 2024-03-28T16:07:20.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.961+0000 7f9d9cb8d740 1 -- start start 2024-03-28T16:07:20.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.961+0000 7f9d9cb8d740 1 Processor -- start 2024-03-28T16:07:20.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.962+0000 7f9d9cb8d740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564c67217e80 0x564c67221e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:20.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.962+0000 7f9d9cb8d740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564c67218870 0x564c67222370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:20.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.962+0000 7f9d9cb8d740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564c6721d9e0 0x564c672293b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:20.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.962+0000 7f9d9cb8d740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c67220930 con 0x564c67217e80 2024-03-28T16:07:20.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.962+0000 7f9d9cb8d740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c67138e80 con 0x564c6721d9e0 2024-03-28T16:07:20.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.962+0000 7f9d9cb8d740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c67220ab0 con 0x564c67218870 2024-03-28T16:07:20.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.962+0000 7f9d9cb8d740 10 client.0 _ll_register_callbacks cb 0x564c6714b130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:07:20.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.962+0000 7f9d92a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564c67217e80 0x564c67221e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:20.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.962+0000 7f9d92a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564c67217e80 0x564c67221e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58664/0 (socket says 172.21.15.67:58664) 2024-03-28T16:07:20.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.962+0000 7f9d92a00640 1 -- 172.21.15.67:0/1821179951 learned_addr learned my addr 172.21.15.67:0/1821179951 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:07:20.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.963+0000 7f9d92a00640 1 -- 172.21.15.67:0/1821179951 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564c67218870 msgr2=0x564c67222370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:20.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.963+0000 7f9d92a00640 1 --2- 172.21.15.67:0/1821179951 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564c67218870 0x564c67222370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:20.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.963+0000 7f9d92a00640 1 -- 172.21.15.67:0/1821179951 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564c6721d9e0 msgr2=0x564c672293b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:20.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.963+0000 7f9d92a00640 1 --2- 172.21.15.67:0/1821179951 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564c6721d9e0 0x564c672293b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:20.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.963+0000 7f9d92a00640 1 -- 172.21.15.67:0/1821179951 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564c66f5b7d0 con 0x564c67217e80 2024-03-28T16:07:20.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.963+0000 7f9d92a00640 1 --2- 172.21.15.67:0/1821179951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564c67217e80 0x564c67221e30 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f9d84009760 tx=0x7f9d84004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac1fb32279d72d38 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:20.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.963+0000 7f9d80c00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:07:20.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.964+0000 7f9d80c00640 1 -- 172.21.15.67:0/1821179951 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d8400ec60 con 0x564c67217e80 2024-03-28T16:07:20.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.964+0000 7f9d80c00640 1 -- 172.21.15.67:0/1821179951 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f9d8400ee00 con 0x564c67217e80 2024-03-28T16:07:20.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.964+0000 7f9d9cb8d740 1 -- 172.21.15.67:0/1821179951 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9d7c004380 con 0x564c67217e80 2024-03-28T16:07:20.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.965+0000 7f9d80c00640 1 -- 172.21.15.67:0/1821179951 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9d8401f420 con 0x564c67217e80 2024-03-28T16:07:20.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.965+0000 7f9d9cb8d740 10 client.15807 fetch_fsmap learned FSMap version 19 2024-03-28T16:07:20.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.965+0000 7f9d9cb8d740 1 -- 172.21.15.67:0/1821179951 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x564c66e913f0 con 0x564c67217e80 2024-03-28T16:07:20.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.965+0000 7f9d80c00640 1 -- 172.21.15.67:0/1821179951 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9d7c004380 con 0x564c67217e80 2024-03-28T16:07:20.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.965+0000 7f9d80c00640 1 -- 172.21.15.67:0/1821179951 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f9d8400f040 con 0x564c67217e80 2024-03-28T16:07:20.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.965+0000 7f9d9cb8d740 10 client.15807 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:07:20.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.965+0000 7f9d9cb8d740 10 client.15807 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:07:20.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.965+0000 7f9d9cb8d740 20 client.15807 populate_metadata read hostname 'smithi067' 2024-03-28T16:07:20.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.965+0000 7f9d9cb8d740 1 --2- 172.21.15.67:0/1821179951 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x564c6722dd40 0x564c67230200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:20.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.965+0000 7f9d9cb8d740 4 client.15807 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:07:20.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.965+0000 7f9d9cb8d740 1 -- 172.21.15.67:0/1821179951 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564c6718a790 con 0x564c6722dd40 2024-03-28T16:07:20.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.966+0000 7f9d8a000640 1 --2- 172.21.15.67:0/1821179951 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x564c6722dd40 0x564c67230200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:20.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.967+0000 7f9d8a000640 1 --2- 172.21.15.67:0/1821179951 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x564c6722dd40 0x564c67230200 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=72b05586e2be8be9 server_cookie=68db55fa8c67bdb6 in_seq=0 out_seq=0 2024-03-28T16:07:20.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.967+0000 7f9d80c00640 10 client.15807 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:07:20.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.968+0000 7f9d80c00640 1 -- 172.21.15.67:0/1821179951 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x564c6718a790 con 0x564c6722dd40 2024-03-28T16:07:20.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.968+0000 7f9d80c00640 10 client.15807 handle_command_reply: tid=0 2024-03-28T16:07:20.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.968+0000 7f9d9cb8d740 1 client.15807 shutdown 2024-03-28T16:07:20.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.968+0000 7f9d9cb8d740 1 -- 172.21.15.67:0/1821179951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564c67217e80 msgr2=0x564c67221e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:20.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.968+0000 7f9d9cb8d740 1 --2- 172.21.15.67:0/1821179951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564c67217e80 0x564c67221e30 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f9d84009760 tx=0x7f9d84004a40 comp rx=0 tx=0).stop 2024-03-28T16:07:20.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.968+0000 7f9d9cb8d740 1 -- 172.21.15.67:0/1821179951 shutdown_connections 2024-03-28T16:07:20.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.968+0000 7f9d9cb8d740 1 --2- 172.21.15.67:0/1821179951 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x564c6722dd40 0x564c67230200 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:20.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.968+0000 7f9d9cb8d740 1 --2- 172.21.15.67:0/1821179951 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564c6721d9e0 0x564c672293b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:20.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.968+0000 7f9d9cb8d740 1 --2- 172.21.15.67:0/1821179951 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564c67218870 0x564c67222370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:20.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.968+0000 7f9d9cb8d740 1 --2- 172.21.15.67:0/1821179951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564c67217e80 0x564c67221e30 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:20.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.968+0000 7f9d9cb8d740 1 -- 172.21.15.67:0/1821179951 >> 172.21.15.67:0/1821179951 conn(0x564c671e0200 msgr2=0x564c67212a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:20.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.969+0000 7f9d80c00640 0 client.15807 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:07:20.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.969+0000 7f9d9cb8d740 1 -- 172.21.15.67:0/1821179951 shutdown_connections 2024-03-28T16:07:20.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.969+0000 7f9d9cb8d740 1 -- 172.21.15.67:0/1821179951 wait complete. 2024-03-28T16:07:20.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.969+0000 7f9d9cb8d740 20 client.15807 trim_cache size 0 max 16384 2024-03-28T16:07:20.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.994+0000 7f9d9cb8d740 1 Processor -- start 2024-03-28T16:07:20.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.994+0000 7f9d9cb8d740 1 -- start start 2024-03-28T16:07:20.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.995+0000 7f9d9cb8d740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564c67217e80 0x564c672220b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:20.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.995+0000 7f9d9cb8d740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564c67218870 0x564c67229370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:20.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.995+0000 7f9d9cb8d740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564c6721d9e0 0x564c67219100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:20.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.995+0000 7f9d9cb8d740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d941a8d00 con 0x564c6721d9e0 2024-03-28T16:07:20.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.995+0000 7f9d9cb8d740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d941ab2c0 con 0x564c67218870 2024-03-28T16:07:20.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.995+0000 7f9d9cb8d740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d941a9300 con 0x564c67217e80 2024-03-28T16:07:20.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.995+0000 7f9d93400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564c6721d9e0 0x564c67219100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:20.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.995+0000 7f9d93400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564c6721d9e0 0x564c67219100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58670/0 (socket says 172.21.15.67:58670) 2024-03-28T16:07:20.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.995+0000 7f9d93400640 1 -- 172.21.15.67:0/275084680 learned_addr learned my addr 172.21.15.67:0/275084680 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:07:20.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.995+0000 7f9d8a000640 1 --2- 172.21.15.67:0/275084680 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564c67218870 0x564c67229370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:20.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.996+0000 7f9d93400640 1 -- 172.21.15.67:0/275084680 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564c67217e80 msgr2=0x564c672220b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:20.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.996+0000 7f9d93400640 1 --2- 172.21.15.67:0/275084680 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564c67217e80 0x564c672220b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:20.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.996+0000 7f9d93400640 1 -- 172.21.15.67:0/275084680 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564c67218870 msgr2=0x564c67229370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:20.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.996+0000 7f9d93400640 1 --2- 172.21.15.67:0/275084680 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564c67218870 0x564c67229370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:20.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.996+0000 7f9d93400640 1 -- 172.21.15.67:0/275084680 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564c66e510f0 con 0x564c6721d9e0 2024-03-28T16:07:20.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.996+0000 7f9d8a000640 1 --2- 172.21.15.67:0/275084680 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564c67218870 0x564c67229370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T16:07:20.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.996+0000 7f9d93400640 1 --2- 172.21.15.67:0/275084680 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564c6721d9e0 0x564c67219100 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c009610 tx=0x7f9d8c005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96e2ab6c1b403788 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:20.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.996+0000 7f9d83400640 1 -- 172.21.15.67:0/275084680 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d8c020040 con 0x564c6721d9e0 2024-03-28T16:07:20.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.996+0000 7f9d83400640 1 -- 172.21.15.67:0/275084680 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9d8c09ece0 con 0x564c6721d9e0 2024-03-28T16:07:20.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.996+0000 7f9d83400640 1 -- 172.21.15.67:0/275084680 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d8c0973a0 con 0x564c6721d9e0 2024-03-28T16:07:21.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.997+0000 7f9d9cb8d740 1 -- 172.21.15.67:0/275084680 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564c6721d9e0 msgr2=0x564c67219100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:21.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.997+0000 7f9d9cb8d740 1 --2- 172.21.15.67:0/275084680 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564c6721d9e0 0x564c67219100 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c009610 tx=0x7f9d8c005740 comp rx=0 tx=0).stop 2024-03-28T16:07:21.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.997+0000 7f9d9cb8d740 1 -- 172.21.15.67:0/275084680 shutdown_connections 2024-03-28T16:07:21.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.997+0000 7f9d9cb8d740 1 --2- 172.21.15.67:0/275084680 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564c6721d9e0 0x564c67219100 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:21.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.997+0000 7f9d9cb8d740 1 --2- 172.21.15.67:0/275084680 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564c67218870 0x564c67229370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:21.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.997+0000 7f9d9cb8d740 1 --2- 172.21.15.67:0/275084680 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564c67217e80 0x564c672220b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:21.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.997+0000 7f9d9cb8d740 1 -- 172.21.15.67:0/275084680 >> 172.21.15.67:0/275084680 conn(0x564c671e0200 msgr2=0x564c672172c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:21.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.998+0000 7f9d9cb8d740 1 -- 172.21.15.67:0/275084680 shutdown_connections 2024-03-28T16:07:21.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.998+0000 7f9d9cb8d740 1 -- 172.21.15.67:0/275084680 wait complete. 2024-03-28T16:07:21.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.998+0000 7f9d9cb8d740 1 -- start start 2024-03-28T16:07:21.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.998+0000 7f9d9cb8d740 1 Processor -- start 2024-03-28T16:07:21.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.999+0000 7f9d9cb8d740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564c67217e80 0x564c67226e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:21.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.999+0000 7f9d9cb8d740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564c67218870 0x564c67227340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:21.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.999+0000 7f9d9cb8d740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564c6721d9e0 0x564c67228290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:21.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.999+0000 7f9d9cb8d740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d941a9940 con 0x564c67217e80 2024-03-28T16:07:21.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.999+0000 7f9d9cb8d740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d941aac60 con 0x564c6721d9e0 2024-03-28T16:07:21.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.999+0000 7f9d9cb8d740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d941a9fa0 con 0x564c67218870 2024-03-28T16:07:21.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.999+0000 7f9d9cb8d740 10 client.0 _ll_register_callbacks cb 0x7f9d84009b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:07:21.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.999+0000 7f9d92a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564c67217e80 0x564c67226e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:21.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.999+0000 7f9d92a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564c67217e80 0x564c67226e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58674/0 (socket says 172.21.15.67:58674) 2024-03-28T16:07:21.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:20.999+0000 7f9d92a00640 1 -- 172.21.15.67:0/1564840584 learned_addr learned my addr 172.21.15.67:0/1564840584 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:07:21.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.000+0000 7f9d93400640 1 --2- 172.21.15.67:0/1564840584 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564c6721d9e0 0x564c67228290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:21.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.000+0000 7f9d92a00640 1 -- 172.21.15.67:0/1564840584 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564c67218870 msgr2=0x564c67227340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:21.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.000+0000 7f9d92a00640 1 --2- 172.21.15.67:0/1564840584 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564c67218870 0x564c67227340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:21.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.000+0000 7f9d92a00640 1 -- 172.21.15.67:0/1564840584 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564c6721d9e0 msgr2=0x564c67228290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:21.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.000+0000 7f9d92a00640 1 --2- 172.21.15.67:0/1564840584 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564c6721d9e0 0x564c67228290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:21.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.000+0000 7f9d92a00640 1 -- 172.21.15.67:0/1564840584 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564c67228880 con 0x564c67217e80 2024-03-28T16:07:21.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.000+0000 7f9d93400640 1 --2- 172.21.15.67:0/1564840584 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564c6721d9e0 0x564c67228290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:07:21.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.000+0000 7f9d92a00640 1 --2- 172.21.15.67:0/1564840584 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564c67217e80 0x564c67226e00 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f9d84009470 tx=0x7f9d84011c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f618386d87f720c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:21.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.000+0000 7f9d80c00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:07:21.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.001+0000 7f9d80c00640 1 -- 172.21.15.67:0/1564840584 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d84012500 con 0x564c67217e80 2024-03-28T16:07:21.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.001+0000 7f9d80c00640 1 -- 172.21.15.67:0/1564840584 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f9d84027950 con 0x564c67217e80 2024-03-28T16:07:21.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.001+0000 7f9d9cb8d740 1 -- 172.21.15.67:0/1564840584 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9d8c09ece0 con 0x564c67217e80 2024-03-28T16:07:21.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.001+0000 7f9d80c00640 1 -- 172.21.15.67:0/1564840584 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9d84027b90 con 0x564c67217e80 2024-03-28T16:07:21.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.001+0000 7f9d9cb8d740 10 client.15813 fetch_fsmap learned FSMap version 19 2024-03-28T16:07:21.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.001+0000 7f9d9cb8d740 1 -- 172.21.15.67:0/1564840584 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x564c67289990 con 0x564c67217e80 2024-03-28T16:07:21.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.001+0000 7f9d80c00640 1 -- 172.21.15.67:0/1564840584 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9d8c09ece0 con 0x564c67217e80 2024-03-28T16:07:21.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.001+0000 7f9d80c00640 1 -- 172.21.15.67:0/1564840584 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f9d84032450 con 0x564c67217e80 2024-03-28T16:07:21.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.002+0000 7f9d9cb8d740 10 client.15813 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:07:21.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.002+0000 7f9d9cb8d740 10 client.15813 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:07:21.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.002+0000 7f9d9cb8d740 20 client.15813 populate_metadata read hostname 'smithi067' 2024-03-28T16:07:21.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.002+0000 7f9d9cb8d740 1 --2- 172.21.15.67:0/1564840584 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x564c6728a310 0x564c6728c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:21.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.002+0000 7f9d9cb8d740 4 client.15813 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T16:07:21.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.002+0000 7f9d9cb8d740 1 -- 172.21.15.67:0/1564840584 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x564c67184ae0 con 0x564c6728a310 2024-03-28T16:07:21.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.002+0000 7f9d8a000640 1 --2- 172.21.15.67:0/1564840584 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x564c6728a310 0x564c6728c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:21.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.003+0000 7f9d8a000640 1 --2- 172.21.15.67:0/1564840584 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x564c6728a310 0x564c6728c730 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ce436ab61f3586d server_cookie=e6c7d71978d9f8bb in_seq=0 out_seq=0 2024-03-28T16:07:21.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.003+0000 7f9d80c00640 10 client.15813 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:07:21.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.004+0000 7f9d80c00640 1 -- 172.21.15.67:0/1564840584 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x564c67184ae0 con 0x564c6728a310 2024-03-28T16:07:21.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.004+0000 7f9d80c00640 10 client.15813 handle_command_reply: tid=0 2024-03-28T16:07:21.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.004+0000 7f9d9cb8d740 1 client.15813 shutdown 2024-03-28T16:07:21.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.004+0000 7f9d9cb8d740 1 -- 172.21.15.67:0/1564840584 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564c67217e80 msgr2=0x564c67226e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:21.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.004+0000 7f9d9cb8d740 1 --2- 172.21.15.67:0/1564840584 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564c67217e80 0x564c67226e00 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f9d84009470 tx=0x7f9d84011c90 comp rx=0 tx=0).stop 2024-03-28T16:07:21.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.004+0000 7f9d9cb8d740 1 -- 172.21.15.67:0/1564840584 shutdown_connections 2024-03-28T16:07:21.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.004+0000 7f9d9cb8d740 1 --2- 172.21.15.67:0/1564840584 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x564c6728a310 0x564c6728c730 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:21.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.004+0000 7f9d9cb8d740 1 --2- 172.21.15.67:0/1564840584 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564c6721d9e0 0x564c67228290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:21.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.004+0000 7f9d9cb8d740 1 --2- 172.21.15.67:0/1564840584 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564c67218870 0x564c67227340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:21.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.004+0000 7f9d9cb8d740 1 --2- 172.21.15.67:0/1564840584 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564c67217e80 0x564c67226e00 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:21.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.004+0000 7f9d9cb8d740 1 -- 172.21.15.67:0/1564840584 >> 172.21.15.67:0/1564840584 conn(0x564c671e0200 msgr2=0x564c672172c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:21.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.004+0000 7f9d80c00640 0 client.15813 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:07:21.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.004+0000 7f9d9cb8d740 1 -- 172.21.15.67:0/1564840584 shutdown_connections 2024-03-28T16:07:21.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.004+0000 7f9d9cb8d740 1 -- 172.21.15.67:0/1564840584 wait complete. 2024-03-28T16:07:21.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.005+0000 7f9d9cb8d740 20 client.15813 trim_cache size 0 max 16384 2024-03-28T16:07:21.008 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-28T16:07:21.008 INFO:teuthology.orchestra.run.smithi067.stdout: "status": "no active scrubs running", 2024-03-28T16:07:21.008 INFO:teuthology.orchestra.run.smithi067.stdout: "scrubs": {} 2024-03-28T16:07:21.008 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-28T16:07:21.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.006+0000 7f9d99e00640 1 -- 172.21.15.67:0/2660303722 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f9d780a2800 msgr2=0x7f9d780a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:21.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.006+0000 7f9d99e00640 1 --2- 172.21.15.67:0/2660303722 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f9d780a2800 0x7f9d780a4cc0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x564c67218c70 tx=0x7f9d84009580 comp rx=0 tx=0).stop 2024-03-28T16:07:21.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.006+0000 7f9d99e00640 1 -- 172.21.15.67:0/2660303722 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d941073a0 msgr2=0x7f9d94100630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:21.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.006+0000 7f9d99e00640 1 --2- 172.21.15.67:0/2660303722 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d941073a0 0x7f9d94100630 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c00d750 tx=0x7f9d8c00dc10 comp rx=0 tx=0).stop 2024-03-28T16:07:21.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.006+0000 7f9d99e00640 1 -- 172.21.15.67:0/2660303722 shutdown_connections 2024-03-28T16:07:21.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.006+0000 7f9d99e00640 1 --2- 172.21.15.67:0/2660303722 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f9d780a2800 0x7f9d780a4cc0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:21.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.006+0000 7f9d99e00640 1 --2- 172.21.15.67:0/2660303722 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d941073a0 0x7f9d94100630 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:21.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.006+0000 7f9d99e00640 1 --2- 172.21.15.67:0/2660303722 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9d94104a70 0x7f9d941000f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:21.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.006+0000 7f9d99e00640 1 --2- 172.21.15.67:0/2660303722 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d94102140 0x7f9d94101ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:21.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.006+0000 7f9d99e00640 1 -- 172.21.15.67:0/2660303722 >> 172.21.15.67:0/2660303722 conn(0x7f9d940fbb30 msgr2=0x7f9d94103ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:21.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.007+0000 7f9d99e00640 1 -- 172.21.15.67:0/2660303722 shutdown_connections 2024-03-28T16:07:21.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.007+0000 7f9d99e00640 1 -- 172.21.15.67:0/2660303722 wait complete. 2024-03-28T16:07:21.025 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:07:20 smithi067 ceph-mon[31707]: pgmap v552: 97 pgs: 97 active+clean; 1.1 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 32 MiB/s wr, 10 op/s 2024-03-28T16:07:21.031 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T16:07:21.031 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T16:07:21.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:07:20 smithi183 ceph-mon[35580]: pgmap v552: 97 pgs: 97 active+clean; 1.1 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 32 MiB/s wr, 10 op/s 2024-03-28T16:07:21.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.300+0000 7f4b62e00640 1 -- 172.21.15.67:0/4124816228 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b5c1048a0 msgr2=0x7f4b5c106c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:21.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.300+0000 7f4b62e00640 1 --2- 172.21.15.67:0/4124816228 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b5c1048a0 0x7f4b5c106c90 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f4b48009900 tx=0x7f4b4802bc30 comp rx=0 tx=0).stop 2024-03-28T16:07:21.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.300+0000 7f4b62e00640 1 -- 172.21.15.67:0/4124816228 shutdown_connections 2024-03-28T16:07:21.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.300+0000 7f4b62e00640 1 --2- 172.21.15.67:0/4124816228 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4b5c1071d0 0x7f4b5c109620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:21.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.300+0000 7f4b62e00640 1 --2- 172.21.15.67:0/4124816228 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b5c1048a0 0x7f4b5c106c90 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:21.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.300+0000 7f4b62e00640 1 --2- 172.21.15.67:0/4124816228 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b5c101f70 0x7f4b5c104360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:21.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.300+0000 7f4b62e00640 1 -- 172.21.15.67:0/4124816228 >> 172.21.15.67:0/4124816228 conn(0x7f4b5c0fb980 msgr2=0x7f4b5c0fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:21.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.301+0000 7f4b62e00640 1 -- 172.21.15.67:0/4124816228 shutdown_connections 2024-03-28T16:07:21.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.301+0000 7f4b62e00640 1 -- 172.21.15.67:0/4124816228 wait complete. 2024-03-28T16:07:21.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:07:20 smithi150 ceph-mon[34825]: pgmap v552: 97 pgs: 97 active+clean; 1.1 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 32 MiB/s wr, 10 op/s 2024-03-28T16:07:21.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.302+0000 7f4b62e00640 1 Processor -- start 2024-03-28T16:07:21.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.302+0000 7f4b62e00640 1 -- start start 2024-03-28T16:07:21.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.303+0000 7f4b62e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4b5c101f70 0x7f4b5c197750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:21.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.303+0000 7f4b62e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b5c1048a0 0x7f4b5c197c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:21.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.303+0000 7f4b62e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b5c1071d0 0x7f4b5c19ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:21.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.303+0000 7f4b62e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b5c10d070 con 0x7f4b5c1071d0 2024-03-28T16:07:21.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.303+0000 7f4b62e00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b5c10cef0 con 0x7f4b5c1048a0 2024-03-28T16:07:21.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.303+0000 7f4b62e00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b5c10d1f0 con 0x7f4b5c101f70 2024-03-28T16:07:21.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.304+0000 7f4b61a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4b5c101f70 0x7f4b5c197750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:21.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.304+0000 7f4b61a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4b5c101f70 0x7f4b5c197750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.67:58120/0 (socket says 172.21.15.67:58120) 2024-03-28T16:07:21.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.304+0000 7f4b61a00640 1 -- 172.21.15.67:0/4050106436 learned_addr learned my addr 172.21.15.67:0/4050106436 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:07:21.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.304+0000 7f4b61000640 1 --2- 172.21.15.67:0/4050106436 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b5c1048a0 0x7f4b5c197c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:21.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.304+0000 7f4b62400640 1 --2- 172.21.15.67:0/4050106436 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b5c1071d0 0x7f4b5c19ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:21.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.305+0000 7f4b61a00640 1 -- 172.21.15.67:0/4050106436 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b5c1048a0 msgr2=0x7f4b5c197c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:21.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.305+0000 7f4b61a00640 1 --2- 172.21.15.67:0/4050106436 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b5c1048a0 0x7f4b5c197c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:21.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.305+0000 7f4b61a00640 1 -- 172.21.15.67:0/4050106436 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b5c1071d0 msgr2=0x7f4b5c19ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:21.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.305+0000 7f4b61a00640 1 --2- 172.21.15.67:0/4050106436 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b5c1071d0 0x7f4b5c19ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:21.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.305+0000 7f4b61a00640 1 -- 172.21.15.67:0/4050106436 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b5c06b1f0 con 0x7f4b5c101f70 2024-03-28T16:07:21.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.305+0000 7f4b61a00640 1 --2- 172.21.15.67:0/4050106436 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4b5c101f70 0x7f4b5c197750 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4b5400ba50 tx=0x7f4b5400bf10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:21.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.306+0000 7f4b5a000640 1 -- 172.21.15.67:0/4050106436 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b54019070 con 0x7f4b5c101f70 2024-03-28T16:07:21.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.306+0000 7f4b5a000640 1 -- 172.21.15.67:0/4050106436 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4b5400cbe0 con 0x7f4b5c101f70 2024-03-28T16:07:21.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.306+0000 7f4b62e00640 1 -- 172.21.15.67:0/4050106436 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b5c06b4e0 con 0x7f4b5c101f70 2024-03-28T16:07:21.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.306+0000 7f4b5a000640 1 -- 172.21.15.67:0/4050106436 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b540075d0 con 0x7f4b5c101f70 2024-03-28T16:07:21.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.306+0000 7f4b62e00640 1 -- 172.21.15.67:0/4050106436 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b5c06ba80 con 0x7f4b5c101f70 2024-03-28T16:07:21.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.308+0000 7f4b62e00640 1 -- 172.21.15.67:0/4050106436 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b5c0fd470 con 0x7f4b5c101f70 2024-03-28T16:07:21.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.312+0000 7f4b5a000640 1 -- 172.21.15.67:0/4050106436 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f4b5401c070 con 0x7f4b5c101f70 2024-03-28T16:07:21.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.313+0000 7f4b5a000640 1 --2- 172.21.15.67:0/4050106436 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f4b440a2800 0x7f4b440a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:21.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.314+0000 7f4b5a000640 1 -- 172.21.15.67:0/4050106436 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f4b540ce4f0 con 0x7f4b5c101f70 2024-03-28T16:07:21.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.314+0000 7f4b61000640 1 --2- 172.21.15.67:0/4050106436 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f4b440a2800 0x7f4b440a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:21.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.315+0000 7f4b61000640 1 --2- 172.21.15.67:0/4050106436 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f4b440a2800 0x7f4b440a4cc0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f4b48009230 tx=0x7f4b48005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:21.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.329+0000 7f4b5a000640 1 -- 172.21.15.67:0/4050106436 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4b5409af50 con 0x7f4b5c101f70 2024-03-28T16:07:21.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.685+0000 7f4b62e00640 1 -- 172.21.15.67:0/4050106436 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b5c02d070 con 0x7f4b5c101f70 2024-03-28T16:07:21.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.688+0000 7f4b5a000640 1 -- 172.21.15.67:0/4050106436 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11341 (secure 0 0 0) 0x7f4b5409eff0 con 0x7f4b5c101f70 2024-03-28T16:07:21.688 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:07:21.688 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2008297659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2008297659},{"type":"v1","addr":"172.21.15.67:6835","nonce":2008297659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14532,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/320295745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":320295745},{"type":"v1","addr":"172.21.15.67:6837","nonce":320295745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6839/691821273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":691821273},{"type":"v1","addr":"172.21.15.67:6839","nonce":691821273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6841/3211254875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6840","nonce":3211254875},{"type":"v1","addr":"172.21.15.67:6841","nonce":3211254875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2129665423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2129665423},{"type":"v1","addr":"172.21.15.183:6833","nonce":2129665423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24496,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/1589784425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":1589784425},{"type":"v1","addr":"172.21.15.183:6835","nonce":1589784425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24508,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/2185863835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":2185863835},{"type":"v1","addr":"172.21.15.183:6837","nonce":2185863835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24520,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6839/1242814398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":1242814398},{"type":"v1","addr":"172.21.15.183:6839","nonce":1242814398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":34409,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1008335521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1008335521},{"type":"v1","addr":"172.21.15.150:6833","nonce":1008335521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":34445,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3269935695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3269935695},{"type":"v1","addr":"172.21.15.150:6835","nonce":3269935695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T16:01:07.696824+0000","modified":"2024-03-28T16:01:11.997773+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":34508},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34475":{"gid":34475,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/1112886277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":1112886277},{"type":"v1","addr":"172.21.15.150:6837","nonce":1112886277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_34508":{"gid":34508,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6839/1211011614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":1211011614},{"type":"v1","addr":"172.21.15.150:6839","nonce":1211011614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34508,"qdb_cluster":[34508]},"id":1}]} 2024-03-28T16:07:21.689 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 19 2024-03-28T16:07:21.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.691+0000 7f4b62e00640 1 -- 172.21.15.67:0/4050106436 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f4b440a2800 msgr2=0x7f4b440a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:21.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.691+0000 7f4b62e00640 1 --2- 172.21.15.67:0/4050106436 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f4b440a2800 0x7f4b440a4cc0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f4b48009230 tx=0x7f4b48005fd0 comp rx=0 tx=0).stop 2024-03-28T16:07:21.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.691+0000 7f4b62e00640 1 -- 172.21.15.67:0/4050106436 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4b5c101f70 msgr2=0x7f4b5c197750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:21.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.691+0000 7f4b62e00640 1 --2- 172.21.15.67:0/4050106436 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4b5c101f70 0x7f4b5c197750 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4b5400ba50 tx=0x7f4b5400bf10 comp rx=0 tx=0).stop 2024-03-28T16:07:21.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.691+0000 7f4b62e00640 1 -- 172.21.15.67:0/4050106436 shutdown_connections 2024-03-28T16:07:21.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.691+0000 7f4b62e00640 1 --2- 172.21.15.67:0/4050106436 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f4b440a2800 0x7f4b440a4cc0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:21.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.691+0000 7f4b62e00640 1 --2- 172.21.15.67:0/4050106436 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b5c1071d0 0x7f4b5c19ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:21.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.691+0000 7f4b62e00640 1 --2- 172.21.15.67:0/4050106436 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b5c1048a0 0x7f4b5c197c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:21.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.691+0000 7f4b62e00640 1 --2- 172.21.15.67:0/4050106436 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4b5c101f70 0x7f4b5c197750 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:21.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.691+0000 7f4b62e00640 1 -- 172.21.15.67:0/4050106436 >> 172.21.15.67:0/4050106436 conn(0x7f4b5c0fb980 msgr2=0x7f4b5c102970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:21.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.691+0000 7f4b62e00640 1 -- 172.21.15.67:0/4050106436 shutdown_connections 2024-03-28T16:07:21.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.691+0000 7f4b62e00640 1 -- 172.21.15.67:0/4050106436 wait complete. 2024-03-28T16:07:21.715 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T16:07:21.715 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T16:07:21.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.980+0000 7f1258400640 1 -- 172.21.15.67:0/2907794224 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12501016d0 msgr2=0x7f1250101af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:21.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.980+0000 7f1258400640 1 --2- 172.21.15.67:0/2907794224 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12501016d0 0x7f1250101af0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f1244009900 tx=0x7f124402bc30 comp rx=0 tx=0).stop 2024-03-28T16:07:21.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.980+0000 7f1258400640 1 -- 172.21.15.67:0/2907794224 shutdown_connections 2024-03-28T16:07:21.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.980+0000 7f1258400640 1 --2- 172.21.15.67:0/2907794224 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1250103280 0x7f1250109b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:21.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.980+0000 7f1258400640 1 --2- 172.21.15.67:0/2907794224 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12501028c0 0x7f1250102d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:21.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.980+0000 7f1258400640 1 --2- 172.21.15.67:0/2907794224 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12501016d0 0x7f1250101af0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:21.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.980+0000 7f1258400640 1 -- 172.21.15.67:0/2907794224 >> 172.21.15.67:0/2907794224 conn(0x7f12500fcc40 msgr2=0x7f12500ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:21.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.980+0000 7f1258400640 1 -- 172.21.15.67:0/2907794224 shutdown_connections 2024-03-28T16:07:21.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.980+0000 7f1258400640 1 -- 172.21.15.67:0/2907794224 wait complete. 2024-03-28T16:07:21.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.982+0000 7f1258400640 1 Processor -- start 2024-03-28T16:07:21.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.983+0000 7f1258400640 1 -- start start 2024-03-28T16:07:21.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.983+0000 7f1258400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f12501016d0 0x7f1250110430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:21.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.983+0000 7f1258400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12501028c0 0x7f1250110970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:21.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.983+0000 7f1258400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1250103280 0x7f125010cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:21.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.984+0000 7f1258400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1250074030 con 0x7f12501028c0 2024-03-28T16:07:21.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.984+0000 7f1258400640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1250073eb0 con 0x7f1250103280 2024-03-28T16:07:21.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.984+0000 7f1258400640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12500741b0 con 0x7f12501016d0 2024-03-28T16:07:21.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.984+0000 7f1255000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12501028c0 0x7f1250110970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:21.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.984+0000 7f1255000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12501028c0 0x7f1250110970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58712/0 (socket says 172.21.15.67:58712) 2024-03-28T16:07:21.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.984+0000 7f1255000640 1 -- 172.21.15.67:0/2032214024 learned_addr learned my addr 172.21.15.67:0/2032214024 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:07:21.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.985+0000 7f1255000640 1 -- 172.21.15.67:0/2032214024 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f12501016d0 msgr2=0x7f1250110430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:21.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.985+0000 7f1255000640 1 --2- 172.21.15.67:0/2032214024 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f12501016d0 0x7f1250110430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:21.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.985+0000 7f1255000640 1 -- 172.21.15.67:0/2032214024 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1250103280 msgr2=0x7f125010cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:21.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.985+0000 7f1255000640 1 --2- 172.21.15.67:0/2032214024 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1250103280 0x7f125010cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:21.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.985+0000 7f1255000640 1 -- 172.21.15.67:0/2032214024 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f125010d860 con 0x7f12501028c0 2024-03-28T16:07:21.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.985+0000 7f1255000640 1 --2- 172.21.15.67:0/2032214024 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12501028c0 0x7f1250110970 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f123c00e9f0 tx=0x7f123c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:21.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.986+0000 7f124e000640 1 -- 172.21.15.67:0/2032214024 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f123c00cd80 con 0x7f12501028c0 2024-03-28T16:07:21.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.986+0000 7f1258400640 1 -- 172.21.15.67:0/2032214024 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1250199740 con 0x7f12501028c0 2024-03-28T16:07:21.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.986+0000 7f124e000640 1 -- 172.21.15.67:0/2032214024 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f123c0108e0 con 0x7f12501028c0 2024-03-28T16:07:21.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.986+0000 7f124e000640 1 -- 172.21.15.67:0/2032214024 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f123c0188e0 con 0x7f12501028c0 2024-03-28T16:07:21.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.987+0000 7f1258400640 1 -- 172.21.15.67:0/2032214024 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1250199ce0 con 0x7f12501028c0 2024-03-28T16:07:21.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.991+0000 7f124e000640 1 -- 172.21.15.67:0/2032214024 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f123c010a90 con 0x7f12501028c0 2024-03-28T16:07:21.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.993+0000 7f124e000640 1 --2- 172.21.15.67:0/2032214024 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f12380a28d0 0x7f12380a4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:21.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.994+0000 7f124e000640 1 -- 172.21.15.67:0/2032214024 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f123c0cee40 con 0x7f12501028c0 2024-03-28T16:07:21.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.994+0000 7f1255a00640 1 --2- 172.21.15.67:0/2032214024 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f12380a28d0 0x7f12380a4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:21.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.995+0000 7f1255a00640 1 --2- 172.21.15.67:0/2032214024 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f12380a28d0 0x7f12380a4d90 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f1244005ab0 tx=0x7f1244000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:21.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.996+0000 7f125b1b2740 1 Processor -- start 2024-03-28T16:07:21.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.996+0000 7f125b1b2740 1 -- start start 2024-03-28T16:07:21.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.996+0000 7f125b1b2740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5558f3b4f810 0x5558f3b4fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:21.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.996+0000 7f125b1b2740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5558f3b509e0 0x5558f3b50e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:21.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.996+0000 7f125b1b2740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558f3b513a0 0x5558f3b57c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:21.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.996+0000 7f125b1b2740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558f3ab42c0 con 0x5558f3b513a0 2024-03-28T16:07:21.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.997+0000 7f125b1b2740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558f3a5cc90 con 0x5558f3b509e0 2024-03-28T16:07:21.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.997+0000 7f125b1b2740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558f3ab0360 con 0x5558f3b4f810 2024-03-28T16:07:21.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.997+0000 7f1256400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558f3b513a0 0x5558f3b57c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:21.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.997+0000 7f1256400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558f3b513a0 0x5558f3b57c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58720/0 (socket says 172.21.15.67:58720) 2024-03-28T16:07:21.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.997+0000 7f1256400640 1 -- 172.21.15.67:0/1306758739 learned_addr learned my addr 172.21.15.67:0/1306758739 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:07:21.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.997+0000 7f1255000640 1 --2- 172.21.15.67:0/1306758739 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5558f3b509e0 0x5558f3b50e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:21.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.997+0000 7f1255a00640 1 --2- 172.21.15.67:0/1306758739 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5558f3b4f810 0x5558f3b4fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:21.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.997+0000 7f1256400640 1 -- 172.21.15.67:0/1306758739 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5558f3b4f810 msgr2=0x5558f3b4fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:21.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.997+0000 7f1256400640 1 --2- 172.21.15.67:0/1306758739 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5558f3b4f810 0x5558f3b4fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:21.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.997+0000 7f1256400640 1 -- 172.21.15.67:0/1306758739 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5558f3b509e0 msgr2=0x5558f3b50e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:21.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.997+0000 7f1256400640 1 --2- 172.21.15.67:0/1306758739 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5558f3b509e0 0x5558f3b50e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:21.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.997+0000 7f1256400640 1 -- 172.21.15.67:0/1306758739 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5558f3a81080 con 0x5558f3b513a0 2024-03-28T16:07:21.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.998+0000 7f1256400640 1 --2- 172.21.15.67:0/1306758739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558f3b513a0 0x5558f3b57c30 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f1248009d80 tx=0x7f124800e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d987a26d49606d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:21.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.998+0000 7f1242a00640 1 -- 172.21.15.67:0/1306758739 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1248016070 con 0x5558f3b513a0 2024-03-28T16:07:21.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.998+0000 7f1242a00640 1 -- 172.21.15.67:0/1306758739 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1248005020 con 0x5558f3b513a0 2024-03-28T16:07:21.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.998+0000 7f1242a00640 1 -- 172.21.15.67:0/1306758739 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12480116d0 con 0x5558f3b513a0 2024-03-28T16:07:22.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.999+0000 7f125b1b2740 1 -- 172.21.15.67:0/1306758739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558f3b513a0 msgr2=0x5558f3b57c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:22.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.999+0000 7f125b1b2740 1 --2- 172.21.15.67:0/1306758739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558f3b513a0 0x5558f3b57c30 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f1248009d80 tx=0x7f124800e6c0 comp rx=0 tx=0).stop 2024-03-28T16:07:22.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.999+0000 7f125b1b2740 1 -- 172.21.15.67:0/1306758739 shutdown_connections 2024-03-28T16:07:22.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.999+0000 7f125b1b2740 1 --2- 172.21.15.67:0/1306758739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558f3b513a0 0x5558f3b57c30 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:22.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.999+0000 7f125b1b2740 1 --2- 172.21.15.67:0/1306758739 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5558f3b509e0 0x5558f3b50e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:22.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.999+0000 7f125b1b2740 1 --2- 172.21.15.67:0/1306758739 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5558f3b4f810 0x5558f3b4fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:22.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:21.999+0000 7f125b1b2740 1 -- 172.21.15.67:0/1306758739 >> 172.21.15.67:0/1306758739 conn(0x5558f3b4ac00 msgr2=0x5558f3b4b000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:22.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.000+0000 7f125b1b2740 1 -- 172.21.15.67:0/1306758739 shutdown_connections 2024-03-28T16:07:22.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.000+0000 7f125b1b2740 1 -- 172.21.15.67:0/1306758739 wait complete. 2024-03-28T16:07:22.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.000+0000 7f125b1b2740 1 -- start start 2024-03-28T16:07:22.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.000+0000 7f125b1b2740 1 Processor -- start 2024-03-28T16:07:22.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.001+0000 7f125b1b2740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558f3b4f810 0x5558f3b59a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:22.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.001+0000 7f125b1b2740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5558f3b509e0 0x5558f3b59f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:22.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.001+0000 7f125b1b2740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5558f3b513a0 0x5558f3b60fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:22.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.001+0000 7f125b1b2740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558f3b583b0 con 0x5558f3b4f810 2024-03-28T16:07:22.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.001+0000 7f125b1b2740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558f3a700d0 con 0x5558f3b509e0 2024-03-28T16:07:22.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.001+0000 7f125b1b2740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558f3b58530 con 0x5558f3b513a0 2024-03-28T16:07:22.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.001+0000 7f125b1b2740 10 client.0 _ll_register_callbacks cb 0x5558f3a74e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:07:22.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.002+0000 7f1255a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558f3b4f810 0x5558f3b59a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:22.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.002+0000 7f1255a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558f3b4f810 0x5558f3b59a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58730/0 (socket says 172.21.15.67:58730) 2024-03-28T16:07:22.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.002+0000 7f1255a00640 1 -- 172.21.15.67:0/1736181772 learned_addr learned my addr 172.21.15.67:0/1736181772 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:07:22.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.002+0000 7f1255000640 1 --2- 172.21.15.67:0/1736181772 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5558f3b509e0 0x5558f3b59f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:22.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.002+0000 7f1256400640 1 --2- 172.21.15.67:0/1736181772 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5558f3b513a0 0x5558f3b60fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:22.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.002+0000 7f1255a00640 1 -- 172.21.15.67:0/1736181772 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5558f3b513a0 msgr2=0x5558f3b60fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:22.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.002+0000 7f1255a00640 1 --2- 172.21.15.67:0/1736181772 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5558f3b513a0 0x5558f3b60fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:22.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.002+0000 7f1255a00640 1 -- 172.21.15.67:0/1736181772 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5558f3b509e0 msgr2=0x5558f3b59f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:22.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.002+0000 7f1255a00640 1 --2- 172.21.15.67:0/1736181772 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5558f3b509e0 0x5558f3b59f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:22.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.002+0000 7f1255a00640 1 -- 172.21.15.67:0/1736181772 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5558f37c83f0 con 0x5558f3b4f810 2024-03-28T16:07:22.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.003+0000 7f1255a00640 1 --2- 172.21.15.67:0/1736181772 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558f3b4f810 0x5558f3b59a50 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f1250102720 tx=0x7f1244009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9a2fa6d0f44f61d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:22.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.003+0000 7f1237400640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:07:22.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.003+0000 7f1237400640 1 -- 172.21.15.67:0/1736181772 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f124404f020 con 0x5558f3b4f810 2024-03-28T16:07:22.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.004+0000 7f125b1b2740 1 -- 172.21.15.67:0/1736181772 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1248005020 con 0x5558f3b4f810 2024-03-28T16:07:22.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.004+0000 7f1237400640 1 -- 172.21.15.67:0/1736181772 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f12440037f0 con 0x5558f3b4f810 2024-03-28T16:07:22.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.004+0000 7f1237400640 1 -- 172.21.15.67:0/1736181772 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1244045040 con 0x5558f3b4f810 2024-03-28T16:07:22.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.004+0000 7f125b1b2740 10 client.15831 fetch_fsmap learned FSMap version 19 2024-03-28T16:07:22.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.004+0000 7f125b1b2740 1 -- 172.21.15.67:0/1736181772 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5558f38927d0 con 0x5558f3b4f810 2024-03-28T16:07:22.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.005+0000 7f1237400640 1 -- 172.21.15.67:0/1736181772 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1248005020 con 0x5558f3b4f810 2024-03-28T16:07:22.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.006+0000 7f1237400640 1 -- 172.21.15.67:0/1736181772 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f124405c020 con 0x5558f3b4f810 2024-03-28T16:07:22.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.006+0000 7f125b1b2740 10 client.15831 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:07:22.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.006+0000 7f125b1b2740 10 client.15831 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:07:22.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.006+0000 7f125b1b2740 20 client.15831 populate_metadata read hostname 'smithi067' 2024-03-28T16:07:22.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.006+0000 7f125b1b2740 1 --2- 172.21.15.67:0/1736181772 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5558f3b65610 0x5558f3b67a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:22.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.006+0000 7f125b1b2740 4 client.15831 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:07:22.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.006+0000 7f125b1b2740 1 -- 172.21.15.67:0/1736181772 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5558f3ac1800 con 0x5558f3b65610 2024-03-28T16:07:22.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.007+0000 7f1255000640 1 --2- 172.21.15.67:0/1736181772 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5558f3b65610 0x5558f3b67a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:22.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.008+0000 7f1255000640 1 --2- 172.21.15.67:0/1736181772 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5558f3b65610 0x5558f3b67a80 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ccc422778e587f92 server_cookie=f97837f121649257 in_seq=0 out_seq=0 2024-03-28T16:07:22.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.008+0000 7f1237400640 10 client.15831 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:07:22.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.010+0000 7f1237400640 1 -- 172.21.15.67:0/1736181772 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5558f3ac1800 con 0x5558f3b65610 2024-03-28T16:07:22.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.010+0000 7f1237400640 10 client.15831 handle_command_reply: tid=0 2024-03-28T16:07:22.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.010+0000 7f125b1b2740 1 client.15831 shutdown 2024-03-28T16:07:22.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.010+0000 7f125b1b2740 1 -- 172.21.15.67:0/1736181772 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558f3b4f810 msgr2=0x5558f3b59a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:22.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.011+0000 7f125b1b2740 1 --2- 172.21.15.67:0/1736181772 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558f3b4f810 0x5558f3b59a50 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f1250102720 tx=0x7f1244009260 comp rx=0 tx=0).stop 2024-03-28T16:07:22.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.011+0000 7f125b1b2740 1 -- 172.21.15.67:0/1736181772 shutdown_connections 2024-03-28T16:07:22.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.011+0000 7f125b1b2740 1 --2- 172.21.15.67:0/1736181772 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5558f3b65610 0x5558f3b67a80 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:22.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.011+0000 7f125b1b2740 1 --2- 172.21.15.67:0/1736181772 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5558f3b513a0 0x5558f3b60fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:22.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.011+0000 7f125b1b2740 1 --2- 172.21.15.67:0/1736181772 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5558f3b509e0 0x5558f3b59f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:22.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.011+0000 7f125b1b2740 1 --2- 172.21.15.67:0/1736181772 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558f3b4f810 0x5558f3b59a50 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:22.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.011+0000 7f125b1b2740 1 -- 172.21.15.67:0/1736181772 >> 172.21.15.67:0/1736181772 conn(0x5558f3b4ac00 msgr2=0x5558f3b49eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:22.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.011+0000 7f1237400640 0 client.15831 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:07:22.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.011+0000 7f125b1b2740 1 -- 172.21.15.67:0/1736181772 shutdown_connections 2024-03-28T16:07:22.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.011+0000 7f125b1b2740 1 -- 172.21.15.67:0/1736181772 wait complete. 2024-03-28T16:07:22.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.011+0000 7f125b1b2740 20 client.15831 trim_cache size 0 max 16384 2024-03-28T16:07:22.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.029+0000 7f125b1b2740 1 Processor -- start 2024-03-28T16:07:22.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.029+0000 7f125b1b2740 1 -- start start 2024-03-28T16:07:22.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.030+0000 7f125b1b2740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5558f3b4f810 0x5558f3b53c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:22.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.030+0000 7f125b1b2740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5558f3b509e0 0x5558f3b60f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:22.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.030+0000 7f125b1b2740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558f3b513a0 0x5558f3b54150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:22.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.030+0000 7f125b1b2740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12501a0970 con 0x5558f3b513a0 2024-03-28T16:07:22.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.030+0000 7f125b1b2740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12501b7e70 con 0x5558f3b509e0 2024-03-28T16:07:22.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.030+0000 7f125b1b2740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12501a0f70 con 0x5558f3b4f810 2024-03-28T16:07:22.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.030+0000 7f1256400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558f3b513a0 0x5558f3b54150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:22.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.030+0000 7f1256400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558f3b513a0 0x5558f3b54150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58732/0 (socket says 172.21.15.67:58732) 2024-03-28T16:07:22.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.030+0000 7f1256400640 1 -- 172.21.15.67:0/3624032814 learned_addr learned my addr 172.21.15.67:0/3624032814 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:07:22.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.031+0000 7f1255000640 1 --2- 172.21.15.67:0/3624032814 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5558f3b509e0 0x5558f3b60f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:22.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.031+0000 7f1256400640 1 -- 172.21.15.67:0/3624032814 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5558f3b4f810 msgr2=0x5558f3b53c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:22.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.031+0000 7f1256400640 1 --2- 172.21.15.67:0/3624032814 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5558f3b4f810 0x5558f3b53c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:22.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.031+0000 7f1256400640 1 -- 172.21.15.67:0/3624032814 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5558f3b509e0 msgr2=0x5558f3b60f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:22.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.031+0000 7f1256400640 1 --2- 172.21.15.67:0/3624032814 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5558f3b509e0 0x5558f3b60f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:22.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.031+0000 7f1256400640 1 -- 172.21.15.67:0/3624032814 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5558f3b67df0 con 0x5558f3b513a0 2024-03-28T16:07:22.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.031+0000 7f1255000640 1 --2- 172.21.15.67:0/3624032814 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5558f3b509e0 0x5558f3b60f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:07:22.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.031+0000 7f1256400640 1 --2- 172.21.15.67:0/3624032814 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558f3b513a0 0x5558f3b54150 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x5558f3b4bb70 tx=0x7f1248009470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74192262ebde58f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:22.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.031+0000 7f1242000640 1 -- 172.21.15.67:0/3624032814 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1248028070 con 0x5558f3b513a0 2024-03-28T16:07:22.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.032+0000 7f1242000640 1 -- 172.21.15.67:0/3624032814 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1248003e20 con 0x5558f3b513a0 2024-03-28T16:07:22.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.032+0000 7f1242000640 1 -- 172.21.15.67:0/3624032814 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1248016040 con 0x5558f3b513a0 2024-03-28T16:07:22.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.032+0000 7f125b1b2740 1 -- 172.21.15.67:0/3624032814 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558f3b513a0 msgr2=0x5558f3b54150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:22.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.032+0000 7f125b1b2740 1 --2- 172.21.15.67:0/3624032814 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558f3b513a0 0x5558f3b54150 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x5558f3b4bb70 tx=0x7f1248009470 comp rx=0 tx=0).stop 2024-03-28T16:07:22.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.032+0000 7f125b1b2740 1 -- 172.21.15.67:0/3624032814 shutdown_connections 2024-03-28T16:07:22.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.032+0000 7f125b1b2740 1 --2- 172.21.15.67:0/3624032814 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558f3b513a0 0x5558f3b54150 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:22.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.032+0000 7f125b1b2740 1 --2- 172.21.15.67:0/3624032814 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5558f3b509e0 0x5558f3b60f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:22.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.032+0000 7f125b1b2740 1 --2- 172.21.15.67:0/3624032814 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5558f3b4f810 0x5558f3b53c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:22.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.032+0000 7f125b1b2740 1 -- 172.21.15.67:0/3624032814 >> 172.21.15.67:0/3624032814 conn(0x5558f3b4ac00 msgr2=0x5558f3b4e6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:22.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.033+0000 7f125b1b2740 1 -- 172.21.15.67:0/3624032814 shutdown_connections 2024-03-28T16:07:22.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.033+0000 7f125b1b2740 1 -- 172.21.15.67:0/3624032814 wait complete. 2024-03-28T16:07:22.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.033+0000 7f125b1b2740 1 -- start start 2024-03-28T16:07:22.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.033+0000 7f125b1b2740 1 Processor -- start 2024-03-28T16:07:22.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.034+0000 7f125b1b2740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558f3b4f810 0x5558f3b5ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:22.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.034+0000 7f125b1b2740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5558f3b509e0 0x5558f3b5f320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:22.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.034+0000 7f125b1b2740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5558f3b513a0 0x5558f3b60330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:22.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.034+0000 7f125b1b2740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12501a1570 con 0x5558f3b4f810 2024-03-28T16:07:22.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.034+0000 7f125b1b2740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12501b7810 con 0x5558f3b513a0 2024-03-28T16:07:22.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.034+0000 7f125b1b2740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12501a1bd0 con 0x5558f3b509e0 2024-03-28T16:07:22.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.034+0000 7f125b1b2740 10 client.0 _ll_register_callbacks cb 0x7f123c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:07:22.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.034+0000 7f1255a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558f3b4f810 0x5558f3b5ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:22.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.034+0000 7f1255a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558f3b4f810 0x5558f3b5ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58738/0 (socket says 172.21.15.67:58738) 2024-03-28T16:07:22.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.034+0000 7f1255a00640 1 -- 172.21.15.67:0/1720192297 learned_addr learned my addr 172.21.15.67:0/1720192297 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:07:22.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.034+0000 7f1256400640 1 --2- 172.21.15.67:0/1720192297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5558f3b513a0 0x5558f3b60330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:22.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.034+0000 7f1255a00640 1 -- 172.21.15.67:0/1720192297 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5558f3b509e0 msgr2=0x5558f3b5f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:22.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.034+0000 7f1255a00640 1 --2- 172.21.15.67:0/1720192297 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5558f3b509e0 0x5558f3b5f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:22.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.034+0000 7f1255a00640 1 -- 172.21.15.67:0/1720192297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5558f3b513a0 msgr2=0x5558f3b60330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:22.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.034+0000 7f1255a00640 1 --2- 172.21.15.67:0/1720192297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5558f3b513a0 0x5558f3b60330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:22.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.035+0000 7f1255a00640 1 -- 172.21.15.67:0/1720192297 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5558f3b60920 con 0x5558f3b4f810 2024-03-28T16:07:22.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.035+0000 7f1256400640 1 --2- 172.21.15.67:0/1720192297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5558f3b513a0 0x5558f3b60330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:07:22.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.035+0000 7f1255a00640 1 --2- 172.21.15.67:0/1720192297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558f3b4f810 0x5558f3b5ede0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f1244009820 tx=0x7f124402c320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3e379fae72f7944 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:22.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.035+0000 7f1237400640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:07:22.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.035+0000 7f1237400640 1 -- 172.21.15.67:0/1720192297 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f124404f030 con 0x5558f3b4f810 2024-03-28T16:07:22.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.035+0000 7f1237400640 1 -- 172.21.15.67:0/1720192297 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f124404bce0 con 0x5558f3b4f810 2024-03-28T16:07:22.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.035+0000 7f125b1b2740 1 -- 172.21.15.67:0/1720192297 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1248003e20 con 0x5558f3b4f810 2024-03-28T16:07:22.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.036+0000 7f1237400640 1 -- 172.21.15.67:0/1720192297 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f124406c050 con 0x5558f3b4f810 2024-03-28T16:07:22.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.036+0000 7f125b1b2740 10 client.15837 fetch_fsmap learned FSMap version 19 2024-03-28T16:07:22.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.036+0000 7f125b1b2740 1 -- 172.21.15.67:0/1720192297 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5558f3bc1550 con 0x5558f3b4f810 2024-03-28T16:07:22.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.036+0000 7f1237400640 1 -- 172.21.15.67:0/1720192297 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1248003e20 con 0x5558f3b4f810 2024-03-28T16:07:22.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.036+0000 7f1237400640 1 -- 172.21.15.67:0/1720192297 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f124405c040 con 0x5558f3b4f810 2024-03-28T16:07:22.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.036+0000 7f125b1b2740 10 client.15837 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:07:22.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.036+0000 7f125b1b2740 10 client.15837 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:07:22.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.036+0000 7f125b1b2740 20 client.15837 populate_metadata read hostname 'smithi067' 2024-03-28T16:07:22.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.036+0000 7f125b1b2740 1 --2- 172.21.15.67:0/1720192297 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5558f3bc1ed0 0x5558f3bc42f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:22.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.036+0000 7f125b1b2740 4 client.15837 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T16:07:22.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.036+0000 7f125b1b2740 1 -- 172.21.15.67:0/1720192297 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5558f3abbb50 con 0x5558f3bc1ed0 2024-03-28T16:07:22.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.037+0000 7f1255000640 1 --2- 172.21.15.67:0/1720192297 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5558f3bc1ed0 0x5558f3bc42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:22.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.038+0000 7f1255000640 1 --2- 172.21.15.67:0/1720192297 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5558f3bc1ed0 0x5558f3bc42f0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb25504cd0e3bc7a server_cookie=da9289132a840ad4 in_seq=0 out_seq=0 2024-03-28T16:07:22.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.038+0000 7f1237400640 10 client.15837 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:07:22.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.038+0000 7f1237400640 1 -- 172.21.15.67:0/1720192297 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5558f3abbb50 con 0x5558f3bc1ed0 2024-03-28T16:07:22.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.038+0000 7f1237400640 10 client.15837 handle_command_reply: tid=0 2024-03-28T16:07:22.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.038+0000 7f125b1b2740 1 client.15837 shutdown 2024-03-28T16:07:22.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.039+0000 7f125b1b2740 1 -- 172.21.15.67:0/1720192297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558f3b4f810 msgr2=0x5558f3b5ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:22.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.039+0000 7f125b1b2740 1 --2- 172.21.15.67:0/1720192297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558f3b4f810 0x5558f3b5ede0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f1244009820 tx=0x7f124402c320 comp rx=0 tx=0).stop 2024-03-28T16:07:22.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.039+0000 7f125b1b2740 1 -- 172.21.15.67:0/1720192297 shutdown_connections 2024-03-28T16:07:22.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.039+0000 7f125b1b2740 1 --2- 172.21.15.67:0/1720192297 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5558f3bc1ed0 0x5558f3bc42f0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:22.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.039+0000 7f125b1b2740 1 --2- 172.21.15.67:0/1720192297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5558f3b513a0 0x5558f3b60330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:22.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.039+0000 7f125b1b2740 1 --2- 172.21.15.67:0/1720192297 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5558f3b509e0 0x5558f3b5f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:22.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.039+0000 7f125b1b2740 1 --2- 172.21.15.67:0/1720192297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558f3b4f810 0x5558f3b5ede0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:22.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.039+0000 7f1237400640 0 client.15837 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:07:22.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.039+0000 7f125b1b2740 1 -- 172.21.15.67:0/1720192297 >> 172.21.15.67:0/1720192297 conn(0x5558f3b4ac00 msgr2=0x5558f3b49620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:22.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.039+0000 7f125b1b2740 1 -- 172.21.15.67:0/1720192297 shutdown_connections 2024-03-28T16:07:22.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.039+0000 7f125b1b2740 1 -- 172.21.15.67:0/1720192297 wait complete. 2024-03-28T16:07:22.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.040+0000 7f125b1b2740 20 client.15837 trim_cache size 0 max 16384 2024-03-28T16:07:22.041 INFO:teuthology.orchestra.run.smithi067.stdout:[] 2024-03-28T16:07:22.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.040+0000 7f1258400640 1 -- 172.21.15.67:0/2032214024 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f12380a28d0 msgr2=0x7f12380a4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:22.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.040+0000 7f1258400640 1 --2- 172.21.15.67:0/2032214024 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f12380a28d0 0x7f12380a4d90 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f1244005ab0 tx=0x7f1244000bc0 comp rx=0 tx=0).stop 2024-03-28T16:07:22.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.041+0000 7f1258400640 1 -- 172.21.15.67:0/2032214024 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12501028c0 msgr2=0x7f1250110970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:22.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.041+0000 7f1258400640 1 --2- 172.21.15.67:0/2032214024 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12501028c0 0x7f1250110970 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f123c00e9f0 tx=0x7f123c00eeb0 comp rx=0 tx=0).stop 2024-03-28T16:07:22.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.041+0000 7f1258400640 1 -- 172.21.15.67:0/2032214024 shutdown_connections 2024-03-28T16:07:22.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.041+0000 7f1258400640 1 --2- 172.21.15.67:0/2032214024 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f12380a28d0 0x7f12380a4d90 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:22.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.041+0000 7f1258400640 1 --2- 172.21.15.67:0/2032214024 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1250103280 0x7f125010cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:22.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.041+0000 7f1258400640 1 --2- 172.21.15.67:0/2032214024 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12501028c0 0x7f1250110970 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:22.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.041+0000 7f1258400640 1 --2- 172.21.15.67:0/2032214024 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f12501016d0 0x7f1250110430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:22.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.041+0000 7f1258400640 1 -- 172.21.15.67:0/2032214024 >> 172.21.15.67:0/2032214024 conn(0x7f12500fcc40 msgr2=0x7f12500fd300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:22.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.041+0000 7f1258400640 1 -- 172.21.15.67:0/2032214024 shutdown_connections 2024-03-28T16:07:22.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:22.043+0000 7f1258400640 1 -- 172.21.15.67:0/2032214024 wait complete. 2024-03-28T16:07:22.064 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:07:21 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/4050106436' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:07:22.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:07:21 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/4050106436' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:07:22.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:07:21 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/4050106436' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:07:23.067 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T16:07:23.067 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T16:07:23.068 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T16:07:23.168 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:07:22 smithi067 ceph-mon[31707]: pgmap v553: 97 pgs: 97 active+clean; 785 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 30 MiB/s wr, 23 op/s 2024-03-28T16:07:23.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:07:22 smithi150 ceph-mon[34825]: pgmap v553: 97 pgs: 97 active+clean; 785 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 30 MiB/s wr, 23 op/s 2024-03-28T16:07:23.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.339+0000 7f9e6e800640 1 -- 172.21.15.67:0/3134495935 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e68103260 msgr2=0x7f9e68109af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:23.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.339+0000 7f9e6e800640 1 --2- 172.21.15.67:0/3134495935 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e68103260 0x7f9e68109af0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f9e60009ca0 tx=0x7f9e6002c100 comp rx=0 tx=0).stop 2024-03-28T16:07:23.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.339+0000 7f9e6e800640 1 -- 172.21.15.67:0/3134495935 shutdown_connections 2024-03-28T16:07:23.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.340+0000 7f9e6e800640 1 --2- 172.21.15.67:0/3134495935 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e68103260 0x7f9e68109af0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:23.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.340+0000 7f9e6e800640 1 --2- 172.21.15.67:0/3134495935 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9e681028a0 0x7f9e68102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:23.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.340+0000 7f9e6e800640 1 --2- 172.21.15.67:0/3134495935 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9e681016b0 0x7f9e68101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:23.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.340+0000 7f9e6e800640 1 -- 172.21.15.67:0/3134495935 >> 172.21.15.67:0/3134495935 conn(0x7f9e680fcc40 msgr2=0x7f9e680ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:23.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.340+0000 7f9e6e800640 1 -- 172.21.15.67:0/3134495935 shutdown_connections 2024-03-28T16:07:23.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.340+0000 7f9e6e800640 1 -- 172.21.15.67:0/3134495935 wait complete. 2024-03-28T16:07:23.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.341+0000 7f9e6e800640 1 Processor -- start 2024-03-28T16:07:23.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.341+0000 7f9e6e800640 1 -- start start 2024-03-28T16:07:23.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.342+0000 7f9e6e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e681016b0 0x7f9e68195690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:23.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.342+0000 7f9e6e800640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9e681028a0 0x7f9e68195bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:23.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.342+0000 7f9e6e800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9e68103260 0x7f9e6819cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:23.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.342+0000 7f9e6e800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e6810b000 con 0x7f9e681016b0 2024-03-28T16:07:23.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.342+0000 7f9e6e800640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e6810ae80 con 0x7f9e681028a0 2024-03-28T16:07:23.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.342+0000 7f9e6e800640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e6810b180 con 0x7f9e68103260 2024-03-28T16:07:23.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.342+0000 7f9e67400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e681016b0 0x7f9e68195690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:23.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.343+0000 7f9e67400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e681016b0 0x7f9e68195690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58760/0 (socket says 172.21.15.67:58760) 2024-03-28T16:07:23.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.343+0000 7f9e67400640 1 -- 172.21.15.67:0/1707274791 learned_addr learned my addr 172.21.15.67:0/1707274791 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:07:23.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.343+0000 7f9e66a00640 1 --2- 172.21.15.67:0/1707274791 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9e681028a0 0x7f9e68195bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:23.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.343+0000 7f9e67400640 1 -- 172.21.15.67:0/1707274791 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9e68103260 msgr2=0x7f9e6819cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:23.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.343+0000 7f9e67400640 1 --2- 172.21.15.67:0/1707274791 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9e68103260 0x7f9e6819cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:23.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.343+0000 7f9e67400640 1 -- 172.21.15.67:0/1707274791 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9e681028a0 msgr2=0x7f9e68195bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:23.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.343+0000 7f9e67400640 1 --2- 172.21.15.67:0/1707274791 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9e681028a0 0x7f9e68195bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:23.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.343+0000 7f9e67400640 1 -- 172.21.15.67:0/1707274791 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e6819d2b0 con 0x7f9e681016b0 2024-03-28T16:07:23.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.344+0000 7f9e67400640 1 --2- 172.21.15.67:0/1707274791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e681016b0 0x7f9e68195690 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f9e5800d770 tx=0x7f9e5800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:23.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.344+0000 7f9e5fe00640 1 -- 172.21.15.67:0/1707274791 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e58017070 con 0x7f9e681016b0 2024-03-28T16:07:23.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.344+0000 7f9e5fe00640 1 -- 172.21.15.67:0/1707274791 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9e58009c30 con 0x7f9e681016b0 2024-03-28T16:07:23.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.344+0000 7f9e6e800640 1 -- 172.21.15.67:0/1707274791 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e6819d5a0 con 0x7f9e681016b0 2024-03-28T16:07:23.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.344+0000 7f9e5fe00640 1 -- 172.21.15.67:0/1707274791 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e58003d00 con 0x7f9e681016b0 2024-03-28T16:07:23.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.344+0000 7f9e6e800640 1 -- 172.21.15.67:0/1707274791 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e6819daf0 con 0x7f9e681016b0 2024-03-28T16:07:23.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.350+0000 7f9e7159f740 1 Processor -- start 2024-03-28T16:07:23.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.350+0000 7f9e5fe00640 1 -- 172.21.15.67:0/1707274791 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f9e58003ea0 con 0x7f9e681016b0 2024-03-28T16:07:23.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.350+0000 7f9e7159f740 1 -- start start 2024-03-28T16:07:23.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.350+0000 7f9e7159f740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563670da80a0 0x563670da84c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:23.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.351+0000 7f9e7159f740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563670da8a90 0x563670dad590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:23.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.351+0000 7f9e7159f740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563670dadc00 0x563670db0090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:23.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.351+0000 7f9e7159f740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x563670d0deb0 con 0x563670dadc00 2024-03-28T16:07:23.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.351+0000 7f9e7159f740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x563670cb6a60 con 0x563670da80a0 2024-03-28T16:07:23.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.351+0000 7f9e7159f740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x563670d09f50 con 0x563670da8a90 2024-03-28T16:07:23.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:07:22 smithi183 ceph-mon[35580]: pgmap v553: 97 pgs: 97 active+clean; 785 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 30 MiB/s wr, 23 op/s 2024-03-28T16:07:23.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.351+0000 7f9e66a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563670dadc00 0x563670db0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:23.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.351+0000 7f9e66a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563670dadc00 0x563670db0090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58776/0 (socket says 172.21.15.67:58776) 2024-03-28T16:07:23.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.351+0000 7f9e66a00640 1 -- 172.21.15.67:0/3819063152 learned_addr learned my addr 172.21.15.67:0/3819063152 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:07:23.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.351+0000 7f9e67e00640 1 --2- 172.21.15.67:0/3819063152 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563670da80a0 0x563670da84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:23.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.352+0000 7f9e67400640 1 --2- 172.21.15.67:0/3819063152 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563670da8a90 0x563670dad590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:23.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.352+0000 7f9e66a00640 1 -- 172.21.15.67:0/3819063152 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563670da8a90 msgr2=0x563670dad590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:23.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.352+0000 7f9e66a00640 1 --2- 172.21.15.67:0/3819063152 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563670da8a90 0x563670dad590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:23.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.352+0000 7f9e66a00640 1 -- 172.21.15.67:0/3819063152 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563670da80a0 msgr2=0x563670da84c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:23.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.352+0000 7f9e66a00640 1 --2- 172.21.15.67:0/3819063152 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563670da80a0 0x563670da84c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:23.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.352+0000 7f9e66a00640 1 -- 172.21.15.67:0/3819063152 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563670aea7e0 con 0x563670dadc00 2024-03-28T16:07:23.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.352+0000 7f9e67400640 1 --2- 172.21.15.67:0/3819063152 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563670da8a90 0x563670dad590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:07:23.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.352+0000 7f9e66a00640 1 --2- 172.21.15.67:0/3819063152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563670dadc00 0x563670db0090 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f9e5400beb0 tx=0x7f9e54013520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c04d85db41f13fa server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:23.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.352+0000 7f9e5fe00640 1 --2- 172.21.15.67:0/1707274791 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f9e4c0a2800 0x7f9e4c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:23.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.353+0000 7f9e5fe00640 1 -- 172.21.15.67:0/1707274791 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f9e580ce750 con 0x7f9e681016b0 2024-03-28T16:07:23.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.353+0000 7f9e5cc00640 1 -- 172.21.15.67:0/3819063152 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e54019670 con 0x563670dadc00 2024-03-28T16:07:23.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.353+0000 7f9e67e00640 1 --2- 172.21.15.67:0/1707274791 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f9e4c0a2800 0x7f9e4c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:23.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.353+0000 7f9e5cc00640 1 -- 172.21.15.67:0/3819063152 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9e5401e070 con 0x563670dadc00 2024-03-28T16:07:23.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.353+0000 7f9e5cc00640 1 -- 172.21.15.67:0/3819063152 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e54019670 con 0x563670dadc00 2024-03-28T16:07:23.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.354+0000 7f9e7159f740 1 -- 172.21.15.67:0/3819063152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563670dadc00 msgr2=0x563670db0090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:23.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.354+0000 7f9e7159f740 1 --2- 172.21.15.67:0/3819063152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563670dadc00 0x563670db0090 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f9e5400beb0 tx=0x7f9e54013520 comp rx=0 tx=0).stop 2024-03-28T16:07:23.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.355+0000 7f9e7159f740 1 -- 172.21.15.67:0/3819063152 shutdown_connections 2024-03-28T16:07:23.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.355+0000 7f9e7159f740 1 --2- 172.21.15.67:0/3819063152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563670dadc00 0x563670db0090 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:23.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.355+0000 7f9e7159f740 1 --2- 172.21.15.67:0/3819063152 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563670da8a90 0x563670dad590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:23.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.355+0000 7f9e7159f740 1 --2- 172.21.15.67:0/3819063152 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563670da80a0 0x563670da84c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:23.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.355+0000 7f9e7159f740 1 -- 172.21.15.67:0/3819063152 >> 172.21.15.67:0/3819063152 conn(0x563670da3610 msgr2=0x563670da3a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:23.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.355+0000 7f9e67e00640 1 --2- 172.21.15.67:0/1707274791 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f9e4c0a2800 0x7f9e4c0a4cc0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f9e60005cc0 tx=0x7f9e60005c50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:23.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.355+0000 7f9e7159f740 1 -- 172.21.15.67:0/3819063152 shutdown_connections 2024-03-28T16:07:23.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.355+0000 7f9e7159f740 1 -- 172.21.15.67:0/3819063152 wait complete. 2024-03-28T16:07:23.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.356+0000 7f9e7159f740 1 -- start start 2024-03-28T16:07:23.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.356+0000 7f9e7159f740 1 Processor -- start 2024-03-28T16:07:23.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.356+0000 7f9e7159f740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563670da80a0 0x563670db21f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:23.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.356+0000 7f9e7159f740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563670da8a90 0x563670db2730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:23.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.356+0000 7f9e7159f740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563670dadc00 0x563670db9770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:23.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.356+0000 7f9e7159f740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x563670db0bb0 con 0x563670dadc00 2024-03-28T16:07:23.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.356+0000 7f9e7159f740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x563670cc9eb0 con 0x563670da8a90 2024-03-28T16:07:23.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.356+0000 7f9e7159f740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x563670db0d30 con 0x563670da80a0 2024-03-28T16:07:23.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.357+0000 7f9e7159f740 10 client.0 _ll_register_callbacks cb 0x563670cbe150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:07:23.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.357+0000 7f9e67e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563670dadc00 0x563670db9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:23.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.357+0000 7f9e67e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563670dadc00 0x563670db9770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58782/0 (socket says 172.21.15.67:58782) 2024-03-28T16:07:23.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.357+0000 7f9e67e00640 1 -- 172.21.15.67:0/1968184279 learned_addr learned my addr 172.21.15.67:0/1968184279 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:07:23.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.357+0000 7f9e66a00640 1 --2- 172.21.15.67:0/1968184279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563670da8a90 0x563670db2730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:23.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.357+0000 7f9e67400640 1 --2- 172.21.15.67:0/1968184279 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563670da80a0 0x563670db21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:23.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.357+0000 7f9e67e00640 1 -- 172.21.15.67:0/1968184279 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563670da80a0 msgr2=0x563670db21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:23.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.357+0000 7f9e67e00640 1 --2- 172.21.15.67:0/1968184279 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563670da80a0 0x563670db21f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:23.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.357+0000 7f9e67e00640 1 -- 172.21.15.67:0/1968184279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563670da8a90 msgr2=0x563670db2730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:23.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.357+0000 7f9e67e00640 1 --2- 172.21.15.67:0/1968184279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563670da8a90 0x563670db2730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:23.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.357+0000 7f9e67e00640 1 -- 172.21.15.67:0/1968184279 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563670b4af70 con 0x563670dadc00 2024-03-28T16:07:23.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.357+0000 7f9e67400640 1 --2- 172.21.15.67:0/1968184279 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563670da80a0 0x563670db21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:07:23.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.358+0000 7f9e67e00640 1 --2- 172.21.15.67:0/1968184279 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563670dadc00 0x563670db9770 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f9e60036000 tx=0x7f9e600095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=218e68f592e1623 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:23.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.358+0000 7f9e51600640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:07:23.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.358+0000 7f9e51600640 1 -- 172.21.15.67:0/1968184279 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e60044070 con 0x563670dadc00 2024-03-28T16:07:23.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.358+0000 7f9e51600640 1 -- 172.21.15.67:0/1968184279 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f9e600041b0 con 0x563670dadc00 2024-03-28T16:07:23.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.358+0000 7f9e7159f740 1 -- 172.21.15.67:0/1968184279 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9e5401e070 con 0x563670dadc00 2024-03-28T16:07:23.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.358+0000 7f9e51600640 1 -- 172.21.15.67:0/1968184279 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9e60039040 con 0x563670dadc00 2024-03-28T16:07:23.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.358+0000 7f9e7159f740 10 client.15849 fetch_fsmap learned FSMap version 19 2024-03-28T16:07:23.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.359+0000 7f9e7159f740 1 -- 172.21.15.67:0/1968184279 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5636709e2170 con 0x563670dadc00 2024-03-28T16:07:23.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.359+0000 7f9e51600640 1 -- 172.21.15.67:0/1968184279 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9e5401e070 con 0x563670dadc00 2024-03-28T16:07:23.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.359+0000 7f9e51600640 1 -- 172.21.15.67:0/1968184279 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f9e6004ba20 con 0x563670dadc00 2024-03-28T16:07:23.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.359+0000 7f9e7159f740 10 client.15849 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:07:23.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.359+0000 7f9e7159f740 10 client.15849 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:07:23.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.359+0000 7f9e7159f740 20 client.15849 populate_metadata read hostname 'smithi067' 2024-03-28T16:07:23.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.359+0000 7f9e7159f740 1 --2- 172.21.15.67:0/1968184279 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x563670dbe110 0x563670dc05b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:23.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.359+0000 7f9e7159f740 4 client.15849 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:07:23.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.359+0000 7f9e7159f740 1 -- 172.21.15.67:0/1968184279 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563670d1b3f0 con 0x563670dbe110 2024-03-28T16:07:23.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.360+0000 7f9e67400640 1 --2- 172.21.15.67:0/1968184279 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x563670dbe110 0x563670dc05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:23.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.362+0000 7f9e67400640 1 --2- 172.21.15.67:0/1968184279 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x563670dbe110 0x563670dc05b0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cd493d9ed1b79006 server_cookie=b28d6035751bf44e in_seq=0 out_seq=0 2024-03-28T16:07:23.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.362+0000 7f9e51600640 10 client.15849 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:07:23.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.363+0000 7f9e51600640 1 -- 172.21.15.67:0/1968184279 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x563670d1b3f0 con 0x563670dbe110 2024-03-28T16:07:23.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.363+0000 7f9e51600640 10 client.15849 handle_command_reply: tid=0 2024-03-28T16:07:23.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.363+0000 7f9e7159f740 1 client.15849 shutdown 2024-03-28T16:07:23.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.363+0000 7f9e7159f740 1 -- 172.21.15.67:0/1968184279 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563670dadc00 msgr2=0x563670db9770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:23.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.363+0000 7f9e7159f740 1 --2- 172.21.15.67:0/1968184279 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563670dadc00 0x563670db9770 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f9e60036000 tx=0x7f9e600095c0 comp rx=0 tx=0).stop 2024-03-28T16:07:23.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.363+0000 7f9e7159f740 1 -- 172.21.15.67:0/1968184279 shutdown_connections 2024-03-28T16:07:23.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.363+0000 7f9e7159f740 1 --2- 172.21.15.67:0/1968184279 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x563670dbe110 0x563670dc05b0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:23.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.363+0000 7f9e7159f740 1 --2- 172.21.15.67:0/1968184279 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563670dadc00 0x563670db9770 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:23.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.363+0000 7f9e7159f740 1 --2- 172.21.15.67:0/1968184279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563670da8a90 0x563670db2730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:23.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.363+0000 7f9e51600640 0 client.15849 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:07:23.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.363+0000 7f9e7159f740 1 --2- 172.21.15.67:0/1968184279 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563670da80a0 0x563670db21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:23.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.363+0000 7f9e7159f740 1 -- 172.21.15.67:0/1968184279 >> 172.21.15.67:0/1968184279 conn(0x563670da3610 msgr2=0x563670da2830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:23.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.363+0000 7f9e7159f740 1 -- 172.21.15.67:0/1968184279 shutdown_connections 2024-03-28T16:07:23.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.363+0000 7f9e7159f740 1 -- 172.21.15.67:0/1968184279 wait complete. 2024-03-28T16:07:23.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.364+0000 7f9e7159f740 20 client.15849 trim_cache size 0 max 16384 2024-03-28T16:07:23.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.389+0000 7f9e7159f740 1 Processor -- start 2024-03-28T16:07:23.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.390+0000 7f9e7159f740 1 -- start start 2024-03-28T16:07:23.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.390+0000 7f9e7159f740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563670da80a0 0x563670da26b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:23.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.390+0000 7f9e7159f740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563670da8a90 0x563670da2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:23.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.390+0000 7f9e7159f740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563670dadc00 0x563670db9730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:23.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.390+0000 7f9e7159f740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e4800c3c0 con 0x563670da80a0 2024-03-28T16:07:23.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.390+0000 7f9e7159f740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e4800e980 con 0x563670dadc00 2024-03-28T16:07:23.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.390+0000 7f9e7159f740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e4800c9c0 con 0x563670da8a90 2024-03-28T16:07:23.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.391+0000 7f9e67400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563670da80a0 0x563670da26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:23.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.391+0000 7f9e67400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563670da80a0 0x563670da26b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58792/0 (socket says 172.21.15.67:58792) 2024-03-28T16:07:23.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.391+0000 7f9e67400640 1 -- 172.21.15.67:0/2112601889 learned_addr learned my addr 172.21.15.67:0/2112601889 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:07:23.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.391+0000 7f9e66a00640 1 --2- 172.21.15.67:0/2112601889 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563670da8a90 0x563670da2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:23.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.391+0000 7f9e67e00640 1 --2- 172.21.15.67:0/2112601889 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563670dadc00 0x563670db9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:23.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.392+0000 7f9e67400640 1 -- 172.21.15.67:0/2112601889 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563670da8a90 msgr2=0x563670da2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:23.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.392+0000 7f9e67400640 1 --2- 172.21.15.67:0/2112601889 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563670da8a90 0x563670da2bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:23.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.392+0000 7f9e67400640 1 -- 172.21.15.67:0/2112601889 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563670dadc00 msgr2=0x563670db9730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:23.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.392+0000 7f9e67400640 1 --2- 172.21.15.67:0/2112601889 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563670dadc00 0x563670db9730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:23.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.392+0000 7f9e67400640 1 -- 172.21.15.67:0/2112601889 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563670da4080 con 0x563670da80a0 2024-03-28T16:07:23.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.392+0000 7f9e67400640 1 --2- 172.21.15.67:0/2112601889 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563670da80a0 0x563670da26b0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f9e58097ad0 tx=0x7f9e580bb040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cef3e5dca138719 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:23.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.393+0000 7f9e53e00640 1 -- 172.21.15.67:0/2112601889 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e58020040 con 0x563670da80a0 2024-03-28T16:07:23.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.393+0000 7f9e53e00640 1 -- 172.21.15.67:0/2112601889 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9e5809b100 con 0x563670da80a0 2024-03-28T16:07:23.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.393+0000 7f9e53e00640 1 -- 172.21.15.67:0/2112601889 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e58097770 con 0x563670da80a0 2024-03-28T16:07:23.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.395+0000 7f9e7159f740 1 -- 172.21.15.67:0/2112601889 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563670da80a0 msgr2=0x563670da26b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:23.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.395+0000 7f9e7159f740 1 --2- 172.21.15.67:0/2112601889 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563670da80a0 0x563670da26b0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f9e58097ad0 tx=0x7f9e580bb040 comp rx=0 tx=0).stop 2024-03-28T16:07:23.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.395+0000 7f9e7159f740 1 -- 172.21.15.67:0/2112601889 shutdown_connections 2024-03-28T16:07:23.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.395+0000 7f9e7159f740 1 --2- 172.21.15.67:0/2112601889 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563670dadc00 0x563670db9730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:23.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.395+0000 7f9e7159f740 1 --2- 172.21.15.67:0/2112601889 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563670da8a90 0x563670da2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:23.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.395+0000 7f9e7159f740 1 --2- 172.21.15.67:0/2112601889 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563670da80a0 0x563670da26b0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:23.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.395+0000 7f9e7159f740 1 -- 172.21.15.67:0/2112601889 >> 172.21.15.67:0/2112601889 conn(0x563670da3610 msgr2=0x563670da1fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:23.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.395+0000 7f9e7159f740 1 -- 172.21.15.67:0/2112601889 shutdown_connections 2024-03-28T16:07:23.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.395+0000 7f9e7159f740 1 -- 172.21.15.67:0/2112601889 wait complete. 2024-03-28T16:07:23.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.396+0000 7f9e7159f740 1 -- start start 2024-03-28T16:07:23.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.397+0000 7f9e7159f740 1 Processor -- start 2024-03-28T16:07:23.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.398+0000 7f9e7159f740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563670da8a90 0x563670db7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:23.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.398+0000 7f9e7159f740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563670dadc00 0x563670db84e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:23.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.398+0000 7f9e7159f740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563670db8a20 0x563670e19a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:23.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.398+0000 7f9e7159f740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e4800d000 con 0x563670dadc00 2024-03-28T16:07:23.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.398+0000 7f9e7159f740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e4800e320 con 0x563670db8a20 2024-03-28T16:07:23.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.398+0000 7f9e7159f740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e4800d660 con 0x563670da8a90 2024-03-28T16:07:23.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.398+0000 7f9e7159f740 10 client.0 _ll_register_callbacks cb 0x7f9e60009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:07:23.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.399+0000 7f9e66a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563670dadc00 0x563670db84e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:23.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.399+0000 7f9e66a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563670dadc00 0x563670db84e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58808/0 (socket says 172.21.15.67:58808) 2024-03-28T16:07:23.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.399+0000 7f9e66a00640 1 -- 172.21.15.67:0/1590995172 learned_addr learned my addr 172.21.15.67:0/1590995172 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:07:23.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.399+0000 7f9e67e00640 1 --2- 172.21.15.67:0/1590995172 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563670db8a20 0x563670e19a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:23.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.400+0000 7f9e67400640 1 --2- 172.21.15.67:0/1590995172 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563670da8a90 0x563670db7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:23.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.400+0000 7f9e66a00640 1 -- 172.21.15.67:0/1590995172 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563670da8a90 msgr2=0x563670db7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:23.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.400+0000 7f9e66a00640 1 --2- 172.21.15.67:0/1590995172 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563670da8a90 0x563670db7fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:23.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.400+0000 7f9e66a00640 1 -- 172.21.15.67:0/1590995172 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563670db8a20 msgr2=0x563670e19a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:23.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.400+0000 7f9e66a00640 1 --2- 172.21.15.67:0/1590995172 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563670db8a20 0x563670e19a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:23.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.400+0000 7f9e66a00640 1 -- 172.21.15.67:0/1590995172 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563670e19fc0 con 0x563670dadc00 2024-03-28T16:07:23.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.401+0000 7f9e66a00640 1 --2- 172.21.15.67:0/1590995172 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563670dadc00 0x563670db84e0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f9e5400beb0 tx=0x7f9e54005250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5e3ed98367b85f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:23.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.401+0000 7f9e51600640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:07:23.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.401+0000 7f9e51600640 1 -- 172.21.15.67:0/1590995172 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e540055c0 con 0x563670dadc00 2024-03-28T16:07:23.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.401+0000 7f9e51600640 1 -- 172.21.15.67:0/1590995172 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f9e54004680 con 0x563670dadc00 2024-03-28T16:07:23.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.402+0000 7f9e7159f740 1 -- 172.21.15.67:0/1590995172 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9e5809b100 con 0x563670dadc00 2024-03-28T16:07:23.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.402+0000 7f9e51600640 1 -- 172.21.15.67:0/1590995172 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9e540048c0 con 0x563670dadc00 2024-03-28T16:07:23.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.402+0000 7f9e7159f740 10 client.15855 fetch_fsmap learned FSMap version 19 2024-03-28T16:07:23.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.402+0000 7f9e7159f740 1 -- 172.21.15.67:0/1590995172 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x563670e1b2e0 con 0x563670dadc00 2024-03-28T16:07:23.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.402+0000 7f9e51600640 1 -- 172.21.15.67:0/1590995172 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9e5809b100 con 0x563670dadc00 2024-03-28T16:07:23.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.403+0000 7f9e51600640 1 -- 172.21.15.67:0/1590995172 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f9e54019b50 con 0x563670dadc00 2024-03-28T16:07:23.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.403+0000 7f9e7159f740 10 client.15855 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:07:23.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.403+0000 7f9e7159f740 10 client.15855 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:07:23.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.403+0000 7f9e7159f740 20 client.15855 populate_metadata read hostname 'smithi067' 2024-03-28T16:07:23.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.403+0000 7f9e7159f740 1 --2- 172.21.15.67:0/1590995172 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x563670e1b940 0x563670e2f280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:23.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.403+0000 7f9e7159f740 4 client.15855 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T16:07:23.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.403+0000 7f9e7159f740 1 -- 172.21.15.67:0/1590995172 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x563670d15740 con 0x563670e1b940 2024-03-28T16:07:23.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.404+0000 7f9e67400640 1 --2- 172.21.15.67:0/1590995172 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x563670e1b940 0x563670e2f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:23.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.405+0000 7f9e67400640 1 --2- 172.21.15.67:0/1590995172 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x563670e1b940 0x563670e2f280 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a18d29adef24878d server_cookie=4bb5ece9fad38de6 in_seq=0 out_seq=0 2024-03-28T16:07:23.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.405+0000 7f9e51600640 10 client.15855 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:07:23.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.407+0000 7f9e51600640 1 -- 172.21.15.67:0/1590995172 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x563670d15740 con 0x563670e1b940 2024-03-28T16:07:23.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.407+0000 7f9e51600640 10 client.15855 handle_command_reply: tid=0 2024-03-28T16:07:23.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.407+0000 7f9e7159f740 1 client.15855 shutdown 2024-03-28T16:07:23.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.407+0000 7f9e7159f740 1 -- 172.21.15.67:0/1590995172 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563670dadc00 msgr2=0x563670db84e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:23.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.407+0000 7f9e7159f740 1 --2- 172.21.15.67:0/1590995172 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563670dadc00 0x563670db84e0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f9e5400beb0 tx=0x7f9e54005250 comp rx=0 tx=0).stop 2024-03-28T16:07:23.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.408+0000 7f9e7159f740 1 -- 172.21.15.67:0/1590995172 shutdown_connections 2024-03-28T16:07:23.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.408+0000 7f9e7159f740 1 --2- 172.21.15.67:0/1590995172 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x563670e1b940 0x563670e2f280 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:23.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.408+0000 7f9e7159f740 1 --2- 172.21.15.67:0/1590995172 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563670db8a20 0x563670e19a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:23.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.408+0000 7f9e7159f740 1 --2- 172.21.15.67:0/1590995172 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563670dadc00 0x563670db84e0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:23.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.408+0000 7f9e7159f740 1 --2- 172.21.15.67:0/1590995172 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563670da8a90 0x563670db7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:23.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.408+0000 7f9e51600640 0 client.15855 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:07:23.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.408+0000 7f9e7159f740 1 -- 172.21.15.67:0/1590995172 >> 172.21.15.67:0/1590995172 conn(0x563670da3610 msgr2=0x563670da9d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:23.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.408+0000 7f9e7159f740 1 -- 172.21.15.67:0/1590995172 shutdown_connections 2024-03-28T16:07:23.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.408+0000 7f9e7159f740 1 -- 172.21.15.67:0/1590995172 wait complete. 2024-03-28T16:07:23.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.408+0000 7f9e7159f740 20 client.15855 trim_cache size 0 max 16384 2024-03-28T16:07:23.409 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-28T16:07:23.410 INFO:teuthology.orchestra.run.smithi067.stdout: "return_code": 0, 2024-03-28T16:07:23.410 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_tag": "4e5f268b-43bd-49e1-a118-21dfec109591", 2024-03-28T16:07:23.410 INFO:teuthology.orchestra.run.smithi067.stdout: "mode": "asynchronous" 2024-03-28T16:07:23.410 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-28T16:07:23.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.409+0000 7f9e6e800640 1 -- 172.21.15.67:0/1707274791 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f9e4c0a2800 msgr2=0x7f9e4c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:23.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.409+0000 7f9e6e800640 1 --2- 172.21.15.67:0/1707274791 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f9e4c0a2800 0x7f9e4c0a4cc0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f9e60005cc0 tx=0x7f9e60005c50 comp rx=0 tx=0).stop 2024-03-28T16:07:23.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.409+0000 7f9e6e800640 1 -- 172.21.15.67:0/1707274791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e681016b0 msgr2=0x7f9e68195690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:23.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.409+0000 7f9e6e800640 1 --2- 172.21.15.67:0/1707274791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e681016b0 0x7f9e68195690 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f9e5800d770 tx=0x7f9e5800dc30 comp rx=0 tx=0).stop 2024-03-28T16:07:23.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.409+0000 7f9e6e800640 1 -- 172.21.15.67:0/1707274791 shutdown_connections 2024-03-28T16:07:23.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.409+0000 7f9e6e800640 1 --2- 172.21.15.67:0/1707274791 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f9e4c0a2800 0x7f9e4c0a4cc0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:23.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.409+0000 7f9e6e800640 1 --2- 172.21.15.67:0/1707274791 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9e68103260 0x7f9e6819cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:23.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.409+0000 7f9e6e800640 1 --2- 172.21.15.67:0/1707274791 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9e681028a0 0x7f9e68195bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:23.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.409+0000 7f9e6e800640 1 --2- 172.21.15.67:0/1707274791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e681016b0 0x7f9e68195690 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:23.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.410+0000 7f9e6e800640 1 -- 172.21.15.67:0/1707274791 >> 172.21.15.67:0/1707274791 conn(0x7f9e680fcc40 msgr2=0x7f9e68105bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:23.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.410+0000 7f9e6e800640 1 -- 172.21.15.67:0/1707274791 shutdown_connections 2024-03-28T16:07:23.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.410+0000 7f9e6e800640 1 -- 172.21.15.67:0/1707274791 wait complete. 2024-03-28T16:07:23.433 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T16:07:23.502 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T16:07:23.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.764+0000 7f9ac3800640 1 -- 172.21.15.67:0/255952023 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9abc104ad0 msgr2=0x7f9abc104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:23.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.764+0000 7f9ac3800640 1 --2- 172.21.15.67:0/255952023 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9abc104ad0 0x7f9abc104f50 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f9aa8009900 tx=0x7f9aa802bc30 comp rx=0 tx=0).stop 2024-03-28T16:07:23.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.764+0000 7f9ac3800640 1 -- 172.21.15.67:0/255952023 shutdown_connections 2024-03-28T16:07:23.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.764+0000 7f9ac3800640 1 --2- 172.21.15.67:0/255952023 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9abc105490 0x7f9abc10bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:23.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.764+0000 7f9ac3800640 1 --2- 172.21.15.67:0/255952023 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9abc104ad0 0x7f9abc104f50 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:23.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.764+0000 7f9ac3800640 1 --2- 172.21.15.67:0/255952023 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9abc1038e0 0x7f9abc103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:23.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.764+0000 7f9ac3800640 1 -- 172.21.15.67:0/255952023 >> 172.21.15.67:0/255952023 conn(0x7f9abc0fee50 msgr2=0x7f9abc101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:23.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.766+0000 7f9ac3800640 1 -- 172.21.15.67:0/255952023 shutdown_connections 2024-03-28T16:07:23.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.766+0000 7f9ac3800640 1 -- 172.21.15.67:0/255952023 wait complete. 2024-03-28T16:07:23.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.767+0000 7f9ac3800640 1 Processor -- start 2024-03-28T16:07:23.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.767+0000 7f9ac3800640 1 -- start start 2024-03-28T16:07:23.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.767+0000 7f9ac3800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9abc1038e0 0x7f9abc199af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:23.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.768+0000 7f9ac3800640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9abc104ad0 0x7f9abc19a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:23.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.768+0000 7f9ac3800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9abc105490 0x7f9abc1a10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:23.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.768+0000 7f9ac3800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9abc10f440 con 0x7f9abc105490 2024-03-28T16:07:23.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.768+0000 7f9ac3800640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9abc10f2c0 con 0x7f9abc104ad0 2024-03-28T16:07:23.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.768+0000 7f9ac3800640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9abc10f5c0 con 0x7f9abc1038e0 2024-03-28T16:07:23.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.768+0000 7f9ac1800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9abc105490 0x7f9abc1a10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:23.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.768+0000 7f9ac1800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9abc105490 0x7f9abc1a10a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58836/0 (socket says 172.21.15.67:58836) 2024-03-28T16:07:23.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.768+0000 7f9ac1800640 1 -- 172.21.15.67:0/487287763 learned_addr learned my addr 172.21.15.67:0/487287763 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:07:23.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.768+0000 7f9abbe00640 1 --2- 172.21.15.67:0/487287763 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9abc104ad0 0x7f9abc19a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:23.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.768+0000 7f9ac0e00640 1 --2- 172.21.15.67:0/487287763 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9abc1038e0 0x7f9abc199af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:23.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.769+0000 7f9ac1800640 1 -- 172.21.15.67:0/487287763 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9abc1038e0 msgr2=0x7f9abc199af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:23.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.769+0000 7f9ac1800640 1 --2- 172.21.15.67:0/487287763 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9abc1038e0 0x7f9abc199af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:23.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.769+0000 7f9ac1800640 1 -- 172.21.15.67:0/487287763 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9abc104ad0 msgr2=0x7f9abc19a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:23.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.769+0000 7f9ac1800640 1 --2- 172.21.15.67:0/487287763 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9abc104ad0 0x7f9abc19a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:23.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.769+0000 7f9ac1800640 1 -- 172.21.15.67:0/487287763 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9abc1a1710 con 0x7f9abc105490 2024-03-28T16:07:23.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.769+0000 7f9ac1800640 1 --2- 172.21.15.67:0/487287763 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9abc105490 0x7f9abc1a10a0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f9ab400eaa0 tx=0x7f9ab400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:23.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.769+0000 7f9ab9600640 1 -- 172.21.15.67:0/487287763 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab400cdb0 con 0x7f9abc105490 2024-03-28T16:07:23.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.769+0000 7f9ac3800640 1 -- 172.21.15.67:0/487287763 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9abc1a1a00 con 0x7f9abc105490 2024-03-28T16:07:23.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.769+0000 7f9ab9600640 1 -- 172.21.15.67:0/487287763 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9ab4005020 con 0x7f9abc105490 2024-03-28T16:07:23.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.770+0000 7f9ab9600640 1 -- 172.21.15.67:0/487287763 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab40116d0 con 0x7f9abc105490 2024-03-28T16:07:23.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.770+0000 7f9ac3800640 1 -- 172.21.15.67:0/487287763 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9abc1a1ed0 con 0x7f9abc105490 2024-03-28T16:07:23.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.774+0000 7f9ac641c740 1 Processor -- start 2024-03-28T16:07:23.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.774+0000 7f9ab9600640 1 -- 172.21.15.67:0/487287763 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f9ab40051d0 con 0x7f9abc105490 2024-03-28T16:07:23.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.774+0000 7f9ac641c740 1 -- start start 2024-03-28T16:07:23.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.774+0000 7f9ac641c740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c13cefde80 0x55c13cefe2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:23.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.774+0000 7f9ac641c740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c13cefe870 0x55c13cf03370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:23.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.774+0000 7f9ac641c740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c13cf039e0 0x55c13cf05e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:23.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.774+0000 7f9ac641c740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c13ce63250 con 0x55c13cefe870 2024-03-28T16:07:23.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.774+0000 7f9ac641c740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c13ce0ba30 con 0x55c13cefde80 2024-03-28T16:07:23.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.774+0000 7f9ac641c740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c13ce5f2f0 con 0x55c13cf039e0 2024-03-28T16:07:23.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.775+0000 7f9ac0e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c13cefe870 0x55c13cf03370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:23.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.775+0000 7f9ac0e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c13cefe870 0x55c13cf03370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58850/0 (socket says 172.21.15.67:58850) 2024-03-28T16:07:23.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.775+0000 7f9ac0e00640 1 -- 172.21.15.67:0/3855907858 learned_addr learned my addr 172.21.15.67:0/3855907858 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:07:23.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.775+0000 7f9abbe00640 1 --2- 172.21.15.67:0/3855907858 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c13cefde80 0x55c13cefe2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:23.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.775+0000 7f9ab9600640 1 --2- 172.21.15.67:0/487287763 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f9aa40a2880 0x7f9aa40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:23.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.775+0000 7f9abbe00640 1 --2- 172.21.15.67:0/3855907858 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c13cf039e0 0x55c13cf05e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:23.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.775+0000 7f9ab9600640 1 -- 172.21.15.67:0/487287763 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f9ab40cecc0 con 0x7f9abc105490 2024-03-28T16:07:23.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.776+0000 7f9ac0e00640 1 -- 172.21.15.67:0/3855907858 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c13cf039e0 msgr2=0x55c13cf05e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:23.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.776+0000 7f9ac0e00640 1 --2- 172.21.15.67:0/3855907858 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c13cf039e0 0x55c13cf05e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:23.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.776+0000 7f9ac0e00640 1 -- 172.21.15.67:0/3855907858 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c13cefde80 msgr2=0x55c13cefe2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:23.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.776+0000 7f9ac0e00640 1 --2- 172.21.15.67:0/3855907858 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c13cefde80 0x55c13cefe2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:23.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.776+0000 7f9ac0e00640 1 -- 172.21.15.67:0/3855907858 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c13cdf6690 con 0x55c13cefe870 2024-03-28T16:07:23.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.776+0000 7f9ac1800640 1 --2- 172.21.15.67:0/487287763 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f9aa40a2880 0x7f9aa40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:23.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.776+0000 7f9ac0e00640 1 --2- 172.21.15.67:0/3855907858 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c13cefe870 0x55c13cf03370 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f9ab000bc90 tx=0x7f9ab000f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3accd915d24dc1f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:23.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.777+0000 7f9aae000640 1 -- 172.21.15.67:0/3855907858 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab0015070 con 0x55c13cefe870 2024-03-28T16:07:23.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.777+0000 7f9aae000640 1 -- 172.21.15.67:0/3855907858 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9ab00035f0 con 0x55c13cefe870 2024-03-28T16:07:23.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.777+0000 7f9aae000640 1 -- 172.21.15.67:0/3855907858 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab00116a0 con 0x55c13cefe870 2024-03-28T16:07:23.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.778+0000 7f9ac1800640 1 --2- 172.21.15.67:0/487287763 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f9aa40a2880 0x7f9aa40a4d40 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f9ab4097a70 tx=0x7f9ab400cf50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:23.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.778+0000 7f9ac641c740 1 -- 172.21.15.67:0/3855907858 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c13cefe870 msgr2=0x55c13cf03370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:23.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.778+0000 7f9ac641c740 1 --2- 172.21.15.67:0/3855907858 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c13cefe870 0x55c13cf03370 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f9ab000bc90 tx=0x7f9ab000f730 comp rx=0 tx=0).stop 2024-03-28T16:07:23.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.778+0000 7f9ac641c740 1 -- 172.21.15.67:0/3855907858 shutdown_connections 2024-03-28T16:07:23.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.778+0000 7f9ac641c740 1 --2- 172.21.15.67:0/3855907858 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c13cf039e0 0x55c13cf05e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:23.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.778+0000 7f9ac641c740 1 --2- 172.21.15.67:0/3855907858 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c13cefe870 0x55c13cf03370 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:23.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.778+0000 7f9ac641c740 1 --2- 172.21.15.67:0/3855907858 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c13cefde80 0x55c13cefe2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:23.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.778+0000 7f9ac641c740 1 -- 172.21.15.67:0/3855907858 >> 172.21.15.67:0/3855907858 conn(0x55c13cec6200 msgr2=0x55c13cef97f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:23.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.778+0000 7f9ac641c740 1 -- 172.21.15.67:0/3855907858 shutdown_connections 2024-03-28T16:07:23.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.778+0000 7f9ac641c740 1 -- 172.21.15.67:0/3855907858 wait complete. 2024-03-28T16:07:23.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.779+0000 7f9ac641c740 1 -- start start 2024-03-28T16:07:23.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.779+0000 7f9ac641c740 1 Processor -- start 2024-03-28T16:07:23.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.780+0000 7f9ac641c740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c13cefde80 0x55c13cf07dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:23.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.780+0000 7f9ac641c740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c13cefe870 0x55c13cf08310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:23.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.780+0000 7f9ac641c740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c13cf039e0 0x55c13cf0f350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:23.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.780+0000 7f9ac641c740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c13cf06990 con 0x55c13cf039e0 2024-03-28T16:07:23.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.780+0000 7f9ac641c740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c13ce1ee80 con 0x55c13cefe870 2024-03-28T16:07:23.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.780+0000 7f9ac641c740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c13cf06b10 con 0x55c13cefde80 2024-03-28T16:07:23.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.780+0000 7f9ac641c740 10 client.0 _ll_register_callbacks cb 0x55c13ce31130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:07:23.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.780+0000 7f9abbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c13cf039e0 0x55c13cf0f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:23.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.780+0000 7f9abbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c13cf039e0 0x55c13cf0f350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58856/0 (socket says 172.21.15.67:58856) 2024-03-28T16:07:23.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.780+0000 7f9abbe00640 1 -- 172.21.15.67:0/2955000092 learned_addr learned my addr 172.21.15.67:0/2955000092 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:07:23.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.781+0000 7f9abbe00640 1 --2- 172.21.15.67:0/2955000092 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c13cefde80 0x55c13cf07dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:23.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.781+0000 7f9abbe00640 1 -- 172.21.15.67:0/2955000092 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c13cefde80 msgr2=0x55c13cf07dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:23.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.781+0000 7f9abbe00640 1 --2- 172.21.15.67:0/2955000092 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c13cefde80 0x55c13cf07dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:23.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.781+0000 7f9abbe00640 1 -- 172.21.15.67:0/2955000092 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c13cefe870 msgr2=0x55c13cf08310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:23.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.781+0000 7f9abbe00640 1 --2- 172.21.15.67:0/2955000092 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c13cefe870 0x55c13cf08310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:23.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.781+0000 7f9abbe00640 1 -- 172.21.15.67:0/2955000092 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c13cc417d0 con 0x55c13cf039e0 2024-03-28T16:07:23.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.781+0000 7f9abbe00640 1 --2- 172.21.15.67:0/2955000092 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c13cf039e0 0x55c13cf0f350 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f9aa8009230 tx=0x7f9aa802bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a060c98b631f14e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:23.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.781+0000 7f9aa2a00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:07:23.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.781+0000 7f9aa2a00640 1 -- 172.21.15.67:0/2955000092 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aa8049030 con 0x55c13cf039e0 2024-03-28T16:07:23.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.782+0000 7f9aa2a00640 1 -- 172.21.15.67:0/2955000092 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f9aa803a040 con 0x55c13cf039e0 2024-03-28T16:07:23.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.782+0000 7f9ac641c740 1 -- 172.21.15.67:0/2955000092 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9ab00035f0 con 0x55c13cf039e0 2024-03-28T16:07:23.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.782+0000 7f9aa2a00640 1 -- 172.21.15.67:0/2955000092 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9aa8005540 con 0x55c13cf039e0 2024-03-28T16:07:23.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.782+0000 7f9ac641c740 10 client.15867 fetch_fsmap learned FSMap version 19 2024-03-28T16:07:23.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.782+0000 7f9ac641c740 1 -- 172.21.15.67:0/2955000092 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55c13cb773f0 con 0x55c13cf039e0 2024-03-28T16:07:23.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.782+0000 7f9aa2a00640 1 -- 172.21.15.67:0/2955000092 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9ab00035f0 con 0x55c13cf039e0 2024-03-28T16:07:23.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.782+0000 7f9aa2a00640 1 -- 172.21.15.67:0/2955000092 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f9aa80037f0 con 0x55c13cf039e0 2024-03-28T16:07:23.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.783+0000 7f9ac641c740 10 client.15867 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:07:23.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.783+0000 7f9ac641c740 10 client.15867 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:07:23.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.783+0000 7f9ac641c740 20 client.15867 populate_metadata read hostname 'smithi067' 2024-03-28T16:07:23.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.783+0000 7f9ac641c740 1 --2- 172.21.15.67:0/2955000092 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55c13cf13ba0 0x55c13cf16060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:23.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.783+0000 7f9ac641c740 4 client.15867 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:07:23.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.783+0000 7f9ac641c740 1 -- 172.21.15.67:0/2955000092 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c13ce70790 con 0x55c13cf13ba0 2024-03-28T16:07:23.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.784+0000 7f9ac0e00640 1 --2- 172.21.15.67:0/2955000092 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55c13cf13ba0 0x55c13cf16060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:23.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.785+0000 7f9ac0e00640 1 --2- 172.21.15.67:0/2955000092 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55c13cf13ba0 0x55c13cf16060 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=53f058b740e0e83e server_cookie=7e4d7a0f27dd00c8 in_seq=0 out_seq=0 2024-03-28T16:07:23.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.785+0000 7f9aa2a00640 10 client.15867 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:07:23.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.787+0000 7f9aa2a00640 1 -- 172.21.15.67:0/2955000092 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c13ce70790 con 0x55c13cf13ba0 2024-03-28T16:07:23.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.787+0000 7f9aa2a00640 10 client.15867 handle_command_reply: tid=0 2024-03-28T16:07:23.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.787+0000 7f9ac641c740 1 client.15867 shutdown 2024-03-28T16:07:23.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.787+0000 7f9ac641c740 1 -- 172.21.15.67:0/2955000092 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c13cf039e0 msgr2=0x55c13cf0f350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:23.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.787+0000 7f9ac641c740 1 --2- 172.21.15.67:0/2955000092 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c13cf039e0 0x55c13cf0f350 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f9aa8009230 tx=0x7f9aa802bc30 comp rx=0 tx=0).stop 2024-03-28T16:07:23.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.788+0000 7f9ac641c740 1 -- 172.21.15.67:0/2955000092 shutdown_connections 2024-03-28T16:07:23.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.788+0000 7f9ac641c740 1 --2- 172.21.15.67:0/2955000092 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55c13cf13ba0 0x55c13cf16060 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:23.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.788+0000 7f9ac641c740 1 --2- 172.21.15.67:0/2955000092 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c13cf039e0 0x55c13cf0f350 secure :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f9aa8009230 tx=0x7f9aa802bc30 comp rx=0 tx=0).stop 2024-03-28T16:07:23.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.788+0000 7f9ac641c740 1 --2- 172.21.15.67:0/2955000092 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c13cefe870 0x55c13cf08310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:23.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.788+0000 7f9ac641c740 1 --2- 172.21.15.67:0/2955000092 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c13cefde80 0x55c13cf07dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:23.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.788+0000 7f9ac641c740 1 -- 172.21.15.67:0/2955000092 >> 172.21.15.67:0/2955000092 conn(0x55c13cec6200 msgr2=0x55c13cef8a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:23.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.788+0000 7f9aa2a00640 0 client.15867 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:07:23.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.788+0000 7f9ac641c740 1 -- 172.21.15.67:0/2955000092 shutdown_connections 2024-03-28T16:07:23.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.788+0000 7f9ac641c740 1 -- 172.21.15.67:0/2955000092 wait complete. 2024-03-28T16:07:23.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.789+0000 7f9ac641c740 20 client.15867 trim_cache size 0 max 16384 2024-03-28T16:07:23.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.814+0000 7f9ac641c740 1 Processor -- start 2024-03-28T16:07:23.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.814+0000 7f9ac641c740 1 -- start start 2024-03-28T16:07:23.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.815+0000 7f9ac641c740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c13cefde80 0x55c13cf15ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:23.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.815+0000 7f9ac641c740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c13cefe870 0x55c13cf0f310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:23.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.815+0000 7f9ac641c740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c13ceffa20 0x55c13cef8200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:23.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.815+0000 7f9ac641c740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9abc1a8c00 con 0x55c13ceffa20 2024-03-28T16:07:23.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.815+0000 7f9ac641c740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9abc1ab1c0 con 0x55c13cefde80 2024-03-28T16:07:23.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.815+0000 7f9ac641c740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9abc1a9200 con 0x55c13cefe870 2024-03-28T16:07:23.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.815+0000 7f9abbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c13ceffa20 0x55c13cef8200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:23.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.815+0000 7f9abbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c13ceffa20 0x55c13cef8200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58860/0 (socket says 172.21.15.67:58860) 2024-03-28T16:07:23.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.815+0000 7f9abbe00640 1 -- 172.21.15.67:0/2722865380 learned_addr learned my addr 172.21.15.67:0/2722865380 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:07:23.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.815+0000 7f9abbe00640 1 --2- 172.21.15.67:0/2722865380 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c13cefde80 0x55c13cf15ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:23.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.816+0000 7f9ac0e00640 1 --2- 172.21.15.67:0/2722865380 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c13cefe870 0x55c13cf0f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:23.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.816+0000 7f9abbe00640 1 -- 172.21.15.67:0/2722865380 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c13cefe870 msgr2=0x55c13cf0f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:23.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.816+0000 7f9abbe00640 1 --2- 172.21.15.67:0/2722865380 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c13cefe870 0x55c13cf0f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:23.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.816+0000 7f9abbe00640 1 -- 172.21.15.67:0/2722865380 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c13cefde80 msgr2=0x55c13cf15ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:23.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.816+0000 7f9abbe00640 1 --2- 172.21.15.67:0/2722865380 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c13cefde80 0x55c13cf15ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:23.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.816+0000 7f9abbe00640 1 -- 172.21.15.67:0/2722865380 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c13cb370f0 con 0x55c13ceffa20 2024-03-28T16:07:23.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.816+0000 7f9abbe00640 1 --2- 172.21.15.67:0/2722865380 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c13ceffa20 0x55c13cef8200 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f9aa8009680 tx=0x7f9aa8045040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37b90076c6b7c3db server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:23.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.816+0000 7f9aad600640 1 -- 172.21.15.67:0/2722865380 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aa8049040 con 0x55c13ceffa20 2024-03-28T16:07:23.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.817+0000 7f9aad600640 1 -- 172.21.15.67:0/2722865380 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9aa804a040 con 0x55c13ceffa20 2024-03-28T16:07:23.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.817+0000 7f9aad600640 1 -- 172.21.15.67:0/2722865380 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aa8004c90 con 0x55c13ceffa20 2024-03-28T16:07:23.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.817+0000 7f9ac641c740 1 -- 172.21.15.67:0/2722865380 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c13ceffa20 msgr2=0x55c13cef8200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:23.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.817+0000 7f9ac641c740 1 --2- 172.21.15.67:0/2722865380 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c13ceffa20 0x55c13cef8200 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f9aa8009680 tx=0x7f9aa8045040 comp rx=0 tx=0).stop 2024-03-28T16:07:23.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.818+0000 7f9ac641c740 1 -- 172.21.15.67:0/2722865380 shutdown_connections 2024-03-28T16:07:23.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.818+0000 7f9ac641c740 1 --2- 172.21.15.67:0/2722865380 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c13ceffa20 0x55c13cef8200 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:23.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.818+0000 7f9ac641c740 1 --2- 172.21.15.67:0/2722865380 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c13cefe870 0x55c13cf0f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:23.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.818+0000 7f9ac641c740 1 --2- 172.21.15.67:0/2722865380 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c13cefde80 0x55c13cf15ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:23.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.818+0000 7f9ac641c740 1 -- 172.21.15.67:0/2722865380 >> 172.21.15.67:0/2722865380 conn(0x55c13cec6200 msgr2=0x55c13ceff310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:23.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.818+0000 7f9ac641c740 1 -- 172.21.15.67:0/2722865380 shutdown_connections 2024-03-28T16:07:23.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.818+0000 7f9ac641c740 1 -- 172.21.15.67:0/2722865380 wait complete. 2024-03-28T16:07:23.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.818+0000 7f9ac641c740 1 -- start start 2024-03-28T16:07:23.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.819+0000 7f9ac641c740 1 Processor -- start 2024-03-28T16:07:23.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.819+0000 7f9ac641c740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c13cefde80 0x55c13cf0d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:23.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.819+0000 7f9ac641c740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c13cefe870 0x55c13cf0d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:23.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.819+0000 7f9ac641c740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c13ceffa20 0x55c13cf0e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:23.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.819+0000 7f9ac641c740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9abc1a9840 con 0x55c13cefe870 2024-03-28T16:07:23.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.819+0000 7f9ac641c740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9abc1aab60 con 0x55c13cefde80 2024-03-28T16:07:23.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.819+0000 7f9ac641c740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9abc1a9ea0 con 0x55c13ceffa20 2024-03-28T16:07:23.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.819+0000 7f9ac641c740 10 client.0 _ll_register_callbacks cb 0x7f9aa802cae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:07:23.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.820+0000 7f9ac0e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c13cefe870 0x55c13cf0d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:23.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.820+0000 7f9ac0e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c13cefe870 0x55c13cf0d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58872/0 (socket says 172.21.15.67:58872) 2024-03-28T16:07:23.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.820+0000 7f9ac0e00640 1 -- 172.21.15.67:0/986477383 learned_addr learned my addr 172.21.15.67:0/986477383 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:07:23.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.820+0000 7f9abbe00640 1 --2- 172.21.15.67:0/986477383 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c13cefde80 0x55c13cf0d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:23.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.820+0000 7f9ac0e00640 1 -- 172.21.15.67:0/986477383 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c13ceffa20 msgr2=0x55c13cf0e6b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T16:07:23.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.820+0000 7f9ac0e00640 1 --2- 172.21.15.67:0/986477383 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c13ceffa20 0x55c13cf0e6b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:23.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.820+0000 7f9ac0e00640 1 -- 172.21.15.67:0/986477383 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c13cefde80 msgr2=0x55c13cf0d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:23.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.820+0000 7f9ac0e00640 1 --2- 172.21.15.67:0/986477383 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c13cefde80 0x55c13cf0d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:23.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.820+0000 7f9ac0e00640 1 -- 172.21.15.67:0/986477383 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c13cf0eca0 con 0x55c13cefe870 2024-03-28T16:07:23.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.820+0000 7f9abbe00640 1 --2- 172.21.15.67:0/986477383 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c13cefde80 0x55c13cf0d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T16:07:23.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.821+0000 7f9ac0e00640 1 --2- 172.21.15.67:0/986477383 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c13cefe870 0x55c13cf0d6f0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f9ab000f770 tx=0x7f9ab000b450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b00d5a0c38f2ed3e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:23.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.821+0000 7f9aa2a00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:07:23.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.821+0000 7f9aa2a00640 1 -- 172.21.15.67:0/986477383 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab0015040 con 0x55c13cefe870 2024-03-28T16:07:23.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.821+0000 7f9aa2a00640 1 -- 172.21.15.67:0/986477383 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f9ab0003eb0 con 0x55c13cefe870 2024-03-28T16:07:23.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.821+0000 7f9ac641c740 1 -- 172.21.15.67:0/986477383 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9aa804a040 con 0x55c13cefe870 2024-03-28T16:07:23.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.822+0000 7f9aa2a00640 1 -- 172.21.15.67:0/986477383 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9ab000b500 con 0x55c13cefe870 2024-03-28T16:07:23.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.822+0000 7f9ac641c740 10 client.15873 fetch_fsmap learned FSMap version 19 2024-03-28T16:07:23.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.822+0000 7f9ac641c740 1 -- 172.21.15.67:0/986477383 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55c13cf6fc90 con 0x55c13cefe870 2024-03-28T16:07:23.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.822+0000 7f9aa2a00640 1 -- 172.21.15.67:0/986477383 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9aa804a040 con 0x55c13cefe870 2024-03-28T16:07:23.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.822+0000 7f9aa2a00640 1 -- 172.21.15.67:0/986477383 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f9ab00035f0 con 0x55c13cefe870 2024-03-28T16:07:23.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.822+0000 7f9ac641c740 10 client.15873 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:07:23.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.822+0000 7f9ac641c740 10 client.15873 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:07:23.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.822+0000 7f9ac641c740 20 client.15873 populate_metadata read hostname 'smithi067' 2024-03-28T16:07:23.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.822+0000 7f9ac641c740 1 --2- 172.21.15.67:0/986477383 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55c13cf703e0 0x55c13cf72800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:23.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.822+0000 7f9ac641c740 4 client.15873 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T16:07:23.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.822+0000 7f9ac641c740 1 -- 172.21.15.67:0/986477383 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55c13ce6aae0 con 0x55c13cf703e0 2024-03-28T16:07:23.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.823+0000 7f9abbe00640 1 --2- 172.21.15.67:0/986477383 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55c13cf703e0 0x55c13cf72800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:23.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.824+0000 7f9abbe00640 1 --2- 172.21.15.67:0/986477383 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55c13cf703e0 0x55c13cf72800 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=804bbcfbd79c25b server_cookie=fffdea1b6e35972b in_seq=0 out_seq=0 2024-03-28T16:07:23.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.824+0000 7f9aa2a00640 10 client.15873 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:07:23.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.825+0000 7f9aa2a00640 1 -- 172.21.15.67:0/986477383 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55c13ce6aae0 con 0x55c13cf703e0 2024-03-28T16:07:23.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.825+0000 7f9aa2a00640 10 client.15873 handle_command_reply: tid=0 2024-03-28T16:07:23.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.825+0000 7f9ac641c740 1 client.15873 shutdown 2024-03-28T16:07:23.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.825+0000 7f9ac641c740 1 -- 172.21.15.67:0/986477383 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c13cefe870 msgr2=0x55c13cf0d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:23.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.825+0000 7f9ac641c740 1 --2- 172.21.15.67:0/986477383 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c13cefe870 0x55c13cf0d6f0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f9ab000f770 tx=0x7f9ab000b450 comp rx=0 tx=0).stop 2024-03-28T16:07:23.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.825+0000 7f9ac641c740 1 -- 172.21.15.67:0/986477383 shutdown_connections 2024-03-28T16:07:23.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.825+0000 7f9ac641c740 1 --2- 172.21.15.67:0/986477383 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55c13cf703e0 0x55c13cf72800 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:23.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.825+0000 7f9ac641c740 1 --2- 172.21.15.67:0/986477383 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c13ceffa20 0x55c13cf0e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:23.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.825+0000 7f9ac641c740 1 --2- 172.21.15.67:0/986477383 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c13cefe870 0x55c13cf0d6f0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:23.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.825+0000 7f9ac641c740 1 --2- 172.21.15.67:0/986477383 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c13cefde80 0x55c13cf0d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:23.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.826+0000 7f9aa2a00640 0 client.15873 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:07:23.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.825+0000 7f9ac641c740 1 -- 172.21.15.67:0/986477383 >> 172.21.15.67:0/986477383 conn(0x55c13cec6200 msgr2=0x55c13cef8200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:23.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.826+0000 7f9ac641c740 1 -- 172.21.15.67:0/986477383 shutdown_connections 2024-03-28T16:07:23.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.826+0000 7f9ac641c740 1 -- 172.21.15.67:0/986477383 wait complete. 2024-03-28T16:07:23.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.826+0000 7f9ac641c740 20 client.15873 trim_cache size 0 max 16384 2024-03-28T16:07:23.831 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-28T16:07:23.831 INFO:teuthology.orchestra.run.smithi067.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T16:07:23.831 INFO:teuthology.orchestra.run.smithi067.stdout: "scrubs": { 2024-03-28T16:07:23.831 INFO:teuthology.orchestra.run.smithi067.stdout: "4e5f268b-43bd-49e1-a118-21dfec109591": { 2024-03-28T16:07:23.831 INFO:teuthology.orchestra.run.smithi067.stdout: "path": "/", 2024-03-28T16:07:23.831 INFO:teuthology.orchestra.run.smithi067.stdout: "tag": "4e5f268b-43bd-49e1-a118-21dfec109591", 2024-03-28T16:07:23.831 INFO:teuthology.orchestra.run.smithi067.stdout: "options": "recursive,force" 2024-03-28T16:07:23.831 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:07:23.831 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:07:23.831 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-28T16:07:23.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.827+0000 7f9ac3800640 1 -- 172.21.15.67:0/487287763 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f9aa40a2880 msgr2=0x7f9aa40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:23.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.827+0000 7f9ac3800640 1 --2- 172.21.15.67:0/487287763 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f9aa40a2880 0x7f9aa40a4d40 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f9ab4097a70 tx=0x7f9ab400cf50 comp rx=0 tx=0).stop 2024-03-28T16:07:23.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.827+0000 7f9ac3800640 1 -- 172.21.15.67:0/487287763 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9abc105490 msgr2=0x7f9abc1a10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:23.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.827+0000 7f9ac3800640 1 --2- 172.21.15.67:0/487287763 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9abc105490 0x7f9abc1a10a0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f9ab400eaa0 tx=0x7f9ab400ef60 comp rx=0 tx=0).stop 2024-03-28T16:07:23.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.827+0000 7f9ac3800640 1 -- 172.21.15.67:0/487287763 shutdown_connections 2024-03-28T16:07:23.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.827+0000 7f9ac3800640 1 --2- 172.21.15.67:0/487287763 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f9aa40a2880 0x7f9aa40a4d40 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:23.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.827+0000 7f9ac3800640 1 --2- 172.21.15.67:0/487287763 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9abc105490 0x7f9abc1a10a0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:23.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.827+0000 7f9ac3800640 1 --2- 172.21.15.67:0/487287763 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9abc104ad0 0x7f9abc19a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:23.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.827+0000 7f9ac3800640 1 --2- 172.21.15.67:0/487287763 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9abc1038e0 0x7f9abc199af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:23.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.827+0000 7f9ac3800640 1 -- 172.21.15.67:0/487287763 >> 172.21.15.67:0/487287763 conn(0x7f9abc0fee50 msgr2=0x7f9abc107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:23.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.827+0000 7f9ac3800640 1 -- 172.21.15.67:0/487287763 shutdown_connections 2024-03-28T16:07:23.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:23.828+0000 7f9ac3800640 1 -- 172.21.15.67:0/487287763 wait complete. 2024-03-28T16:07:23.849 INFO:tasks.cephfs.filesystem:scrub status for tag:4e5f268b-43bd-49e1-a118-21dfec109591 - {'path': '/', 'tag': '4e5f268b-43bd-49e1-a118-21dfec109591', 'options': 'recursive,force'} 2024-03-28T16:07:24.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:07:23 smithi150 ceph-mon[34825]: scrub queued for path: / 2024-03-28T16:07:24.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:07:23 smithi150 ceph-mon[34825]: scrub summary: idle+waiting paths [/] 2024-03-28T16:07:24.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:07:23 smithi150 ceph-mon[34825]: scrub summary: active paths [/] 2024-03-28T16:07:24.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:07:23 smithi150 ceph-mon[34825]: scrub summary: idle+waiting paths [/] 2024-03-28T16:07:24.321 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:07:23 smithi067 ceph-mon[31707]: scrub queued for path: / 2024-03-28T16:07:24.321 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:07:23 smithi067 ceph-mon[31707]: scrub summary: idle+waiting paths [/] 2024-03-28T16:07:24.321 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:07:23 smithi067 ceph-mon[31707]: scrub summary: active paths [/] 2024-03-28T16:07:24.321 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:07:23 smithi067 ceph-mon[31707]: scrub summary: idle+waiting paths [/] 2024-03-28T16:07:24.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:07:23 smithi183 ceph-mon[35580]: scrub queued for path: / 2024-03-28T16:07:24.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:07:23 smithi183 ceph-mon[35580]: scrub summary: idle+waiting paths [/] 2024-03-28T16:07:24.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:07:23 smithi183 ceph-mon[35580]: scrub summary: active paths [/] 2024-03-28T16:07:24.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:07:23 smithi183 ceph-mon[35580]: scrub summary: idle+waiting paths [/] 2024-03-28T16:07:25.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:07:24 smithi150 ceph-mon[34825]: pgmap v554: 97 pgs: 97 active+clean; 773 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 62 MiB/s wr, 39 op/s 2024-03-28T16:07:25.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:07:24 smithi067 ceph-mon[31707]: pgmap v554: 97 pgs: 97 active+clean; 773 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 62 MiB/s wr, 39 op/s 2024-03-28T16:07:25.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:07:24 smithi183 ceph-mon[35580]: pgmap v554: 97 pgs: 97 active+clean; 773 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 62 MiB/s wr, 39 op/s 2024-03-28T16:07:27.153 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:07:26 smithi183 ceph-mon[35580]: pgmap v555: 97 pgs: 97 active+clean; 1.2 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 105 MiB/s wr, 52 op/s 2024-03-28T16:07:27.153 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:07:26 smithi183 ceph-mon[35580]: scrub summary: idle 2024-03-28T16:07:27.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:07:26 smithi150 ceph-mon[34825]: pgmap v555: 97 pgs: 97 active+clean; 1.2 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 105 MiB/s wr, 52 op/s 2024-03-28T16:07:27.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:07:26 smithi150 ceph-mon[34825]: scrub summary: idle 2024-03-28T16:07:27.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:07:26 smithi067 ceph-mon[31707]: pgmap v555: 97 pgs: 97 active+clean; 1.2 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 105 MiB/s wr, 52 op/s 2024-03-28T16:07:27.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:07:26 smithi067 ceph-mon[31707]: scrub summary: idle 2024-03-28T16:07:29.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:07:28 smithi150 ceph-mon[34825]: pgmap v556: 97 pgs: 97 active+clean; 1.4 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 120 MiB/s wr, 55 op/s 2024-03-28T16:07:29.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:07:28 smithi067 ceph-mon[31707]: pgmap v556: 97 pgs: 97 active+clean; 1.4 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 120 MiB/s wr, 55 op/s 2024-03-28T16:07:29.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:07:28 smithi183 ceph-mon[35580]: pgmap v556: 97 pgs: 97 active+clean; 1.4 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 120 MiB/s wr, 55 op/s 2024-03-28T16:07:31.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:07:30 smithi150 ceph-mon[34825]: pgmap v557: 97 pgs: 97 active+clean; 1.7 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 142 MiB/s wr, 61 op/s 2024-03-28T16:07:31.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:07:30 smithi067 ceph-mon[31707]: pgmap v557: 97 pgs: 97 active+clean; 1.7 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 142 MiB/s wr, 61 op/s 2024-03-28T16:07:31.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:07:30 smithi183 ceph-mon[35580]: pgmap v557: 97 pgs: 97 active+clean; 1.7 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 142 MiB/s wr, 61 op/s 2024-03-28T16:07:33.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:07:32 smithi150 ceph-mon[34825]: pgmap v558: 97 pgs: 97 active+clean; 2.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 173 MiB/s wr, 69 op/s 2024-03-28T16:07:33.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:07:32 smithi067 ceph-mon[31707]: pgmap v558: 97 pgs: 97 active+clean; 2.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 173 MiB/s wr, 69 op/s 2024-03-28T16:07:33.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:07:32 smithi183 ceph-mon[35580]: pgmap v558: 97 pgs: 97 active+clean; 2.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 173 MiB/s wr, 69 op/s 2024-03-28T16:07:35.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:07:34 smithi150 ceph-mon[34825]: pgmap v559: 97 pgs: 97 active+clean; 2.4 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 174 MiB/s wr, 55 op/s 2024-03-28T16:07:35.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:07:34 smithi067 ceph-mon[31707]: pgmap v559: 97 pgs: 97 active+clean; 2.4 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 174 MiB/s wr, 55 op/s 2024-03-28T16:07:35.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:07:34 smithi183 ceph-mon[35580]: pgmap v559: 97 pgs: 97 active+clean; 2.4 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 174 MiB/s wr, 55 op/s 2024-03-28T16:07:37.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:07:36 smithi150 ceph-mon[34825]: pgmap v560: 97 pgs: 97 active+clean; 2.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 164 MiB/s wr, 45 op/s 2024-03-28T16:07:37.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:07:36 smithi067 ceph-mon[31707]: pgmap v560: 97 pgs: 97 active+clean; 2.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 164 MiB/s wr, 45 op/s 2024-03-28T16:07:37.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:07:36 smithi183 ceph-mon[35580]: pgmap v560: 97 pgs: 97 active+clean; 2.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 164 MiB/s wr, 45 op/s 2024-03-28T16:07:39.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:07:38 smithi150 ceph-mon[34825]: pgmap v561: 97 pgs: 97 active+clean; 2.8 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 135 MiB/s wr, 35 op/s 2024-03-28T16:07:39.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:07:38 smithi067 ceph-mon[31707]: pgmap v561: 97 pgs: 97 active+clean; 2.8 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 135 MiB/s wr, 35 op/s 2024-03-28T16:07:39.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:07:38 smithi183 ceph-mon[35580]: pgmap v561: 97 pgs: 97 active+clean; 2.8 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 135 MiB/s wr, 35 op/s 2024-03-28T16:07:41.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:07:40 smithi150 ceph-mon[34825]: pgmap v562: 97 pgs: 97 active+clean; 3.1 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 141 MiB/s wr, 37 op/s 2024-03-28T16:07:41.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:07:40 smithi067 ceph-mon[31707]: pgmap v562: 97 pgs: 97 active+clean; 3.1 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 141 MiB/s wr, 37 op/s 2024-03-28T16:07:41.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:07:40 smithi183 ceph-mon[35580]: pgmap v562: 97 pgs: 97 active+clean; 3.1 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 141 MiB/s wr, 37 op/s 2024-03-28T16:07:43.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:07:42 smithi150 ceph-mon[34825]: pgmap v563: 97 pgs: 97 active+clean; 3.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 148 MiB/s wr, 39 op/s 2024-03-28T16:07:43.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:07:42 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T16:07:43.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:07:42 smithi067 ceph-mon[31707]: pgmap v563: 97 pgs: 97 active+clean; 3.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 148 MiB/s wr, 39 op/s 2024-03-28T16:07:43.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:07:42 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T16:07:43.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:07:42 smithi183 ceph-mon[35580]: pgmap v563: 97 pgs: 97 active+clean; 3.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 148 MiB/s wr, 39 op/s 2024-03-28T16:07:43.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:07:42 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T16:07:44.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:07:43 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:07:44.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:07:43 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T16:07:44.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:07:43 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:07:44.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:07:43 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:07:44.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:07:43 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T16:07:44.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:07:43 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:07:44.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:07:43 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:07:44.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:07:43 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T16:07:44.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:07:43 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:07:45.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:07:44 smithi150 ceph-mon[34825]: pgmap v564: 97 pgs: 97 active+clean; 3.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 140 MiB/s wr, 37 op/s 2024-03-28T16:07:45.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:07:44 smithi067 ceph-mon[31707]: pgmap v564: 97 pgs: 97 active+clean; 3.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 140 MiB/s wr, 37 op/s 2024-03-28T16:07:45.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:07:44 smithi183 ceph-mon[35580]: pgmap v564: 97 pgs: 97 active+clean; 3.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 140 MiB/s wr, 37 op/s 2024-03-28T16:07:47.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:07:46 smithi150 ceph-mon[34825]: pgmap v565: 97 pgs: 97 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 154 MiB/s wr, 41 op/s 2024-03-28T16:07:47.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:07:46 smithi067 ceph-mon[31707]: pgmap v565: 97 pgs: 97 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 154 MiB/s wr, 41 op/s 2024-03-28T16:07:47.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:07:46 smithi183 ceph-mon[35580]: pgmap v565: 97 pgs: 97 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 154 MiB/s wr, 41 op/s 2024-03-28T16:07:49.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:07:48 smithi150 ceph-mon[34825]: pgmap v566: 97 pgs: 97 active+clean; 4.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 147 MiB/s wr, 38 op/s 2024-03-28T16:07:49.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:07:48 smithi067 ceph-mon[31707]: pgmap v566: 97 pgs: 97 active+clean; 4.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 147 MiB/s wr, 38 op/s 2024-03-28T16:07:49.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:07:48 smithi183 ceph-mon[35580]: pgmap v566: 97 pgs: 97 active+clean; 4.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 147 MiB/s wr, 38 op/s 2024-03-28T16:07:51.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:07:51 smithi150 ceph-mon[34825]: pgmap v567: 97 pgs: 97 active+clean; 4.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 158 MiB/s wr, 41 op/s 2024-03-28T16:07:51.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:07:51 smithi067 ceph-mon[31707]: pgmap v567: 97 pgs: 97 active+clean; 4.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 158 MiB/s wr, 41 op/s 2024-03-28T16:07:51.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:07:51 smithi183 ceph-mon[35580]: pgmap v567: 97 pgs: 97 active+clean; 4.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 158 MiB/s wr, 41 op/s 2024-03-28T16:07:53.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:07:53 smithi150 ceph-mon[34825]: pgmap v568: 97 pgs: 97 active+clean; 5.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 165 MiB/s wr, 44 op/s 2024-03-28T16:07:53.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:07:53 smithi067 ceph-mon[31707]: pgmap v568: 97 pgs: 97 active+clean; 5.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 165 MiB/s wr, 44 op/s 2024-03-28T16:07:53.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:07:53 smithi183 ceph-mon[35580]: pgmap v568: 97 pgs: 97 active+clean; 5.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 165 MiB/s wr, 44 op/s 2024-03-28T16:07:53.850 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T16:07:53.850 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T16:07:54.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.132+0000 7fca43400640 1 -- 172.21.15.67:0/3778535606 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca3c109870 msgr2=0x7fca3c110100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:54.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.132+0000 7fca43400640 1 --2- 172.21.15.67:0/3778535606 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca3c109870 0x7fca3c110100 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fca34009ca0 tx=0x7fca3402c120 comp rx=0 tx=0).stop 2024-03-28T16:07:54.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.132+0000 7fca43400640 1 -- 172.21.15.67:0/3778535606 shutdown_connections 2024-03-28T16:07:54.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.132+0000 7fca43400640 1 --2- 172.21.15.67:0/3778535606 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca3c109870 0x7fca3c110100 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:54.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.132+0000 7fca43400640 1 --2- 172.21.15.67:0/3778535606 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fca3c108eb0 0x7fca3c109330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:54.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.132+0000 7fca43400640 1 --2- 172.21.15.67:0/3778535606 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca3c107cc0 0x7fca3c1080e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:54.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.132+0000 7fca43400640 1 -- 172.21.15.67:0/3778535606 >> 172.21.15.67:0/3778535606 conn(0x7fca3c076c60 msgr2=0x7fca3c0790a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:54.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.132+0000 7fca43400640 1 -- 172.21.15.67:0/3778535606 shutdown_connections 2024-03-28T16:07:54.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.132+0000 7fca43400640 1 -- 172.21.15.67:0/3778535606 wait complete. 2024-03-28T16:07:54.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.134+0000 7fca43400640 1 Processor -- start 2024-03-28T16:07:54.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.134+0000 7fca43400640 1 -- start start 2024-03-28T16:07:54.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.134+0000 7fca43400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca3c107cc0 0x7fca3c19df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:54.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.134+0000 7fca43400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca3c108eb0 0x7fca3c19e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:54.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.135+0000 7fca43400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fca3c109870 0x7fca3c1a54c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:54.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.135+0000 7fca43400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca3c113870 con 0x7fca3c108eb0 2024-03-28T16:07:54.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.135+0000 7fca43400640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca3c1136f0 con 0x7fca3c107cc0 2024-03-28T16:07:54.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.135+0000 7fca43400640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca3c1139f0 con 0x7fca3c109870 2024-03-28T16:07:54.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.135+0000 7fca3be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca3c108eb0 0x7fca3c19e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:54.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.135+0000 7fca3be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca3c108eb0 0x7fca3c19e450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59154/0 (socket says 172.21.15.67:59154) 2024-03-28T16:07:54.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.135+0000 7fca3be00640 1 -- 172.21.15.67:0/2044817935 learned_addr learned my addr 172.21.15.67:0/2044817935 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:07:54.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.135+0000 7fca41400640 1 --2- 172.21.15.67:0/2044817935 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fca3c109870 0x7fca3c1a54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:54.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.136+0000 7fca40a00640 1 --2- 172.21.15.67:0/2044817935 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca3c107cc0 0x7fca3c19df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:54.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.136+0000 7fca3be00640 1 -- 172.21.15.67:0/2044817935 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fca3c109870 msgr2=0x7fca3c1a54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:54.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.136+0000 7fca3be00640 1 --2- 172.21.15.67:0/2044817935 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fca3c109870 0x7fca3c1a54c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:54.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.136+0000 7fca3be00640 1 -- 172.21.15.67:0/2044817935 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca3c107cc0 msgr2=0x7fca3c19df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:54.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.136+0000 7fca3be00640 1 --2- 172.21.15.67:0/2044817935 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca3c107cc0 0x7fca3c19df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:54.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.136+0000 7fca3be00640 1 -- 172.21.15.67:0/2044817935 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca3c1a5b30 con 0x7fca3c108eb0 2024-03-28T16:07:54.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.136+0000 7fca3be00640 1 --2- 172.21.15.67:0/2044817935 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca3c108eb0 0x7fca3c19e450 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fca2800b450 tx=0x7fca2800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:54.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.137+0000 7fca39600640 1 -- 172.21.15.67:0/2044817935 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca28015020 con 0x7fca3c108eb0 2024-03-28T16:07:54.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.137+0000 7fca39600640 1 -- 172.21.15.67:0/2044817935 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fca28011d40 con 0x7fca3c108eb0 2024-03-28T16:07:54.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.137+0000 7fca39600640 1 -- 172.21.15.67:0/2044817935 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca2800f340 con 0x7fca3c108eb0 2024-03-28T16:07:54.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.137+0000 7fca43400640 1 -- 172.21.15.67:0/2044817935 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca3c1a5e20 con 0x7fca3c108eb0 2024-03-28T16:07:54.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.137+0000 7fca43400640 1 -- 172.21.15.67:0/2044817935 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca3c1a6230 con 0x7fca3c108eb0 2024-03-28T16:07:54.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.144+0000 7fca39600640 1 -- 172.21.15.67:0/2044817935 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fca28018020 con 0x7fca3c108eb0 2024-03-28T16:07:54.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.147+0000 7fca39600640 1 --2- 172.21.15.67:0/2044817935 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fca240a2880 0x7fca240a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:54.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.147+0000 7fca39600640 1 -- 172.21.15.67:0/2044817935 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fca280ce4a0 con 0x7fca3c108eb0 2024-03-28T16:07:54.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.147+0000 7fca40a00640 1 --2- 172.21.15.67:0/2044817935 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fca240a2880 0x7fca240a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:54.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.148+0000 7fca46068740 1 Processor -- start 2024-03-28T16:07:54.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.148+0000 7fca46068740 1 -- start start 2024-03-28T16:07:54.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.148+0000 7fca46068740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56352dfa7e80 0x56352dfa82a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:54.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.148+0000 7fca46068740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56352dfa8870 0x56352dfad370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:54.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.148+0000 7fca46068740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56352dfad9e0 0x56352dfafe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:54.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.148+0000 7fca46068740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x56352df0d250 con 0x56352dfad9e0 2024-03-28T16:07:54.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.148+0000 7fca46068740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x56352deb5a30 con 0x56352dfa8870 2024-03-28T16:07:54.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.148+0000 7fca46068740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x56352df092f0 con 0x56352dfa7e80 2024-03-28T16:07:54.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.148+0000 7fca40a00640 1 --2- 172.21.15.67:0/2044817935 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fca240a2880 0x7fca240a4d40 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fca30005fd0 tx=0x7fca30005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:54.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.149+0000 7fca3be00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56352dfa8870 0x56352dfad370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:54.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.149+0000 7fca3be00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56352dfa8870 0x56352dfad370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.67:36694/0 (socket says 172.21.15.67:36694) 2024-03-28T16:07:54.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.149+0000 7fca3be00640 1 -- 172.21.15.67:0/1824408960 learned_addr learned my addr 172.21.15.67:0/1824408960 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:07:54.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.150+0000 7fca40a00640 1 --2- 172.21.15.67:0/1824408960 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56352dfa7e80 0x56352dfa82a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:54.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.150+0000 7fca3be00640 1 -- 172.21.15.67:0/1824408960 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56352dfa7e80 msgr2=0x56352dfa82a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:54.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.150+0000 7fca3be00640 1 --2- 172.21.15.67:0/1824408960 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56352dfa7e80 0x56352dfa82a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:54.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.150+0000 7fca3be00640 1 -- 172.21.15.67:0/1824408960 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56352dfad9e0 msgr2=0x56352dfafe70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T16:07:54.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.150+0000 7fca3be00640 1 --2- 172.21.15.67:0/1824408960 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56352dfad9e0 0x56352dfafe70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:54.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.150+0000 7fca3be00640 1 -- 172.21.15.67:0/1824408960 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56352dea0690 con 0x56352dfa8870 2024-03-28T16:07:54.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.151+0000 7fca3be00640 1 --2- 172.21.15.67:0/1824408960 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56352dfa8870 0x56352dfad370 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fca280972b0 tx=0x7fca2809aa90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8aa24cff207debc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:54.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.151+0000 7fca2dc00640 1 -- 172.21.15.67:0/1824408960 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca2809ae50 con 0x56352dfa8870 2024-03-28T16:07:54.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.151+0000 7fca2dc00640 1 -- 172.21.15.67:0/1824408960 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fca2809d070 con 0x56352dfa8870 2024-03-28T16:07:54.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.151+0000 7fca2dc00640 1 -- 172.21.15.67:0/1824408960 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca280ce110 con 0x56352dfa8870 2024-03-28T16:07:54.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.152+0000 7fca46068740 1 -- 172.21.15.67:0/1824408960 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56352dfa8870 msgr2=0x56352dfad370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:54.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.152+0000 7fca46068740 1 --2- 172.21.15.67:0/1824408960 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56352dfa8870 0x56352dfad370 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fca280972b0 tx=0x7fca2809aa90 comp rx=0 tx=0).stop 2024-03-28T16:07:54.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.152+0000 7fca46068740 1 -- 172.21.15.67:0/1824408960 shutdown_connections 2024-03-28T16:07:54.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.152+0000 7fca46068740 1 --2- 172.21.15.67:0/1824408960 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56352dfad9e0 0x56352dfafe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:54.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.152+0000 7fca46068740 1 --2- 172.21.15.67:0/1824408960 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56352dfa8870 0x56352dfad370 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:54.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.152+0000 7fca46068740 1 --2- 172.21.15.67:0/1824408960 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56352dfa7e80 0x56352dfa82a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:54.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.152+0000 7fca46068740 1 -- 172.21.15.67:0/1824408960 >> 172.21.15.67:0/1824408960 conn(0x56352df70200 msgr2=0x56352dfa37f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:54.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.152+0000 7fca46068740 1 -- 172.21.15.67:0/1824408960 shutdown_connections 2024-03-28T16:07:54.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.152+0000 7fca46068740 1 -- 172.21.15.67:0/1824408960 wait complete. 2024-03-28T16:07:54.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.153+0000 7fca46068740 1 -- start start 2024-03-28T16:07:54.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.153+0000 7fca46068740 1 Processor -- start 2024-03-28T16:07:54.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.154+0000 7fca46068740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56352dfa7e80 0x56352dfc2e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:54.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.154+0000 7fca46068740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56352dfa8870 0x56352dfc3380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:54.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.154+0000 7fca46068740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56352dfad9e0 0x56352dfca3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:54.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.154+0000 7fca46068740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x56352dfc19a0 con 0x56352dfa7e80 2024-03-28T16:07:54.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.154+0000 7fca46068740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x56352dec8e80 con 0x56352dfa8870 2024-03-28T16:07:54.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.154+0000 7fca46068740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x56352dfc1b20 con 0x56352dfad9e0 2024-03-28T16:07:54.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.154+0000 7fca46068740 10 client.0 _ll_register_callbacks cb 0x56352dedb130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:07:54.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.154+0000 7fca40a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56352dfa7e80 0x56352dfc2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:54.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.154+0000 7fca3be00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56352dfa8870 0x56352dfc3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:54.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.154+0000 7fca40a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56352dfa7e80 0x56352dfc2e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59166/0 (socket says 172.21.15.67:59166) 2024-03-28T16:07:54.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.154+0000 7fca40a00640 1 -- 172.21.15.67:0/1652466164 learned_addr learned my addr 172.21.15.67:0/1652466164 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:07:54.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.155+0000 7fca41400640 1 --2- 172.21.15.67:0/1652466164 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56352dfad9e0 0x56352dfca3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:54.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.155+0000 7fca40a00640 1 -- 172.21.15.67:0/1652466164 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56352dfad9e0 msgr2=0x56352dfca3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:54.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.155+0000 7fca40a00640 1 --2- 172.21.15.67:0/1652466164 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56352dfad9e0 0x56352dfca3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:54.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.155+0000 7fca40a00640 1 -- 172.21.15.67:0/1652466164 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56352dfa8870 msgr2=0x56352dfc3380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:54.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.155+0000 7fca40a00640 1 --2- 172.21.15.67:0/1652466164 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56352dfa8870 0x56352dfc3380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:54.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.155+0000 7fca40a00640 1 -- 172.21.15.67:0/1652466164 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56352dceb7d0 con 0x56352dfa7e80 2024-03-28T16:07:54.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.155+0000 7fca3be00640 1 --2- 172.21.15.67:0/1652466164 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56352dfa8870 0x56352dfc3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T16:07:54.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.155+0000 7fca40a00640 1 --2- 172.21.15.67:0/1652466164 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56352dfa7e80 0x56352dfc2e40 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fca30011440 tx=0x7fca300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8e6279a38ce4d03 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:54.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.155+0000 7fca22000640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:07:54.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.156+0000 7fca22000640 1 -- 172.21.15.67:0/1652466164 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca3001f840 con 0x56352dfa7e80 2024-03-28T16:07:54.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.156+0000 7fca22000640 1 -- 172.21.15.67:0/1652466164 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fca3001f9e0 con 0x56352dfa7e80 2024-03-28T16:07:54.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.156+0000 7fca46068740 1 -- 172.21.15.67:0/1652466164 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fca2809d070 con 0x56352dfa7e80 2024-03-28T16:07:54.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.156+0000 7fca22000640 1 -- 172.21.15.67:0/1652466164 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fca3001fbe0 con 0x56352dfa7e80 2024-03-28T16:07:54.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.156+0000 7fca46068740 10 client.15882 fetch_fsmap learned FSMap version 19 2024-03-28T16:07:54.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.156+0000 7fca46068740 1 -- 172.21.15.67:0/1652466164 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x56352dc213f0 con 0x56352dfa7e80 2024-03-28T16:07:54.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.157+0000 7fca22000640 1 -- 172.21.15.67:0/1652466164 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fca2809d070 con 0x56352dfa7e80 2024-03-28T16:07:54.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.157+0000 7fca22000640 1 -- 172.21.15.67:0/1652466164 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fca300103d0 con 0x56352dfa7e80 2024-03-28T16:07:54.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.157+0000 7fca46068740 10 client.15882 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:07:54.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.157+0000 7fca46068740 10 client.15882 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:07:54.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.157+0000 7fca46068740 20 client.15882 populate_metadata read hostname 'smithi067' 2024-03-28T16:07:54.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.157+0000 7fca46068740 1 --2- 172.21.15.67:0/1652466164 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x56352dfcecc0 0x56352dfd1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:54.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.157+0000 7fca46068740 4 client.15882 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:07:54.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.157+0000 7fca46068740 1 -- 172.21.15.67:0/1652466164 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56352df1a790 con 0x56352dfcecc0 2024-03-28T16:07:54.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.158+0000 7fca3be00640 1 --2- 172.21.15.67:0/1652466164 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x56352dfcecc0 0x56352dfd1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:54.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.161+0000 7fca3be00640 1 --2- 172.21.15.67:0/1652466164 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x56352dfcecc0 0x56352dfd1100 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9db06945dadfad87 server_cookie=d68e6840648690b7 in_seq=0 out_seq=0 2024-03-28T16:07:54.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.161+0000 7fca22000640 10 client.15882 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:07:54.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.162+0000 7fca22000640 1 -- 172.21.15.67:0/1652466164 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56352df1a790 con 0x56352dfcecc0 2024-03-28T16:07:54.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.162+0000 7fca22000640 10 client.15882 handle_command_reply: tid=0 2024-03-28T16:07:54.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.162+0000 7fca46068740 1 client.15882 shutdown 2024-03-28T16:07:54.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.163+0000 7fca46068740 1 -- 172.21.15.67:0/1652466164 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56352dfa7e80 msgr2=0x56352dfc2e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:54.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.163+0000 7fca46068740 1 --2- 172.21.15.67:0/1652466164 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56352dfa7e80 0x56352dfc2e40 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fca30011440 tx=0x7fca300046c0 comp rx=0 tx=0).stop 2024-03-28T16:07:54.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.163+0000 7fca46068740 1 -- 172.21.15.67:0/1652466164 shutdown_connections 2024-03-28T16:07:54.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.163+0000 7fca46068740 1 --2- 172.21.15.67:0/1652466164 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x56352dfcecc0 0x56352dfd1100 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:54.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.163+0000 7fca46068740 1 --2- 172.21.15.67:0/1652466164 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56352dfad9e0 0x56352dfca3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:54.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.163+0000 7fca46068740 1 --2- 172.21.15.67:0/1652466164 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56352dfa8870 0x56352dfc3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:54.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.163+0000 7fca46068740 1 --2- 172.21.15.67:0/1652466164 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56352dfa7e80 0x56352dfc2e40 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:54.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.163+0000 7fca22000640 0 client.15882 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:07:54.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.163+0000 7fca46068740 1 -- 172.21.15.67:0/1652466164 >> 172.21.15.67:0/1652466164 conn(0x56352df70200 msgr2=0x56352dfa2a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:54.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.163+0000 7fca46068740 1 -- 172.21.15.67:0/1652466164 shutdown_connections 2024-03-28T16:07:54.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.163+0000 7fca46068740 1 -- 172.21.15.67:0/1652466164 wait complete. 2024-03-28T16:07:54.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.163+0000 7fca46068740 20 client.15882 trim_cache size 0 max 16384 2024-03-28T16:07:54.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.184+0000 7fca46068740 1 Processor -- start 2024-03-28T16:07:54.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.184+0000 7fca46068740 1 -- start start 2024-03-28T16:07:54.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.184+0000 7fca46068740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56352dfa7e80 0x56352dfa3a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:54.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.185+0000 7fca46068740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56352dfa8870 0x56352dfca380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:54.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.185+0000 7fca46068740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56352dfad9e0 0x56352dfc2a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:54.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.185+0000 7fca46068740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca3c080ac0 con 0x56352dfa8870 2024-03-28T16:07:54.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.185+0000 7fca46068740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca3c1a6e40 con 0x56352dfa7e80 2024-03-28T16:07:54.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.185+0000 7fca46068740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca3c0810c0 con 0x56352dfad9e0 2024-03-28T16:07:54.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.185+0000 7fca3be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56352dfa8870 0x56352dfca380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:54.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.185+0000 7fca3be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56352dfa8870 0x56352dfca380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59180/0 (socket says 172.21.15.67:59180) 2024-03-28T16:07:54.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.185+0000 7fca3be00640 1 -- 172.21.15.67:0/2245903006 learned_addr learned my addr 172.21.15.67:0/2245903006 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:07:54.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.185+0000 7fca40a00640 1 --2- 172.21.15.67:0/2245903006 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56352dfa7e80 0x56352dfa3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:54.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.185+0000 7fca41400640 1 --2- 172.21.15.67:0/2245903006 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56352dfad9e0 0x56352dfc2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:54.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.185+0000 7fca3be00640 1 -- 172.21.15.67:0/2245903006 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56352dfad9e0 msgr2=0x56352dfc2a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:54.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.185+0000 7fca3be00640 1 --2- 172.21.15.67:0/2245903006 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56352dfad9e0 0x56352dfc2a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:54.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.185+0000 7fca3be00640 1 -- 172.21.15.67:0/2245903006 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56352dfa7e80 msgr2=0x56352dfa3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:54.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.185+0000 7fca3be00640 1 --2- 172.21.15.67:0/2245903006 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56352dfa7e80 0x56352dfa3a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:54.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.185+0000 7fca3be00640 1 -- 172.21.15.67:0/2245903006 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56352dbe10f0 con 0x56352dfa8870 2024-03-28T16:07:54.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.186+0000 7fca3be00640 1 --2- 172.21.15.67:0/2245903006 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56352dfa8870 0x56352dfca380 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fca280972b0 tx=0x7fca2801c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfbae139ad48ecb0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:54.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.186+0000 7fca2d200640 1 -- 172.21.15.67:0/2245903006 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca2809ae50 con 0x56352dfa8870 2024-03-28T16:07:54.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.186+0000 7fca2d200640 1 -- 172.21.15.67:0/2245903006 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fca2809d040 con 0x56352dfa8870 2024-03-28T16:07:54.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.186+0000 7fca2d200640 1 -- 172.21.15.67:0/2245903006 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca280841b0 con 0x56352dfa8870 2024-03-28T16:07:54.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.187+0000 7fca46068740 1 -- 172.21.15.67:0/2245903006 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56352dfa8870 msgr2=0x56352dfca380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:54.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.187+0000 7fca46068740 1 --2- 172.21.15.67:0/2245903006 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56352dfa8870 0x56352dfca380 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fca280972b0 tx=0x7fca2801c040 comp rx=0 tx=0).stop 2024-03-28T16:07:54.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.187+0000 7fca46068740 1 -- 172.21.15.67:0/2245903006 shutdown_connections 2024-03-28T16:07:54.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.187+0000 7fca46068740 1 --2- 172.21.15.67:0/2245903006 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56352dfad9e0 0x56352dfc2a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:54.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.187+0000 7fca46068740 1 --2- 172.21.15.67:0/2245903006 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56352dfa8870 0x56352dfca380 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:54.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.187+0000 7fca46068740 1 --2- 172.21.15.67:0/2245903006 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56352dfa7e80 0x56352dfa3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:54.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.187+0000 7fca46068740 1 -- 172.21.15.67:0/2245903006 >> 172.21.15.67:0/2245903006 conn(0x56352df70200 msgr2=0x56352dfa3310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:54.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.187+0000 7fca46068740 1 -- 172.21.15.67:0/2245903006 shutdown_connections 2024-03-28T16:07:54.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.187+0000 7fca46068740 1 -- 172.21.15.67:0/2245903006 wait complete. 2024-03-28T16:07:54.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.188+0000 7fca46068740 1 -- start start 2024-03-28T16:07:54.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.188+0000 7fca46068740 1 Processor -- start 2024-03-28T16:07:54.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.188+0000 7fca46068740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56352dfa7e80 0x56352dfc7a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:54.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.188+0000 7fca46068740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56352dfa8870 0x56352dfc7f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:54.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.188+0000 7fca46068740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56352dfad9e0 0x56352dfc8f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:54.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.188+0000 7fca46068740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca3c081700 con 0x56352dfad9e0 2024-03-28T16:07:54.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.188+0000 7fca46068740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca3c1a67e0 con 0x56352dfa8870 2024-03-28T16:07:54.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.188+0000 7fca46068740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca3c081d60 con 0x56352dfa7e80 2024-03-28T16:07:54.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.189+0000 7fca46068740 10 client.0 _ll_register_callbacks cb 0x7fca30015070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:07:54.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.189+0000 7fca41400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56352dfad9e0 0x56352dfc8f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:54.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.189+0000 7fca41400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56352dfad9e0 0x56352dfc8f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59184/0 (socket says 172.21.15.67:59184) 2024-03-28T16:07:54.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.189+0000 7fca41400640 1 -- 172.21.15.67:0/430801710 learned_addr learned my addr 172.21.15.67:0/430801710 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:07:54.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.189+0000 7fca40a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56352dfa7e80 0x56352dfc7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:54.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.189+0000 7fca3be00640 1 --2- 172.21.15.67:0/430801710 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56352dfa8870 0x56352dfc7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:54.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.190+0000 7fca41400640 1 -- 172.21.15.67:0/430801710 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56352dfa7e80 msgr2=0x56352dfc7a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:54.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.190+0000 7fca41400640 1 --2- 172.21.15.67:0/430801710 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56352dfa7e80 0x56352dfc7a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:54.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.190+0000 7fca41400640 1 -- 172.21.15.67:0/430801710 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56352dfa8870 msgr2=0x56352dfc7f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:54.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.190+0000 7fca41400640 1 --2- 172.21.15.67:0/430801710 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56352dfa8870 0x56352dfc7f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:54.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.190+0000 7fca41400640 1 -- 172.21.15.67:0/430801710 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56352dfc9520 con 0x56352dfad9e0 2024-03-28T16:07:54.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.190+0000 7fca41400640 1 --2- 172.21.15.67:0/430801710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56352dfad9e0 0x56352dfc8f30 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fca34032380 tx=0x7fca34009870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e78a9c67dc1dec0e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:54.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.190+0000 7fca22000640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:07:54.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.190+0000 7fca22000640 1 -- 172.21.15.67:0/430801710 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca34044030 con 0x56352dfad9e0 2024-03-28T16:07:54.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.191+0000 7fca46068740 1 -- 172.21.15.67:0/430801710 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fca2809d040 con 0x56352dfad9e0 2024-03-28T16:07:54.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.191+0000 7fca22000640 1 -- 172.21.15.67:0/430801710 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fca34032e00 con 0x56352dfad9e0 2024-03-28T16:07:54.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.191+0000 7fca22000640 1 -- 172.21.15.67:0/430801710 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fca34039040 con 0x56352dfad9e0 2024-03-28T16:07:54.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.191+0000 7fca46068740 10 client.15888 fetch_fsmap learned FSMap version 19 2024-03-28T16:07:54.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.191+0000 7fca46068740 1 -- 172.21.15.67:0/430801710 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x56352e02a590 con 0x56352dfad9e0 2024-03-28T16:07:54.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.191+0000 7fca22000640 1 -- 172.21.15.67:0/430801710 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fca2809d040 con 0x56352dfad9e0 2024-03-28T16:07:54.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.192+0000 7fca22000640 1 -- 172.21.15.67:0/430801710 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fca34004270 con 0x56352dfad9e0 2024-03-28T16:07:54.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.192+0000 7fca46068740 10 client.15888 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:07:54.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.192+0000 7fca46068740 10 client.15888 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:07:54.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.192+0000 7fca46068740 20 client.15888 populate_metadata read hostname 'smithi067' 2024-03-28T16:07:54.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.192+0000 7fca46068740 1 --2- 172.21.15.67:0/430801710 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x56352e02afc0 0x56352e02d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:54.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.192+0000 7fca46068740 4 client.15888 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T16:07:54.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.192+0000 7fca46068740 1 -- 172.21.15.67:0/430801710 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x56352df14ae0 con 0x56352e02afc0 2024-03-28T16:07:54.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.193+0000 7fca40a00640 1 --2- 172.21.15.67:0/430801710 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x56352e02afc0 0x56352e02d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:54.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.194+0000 7fca40a00640 1 --2- 172.21.15.67:0/430801710 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x56352e02afc0 0x56352e02d3e0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f90dde8d16e84d5c server_cookie=8892446ed34eab12 in_seq=0 out_seq=0 2024-03-28T16:07:54.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.194+0000 7fca22000640 10 client.15888 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:07:54.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.195+0000 7fca22000640 1 -- 172.21.15.67:0/430801710 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x56352df14ae0 con 0x56352e02afc0 2024-03-28T16:07:54.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.195+0000 7fca22000640 10 client.15888 handle_command_reply: tid=0 2024-03-28T16:07:54.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.195+0000 7fca46068740 1 client.15888 shutdown 2024-03-28T16:07:54.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.195+0000 7fca46068740 1 -- 172.21.15.67:0/430801710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56352dfad9e0 msgr2=0x56352dfc8f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:54.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.195+0000 7fca46068740 1 --2- 172.21.15.67:0/430801710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56352dfad9e0 0x56352dfc8f30 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fca34032380 tx=0x7fca34009870 comp rx=0 tx=0).stop 2024-03-28T16:07:54.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.195+0000 7fca46068740 1 -- 172.21.15.67:0/430801710 shutdown_connections 2024-03-28T16:07:54.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.195+0000 7fca46068740 1 --2- 172.21.15.67:0/430801710 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x56352e02afc0 0x56352e02d3e0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:54.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.195+0000 7fca46068740 1 --2- 172.21.15.67:0/430801710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56352dfad9e0 0x56352dfc8f30 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:54.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.195+0000 7fca22000640 0 client.15888 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:07:54.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.195+0000 7fca46068740 1 --2- 172.21.15.67:0/430801710 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56352dfa8870 0x56352dfc7f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:54.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.195+0000 7fca46068740 1 --2- 172.21.15.67:0/430801710 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56352dfa7e80 0x56352dfc7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:54.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.195+0000 7fca46068740 1 -- 172.21.15.67:0/430801710 >> 172.21.15.67:0/430801710 conn(0x56352df70200 msgr2=0x56352dfc2a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:54.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.196+0000 7fca46068740 1 -- 172.21.15.67:0/430801710 shutdown_connections 2024-03-28T16:07:54.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.196+0000 7fca46068740 1 -- 172.21.15.67:0/430801710 wait complete. 2024-03-28T16:07:54.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.196+0000 7fca46068740 20 client.15888 trim_cache size 0 max 16384 2024-03-28T16:07:54.198 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-28T16:07:54.198 INFO:teuthology.orchestra.run.smithi067.stdout: "status": "no active scrubs running", 2024-03-28T16:07:54.198 INFO:teuthology.orchestra.run.smithi067.stdout: "scrubs": {} 2024-03-28T16:07:54.198 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-28T16:07:54.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.197+0000 7fca43400640 1 -- 172.21.15.67:0/2044817935 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fca240a2880 msgr2=0x7fca240a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:54.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.197+0000 7fca43400640 1 --2- 172.21.15.67:0/2044817935 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fca240a2880 0x7fca240a4d40 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fca30005fd0 tx=0x7fca30005850 comp rx=0 tx=0).stop 2024-03-28T16:07:54.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.197+0000 7fca43400640 1 -- 172.21.15.67:0/2044817935 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca3c108eb0 msgr2=0x7fca3c19e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:54.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.197+0000 7fca43400640 1 --2- 172.21.15.67:0/2044817935 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca3c108eb0 0x7fca3c19e450 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fca2800b450 tx=0x7fca2800b910 comp rx=0 tx=0).stop 2024-03-28T16:07:54.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.197+0000 7fca43400640 1 -- 172.21.15.67:0/2044817935 shutdown_connections 2024-03-28T16:07:54.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.197+0000 7fca43400640 1 --2- 172.21.15.67:0/2044817935 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fca240a2880 0x7fca240a4d40 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:54.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.197+0000 7fca43400640 1 --2- 172.21.15.67:0/2044817935 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fca3c109870 0x7fca3c1a54c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:54.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.197+0000 7fca43400640 1 --2- 172.21.15.67:0/2044817935 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca3c108eb0 0x7fca3c19e450 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:54.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.197+0000 7fca43400640 1 --2- 172.21.15.67:0/2044817935 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca3c107cc0 0x7fca3c19df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:54.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.197+0000 7fca43400640 1 -- 172.21.15.67:0/2044817935 >> 172.21.15.67:0/2044817935 conn(0x7fca3c076c60 msgr2=0x7fca3c10c1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:54.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.197+0000 7fca43400640 1 -- 172.21.15.67:0/2044817935 shutdown_connections 2024-03-28T16:07:54.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.198+0000 7fca43400640 1 -- 172.21.15.67:0/2044817935 wait complete. 2024-03-28T16:07:54.224 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T16:07:54.224 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T16:07:54.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.487+0000 7f189ba00640 1 -- 172.21.15.67:0/3288719938 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18941073c0 msgr2=0x7f1894109810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:54.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.487+0000 7f189ba00640 1 --2- 172.21.15.67:0/3288719938 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18941073c0 0x7f1894109810 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f188c009ca0 tx=0x7f188c02c100 comp rx=0 tx=0).stop 2024-03-28T16:07:54.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.487+0000 7f189ba00640 1 -- 172.21.15.67:0/3288719938 shutdown_connections 2024-03-28T16:07:54.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.487+0000 7f189ba00640 1 --2- 172.21.15.67:0/3288719938 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18941073c0 0x7f1894109810 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:54.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.487+0000 7f189ba00640 1 --2- 172.21.15.67:0/3288719938 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1894104a90 0x7f1894106e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:54.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.487+0000 7f189ba00640 1 --2- 172.21.15.67:0/3288719938 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1894102160 0x7f1894104550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:54.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.487+0000 7f189ba00640 1 -- 172.21.15.67:0/3288719938 >> 172.21.15.67:0/3288719938 conn(0x7f18940fbb50 msgr2=0x7f18940fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:54.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.488+0000 7f189ba00640 1 -- 172.21.15.67:0/3288719938 shutdown_connections 2024-03-28T16:07:54.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.488+0000 7f189ba00640 1 -- 172.21.15.67:0/3288719938 wait complete. 2024-03-28T16:07:54.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.489+0000 7f189ba00640 1 Processor -- start 2024-03-28T16:07:54.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.489+0000 7f189ba00640 1 -- start start 2024-03-28T16:07:54.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.489+0000 7f189ba00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1894102160 0x7f1894101ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:54.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.490+0000 7f189ba00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1894104a90 0x7f1894100110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:54.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.490+0000 7f189ba00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f18941073c0 0x7f1894100650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:54.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.490+0000 7f189ba00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f189410d300 con 0x7f1894104a90 2024-03-28T16:07:54.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.490+0000 7f189ba00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f189410d180 con 0x7f1894102160 2024-03-28T16:07:54.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.490+0000 7f189ba00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f189410d480 con 0x7f18941073c0 2024-03-28T16:07:54.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.490+0000 7f1893e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1894104a90 0x7f1894100110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:54.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.490+0000 7f1893e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1894104a90 0x7f1894100110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59204/0 (socket says 172.21.15.67:59204) 2024-03-28T16:07:54.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.490+0000 7f1893e00640 1 -- 172.21.15.67:0/2303138524 learned_addr learned my addr 172.21.15.67:0/2303138524 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:07:54.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.490+0000 7f1899000640 1 --2- 172.21.15.67:0/2303138524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1894102160 0x7f1894101ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:54.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.491+0000 7f1893e00640 1 -- 172.21.15.67:0/2303138524 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f18941073c0 msgr2=0x7f1894100650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:54.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.491+0000 7f1893e00640 1 --2- 172.21.15.67:0/2303138524 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f18941073c0 0x7f1894100650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:54.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.491+0000 7f1893e00640 1 -- 172.21.15.67:0/2303138524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1894102160 msgr2=0x7f1894101ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:54.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.491+0000 7f1893e00640 1 --2- 172.21.15.67:0/2303138524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1894102160 0x7f1894101ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:54.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.491+0000 7f1893e00640 1 -- 172.21.15.67:0/2303138524 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1894100ee0 con 0x7f1894104a90 2024-03-28T16:07:54.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.491+0000 7f1893e00640 1 --2- 172.21.15.67:0/2303138524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1894104a90 0x7f1894100110 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f188000b450 tx=0x7f188000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:54.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.492+0000 7f1891600640 1 -- 172.21.15.67:0/2303138524 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1880015020 con 0x7f1894104a90 2024-03-28T16:07:54.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.492+0000 7f189ba00640 1 -- 172.21.15.67:0/2303138524 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18941a1b50 con 0x7f1894104a90 2024-03-28T16:07:54.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.492+0000 7f1891600640 1 -- 172.21.15.67:0/2303138524 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1880011d40 con 0x7f1894104a90 2024-03-28T16:07:54.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.492+0000 7f1891600640 1 -- 172.21.15.67:0/2303138524 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f188000f340 con 0x7f1894104a90 2024-03-28T16:07:54.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.492+0000 7f189ba00640 1 -- 172.21.15.67:0/2303138524 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18941a2120 con 0x7f1894104a90 2024-03-28T16:07:54.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.494+0000 7f189ba00640 1 -- 172.21.15.67:0/2303138524 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18940fd640 con 0x7f1894104a90 2024-03-28T16:07:54.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.496+0000 7f1891600640 1 -- 172.21.15.67:0/2303138524 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f1880018020 con 0x7f1894104a90 2024-03-28T16:07:54.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.498+0000 7f1891600640 1 --2- 172.21.15.67:0/2303138524 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f187c0a2880 0x7f187c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:54.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.498+0000 7f1891600640 1 -- 172.21.15.67:0/2303138524 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f18800ce690 con 0x7f1894104a90 2024-03-28T16:07:54.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.499+0000 7f1899000640 1 --2- 172.21.15.67:0/2303138524 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f187c0a2880 0x7f187c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:54.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.500+0000 7f1899000640 1 --2- 172.21.15.67:0/2303138524 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f187c0a2880 0x7f187c0a4d40 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f1888009890 tx=0x7f1888008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:54.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.508+0000 7f1891600640 1 -- 172.21.15.67:0/2303138524 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f188001f020 con 0x7f1894104a90 2024-03-28T16:07:54.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.848+0000 7f189ba00640 1 -- 172.21.15.67:0/2303138524 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f18940622a0 con 0x7f1894104a90 2024-03-28T16:07:54.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.849+0000 7f1891600640 1 -- 172.21.15.67:0/2303138524 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11341 (secure 0 0 0) 0x7f188000d040 con 0x7f1894104a90 2024-03-28T16:07:54.851 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:07:54.851 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2008297659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2008297659},{"type":"v1","addr":"172.21.15.67:6835","nonce":2008297659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14532,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/320295745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":320295745},{"type":"v1","addr":"172.21.15.67:6837","nonce":320295745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6839/691821273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":691821273},{"type":"v1","addr":"172.21.15.67:6839","nonce":691821273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6841/3211254875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6840","nonce":3211254875},{"type":"v1","addr":"172.21.15.67:6841","nonce":3211254875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2129665423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2129665423},{"type":"v1","addr":"172.21.15.183:6833","nonce":2129665423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24496,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/1589784425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":1589784425},{"type":"v1","addr":"172.21.15.183:6835","nonce":1589784425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24508,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/2185863835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":2185863835},{"type":"v1","addr":"172.21.15.183:6837","nonce":2185863835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24520,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6839/1242814398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":1242814398},{"type":"v1","addr":"172.21.15.183:6839","nonce":1242814398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":34409,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1008335521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1008335521},{"type":"v1","addr":"172.21.15.150:6833","nonce":1008335521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":34445,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3269935695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3269935695},{"type":"v1","addr":"172.21.15.150:6835","nonce":3269935695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T16:01:07.696824+0000","modified":"2024-03-28T16:01:11.997773+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":34508},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34475":{"gid":34475,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/1112886277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":1112886277},{"type":"v1","addr":"172.21.15.150:6837","nonce":1112886277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_34508":{"gid":34508,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6839/1211011614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":1211011614},{"type":"v1","addr":"172.21.15.150:6839","nonce":1211011614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34508,"qdb_cluster":[34508]},"id":1}]} 2024-03-28T16:07:54.851 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 19 2024-03-28T16:07:54.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.853+0000 7f189ba00640 1 -- 172.21.15.67:0/2303138524 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f187c0a2880 msgr2=0x7f187c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:54.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.853+0000 7f189ba00640 1 --2- 172.21.15.67:0/2303138524 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f187c0a2880 0x7f187c0a4d40 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f1888009890 tx=0x7f1888008040 comp rx=0 tx=0).stop 2024-03-28T16:07:54.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.853+0000 7f189ba00640 1 -- 172.21.15.67:0/2303138524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1894104a90 msgr2=0x7f1894100110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:54.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.853+0000 7f189ba00640 1 --2- 172.21.15.67:0/2303138524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1894104a90 0x7f1894100110 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f188000b450 tx=0x7f188000b910 comp rx=0 tx=0).stop 2024-03-28T16:07:54.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.853+0000 7f189ba00640 1 -- 172.21.15.67:0/2303138524 shutdown_connections 2024-03-28T16:07:54.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.854+0000 7f189ba00640 1 --2- 172.21.15.67:0/2303138524 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f187c0a2880 0x7f187c0a4d40 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:54.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.854+0000 7f189ba00640 1 --2- 172.21.15.67:0/2303138524 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f18941073c0 0x7f1894100650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:54.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.854+0000 7f189ba00640 1 --2- 172.21.15.67:0/2303138524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1894104a90 0x7f1894100110 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:54.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.854+0000 7f189ba00640 1 --2- 172.21.15.67:0/2303138524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1894102160 0x7f1894101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:54.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.854+0000 7f189ba00640 1 -- 172.21.15.67:0/2303138524 >> 172.21.15.67:0/2303138524 conn(0x7f18940fbb50 msgr2=0x7f1894102c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:54.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.854+0000 7f189ba00640 1 -- 172.21.15.67:0/2303138524 shutdown_connections 2024-03-28T16:07:54.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:54.854+0000 7f189ba00640 1 -- 172.21.15.67:0/2303138524 wait complete. 2024-03-28T16:07:54.879 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T16:07:54.879 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T16:07:55.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.116+0000 7f44c2c00640 1 -- 172.21.15.67:0/1907042616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44bc0fff10 msgr2=0x7f44bc102340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:55.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.116+0000 7f44c2c00640 1 --2- 172.21.15.67:0/1907042616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44bc0fff10 0x7f44bc102340 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f44ac009900 tx=0x7f44ac02bc30 comp rx=0 tx=0).stop 2024-03-28T16:07:55.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.116+0000 7f44c2c00640 1 -- 172.21.15.67:0/1907042616 shutdown_connections 2024-03-28T16:07:55.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.116+0000 7f44c2c00640 1 --2- 172.21.15.67:0/1907042616 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44bc105230 0x7f44bc1056b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:55.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.116+0000 7f44c2c00640 1 --2- 172.21.15.67:0/1907042616 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f44bc102880 0x7f44bc104cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:55.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.116+0000 7f44c2c00640 1 --2- 172.21.15.67:0/1907042616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44bc0fff10 0x7f44bc102340 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:55.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.116+0000 7f44c2c00640 1 -- 172.21.15.67:0/1907042616 >> 172.21.15.67:0/1907042616 conn(0x7f44bc0fbb30 msgr2=0x7f44bc0fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:55.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.116+0000 7f44c2c00640 1 -- 172.21.15.67:0/1907042616 shutdown_connections 2024-03-28T16:07:55.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.116+0000 7f44c2c00640 1 -- 172.21.15.67:0/1907042616 wait complete. 2024-03-28T16:07:55.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.117+0000 7f44c2c00640 1 Processor -- start 2024-03-28T16:07:55.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.117+0000 7f44c2c00640 1 -- start start 2024-03-28T16:07:55.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.118+0000 7f44c2c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44bc0fff10 0x7f44bc19df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:55.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.118+0000 7f44c2c00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44bc102880 0x7f44bc19e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:55.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.118+0000 7f44c2c00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f44bc105230 0x7f44bc1a5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:55.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.118+0000 7f44c2c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44bc113810 con 0x7f44bc0fff10 2024-03-28T16:07:55.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.118+0000 7f44c2c00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44bc113690 con 0x7f44bc102880 2024-03-28T16:07:55.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.118+0000 7f44c2c00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44bc113990 con 0x7f44bc105230 2024-03-28T16:07:55.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.118+0000 7f44bbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44bc0fff10 0x7f44bc19df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:55.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.118+0000 7f44bbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44bc0fff10 0x7f44bc19df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59222/0 (socket says 172.21.15.67:59222) 2024-03-28T16:07:55.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.118+0000 7f44bbe00640 1 -- 172.21.15.67:0/3798269576 learned_addr learned my addr 172.21.15.67:0/3798269576 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:07:55.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.118+0000 7f44bb400640 1 --2- 172.21.15.67:0/3798269576 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44bc102880 0x7f44bc19e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:55.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.119+0000 7f44bbe00640 1 -- 172.21.15.67:0/3798269576 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f44bc105230 msgr2=0x7f44bc1a5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:55.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.119+0000 7f44bbe00640 1 --2- 172.21.15.67:0/3798269576 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f44bc105230 0x7f44bc1a5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:55.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.119+0000 7f44bbe00640 1 -- 172.21.15.67:0/3798269576 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44bc102880 msgr2=0x7f44bc19e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:55.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.119+0000 7f44bbe00640 1 --2- 172.21.15.67:0/3798269576 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44bc102880 0x7f44bc19e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:55.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.119+0000 7f44bbe00640 1 -- 172.21.15.67:0/3798269576 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44bc1a5bb0 con 0x7f44bc0fff10 2024-03-28T16:07:55.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.119+0000 7f44bbe00640 1 --2- 172.21.15.67:0/3798269576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44bc0fff10 0x7f44bc19df90 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f44ac006010 tx=0x7f44ac0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:55.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.119+0000 7f44b8c00640 1 -- 172.21.15.67:0/3798269576 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44ac043020 con 0x7f44bc0fff10 2024-03-28T16:07:55.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.120+0000 7f44c2c00640 1 -- 172.21.15.67:0/3798269576 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44bc1a5e40 con 0x7f44bc0fff10 2024-03-28T16:07:55.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.120+0000 7f44b8c00640 1 -- 172.21.15.67:0/3798269576 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f44ac004aa0 con 0x7f44bc0fff10 2024-03-28T16:07:55.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.120+0000 7f44b8c00640 1 -- 172.21.15.67:0/3798269576 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44ac003bb0 con 0x7f44bc0fff10 2024-03-28T16:07:55.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.120+0000 7f44c2c00640 1 -- 172.21.15.67:0/3798269576 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44bc1a6330 con 0x7f44bc0fff10 2024-03-28T16:07:55.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.125+0000 7f44c5996740 1 Processor -- start 2024-03-28T16:07:55.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.125+0000 7f44c5996740 1 -- start start 2024-03-28T16:07:55.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.125+0000 7f44b8c00640 1 -- 172.21.15.67:0/3798269576 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f44ac046020 con 0x7f44bc0fff10 2024-03-28T16:07:55.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.125+0000 7f44c5996740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557e9d18e810 0x557e9d18ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:55.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.126+0000 7f44c5996740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557e9d18f9e0 0x557e9d18fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:55.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.126+0000 7f44c5996740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557e9d1903a0 0x557e9d196c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:55.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.126+0000 7f44c5996740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e9d0f32c0 con 0x557e9d1903a0 2024-03-28T16:07:55.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.126+0000 7f44c5996740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e9d09bc90 con 0x557e9d18f9e0 2024-03-28T16:07:55.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.126+0000 7f44c5996740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e9d0ef360 con 0x557e9d18e810 2024-03-28T16:07:55.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.126+0000 7f44bb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557e9d1903a0 0x557e9d196c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:55.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.126+0000 7f44bb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557e9d1903a0 0x557e9d196c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59230/0 (socket says 172.21.15.67:59230) 2024-03-28T16:07:55.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.126+0000 7f44bb400640 1 -- 172.21.15.67:0/2811904085 learned_addr learned my addr 172.21.15.67:0/2811904085 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:07:55.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.126+0000 7f44bbe00640 1 --2- 172.21.15.67:0/2811904085 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557e9d18f9e0 0x557e9d18fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:55.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.126+0000 7f44c0c00640 1 --2- 172.21.15.67:0/2811904085 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557e9d18e810 0x557e9d18ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:55.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.127+0000 7f44bb400640 1 -- 172.21.15.67:0/2811904085 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557e9d18e810 msgr2=0x557e9d18ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:55.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.127+0000 7f44bb400640 1 --2- 172.21.15.67:0/2811904085 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557e9d18e810 0x557e9d18ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:55.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.127+0000 7f44bb400640 1 -- 172.21.15.67:0/2811904085 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557e9d18f9e0 msgr2=0x557e9d18fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:55.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.127+0000 7f44bb400640 1 --2- 172.21.15.67:0/2811904085 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557e9d18f9e0 0x557e9d18fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:55.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.127+0000 7f44bb400640 1 -- 172.21.15.67:0/2811904085 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557e9d0c0080 con 0x557e9d1903a0 2024-03-28T16:07:55.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.127+0000 7f44b8c00640 1 --2- 172.21.15.67:0/3798269576 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f44a00a2800 0x7f44a00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:55.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.128+0000 7f44b8c00640 1 -- 172.21.15.67:0/3798269576 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f44ac0f0eb0 con 0x7f44bc0fff10 2024-03-28T16:07:55.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.128+0000 7f44bb400640 1 --2- 172.21.15.67:0/2811904085 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557e9d1903a0 0x557e9d196c30 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f44a8009a40 tx=0x7f44a8009f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1013af253850e1d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:55.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.128+0000 7f44c0c00640 1 --2- 172.21.15.67:0/3798269576 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f44a00a2800 0x7f44a00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:55.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.128+0000 7f44b1200640 1 -- 172.21.15.67:0/2811904085 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44a8017070 con 0x557e9d1903a0 2024-03-28T16:07:55.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.128+0000 7f44b1200640 1 -- 172.21.15.67:0/2811904085 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f44a8013470 con 0x557e9d1903a0 2024-03-28T16:07:55.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.128+0000 7f44b1200640 1 -- 172.21.15.67:0/2811904085 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44a80125d0 con 0x557e9d1903a0 2024-03-28T16:07:55.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.129+0000 7f44c5996740 1 -- 172.21.15.67:0/2811904085 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557e9d1903a0 msgr2=0x557e9d196c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:55.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.129+0000 7f44c0c00640 1 --2- 172.21.15.67:0/3798269576 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f44a00a2800 0x7f44a00a4cc0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f44b4009270 tx=0x7f44b4009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:55.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.129+0000 7f44c5996740 1 --2- 172.21.15.67:0/2811904085 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557e9d1903a0 0x557e9d196c30 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f44a8009a40 tx=0x7f44a8009f00 comp rx=0 tx=0).stop 2024-03-28T16:07:55.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.129+0000 7f44c5996740 1 -- 172.21.15.67:0/2811904085 shutdown_connections 2024-03-28T16:07:55.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.129+0000 7f44c5996740 1 --2- 172.21.15.67:0/2811904085 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557e9d1903a0 0x557e9d196c30 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:55.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.129+0000 7f44c5996740 1 --2- 172.21.15.67:0/2811904085 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557e9d18f9e0 0x557e9d18fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:55.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.129+0000 7f44c5996740 1 --2- 172.21.15.67:0/2811904085 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557e9d18e810 0x557e9d18ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:55.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.129+0000 7f44c5996740 1 -- 172.21.15.67:0/2811904085 >> 172.21.15.67:0/2811904085 conn(0x557e9d189c00 msgr2=0x557e9d18a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:55.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.129+0000 7f44c5996740 1 -- 172.21.15.67:0/2811904085 shutdown_connections 2024-03-28T16:07:55.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.130+0000 7f44c5996740 1 -- 172.21.15.67:0/2811904085 wait complete. 2024-03-28T16:07:55.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.130+0000 7f44c5996740 1 -- start start 2024-03-28T16:07:55.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.130+0000 7f44c5996740 1 Processor -- start 2024-03-28T16:07:55.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.131+0000 7f44c5996740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557e9d18e810 0x557e9d198a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:55.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.131+0000 7f44c5996740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557e9d18f9e0 0x557e9d198f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:55.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.131+0000 7f44c5996740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557e9d1903a0 0x557e9d19ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:55.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.131+0000 7f44c5996740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e9d1973b0 con 0x557e9d1903a0 2024-03-28T16:07:55.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.131+0000 7f44c5996740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e9d0af0d0 con 0x557e9d18e810 2024-03-28T16:07:55.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.131+0000 7f44c5996740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e9d197530 con 0x557e9d18f9e0 2024-03-28T16:07:55.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.131+0000 7f44c5996740 10 client.0 _ll_register_callbacks cb 0x557e9d0b3e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:07:55.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.131+0000 7f44c0c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557e9d1903a0 0x557e9d19ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:55.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.132+0000 7f44c0c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557e9d1903a0 0x557e9d19ffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59238/0 (socket says 172.21.15.67:59238) 2024-03-28T16:07:55.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.132+0000 7f44c0c00640 1 -- 172.21.15.67:0/1752983503 learned_addr learned my addr 172.21.15.67:0/1752983503 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:07:55.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.132+0000 7f44bbe00640 1 --2- 172.21.15.67:0/1752983503 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557e9d18e810 0x557e9d198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:55.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.132+0000 7f44c0c00640 1 -- 172.21.15.67:0/1752983503 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557e9d18f9e0 msgr2=0x557e9d198f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:55.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.132+0000 7f44c0c00640 1 --2- 172.21.15.67:0/1752983503 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557e9d18f9e0 0x557e9d198f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:55.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.132+0000 7f44c0c00640 1 -- 172.21.15.67:0/1752983503 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557e9d18e810 msgr2=0x557e9d198a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:55.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.132+0000 7f44c0c00640 1 --2- 172.21.15.67:0/1752983503 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557e9d18e810 0x557e9d198a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:55.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.132+0000 7f44c0c00640 1 -- 172.21.15.67:0/1752983503 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557e9ce073f0 con 0x557e9d1903a0 2024-03-28T16:07:55.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.133+0000 7f44c0c00640 1 --2- 172.21.15.67:0/1752983503 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557e9d1903a0 0x557e9d19ffd0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f44b4009800 tx=0x7f44b4009200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eef6186e01a7210c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:55.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.133+0000 7f44a5600640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:07:55.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.133+0000 7f44a5600640 1 -- 172.21.15.67:0/1752983503 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44b4023070 con 0x557e9d1903a0 2024-03-28T16:07:55.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.133+0000 7f44c5996740 1 -- 172.21.15.67:0/1752983503 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f44a8013470 con 0x557e9d1903a0 2024-03-28T16:07:55.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.133+0000 7f44a5600640 1 -- 172.21.15.67:0/1752983503 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f44b40119b0 con 0x557e9d1903a0 2024-03-28T16:07:55.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.134+0000 7f44a5600640 1 -- 172.21.15.67:0/1752983503 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f44b402f050 con 0x557e9d1903a0 2024-03-28T16:07:55.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.134+0000 7f44c5996740 10 client.15906 fetch_fsmap learned FSMap version 19 2024-03-28T16:07:55.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.134+0000 7f44c5996740 1 -- 172.21.15.67:0/1752983503 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x557e9ced17d0 con 0x557e9d1903a0 2024-03-28T16:07:55.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.134+0000 7f44a5600640 1 -- 172.21.15.67:0/1752983503 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f44a8013470 con 0x557e9d1903a0 2024-03-28T16:07:55.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.135+0000 7f44a5600640 1 -- 172.21.15.67:0/1752983503 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f44b400f040 con 0x557e9d1903a0 2024-03-28T16:07:55.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.135+0000 7f44c5996740 10 client.15906 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:07:55.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.135+0000 7f44c5996740 10 client.15906 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:07:55.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.135+0000 7f44c5996740 20 client.15906 populate_metadata read hostname 'smithi067' 2024-03-28T16:07:55.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.135+0000 7f44c5996740 1 --2- 172.21.15.67:0/1752983503 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x557e9d1a4570 0x557e9d1a69e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:55.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.135+0000 7f44c5996740 4 client.15906 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:07:55.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.135+0000 7f44c5996740 1 -- 172.21.15.67:0/1752983503 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557e9d100800 con 0x557e9d1a4570 2024-03-28T16:07:55.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.137+0000 7f44bbe00640 1 --2- 172.21.15.67:0/1752983503 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x557e9d1a4570 0x557e9d1a69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:55.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.137+0000 7f44a5600640 10 client.15906 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:07:55.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.137+0000 7f44bbe00640 1 --2- 172.21.15.67:0/1752983503 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x557e9d1a4570 0x557e9d1a69e0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c3fd77765b6943a2 server_cookie=9ea8497d7a42e510 in_seq=0 out_seq=0 2024-03-28T16:07:55.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.138+0000 7f44a5600640 1 -- 172.21.15.67:0/1752983503 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x557e9d100800 con 0x557e9d1a4570 2024-03-28T16:07:55.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.139+0000 7f44a5600640 10 client.15906 handle_command_reply: tid=0 2024-03-28T16:07:55.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.139+0000 7f44c5996740 1 client.15906 shutdown 2024-03-28T16:07:55.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.139+0000 7f44c5996740 1 -- 172.21.15.67:0/1752983503 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557e9d1903a0 msgr2=0x557e9d19ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:55.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.139+0000 7f44c5996740 1 --2- 172.21.15.67:0/1752983503 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557e9d1903a0 0x557e9d19ffd0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f44b4009800 tx=0x7f44b4009200 comp rx=0 tx=0).stop 2024-03-28T16:07:55.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.139+0000 7f44c5996740 1 -- 172.21.15.67:0/1752983503 shutdown_connections 2024-03-28T16:07:55.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.139+0000 7f44c5996740 1 --2- 172.21.15.67:0/1752983503 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x557e9d1a4570 0x557e9d1a69e0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:55.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.139+0000 7f44c5996740 1 --2- 172.21.15.67:0/1752983503 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557e9d1903a0 0x557e9d19ffd0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:55.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.139+0000 7f44c5996740 1 --2- 172.21.15.67:0/1752983503 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557e9d18f9e0 0x557e9d198f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:55.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.139+0000 7f44c5996740 1 --2- 172.21.15.67:0/1752983503 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557e9d18e810 0x557e9d198a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:55.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.139+0000 7f44c5996740 1 -- 172.21.15.67:0/1752983503 >> 172.21.15.67:0/1752983503 conn(0x557e9d189c00 msgr2=0x557e9d188eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:55.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.139+0000 7f44a5600640 0 client.15906 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:07:55.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.139+0000 7f44c5996740 1 -- 172.21.15.67:0/1752983503 shutdown_connections 2024-03-28T16:07:55.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.140+0000 7f44c5996740 1 -- 172.21.15.67:0/1752983503 wait complete. 2024-03-28T16:07:55.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.140+0000 7f44c5996740 20 client.15906 trim_cache size 0 max 16384 2024-03-28T16:07:55.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.155+0000 7f44c5996740 1 Processor -- start 2024-03-28T16:07:55.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.155+0000 7f44c5996740 1 -- start start 2024-03-28T16:07:55.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.156+0000 7f44c5996740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557e9d18e810 0x557e9d192c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:55.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.156+0000 7f44c5996740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557e9d18f9e0 0x557e9d19ff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:55.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.156+0000 7f44c5996740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557e9d1903a0 0x557e9d193150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:55.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.156+0000 7f44c5996740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44bc1ad100 con 0x557e9d18e810 2024-03-28T16:07:55.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.156+0000 7f44c5996740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44bc1af6c0 con 0x557e9d18f9e0 2024-03-28T16:07:55.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.156+0000 7f44c5996740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44bc1ad700 con 0x557e9d1903a0 2024-03-28T16:07:55.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.156+0000 7f44bbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557e9d18e810 0x557e9d192c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:55.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.156+0000 7f44bbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557e9d18e810 0x557e9d192c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59246/0 (socket says 172.21.15.67:59246) 2024-03-28T16:07:55.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.156+0000 7f44bbe00640 1 -- 172.21.15.67:0/2310010269 learned_addr learned my addr 172.21.15.67:0/2310010269 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:07:55.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.156+0000 7f44c0c00640 1 --2- 172.21.15.67:0/2310010269 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557e9d1903a0 0x557e9d193150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:55.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.156+0000 7f44bb400640 1 --2- 172.21.15.67:0/2310010269 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557e9d18f9e0 0x557e9d19ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:55.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.157+0000 7f44bbe00640 1 -- 172.21.15.67:0/2310010269 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557e9d1903a0 msgr2=0x557e9d193150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:55.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.157+0000 7f44bbe00640 1 --2- 172.21.15.67:0/2310010269 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557e9d1903a0 0x557e9d193150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:55.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.157+0000 7f44bbe00640 1 -- 172.21.15.67:0/2310010269 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557e9d18f9e0 msgr2=0x557e9d19ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:55.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.157+0000 7f44bbe00640 1 --2- 172.21.15.67:0/2310010269 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557e9d18f9e0 0x557e9d19ff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:55.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.157+0000 7f44bbe00640 1 -- 172.21.15.67:0/2310010269 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557e9d1a6d50 con 0x557e9d18e810 2024-03-28T16:07:55.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.157+0000 7f44bbe00640 1 --2- 172.21.15.67:0/2310010269 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557e9d18e810 0x557e9d192c10 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f44ac000c00 tx=0x7f44ac0f0040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1900713e744bfa4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:55.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.157+0000 7f44a7e00640 1 -- 172.21.15.67:0/2310010269 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44ac0bd5d0 con 0x557e9d18e810 2024-03-28T16:07:55.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.158+0000 7f44a7e00640 1 -- 172.21.15.67:0/2310010269 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f44ac0bdf00 con 0x557e9d18e810 2024-03-28T16:07:55.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.158+0000 7f44a7e00640 1 -- 172.21.15.67:0/2310010269 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44ac0bd160 con 0x557e9d18e810 2024-03-28T16:07:55.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.159+0000 7f44c5996740 1 -- 172.21.15.67:0/2310010269 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557e9d18e810 msgr2=0x557e9d192c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:55.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.159+0000 7f44c5996740 1 --2- 172.21.15.67:0/2310010269 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557e9d18e810 0x557e9d192c10 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f44ac000c00 tx=0x7f44ac0f0040 comp rx=0 tx=0).stop 2024-03-28T16:07:55.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.159+0000 7f44c5996740 1 -- 172.21.15.67:0/2310010269 shutdown_connections 2024-03-28T16:07:55.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.159+0000 7f44c5996740 1 --2- 172.21.15.67:0/2310010269 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557e9d1903a0 0x557e9d193150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:55.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.159+0000 7f44c5996740 1 --2- 172.21.15.67:0/2310010269 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557e9d18f9e0 0x557e9d19ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:55.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.159+0000 7f44c5996740 1 --2- 172.21.15.67:0/2310010269 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557e9d18e810 0x557e9d192c10 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:55.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.159+0000 7f44c5996740 1 -- 172.21.15.67:0/2310010269 >> 172.21.15.67:0/2310010269 conn(0x557e9d189c00 msgr2=0x557e9d18d6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:55.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.159+0000 7f44c5996740 1 -- 172.21.15.67:0/2310010269 shutdown_connections 2024-03-28T16:07:55.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.159+0000 7f44c5996740 1 -- 172.21.15.67:0/2310010269 wait complete. 2024-03-28T16:07:55.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.160+0000 7f44c5996740 1 -- start start 2024-03-28T16:07:55.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.160+0000 7f44c5996740 1 Processor -- start 2024-03-28T16:07:55.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.160+0000 7f44c5996740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557e9d18e810 0x557e9d19d770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:55.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.161+0000 7f44c5996740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557e9d18f9e0 0x557e9d19dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:55.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.161+0000 7f44c5996740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557e9d1903a0 0x557e9d19ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:55.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.161+0000 7f44c5996740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44bc1add40 con 0x557e9d1903a0 2024-03-28T16:07:55.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.161+0000 7f44c5996740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44bc1af060 con 0x557e9d18f9e0 2024-03-28T16:07:55.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.161+0000 7f44c5996740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44bc1ae3a0 con 0x557e9d18e810 2024-03-28T16:07:55.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.161+0000 7f44c5996740 10 client.0 _ll_register_callbacks cb 0x7f44b4016070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:07:55.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.161+0000 7f44c0c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557e9d1903a0 0x557e9d19ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:55.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.162+0000 7f44c0c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557e9d1903a0 0x557e9d19ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59262/0 (socket says 172.21.15.67:59262) 2024-03-28T16:07:55.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.162+0000 7f44c0c00640 1 -- 172.21.15.67:0/174374635 learned_addr learned my addr 172.21.15.67:0/174374635 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:07:55.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.162+0000 7f44bbe00640 1 --2- 172.21.15.67:0/174374635 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557e9d18e810 0x557e9d19d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:55.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.162+0000 7f44bb400640 1 --2- 172.21.15.67:0/174374635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557e9d18f9e0 0x557e9d19dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:55.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.162+0000 7f44c0c00640 1 -- 172.21.15.67:0/174374635 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557e9d18e810 msgr2=0x557e9d19d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:55.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.162+0000 7f44c0c00640 1 --2- 172.21.15.67:0/174374635 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557e9d18e810 0x557e9d19d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:55.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.163+0000 7f44c0c00640 1 -- 172.21.15.67:0/174374635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557e9d18f9e0 msgr2=0x557e9d19dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:55.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.163+0000 7f44c0c00640 1 --2- 172.21.15.67:0/174374635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557e9d18f9e0 0x557e9d19dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:55.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.163+0000 7f44c0c00640 1 -- 172.21.15.67:0/174374635 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557e9d19f2b0 con 0x557e9d1903a0 2024-03-28T16:07:55.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.163+0000 7f44bb400640 1 --2- 172.21.15.67:0/174374635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557e9d18f9e0 0x557e9d19dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T16:07:55.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.163+0000 7f44c0c00640 1 --2- 172.21.15.67:0/174374635 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557e9d1903a0 0x557e9d19ecc0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f44b4005d70 tx=0x7f44b4027720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d92c172856bdf5c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:55.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.163+0000 7f44a5600640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:07:55.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.163+0000 7f44a5600640 1 -- 172.21.15.67:0/174374635 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44b4023050 con 0x557e9d1903a0 2024-03-28T16:07:55.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.164+0000 7f44a5600640 1 -- 172.21.15.67:0/174374635 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f44b4003970 con 0x557e9d1903a0 2024-03-28T16:07:55.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.164+0000 7f44c5996740 1 -- 172.21.15.67:0/174374635 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f44ac0bdf00 con 0x557e9d1903a0 2024-03-28T16:07:55.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.164+0000 7f44a5600640 1 -- 172.21.15.67:0/174374635 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f44b4003b70 con 0x557e9d1903a0 2024-03-28T16:07:55.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.164+0000 7f44c5996740 10 client.15912 fetch_fsmap learned FSMap version 19 2024-03-28T16:07:55.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.164+0000 7f44c5996740 1 -- 172.21.15.67:0/174374635 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x557e9d1fff30 con 0x557e9d1903a0 2024-03-28T16:07:55.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.165+0000 7f44a5600640 1 -- 172.21.15.67:0/174374635 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f44ac0bdf00 con 0x557e9d1903a0 2024-03-28T16:07:55.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.165+0000 7f44a5600640 1 -- 172.21.15.67:0/174374635 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f44b4027990 con 0x557e9d1903a0 2024-03-28T16:07:55.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.165+0000 7f44c5996740 10 client.15912 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:07:55.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.165+0000 7f44c5996740 10 client.15912 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:07:55.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.165+0000 7f44c5996740 20 client.15912 populate_metadata read hostname 'smithi067' 2024-03-28T16:07:55.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.165+0000 7f44c5996740 1 --2- 172.21.15.67:0/174374635 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x557e9d2007a0 0x557e9d202bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:55.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.165+0000 7f44c5996740 4 client.15912 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T16:07:55.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.165+0000 7f44c5996740 1 -- 172.21.15.67:0/174374635 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x557e9d0fab50 con 0x557e9d2007a0 2024-03-28T16:07:55.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.166+0000 7f44bbe00640 1 --2- 172.21.15.67:0/174374635 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x557e9d2007a0 0x557e9d202bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:55.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.167+0000 7f44bbe00640 1 --2- 172.21.15.67:0/174374635 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x557e9d2007a0 0x557e9d202bc0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fdc168c7528a3698 server_cookie=7121dc4f878819c5 in_seq=0 out_seq=0 2024-03-28T16:07:55.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.167+0000 7f44a5600640 10 client.15912 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:07:55.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.168+0000 7f44a5600640 1 -- 172.21.15.67:0/174374635 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x557e9d0fab50 con 0x557e9d2007a0 2024-03-28T16:07:55.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.168+0000 7f44a5600640 10 client.15912 handle_command_reply: tid=0 2024-03-28T16:07:55.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.168+0000 7f44c5996740 1 client.15912 shutdown 2024-03-28T16:07:55.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.169+0000 7f44c5996740 1 -- 172.21.15.67:0/174374635 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557e9d1903a0 msgr2=0x557e9d19ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:55.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.169+0000 7f44c5996740 1 --2- 172.21.15.67:0/174374635 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557e9d1903a0 0x557e9d19ecc0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f44b4005d70 tx=0x7f44b4027720 comp rx=0 tx=0).stop 2024-03-28T16:07:55.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.169+0000 7f44c5996740 1 -- 172.21.15.67:0/174374635 shutdown_connections 2024-03-28T16:07:55.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.169+0000 7f44c5996740 1 --2- 172.21.15.67:0/174374635 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x557e9d2007a0 0x557e9d202bc0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:55.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.169+0000 7f44c5996740 1 --2- 172.21.15.67:0/174374635 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557e9d1903a0 0x557e9d19ecc0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:55.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.169+0000 7f44c5996740 1 --2- 172.21.15.67:0/174374635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557e9d18f9e0 0x557e9d19dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:55.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.169+0000 7f44c5996740 1 --2- 172.21.15.67:0/174374635 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557e9d18e810 0x557e9d19d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:55.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.169+0000 7f44c5996740 1 -- 172.21.15.67:0/174374635 >> 172.21.15.67:0/174374635 conn(0x557e9d189c00 msgr2=0x557e9d188620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:55.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.170+0000 7f44a5600640 0 client.15912 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:07:55.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.171+0000 7f44c5996740 1 -- 172.21.15.67:0/174374635 shutdown_connections 2024-03-28T16:07:55.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.171+0000 7f44c5996740 1 -- 172.21.15.67:0/174374635 wait complete. 2024-03-28T16:07:55.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.171+0000 7f44c5996740 20 client.15912 trim_cache size 0 max 16384 2024-03-28T16:07:55.173 INFO:teuthology.orchestra.run.smithi067.stdout:[] 2024-03-28T16:07:55.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.172+0000 7f44c2c00640 1 -- 172.21.15.67:0/3798269576 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f44a00a2800 msgr2=0x7f44a00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:55.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.172+0000 7f44c2c00640 1 --2- 172.21.15.67:0/3798269576 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f44a00a2800 0x7f44a00a4cc0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f44b4009270 tx=0x7f44b4009500 comp rx=0 tx=0).stop 2024-03-28T16:07:55.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.172+0000 7f44c2c00640 1 -- 172.21.15.67:0/3798269576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44bc0fff10 msgr2=0x7f44bc19df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:55.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.172+0000 7f44c2c00640 1 --2- 172.21.15.67:0/3798269576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44bc0fff10 0x7f44bc19df90 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f44ac006010 tx=0x7f44ac0042f0 comp rx=0 tx=0).stop 2024-03-28T16:07:55.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.173+0000 7f44c2c00640 1 -- 172.21.15.67:0/3798269576 shutdown_connections 2024-03-28T16:07:55.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.173+0000 7f44c2c00640 1 --2- 172.21.15.67:0/3798269576 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f44a00a2800 0x7f44a00a4cc0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:55.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.173+0000 7f44c2c00640 1 --2- 172.21.15.67:0/3798269576 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f44bc105230 0x7f44bc1a5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:55.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.173+0000 7f44c2c00640 1 --2- 172.21.15.67:0/3798269576 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44bc102880 0x7f44bc19e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:55.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.173+0000 7f44c2c00640 1 --2- 172.21.15.67:0/3798269576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44bc0fff10 0x7f44bc19df90 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:55.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.173+0000 7f44c2c00640 1 -- 172.21.15.67:0/3798269576 >> 172.21.15.67:0/3798269576 conn(0x7f44bc0fbb30 msgr2=0x7f44bc1033a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:55.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.173+0000 7f44c2c00640 1 -- 172.21.15.67:0/3798269576 shutdown_connections 2024-03-28T16:07:55.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:55.173+0000 7f44c2c00640 1 -- 172.21.15.67:0/3798269576 wait complete. 2024-03-28T16:07:55.197 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:07:55 smithi067 ceph-mon[31707]: pgmap v569: 97 pgs: 97 active+clean; 5.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 158 MiB/s wr, 41 op/s 2024-03-28T16:07:55.197 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:07:55 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/2303138524' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:07:55.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:07:55 smithi150 ceph-mon[34825]: pgmap v569: 97 pgs: 97 active+clean; 5.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 158 MiB/s wr, 41 op/s 2024-03-28T16:07:55.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:07:55 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/2303138524' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:07:55.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:07:55 smithi183 ceph-mon[35580]: pgmap v569: 97 pgs: 97 active+clean; 5.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 158 MiB/s wr, 41 op/s 2024-03-28T16:07:55.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:07:55 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/2303138524' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:07:56.199 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T16:07:56.200 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T16:07:56.200 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T16:07:56.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:07:56 smithi183 ceph-mon[35580]: pgmap v570: 97 pgs: 97 active+clean; 5.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 162 MiB/s wr, 43 op/s 2024-03-28T16:07:56.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.472+0000 7f806da00640 1 -- 172.21.15.67:0/393676059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8068108eb0 msgr2=0x7f8068109330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:56.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.472+0000 7f806da00640 1 --2- 172.21.15.67:0/393676059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8068108eb0 0x7f8068109330 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f8058009900 tx=0x7f805802bc30 comp rx=0 tx=0).stop 2024-03-28T16:07:56.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.472+0000 7f806da00640 1 -- 172.21.15.67:0/393676059 shutdown_connections 2024-03-28T16:07:56.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.472+0000 7f806da00640 1 --2- 172.21.15.67:0/393676059 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8068109870 0x7f8068110100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:56.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.472+0000 7f806da00640 1 --2- 172.21.15.67:0/393676059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8068108eb0 0x7f8068109330 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:56.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.472+0000 7f806da00640 1 --2- 172.21.15.67:0/393676059 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8068107cc0 0x7f80681080e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:56.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.472+0000 7f806da00640 1 -- 172.21.15.67:0/393676059 >> 172.21.15.67:0/393676059 conn(0x7f8068076c60 msgr2=0x7f80680790a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:56.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.472+0000 7f806da00640 1 -- 172.21.15.67:0/393676059 shutdown_connections 2024-03-28T16:07:56.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.472+0000 7f806da00640 1 -- 172.21.15.67:0/393676059 wait complete. 2024-03-28T16:07:56.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.473+0000 7f806da00640 1 Processor -- start 2024-03-28T16:07:56.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.473+0000 7f806da00640 1 -- start start 2024-03-28T16:07:56.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.474+0000 7f806da00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8068107cc0 0x7f806819df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:56.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.474+0000 7f806da00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8068108eb0 0x7f806819e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:56.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.474+0000 7f806da00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8068109870 0x7f80681a54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:56.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.474+0000 7f806da00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8068113820 con 0x7f8068108eb0 2024-03-28T16:07:56.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.474+0000 7f806da00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80681136a0 con 0x7f8068109870 2024-03-28T16:07:56.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.474+0000 7f806da00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80681139a0 con 0x7f8068107cc0 2024-03-28T16:07:56.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.474+0000 7f8066000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8068108eb0 0x7f806819e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:56.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.474+0000 7f8066000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8068108eb0 0x7f806819e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59270/0 (socket says 172.21.15.67:59270) 2024-03-28T16:07:56.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.474+0000 7f8066000640 1 -- 172.21.15.67:0/3962349327 learned_addr learned my addr 172.21.15.67:0/3962349327 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:07:56.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.475+0000 7f8066a00640 1 --2- 172.21.15.67:0/3962349327 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8068107cc0 0x7f806819df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:56.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.475+0000 7f8066000640 1 -- 172.21.15.67:0/3962349327 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8068107cc0 msgr2=0x7f806819df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:56.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.475+0000 7f8066000640 1 --2- 172.21.15.67:0/3962349327 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8068107cc0 0x7f806819df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:56.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.475+0000 7f8066000640 1 -- 172.21.15.67:0/3962349327 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8068109870 msgr2=0x7f80681a54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:56.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.475+0000 7f8066000640 1 --2- 172.21.15.67:0/3962349327 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8068109870 0x7f80681a54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:56.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.475+0000 7f8066000640 1 -- 172.21.15.67:0/3962349327 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80681a5b50 con 0x7f8068108eb0 2024-03-28T16:07:56.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.475+0000 7f8066a00640 1 --2- 172.21.15.67:0/3962349327 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8068107cc0 0x7f806819df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:07:56.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.475+0000 7f8066000640 1 --2- 172.21.15.67:0/3962349327 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8068108eb0 0x7f806819e470 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f8058000c00 tx=0x7f80580042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:56.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.476+0000 7f805f400640 1 -- 172.21.15.67:0/3962349327 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8058043020 con 0x7f8068108eb0 2024-03-28T16:07:56.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.476+0000 7f805f400640 1 -- 172.21.15.67:0/3962349327 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f80580054c0 con 0x7f8068108eb0 2024-03-28T16:07:56.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.476+0000 7f806da00640 1 -- 172.21.15.67:0/3962349327 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80681a5de0 con 0x7f8068108eb0 2024-03-28T16:07:56.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.476+0000 7f805f400640 1 -- 172.21.15.67:0/3962349327 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8058003d70 con 0x7f8068108eb0 2024-03-28T16:07:56.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.476+0000 7f806da00640 1 -- 172.21.15.67:0/3962349327 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80681a62a0 con 0x7f8068108eb0 2024-03-28T16:07:56.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.479+0000 7f805f400640 1 -- 172.21.15.67:0/3962349327 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f8058046020 con 0x7f8068108eb0 2024-03-28T16:07:56.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.481+0000 7f8070322740 1 Processor -- start 2024-03-28T16:07:56.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.481+0000 7f8070322740 1 -- start start 2024-03-28T16:07:56.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.481+0000 7f8070322740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564661cc70a0 0x564661cc74c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:56.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.481+0000 7f8070322740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564661cc7a90 0x564661ccc590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:56.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.481+0000 7f8070322740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564661cccc00 0x564661ccf090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:56.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.481+0000 7f8070322740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x564661c2ceb0 con 0x564661cccc00 2024-03-28T16:07:56.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.481+0000 7f8070322740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x564661bd5a60 con 0x564661cc7a90 2024-03-28T16:07:56.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.481+0000 7f8070322740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x564661c28f50 con 0x564661cc70a0 2024-03-28T16:07:56.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.481+0000 7f805f400640 1 --2- 172.21.15.67:0/3962349327 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f804c0a2800 0x7f804c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:56.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.482+0000 7f805f400640 1 -- 172.21.15.67:0/3962349327 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f80580f0f60 con 0x7f8068108eb0 2024-03-28T16:07:56.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.482+0000 7f8067400640 1 --2- 172.21.15.67:0/3962349327 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f804c0a2800 0x7f804c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:56.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.482+0000 7f8067400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564661cc70a0 0x564661cc74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:56.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.482+0000 7f8067400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564661cc70a0 0x564661cc74c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.67:42850/0 (socket says 172.21.15.67:42850) 2024-03-28T16:07:56.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.482+0000 7f8067400640 1 -- 172.21.15.67:0/838483320 learned_addr learned my addr 172.21.15.67:0/838483320 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:07:56.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.483+0000 7f8066a00640 1 --2- 172.21.15.67:0/838483320 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564661cc7a90 0x564661ccc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:56.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.483+0000 7f8066000640 1 --2- 172.21.15.67:0/838483320 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564661cccc00 0x564661ccf090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:56.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.483+0000 7f8067400640 1 --2- 172.21.15.67:0/3962349327 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f804c0a2800 0x7f804c0a4cc0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f8060012760 tx=0x7f8060009cb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:56.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.483+0000 7f8067400640 1 -- 172.21.15.67:0/838483320 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564661cc7a90 msgr2=0x564661ccc590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:56.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.483+0000 7f8067400640 1 --2- 172.21.15.67:0/838483320 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564661cc7a90 0x564661ccc590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:56.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.483+0000 7f8067400640 1 -- 172.21.15.67:0/838483320 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564661cccc00 msgr2=0x564661ccf090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:56.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.483+0000 7f8067400640 1 --2- 172.21.15.67:0/838483320 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564661cccc00 0x564661ccf090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:56.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.483+0000 7f8067400640 1 -- 172.21.15.67:0/838483320 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564661a097e0 con 0x564661cc70a0 2024-03-28T16:07:56.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.483+0000 7f8066000640 1 --2- 172.21.15.67:0/838483320 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564661cccc00 0x564661ccf090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T16:07:56.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.484+0000 7f8067400640 1 --2- 172.21.15.67:0/838483320 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564661cc70a0 0x564661cc74c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f806000f8f0 tx=0x7f806001b000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bfc3778ff4b21184 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:56.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.484+0000 7f8057e00640 1 -- 172.21.15.67:0/838483320 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f806000fdb0 con 0x564661cc70a0 2024-03-28T16:07:56.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.485+0000 7f8057e00640 1 -- 172.21.15.67:0/838483320 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f806000d7d0 con 0x564661cc70a0 2024-03-28T16:07:56.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.485+0000 7f8057e00640 1 -- 172.21.15.67:0/838483320 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8060005420 con 0x564661cc70a0 2024-03-28T16:07:56.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.486+0000 7f8070322740 1 -- 172.21.15.67:0/838483320 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564661cc70a0 msgr2=0x564661cc74c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:56.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.486+0000 7f8070322740 1 --2- 172.21.15.67:0/838483320 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564661cc70a0 0x564661cc74c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f806000f8f0 tx=0x7f806001b000 comp rx=0 tx=0).stop 2024-03-28T16:07:56.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.487+0000 7f8070322740 1 -- 172.21.15.67:0/838483320 shutdown_connections 2024-03-28T16:07:56.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.487+0000 7f8070322740 1 --2- 172.21.15.67:0/838483320 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564661cccc00 0x564661ccf090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:56.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.487+0000 7f8070322740 1 --2- 172.21.15.67:0/838483320 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564661cc7a90 0x564661ccc590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:56.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.487+0000 7f8070322740 1 --2- 172.21.15.67:0/838483320 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564661cc70a0 0x564661cc74c0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:56.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.487+0000 7f8070322740 1 -- 172.21.15.67:0/838483320 >> 172.21.15.67:0/838483320 conn(0x564661cc2610 msgr2=0x564661cc2a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:56.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.487+0000 7f8070322740 1 -- 172.21.15.67:0/838483320 shutdown_connections 2024-03-28T16:07:56.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.487+0000 7f8070322740 1 -- 172.21.15.67:0/838483320 wait complete. 2024-03-28T16:07:56.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.487+0000 7f8070322740 1 -- start start 2024-03-28T16:07:56.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.488+0000 7f8070322740 1 Processor -- start 2024-03-28T16:07:56.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.488+0000 7f8070322740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564661cc70a0 0x564661cd12b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:56.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.488+0000 7f8070322740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564661cc7a90 0x564661cd17f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:56.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.488+0000 7f8070322740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564661cccc00 0x564661cd8830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:56.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.488+0000 7f8070322740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x564661ccfb50 con 0x564661cc70a0 2024-03-28T16:07:56.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.488+0000 7f8070322740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x564661be8eb0 con 0x564661cccc00 2024-03-28T16:07:56.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.488+0000 7f8070322740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x564661ccfcd0 con 0x564661cc7a90 2024-03-28T16:07:56.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.489+0000 7f8070322740 10 client.0 _ll_register_callbacks cb 0x564661bdd150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:07:56.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.489+0000 7f8066a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564661cc70a0 0x564661cd12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:56.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.489+0000 7f8066a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564661cc70a0 0x564661cd12b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59280/0 (socket says 172.21.15.67:59280) 2024-03-28T16:07:56.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.489+0000 7f8066a00640 1 -- 172.21.15.67:0/3500675835 learned_addr learned my addr 172.21.15.67:0/3500675835 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:07:56.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.489+0000 7f8067400640 1 --2- 172.21.15.67:0/3500675835 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564661cccc00 0x564661cd8830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:56.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.489+0000 7f8066000640 1 --2- 172.21.15.67:0/3500675835 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564661cc7a90 0x564661cd17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:56.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.490+0000 7f8066a00640 1 -- 172.21.15.67:0/3500675835 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564661cc7a90 msgr2=0x564661cd17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:56.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.490+0000 7f8066a00640 1 --2- 172.21.15.67:0/3500675835 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564661cc7a90 0x564661cd17f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:56.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.490+0000 7f8066a00640 1 -- 172.21.15.67:0/3500675835 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564661cccc00 msgr2=0x564661cd8830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:56.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.490+0000 7f8066a00640 1 --2- 172.21.15.67:0/3500675835 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564661cccc00 0x564661cd8830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:56.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.490+0000 7f8066a00640 1 -- 172.21.15.67:0/3500675835 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564661a69f70 con 0x564661cc70a0 2024-03-28T16:07:56.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.490+0000 7f8066a00640 1 --2- 172.21.15.67:0/3500675835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564661cc70a0 0x564661cd12b0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f8050010770 tx=0x7f8050010c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=991c7111d0174dcd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:56.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.490+0000 7f8054c00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:07:56.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.490+0000 7f8054c00640 1 -- 172.21.15.67:0/3500675835 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f805000ed40 con 0x564661cc70a0 2024-03-28T16:07:56.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.490+0000 7f8054c00640 1 -- 172.21.15.67:0/3500675835 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f8050005020 con 0x564661cc70a0 2024-03-28T16:07:56.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.490+0000 7f8070322740 1 -- 172.21.15.67:0/3500675835 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f806000d7d0 con 0x564661cc70a0 2024-03-28T16:07:56.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.491+0000 7f8054c00640 1 -- 172.21.15.67:0/3500675835 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8050005220 con 0x564661cc70a0 2024-03-28T16:07:56.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.491+0000 7f8070322740 10 client.15924 fetch_fsmap learned FSMap version 19 2024-03-28T16:07:56.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.491+0000 7f8070322740 1 -- 172.21.15.67:0/3500675835 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x564661901170 con 0x564661cc70a0 2024-03-28T16:07:56.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.491+0000 7f8054c00640 1 -- 172.21.15.67:0/3500675835 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f806000d7d0 con 0x564661cc70a0 2024-03-28T16:07:56.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.492+0000 7f8054c00640 1 -- 172.21.15.67:0/3500675835 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f80500049a0 con 0x564661cc70a0 2024-03-28T16:07:56.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.492+0000 7f8070322740 10 client.15924 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:07:56.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.492+0000 7f8070322740 10 client.15924 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:07:56.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.492+0000 7f8070322740 20 client.15924 populate_metadata read hostname 'smithi067' 2024-03-28T16:07:56.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.492+0000 7f8070322740 1 --2- 172.21.15.67:0/3500675835 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x564661cdd180 0x564661cdf600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:56.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.492+0000 7f8070322740 4 client.15924 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:07:56.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.492+0000 7f8070322740 1 -- 172.21.15.67:0/3500675835 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564661c3a3f0 con 0x564661cdd180 2024-03-28T16:07:56.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.493+0000 7f8066000640 1 --2- 172.21.15.67:0/3500675835 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x564661cdd180 0x564661cdf600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:56.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.494+0000 7f8066000640 1 --2- 172.21.15.67:0/3500675835 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x564661cdd180 0x564661cdf600 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cfaf8e9adebbef54 server_cookie=39d495ed002c5c20 in_seq=0 out_seq=0 2024-03-28T16:07:56.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.494+0000 7f8054c00640 10 client.15924 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:07:56.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.496+0000 7f8054c00640 1 -- 172.21.15.67:0/3500675835 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x564661c3a3f0 con 0x564661cdd180 2024-03-28T16:07:56.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.496+0000 7f8054c00640 10 client.15924 handle_command_reply: tid=0 2024-03-28T16:07:56.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.496+0000 7f8070322740 1 client.15924 shutdown 2024-03-28T16:07:56.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.496+0000 7f8070322740 1 -- 172.21.15.67:0/3500675835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564661cc70a0 msgr2=0x564661cd12b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:56.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.496+0000 7f8070322740 1 --2- 172.21.15.67:0/3500675835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564661cc70a0 0x564661cd12b0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f8050010770 tx=0x7f8050010c30 comp rx=0 tx=0).stop 2024-03-28T16:07:56.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.496+0000 7f8070322740 1 -- 172.21.15.67:0/3500675835 shutdown_connections 2024-03-28T16:07:56.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.496+0000 7f8070322740 1 --2- 172.21.15.67:0/3500675835 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x564661cdd180 0x564661cdf600 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:56.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.496+0000 7f8070322740 1 --2- 172.21.15.67:0/3500675835 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564661cccc00 0x564661cd8830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:56.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.496+0000 7f8070322740 1 --2- 172.21.15.67:0/3500675835 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564661cc7a90 0x564661cd17f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:56.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.496+0000 7f8070322740 1 --2- 172.21.15.67:0/3500675835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564661cc70a0 0x564661cd12b0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:56.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.496+0000 7f8054c00640 0 client.15924 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:07:56.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.496+0000 7f8070322740 1 -- 172.21.15.67:0/3500675835 >> 172.21.15.67:0/3500675835 conn(0x564661cc2610 msgr2=0x564661cc1830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:56.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.496+0000 7f8070322740 1 -- 172.21.15.67:0/3500675835 shutdown_connections 2024-03-28T16:07:56.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.496+0000 7f8070322740 1 -- 172.21.15.67:0/3500675835 wait complete. 2024-03-28T16:07:56.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.497+0000 7f8070322740 20 client.15924 trim_cache size 0 max 16384 2024-03-28T16:07:56.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.521+0000 7f8070322740 1 Processor -- start 2024-03-28T16:07:56.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.522+0000 7f8070322740 1 -- start start 2024-03-28T16:07:56.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.522+0000 7f8070322740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564661cc7a90 0x564661cd0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:56.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.522+0000 7f8070322740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564661cccc00 0x564661cd87c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:56.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.522+0000 7f8070322740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564661cc2e50 0x564661cd0e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:56.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.522+0000 7f8070322740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80681acf90 con 0x564661cc7a90 2024-03-28T16:07:56.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.522+0000 7f8070322740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80681af550 con 0x564661cccc00 2024-03-28T16:07:56.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.522+0000 7f8070322740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80681ad590 con 0x564661cc2e50 2024-03-28T16:07:56.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.523+0000 7f8066a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564661cc7a90 0x564661cd0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:56.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.523+0000 7f8066000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564661cccc00 0x564661cd87c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:56.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.523+0000 7f8066000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564661cccc00 0x564661cd87c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.67:36832/0 (socket says 172.21.15.67:36832) 2024-03-28T16:07:56.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.523+0000 7f8066000640 1 -- 172.21.15.67:0/2540732341 learned_addr learned my addr 172.21.15.67:0/2540732341 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:07:56.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.523+0000 7f8066a00640 1 -- 172.21.15.67:0/2540732341 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564661cc2e50 msgr2=0x564661cd0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:56.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.524+0000 7f8066a00640 1 --2- 172.21.15.67:0/2540732341 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564661cc2e50 0x564661cd0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:56.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.524+0000 7f8066a00640 1 -- 172.21.15.67:0/2540732341 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564661cccc00 msgr2=0x564661cd87c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:56.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.524+0000 7f8066a00640 1 --2- 172.21.15.67:0/2540732341 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564661cccc00 0x564661cd87c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:56.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.524+0000 7f8066a00640 1 -- 172.21.15.67:0/2540732341 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564661ccbf20 con 0x564661cc7a90 2024-03-28T16:07:56.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.524+0000 7f8066000640 1 --2- 172.21.15.67:0/2540732341 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564661cccc00 0x564661cd87c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T16:07:56.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.524+0000 7f8066a00640 1 --2- 172.21.15.67:0/2540732341 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564661cc7a90 0x564661cd0900 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f8050010740 tx=0x7f805000bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=388c139086815692 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:56.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.525+0000 7f8057400640 1 -- 172.21.15.67:0/2540732341 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8050029040 con 0x564661cc7a90 2024-03-28T16:07:56.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.525+0000 7f8057400640 1 -- 172.21.15.67:0/2540732341 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f80500053c0 con 0x564661cc7a90 2024-03-28T16:07:56.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.525+0000 7f8057400640 1 -- 172.21.15.67:0/2540732341 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8050004680 con 0x564661cc7a90 2024-03-28T16:07:56.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.526+0000 7f8070322740 1 -- 172.21.15.67:0/2540732341 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564661cc7a90 msgr2=0x564661cd0900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:56.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.526+0000 7f8070322740 1 --2- 172.21.15.67:0/2540732341 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564661cc7a90 0x564661cd0900 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f8050010740 tx=0x7f805000bf30 comp rx=0 tx=0).stop 2024-03-28T16:07:56.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.526+0000 7f8070322740 1 -- 172.21.15.67:0/2540732341 shutdown_connections 2024-03-28T16:07:56.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.526+0000 7f8070322740 1 --2- 172.21.15.67:0/2540732341 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564661cc2e50 0x564661cd0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:56.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.526+0000 7f8070322740 1 --2- 172.21.15.67:0/2540732341 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564661cccc00 0x564661cd87c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:56.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.526+0000 7f8070322740 1 --2- 172.21.15.67:0/2540732341 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564661cc7a90 0x564661cd0900 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:56.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.526+0000 7f8070322740 1 -- 172.21.15.67:0/2540732341 >> 172.21.15.67:0/2540732341 conn(0x564661cc2610 msgr2=0x564661cc6100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:56.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.526+0000 7f8070322740 1 -- 172.21.15.67:0/2540732341 shutdown_connections 2024-03-28T16:07:56.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.526+0000 7f8070322740 1 -- 172.21.15.67:0/2540732341 wait complete. 2024-03-28T16:07:56.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.527+0000 7f8070322740 1 -- start start 2024-03-28T16:07:56.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.527+0000 7f8070322740 1 Processor -- start 2024-03-28T16:07:56.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.527+0000 7f8070322740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564661cc2e50 0x564661cd84f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:56.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.527+0000 7f8070322740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564661cc7a90 0x564661cd6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:56.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.528+0000 7f8070322740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564661cccc00 0x564661cd7100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:56.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.528+0000 7f8070322740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80681adbd0 con 0x564661cccc00 2024-03-28T16:07:56.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.528+0000 7f8070322740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80681aeef0 con 0x564661cc7a90 2024-03-28T16:07:56.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.528+0000 7f8070322740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80681ae230 con 0x564661cc2e50 2024-03-28T16:07:56.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.528+0000 7f8070322740 10 client.0 _ll_register_callbacks cb 0x7f805000b650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:07:56.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.528+0000 7f8067400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564661cccc00 0x564661cd7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:56.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.528+0000 7f8067400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564661cccc00 0x564661cd7100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59302/0 (socket says 172.21.15.67:59302) 2024-03-28T16:07:56.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.528+0000 7f8067400640 1 -- 172.21.15.67:0/3337557928 learned_addr learned my addr 172.21.15.67:0/3337557928 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:07:56.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.528+0000 7f8066000640 1 --2- 172.21.15.67:0/3337557928 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564661cc7a90 0x564661cd6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:56.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.529+0000 7f8067400640 1 -- 172.21.15.67:0/3337557928 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564661cc2e50 msgr2=0x564661cd84f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:56.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.529+0000 7f8067400640 1 --2- 172.21.15.67:0/3337557928 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564661cc2e50 0x564661cd84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:56.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.529+0000 7f8067400640 1 -- 172.21.15.67:0/3337557928 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564661cc7a90 msgr2=0x564661cd6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:56.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.529+0000 7f8067400640 1 --2- 172.21.15.67:0/3337557928 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564661cc7a90 0x564661cd6bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:56.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.529+0000 7f8067400640 1 -- 172.21.15.67:0/3337557928 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564661cd76c0 con 0x564661cccc00 2024-03-28T16:07:56.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.530+0000 7f8054c00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:07:56.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.530+0000 7f8067400640 1 --2- 172.21.15.67:0/3337557928 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564661cccc00 0x564661cd7100 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f806000f8f0 tx=0x7f806000f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f518dd9292a63af server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:56.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.530+0000 7f8054c00640 1 -- 172.21.15.67:0/3337557928 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8060004b20 con 0x564661cccc00 2024-03-28T16:07:56.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.530+0000 7f8054c00640 1 -- 172.21.15.67:0/3337557928 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f8060004cc0 con 0x564661cccc00 2024-03-28T16:07:56.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.530+0000 7f8070322740 1 -- 172.21.15.67:0/3337557928 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f80500053c0 con 0x564661cccc00 2024-03-28T16:07:56.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.531+0000 7f8054c00640 1 -- 172.21.15.67:0/3337557928 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f806000d7d0 con 0x564661cccc00 2024-03-28T16:07:56.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.531+0000 7f8070322740 10 client.15930 fetch_fsmap learned FSMap version 19 2024-03-28T16:07:56.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.531+0000 7f8070322740 1 -- 172.21.15.67:0/3337557928 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x564661d398e0 con 0x564661cccc00 2024-03-28T16:07:56.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.531+0000 7f8054c00640 1 -- 172.21.15.67:0/3337557928 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f80500053c0 con 0x564661cccc00 2024-03-28T16:07:56.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.531+0000 7f8054c00640 1 -- 172.21.15.67:0/3337557928 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f8060037030 con 0x564661cccc00 2024-03-28T16:07:56.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.531+0000 7f8070322740 10 client.15930 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:07:56.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.531+0000 7f8070322740 10 client.15930 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:07:56.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.531+0000 7f8070322740 20 client.15930 populate_metadata read hostname 'smithi067' 2024-03-28T16:07:56.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.531+0000 7f8070322740 1 --2- 172.21.15.67:0/3337557928 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x564661d39fd0 0x564661d3c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:56.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.531+0000 7f8070322740 4 client.15930 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T16:07:56.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.531+0000 7f8070322740 1 -- 172.21.15.67:0/3337557928 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x564661c34740 con 0x564661d39fd0 2024-03-28T16:07:56.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.533+0000 7f8066a00640 1 --2- 172.21.15.67:0/3337557928 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x564661d39fd0 0x564661d3c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:56.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.534+0000 7f8066a00640 1 --2- 172.21.15.67:0/3337557928 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x564661d39fd0 0x564661d3c3f0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b68bbd3cc63db312 server_cookie=72c36b33ac5a1559 in_seq=0 out_seq=0 2024-03-28T16:07:56.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.534+0000 7f8054c00640 10 client.15930 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:07:56.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.536+0000 7f8054c00640 1 -- 172.21.15.67:0/3337557928 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x564661c34740 con 0x564661d39fd0 2024-03-28T16:07:56.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.536+0000 7f8054c00640 10 client.15930 handle_command_reply: tid=0 2024-03-28T16:07:56.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.536+0000 7f8070322740 1 client.15930 shutdown 2024-03-28T16:07:56.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.536+0000 7f8070322740 1 -- 172.21.15.67:0/3337557928 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564661cccc00 msgr2=0x564661cd7100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:56.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.536+0000 7f8070322740 1 --2- 172.21.15.67:0/3337557928 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564661cccc00 0x564661cd7100 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f806000f8f0 tx=0x7f806000f740 comp rx=0 tx=0).stop 2024-03-28T16:07:56.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.536+0000 7f8070322740 1 -- 172.21.15.67:0/3337557928 shutdown_connections 2024-03-28T16:07:56.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.536+0000 7f8070322740 1 --2- 172.21.15.67:0/3337557928 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x564661d39fd0 0x564661d3c3f0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:56.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.536+0000 7f8070322740 1 --2- 172.21.15.67:0/3337557928 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564661cccc00 0x564661cd7100 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:56.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.536+0000 7f8070322740 1 --2- 172.21.15.67:0/3337557928 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564661cc7a90 0x564661cd6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:56.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.536+0000 7f8070322740 1 --2- 172.21.15.67:0/3337557928 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564661cc2e50 0x564661cd84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:56.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.536+0000 7f8070322740 1 -- 172.21.15.67:0/3337557928 >> 172.21.15.67:0/3337557928 conn(0x564661cc2610 msgr2=0x564661cc8320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:56.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.536+0000 7f8054c00640 0 client.15930 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:07:56.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.537+0000 7f8070322740 1 -- 172.21.15.67:0/3337557928 shutdown_connections 2024-03-28T16:07:56.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.537+0000 7f8070322740 1 -- 172.21.15.67:0/3337557928 wait complete. 2024-03-28T16:07:56.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.537+0000 7f8070322740 20 client.15930 trim_cache size 0 max 16384 2024-03-28T16:07:56.539 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-28T16:07:56.539 INFO:teuthology.orchestra.run.smithi067.stdout: "return_code": 0, 2024-03-28T16:07:56.539 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_tag": "1af09c16-5057-44a7-abc2-47e83d7c6534", 2024-03-28T16:07:56.539 INFO:teuthology.orchestra.run.smithi067.stdout: "mode": "asynchronous" 2024-03-28T16:07:56.539 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-28T16:07:56.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.537+0000 7f806da00640 1 -- 172.21.15.67:0/3962349327 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f804c0a2800 msgr2=0x7f804c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:56.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.537+0000 7f806da00640 1 --2- 172.21.15.67:0/3962349327 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f804c0a2800 0x7f804c0a4cc0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f8060012760 tx=0x7f8060009cb0 comp rx=0 tx=0).stop 2024-03-28T16:07:56.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.537+0000 7f806da00640 1 -- 172.21.15.67:0/3962349327 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8068108eb0 msgr2=0x7f806819e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:56.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.537+0000 7f806da00640 1 --2- 172.21.15.67:0/3962349327 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8068108eb0 0x7f806819e470 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f8058000c00 tx=0x7f80580042f0 comp rx=0 tx=0).stop 2024-03-28T16:07:56.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.538+0000 7f806da00640 1 -- 172.21.15.67:0/3962349327 shutdown_connections 2024-03-28T16:07:56.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.538+0000 7f806da00640 1 --2- 172.21.15.67:0/3962349327 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f804c0a2800 0x7f804c0a4cc0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:56.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.538+0000 7f806da00640 1 --2- 172.21.15.67:0/3962349327 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8068109870 0x7f80681a54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:56.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.538+0000 7f806da00640 1 --2- 172.21.15.67:0/3962349327 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8068108eb0 0x7f806819e470 secure :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f8058000c00 tx=0x7f80580042f0 comp rx=0 tx=0).stop 2024-03-28T16:07:56.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.538+0000 7f806da00640 1 --2- 172.21.15.67:0/3962349327 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8068107cc0 0x7f806819df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:56.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.538+0000 7f806da00640 1 -- 172.21.15.67:0/3962349327 >> 172.21.15.67:0/3962349327 conn(0x7f8068076c60 msgr2=0x7f806810c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:56.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.538+0000 7f806da00640 1 -- 172.21.15.67:0/3962349327 shutdown_connections 2024-03-28T16:07:56.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.538+0000 7f806da00640 1 -- 172.21.15.67:0/3962349327 wait complete. 2024-03-28T16:07:56.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:07:56 smithi150 ceph-mon[34825]: pgmap v570: 97 pgs: 97 active+clean; 5.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 162 MiB/s wr, 43 op/s 2024-03-28T16:07:56.561 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:07:56 smithi067 ceph-mon[31707]: pgmap v570: 97 pgs: 97 active+clean; 5.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 162 MiB/s wr, 43 op/s 2024-03-28T16:07:56.563 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T16:07:56.563 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T16:07:56.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.828+0000 7f1b41400640 1 -- 172.21.15.67:0/4167636134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b3c1038e0 msgr2=0x7f1b3c103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:56.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.828+0000 7f1b41400640 1 --2- 172.21.15.67:0/4167636134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b3c1038e0 0x7f1b3c103d00 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c009900 tx=0x7f1b2c02bc30 comp rx=0 tx=0).stop 2024-03-28T16:07:56.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.829+0000 7f1b41400640 1 -- 172.21.15.67:0/4167636134 shutdown_connections 2024-03-28T16:07:56.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.829+0000 7f1b41400640 1 --2- 172.21.15.67:0/4167636134 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b3c105490 0x7f1b3c10bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:56.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.829+0000 7f1b41400640 1 --2- 172.21.15.67:0/4167636134 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1b3c104ad0 0x7f1b3c104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:56.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.829+0000 7f1b41400640 1 --2- 172.21.15.67:0/4167636134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b3c1038e0 0x7f1b3c103d00 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:56.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.829+0000 7f1b41400640 1 -- 172.21.15.67:0/4167636134 >> 172.21.15.67:0/4167636134 conn(0x7f1b3c0fee50 msgr2=0x7f1b3c101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:56.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.829+0000 7f1b41400640 1 -- 172.21.15.67:0/4167636134 shutdown_connections 2024-03-28T16:07:56.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.829+0000 7f1b41400640 1 -- 172.21.15.67:0/4167636134 wait complete. 2024-03-28T16:07:56.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.830+0000 7f1b41400640 1 Processor -- start 2024-03-28T16:07:56.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.830+0000 7f1b41400640 1 -- start start 2024-03-28T16:07:56.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.830+0000 7f1b41400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b3c1038e0 0x7f1b3c199b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:56.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.831+0000 7f1b41400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b3c104ad0 0x7f1b3c19a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:56.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.831+0000 7f1b41400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1b3c105490 0x7f1b3c1a1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:56.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.831+0000 7f1b41400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b3c10f450 con 0x7f1b3c104ad0 2024-03-28T16:07:56.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.831+0000 7f1b41400640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b3c10f2d0 con 0x7f1b3c1038e0 2024-03-28T16:07:56.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.831+0000 7f1b41400640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b3c10f5d0 con 0x7f1b3c105490 2024-03-28T16:07:56.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.831+0000 7f1b3a800640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b3c1038e0 0x7f1b3c199b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:56.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.831+0000 7f1b3a800640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b3c1038e0 0x7f1b3c199b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.67:36866/0 (socket says 172.21.15.67:36866) 2024-03-28T16:07:56.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.831+0000 7f1b3a800640 1 -- 172.21.15.67:0/1670506272 learned_addr learned my addr 172.21.15.67:0/1670506272 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:07:56.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.831+0000 7f1b39e00640 1 --2- 172.21.15.67:0/1670506272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b3c104ad0 0x7f1b3c19a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:56.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.832+0000 7f1b3b200640 1 --2- 172.21.15.67:0/1670506272 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1b3c105490 0x7f1b3c1a1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:56.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.832+0000 7f1b3a800640 1 -- 172.21.15.67:0/1670506272 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1b3c105490 msgr2=0x7f1b3c1a1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:56.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.832+0000 7f1b3a800640 1 --2- 172.21.15.67:0/1670506272 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1b3c105490 0x7f1b3c1a1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:56.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.832+0000 7f1b3a800640 1 -- 172.21.15.67:0/1670506272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b3c104ad0 msgr2=0x7f1b3c19a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:56.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.832+0000 7f1b3a800640 1 --2- 172.21.15.67:0/1670506272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b3c104ad0 0x7f1b3c19a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:56.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.832+0000 7f1b3a800640 1 -- 172.21.15.67:0/1670506272 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b3c1a17a0 con 0x7f1b3c1038e0 2024-03-28T16:07:56.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.832+0000 7f1b39e00640 1 --2- 172.21.15.67:0/1670506272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b3c104ad0 0x7f1b3c19a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T16:07:56.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.833+0000 7f1b3a800640 1 --2- 172.21.15.67:0/1670506272 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b3c1038e0 0x7f1b3c199b80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c000c00 tx=0x7f1b2c0040c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:56.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.833+0000 7f1b33400640 1 -- 172.21.15.67:0/1670506272 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b2c043020 con 0x7f1b3c1038e0 2024-03-28T16:07:56.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.834+0000 7f1b41400640 1 -- 172.21.15.67:0/1670506272 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b3c1a1a30 con 0x7f1b3c1038e0 2024-03-28T16:07:56.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.834+0000 7f1b41400640 1 -- 172.21.15.67:0/1670506272 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b3c1a1fd0 con 0x7f1b3c1038e0 2024-03-28T16:07:56.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.834+0000 7f1b33400640 1 -- 172.21.15.67:0/1670506272 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1b2c0044c0 con 0x7f1b3c1038e0 2024-03-28T16:07:56.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.834+0000 7f1b33400640 1 -- 172.21.15.67:0/1670506272 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b2c003c00 con 0x7f1b3c1038e0 2024-03-28T16:07:56.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.839+0000 7f1b4400f740 1 Processor -- start 2024-03-28T16:07:56.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.839+0000 7f1b4400f740 1 -- start start 2024-03-28T16:07:56.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.840+0000 7f1b4400f740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ab6e676e80 0x55ab6e6772a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:56.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.840+0000 7f1b4400f740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ab6e677870 0x55ab6e67c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:56.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.840+0000 7f1b4400f740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ab6e67c9e0 0x55ab6e67ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:56.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.840+0000 7f1b4400f740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab6e5dc250 con 0x55ab6e677870 2024-03-28T16:07:56.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.840+0000 7f1b4400f740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab6e584a30 con 0x55ab6e676e80 2024-03-28T16:07:56.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.840+0000 7f1b4400f740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab6e5d82f0 con 0x55ab6e67c9e0 2024-03-28T16:07:56.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.840+0000 7f1b33400640 1 -- 172.21.15.67:0/1670506272 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f1b2c046020 con 0x7f1b3c1038e0 2024-03-28T16:07:56.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.841+0000 7f1b3a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ab6e677870 0x55ab6e67c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:56.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.841+0000 7f1b3a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ab6e677870 0x55ab6e67c370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59328/0 (socket says 172.21.15.67:59328) 2024-03-28T16:07:56.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.841+0000 7f1b3a800640 1 -- 172.21.15.67:0/2387305214 learned_addr learned my addr 172.21.15.67:0/2387305214 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:07:56.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.841+0000 7f1b39e00640 1 --2- 172.21.15.67:0/2387305214 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ab6e67c9e0 0x55ab6e67ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:56.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.841+0000 7f1b3b200640 1 --2- 172.21.15.67:0/2387305214 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ab6e676e80 0x55ab6e6772a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:56.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.841+0000 7f1b3a800640 1 -- 172.21.15.67:0/2387305214 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ab6e67c9e0 msgr2=0x55ab6e67ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:56.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.841+0000 7f1b3a800640 1 --2- 172.21.15.67:0/2387305214 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ab6e67c9e0 0x55ab6e67ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:56.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.841+0000 7f1b3a800640 1 -- 172.21.15.67:0/2387305214 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ab6e676e80 msgr2=0x55ab6e6772a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:56.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.841+0000 7f1b3a800640 1 --2- 172.21.15.67:0/2387305214 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ab6e676e80 0x55ab6e6772a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:56.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.841+0000 7f1b3a800640 1 -- 172.21.15.67:0/2387305214 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ab6e56f690 con 0x55ab6e677870 2024-03-28T16:07:56.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.842+0000 7f1b3a800640 1 --2- 172.21.15.67:0/2387305214 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ab6e677870 0x55ab6e67c370 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c0f4ca0 tx=0x7f1b2c0f4cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=309ee863dedb0e14 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:56.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.842+0000 7f1b27400640 1 -- 172.21.15.67:0/2387305214 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b2c0f9d80 con 0x55ab6e677870 2024-03-28T16:07:56.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.842+0000 7f1b27400640 1 -- 172.21.15.67:0/2387305214 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1b2c0fe070 con 0x55ab6e677870 2024-03-28T16:07:56.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.842+0000 7f1b27400640 1 -- 172.21.15.67:0/2387305214 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b2c0f7650 con 0x55ab6e677870 2024-03-28T16:07:56.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.843+0000 7f1b33400640 1 --2- 172.21.15.67:0/1670506272 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f1b200a2880 0x7f1b200a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:56.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.843+0000 7f1b4400f740 1 -- 172.21.15.67:0/2387305214 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ab6e677870 msgr2=0x55ab6e67c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:56.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.843+0000 7f1b4400f740 1 --2- 172.21.15.67:0/2387305214 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ab6e677870 0x55ab6e67c370 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c0f4ca0 tx=0x7f1b2c0f4cd0 comp rx=0 tx=0).stop 2024-03-28T16:07:56.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.843+0000 7f1b33400640 1 -- 172.21.15.67:0/1670506272 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f1b2c0f0c90 con 0x7f1b3c1038e0 2024-03-28T16:07:56.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.843+0000 7f1b4400f740 1 -- 172.21.15.67:0/2387305214 shutdown_connections 2024-03-28T16:07:56.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.843+0000 7f1b4400f740 1 --2- 172.21.15.67:0/2387305214 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ab6e67c9e0 0x55ab6e67ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:56.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.843+0000 7f1b4400f740 1 --2- 172.21.15.67:0/2387305214 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ab6e677870 0x55ab6e67c370 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:56.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.843+0000 7f1b4400f740 1 --2- 172.21.15.67:0/2387305214 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ab6e676e80 0x55ab6e6772a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:56.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.843+0000 7f1b4400f740 1 -- 172.21.15.67:0/2387305214 >> 172.21.15.67:0/2387305214 conn(0x55ab6e63f200 msgr2=0x55ab6e6727f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:56.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.844+0000 7f1b4400f740 1 -- 172.21.15.67:0/2387305214 shutdown_connections 2024-03-28T16:07:56.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.844+0000 7f1b3b200640 1 --2- 172.21.15.67:0/1670506272 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f1b200a2880 0x7f1b200a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:56.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.844+0000 7f1b4400f740 1 -- 172.21.15.67:0/2387305214 wait complete. 2024-03-28T16:07:56.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.844+0000 7f1b4400f740 1 -- start start 2024-03-28T16:07:56.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.845+0000 7f1b4400f740 1 Processor -- start 2024-03-28T16:07:56.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.845+0000 7f1b3b200640 1 --2- 172.21.15.67:0/1670506272 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f1b200a2880 0x7f1b200a4d40 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x55ab6e6780d0 tx=0x7f1b34018040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:56.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.845+0000 7f1b4400f740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ab6e676e80 0x55ab6e680cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:56.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.845+0000 7f1b4400f740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ab6e67c9e0 0x55ab6e681210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:56.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.845+0000 7f1b4400f740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ab6e686240 0x55ab6e688630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:56.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.845+0000 7f1b4400f740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab6e67f750 con 0x55ab6e67c9e0 2024-03-28T16:07:56.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.845+0000 7f1b4400f740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab6e597e80 con 0x55ab6e686240 2024-03-28T16:07:56.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.845+0000 7f1b4400f740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab6e67f8d0 con 0x55ab6e676e80 2024-03-28T16:07:56.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.845+0000 7f1b4400f740 10 client.0 _ll_register_callbacks cb 0x55ab6e5aa130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:07:56.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.845+0000 7f1b39e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ab6e67c9e0 0x55ab6e681210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:56.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.845+0000 7f1b39e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ab6e67c9e0 0x55ab6e681210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59340/0 (socket says 172.21.15.67:59340) 2024-03-28T16:07:56.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.845+0000 7f1b39e00640 1 -- 172.21.15.67:0/1913713105 learned_addr learned my addr 172.21.15.67:0/1913713105 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:07:56.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.846+0000 7f1b3b200640 1 --2- 172.21.15.67:0/1913713105 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ab6e686240 0x55ab6e688630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:56.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.846+0000 7f1b39e00640 1 -- 172.21.15.67:0/1913713105 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ab6e676e80 msgr2=0x55ab6e680cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:56.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.846+0000 7f1b39e00640 1 --2- 172.21.15.67:0/1913713105 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ab6e676e80 0x55ab6e680cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:56.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.846+0000 7f1b39e00640 1 -- 172.21.15.67:0/1913713105 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ab6e686240 msgr2=0x55ab6e688630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:56.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.846+0000 7f1b39e00640 1 --2- 172.21.15.67:0/1913713105 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ab6e686240 0x55ab6e688630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:56.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.846+0000 7f1b39e00640 1 -- 172.21.15.67:0/1913713105 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ab6e3ba7d0 con 0x55ab6e67c9e0 2024-03-28T16:07:56.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.846+0000 7f1b39e00640 1 --2- 172.21.15.67:0/1913713105 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ab6e67c9e0 0x55ab6e681210 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f1b28004450 tx=0x7f1b280039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd59dfadd974156d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:56.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.846+0000 7f1b1be00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:07:56.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.847+0000 7f1b1be00640 1 -- 172.21.15.67:0/1913713105 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b28005020 con 0x55ab6e67c9e0 2024-03-28T16:07:56.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.847+0000 7f1b1be00640 1 -- 172.21.15.67:0/1913713105 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f1b280051c0 con 0x55ab6e67c9e0 2024-03-28T16:07:56.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.847+0000 7f1b4400f740 1 -- 172.21.15.67:0/1913713105 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1b2c0fe070 con 0x55ab6e67c9e0 2024-03-28T16:07:56.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.847+0000 7f1b1be00640 1 -- 172.21.15.67:0/1913713105 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1b280053c0 con 0x55ab6e67c9e0 2024-03-28T16:07:56.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.847+0000 7f1b4400f740 10 client.15942 fetch_fsmap learned FSMap version 19 2024-03-28T16:07:56.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.847+0000 7f1b4400f740 1 -- 172.21.15.67:0/1913713105 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55ab6e2f03f0 con 0x55ab6e67c9e0 2024-03-28T16:07:56.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.848+0000 7f1b1be00640 1 -- 172.21.15.67:0/1913713105 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1b2c0fe070 con 0x55ab6e67c9e0 2024-03-28T16:07:56.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.848+0000 7f1b1be00640 1 -- 172.21.15.67:0/1913713105 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f1b280119e0 con 0x55ab6e67c9e0 2024-03-28T16:07:56.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.848+0000 7f1b4400f740 10 client.15942 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:07:56.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.848+0000 7f1b4400f740 10 client.15942 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:07:56.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.848+0000 7f1b4400f740 20 client.15942 populate_metadata read hostname 'smithi067' 2024-03-28T16:07:56.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.848+0000 7f1b4400f740 1 --2- 172.21.15.67:0/1913713105 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55ab6e68cfc0 0x55ab6e68f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:56.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.848+0000 7f1b4400f740 4 client.15942 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:07:56.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.848+0000 7f1b4400f740 1 -- 172.21.15.67:0/1913713105 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ab6e5e9790 con 0x55ab6e68cfc0 2024-03-28T16:07:56.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.849+0000 7f1b3a800640 1 --2- 172.21.15.67:0/1913713105 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55ab6e68cfc0 0x55ab6e68f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:56.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.849+0000 7f1b1be00640 10 client.15942 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:07:56.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.849+0000 7f1b3a800640 1 --2- 172.21.15.67:0/1913713105 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55ab6e68cfc0 0x55ab6e68f460 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9fc6ccbc6a3e136c server_cookie=8dff67873dd05f38 in_seq=0 out_seq=0 2024-03-28T16:07:56.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.854+0000 7f1b1be00640 1 -- 172.21.15.67:0/1913713105 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55ab6e5e9790 con 0x55ab6e68cfc0 2024-03-28T16:07:56.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.854+0000 7f1b1be00640 10 client.15942 handle_command_reply: tid=0 2024-03-28T16:07:56.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.854+0000 7f1b4400f740 1 client.15942 shutdown 2024-03-28T16:07:56.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.855+0000 7f1b4400f740 1 -- 172.21.15.67:0/1913713105 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ab6e67c9e0 msgr2=0x55ab6e681210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:56.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.855+0000 7f1b4400f740 1 --2- 172.21.15.67:0/1913713105 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ab6e67c9e0 0x55ab6e681210 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f1b28004450 tx=0x7f1b280039c0 comp rx=0 tx=0).stop 2024-03-28T16:07:56.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.855+0000 7f1b4400f740 1 -- 172.21.15.67:0/1913713105 shutdown_connections 2024-03-28T16:07:56.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.855+0000 7f1b4400f740 1 --2- 172.21.15.67:0/1913713105 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55ab6e68cfc0 0x55ab6e68f460 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:56.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.855+0000 7f1b4400f740 1 --2- 172.21.15.67:0/1913713105 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ab6e686240 0x55ab6e688630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:56.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.855+0000 7f1b4400f740 1 --2- 172.21.15.67:0/1913713105 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ab6e67c9e0 0x55ab6e681210 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:56.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.855+0000 7f1b4400f740 1 --2- 172.21.15.67:0/1913713105 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ab6e676e80 0x55ab6e680cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:56.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.855+0000 7f1b4400f740 1 -- 172.21.15.67:0/1913713105 >> 172.21.15.67:0/1913713105 conn(0x55ab6e63f200 msgr2=0x55ab6e671a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:56.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.855+0000 7f1b1be00640 0 client.15942 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:07:56.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.855+0000 7f1b4400f740 1 -- 172.21.15.67:0/1913713105 shutdown_connections 2024-03-28T16:07:56.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.855+0000 7f1b4400f740 1 -- 172.21.15.67:0/1913713105 wait complete. 2024-03-28T16:07:56.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.855+0000 7f1b4400f740 20 client.15942 trim_cache size 0 max 16384 2024-03-28T16:07:56.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.884+0000 7f1b4400f740 1 Processor -- start 2024-03-28T16:07:56.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.885+0000 7f1b4400f740 1 -- start start 2024-03-28T16:07:56.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.885+0000 7f1b4400f740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ab6e676e80 0x55ab6e6802f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:56.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.885+0000 7f1b4400f740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ab6e67c9e0 0x55ab6e678a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:56.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.885+0000 7f1b4400f740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ab6e686240 0x55ab6e693d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:56.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.885+0000 7f1b4400f740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b3c1a48d0 con 0x55ab6e67c9e0 2024-03-28T16:07:56.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.885+0000 7f1b4400f740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b3c1a6e70 con 0x55ab6e676e80 2024-03-28T16:07:56.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.885+0000 7f1b4400f740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b3c1a4ed0 con 0x55ab6e686240 2024-03-28T16:07:56.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.885+0000 7f1b39e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ab6e67c9e0 0x55ab6e678a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:56.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.886+0000 7f1b39e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ab6e67c9e0 0x55ab6e678a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59346/0 (socket says 172.21.15.67:59346) 2024-03-28T16:07:56.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.886+0000 7f1b39e00640 1 -- 172.21.15.67:0/1581254993 learned_addr learned my addr 172.21.15.67:0/1581254993 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:07:56.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.886+0000 7f1b3a800640 1 --2- 172.21.15.67:0/1581254993 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ab6e676e80 0x55ab6e6802f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:56.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.886+0000 7f1b3b200640 1 --2- 172.21.15.67:0/1581254993 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ab6e686240 0x55ab6e693d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:56.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.886+0000 7f1b39e00640 1 -- 172.21.15.67:0/1581254993 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ab6e686240 msgr2=0x55ab6e693d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:56.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.886+0000 7f1b39e00640 1 --2- 172.21.15.67:0/1581254993 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ab6e686240 0x55ab6e693d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:56.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.886+0000 7f1b39e00640 1 -- 172.21.15.67:0/1581254993 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ab6e676e80 msgr2=0x55ab6e6802f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:56.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.886+0000 7f1b39e00640 1 --2- 172.21.15.67:0/1581254993 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ab6e676e80 0x55ab6e6802f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:56.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.886+0000 7f1b39e00640 1 -- 172.21.15.67:0/1581254993 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ab6e2b00f0 con 0x55ab6e67c9e0 2024-03-28T16:07:56.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.887+0000 7f1b39e00640 1 --2- 172.21.15.67:0/1581254993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ab6e67c9e0 0x55ab6e678a20 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f1b28011d50 tx=0x7f1b2800b540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6aa2e1a02c8bcde0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:56.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.887+0000 7f1b26a00640 1 -- 172.21.15.67:0/1581254993 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b28034070 con 0x55ab6e67c9e0 2024-03-28T16:07:56.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.887+0000 7f1b26a00640 1 -- 172.21.15.67:0/1581254993 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1b2801b470 con 0x55ab6e67c9e0 2024-03-28T16:07:56.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.887+0000 7f1b26a00640 1 -- 172.21.15.67:0/1581254993 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b2802b040 con 0x55ab6e67c9e0 2024-03-28T16:07:56.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.888+0000 7f1b4400f740 1 -- 172.21.15.67:0/1581254993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ab6e67c9e0 msgr2=0x55ab6e678a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:56.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.888+0000 7f1b4400f740 1 --2- 172.21.15.67:0/1581254993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ab6e67c9e0 0x55ab6e678a20 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f1b28011d50 tx=0x7f1b2800b540 comp rx=0 tx=0).stop 2024-03-28T16:07:56.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.888+0000 7f1b4400f740 1 -- 172.21.15.67:0/1581254993 shutdown_connections 2024-03-28T16:07:56.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.888+0000 7f1b4400f740 1 --2- 172.21.15.67:0/1581254993 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ab6e686240 0x55ab6e693d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:56.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.888+0000 7f1b4400f740 1 --2- 172.21.15.67:0/1581254993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ab6e67c9e0 0x55ab6e678a20 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:56.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.888+0000 7f1b4400f740 1 --2- 172.21.15.67:0/1581254993 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ab6e676e80 0x55ab6e6802f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:56.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.889+0000 7f1b4400f740 1 -- 172.21.15.67:0/1581254993 >> 172.21.15.67:0/1581254993 conn(0x55ab6e63f200 msgr2=0x55ab6e678310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:56.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.889+0000 7f1b4400f740 1 -- 172.21.15.67:0/1581254993 shutdown_connections 2024-03-28T16:07:56.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.889+0000 7f1b4400f740 1 -- 172.21.15.67:0/1581254993 wait complete. 2024-03-28T16:07:56.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.889+0000 7f1b4400f740 1 -- start start 2024-03-28T16:07:56.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.890+0000 7f1b4400f740 1 Processor -- start 2024-03-28T16:07:56.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.890+0000 7f1b4400f740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ab6e676e80 0x55ab6e69e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:56.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.890+0000 7f1b4400f740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ab6e67c9e0 0x55ab6e69ec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:56.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.890+0000 7f1b4400f740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ab6e686240 0x55ab6e6e7e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:56.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.890+0000 7f1b4400f740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b3c1a54f0 con 0x55ab6e67c9e0 2024-03-28T16:07:56.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.890+0000 7f1b4400f740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b3c1a6810 con 0x55ab6e686240 2024-03-28T16:07:56.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.890+0000 7f1b4400f740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b3c1a5b50 con 0x55ab6e676e80 2024-03-28T16:07:56.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.890+0000 7f1b4400f740 10 client.0 _ll_register_callbacks cb 0x7f1b2800bc60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:07:56.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.891+0000 7f1b39e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ab6e67c9e0 0x55ab6e69ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:56.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.891+0000 7f1b39e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ab6e67c9e0 0x55ab6e69ec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59348/0 (socket says 172.21.15.67:59348) 2024-03-28T16:07:56.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.891+0000 7f1b39e00640 1 -- 172.21.15.67:0/2462329380 learned_addr learned my addr 172.21.15.67:0/2462329380 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:07:56.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.891+0000 7f1b3b200640 1 --2- 172.21.15.67:0/2462329380 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ab6e686240 0x55ab6e6e7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:56.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.891+0000 7f1b3a800640 1 --2- 172.21.15.67:0/2462329380 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ab6e676e80 0x55ab6e69e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:56.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.891+0000 7f1b39e00640 1 -- 172.21.15.67:0/2462329380 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ab6e676e80 msgr2=0x55ab6e69e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:56.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.892+0000 7f1b39e00640 1 --2- 172.21.15.67:0/2462329380 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ab6e676e80 0x55ab6e69e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:56.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.892+0000 7f1b39e00640 1 -- 172.21.15.67:0/2462329380 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ab6e686240 msgr2=0x55ab6e6e7e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:56.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.892+0000 7f1b39e00640 1 --2- 172.21.15.67:0/2462329380 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ab6e686240 0x55ab6e6e7e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:56.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.892+0000 7f1b39e00640 1 -- 172.21.15.67:0/2462329380 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ab6e6e8470 con 0x55ab6e67c9e0 2024-03-28T16:07:56.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.892+0000 7f1b3b200640 1 --2- 172.21.15.67:0/2462329380 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ab6e686240 0x55ab6e6e7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T16:07:56.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.892+0000 7f1b3a800640 1 --2- 172.21.15.67:0/2462329380 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ab6e676e80 0x55ab6e69e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:07:56.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.892+0000 7f1b39e00640 1 --2- 172.21.15.67:0/2462329380 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ab6e67c9e0 0x55ab6e69ec90 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f1b28032040 tx=0x7f1b2800bd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea8cff2ef35a9350 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:56.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.892+0000 7f1b1be00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:07:56.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.892+0000 7f1b1be00640 1 -- 172.21.15.67:0/2462329380 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b2803d060 con 0x55ab6e67c9e0 2024-03-28T16:07:56.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.892+0000 7f1b1be00640 1 -- 172.21.15.67:0/2462329380 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f1b2801b720 con 0x55ab6e67c9e0 2024-03-28T16:07:56.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.893+0000 7f1b4400f740 1 -- 172.21.15.67:0/2462329380 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1b2801b470 con 0x55ab6e67c9e0 2024-03-28T16:07:56.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.893+0000 7f1b1be00640 1 -- 172.21.15.67:0/2462329380 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1b2802b070 con 0x55ab6e67c9e0 2024-03-28T16:07:56.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.893+0000 7f1b4400f740 10 client.15948 fetch_fsmap learned FSMap version 19 2024-03-28T16:07:56.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.893+0000 7f1b4400f740 1 -- 172.21.15.67:0/2462329380 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55ab6e6e9720 con 0x55ab6e67c9e0 2024-03-28T16:07:56.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.893+0000 7f1b1be00640 1 -- 172.21.15.67:0/2462329380 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1b2801b470 con 0x55ab6e67c9e0 2024-03-28T16:07:56.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.893+0000 7f1b1be00640 1 -- 172.21.15.67:0/2462329380 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f1b28037070 con 0x55ab6e67c9e0 2024-03-28T16:07:56.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.894+0000 7f1b4400f740 10 client.15948 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:07:56.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.894+0000 7f1b4400f740 10 client.15948 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:07:56.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.894+0000 7f1b4400f740 20 client.15948 populate_metadata read hostname 'smithi067' 2024-03-28T16:07:56.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.894+0000 7f1b4400f740 1 --2- 172.21.15.67:0/2462329380 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55ab6e6e9f60 0x55ab6e6ec380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:56.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.894+0000 7f1b4400f740 4 client.15948 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T16:07:56.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.894+0000 7f1b4400f740 1 -- 172.21.15.67:0/2462329380 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55ab6e680ac0 con 0x55ab6e6e9f60 2024-03-28T16:07:56.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.894+0000 7f1b3a800640 1 --2- 172.21.15.67:0/2462329380 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55ab6e6e9f60 0x55ab6e6ec380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:56.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.895+0000 7f1b3a800640 1 --2- 172.21.15.67:0/2462329380 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55ab6e6e9f60 0x55ab6e6ec380 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d27813384890288 server_cookie=565ca2f25e2a90c8 in_seq=0 out_seq=0 2024-03-28T16:07:56.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.895+0000 7f1b1be00640 10 client.15948 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:07:56.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.895+0000 7f1b1be00640 1 -- 172.21.15.67:0/2462329380 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55ab6e680ac0 con 0x55ab6e6e9f60 2024-03-28T16:07:56.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.895+0000 7f1b1be00640 10 client.15948 handle_command_reply: tid=0 2024-03-28T16:07:56.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.895+0000 7f1b4400f740 1 client.15948 shutdown 2024-03-28T16:07:56.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.896+0000 7f1b4400f740 1 -- 172.21.15.67:0/2462329380 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ab6e67c9e0 msgr2=0x55ab6e69ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:56.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.896+0000 7f1b4400f740 1 --2- 172.21.15.67:0/2462329380 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ab6e67c9e0 0x55ab6e69ec90 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f1b28032040 tx=0x7f1b2800bd10 comp rx=0 tx=0).stop 2024-03-28T16:07:56.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.896+0000 7f1b4400f740 1 -- 172.21.15.67:0/2462329380 shutdown_connections 2024-03-28T16:07:56.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.896+0000 7f1b4400f740 1 --2- 172.21.15.67:0/2462329380 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55ab6e6e9f60 0x55ab6e6ec380 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:56.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.896+0000 7f1b4400f740 1 --2- 172.21.15.67:0/2462329380 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ab6e686240 0x55ab6e6e7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:56.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.896+0000 7f1b4400f740 1 --2- 172.21.15.67:0/2462329380 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ab6e67c9e0 0x55ab6e69ec90 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:56.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.896+0000 7f1b4400f740 1 --2- 172.21.15.67:0/2462329380 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ab6e676e80 0x55ab6e69e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:56.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.896+0000 7f1b4400f740 1 -- 172.21.15.67:0/2462329380 >> 172.21.15.67:0/2462329380 conn(0x55ab6e63f200 msgr2=0x55ab6e678ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:56.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.896+0000 7f1b1be00640 0 client.15948 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:07:56.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.896+0000 7f1b4400f740 1 -- 172.21.15.67:0/2462329380 shutdown_connections 2024-03-28T16:07:56.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.896+0000 7f1b4400f740 1 -- 172.21.15.67:0/2462329380 wait complete. 2024-03-28T16:07:56.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.896+0000 7f1b4400f740 20 client.15948 trim_cache size 0 max 16384 2024-03-28T16:07:56.899 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-28T16:07:56.899 INFO:teuthology.orchestra.run.smithi067.stdout: "status": "no active scrubs running", 2024-03-28T16:07:56.899 INFO:teuthology.orchestra.run.smithi067.stdout: "scrubs": {} 2024-03-28T16:07:56.900 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-28T16:07:56.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.897+0000 7f1b41400640 1 -- 172.21.15.67:0/1670506272 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f1b200a2880 msgr2=0x7f1b200a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:56.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.897+0000 7f1b41400640 1 --2- 172.21.15.67:0/1670506272 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f1b200a2880 0x7f1b200a4d40 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x55ab6e6780d0 tx=0x7f1b34018040 comp rx=0 tx=0).stop 2024-03-28T16:07:56.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.897+0000 7f1b41400640 1 -- 172.21.15.67:0/1670506272 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b3c1038e0 msgr2=0x7f1b3c199b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:56.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.897+0000 7f1b41400640 1 --2- 172.21.15.67:0/1670506272 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b3c1038e0 0x7f1b3c199b80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c000c00 tx=0x7f1b2c0040c0 comp rx=0 tx=0).stop 2024-03-28T16:07:56.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.897+0000 7f1b41400640 1 -- 172.21.15.67:0/1670506272 shutdown_connections 2024-03-28T16:07:56.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.897+0000 7f1b41400640 1 --2- 172.21.15.67:0/1670506272 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f1b200a2880 0x7f1b200a4d40 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:56.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.897+0000 7f1b41400640 1 --2- 172.21.15.67:0/1670506272 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1b3c105490 0x7f1b3c1a1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:56.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.897+0000 7f1b41400640 1 --2- 172.21.15.67:0/1670506272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b3c104ad0 0x7f1b3c19a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:56.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.897+0000 7f1b41400640 1 --2- 172.21.15.67:0/1670506272 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b3c1038e0 0x7f1b3c199b80 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:56.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.897+0000 7f1b41400640 1 -- 172.21.15.67:0/1670506272 >> 172.21.15.67:0/1670506272 conn(0x7f1b3c0fee50 msgr2=0x7f1b3c107ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:56.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.897+0000 7f1b41400640 1 -- 172.21.15.67:0/1670506272 shutdown_connections 2024-03-28T16:07:56.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:56.898+0000 7f1b41400640 1 -- 172.21.15.67:0/1670506272 wait complete. 2024-03-28T16:07:56.926 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T16:07:56.926 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T16:07:57.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.169+0000 7fdc51200640 1 -- 172.21.15.67:0/3203169189 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc4c102200 msgr2=0x7fdc4c1045f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:57.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.169+0000 7fdc51200640 1 --2- 172.21.15.67:0/3203169189 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc4c102200 0x7fdc4c1045f0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c009900 tx=0x7fdc3c02bc30 comp rx=0 tx=0).stop 2024-03-28T16:07:57.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.169+0000 7fdc51200640 1 -- 172.21.15.67:0/3203169189 shutdown_connections 2024-03-28T16:07:57.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.169+0000 7fdc51200640 1 --2- 172.21.15.67:0/3203169189 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdc4c107460 0x7fdc4c109850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:57.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.169+0000 7fdc51200640 1 --2- 172.21.15.67:0/3203169189 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdc4c104b30 0x7fdc4c106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:57.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.169+0000 7fdc51200640 1 --2- 172.21.15.67:0/3203169189 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc4c102200 0x7fdc4c1045f0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:57.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.169+0000 7fdc51200640 1 -- 172.21.15.67:0/3203169189 >> 172.21.15.67:0/3203169189 conn(0x7fdc4c0fbb30 msgr2=0x7fdc4c0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:57.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.169+0000 7fdc51200640 1 -- 172.21.15.67:0/3203169189 shutdown_connections 2024-03-28T16:07:57.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.169+0000 7fdc51200640 1 -- 172.21.15.67:0/3203169189 wait complete. 2024-03-28T16:07:57.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.170+0000 7fdc51200640 1 Processor -- start 2024-03-28T16:07:57.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.170+0000 7fdc51200640 1 -- start start 2024-03-28T16:07:57.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.171+0000 7fdc51200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdc4c102200 0x7fdc4c199b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:57.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.171+0000 7fdc51200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc4c104b30 0x7fdc4c19a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:57.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.171+0000 7fdc51200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdc4c107460 0x7fdc4c1a10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:57.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.171+0000 7fdc51200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc4c10d270 con 0x7fdc4c104b30 2024-03-28T16:07:57.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.171+0000 7fdc51200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc4c10d0f0 con 0x7fdc4c102200 2024-03-28T16:07:57.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.171+0000 7fdc51200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc4c10d3f0 con 0x7fdc4c107460 2024-03-28T16:07:57.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.171+0000 7fdc43e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc4c104b30 0x7fdc4c19a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:57.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.171+0000 7fdc49e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdc4c102200 0x7fdc4c199b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:57.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.171+0000 7fdc49e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdc4c102200 0x7fdc4c199b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.67:36902/0 (socket says 172.21.15.67:36902) 2024-03-28T16:07:57.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.171+0000 7fdc49e00640 1 -- 172.21.15.67:0/2313112618 learned_addr learned my addr 172.21.15.67:0/2313112618 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:07:57.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.172+0000 7fdc4a800640 1 --2- 172.21.15.67:0/2313112618 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdc4c107460 0x7fdc4c1a10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:57.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.172+0000 7fdc49e00640 1 -- 172.21.15.67:0/2313112618 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdc4c107460 msgr2=0x7fdc4c1a10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:57.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.172+0000 7fdc49e00640 1 --2- 172.21.15.67:0/2313112618 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdc4c107460 0x7fdc4c1a10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:57.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.172+0000 7fdc49e00640 1 -- 172.21.15.67:0/2313112618 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc4c104b30 msgr2=0x7fdc4c19a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:57.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.172+0000 7fdc49e00640 1 --2- 172.21.15.67:0/2313112618 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc4c104b30 0x7fdc4c19a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:57.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.172+0000 7fdc49e00640 1 -- 172.21.15.67:0/2313112618 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc4c1a1740 con 0x7fdc4c102200 2024-03-28T16:07:57.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.172+0000 7fdc4a800640 1 --2- 172.21.15.67:0/2313112618 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdc4c107460 0x7fdc4c1a10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T16:07:57.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.173+0000 7fdc49e00640 1 --2- 172.21.15.67:0/2313112618 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdc4c102200 0x7fdc4c199b20 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c000c00 tx=0x7fdc3c0042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:57.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.173+0000 7fdc42a00640 1 -- 172.21.15.67:0/2313112618 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc3c043020 con 0x7fdc4c102200 2024-03-28T16:07:57.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.173+0000 7fdc42a00640 1 -- 172.21.15.67:0/2313112618 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fdc3c004aa0 con 0x7fdc4c102200 2024-03-28T16:07:57.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.173+0000 7fdc42a00640 1 -- 172.21.15.67:0/2313112618 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc3c003d70 con 0x7fdc4c102200 2024-03-28T16:07:57.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.173+0000 7fdc51200640 1 -- 172.21.15.67:0/2313112618 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc4c1a19d0 con 0x7fdc4c102200 2024-03-28T16:07:57.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.173+0000 7fdc51200640 1 -- 172.21.15.67:0/2313112618 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc4c1a1ed0 con 0x7fdc4c102200 2024-03-28T16:07:57.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.175+0000 7fdc51200640 1 -- 172.21.15.67:0/2313112618 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc4c108f50 con 0x7fdc4c102200 2024-03-28T16:07:57.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.179+0000 7fdc42a00640 1 -- 172.21.15.67:0/2313112618 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fdc3c046020 con 0x7fdc4c102200 2024-03-28T16:07:57.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.181+0000 7fdc42a00640 1 --2- 172.21.15.67:0/2313112618 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fdc300a2880 0x7fdc300a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:57.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.181+0000 7fdc42a00640 1 -- 172.21.15.67:0/2313112618 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fdc3c0f0f30 con 0x7fdc4c102200 2024-03-28T16:07:57.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.181+0000 7fdc43e00640 1 --2- 172.21.15.67:0/2313112618 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fdc300a2880 0x7fdc300a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:57.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.183+0000 7fdc43e00640 1 --2- 172.21.15.67:0/2313112618 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fdc300a2880 0x7fdc300a4d40 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c19b030 tx=0x7fdc34005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:57.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.188+0000 7fdc42a00640 1 -- 172.21.15.67:0/2313112618 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fdc3c04d070 con 0x7fdc4c102200 2024-03-28T16:07:57.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:07:57 smithi067 ceph-mon[31707]: scrub queued for path: / 2024-03-28T16:07:57.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:07:57 smithi067 ceph-mon[31707]: scrub summary: idle+waiting paths [/] 2024-03-28T16:07:57.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:07:57 smithi067 ceph-mon[31707]: scrub summary: active paths [/] 2024-03-28T16:07:57.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:07:57 smithi067 ceph-mon[31707]: scrub summary: idle+waiting paths [/] 2024-03-28T16:07:57.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:07:57 smithi067 ceph-mon[31707]: scrub summary: idle 2024-03-28T16:07:57.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:07:57 smithi183 ceph-mon[35580]: scrub queued for path: / 2024-03-28T16:07:57.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:07:57 smithi183 ceph-mon[35580]: scrub summary: idle+waiting paths [/] 2024-03-28T16:07:57.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:07:57 smithi183 ceph-mon[35580]: scrub summary: active paths [/] 2024-03-28T16:07:57.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:07:57 smithi183 ceph-mon[35580]: scrub summary: idle+waiting paths [/] 2024-03-28T16:07:57.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:07:57 smithi183 ceph-mon[35580]: scrub summary: idle 2024-03-28T16:07:57.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.538+0000 7fdc51200640 1 -- 172.21.15.67:0/2313112618 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdc4c02d070 con 0x7fdc4c102200 2024-03-28T16:07:57.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.541+0000 7fdc42a00640 1 -- 172.21.15.67:0/2313112618 <== mon.2 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11341 (secure 0 0 0) 0x7fdc3c0bdfe0 con 0x7fdc4c102200 2024-03-28T16:07:57.542 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:07:57.542 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2008297659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2008297659},{"type":"v1","addr":"172.21.15.67:6835","nonce":2008297659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14532,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/320295745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":320295745},{"type":"v1","addr":"172.21.15.67:6837","nonce":320295745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6839/691821273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":691821273},{"type":"v1","addr":"172.21.15.67:6839","nonce":691821273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6841/3211254875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6840","nonce":3211254875},{"type":"v1","addr":"172.21.15.67:6841","nonce":3211254875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2129665423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2129665423},{"type":"v1","addr":"172.21.15.183:6833","nonce":2129665423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24496,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/1589784425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":1589784425},{"type":"v1","addr":"172.21.15.183:6835","nonce":1589784425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24508,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/2185863835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":2185863835},{"type":"v1","addr":"172.21.15.183:6837","nonce":2185863835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24520,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6839/1242814398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":1242814398},{"type":"v1","addr":"172.21.15.183:6839","nonce":1242814398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":34409,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1008335521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1008335521},{"type":"v1","addr":"172.21.15.150:6833","nonce":1008335521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":34445,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3269935695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3269935695},{"type":"v1","addr":"172.21.15.150:6835","nonce":3269935695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T16:01:07.696824+0000","modified":"2024-03-28T16:01:11.997773+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":34508},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34475":{"gid":34475,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/1112886277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":1112886277},{"type":"v1","addr":"172.21.15.150:6837","nonce":1112886277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_34508":{"gid":34508,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6839/1211011614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":1211011614},{"type":"v1","addr":"172.21.15.150:6839","nonce":1211011614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34508,"qdb_cluster":[34508]},"id":1}]} 2024-03-28T16:07:57.542 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 19 2024-03-28T16:07:57.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.545+0000 7fdc51200640 1 -- 172.21.15.67:0/2313112618 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fdc300a2880 msgr2=0x7fdc300a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:57.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.545+0000 7fdc51200640 1 --2- 172.21.15.67:0/2313112618 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fdc300a2880 0x7fdc300a4d40 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c19b030 tx=0x7fdc34005f90 comp rx=0 tx=0).stop 2024-03-28T16:07:57.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.545+0000 7fdc51200640 1 -- 172.21.15.67:0/2313112618 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdc4c102200 msgr2=0x7fdc4c199b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:57.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.545+0000 7fdc51200640 1 --2- 172.21.15.67:0/2313112618 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdc4c102200 0x7fdc4c199b20 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c000c00 tx=0x7fdc3c0042f0 comp rx=0 tx=0).stop 2024-03-28T16:07:57.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.545+0000 7fdc51200640 1 -- 172.21.15.67:0/2313112618 shutdown_connections 2024-03-28T16:07:57.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.545+0000 7fdc51200640 1 --2- 172.21.15.67:0/2313112618 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fdc300a2880 0x7fdc300a4d40 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:57.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.545+0000 7fdc51200640 1 --2- 172.21.15.67:0/2313112618 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdc4c107460 0x7fdc4c1a10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:57.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.545+0000 7fdc51200640 1 --2- 172.21.15.67:0/2313112618 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc4c104b30 0x7fdc4c19a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:57.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.545+0000 7fdc51200640 1 --2- 172.21.15.67:0/2313112618 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdc4c102200 0x7fdc4c199b20 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:57.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.545+0000 7fdc51200640 1 -- 172.21.15.67:0/2313112618 >> 172.21.15.67:0/2313112618 conn(0x7fdc4c0fbb30 msgr2=0x7fdc4c101910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:57.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.545+0000 7fdc51200640 1 -- 172.21.15.67:0/2313112618 shutdown_connections 2024-03-28T16:07:57.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.545+0000 7fdc51200640 1 -- 172.21.15.67:0/2313112618 wait complete. 2024-03-28T16:07:57.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:07:57 smithi150 ceph-mon[34825]: scrub queued for path: / 2024-03-28T16:07:57.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:07:57 smithi150 ceph-mon[34825]: scrub summary: idle+waiting paths [/] 2024-03-28T16:07:57.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:07:57 smithi150 ceph-mon[34825]: scrub summary: active paths [/] 2024-03-28T16:07:57.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:07:57 smithi150 ceph-mon[34825]: scrub summary: idle+waiting paths [/] 2024-03-28T16:07:57.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:07:57 smithi150 ceph-mon[34825]: scrub summary: idle 2024-03-28T16:07:57.570 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T16:07:57.570 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T16:07:57.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.835+0000 7f338a000640 1 -- 172.21.15.67:0/436008686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3384105350 msgr2=0x7f338410bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:57.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.835+0000 7f338a000640 1 --2- 172.21.15.67:0/436008686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3384105350 0x7f338410bbe0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f337c009ca0 tx=0x7f337c02c100 comp rx=0 tx=0).stop 2024-03-28T16:07:57.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.835+0000 7f338a000640 1 -- 172.21.15.67:0/436008686 shutdown_connections 2024-03-28T16:07:57.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.835+0000 7f338a000640 1 --2- 172.21.15.67:0/436008686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3384105350 0x7f338410bbe0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:57.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.835+0000 7f338a000640 1 --2- 172.21.15.67:0/436008686 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3384104990 0x7f3384104e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:57.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.835+0000 7f338a000640 1 --2- 172.21.15.67:0/436008686 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f33841037a0 0x7f3384103bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:57.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.835+0000 7f338a000640 1 -- 172.21.15.67:0/436008686 >> 172.21.15.67:0/436008686 conn(0x7f33840fed10 msgr2=0x7f3384101150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:57.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.836+0000 7f338a000640 1 -- 172.21.15.67:0/436008686 shutdown_connections 2024-03-28T16:07:57.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.836+0000 7f338a000640 1 -- 172.21.15.67:0/436008686 wait complete. 2024-03-28T16:07:57.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.837+0000 7f338a000640 1 Processor -- start 2024-03-28T16:07:57.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.837+0000 7f338a000640 1 -- start start 2024-03-28T16:07:57.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.838+0000 7f338a000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f33841037a0 0x7f3384199920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:57.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.838+0000 7f338a000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3384104990 0x7f3384199e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:57.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.838+0000 7f338a000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3384105350 0x7f33841a0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:57.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.838+0000 7f338a000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f338410f2d0 con 0x7f3384104990 2024-03-28T16:07:57.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.838+0000 7f338a000640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f338410f150 con 0x7f3384105350 2024-03-28T16:07:57.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.838+0000 7f338a000640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f338410f450 con 0x7f33841037a0 2024-03-28T16:07:57.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.838+0000 7f337be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3384104990 0x7f3384199e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:57.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.838+0000 7f337be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3384104990 0x7f3384199e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59378/0 (socket says 172.21.15.67:59378) 2024-03-28T16:07:57.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.839+0000 7f337be00640 1 -- 172.21.15.67:0/329966084 learned_addr learned my addr 172.21.15.67:0/329966084 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:07:57.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.839+0000 7f3389600640 1 --2- 172.21.15.67:0/329966084 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3384105350 0x7f33841a0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:57.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.839+0000 7f3388c00640 1 --2- 172.21.15.67:0/329966084 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f33841037a0 0x7f3384199920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:57.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.839+0000 7f337be00640 1 -- 172.21.15.67:0/329966084 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f33841037a0 msgr2=0x7f3384199920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:57.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.839+0000 7f337be00640 1 --2- 172.21.15.67:0/329966084 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f33841037a0 0x7f3384199920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:57.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.839+0000 7f337be00640 1 -- 172.21.15.67:0/329966084 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3384105350 msgr2=0x7f33841a0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:57.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.839+0000 7f337be00640 1 --2- 172.21.15.67:0/329966084 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3384105350 0x7f33841a0ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:57.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.839+0000 7f337be00640 1 -- 172.21.15.67:0/329966084 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33841a1540 con 0x7f3384104990 2024-03-28T16:07:57.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.839+0000 7f3389600640 1 --2- 172.21.15.67:0/329966084 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3384105350 0x7f33841a0ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T16:07:57.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.840+0000 7f337be00640 1 --2- 172.21.15.67:0/329966084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3384104990 0x7f3384199e60 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f336c00e9f0 tx=0x7f336c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:57.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.840+0000 7f3382000640 1 -- 172.21.15.67:0/329966084 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f336c00cd80 con 0x7f3384104990 2024-03-28T16:07:57.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.840+0000 7f3382000640 1 -- 172.21.15.67:0/329966084 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f336c0108e0 con 0x7f3384104990 2024-03-28T16:07:57.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.840+0000 7f3382000640 1 -- 172.21.15.67:0/329966084 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f336c018920 con 0x7f3384104990 2024-03-28T16:07:57.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.841+0000 7f338a000640 1 -- 172.21.15.67:0/329966084 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33841a1830 con 0x7f3384104990 2024-03-28T16:07:57.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.841+0000 7f338a000640 1 -- 172.21.15.67:0/329966084 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33841a1d30 con 0x7f3384104990 2024-03-28T16:07:57.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.846+0000 7f338e39a740 1 Processor -- start 2024-03-28T16:07:57.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.846+0000 7f338e39a740 1 -- start start 2024-03-28T16:07:57.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.846+0000 7f338e39a740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e828dc79f0 0x55e828dc7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:57.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.846+0000 7f338e39a740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e828dc8bc0 0x55e828dc9040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:57.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.846+0000 7f338e39a740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e828dc9580 0x55e828dcfe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:57.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.847+0000 7f338e39a740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e828d2c2c0 con 0x55e828dc79f0 2024-03-28T16:07:57.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.847+0000 7f338e39a740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e828cd4c90 con 0x55e828dc8bc0 2024-03-28T16:07:57.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.847+0000 7f338e39a740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e828d28360 con 0x55e828dc9580 2024-03-28T16:07:57.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.847+0000 7f3389600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e828dc79f0 0x55e828dc7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:57.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.847+0000 7f3389600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e828dc79f0 0x55e828dc7e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59386/0 (socket says 172.21.15.67:59386) 2024-03-28T16:07:57.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.847+0000 7f3389600640 1 -- 172.21.15.67:0/2712617889 learned_addr learned my addr 172.21.15.67:0/2712617889 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:07:57.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.847+0000 7f3382000640 1 -- 172.21.15.67:0/329966084 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f336c010a90 con 0x7f3384104990 2024-03-28T16:07:57.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.847+0000 7f3388c00640 1 --2- 172.21.15.67:0/2712617889 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e828dc8bc0 0x55e828dc9040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:57.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.848+0000 7f3389600640 1 -- 172.21.15.67:0/2712617889 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e828dc9580 msgr2=0x55e828dcfe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:57.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.848+0000 7f3389600640 1 --2- 172.21.15.67:0/2712617889 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e828dc9580 0x55e828dcfe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:57.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.848+0000 7f3389600640 1 -- 172.21.15.67:0/2712617889 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e828dc8bc0 msgr2=0x55e828dc9040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:57.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.848+0000 7f3389600640 1 --2- 172.21.15.67:0/2712617889 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e828dc8bc0 0x55e828dc9040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:57.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.848+0000 7f3389600640 1 -- 172.21.15.67:0/2712617889 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e828cf9080 con 0x55e828dc79f0 2024-03-28T16:07:57.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.848+0000 7f3388c00640 1 --2- 172.21.15.67:0/2712617889 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e828dc8bc0 0x55e828dc9040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:07:57.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.848+0000 7f3389600640 1 --2- 172.21.15.67:0/2712617889 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e828dc79f0 0x55e828dc7e10 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f337c005e30 tx=0x7f337c005f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0b87b87bc8d7240 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:57.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.849+0000 7f3379c00640 1 -- 172.21.15.67:0/2712617889 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f337c043070 con 0x55e828dc79f0 2024-03-28T16:07:57.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.849+0000 7f3379c00640 1 -- 172.21.15.67:0/2712617889 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f337c003e20 con 0x55e828dc79f0 2024-03-28T16:07:57.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.849+0000 7f3379c00640 1 -- 172.21.15.67:0/2712617889 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f337c039040 con 0x55e828dc79f0 2024-03-28T16:07:57.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.849+0000 7f3382000640 1 --2- 172.21.15.67:0/329966084 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f33680a2880 0x7f33680a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:57.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.849+0000 7f3382000640 1 -- 172.21.15.67:0/329966084 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f336c0cece0 con 0x7f3384104990 2024-03-28T16:07:57.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.850+0000 7f3388c00640 1 --2- 172.21.15.67:0/329966084 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f33680a2880 0x7f33680a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:57.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.850+0000 7f338e39a740 1 -- 172.21.15.67:0/2712617889 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e828dc79f0 msgr2=0x55e828dc7e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:57.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.850+0000 7f338e39a740 1 --2- 172.21.15.67:0/2712617889 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e828dc79f0 0x55e828dc7e10 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f337c005e30 tx=0x7f337c005f80 comp rx=0 tx=0).stop 2024-03-28T16:07:57.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.850+0000 7f338e39a740 1 -- 172.21.15.67:0/2712617889 shutdown_connections 2024-03-28T16:07:57.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.850+0000 7f338e39a740 1 --2- 172.21.15.67:0/2712617889 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e828dc9580 0x55e828dcfe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:57.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.850+0000 7f338e39a740 1 --2- 172.21.15.67:0/2712617889 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e828dc8bc0 0x55e828dc9040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:57.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.850+0000 7f338e39a740 1 --2- 172.21.15.67:0/2712617889 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e828dc79f0 0x55e828dc7e10 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:57.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.850+0000 7f338e39a740 1 -- 172.21.15.67:0/2712617889 >> 172.21.15.67:0/2712617889 conn(0x55e828dc2de0 msgr2=0x55e828dc31e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:57.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.850+0000 7f338e39a740 1 -- 172.21.15.67:0/2712617889 shutdown_connections 2024-03-28T16:07:57.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.850+0000 7f338e39a740 1 -- 172.21.15.67:0/2712617889 wait complete. 2024-03-28T16:07:57.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.851+0000 7f3388c00640 1 --2- 172.21.15.67:0/329966084 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f33680a2880 0x7f33680a4d40 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x55e828dc3d50 tx=0x7f3374005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:57.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.851+0000 7f338e39a740 1 -- start start 2024-03-28T16:07:57.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.851+0000 7f338e39a740 1 Processor -- start 2024-03-28T16:07:57.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.851+0000 7f338e39a740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e828dc79f0 0x55e828dd1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:57.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.852+0000 7f338e39a740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e828dc8bc0 0x55e828dd21e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:57.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.852+0000 7f338e39a740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e828dc9580 0x55e828dd9220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:57.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.852+0000 7f338e39a740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e828dd0530 con 0x55e828dc79f0 2024-03-28T16:07:57.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.852+0000 7f338e39a740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e828ce80d0 con 0x55e828dc8bc0 2024-03-28T16:07:57.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.852+0000 7f338e39a740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e828dd06b0 con 0x55e828dc9580 2024-03-28T16:07:57.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.852+0000 7f338e39a740 10 client.0 _ll_register_callbacks cb 0x55e828cece80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:07:57.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.852+0000 7f3388c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e828dc79f0 0x55e828dd1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:57.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.852+0000 7f3388c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e828dc79f0 0x55e828dd1ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59394/0 (socket says 172.21.15.67:59394) 2024-03-28T16:07:57.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.852+0000 7f3388c00640 1 -- 172.21.15.67:0/4049835769 learned_addr learned my addr 172.21.15.67:0/4049835769 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:07:57.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.852+0000 7f3389600640 1 --2- 172.21.15.67:0/4049835769 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e828dc9580 0x55e828dd9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:57.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.852+0000 7f337be00640 1 --2- 172.21.15.67:0/4049835769 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e828dc8bc0 0x55e828dd21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:57.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.852+0000 7f3388c00640 1 -- 172.21.15.67:0/4049835769 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e828dc9580 msgr2=0x55e828dd9220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:57.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.852+0000 7f3388c00640 1 --2- 172.21.15.67:0/4049835769 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e828dc9580 0x55e828dd9220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:57.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.852+0000 7f3388c00640 1 -- 172.21.15.67:0/4049835769 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e828dc8bc0 msgr2=0x55e828dd21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:57.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.852+0000 7f3388c00640 1 --2- 172.21.15.67:0/4049835769 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e828dc8bc0 0x55e828dd21e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:57.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.852+0000 7f3388c00640 1 -- 172.21.15.67:0/4049835769 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e828a403f0 con 0x55e828dc79f0 2024-03-28T16:07:57.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.853+0000 7f337be00640 1 --2- 172.21.15.67:0/4049835769 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e828dc8bc0 0x55e828dd21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:07:57.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.853+0000 7f3388c00640 1 --2- 172.21.15.67:0/4049835769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e828dc79f0 0x55e828dd1ca0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f3374011890 tx=0x7f33740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bebf7c76a00db040 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:57.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.853+0000 7f3372000640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:07:57.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.853+0000 7f3372000640 1 -- 172.21.15.67:0/4049835769 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3374004030 con 0x55e828dc79f0 2024-03-28T16:07:57.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.853+0000 7f3372000640 1 -- 172.21.15.67:0/4049835769 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f33740041d0 con 0x55e828dc79f0 2024-03-28T16:07:57.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.853+0000 7f338e39a740 1 -- 172.21.15.67:0/4049835769 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f337c003e20 con 0x55e828dc79f0 2024-03-28T16:07:57.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.854+0000 7f3372000640 1 -- 172.21.15.67:0/4049835769 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3374004410 con 0x55e828dc79f0 2024-03-28T16:07:57.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.854+0000 7f338e39a740 10 client.15966 fetch_fsmap learned FSMap version 19 2024-03-28T16:07:57.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.854+0000 7f338e39a740 1 -- 172.21.15.67:0/4049835769 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55e828b0a7d0 con 0x55e828dc79f0 2024-03-28T16:07:57.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.854+0000 7f3372000640 1 -- 172.21.15.67:0/4049835769 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f337c003e20 con 0x55e828dc79f0 2024-03-28T16:07:57.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.854+0000 7f3372000640 1 -- 172.21.15.67:0/4049835769 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f337401a040 con 0x55e828dc79f0 2024-03-28T16:07:57.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.855+0000 7f338e39a740 10 client.15966 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:07:57.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.855+0000 7f338e39a740 10 client.15966 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:07:57.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.855+0000 7f338e39a740 20 client.15966 populate_metadata read hostname 'smithi067' 2024-03-28T16:07:57.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.855+0000 7f338e39a740 1 --2- 172.21.15.67:0/4049835769 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55e828ddd6c0 0x55e828ddfb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:57.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.855+0000 7f338e39a740 4 client.15966 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:07:57.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.855+0000 7f338e39a740 1 -- 172.21.15.67:0/4049835769 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e828d39800 con 0x55e828ddd6c0 2024-03-28T16:07:57.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.855+0000 7f337be00640 1 --2- 172.21.15.67:0/4049835769 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55e828ddd6c0 0x55e828ddfb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:57.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.856+0000 7f337be00640 1 --2- 172.21.15.67:0/4049835769 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55e828ddd6c0 0x55e828ddfb80 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a835702777d95ba6 server_cookie=600b4e850c746c35 in_seq=0 out_seq=0 2024-03-28T16:07:57.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.856+0000 7f3372000640 10 client.15966 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:07:57.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.860+0000 7f3372000640 1 -- 172.21.15.67:0/4049835769 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55e828d39800 con 0x55e828ddd6c0 2024-03-28T16:07:57.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.860+0000 7f3372000640 10 client.15966 handle_command_reply: tid=0 2024-03-28T16:07:57.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.860+0000 7f338e39a740 1 client.15966 shutdown 2024-03-28T16:07:57.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.860+0000 7f338e39a740 1 -- 172.21.15.67:0/4049835769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e828dc79f0 msgr2=0x55e828dd1ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:57.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.860+0000 7f338e39a740 1 --2- 172.21.15.67:0/4049835769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e828dc79f0 0x55e828dd1ca0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f3374011890 tx=0x7f33740046c0 comp rx=0 tx=0).stop 2024-03-28T16:07:57.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.860+0000 7f338e39a740 1 -- 172.21.15.67:0/4049835769 shutdown_connections 2024-03-28T16:07:57.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.860+0000 7f338e39a740 1 --2- 172.21.15.67:0/4049835769 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55e828ddd6c0 0x55e828ddfb80 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:57.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.860+0000 7f338e39a740 1 --2- 172.21.15.67:0/4049835769 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e828dc9580 0x55e828dd9220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:57.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.860+0000 7f338e39a740 1 --2- 172.21.15.67:0/4049835769 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e828dc8bc0 0x55e828dd21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:57.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.860+0000 7f338e39a740 1 --2- 172.21.15.67:0/4049835769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e828dc79f0 0x55e828dd1ca0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:57.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.861+0000 7f338e39a740 1 -- 172.21.15.67:0/4049835769 >> 172.21.15.67:0/4049835769 conn(0x55e828dc2de0 msgr2=0x55e828dc2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:57.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.861+0000 7f3372000640 0 client.15966 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:07:57.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.861+0000 7f338e39a740 1 -- 172.21.15.67:0/4049835769 shutdown_connections 2024-03-28T16:07:57.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.861+0000 7f338e39a740 1 -- 172.21.15.67:0/4049835769 wait complete. 2024-03-28T16:07:57.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.861+0000 7f338e39a740 20 client.15966 trim_cache size 0 max 16384 2024-03-28T16:07:57.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.882+0000 7f338e39a740 1 Processor -- start 2024-03-28T16:07:57.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.882+0000 7f338e39a740 1 -- start start 2024-03-28T16:07:57.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.883+0000 7f338e39a740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e828dc79f0 0x55e828dc6750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:57.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.883+0000 7f338e39a740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e828dc8bc0 0x55e828dd91e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:57.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.883+0000 7f338e39a740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e828dc9580 0x55e828dcbc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:57.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.883+0000 7f338e39a740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33841a8a30 con 0x55e828dc8bc0 2024-03-28T16:07:57.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.883+0000 7f338e39a740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33841aaff0 con 0x55e828dc9580 2024-03-28T16:07:57.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.883+0000 7f338e39a740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33841a9030 con 0x55e828dc79f0 2024-03-28T16:07:57.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.883+0000 7f337be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e828dc8bc0 0x55e828dd91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:57.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.883+0000 7f337be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e828dc8bc0 0x55e828dd91e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59406/0 (socket says 172.21.15.67:59406) 2024-03-28T16:07:57.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.883+0000 7f337be00640 1 -- 172.21.15.67:0/2158116471 learned_addr learned my addr 172.21.15.67:0/2158116471 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:07:57.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.884+0000 7f337be00640 1 -- 172.21.15.67:0/2158116471 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e828dc79f0 msgr2=0x55e828dc6750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:57.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.884+0000 7f337be00640 1 --2- 172.21.15.67:0/2158116471 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e828dc79f0 0x55e828dc6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:57.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.884+0000 7f3389600640 1 --2- 172.21.15.67:0/2158116471 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e828dc9580 0x55e828dcbc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:57.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.884+0000 7f337be00640 1 -- 172.21.15.67:0/2158116471 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e828dc9580 msgr2=0x55e828dcbc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:57.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.884+0000 7f337be00640 1 --2- 172.21.15.67:0/2158116471 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e828dc9580 0x55e828dcbc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:57.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.884+0000 7f337be00640 1 -- 172.21.15.67:0/2158116471 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e828dcf7a0 con 0x55e828dc8bc0 2024-03-28T16:07:57.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.884+0000 7f337be00640 1 --2- 172.21.15.67:0/2158116471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e828dc8bc0 0x55e828dd91e0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f336c098110 tx=0x7f336c086040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c1136c00bffca98 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:57.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.884+0000 7f3379200640 1 -- 172.21.15.67:0/2158116471 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f336c0a3e30 con 0x55e828dc8bc0 2024-03-28T16:07:57.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.884+0000 7f3379200640 1 -- 172.21.15.67:0/2158116471 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f336c0997e0 con 0x55e828dc8bc0 2024-03-28T16:07:57.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.884+0000 7f3379200640 1 -- 172.21.15.67:0/2158116471 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f336c09ab90 con 0x55e828dc8bc0 2024-03-28T16:07:57.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.885+0000 7f338e39a740 1 -- 172.21.15.67:0/2158116471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e828dc8bc0 msgr2=0x55e828dd91e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:57.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.885+0000 7f338e39a740 1 --2- 172.21.15.67:0/2158116471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e828dc8bc0 0x55e828dd91e0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f336c098110 tx=0x7f336c086040 comp rx=0 tx=0).stop 2024-03-28T16:07:57.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.885+0000 7f338e39a740 1 -- 172.21.15.67:0/2158116471 shutdown_connections 2024-03-28T16:07:57.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.885+0000 7f338e39a740 1 --2- 172.21.15.67:0/2158116471 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e828dc9580 0x55e828dcbc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:57.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.885+0000 7f338e39a740 1 --2- 172.21.15.67:0/2158116471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e828dc8bc0 0x55e828dd91e0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:57.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.885+0000 7f338e39a740 1 --2- 172.21.15.67:0/2158116471 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e828dc79f0 0x55e828dc6750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:57.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.885+0000 7f338e39a740 1 -- 172.21.15.67:0/2158116471 >> 172.21.15.67:0/2158116471 conn(0x55e828dc2de0 msgr2=0x55e828dc6040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:57.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.886+0000 7f338e39a740 1 -- 172.21.15.67:0/2158116471 shutdown_connections 2024-03-28T16:07:57.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.886+0000 7f338e39a740 1 -- 172.21.15.67:0/2158116471 wait complete. 2024-03-28T16:07:57.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.886+0000 7f338e39a740 1 -- start start 2024-03-28T16:07:57.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.886+0000 7f338e39a740 1 Processor -- start 2024-03-28T16:07:57.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.887+0000 7f338e39a740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e828dc79f0 0x55e828dd5070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:57.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.887+0000 7f338e39a740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e828dc8bc0 0x55e828dd55b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:57.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.887+0000 7f338e39a740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e828dc9580 0x55e828dd85d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:57.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.887+0000 7f338e39a740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33841a9670 con 0x55e828dc8bc0 2024-03-28T16:07:57.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.887+0000 7f338e39a740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33841aa990 con 0x55e828dc79f0 2024-03-28T16:07:57.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.887+0000 7f338e39a740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33841a9cd0 con 0x55e828dc9580 2024-03-28T16:07:57.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.887+0000 7f338e39a740 10 client.0 _ll_register_callbacks cb 0x7f3374005a90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:07:57.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.887+0000 7f337be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e828dc8bc0 0x55e828dd55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:57.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.887+0000 7f337be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e828dc8bc0 0x55e828dd55b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59416/0 (socket says 172.21.15.67:59416) 2024-03-28T16:07:57.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.887+0000 7f337be00640 1 -- 172.21.15.67:0/247230835 learned_addr learned my addr 172.21.15.67:0/247230835 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:07:57.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.887+0000 7f337be00640 1 -- 172.21.15.67:0/247230835 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e828dc9580 msgr2=0x55e828dd85d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:57.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.887+0000 7f337be00640 1 --2- 172.21.15.67:0/247230835 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e828dc9580 0x55e828dd85d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:57.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.887+0000 7f337be00640 1 -- 172.21.15.67:0/247230835 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e828dc79f0 msgr2=0x55e828dd5070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:57.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.887+0000 7f337be00640 1 --2- 172.21.15.67:0/247230835 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e828dc79f0 0x55e828dd5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:57.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.887+0000 7f337be00640 1 -- 172.21.15.67:0/247230835 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e828dd8bc0 con 0x55e828dc8bc0 2024-03-28T16:07:57.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.888+0000 7f337be00640 1 --2- 172.21.15.67:0/247230835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e828dc8bc0 0x55e828dd55b0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f336c00e9c0 tx=0x7f336c0cefe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c4cf66fd05c541f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:57.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.888+0000 7f3372000640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:07:57.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.888+0000 7f3372000640 1 -- 172.21.15.67:0/247230835 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f336c0ae070 con 0x55e828dc8bc0 2024-03-28T16:07:57.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.888+0000 7f3372000640 1 -- 172.21.15.67:0/247230835 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f336c097e20 con 0x55e828dc8bc0 2024-03-28T16:07:57.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.888+0000 7f338e39a740 1 -- 172.21.15.67:0/247230835 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f336c0997e0 con 0x55e828dc8bc0 2024-03-28T16:07:57.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.888+0000 7f3372000640 1 -- 172.21.15.67:0/247230835 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f336c0b3040 con 0x55e828dc8bc0 2024-03-28T16:07:57.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.888+0000 7f338e39a740 10 client.15972 fetch_fsmap learned FSMap version 19 2024-03-28T16:07:57.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.888+0000 7f338e39a740 1 -- 172.21.15.67:0/247230835 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55e828e39780 con 0x55e828dc8bc0 2024-03-28T16:07:57.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.889+0000 7f3372000640 1 -- 172.21.15.67:0/247230835 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f336c0a7080 con 0x55e828dc8bc0 2024-03-28T16:07:57.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.889+0000 7f3372000640 1 -- 172.21.15.67:0/247230835 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f336c09ac90 con 0x55e828dc8bc0 2024-03-28T16:07:57.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.889+0000 7f338e39a740 10 client.15972 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:07:57.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.889+0000 7f338e39a740 10 client.15972 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:07:57.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.889+0000 7f338e39a740 20 client.15972 populate_metadata read hostname 'smithi067' 2024-03-28T16:07:57.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.889+0000 7f338e39a740 1 --2- 172.21.15.67:0/247230835 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55e828e3a060 0x55e828e3c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:57.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.889+0000 7f338e39a740 4 client.15972 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T16:07:57.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.889+0000 7f338e39a740 1 -- 172.21.15.67:0/247230835 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55e828d33b50 con 0x55e828e3a060 2024-03-28T16:07:57.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.890+0000 7f3388c00640 1 --2- 172.21.15.67:0/247230835 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55e828e3a060 0x55e828e3c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:57.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.891+0000 7f3388c00640 1 --2- 172.21.15.67:0/247230835 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55e828e3a060 0x55e828e3c480 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4017cf8eb618f17 server_cookie=bdc7679865a9cb70 in_seq=0 out_seq=0 2024-03-28T16:07:57.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.891+0000 7f3372000640 10 client.15972 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:07:57.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.891+0000 7f3372000640 1 -- 172.21.15.67:0/247230835 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55e828d33b50 con 0x55e828e3a060 2024-03-28T16:07:57.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.891+0000 7f3372000640 10 client.15972 handle_command_reply: tid=0 2024-03-28T16:07:57.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.891+0000 7f338e39a740 1 client.15972 shutdown 2024-03-28T16:07:57.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.891+0000 7f338e39a740 1 -- 172.21.15.67:0/247230835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e828dc8bc0 msgr2=0x55e828dd55b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:57.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.891+0000 7f338e39a740 1 --2- 172.21.15.67:0/247230835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e828dc8bc0 0x55e828dd55b0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f336c00e9c0 tx=0x7f336c0cefe0 comp rx=0 tx=0).stop 2024-03-28T16:07:57.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.892+0000 7f338e39a740 1 -- 172.21.15.67:0/247230835 shutdown_connections 2024-03-28T16:07:57.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.892+0000 7f338e39a740 1 --2- 172.21.15.67:0/247230835 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55e828e3a060 0x55e828e3c480 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:57.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.892+0000 7f338e39a740 1 --2- 172.21.15.67:0/247230835 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e828dc9580 0x55e828dd85d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:57.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.892+0000 7f338e39a740 1 --2- 172.21.15.67:0/247230835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e828dc8bc0 0x55e828dd55b0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:57.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.892+0000 7f338e39a740 1 --2- 172.21.15.67:0/247230835 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e828dc79f0 0x55e828dd5070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:57.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.892+0000 7f3372000640 0 client.15972 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:07:57.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.892+0000 7f338e39a740 1 -- 172.21.15.67:0/247230835 >> 172.21.15.67:0/247230835 conn(0x55e828dc2de0 msgr2=0x55e828dc68d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:57.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.892+0000 7f338e39a740 1 -- 172.21.15.67:0/247230835 shutdown_connections 2024-03-28T16:07:57.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.892+0000 7f338e39a740 1 -- 172.21.15.67:0/247230835 wait complete. 2024-03-28T16:07:57.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.892+0000 7f338e39a740 20 client.15972 trim_cache size 0 max 16384 2024-03-28T16:07:57.896 INFO:teuthology.orchestra.run.smithi067.stdout:[] 2024-03-28T16:07:57.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.893+0000 7f338a000640 1 -- 172.21.15.67:0/329966084 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f33680a2880 msgr2=0x7f33680a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:57.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.893+0000 7f338a000640 1 --2- 172.21.15.67:0/329966084 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f33680a2880 0x7f33680a4d40 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x55e828dc3d50 tx=0x7f3374005f90 comp rx=0 tx=0).stop 2024-03-28T16:07:57.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.893+0000 7f338a000640 1 -- 172.21.15.67:0/329966084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3384104990 msgr2=0x7f3384199e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:57.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.893+0000 7f338a000640 1 --2- 172.21.15.67:0/329966084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3384104990 0x7f3384199e60 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f336c00e9f0 tx=0x7f336c00eeb0 comp rx=0 tx=0).stop 2024-03-28T16:07:57.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.893+0000 7f338a000640 1 -- 172.21.15.67:0/329966084 shutdown_connections 2024-03-28T16:07:57.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.893+0000 7f338a000640 1 --2- 172.21.15.67:0/329966084 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f33680a2880 0x7f33680a4d40 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:57.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.893+0000 7f338a000640 1 --2- 172.21.15.67:0/329966084 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3384105350 0x7f33841a0ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:57.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.893+0000 7f338a000640 1 --2- 172.21.15.67:0/329966084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3384104990 0x7f3384199e60 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:57.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.893+0000 7f338a000640 1 --2- 172.21.15.67:0/329966084 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f33841037a0 0x7f3384199920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:57.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.893+0000 7f338a000640 1 -- 172.21.15.67:0/329966084 >> 172.21.15.67:0/329966084 conn(0x7f33840fed10 msgr2=0x7f3384107c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:57.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.893+0000 7f338a000640 1 -- 172.21.15.67:0/329966084 shutdown_connections 2024-03-28T16:07:57.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:57.893+0000 7f338a000640 1 -- 172.21.15.67:0/329966084 wait complete. 2024-03-28T16:07:58.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:07:58 smithi067 ceph-mon[31707]: pgmap v571: 97 pgs: 97 active+clean; 5.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 149 MiB/s wr, 39 op/s 2024-03-28T16:07:58.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:07:58 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/2313112618' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:07:58.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:07:58 smithi183 ceph-mon[35580]: pgmap v571: 97 pgs: 97 active+clean; 5.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 149 MiB/s wr, 39 op/s 2024-03-28T16:07:58.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:07:58 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/2313112618' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:07:58.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:07:58 smithi150 ceph-mon[34825]: pgmap v571: 97 pgs: 97 active+clean; 5.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 149 MiB/s wr, 39 op/s 2024-03-28T16:07:58.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:07:58 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/2313112618' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:07:58.921 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T16:07:58.921 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T16:07:58.921 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T16:07:59.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.200+0000 7fd817e00640 1 -- 172.21.15.67:0/1704421528 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd818105350 msgr2=0x7fd81810bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:59.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.200+0000 7fd817e00640 1 --2- 172.21.15.67:0/1704421528 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd818105350 0x7fd81810bbe0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fd810009ca0 tx=0x7fd81002c100 comp rx=0 tx=0).stop 2024-03-28T16:07:59.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.200+0000 7fd817e00640 1 -- 172.21.15.67:0/1704421528 shutdown_connections 2024-03-28T16:07:59.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.200+0000 7fd817e00640 1 --2- 172.21.15.67:0/1704421528 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd818105350 0x7fd81810bbe0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:59.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.200+0000 7fd817e00640 1 --2- 172.21.15.67:0/1704421528 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd818104990 0x7fd818104e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:59.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.200+0000 7fd817e00640 1 --2- 172.21.15.67:0/1704421528 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd8181037a0 0x7fd818103bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:59.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.200+0000 7fd817e00640 1 -- 172.21.15.67:0/1704421528 >> 172.21.15.67:0/1704421528 conn(0x7fd8180fed10 msgr2=0x7fd818101150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:59.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.200+0000 7fd817e00640 1 -- 172.21.15.67:0/1704421528 shutdown_connections 2024-03-28T16:07:59.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.200+0000 7fd817e00640 1 -- 172.21.15.67:0/1704421528 wait complete. 2024-03-28T16:07:59.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.201+0000 7fd817e00640 1 Processor -- start 2024-03-28T16:07:59.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.202+0000 7fd817e00640 1 -- start start 2024-03-28T16:07:59.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.202+0000 7fd817e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8181037a0 0x7fd818199940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:59.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.202+0000 7fd817e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd818104990 0x7fd818199e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:59.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.202+0000 7fd817e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd818105350 0x7fd8181a0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:59.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.202+0000 7fd817e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd81810f2d0 con 0x7fd8181037a0 2024-03-28T16:07:59.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.202+0000 7fd817e00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd81810f150 con 0x7fd818104990 2024-03-28T16:07:59.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.202+0000 7fd817e00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd81810f450 con 0x7fd818105350 2024-03-28T16:07:59.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.202+0000 7fd816a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8181037a0 0x7fd818199940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:59.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.203+0000 7fd816a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8181037a0 0x7fd818199940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59432/0 (socket says 172.21.15.67:59432) 2024-03-28T16:07:59.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.203+0000 7fd816a00640 1 -- 172.21.15.67:0/519940886 learned_addr learned my addr 172.21.15.67:0/519940886 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:07:59.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.203+0000 7fd816000640 1 --2- 172.21.15.67:0/519940886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd818104990 0x7fd818199e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:59.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.203+0000 7fd816a00640 1 -- 172.21.15.67:0/519940886 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd818105350 msgr2=0x7fd8181a0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:59.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.203+0000 7fd816a00640 1 --2- 172.21.15.67:0/519940886 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd818105350 0x7fd8181a0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:59.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.203+0000 7fd816a00640 1 -- 172.21.15.67:0/519940886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd818104990 msgr2=0x7fd818199e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:59.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.203+0000 7fd816a00640 1 --2- 172.21.15.67:0/519940886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd818104990 0x7fd818199e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:59.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.203+0000 7fd816a00640 1 -- 172.21.15.67:0/519940886 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8181a1560 con 0x7fd8181037a0 2024-03-28T16:07:59.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.204+0000 7fd816a00640 1 --2- 172.21.15.67:0/519940886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8181037a0 0x7fd818199940 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fd80800d950 tx=0x7fd80800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:59.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.204+0000 7fd80f400640 1 -- 172.21.15.67:0/519940886 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd808017070 con 0x7fd8181037a0 2024-03-28T16:07:59.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.204+0000 7fd80f400640 1 -- 172.21.15.67:0/519940886 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd808009c30 con 0x7fd8181037a0 2024-03-28T16:07:59.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.204+0000 7fd80f400640 1 -- 172.21.15.67:0/519940886 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd80800f620 con 0x7fd8181037a0 2024-03-28T16:07:59.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.204+0000 7fd817e00640 1 -- 172.21.15.67:0/519940886 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8181a1850 con 0x7fd8181037a0 2024-03-28T16:07:59.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.204+0000 7fd817e00640 1 -- 172.21.15.67:0/519940886 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8181a1d50 con 0x7fd8181037a0 2024-03-28T16:07:59.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.210+0000 7fd8205d8740 1 Processor -- start 2024-03-28T16:07:59.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.210+0000 7fd8205d8740 1 -- start start 2024-03-28T16:07:59.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.210+0000 7fd80f400640 1 -- 172.21.15.67:0/519940886 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fd80801a070 con 0x7fd8181037a0 2024-03-28T16:07:59.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.210+0000 7fd8205d8740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5613690f9ac0 0x5613690f9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:59.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.210+0000 7fd8205d8740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5613690fa4b0 0x5613690fefb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:59.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.210+0000 7fd8205d8740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5613690ff620 0x561369101ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:59.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.210+0000 7fd8205d8740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x561369056eb0 con 0x5613690fa4b0 2024-03-28T16:07:59.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.210+0000 7fd8205d8740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x561368fffa60 con 0x5613690ff620 2024-03-28T16:07:59.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.210+0000 7fd8205d8740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x561369052f50 con 0x5613690f9ac0 2024-03-28T16:07:59.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.211+0000 7fd817400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5613690f9ac0 0x5613690f9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:59.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.211+0000 7fd817400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5613690f9ac0 0x5613690f9ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.67:43066/0 (socket says 172.21.15.67:43066) 2024-03-28T16:07:59.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.211+0000 7fd817400640 1 -- 172.21.15.67:0/2325129140 learned_addr learned my addr 172.21.15.67:0/2325129140 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:07:59.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.212+0000 7fd816a00640 1 --2- 172.21.15.67:0/2325129140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5613690fa4b0 0x5613690fefb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:59.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.212+0000 7fd817400640 1 -- 172.21.15.67:0/2325129140 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5613690ff620 msgr2=0x561369101ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:59.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.212+0000 7fd817400640 1 --2- 172.21.15.67:0/2325129140 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5613690ff620 0x561369101ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:59.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.212+0000 7fd817400640 1 -- 172.21.15.67:0/2325129140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5613690fa4b0 msgr2=0x5613690fefb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:59.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.212+0000 7fd817400640 1 --2- 172.21.15.67:0/2325129140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5613690fa4b0 0x5613690fefb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:59.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.212+0000 7fd817400640 1 -- 172.21.15.67:0/2325129140 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561368e337e0 con 0x5613690f9ac0 2024-03-28T16:07:59.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.213+0000 7fd80f400640 1 --2- 172.21.15.67:0/519940886 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fd7fc0a2880 0x7fd7fc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:59.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.213+0000 7fd817400640 1 --2- 172.21.15.67:0/2325129140 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5613690f9ac0 0x5613690f9ee0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd810005e30 tx=0x7fd810005f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa35a6a9a7cda547 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:59.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.213+0000 7fd80f400640 1 -- 172.21.15.67:0/519940886 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fd8080ce4e0 con 0x7fd8181037a0 2024-03-28T16:07:59.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.213+0000 7fd816a00640 1 --2- 172.21.15.67:0/519940886 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fd7fc0a2880 0x7fd7fc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:59.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.214+0000 7fd816a00640 1 --2- 172.21.15.67:0/519940886 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fd7fc0a2880 0x7fd7fc0a4d40 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fd808097290 tx=0x7fd808005f30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:59.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.214+0000 7fd803e00640 1 -- 172.21.15.67:0/2325129140 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd810041020 con 0x5613690f9ac0 2024-03-28T16:07:59.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.214+0000 7fd803e00640 1 -- 172.21.15.67:0/2325129140 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd810032dc0 con 0x5613690f9ac0 2024-03-28T16:07:59.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.215+0000 7fd803e00640 1 -- 172.21.15.67:0/2325129140 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd810039040 con 0x5613690f9ac0 2024-03-28T16:07:59.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.215+0000 7fd8205d8740 1 -- 172.21.15.67:0/2325129140 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5613690f9ac0 msgr2=0x5613690f9ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:59.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.215+0000 7fd8205d8740 1 --2- 172.21.15.67:0/2325129140 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5613690f9ac0 0x5613690f9ee0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd810005e30 tx=0x7fd810005f80 comp rx=0 tx=0).stop 2024-03-28T16:07:59.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.216+0000 7fd8205d8740 1 -- 172.21.15.67:0/2325129140 shutdown_connections 2024-03-28T16:07:59.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.216+0000 7fd8205d8740 1 --2- 172.21.15.67:0/2325129140 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5613690ff620 0x561369101ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:59.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.216+0000 7fd8205d8740 1 --2- 172.21.15.67:0/2325129140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5613690fa4b0 0x5613690fefb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:59.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.216+0000 7fd8205d8740 1 --2- 172.21.15.67:0/2325129140 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5613690f9ac0 0x5613690f9ee0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:59.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.216+0000 7fd8205d8740 1 -- 172.21.15.67:0/2325129140 >> 172.21.15.67:0/2325129140 conn(0x5613690ec820 msgr2=0x5613690ecc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:59.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.216+0000 7fd8205d8740 1 -- 172.21.15.67:0/2325129140 shutdown_connections 2024-03-28T16:07:59.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.216+0000 7fd8205d8740 1 -- 172.21.15.67:0/2325129140 wait complete. 2024-03-28T16:07:59.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.217+0000 7fd8205d8740 1 -- start start 2024-03-28T16:07:59.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.217+0000 7fd8205d8740 1 Processor -- start 2024-03-28T16:07:59.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.218+0000 7fd8205d8740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5613690f9ac0 0x561369103c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:59.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.218+0000 7fd8205d8740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5613690fa4b0 0x5613691041c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:59.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.218+0000 7fd8205d8740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5613690ff620 0x56136910b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:59.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.218+0000 7fd8205d8740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x561369102570 con 0x5613690f9ac0 2024-03-28T16:07:59.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.218+0000 7fd8205d8740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x561369012eb0 con 0x5613690ff620 2024-03-28T16:07:59.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.218+0000 7fd8205d8740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x5613691026f0 con 0x5613690fa4b0 2024-03-28T16:07:59.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.218+0000 7fd8205d8740 10 client.0 _ll_register_callbacks cb 0x561369007150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:07:59.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.218+0000 7fd816000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5613690fa4b0 0x5613691041c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:59.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.219+0000 7fd816000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5613690fa4b0 0x5613691041c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.67:43072/0 (socket says 172.21.15.67:43072) 2024-03-28T16:07:59.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.219+0000 7fd816000640 1 -- 172.21.15.67:0/1375851305 learned_addr learned my addr 172.21.15.67:0/1375851305 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:07:59.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.219+0000 7fd817400640 1 --2- 172.21.15.67:0/1375851305 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5613690ff620 0x56136910b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:59.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.219+0000 7fd817400640 1 --2- 172.21.15.67:0/1375851305 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5613690f9ac0 0x561369103c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:59.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.219+0000 7fd816000640 1 -- 172.21.15.67:0/1375851305 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5613690ff620 msgr2=0x56136910b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:59.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.219+0000 7fd816000640 1 --2- 172.21.15.67:0/1375851305 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5613690ff620 0x56136910b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:59.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.219+0000 7fd816000640 1 -- 172.21.15.67:0/1375851305 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5613690f9ac0 msgr2=0x561369103c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:59.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.219+0000 7fd816000640 1 --2- 172.21.15.67:0/1375851305 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5613690f9ac0 0x561369103c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:59.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.219+0000 7fd816000640 1 -- 172.21.15.67:0/1375851305 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561368e93f70 con 0x5613690fa4b0 2024-03-28T16:07:59.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.220+0000 7fd816000640 1 --2- 172.21.15.67:0/1375851305 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5613690fa4b0 0x5613691041c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd8040116c0 tx=0x7fd804011b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=35870b0d93b1d49b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:59.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.220+0000 7fd800c00640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-03-28T16:07:59.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.220+0000 7fd800c00640 1 -- 172.21.15.67:0/1375851305 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd80400fcd0 con 0x5613690fa4b0 2024-03-28T16:07:59.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.220+0000 7fd8205d8740 1 -- 172.21.15.67:0/1375851305 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd810032dc0 con 0x5613690fa4b0 2024-03-28T16:07:59.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.221+0000 7fd800c00640 1 -- 172.21.15.67:0/1375851305 <== mon.1 v2:172.21.15.183:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fd804004020 con 0x5613690fa4b0 2024-03-28T16:07:59.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.221+0000 7fd800c00640 1 -- 172.21.15.67:0/1375851305 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd80400fe70 con 0x5613690fa4b0 2024-03-28T16:07:59.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.221+0000 7fd8205d8740 10 client.25492 fetch_fsmap learned FSMap version 19 2024-03-28T16:07:59.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.221+0000 7fd8205d8740 1 -- 172.21.15.67:0/1375851305 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x561368d2b170 con 0x5613690fa4b0 2024-03-28T16:07:59.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.222+0000 7fd800c00640 1 -- 172.21.15.67:0/1375851305 <== mon.1 v2:172.21.15.183:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd810032dc0 con 0x5613690fa4b0 2024-03-28T16:07:59.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.222+0000 7fd800c00640 1 -- 172.21.15.67:0/1375851305 <== mon.1 v2:172.21.15.183:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fd804024450 con 0x5613690fa4b0 2024-03-28T16:07:59.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.222+0000 7fd8205d8740 10 client.25492 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:07:59.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.222+0000 7fd8205d8740 10 client.25492 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:07:59.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.222+0000 7fd8205d8740 20 client.25492 populate_metadata read hostname 'smithi067' 2024-03-28T16:07:59.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.222+0000 7fd8205d8740 1 --2- 172.21.15.67:0/1375851305 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x56136910fae0 0x561369111f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:59.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.222+0000 7fd8205d8740 4 client.25492 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:07:59.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.222+0000 7fd8205d8740 1 -- 172.21.15.67:0/1375851305 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5613690643f0 con 0x56136910fae0 2024-03-28T16:07:59.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.224+0000 7fd817400640 1 --2- 172.21.15.67:0/1375851305 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x56136910fae0 0x561369111f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:59.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.226+0000 7fd817400640 1 --2- 172.21.15.67:0/1375851305 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x56136910fae0 0x561369111f20 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b12fd06e2f7d4cbc server_cookie=67e95ab772e1d260 in_seq=0 out_seq=0 2024-03-28T16:07:59.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.226+0000 7fd800c00640 10 client.25492 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:07:59.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.230+0000 7fd800c00640 1 -- 172.21.15.67:0/1375851305 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5613690643f0 con 0x56136910fae0 2024-03-28T16:07:59.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.230+0000 7fd800c00640 10 client.25492 handle_command_reply: tid=0 2024-03-28T16:07:59.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.230+0000 7fd8205d8740 1 client.25492 shutdown 2024-03-28T16:07:59.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.231+0000 7fd8205d8740 1 -- 172.21.15.67:0/1375851305 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5613690fa4b0 msgr2=0x5613691041c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:59.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.231+0000 7fd8205d8740 1 --2- 172.21.15.67:0/1375851305 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5613690fa4b0 0x5613691041c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd8040116c0 tx=0x7fd804011b80 comp rx=0 tx=0).stop 2024-03-28T16:07:59.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.231+0000 7fd8205d8740 1 -- 172.21.15.67:0/1375851305 shutdown_connections 2024-03-28T16:07:59.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.231+0000 7fd8205d8740 1 --2- 172.21.15.67:0/1375851305 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x56136910fae0 0x561369111f20 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:59.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.231+0000 7fd8205d8740 1 --2- 172.21.15.67:0/1375851305 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5613690ff620 0x56136910b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:59.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.231+0000 7fd800c00640 0 client.25492 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:07:59.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.231+0000 7fd8205d8740 1 --2- 172.21.15.67:0/1375851305 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5613690fa4b0 0x5613691041c0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:59.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.231+0000 7fd8205d8740 1 --2- 172.21.15.67:0/1375851305 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5613690f9ac0 0x561369103c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:59.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.231+0000 7fd8205d8740 1 -- 172.21.15.67:0/1375851305 >> 172.21.15.67:0/1375851305 conn(0x5613690ec820 msgr2=0x5613690eba80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:59.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.231+0000 7fd8205d8740 1 -- 172.21.15.67:0/1375851305 shutdown_connections 2024-03-28T16:07:59.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.231+0000 7fd8205d8740 1 -- 172.21.15.67:0/1375851305 wait complete. 2024-03-28T16:07:59.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.231+0000 7fd8205d8740 20 client.25492 trim_cache size 0 max 16384 2024-03-28T16:07:59.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.256+0000 7fd8205d8740 1 Processor -- start 2024-03-28T16:07:59.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.256+0000 7fd8205d8740 1 -- start start 2024-03-28T16:07:59.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.257+0000 7fd8205d8740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5613690f9ac0 0x5613690eb900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:59.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.257+0000 7fd8205d8740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5613690fa4b0 0x56136910b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:59.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.257+0000 7fd8205d8740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5613690ff620 0x5613691167e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:59.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.257+0000 7fd8205d8740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8181a44e0 con 0x5613690ff620 2024-03-28T16:07:59.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.257+0000 7fd8205d8740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8181a6aa0 con 0x5613690f9ac0 2024-03-28T16:07:59.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.257+0000 7fd8205d8740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8181a4ae0 con 0x5613690fa4b0 2024-03-28T16:07:59.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.257+0000 7fd817400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5613690ff620 0x5613691167e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:59.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.257+0000 7fd817400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5613690ff620 0x5613691167e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59466/0 (socket says 172.21.15.67:59466) 2024-03-28T16:07:59.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.257+0000 7fd817400640 1 -- 172.21.15.67:0/2829835154 learned_addr learned my addr 172.21.15.67:0/2829835154 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:07:59.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.257+0000 7fd817400640 1 --2- 172.21.15.67:0/2829835154 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5613690f9ac0 0x5613690eb900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:59.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.258+0000 7fd816000640 1 --2- 172.21.15.67:0/2829835154 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5613690fa4b0 0x56136910b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:59.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.258+0000 7fd817400640 1 -- 172.21.15.67:0/2829835154 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5613690fa4b0 msgr2=0x56136910b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:59.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.258+0000 7fd817400640 1 --2- 172.21.15.67:0/2829835154 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5613690fa4b0 0x56136910b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:59.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.258+0000 7fd817400640 1 -- 172.21.15.67:0/2829835154 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5613690f9ac0 msgr2=0x5613690eb900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:59.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.258+0000 7fd817400640 1 --2- 172.21.15.67:0/2829835154 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5613690f9ac0 0x5613690eb900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:59.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.258+0000 7fd817400640 1 -- 172.21.15.67:0/2829835154 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5613690ed290 con 0x5613690ff620 2024-03-28T16:07:59.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.258+0000 7fd817400640 1 --2- 172.21.15.67:0/2829835154 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5613690ff620 0x5613691167e0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fd81002c640 tx=0x7fd81004a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36080087b6967a09 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:59.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.259+0000 7fd803400640 1 -- 172.21.15.67:0/2829835154 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd810052040 con 0x5613690ff620 2024-03-28T16:07:59.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.259+0000 7fd803400640 1 -- 172.21.15.67:0/2829835154 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd810048440 con 0x5613690ff620 2024-03-28T16:07:59.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.259+0000 7fd803400640 1 -- 172.21.15.67:0/2829835154 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd810041040 con 0x5613690ff620 2024-03-28T16:07:59.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.260+0000 7fd8205d8740 1 -- 172.21.15.67:0/2829835154 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5613690ff620 msgr2=0x5613691167e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:59.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.260+0000 7fd8205d8740 1 --2- 172.21.15.67:0/2829835154 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5613690ff620 0x5613691167e0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fd81002c640 tx=0x7fd81004a040 comp rx=0 tx=0).stop 2024-03-28T16:07:59.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.260+0000 7fd8205d8740 1 -- 172.21.15.67:0/2829835154 shutdown_connections 2024-03-28T16:07:59.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.260+0000 7fd8205d8740 1 --2- 172.21.15.67:0/2829835154 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5613690ff620 0x5613691167e0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:59.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.260+0000 7fd8205d8740 1 --2- 172.21.15.67:0/2829835154 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5613690fa4b0 0x56136910b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:59.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.260+0000 7fd8205d8740 1 --2- 172.21.15.67:0/2829835154 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5613690f9ac0 0x5613690eb900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:59.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.260+0000 7fd8205d8740 1 -- 172.21.15.67:0/2829835154 >> 172.21.15.67:0/2829835154 conn(0x5613690ec820 msgr2=0x5613690eb1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:59.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.260+0000 7fd8205d8740 1 -- 172.21.15.67:0/2829835154 shutdown_connections 2024-03-28T16:07:59.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.260+0000 7fd8205d8740 1 -- 172.21.15.67:0/2829835154 wait complete. 2024-03-28T16:07:59.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.261+0000 7fd8205d8740 1 -- start start 2024-03-28T16:07:59.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.261+0000 7fd8205d8740 1 Processor -- start 2024-03-28T16:07:59.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.262+0000 7fd8205d8740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5613690f9ac0 0x56136910b090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:59.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.262+0000 7fd8205d8740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5613690fa4b0 0x5613691096a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:59.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.262+0000 7fd8205d8740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5613690ff620 0x561369109be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:59.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.262+0000 7fd8205d8740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8181a5120 con 0x5613690fa4b0 2024-03-28T16:07:59.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.262+0000 7fd8205d8740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8181a6440 con 0x5613690f9ac0 2024-03-28T16:07:59.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.262+0000 7fd8205d8740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8181a5780 con 0x5613690ff620 2024-03-28T16:07:59.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.262+0000 7fd8205d8740 10 client.0 _ll_register_callbacks cb 0x7fd804009d80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:07:59.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.262+0000 7fd816000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5613690fa4b0 0x5613691096a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:59.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.262+0000 7fd817400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5613690f9ac0 0x56136910b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:59.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.262+0000 7fd816000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5613690fa4b0 0x5613691096a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59476/0 (socket says 172.21.15.67:59476) 2024-03-28T16:07:59.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.262+0000 7fd816000640 1 -- 172.21.15.67:0/1543024903 learned_addr learned my addr 172.21.15.67:0/1543024903 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:07:59.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.263+0000 7fd817400640 1 --2- 172.21.15.67:0/1543024903 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5613690ff620 0x561369109be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:59.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.263+0000 7fd816000640 1 -- 172.21.15.67:0/1543024903 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5613690ff620 msgr2=0x561369109be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:59.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.263+0000 7fd816000640 1 --2- 172.21.15.67:0/1543024903 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5613690ff620 0x561369109be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:59.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.263+0000 7fd816000640 1 -- 172.21.15.67:0/1543024903 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5613690f9ac0 msgr2=0x56136910b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:59.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.263+0000 7fd816000640 1 --2- 172.21.15.67:0/1543024903 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5613690f9ac0 0x56136910b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:59.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.263+0000 7fd816000640 1 -- 172.21.15.67:0/1543024903 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56136910a350 con 0x5613690fa4b0 2024-03-28T16:07:59.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.263+0000 7fd817400640 1 --2- 172.21.15.67:0/1543024903 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5613690ff620 0x561369109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:07:59.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.263+0000 7fd816000640 1 --2- 172.21.15.67:0/1543024903 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5613690fa4b0 0x5613691096a0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fd804006010 tx=0x7fd804009cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34fa648ff1d77b44 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:59.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.263+0000 7fd800c00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:07:59.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.264+0000 7fd800c00640 1 -- 172.21.15.67:0/1543024903 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd80402a040 con 0x5613690fa4b0 2024-03-28T16:07:59.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.264+0000 7fd800c00640 1 -- 172.21.15.67:0/1543024903 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fd804024450 con 0x5613690fa4b0 2024-03-28T16:07:59.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.264+0000 7fd8205d8740 1 -- 172.21.15.67:0/1543024903 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd810048440 con 0x5613690fa4b0 2024-03-28T16:07:59.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.264+0000 7fd800c00640 1 -- 172.21.15.67:0/1543024903 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd804024650 con 0x5613690fa4b0 2024-03-28T16:07:59.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.264+0000 7fd8205d8740 10 client.15987 fetch_fsmap learned FSMap version 19 2024-03-28T16:07:59.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.264+0000 7fd8205d8740 1 -- 172.21.15.67:0/1543024903 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x56136916c410 con 0x5613690fa4b0 2024-03-28T16:07:59.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.265+0000 7fd800c00640 1 -- 172.21.15.67:0/1543024903 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd810048440 con 0x5613690fa4b0 2024-03-28T16:07:59.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.266+0000 7fd800c00640 1 -- 172.21.15.67:0/1543024903 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fd804003710 con 0x5613690fa4b0 2024-03-28T16:07:59.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.266+0000 7fd8205d8740 10 client.15987 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:07:59.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.266+0000 7fd8205d8740 10 client.15987 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:07:59.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.266+0000 7fd8205d8740 20 client.15987 populate_metadata read hostname 'smithi067' 2024-03-28T16:07:59.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.266+0000 7fd8205d8740 1 --2- 172.21.15.67:0/1543024903 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x56136916cd40 0x56136916f160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:59.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.266+0000 7fd8205d8740 4 client.15987 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T16:07:59.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.266+0000 7fd8205d8740 1 -- 172.21.15.67:0/1543024903 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x56136905e740 con 0x56136916cd40 2024-03-28T16:07:59.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.267+0000 7fd817400640 1 --2- 172.21.15.67:0/1543024903 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x56136916cd40 0x56136916f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:59.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.268+0000 7fd817400640 1 --2- 172.21.15.67:0/1543024903 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x56136916cd40 0x56136916f160 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=374716115adae3c server_cookie=8e41fb22e297de95 in_seq=0 out_seq=0 2024-03-28T16:07:59.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.268+0000 7fd800c00640 10 client.15987 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:07:59.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.272+0000 7fd800c00640 1 -- 172.21.15.67:0/1543024903 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x56136905e740 con 0x56136916cd40 2024-03-28T16:07:59.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.272+0000 7fd800c00640 10 client.15987 handle_command_reply: tid=0 2024-03-28T16:07:59.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.272+0000 7fd8205d8740 1 client.15987 shutdown 2024-03-28T16:07:59.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.273+0000 7fd8205d8740 1 -- 172.21.15.67:0/1543024903 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5613690fa4b0 msgr2=0x5613691096a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:59.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.273+0000 7fd8205d8740 1 --2- 172.21.15.67:0/1543024903 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5613690fa4b0 0x5613691096a0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fd804006010 tx=0x7fd804009cc0 comp rx=0 tx=0).stop 2024-03-28T16:07:59.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.273+0000 7fd8205d8740 1 -- 172.21.15.67:0/1543024903 shutdown_connections 2024-03-28T16:07:59.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.273+0000 7fd8205d8740 1 --2- 172.21.15.67:0/1543024903 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x56136916cd40 0x56136916f160 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:59.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.273+0000 7fd8205d8740 1 --2- 172.21.15.67:0/1543024903 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5613690ff620 0x561369109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:59.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.273+0000 7fd8205d8740 1 --2- 172.21.15.67:0/1543024903 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5613690fa4b0 0x5613691096a0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:59.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.273+0000 7fd8205d8740 1 --2- 172.21.15.67:0/1543024903 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5613690f9ac0 0x56136910b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:59.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.273+0000 7fd8205d8740 1 -- 172.21.15.67:0/1543024903 >> 172.21.15.67:0/1543024903 conn(0x5613690ec820 msgr2=0x5613690eba80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:59.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.273+0000 7fd800c00640 0 client.15987 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:07:59.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.273+0000 7fd8205d8740 1 -- 172.21.15.67:0/1543024903 shutdown_connections 2024-03-28T16:07:59.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.273+0000 7fd8205d8740 1 -- 172.21.15.67:0/1543024903 wait complete. 2024-03-28T16:07:59.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.274+0000 7fd8205d8740 20 client.15987 trim_cache size 0 max 16384 2024-03-28T16:07:59.275 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-28T16:07:59.275 INFO:teuthology.orchestra.run.smithi067.stdout: "return_code": 0, 2024-03-28T16:07:59.275 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_tag": "89c12da2-68a8-4315-aafa-ce5783a950ab", 2024-03-28T16:07:59.275 INFO:teuthology.orchestra.run.smithi067.stdout: "mode": "asynchronous" 2024-03-28T16:07:59.275 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-28T16:07:59.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.275+0000 7fd817e00640 1 -- 172.21.15.67:0/519940886 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fd7fc0a2880 msgr2=0x7fd7fc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:59.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.275+0000 7fd817e00640 1 --2- 172.21.15.67:0/519940886 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fd7fc0a2880 0x7fd7fc0a4d40 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fd808097290 tx=0x7fd808005f30 comp rx=0 tx=0).stop 2024-03-28T16:07:59.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.275+0000 7fd817e00640 1 -- 172.21.15.67:0/519940886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8181037a0 msgr2=0x7fd818199940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:59.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.275+0000 7fd817e00640 1 --2- 172.21.15.67:0/519940886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8181037a0 0x7fd818199940 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fd80800d950 tx=0x7fd80800de10 comp rx=0 tx=0).stop 2024-03-28T16:07:59.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.275+0000 7fd817e00640 1 -- 172.21.15.67:0/519940886 shutdown_connections 2024-03-28T16:07:59.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.275+0000 7fd817e00640 1 --2- 172.21.15.67:0/519940886 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fd7fc0a2880 0x7fd7fc0a4d40 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:59.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.275+0000 7fd817e00640 1 --2- 172.21.15.67:0/519940886 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd818105350 0x7fd8181a0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:59.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.275+0000 7fd817e00640 1 --2- 172.21.15.67:0/519940886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd818104990 0x7fd818199e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:59.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.275+0000 7fd817e00640 1 --2- 172.21.15.67:0/519940886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8181037a0 0x7fd818199940 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:59.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.275+0000 7fd817e00640 1 -- 172.21.15.67:0/519940886 >> 172.21.15.67:0/519940886 conn(0x7fd8180fed10 msgr2=0x7fd818107c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:59.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.275+0000 7fd817e00640 1 -- 172.21.15.67:0/519940886 shutdown_connections 2024-03-28T16:07:59.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.275+0000 7fd817e00640 1 -- 172.21.15.67:0/519940886 wait complete. 2024-03-28T16:07:59.300 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T16:07:59.300 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T16:07:59.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.545+0000 7f0b3f200640 1 -- 172.21.15.67:0/1978368686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b400ffe50 msgr2=0x7f0b40100270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:59.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.545+0000 7f0b3f200640 1 --2- 172.21.15.67:0/1978368686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b400ffe50 0x7f0b40100270 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f0b30009900 tx=0x7f0b3002bc30 comp rx=0 tx=0).stop 2024-03-28T16:07:59.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.545+0000 7f0b3f200640 1 -- 172.21.15.67:0/1978368686 shutdown_connections 2024-03-28T16:07:59.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.545+0000 7f0b3f200640 1 --2- 172.21.15.67:0/1978368686 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b40101150 0x7f0b401015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:59.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.545+0000 7f0b3f200640 1 --2- 172.21.15.67:0/1978368686 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0b401007b0 0x7f0b40100c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:59.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.545+0000 7f0b3f200640 1 --2- 172.21.15.67:0/1978368686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b400ffe50 0x7f0b40100270 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:59.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.545+0000 7f0b3f200640 1 -- 172.21.15.67:0/1978368686 >> 172.21.15.67:0/1978368686 conn(0x7f0b400fba10 msgr2=0x7f0b400fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:59.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.545+0000 7f0b3f200640 1 -- 172.21.15.67:0/1978368686 shutdown_connections 2024-03-28T16:07:59.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.545+0000 7f0b3f200640 1 -- 172.21.15.67:0/1978368686 wait complete. 2024-03-28T16:07:59.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.547+0000 7f0b3f200640 1 Processor -- start 2024-03-28T16:07:59.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.547+0000 7f0b3f200640 1 -- start start 2024-03-28T16:07:59.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.547+0000 7f0b3f200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b400ffe50 0x7f0b401999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:59.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.547+0000 7f0b3f200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0b401007b0 0x7f0b40199f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:59.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.547+0000 7f0b3f200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b40101150 0x7f0b401a0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:59.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.547+0000 7f0b3f200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b4010f340 con 0x7f0b400ffe50 2024-03-28T16:07:59.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.547+0000 7f0b3f200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b4010f1c0 con 0x7f0b40101150 2024-03-28T16:07:59.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.547+0000 7f0b3f200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b4010f4c0 con 0x7f0b401007b0 2024-03-28T16:07:59.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.548+0000 7f0b3de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b400ffe50 0x7f0b401999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:59.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.548+0000 7f0b3de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b400ffe50 0x7f0b401999f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49962/0 (socket says 172.21.15.67:49962) 2024-03-28T16:07:59.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.548+0000 7f0b3de00640 1 -- 172.21.15.67:0/1920674764 learned_addr learned my addr 172.21.15.67:0/1920674764 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:07:59.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.548+0000 7f0b3e800640 1 --2- 172.21.15.67:0/1920674764 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b40101150 0x7f0b401a0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:59.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.548+0000 7f0b3d400640 1 --2- 172.21.15.67:0/1920674764 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0b401007b0 0x7f0b40199f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:59.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.548+0000 7f0b3de00640 1 -- 172.21.15.67:0/1920674764 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0b401007b0 msgr2=0x7f0b40199f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:59.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.548+0000 7f0b3de00640 1 --2- 172.21.15.67:0/1920674764 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0b401007b0 0x7f0b40199f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:59.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.548+0000 7f0b3de00640 1 -- 172.21.15.67:0/1920674764 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b40101150 msgr2=0x7f0b401a0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:59.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.548+0000 7f0b3de00640 1 --2- 172.21.15.67:0/1920674764 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b40101150 0x7f0b401a0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:59.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.548+0000 7f0b3de00640 1 -- 172.21.15.67:0/1920674764 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b401a1610 con 0x7f0b400ffe50 2024-03-28T16:07:59.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.548+0000 7f0b3d400640 1 --2- 172.21.15.67:0/1920674764 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0b401007b0 0x7f0b40199f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:07:59.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.548+0000 7f0b3e800640 1 --2- 172.21.15.67:0/1920674764 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b40101150 0x7f0b401a0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T16:07:59.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.549+0000 7f0b3de00640 1 --2- 172.21.15.67:0/1920674764 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b400ffe50 0x7f0b401999f0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f0b30000c00 tx=0x7f0b300042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:59.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.549+0000 7f0b36a00640 1 -- 172.21.15.67:0/1920674764 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b30043020 con 0x7f0b400ffe50 2024-03-28T16:07:59.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.549+0000 7f0b36a00640 1 -- 172.21.15.67:0/1920674764 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0b300054c0 con 0x7f0b400ffe50 2024-03-28T16:07:59.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.549+0000 7f0b36a00640 1 -- 172.21.15.67:0/1920674764 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b30003d80 con 0x7f0b400ffe50 2024-03-28T16:07:59.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.549+0000 7f0b3f200640 1 -- 172.21.15.67:0/1920674764 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b401a18a0 con 0x7f0b400ffe50 2024-03-28T16:07:59.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.549+0000 7f0b3f200640 1 -- 172.21.15.67:0/1920674764 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b401a1e20 con 0x7f0b400ffe50 2024-03-28T16:07:59.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.553+0000 7f0b47f6b740 1 Processor -- start 2024-03-28T16:07:59.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.553+0000 7f0b47f6b740 1 -- start start 2024-03-28T16:07:59.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.553+0000 7f0b36a00640 1 -- 172.21.15.67:0/1920674764 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f0b30046020 con 0x7f0b400ffe50 2024-03-28T16:07:59.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.553+0000 7f0b47f6b740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5649d7792050 0x5649d7792470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:59.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.553+0000 7f0b47f6b740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5649d7792a40 0x5649d7797540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:59.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.553+0000 7f0b47f6b740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5649d7797bb0 0x5649d779a040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:59.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.553+0000 7f0b47f6b740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649d76f7250 con 0x5649d7792a40 2024-03-28T16:07:59.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.553+0000 7f0b47f6b740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649d769fa30 con 0x5649d7792050 2024-03-28T16:07:59.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.553+0000 7f0b47f6b740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649d76f32f0 con 0x5649d7797bb0 2024-03-28T16:07:59.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.554+0000 7f0b3e800640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5649d7792050 0x5649d7792470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:59.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.554+0000 7f0b3de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5649d7792a40 0x5649d7797540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:59.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.554+0000 7f0b3e800640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5649d7792050 0x5649d7792470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.67:44914/0 (socket says 172.21.15.67:44914) 2024-03-28T16:07:59.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.554+0000 7f0b3e800640 1 -- 172.21.15.67:0/2800153601 learned_addr learned my addr 172.21.15.67:0/2800153601 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:07:59.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.554+0000 7f0b3d400640 1 --2- 172.21.15.67:0/2800153601 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5649d7797bb0 0x5649d779a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:59.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.555+0000 7f0b3e800640 1 -- 172.21.15.67:0/2800153601 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5649d7797bb0 msgr2=0x5649d779a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:59.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.555+0000 7f0b3e800640 1 --2- 172.21.15.67:0/2800153601 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5649d7797bb0 0x5649d779a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:59.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.555+0000 7f0b3e800640 1 -- 172.21.15.67:0/2800153601 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5649d7792a40 msgr2=0x5649d7797540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:59.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.555+0000 7f0b3e800640 1 --2- 172.21.15.67:0/2800153601 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5649d7792a40 0x5649d7797540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:59.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.555+0000 7f0b3e800640 1 -- 172.21.15.67:0/2800153601 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5649d768a690 con 0x5649d7792050 2024-03-28T16:07:59.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.555+0000 7f0b36a00640 1 --2- 172.21.15.67:0/1920674764 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f0b240a2800 0x7f0b240a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:59.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.555+0000 7f0b3e800640 1 --2- 172.21.15.67:0/2800153601 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5649d7792050 0x5649d7792470 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0b3800c450 tx=0x7f0b3800c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f3660bcd93b1c7d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:59.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.555+0000 7f0b36a00640 1 -- 172.21.15.67:0/1920674764 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f0b300f10b0 con 0x7f0b400ffe50 2024-03-28T16:07:59.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.556+0000 7f0b3de00640 1 --2- 172.21.15.67:0/1920674764 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f0b240a2800 0x7f0b240a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:59.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.556+0000 7f0b2b400640 1 -- 172.21.15.67:0/2800153601 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b38007430 con 0x5649d7792050 2024-03-28T16:07:59.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.556+0000 7f0b2b400640 1 -- 172.21.15.67:0/2800153601 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0b38007a80 con 0x5649d7792050 2024-03-28T16:07:59.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.556+0000 7f0b2b400640 1 -- 172.21.15.67:0/2800153601 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b3800aa60 con 0x5649d7792050 2024-03-28T16:07:59.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.557+0000 7f0b47f6b740 1 -- 172.21.15.67:0/2800153601 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5649d7792050 msgr2=0x5649d7792470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:59.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.557+0000 7f0b3de00640 1 --2- 172.21.15.67:0/1920674764 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f0b240a2800 0x7f0b240a4cc0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f0b30006010 tx=0x7f0b300045e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:59.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.557+0000 7f0b47f6b740 1 --2- 172.21.15.67:0/2800153601 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5649d7792050 0x5649d7792470 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0b3800c450 tx=0x7f0b3800c910 comp rx=0 tx=0).stop 2024-03-28T16:07:59.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.557+0000 7f0b47f6b740 1 -- 172.21.15.67:0/2800153601 shutdown_connections 2024-03-28T16:07:59.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.557+0000 7f0b47f6b740 1 --2- 172.21.15.67:0/2800153601 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5649d7797bb0 0x5649d779a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:59.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.557+0000 7f0b47f6b740 1 --2- 172.21.15.67:0/2800153601 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5649d7792a40 0x5649d7797540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:59.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.557+0000 7f0b47f6b740 1 --2- 172.21.15.67:0/2800153601 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5649d7792050 0x5649d7792470 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:59.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.557+0000 7f0b47f6b740 1 -- 172.21.15.67:0/2800153601 >> 172.21.15.67:0/2800153601 conn(0x5649d775a200 msgr2=0x5649d778d9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:59.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.557+0000 7f0b47f6b740 1 -- 172.21.15.67:0/2800153601 shutdown_connections 2024-03-28T16:07:59.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.557+0000 7f0b47f6b740 1 -- 172.21.15.67:0/2800153601 wait complete. 2024-03-28T16:07:59.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.558+0000 7f0b47f6b740 1 -- start start 2024-03-28T16:07:59.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.558+0000 7f0b47f6b740 1 Processor -- start 2024-03-28T16:07:59.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.559+0000 7f0b47f6b740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5649d7792050 0x5649d77a4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:59.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.559+0000 7f0b47f6b740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5649d7792a40 0x5649d77a4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:59.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.559+0000 7f0b47f6b740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5649d7797bb0 0x5649d77abde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:59.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.559+0000 7f0b47f6b740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649d779ab60 con 0x5649d7797bb0 2024-03-28T16:07:59.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.559+0000 7f0b47f6b740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649d76b2e80 con 0x5649d7792a40 2024-03-28T16:07:59.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.559+0000 7f0b47f6b740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649d779ace0 con 0x5649d7792050 2024-03-28T16:07:59.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.559+0000 7f0b47f6b740 10 client.0 _ll_register_callbacks cb 0x5649d76c5130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:07:59.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.559+0000 7f0b3e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5649d7797bb0 0x5649d77abde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:59.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.559+0000 7f0b3e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5649d7797bb0 0x5649d77abde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49990/0 (socket says 172.21.15.67:49990) 2024-03-28T16:07:59.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.560+0000 7f0b3e800640 1 -- 172.21.15.67:0/3462541538 learned_addr learned my addr 172.21.15.67:0/3462541538 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:07:59.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.560+0000 7f0b3e800640 1 -- 172.21.15.67:0/3462541538 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5649d7792050 msgr2=0x5649d77a4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:59.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.560+0000 7f0b3e800640 1 --2- 172.21.15.67:0/3462541538 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5649d7792050 0x5649d77a4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:59.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.560+0000 7f0b3e800640 1 -- 172.21.15.67:0/3462541538 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5649d7792a40 msgr2=0x5649d77a4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:59.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.560+0000 7f0b3e800640 1 --2- 172.21.15.67:0/3462541538 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5649d7792a40 0x5649d77a4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:59.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.560+0000 7f0b3e800640 1 -- 172.21.15.67:0/3462541538 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5649d74d57d0 con 0x5649d7797bb0 2024-03-28T16:07:59.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.561+0000 7f0b3e800640 1 --2- 172.21.15.67:0/3462541538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5649d7797bb0 0x5649d77abde0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f0b38007f70 tx=0x7f0b3800a500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4a2b03197339cff server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:59.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.561+0000 7f0b1fe00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:07:59.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.561+0000 7f0b1fe00640 1 -- 172.21.15.67:0/3462541538 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b38024060 con 0x5649d7797bb0 2024-03-28T16:07:59.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.561+0000 7f0b47f6b740 1 -- 172.21.15.67:0/3462541538 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0b38007a80 con 0x5649d7797bb0 2024-03-28T16:07:59.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.561+0000 7f0b1fe00640 1 -- 172.21.15.67:0/3462541538 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f0b3800a7e0 con 0x5649d7797bb0 2024-03-28T16:07:59.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.561+0000 7f0b1fe00640 1 -- 172.21.15.67:0/3462541538 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0b3801b040 con 0x5649d7797bb0 2024-03-28T16:07:59.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.561+0000 7f0b47f6b740 10 client.15999 fetch_fsmap learned FSMap version 19 2024-03-28T16:07:59.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.562+0000 7f0b47f6b740 1 -- 172.21.15.67:0/3462541538 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5649d740b3f0 con 0x5649d7797bb0 2024-03-28T16:07:59.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.562+0000 7f0b1fe00640 1 -- 172.21.15.67:0/3462541538 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0b38007a80 con 0x5649d7797bb0 2024-03-28T16:07:59.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.562+0000 7f0b1fe00640 1 -- 172.21.15.67:0/3462541538 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f0b38016ab0 con 0x5649d7797bb0 2024-03-28T16:07:59.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.562+0000 7f0b47f6b740 10 client.15999 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:07:59.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.562+0000 7f0b47f6b740 10 client.15999 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:07:59.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.562+0000 7f0b47f6b740 20 client.15999 populate_metadata read hostname 'smithi067' 2024-03-28T16:07:59.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.562+0000 7f0b47f6b740 1 --2- 172.21.15.67:0/3462541538 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5649d77b0380 0x5649d77b2770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:59.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.562+0000 7f0b47f6b740 4 client.15999 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:07:59.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.562+0000 7f0b47f6b740 1 -- 172.21.15.67:0/3462541538 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5649d7704790 con 0x5649d77b0380 2024-03-28T16:07:59.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.564+0000 7f0b3d400640 1 --2- 172.21.15.67:0/3462541538 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5649d77b0380 0x5649d77b2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:59.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.565+0000 7f0b3d400640 1 --2- 172.21.15.67:0/3462541538 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5649d77b0380 0x5649d77b2770 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3fd418d6fc05f7eb server_cookie=eaee06ebb65bc27 in_seq=0 out_seq=0 2024-03-28T16:07:59.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.565+0000 7f0b1fe00640 10 client.15999 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:07:59.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.566+0000 7f0b1fe00640 1 -- 172.21.15.67:0/3462541538 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5649d7704790 con 0x5649d77b0380 2024-03-28T16:07:59.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.566+0000 7f0b1fe00640 10 client.15999 handle_command_reply: tid=0 2024-03-28T16:07:59.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.567+0000 7f0b47f6b740 1 client.15999 shutdown 2024-03-28T16:07:59.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.567+0000 7f0b47f6b740 1 -- 172.21.15.67:0/3462541538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5649d7797bb0 msgr2=0x5649d77abde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:59.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.567+0000 7f0b47f6b740 1 --2- 172.21.15.67:0/3462541538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5649d7797bb0 0x5649d77abde0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f0b38007f70 tx=0x7f0b3800a500 comp rx=0 tx=0).stop 2024-03-28T16:07:59.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.567+0000 7f0b47f6b740 1 -- 172.21.15.67:0/3462541538 shutdown_connections 2024-03-28T16:07:59.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.567+0000 7f0b47f6b740 1 --2- 172.21.15.67:0/3462541538 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5649d77b0380 0x5649d77b2770 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:59.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.567+0000 7f0b47f6b740 1 --2- 172.21.15.67:0/3462541538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5649d7797bb0 0x5649d77abde0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:59.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.567+0000 7f0b47f6b740 1 --2- 172.21.15.67:0/3462541538 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5649d7792a40 0x5649d77a4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:59.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.567+0000 7f0b1fe00640 0 client.15999 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:07:59.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.567+0000 7f0b47f6b740 1 --2- 172.21.15.67:0/3462541538 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5649d7792050 0x5649d77a4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:59.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.567+0000 7f0b47f6b740 1 -- 172.21.15.67:0/3462541538 >> 172.21.15.67:0/3462541538 conn(0x5649d775a200 msgr2=0x5649d778cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:59.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.567+0000 7f0b47f6b740 1 -- 172.21.15.67:0/3462541538 shutdown_connections 2024-03-28T16:07:59.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.567+0000 7f0b47f6b740 1 -- 172.21.15.67:0/3462541538 wait complete. 2024-03-28T16:07:59.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.568+0000 7f0b47f6b740 20 client.15999 trim_cache size 0 max 16384 2024-03-28T16:07:59.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.592+0000 7f0b47f6b740 1 Processor -- start 2024-03-28T16:07:59.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.592+0000 7f0b47f6b740 1 -- start start 2024-03-28T16:07:59.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.593+0000 7f0b47f6b740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5649d7792050 0x5649d77b6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:59.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.593+0000 7f0b47f6b740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5649d7792a40 0x5649d77abda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:59.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.593+0000 7f0b47f6b740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5649d7797bb0 0x5649d77b7500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:59.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.593+0000 7f0b47f6b740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b401a46e0 con 0x5649d7792050 2024-03-28T16:07:59.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.593+0000 7f0b47f6b740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b401a6ca0 con 0x5649d7792a40 2024-03-28T16:07:59.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.593+0000 7f0b47f6b740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b401a4ce0 con 0x5649d7797bb0 2024-03-28T16:07:59.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.593+0000 7f0b3e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5649d7792050 0x5649d77b6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:59.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.593+0000 7f0b3e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5649d7792050 0x5649d77b6fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49998/0 (socket says 172.21.15.67:49998) 2024-03-28T16:07:59.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.593+0000 7f0b3e800640 1 -- 172.21.15.67:0/884236266 learned_addr learned my addr 172.21.15.67:0/884236266 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:07:59.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.594+0000 7f0b3d400640 1 --2- 172.21.15.67:0/884236266 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5649d7792a40 0x5649d77abda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:59.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.594+0000 7f0b3e800640 1 --2- 172.21.15.67:0/884236266 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5649d7797bb0 0x5649d77b7500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:59.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.594+0000 7f0b3e800640 1 -- 172.21.15.67:0/884236266 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5649d7797bb0 msgr2=0x5649d77b7500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:59.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.594+0000 7f0b3e800640 1 --2- 172.21.15.67:0/884236266 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5649d7797bb0 0x5649d77b7500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:59.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.594+0000 7f0b3e800640 1 -- 172.21.15.67:0/884236266 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5649d7792a40 msgr2=0x5649d77abda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:59.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.594+0000 7f0b3e800640 1 --2- 172.21.15.67:0/884236266 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5649d7792a40 0x5649d77abda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:59.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.594+0000 7f0b3e800640 1 -- 172.21.15.67:0/884236266 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5649d77b2ae0 con 0x5649d7792050 2024-03-28T16:07:59.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.595+0000 7f0b3d400640 1 --2- 172.21.15.67:0/884236266 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5649d7792a40 0x5649d77abda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T16:07:59.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.595+0000 7f0b3e800640 1 --2- 172.21.15.67:0/884236266 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5649d7792050 0x5649d77b6fc0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f0b38000f80 tx=0x7f0b3800f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b7d31d7a2b25eb3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:59.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.596+0000 7f0b2aa00640 1 -- 172.21.15.67:0/884236266 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b3802f070 con 0x5649d7792050 2024-03-28T16:07:59.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.596+0000 7f0b2aa00640 1 -- 172.21.15.67:0/884236266 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0b3800a570 con 0x5649d7792050 2024-03-28T16:07:59.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.596+0000 7f0b2aa00640 1 -- 172.21.15.67:0/884236266 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b38024070 con 0x5649d7792050 2024-03-28T16:07:59.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.597+0000 7f0b47f6b740 1 -- 172.21.15.67:0/884236266 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5649d7792050 msgr2=0x5649d77b6fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:59.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.598+0000 7f0b47f6b740 1 --2- 172.21.15.67:0/884236266 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5649d7792050 0x5649d77b6fc0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f0b38000f80 tx=0x7f0b3800f000 comp rx=0 tx=0).stop 2024-03-28T16:07:59.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.598+0000 7f0b47f6b740 1 -- 172.21.15.67:0/884236266 shutdown_connections 2024-03-28T16:07:59.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.598+0000 7f0b47f6b740 1 --2- 172.21.15.67:0/884236266 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5649d7797bb0 0x5649d77b7500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:59.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.598+0000 7f0b47f6b740 1 --2- 172.21.15.67:0/884236266 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5649d7792a40 0x5649d77abda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:59.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.598+0000 7f0b47f6b740 1 --2- 172.21.15.67:0/884236266 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5649d7792050 0x5649d77b6fc0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:59.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.598+0000 7f0b47f6b740 1 -- 172.21.15.67:0/884236266 >> 172.21.15.67:0/884236266 conn(0x5649d775a200 msgr2=0x5649d7793d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:59.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.598+0000 7f0b47f6b740 1 -- 172.21.15.67:0/884236266 shutdown_connections 2024-03-28T16:07:59.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.598+0000 7f0b47f6b740 1 -- 172.21.15.67:0/884236266 wait complete. 2024-03-28T16:07:59.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.598+0000 7f0b47f6b740 1 -- start start 2024-03-28T16:07:59.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.599+0000 7f0b47f6b740 1 Processor -- start 2024-03-28T16:07:59.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.599+0000 7f0b47f6b740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5649d7792050 0x5649d77a98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:59.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.599+0000 7f0b47f6b740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5649d7792a40 0x5649d77a9de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:59.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.599+0000 7f0b47f6b740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5649d7797bb0 0x5649d77aada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:59.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.599+0000 7f0b47f6b740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b401a5320 con 0x5649d7792050 2024-03-28T16:07:59.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.599+0000 7f0b47f6b740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b401a6640 con 0x5649d7792a40 2024-03-28T16:07:59.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.599+0000 7f0b47f6b740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b401a5980 con 0x5649d7797bb0 2024-03-28T16:07:59.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.599+0000 7f0b47f6b740 10 client.0 _ll_register_callbacks cb 0x7f0b38009ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:07:59.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.600+0000 7f0b3e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5649d7792050 0x5649d77a98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:59.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.600+0000 7f0b3e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5649d7792050 0x5649d77a98a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50006/0 (socket says 172.21.15.67:50006) 2024-03-28T16:07:59.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.600+0000 7f0b3e800640 1 -- 172.21.15.67:0/4105683339 learned_addr learned my addr 172.21.15.67:0/4105683339 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:07:59.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.600+0000 7f0b3e800640 1 --2- 172.21.15.67:0/4105683339 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5649d7797bb0 0x5649d77aada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:59.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.600+0000 7f0b3d400640 1 --2- 172.21.15.67:0/4105683339 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5649d7792a40 0x5649d77a9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:59.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.600+0000 7f0b3e800640 1 -- 172.21.15.67:0/4105683339 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5649d7797bb0 msgr2=0x5649d77aada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:59.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.600+0000 7f0b3e800640 1 --2- 172.21.15.67:0/4105683339 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5649d7797bb0 0x5649d77aada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:59.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.600+0000 7f0b3e800640 1 -- 172.21.15.67:0/4105683339 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5649d7792a40 msgr2=0x5649d77a9de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:07:59.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.600+0000 7f0b3e800640 1 --2- 172.21.15.67:0/4105683339 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5649d7792a40 0x5649d77a9de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:59.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.600+0000 7f0b3e800640 1 -- 172.21.15.67:0/4105683339 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5649d73cb0f0 con 0x5649d7792050 2024-03-28T16:07:59.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.601+0000 7f0b3e800640 1 --2- 172.21.15.67:0/4105683339 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5649d7792050 0x5649d77a98a0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f0b38009530 tx=0x7f0b380093d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc2880251d4c1021 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:07:59.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.601+0000 7f0b1fe00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:07:59.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.601+0000 7f0b1fe00640 1 -- 172.21.15.67:0/4105683339 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b3802f070 con 0x5649d7792050 2024-03-28T16:07:59.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.601+0000 7f0b1fe00640 1 -- 172.21.15.67:0/4105683339 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f0b3800a980 con 0x5649d7792050 2024-03-28T16:07:59.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.601+0000 7f0b47f6b740 1 -- 172.21.15.67:0/4105683339 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0b3800a570 con 0x5649d7792050 2024-03-28T16:07:59.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.602+0000 7f0b1fe00640 1 -- 172.21.15.67:0/4105683339 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0b38024070 con 0x5649d7792050 2024-03-28T16:07:59.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.602+0000 7f0b47f6b740 10 client.16005 fetch_fsmap learned FSMap version 19 2024-03-28T16:07:59.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.602+0000 7f0b47f6b740 1 -- 172.21.15.67:0/4105683339 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5649d780bce0 con 0x5649d7792050 2024-03-28T16:07:59.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.602+0000 7f0b1fe00640 1 -- 172.21.15.67:0/4105683339 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0b3800a570 con 0x5649d7792050 2024-03-28T16:07:59.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.602+0000 7f0b1fe00640 1 -- 172.21.15.67:0/4105683339 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f0b3802a560 con 0x5649d7792050 2024-03-28T16:07:59.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.602+0000 7f0b47f6b740 10 client.16005 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:07:59.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.602+0000 7f0b47f6b740 10 client.16005 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:07:59.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.602+0000 7f0b47f6b740 20 client.16005 populate_metadata read hostname 'smithi067' 2024-03-28T16:07:59.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.602+0000 7f0b47f6b740 1 --2- 172.21.15.67:0/4105683339 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5649d780c570 0x5649d780e990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:07:59.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.602+0000 7f0b47f6b740 4 client.16005 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T16:07:59.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.602+0000 7f0b47f6b740 1 -- 172.21.15.67:0/4105683339 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5649d76feae0 con 0x5649d780c570 2024-03-28T16:07:59.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.603+0000 7f0b3d400640 1 --2- 172.21.15.67:0/4105683339 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5649d780c570 0x5649d780e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:07:59.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.604+0000 7f0b3d400640 1 --2- 172.21.15.67:0/4105683339 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5649d780c570 0x5649d780e990 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce1cd9205775c8ce server_cookie=b780c42bef9e023 in_seq=0 out_seq=0 2024-03-28T16:07:59.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.604+0000 7f0b1fe00640 10 client.16005 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:07:59.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.605+0000 7f0b1fe00640 1 -- 172.21.15.67:0/4105683339 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x5649d76feae0 con 0x5649d780c570 2024-03-28T16:07:59.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.605+0000 7f0b1fe00640 10 client.16005 handle_command_reply: tid=0 2024-03-28T16:07:59.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.605+0000 7f0b47f6b740 1 client.16005 shutdown 2024-03-28T16:07:59.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.605+0000 7f0b47f6b740 1 -- 172.21.15.67:0/4105683339 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5649d7792050 msgr2=0x5649d77a98a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:59.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.605+0000 7f0b47f6b740 1 --2- 172.21.15.67:0/4105683339 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5649d7792050 0x5649d77a98a0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f0b38009530 tx=0x7f0b380093d0 comp rx=0 tx=0).stop 2024-03-28T16:07:59.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.606+0000 7f0b47f6b740 1 -- 172.21.15.67:0/4105683339 shutdown_connections 2024-03-28T16:07:59.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.606+0000 7f0b47f6b740 1 --2- 172.21.15.67:0/4105683339 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5649d780c570 0x5649d780e990 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:59.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.606+0000 7f0b47f6b740 1 --2- 172.21.15.67:0/4105683339 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5649d7797bb0 0x5649d77aada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:59.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.606+0000 7f0b47f6b740 1 --2- 172.21.15.67:0/4105683339 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5649d7792a40 0x5649d77a9de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:59.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.606+0000 7f0b1fe00640 0 client.16005 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:07:59.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.606+0000 7f0b47f6b740 1 --2- 172.21.15.67:0/4105683339 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5649d7792050 0x5649d77a98a0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:59.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.606+0000 7f0b47f6b740 1 -- 172.21.15.67:0/4105683339 >> 172.21.15.67:0/4105683339 conn(0x5649d775a200 msgr2=0x5649d7790c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:59.607 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-28T16:07:59.607 INFO:teuthology.orchestra.run.smithi067.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T16:07:59.607 INFO:teuthology.orchestra.run.smithi067.stdout: "scrubs": { 2024-03-28T16:07:59.607 INFO:teuthology.orchestra.run.smithi067.stdout: "89c12da2-68a8-4315-aafa-ce5783a950ab": { 2024-03-28T16:07:59.608 INFO:teuthology.orchestra.run.smithi067.stdout: "path": "/", 2024-03-28T16:07:59.608 INFO:teuthology.orchestra.run.smithi067.stdout: "tag": "89c12da2-68a8-4315-aafa-ce5783a950ab", 2024-03-28T16:07:59.608 INFO:teuthology.orchestra.run.smithi067.stdout: "options": "recursive,force" 2024-03-28T16:07:59.608 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:07:59.608 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:07:59.608 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-28T16:07:59.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.606+0000 7f0b47f6b740 1 -- 172.21.15.67:0/4105683339 shutdown_connections 2024-03-28T16:07:59.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.606+0000 7f0b47f6b740 1 -- 172.21.15.67:0/4105683339 wait complete. 2024-03-28T16:07:59.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.606+0000 7f0b47f6b740 20 client.16005 trim_cache size 0 max 16384 2024-03-28T16:07:59.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.607+0000 7f0b3f200640 1 -- 172.21.15.67:0/1920674764 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f0b240a2800 msgr2=0x7f0b240a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:59.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.607+0000 7f0b3f200640 1 --2- 172.21.15.67:0/1920674764 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f0b240a2800 0x7f0b240a4cc0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f0b30006010 tx=0x7f0b300045e0 comp rx=0 tx=0).stop 2024-03-28T16:07:59.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.607+0000 7f0b3f200640 1 -- 172.21.15.67:0/1920674764 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b400ffe50 msgr2=0x7f0b401999f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:07:59.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.607+0000 7f0b3f200640 1 --2- 172.21.15.67:0/1920674764 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b400ffe50 0x7f0b401999f0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f0b30000c00 tx=0x7f0b300042f0 comp rx=0 tx=0).stop 2024-03-28T16:07:59.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.608+0000 7f0b3f200640 1 -- 172.21.15.67:0/1920674764 shutdown_connections 2024-03-28T16:07:59.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.608+0000 7f0b3f200640 1 --2- 172.21.15.67:0/1920674764 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f0b240a2800 0x7f0b240a4cc0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:59.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.608+0000 7f0b3f200640 1 --2- 172.21.15.67:0/1920674764 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b40101150 0x7f0b401a0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:59.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.608+0000 7f0b3f200640 1 --2- 172.21.15.67:0/1920674764 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0b401007b0 0x7f0b40199f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:59.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.608+0000 7f0b3f200640 1 --2- 172.21.15.67:0/1920674764 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b400ffe50 0x7f0b401999f0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:07:59.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.608+0000 7f0b3f200640 1 -- 172.21.15.67:0/1920674764 >> 172.21.15.67:0/1920674764 conn(0x7f0b400fba10 msgr2=0x7f0b400fd150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:07:59.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.608+0000 7f0b3f200640 1 -- 172.21.15.67:0/1920674764 shutdown_connections 2024-03-28T16:07:59.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:07:59.608+0000 7f0b3f200640 1 -- 172.21.15.67:0/1920674764 wait complete. 2024-03-28T16:07:59.632 INFO:tasks.cephfs.filesystem:scrub status for tag:89c12da2-68a8-4315-aafa-ce5783a950ab - {'path': '/', 'tag': '89c12da2-68a8-4315-aafa-ce5783a950ab', 'options': 'recursive,force'} 2024-03-28T16:08:00.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:08:00 smithi150 ceph-mon[34825]: pgmap v572: 97 pgs: 97 active+clean; 6.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 166 MiB/s wr, 43 op/s 2024-03-28T16:08:00.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:08:00 smithi150 ceph-mon[34825]: scrub queued for path: / 2024-03-28T16:08:00.553 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:08:00 smithi150 ceph-mon[34825]: scrub summary: idle+waiting paths [/] 2024-03-28T16:08:00.553 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:08:00 smithi150 ceph-mon[34825]: scrub summary: active paths [/] 2024-03-28T16:08:00.553 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:08:00 smithi150 ceph-mon[34825]: scrub summary: idle+waiting paths [/] 2024-03-28T16:08:00.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:08:00 smithi067 ceph-mon[31707]: pgmap v572: 97 pgs: 97 active+clean; 6.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 166 MiB/s wr, 43 op/s 2024-03-28T16:08:00.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:08:00 smithi067 ceph-mon[31707]: scrub queued for path: / 2024-03-28T16:08:00.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:08:00 smithi067 ceph-mon[31707]: scrub summary: idle+waiting paths [/] 2024-03-28T16:08:00.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:08:00 smithi067 ceph-mon[31707]: scrub summary: active paths [/] 2024-03-28T16:08:00.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:08:00 smithi067 ceph-mon[31707]: scrub summary: idle+waiting paths [/] 2024-03-28T16:08:00.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:08:00 smithi183 ceph-mon[35580]: pgmap v572: 97 pgs: 97 active+clean; 6.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 166 MiB/s wr, 43 op/s 2024-03-28T16:08:00.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:08:00 smithi183 ceph-mon[35580]: scrub queued for path: / 2024-03-28T16:08:00.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:08:00 smithi183 ceph-mon[35580]: scrub summary: idle+waiting paths [/] 2024-03-28T16:08:00.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:08:00 smithi183 ceph-mon[35580]: scrub summary: active paths [/] 2024-03-28T16:08:00.601 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:08:00 smithi183 ceph-mon[35580]: scrub summary: idle+waiting paths [/] 2024-03-28T16:08:02.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:08:02 smithi150 ceph-mon[34825]: pgmap v573: 97 pgs: 97 active+clean; 6.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 192 MiB/s wr, 51 op/s 2024-03-28T16:08:02.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:08:02 smithi150 ceph-mon[34825]: scrub summary: idle 2024-03-28T16:08:02.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:08:02 smithi067 ceph-mon[31707]: pgmap v573: 97 pgs: 97 active+clean; 6.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 192 MiB/s wr, 51 op/s 2024-03-28T16:08:02.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:08:02 smithi067 ceph-mon[31707]: scrub summary: idle 2024-03-28T16:08:02.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:08:02 smithi183 ceph-mon[35580]: pgmap v573: 97 pgs: 97 active+clean; 6.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 192 MiB/s wr, 51 op/s 2024-03-28T16:08:02.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:08:02 smithi183 ceph-mon[35580]: scrub summary: idle 2024-03-28T16:08:04.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:08:04 smithi067 ceph-mon[31707]: pgmap v574: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 194 MiB/s wr, 50 op/s 2024-03-28T16:08:04.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:08:04 smithi183 ceph-mon[35580]: pgmap v574: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 194 MiB/s wr, 50 op/s 2024-03-28T16:08:04.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:08:04 smithi150 ceph-mon[34825]: pgmap v574: 97 pgs: 97 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 194 MiB/s wr, 50 op/s 2024-03-28T16:08:06.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:08:06 smithi067 ceph-mon[31707]: pgmap v575: 97 pgs: 97 active+clean; 7.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 200 MiB/s wr, 53 op/s 2024-03-28T16:08:06.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:08:06 smithi183 ceph-mon[35580]: pgmap v575: 97 pgs: 97 active+clean; 7.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 200 MiB/s wr, 53 op/s 2024-03-28T16:08:06.605 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:08:06 smithi150 ceph-mon[34825]: pgmap v575: 97 pgs: 97 active+clean; 7.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 200 MiB/s wr, 53 op/s 2024-03-28T16:08:08.569 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:08:08 smithi067 ceph-mon[31707]: pgmap v576: 97 pgs: 97 active+clean; 7.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 180 MiB/s wr, 47 op/s 2024-03-28T16:08:08.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:08:08 smithi183 ceph-mon[35580]: pgmap v576: 97 pgs: 97 active+clean; 7.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 180 MiB/s wr, 47 op/s 2024-03-28T16:08:08.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:08:08 smithi150 ceph-mon[34825]: pgmap v576: 97 pgs: 97 active+clean; 7.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 180 MiB/s wr, 47 op/s 2024-03-28T16:08:10.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:08:10 smithi183 ceph-mon[35580]: pgmap v577: 97 pgs: 97 active+clean; 8.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 175 MiB/s wr, 45 op/s 2024-03-28T16:08:10.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:08:10 smithi150 ceph-mon[34825]: pgmap v577: 97 pgs: 97 active+clean; 8.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 175 MiB/s wr, 45 op/s 2024-03-28T16:08:10.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:08:10 smithi067 ceph-mon[31707]: pgmap v577: 97 pgs: 97 active+clean; 8.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 175 MiB/s wr, 45 op/s 2024-03-28T16:08:12.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:08:12 smithi183 ceph-mon[35580]: pgmap v578: 97 pgs: 97 active+clean; 8.3 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 173 MiB/s wr, 46 op/s 2024-03-28T16:08:12.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:08:12 smithi150 ceph-mon[34825]: pgmap v578: 97 pgs: 97 active+clean; 8.3 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 173 MiB/s wr, 46 op/s 2024-03-28T16:08:12.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:08:12 smithi067 ceph-mon[31707]: pgmap v578: 97 pgs: 97 active+clean; 8.3 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 173 MiB/s wr, 46 op/s 2024-03-28T16:08:14.600 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:08:14 smithi183 ceph-mon[35580]: pgmap v579: 97 pgs: 97 active+clean; 8.7 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 149 MiB/s wr, 38 op/s 2024-03-28T16:08:14.803 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:08:14 smithi150 ceph-mon[34825]: pgmap v579: 97 pgs: 97 active+clean; 8.7 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 149 MiB/s wr, 38 op/s 2024-03-28T16:08:14.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:08:14 smithi067 ceph-mon[31707]: pgmap v579: 97 pgs: 97 active+clean; 8.7 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 149 MiB/s wr, 38 op/s 2024-03-28T16:08:16.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:08:16 smithi150 ceph-mon[34825]: pgmap v580: 97 pgs: 97 active+clean; 9.0 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 143 MiB/s wr, 38 op/s 2024-03-28T16:08:16.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:08:16 smithi067 ceph-mon[31707]: pgmap v580: 97 pgs: 97 active+clean; 9.0 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 143 MiB/s wr, 38 op/s 2024-03-28T16:08:16.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:08:16 smithi183 ceph-mon[35580]: pgmap v580: 97 pgs: 97 active+clean; 9.0 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 143 MiB/s wr, 38 op/s 2024-03-28T16:08:18.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:08:18 smithi150 ceph-mon[34825]: pgmap v581: 97 pgs: 97 active+clean; 9.1 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 127 MiB/s wr, 33 op/s 2024-03-28T16:08:18.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:08:18 smithi067 ceph-mon[31707]: pgmap v581: 97 pgs: 97 active+clean; 9.1 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 127 MiB/s wr, 33 op/s 2024-03-28T16:08:18.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:08:18 smithi183 ceph-mon[35580]: pgmap v581: 97 pgs: 97 active+clean; 9.1 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 127 MiB/s wr, 33 op/s 2024-03-28T16:08:20.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:08:20 smithi150 ceph-mon[34825]: pgmap v582: 97 pgs: 97 active+clean; 9.4 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 139 MiB/s wr, 36 op/s 2024-03-28T16:08:20.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:08:20 smithi067 ceph-mon[31707]: pgmap v582: 97 pgs: 97 active+clean; 9.4 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 139 MiB/s wr, 36 op/s 2024-03-28T16:08:20.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:08:20 smithi183 ceph-mon[35580]: pgmap v582: 97 pgs: 97 active+clean; 9.4 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 139 MiB/s wr, 36 op/s 2024-03-28T16:08:21.119 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-03-28T16:08:21.119 INFO:tasks.workunit.client.0.smithi067.stdout: Children see throughput for 1 initial writers = 162716.39 kB/sec 2024-03-28T16:08:21.119 INFO:tasks.workunit.client.0.smithi067.stdout: Parent sees throughput for 1 initial writers = 162712.08 kB/sec 2024-03-28T16:08:21.119 INFO:tasks.workunit.client.0.smithi067.stdout: Min throughput per process = 162716.39 kB/sec 2024-03-28T16:08:21.119 INFO:tasks.workunit.client.0.smithi067.stdout: Max throughput per process = 162716.39 kB/sec 2024-03-28T16:08:21.119 INFO:tasks.workunit.client.0.smithi067.stdout: Avg throughput per process = 162716.39 kB/sec 2024-03-28T16:08:21.120 INFO:tasks.workunit.client.0.smithi067.stdout: Min xfer = 10485760.00 kB 2024-03-28T16:08:22.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:08:22 smithi150 ceph-mon[34825]: pgmap v583: 97 pgs: 97 active+clean; 9.7 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 148 MiB/s wr, 39 op/s 2024-03-28T16:08:22.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:08:22 smithi067 ceph-mon[31707]: pgmap v583: 97 pgs: 97 active+clean; 9.7 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 148 MiB/s wr, 39 op/s 2024-03-28T16:08:22.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:08:22 smithi183 ceph-mon[35580]: pgmap v583: 97 pgs: 97 active+clean; 9.7 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 148 MiB/s wr, 39 op/s 2024-03-28T16:08:24.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:08:24 smithi150 ceph-mon[34825]: pgmap v584: 97 pgs: 97 active+clean; 9.9 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 134 MiB/s wr, 35 op/s 2024-03-28T16:08:24.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:08:24 smithi067 ceph-mon[31707]: pgmap v584: 97 pgs: 97 active+clean; 9.9 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 134 MiB/s wr, 35 op/s 2024-03-28T16:08:24.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:08:24 smithi183 ceph-mon[35580]: pgmap v584: 97 pgs: 97 active+clean; 9.9 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 134 MiB/s wr, 35 op/s 2024-03-28T16:08:26.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:08:26 smithi150 ceph-mon[34825]: pgmap v585: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 127 MiB/s wr, 34 op/s 2024-03-28T16:08:26.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:08:26 smithi067 ceph-mon[31707]: pgmap v585: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 127 MiB/s wr, 34 op/s 2024-03-28T16:08:26.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:08:26 smithi183 ceph-mon[35580]: pgmap v585: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 127 MiB/s wr, 34 op/s 2024-03-28T16:08:28.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:08:28 smithi150 ceph-mon[34825]: pgmap v586: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 121 MiB/s wr, 32 op/s 2024-03-28T16:08:28.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:08:28 smithi067 ceph-mon[31707]: pgmap v586: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 121 MiB/s wr, 32 op/s 2024-03-28T16:08:28.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:08:28 smithi183 ceph-mon[35580]: pgmap v586: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 121 MiB/s wr, 32 op/s 2024-03-28T16:08:29.633 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T16:08:29.634 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T16:08:29.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.923+0000 7fb342c00640 1 -- 172.21.15.67:0/2235066853 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb33c102120 msgr2=0x7fb33c104510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:08:29.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.923+0000 7fb342c00640 1 --2- 172.21.15.67:0/2235066853 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb33c102120 0x7fb33c104510 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fb32c009900 tx=0x7fb32c02bc30 comp rx=0 tx=0).stop 2024-03-28T16:08:29.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.923+0000 7fb342c00640 1 -- 172.21.15.67:0/2235066853 shutdown_connections 2024-03-28T16:08:29.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.923+0000 7fb342c00640 1 --2- 172.21.15.67:0/2235066853 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb33c107380 0x7fb33c1097d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:29.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.923+0000 7fb342c00640 1 --2- 172.21.15.67:0/2235066853 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb33c104a50 0x7fb33c106e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:29.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.923+0000 7fb342c00640 1 --2- 172.21.15.67:0/2235066853 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb33c102120 0x7fb33c104510 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:29.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.923+0000 7fb342c00640 1 -- 172.21.15.67:0/2235066853 >> 172.21.15.67:0/2235066853 conn(0x7fb33c0fbb30 msgr2=0x7fb33c0fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:08:29.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.923+0000 7fb342c00640 1 -- 172.21.15.67:0/2235066853 shutdown_connections 2024-03-28T16:08:29.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.923+0000 7fb342c00640 1 -- 172.21.15.67:0/2235066853 wait complete. 2024-03-28T16:08:29.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.925+0000 7fb342c00640 1 Processor -- start 2024-03-28T16:08:29.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.925+0000 7fb342c00640 1 -- start start 2024-03-28T16:08:29.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.926+0000 7fb342c00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb33c102120 0x7fb33c199b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:29.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.926+0000 7fb342c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb33c104a50 0x7fb33c19a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:29.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.926+0000 7fb342c00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb33c107380 0x7fb33c1a1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:29.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.926+0000 7fb342c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb33c10d230 con 0x7fb33c104a50 2024-03-28T16:08:29.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.926+0000 7fb342c00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb33c10d0b0 con 0x7fb33c107380 2024-03-28T16:08:29.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.926+0000 7fb342c00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb33c10d3b0 con 0x7fb33c102120 2024-03-28T16:08:29.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.926+0000 7fb33b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb33c104a50 0x7fb33c19a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:08:29.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.926+0000 7fb33b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb33c104a50 0x7fb33c19a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46232/0 (socket says 172.21.15.67:46232) 2024-03-28T16:08:29.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.926+0000 7fb33b400640 1 -- 172.21.15.67:0/3237798689 learned_addr learned my addr 172.21.15.67:0/3237798689 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:08:29.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.927+0000 7fb340c00640 1 --2- 172.21.15.67:0/3237798689 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb33c107380 0x7fb33c1a1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:08:29.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.928+0000 7fb33be00640 1 --2- 172.21.15.67:0/3237798689 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb33c102120 0x7fb33c199b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:08:29.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.928+0000 7fb33b400640 1 -- 172.21.15.67:0/3237798689 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb33c102120 msgr2=0x7fb33c199b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:08:29.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.928+0000 7fb33b400640 1 --2- 172.21.15.67:0/3237798689 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb33c102120 0x7fb33c199b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:29.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.928+0000 7fb33b400640 1 -- 172.21.15.67:0/3237798689 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb33c107380 msgr2=0x7fb33c1a1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:08:29.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.928+0000 7fb33b400640 1 --2- 172.21.15.67:0/3237798689 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb33c107380 0x7fb33c1a1100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:29.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.928+0000 7fb33b400640 1 -- 172.21.15.67:0/3237798689 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb33c1a1770 con 0x7fb33c104a50 2024-03-28T16:08:29.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.929+0000 7fb33b400640 1 --2- 172.21.15.67:0/3237798689 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb33c104a50 0x7fb33c19a090 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fb328005fd0 tx=0x7fb32800dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:08:29.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.929+0000 7fb338c00640 1 -- 172.21.15.67:0/3237798689 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb328017070 con 0x7fb33c104a50 2024-03-28T16:08:29.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.929+0000 7fb342c00640 1 -- 172.21.15.67:0/3237798689 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb33c1a1a60 con 0x7fb33c104a50 2024-03-28T16:08:29.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.929+0000 7fb342c00640 1 -- 172.21.15.67:0/3237798689 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb33c1a2000 con 0x7fb33c104a50 2024-03-28T16:08:29.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.930+0000 7fb338c00640 1 -- 172.21.15.67:0/3237798689 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb328009c30 con 0x7fb33c104a50 2024-03-28T16:08:29.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.930+0000 7fb338c00640 1 -- 172.21.15.67:0/3237798689 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb328003d40 con 0x7fb33c104a50 2024-03-28T16:08:29.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.934+0000 7fb3459ab740 1 Processor -- start 2024-03-28T16:08:29.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.934+0000 7fb3459ab740 1 -- start start 2024-03-28T16:08:29.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.936+0000 7fb3459ab740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5586da646e80 0x5586da6472a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:29.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.936+0000 7fb3459ab740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5586da647870 0x5586da64c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:29.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.936+0000 7fb3459ab740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5586da64c9e0 0x5586da64ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:29.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.936+0000 7fb3459ab740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5586da5ac250 con 0x5586da646e80 2024-03-28T16:08:29.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.936+0000 7fb3459ab740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5586da554a30 con 0x5586da64c9e0 2024-03-28T16:08:29.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.936+0000 7fb3459ab740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x5586da5a82f0 con 0x5586da647870 2024-03-28T16:08:29.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.936+0000 7fb338c00640 1 -- 172.21.15.67:0/3237798689 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fb328004020 con 0x7fb33c104a50 2024-03-28T16:08:29.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.937+0000 7fb340c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5586da646e80 0x5586da6472a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:08:29.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.937+0000 7fb340c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5586da646e80 0x5586da6472a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46238/0 (socket says 172.21.15.67:46238) 2024-03-28T16:08:29.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.937+0000 7fb340c00640 1 -- 172.21.15.67:0/3792079281 learned_addr learned my addr 172.21.15.67:0/3792079281 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:08:29.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.937+0000 7fb33b400640 1 --2- 172.21.15.67:0/3792079281 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5586da64c9e0 0x5586da64ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:08:29.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.938+0000 7fb33be00640 1 --2- 172.21.15.67:0/3792079281 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5586da647870 0x5586da64c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:08:29.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.938+0000 7fb340c00640 1 -- 172.21.15.67:0/3792079281 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5586da647870 msgr2=0x5586da64c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:08:29.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.938+0000 7fb340c00640 1 --2- 172.21.15.67:0/3792079281 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5586da647870 0x5586da64c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:29.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.938+0000 7fb340c00640 1 -- 172.21.15.67:0/3792079281 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5586da64c9e0 msgr2=0x5586da64ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:08:29.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.938+0000 7fb340c00640 1 --2- 172.21.15.67:0/3792079281 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5586da64c9e0 0x5586da64ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:29.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.938+0000 7fb340c00640 1 -- 172.21.15.67:0/3792079281 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5586da53f690 con 0x5586da646e80 2024-03-28T16:08:29.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.938+0000 7fb340c00640 1 --2- 172.21.15.67:0/3792079281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5586da646e80 0x5586da6472a0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fb334009d70 tx=0x7fb33400f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5017d57bd9238299 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:08:29.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.939+0000 7fb331200640 1 -- 172.21.15.67:0/3792079281 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb334003970 con 0x5586da646e80 2024-03-28T16:08:29.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.939+0000 7fb331200640 1 -- 172.21.15.67:0/3792079281 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb334004030 con 0x5586da646e80 2024-03-28T16:08:29.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.939+0000 7fb331200640 1 -- 172.21.15.67:0/3792079281 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb334011730 con 0x5586da646e80 2024-03-28T16:08:29.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.940+0000 7fb3459ab740 1 -- 172.21.15.67:0/3792079281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5586da646e80 msgr2=0x5586da6472a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:08:29.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.940+0000 7fb3459ab740 1 --2- 172.21.15.67:0/3792079281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5586da646e80 0x5586da6472a0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fb334009d70 tx=0x7fb33400f730 comp rx=0 tx=0).stop 2024-03-28T16:08:29.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.940+0000 7fb3459ab740 1 -- 172.21.15.67:0/3792079281 shutdown_connections 2024-03-28T16:08:29.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.940+0000 7fb3459ab740 1 --2- 172.21.15.67:0/3792079281 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5586da64c9e0 0x5586da64ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:29.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.940+0000 7fb3459ab740 1 --2- 172.21.15.67:0/3792079281 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5586da647870 0x5586da64c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:29.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.940+0000 7fb3459ab740 1 --2- 172.21.15.67:0/3792079281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5586da646e80 0x5586da6472a0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:29.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.940+0000 7fb3459ab740 1 -- 172.21.15.67:0/3792079281 >> 172.21.15.67:0/3792079281 conn(0x5586da60f200 msgr2=0x5586da6427f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:08:29.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.941+0000 7fb3459ab740 1 -- 172.21.15.67:0/3792079281 shutdown_connections 2024-03-28T16:08:29.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.941+0000 7fb3459ab740 1 -- 172.21.15.67:0/3792079281 wait complete. 2024-03-28T16:08:29.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.941+0000 7fb338c00640 1 --2- 172.21.15.67:0/3237798689 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb3200a2790 0x7fb3200a4c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:29.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.941+0000 7fb338c00640 1 -- 172.21.15.67:0/3237798689 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fb3280ce6a0 con 0x7fb33c104a50 2024-03-28T16:08:29.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.941+0000 7fb3459ab740 1 -- start start 2024-03-28T16:08:29.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.941+0000 7fb3459ab740 1 Processor -- start 2024-03-28T16:08:29.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.942+0000 7fb33be00640 1 --2- 172.21.15.67:0/3237798689 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb3200a2790 0x7fb3200a4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:08:29.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.942+0000 7fb3459ab740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5586da646e80 0x5586da6596b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:29.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.942+0000 7fb3459ab740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5586da647870 0x5586da659bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:29.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.942+0000 7fb3459ab740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5586da64c9e0 0x5586da660be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:29.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.942+0000 7fb3459ab740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5586da6581a0 con 0x5586da646e80 2024-03-28T16:08:29.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.942+0000 7fb3459ab740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5586da567e80 con 0x5586da647870 2024-03-28T16:08:29.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.942+0000 7fb3459ab740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x5586da658320 con 0x5586da64c9e0 2024-03-28T16:08:29.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.942+0000 7fb3459ab740 10 client.0 _ll_register_callbacks cb 0x5586da57a130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:08:29.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.943+0000 7fb33be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5586da646e80 0x5586da6596b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:08:29.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.943+0000 7fb33be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5586da646e80 0x5586da6596b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46240/0 (socket says 172.21.15.67:46240) 2024-03-28T16:08:29.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.943+0000 7fb33be00640 1 -- 172.21.15.67:0/3526810197 learned_addr learned my addr 172.21.15.67:0/3526810197 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:08:29.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.943+0000 7fb33b400640 1 --2- 172.21.15.67:0/3526810197 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5586da647870 0x5586da659bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:08:29.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.943+0000 7fb33be00640 1 --2- 172.21.15.67:0/3237798689 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb3200a2790 0x7fb3200a4c50 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fb32c006010 tx=0x7fb32c02c8e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:08:29.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.943+0000 7fb33be00640 1 -- 172.21.15.67:0/3526810197 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5586da64c9e0 msgr2=0x5586da660be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:08:29.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.943+0000 7fb33be00640 1 --2- 172.21.15.67:0/3526810197 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5586da64c9e0 0x5586da660be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:29.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.943+0000 7fb33be00640 1 -- 172.21.15.67:0/3526810197 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5586da647870 msgr2=0x5586da659bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:08:29.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.943+0000 7fb33be00640 1 --2- 172.21.15.67:0/3526810197 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5586da647870 0x5586da659bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:29.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.943+0000 7fb33be00640 1 -- 172.21.15.67:0/3526810197 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5586da38a7d0 con 0x5586da646e80 2024-03-28T16:08:29.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.944+0000 7fb33be00640 1 --2- 172.21.15.67:0/3526810197 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5586da646e80 0x5586da6596b0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fb32c009230 tx=0x7fb32c009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3aa2c3c83d68478 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:08:29.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.944+0000 7fb325600640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:08:29.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.944+0000 7fb325600640 1 -- 172.21.15.67:0/3526810197 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb32c050070 con 0x5586da646e80 2024-03-28T16:08:29.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.944+0000 7fb325600640 1 -- 172.21.15.67:0/3526810197 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fb32c0048c0 con 0x5586da646e80 2024-03-28T16:08:29.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.945+0000 7fb3459ab740 1 -- 172.21.15.67:0/3526810197 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb334004030 con 0x5586da646e80 2024-03-28T16:08:29.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.945+0000 7fb325600640 1 -- 172.21.15.67:0/3526810197 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb32c005650 con 0x5586da646e80 2024-03-28T16:08:29.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.945+0000 7fb3459ab740 10 client.16017 fetch_fsmap learned FSMap version 19 2024-03-28T16:08:29.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.945+0000 7fb3459ab740 1 -- 172.21.15.67:0/3526810197 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5586da2c03f0 con 0x5586da646e80 2024-03-28T16:08:29.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.946+0000 7fb325600640 1 -- 172.21.15.67:0/3526810197 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb334004030 con 0x5586da646e80 2024-03-28T16:08:29.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.946+0000 7fb325600640 1 -- 172.21.15.67:0/3526810197 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fb32c003c20 con 0x5586da646e80 2024-03-28T16:08:29.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.946+0000 7fb3459ab740 10 client.16017 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:08:29.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.946+0000 7fb3459ab740 10 client.16017 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:08:29.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.946+0000 7fb3459ab740 20 client.16017 populate_metadata read hostname 'smithi067' 2024-03-28T16:08:29.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.946+0000 7fb3459ab740 1 --2- 172.21.15.67:0/3526810197 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5586da653a90 0x5586da655f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:29.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.946+0000 7fb3459ab740 4 client.16017 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:08:29.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.946+0000 7fb3459ab740 1 -- 172.21.15.67:0/3526810197 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5586da5b9790 con 0x5586da653a90 2024-03-28T16:08:29.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.947+0000 7fb33b400640 1 --2- 172.21.15.67:0/3526810197 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5586da653a90 0x5586da655f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:08:29.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.947+0000 7fb33b400640 1 --2- 172.21.15.67:0/3526810197 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5586da653a90 0x5586da655f50 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=922241dec59e88e7 server_cookie=cd8ee7150b91ee69 in_seq=0 out_seq=0 2024-03-28T16:08:29.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.947+0000 7fb325600640 10 client.16017 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:08:29.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.949+0000 7fb325600640 1 -- 172.21.15.67:0/3526810197 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5586da5b9790 con 0x5586da653a90 2024-03-28T16:08:29.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.949+0000 7fb325600640 10 client.16017 handle_command_reply: tid=0 2024-03-28T16:08:29.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.949+0000 7fb3459ab740 1 client.16017 shutdown 2024-03-28T16:08:29.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.950+0000 7fb3459ab740 1 -- 172.21.15.67:0/3526810197 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5586da646e80 msgr2=0x5586da6596b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:08:29.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.950+0000 7fb3459ab740 1 --2- 172.21.15.67:0/3526810197 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5586da646e80 0x5586da6596b0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fb32c009230 tx=0x7fb32c009260 comp rx=0 tx=0).stop 2024-03-28T16:08:29.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.950+0000 7fb3459ab740 1 -- 172.21.15.67:0/3526810197 shutdown_connections 2024-03-28T16:08:29.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.950+0000 7fb3459ab740 1 --2- 172.21.15.67:0/3526810197 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5586da653a90 0x5586da655f50 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:29.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.950+0000 7fb3459ab740 1 --2- 172.21.15.67:0/3526810197 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5586da64c9e0 0x5586da660be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:29.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.950+0000 7fb3459ab740 1 --2- 172.21.15.67:0/3526810197 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5586da647870 0x5586da659bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:29.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.950+0000 7fb3459ab740 1 --2- 172.21.15.67:0/3526810197 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5586da646e80 0x5586da6596b0 secure :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fb32c009230 tx=0x7fb32c009260 comp rx=0 tx=0).stop 2024-03-28T16:08:29.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.950+0000 7fb3459ab740 1 -- 172.21.15.67:0/3526810197 >> 172.21.15.67:0/3526810197 conn(0x5586da60f200 msgr2=0x5586da641a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:08:29.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.950+0000 7fb325600640 0 client.16017 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:08:29.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.951+0000 7fb3459ab740 1 -- 172.21.15.67:0/3526810197 shutdown_connections 2024-03-28T16:08:29.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.951+0000 7fb3459ab740 1 -- 172.21.15.67:0/3526810197 wait complete. 2024-03-28T16:08:29.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.951+0000 7fb3459ab740 20 client.16017 trim_cache size 0 max 16384 2024-03-28T16:08:29.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.984+0000 7fb3459ab740 1 Processor -- start 2024-03-28T16:08:29.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.984+0000 7fb3459ab740 1 -- start start 2024-03-28T16:08:29.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.985+0000 7fb3459ab740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5586da647870 0x5586da6489f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:29.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.985+0000 7fb3459ab740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5586da64c9e0 0x5586da660ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:29.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.985+0000 7fb3459ab740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5586da648f30 0x5586da645a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:29.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.985+0000 7fb3459ab740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb31c00c3c0 con 0x5586da648f30 2024-03-28T16:08:29.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.985+0000 7fb3459ab740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb31c00e980 con 0x5586da64c9e0 2024-03-28T16:08:29.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.985+0000 7fb3459ab740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb31c00c9c0 con 0x5586da647870 2024-03-28T16:08:29.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.985+0000 7fb33b400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5586da64c9e0 0x5586da660ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:08:29.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.985+0000 7fb33b400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5586da64c9e0 0x5586da660ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.67:36318/0 (socket says 172.21.15.67:36318) 2024-03-28T16:08:29.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.985+0000 7fb33b400640 1 -- 172.21.15.67:0/434848946 learned_addr learned my addr 172.21.15.67:0/434848946 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:08:29.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.985+0000 7fb340c00640 1 --2- 172.21.15.67:0/434848946 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5586da648f30 0x5586da645a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:08:29.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.986+0000 7fb33be00640 1 --2- 172.21.15.67:0/434848946 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5586da647870 0x5586da6489f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:08:29.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.986+0000 7fb33b400640 1 -- 172.21.15.67:0/434848946 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5586da647870 msgr2=0x5586da6489f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:08:29.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.986+0000 7fb33b400640 1 --2- 172.21.15.67:0/434848946 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5586da647870 0x5586da6489f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:29.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.986+0000 7fb33b400640 1 -- 172.21.15.67:0/434848946 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5586da648f30 msgr2=0x5586da645a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:08:29.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.986+0000 7fb33b400640 1 --2- 172.21.15.67:0/434848946 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5586da648f30 0x5586da645a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:29.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.986+0000 7fb33b400640 1 -- 172.21.15.67:0/434848946 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5586da2800f0 con 0x5586da64c9e0 2024-03-28T16:08:29.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.986+0000 7fb33be00640 1 --2- 172.21.15.67:0/434848946 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5586da647870 0x5586da6489f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T16:08:29.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.987+0000 7fb33b400640 1 --2- 172.21.15.67:0/434848946 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5586da64c9e0 0x5586da660ba0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb3280038f0 tx=0x7fb3280bb040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82cabbf38ccac3f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:08:29.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.987+0000 7fb327e00640 1 -- 172.21.15.67:0/434848946 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb32809b050 con 0x5586da64c9e0 2024-03-28T16:08:29.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.987+0000 7fb327e00640 1 -- 172.21.15.67:0/434848946 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb32809f1a0 con 0x5586da64c9e0 2024-03-28T16:08:29.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.987+0000 7fb327e00640 1 -- 172.21.15.67:0/434848946 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb328022d60 con 0x5586da64c9e0 2024-03-28T16:08:29.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.988+0000 7fb3459ab740 1 -- 172.21.15.67:0/434848946 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5586da64c9e0 msgr2=0x5586da660ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:08:29.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.988+0000 7fb3459ab740 1 --2- 172.21.15.67:0/434848946 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5586da64c9e0 0x5586da660ba0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb3280038f0 tx=0x7fb3280bb040 comp rx=0 tx=0).stop 2024-03-28T16:08:29.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.988+0000 7fb3459ab740 1 -- 172.21.15.67:0/434848946 shutdown_connections 2024-03-28T16:08:29.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.988+0000 7fb3459ab740 1 --2- 172.21.15.67:0/434848946 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5586da648f30 0x5586da645a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:29.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.988+0000 7fb3459ab740 1 --2- 172.21.15.67:0/434848946 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5586da64c9e0 0x5586da660ba0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:29.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.988+0000 7fb3459ab740 1 --2- 172.21.15.67:0/434848946 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5586da647870 0x5586da6489f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:29.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.988+0000 7fb3459ab740 1 -- 172.21.15.67:0/434848946 >> 172.21.15.67:0/434848946 conn(0x5586da60f200 msgr2=0x5586da6482e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:08:29.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.989+0000 7fb3459ab740 1 -- 172.21.15.67:0/434848946 shutdown_connections 2024-03-28T16:08:29.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.989+0000 7fb3459ab740 1 -- 172.21.15.67:0/434848946 wait complete. 2024-03-28T16:08:29.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.989+0000 7fb3459ab740 1 -- start start 2024-03-28T16:08:29.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.990+0000 7fb3459ab740 1 Processor -- start 2024-03-28T16:08:29.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.990+0000 7fb3459ab740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5586da647870 0x5586da6607a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:29.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.990+0000 7fb3459ab740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5586da648f30 0x5586da65d350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:29.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.990+0000 7fb3459ab740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5586da64c9e0 0x5586da65d890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:29.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.990+0000 7fb3459ab740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb31c00d000 con 0x5586da64c9e0 2024-03-28T16:08:29.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.990+0000 7fb3459ab740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb31c00e320 con 0x5586da648f30 2024-03-28T16:08:29.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.990+0000 7fb3459ab740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb31c00d660 con 0x5586da647870 2024-03-28T16:08:29.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.990+0000 7fb3459ab740 10 client.0 _ll_register_callbacks cb 0x7fb32c045050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:08:29.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.991+0000 7fb33b400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5586da648f30 0x5586da65d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:08:29.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.991+0000 7fb33b400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5586da648f30 0x5586da65d350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.67:36324/0 (socket says 172.21.15.67:36324) 2024-03-28T16:08:29.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.991+0000 7fb33b400640 1 -- 172.21.15.67:0/1565084340 learned_addr learned my addr 172.21.15.67:0/1565084340 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:08:29.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.991+0000 7fb340c00640 1 --2- 172.21.15.67:0/1565084340 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5586da64c9e0 0x5586da65d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:08:29.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.991+0000 7fb33be00640 1 --2- 172.21.15.67:0/1565084340 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5586da647870 0x5586da6607a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:08:29.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.991+0000 7fb33b400640 1 -- 172.21.15.67:0/1565084340 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5586da647870 msgr2=0x5586da6607a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:08:29.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.991+0000 7fb33b400640 1 --2- 172.21.15.67:0/1565084340 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5586da647870 0x5586da6607a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:29.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.991+0000 7fb33b400640 1 -- 172.21.15.67:0/1565084340 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5586da64c9e0 msgr2=0x5586da65d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:08:29.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.991+0000 7fb33b400640 1 --2- 172.21.15.67:0/1565084340 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5586da64c9e0 0x5586da65d890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:29.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.991+0000 7fb33b400640 1 -- 172.21.15.67:0/1565084340 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5586da65ddd0 con 0x5586da648f30 2024-03-28T16:08:29.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.992+0000 7fb33b400640 1 --2- 172.21.15.67:0/1565084340 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5586da648f30 0x5586da65d350 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb328000c00 tx=0x7fb3280ce9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=676e763b6c70520a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:08:29.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.992+0000 7fb325600640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-28T16:08:29.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.992+0000 7fb325600640 1 -- 172.21.15.67:0/1565084340 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb32809b050 con 0x5586da648f30 2024-03-28T16:08:29.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.992+0000 7fb325600640 1 -- 172.21.15.67:0/1565084340 <== mon.2 v2:172.21.15.150:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fb328022d60 con 0x5586da648f30 2024-03-28T16:08:29.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.992+0000 7fb3459ab740 1 -- 172.21.15.67:0/1565084340 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb32809f1a0 con 0x5586da648f30 2024-03-28T16:08:29.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.992+0000 7fb325600640 1 -- 172.21.15.67:0/1565084340 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb3280a37f0 con 0x5586da648f30 2024-03-28T16:08:29.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.992+0000 7fb3459ab740 10 client.35528 fetch_fsmap learned FSMap version 19 2024-03-28T16:08:29.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.992+0000 7fb3459ab740 1 -- 172.21.15.67:0/1565084340 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5586da6c9d40 con 0x5586da648f30 2024-03-28T16:08:29.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.993+0000 7fb325600640 1 -- 172.21.15.67:0/1565084340 <== mon.2 v2:172.21.15.150:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb32809f1a0 con 0x5586da648f30 2024-03-28T16:08:29.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.993+0000 7fb325600640 1 -- 172.21.15.67:0/1565084340 <== mon.2 v2:172.21.15.150:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fb3280d0bb0 con 0x5586da648f30 2024-03-28T16:08:29.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.993+0000 7fb3459ab740 10 client.35528 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:08:29.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.993+0000 7fb3459ab740 10 client.35528 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:08:29.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.993+0000 7fb3459ab740 20 client.35528 populate_metadata read hostname 'smithi067' 2024-03-28T16:08:29.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.993+0000 7fb3459ab740 1 --2- 172.21.15.67:0/1565084340 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5586da6ca5d0 0x5586da6cc9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:29.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.993+0000 7fb3459ab740 4 client.35528 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T16:08:29.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.993+0000 7fb3459ab740 1 -- 172.21.15.67:0/1565084340 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5586da5b3ae0 con 0x5586da6ca5d0 2024-03-28T16:08:29.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.995+0000 7fb33be00640 1 --2- 172.21.15.67:0/1565084340 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5586da6ca5d0 0x5586da6cc9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:08:29.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.998+0000 7fb33be00640 1 --2- 172.21.15.67:0/1565084340 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5586da6ca5d0 0x5586da6cc9f0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=857b581eb3c168c7 server_cookie=504f823ee6a33c30 in_seq=0 out_seq=0 2024-03-28T16:08:29.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.998+0000 7fb325600640 10 client.35528 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:08:30.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.999+0000 7fb325600640 1 -- 172.21.15.67:0/1565084340 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5586da5b3ae0 con 0x5586da6ca5d0 2024-03-28T16:08:30.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.999+0000 7fb325600640 10 client.35528 handle_command_reply: tid=0 2024-03-28T16:08:30.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.999+0000 7fb3459ab740 1 client.35528 shutdown 2024-03-28T16:08:30.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.999+0000 7fb3459ab740 1 -- 172.21.15.67:0/1565084340 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5586da648f30 msgr2=0x5586da65d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:08:30.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.999+0000 7fb3459ab740 1 --2- 172.21.15.67:0/1565084340 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5586da648f30 0x5586da65d350 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb328000c00 tx=0x7fb3280ce9a0 comp rx=0 tx=0).stop 2024-03-28T16:08:30.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.999+0000 7fb3459ab740 1 -- 172.21.15.67:0/1565084340 shutdown_connections 2024-03-28T16:08:30.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.999+0000 7fb3459ab740 1 --2- 172.21.15.67:0/1565084340 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5586da6ca5d0 0x5586da6cc9f0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:30.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.999+0000 7fb3459ab740 1 --2- 172.21.15.67:0/1565084340 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5586da64c9e0 0x5586da65d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:30.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.999+0000 7fb3459ab740 1 --2- 172.21.15.67:0/1565084340 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5586da648f30 0x5586da65d350 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:30.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.999+0000 7fb3459ab740 1 --2- 172.21.15.67:0/1565084340 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5586da647870 0x5586da6607a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:30.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.999+0000 7fb325600640 0 client.35528 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:08:30.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.999+0000 7fb3459ab740 1 -- 172.21.15.67:0/1565084340 >> 172.21.15.67:0/1565084340 conn(0x5586da60f200 msgr2=0x5586da641200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:08:30.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.999+0000 7fb3459ab740 1 -- 172.21.15.67:0/1565084340 shutdown_connections 2024-03-28T16:08:30.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:29.999+0000 7fb3459ab740 1 -- 172.21.15.67:0/1565084340 wait complete. 2024-03-28T16:08:30.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:30.000+0000 7fb3459ab740 20 client.35528 trim_cache size 0 max 16384 2024-03-28T16:08:30.002 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-28T16:08:30.002 INFO:teuthology.orchestra.run.smithi067.stdout: "status": "no active scrubs running", 2024-03-28T16:08:30.002 INFO:teuthology.orchestra.run.smithi067.stdout: "scrubs": {} 2024-03-28T16:08:30.002 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-28T16:08:30.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:30.000+0000 7fb342c00640 1 -- 172.21.15.67:0/3237798689 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb3200a2790 msgr2=0x7fb3200a4c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:08:30.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:30.000+0000 7fb342c00640 1 --2- 172.21.15.67:0/3237798689 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb3200a2790 0x7fb3200a4c50 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fb32c006010 tx=0x7fb32c02c8e0 comp rx=0 tx=0).stop 2024-03-28T16:08:30.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:30.000+0000 7fb342c00640 1 -- 172.21.15.67:0/3237798689 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb33c104a50 msgr2=0x7fb33c19a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:08:30.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:30.000+0000 7fb342c00640 1 --2- 172.21.15.67:0/3237798689 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb33c104a50 0x7fb33c19a090 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fb328005fd0 tx=0x7fb32800dde0 comp rx=0 tx=0).stop 2024-03-28T16:08:30.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:30.001+0000 7fb342c00640 1 -- 172.21.15.67:0/3237798689 shutdown_connections 2024-03-28T16:08:30.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:30.001+0000 7fb342c00640 1 --2- 172.21.15.67:0/3237798689 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb3200a2790 0x7fb3200a4c50 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:30.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:30.001+0000 7fb342c00640 1 --2- 172.21.15.67:0/3237798689 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb33c107380 0x7fb33c1a1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:30.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:30.001+0000 7fb342c00640 1 --2- 172.21.15.67:0/3237798689 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb33c104a50 0x7fb33c19a090 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:30.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:30.001+0000 7fb342c00640 1 --2- 172.21.15.67:0/3237798689 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb33c102120 0x7fb33c199b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:30.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:30.001+0000 7fb342c00640 1 -- 172.21.15.67:0/3237798689 >> 172.21.15.67:0/3237798689 conn(0x7fb33c0fbb30 msgr2=0x7fb33c1018b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:08:30.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:30.002+0000 7fb342c00640 1 -- 172.21.15.67:0/3237798689 shutdown_connections 2024-03-28T16:08:30.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:30.002+0000 7fb342c00640 1 -- 172.21.15.67:0/3237798689 wait complete. 2024-03-28T16:08:30.037 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T16:08:30.038 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T16:08:30.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:30.313+0000 7fb6b2000640 1 -- 172.21.15.67:0/4030180266 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6ac1028b0 msgr2=0x7fb6ac102d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:08:30.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:30.313+0000 7fb6b2000640 1 --2- 172.21.15.67:0/4030180266 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6ac1028b0 0x7fb6ac102d30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb694009860 tx=0x7fb69402bb70 comp rx=0 tx=0).stop 2024-03-28T16:08:30.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:30.313+0000 7fb6b2000640 1 -- 172.21.15.67:0/4030180266 shutdown_connections 2024-03-28T16:08:30.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:30.313+0000 7fb6b2000640 1 --2- 172.21.15.67:0/4030180266 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6ac103270 0x7fb6ac109b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:30.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:30.313+0000 7fb6b2000640 1 --2- 172.21.15.67:0/4030180266 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6ac1028b0 0x7fb6ac102d30 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:30.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:30.313+0000 7fb6b2000640 1 --2- 172.21.15.67:0/4030180266 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb6ac1016c0 0x7fb6ac101ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:30.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:30.313+0000 7fb6b2000640 1 -- 172.21.15.67:0/4030180266 >> 172.21.15.67:0/4030180266 conn(0x7fb6ac0fcc50 msgr2=0x7fb6ac0ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:08:30.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:30.313+0000 7fb6b2000640 1 -- 172.21.15.67:0/4030180266 shutdown_connections 2024-03-28T16:08:30.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:30.313+0000 7fb6b2000640 1 -- 172.21.15.67:0/4030180266 wait complete. 2024-03-28T16:08:30.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:30.314+0000 7fb6b2000640 1 Processor -- start 2024-03-28T16:08:30.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:30.314+0000 7fb6b2000640 1 -- start start 2024-03-28T16:08:30.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:30.315+0000 7fb6b2000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6ac1016c0 0x7fb6ac075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:30.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:30.315+0000 7fb6b2000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6ac1028b0 0x7fb6ac073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:30.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:30.315+0000 7fb6b2000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb6ac103270 0x7fb6ac073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:30.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:30.315+0000 7fb6b2000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6ac10d1d0 con 0x7fb6ac1028b0 2024-03-28T16:08:30.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:30.315+0000 7fb6b2000640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6ac10d050 con 0x7fb6ac1016c0 2024-03-28T16:08:30.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:30.315+0000 7fb6b2000640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6ac10d350 con 0x7fb6ac103270 2024-03-28T16:08:30.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:30.315+0000 7fb6a2a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6ac1028b0 0x7fb6ac073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:08:30.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:30.315+0000 7fb6ab400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6ac1016c0 0x7fb6ac075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:08:30.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:30.315+0000 7fb6a2a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6ac1028b0 0x7fb6ac073a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46260/0 (socket says 172.21.15.67:46260) 2024-03-28T16:08:30.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:30.315+0000 7fb6a2a00640 1 -- 172.21.15.67:0/655647294 learned_addr learned my addr 172.21.15.67:0/655647294 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:08:30.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:30.315+0000 7fb6ab400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6ac1016c0 0x7fb6ac075450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.67:36342/0 (socket says 172.21.15.67:36342) 2024-03-28T16:08:30.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:30.316+0000 7fb6a2a00640 1 -- 172.21.15.67:0/655647294 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb6ac103270 msgr2=0x7fb6ac073fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:08:30.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:30.316+0000 7fb6a2a00640 1 --2- 172.21.15.67:0/655647294 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb6ac103270 0x7fb6ac073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:30.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:30.316+0000 7fb6a2a00640 1 -- 172.21.15.67:0/655647294 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6ac1016c0 msgr2=0x7fb6ac075450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:08:30.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:30.316+0000 7fb6a2a00640 1 --2- 172.21.15.67:0/655647294 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6ac1016c0 0x7fb6ac075450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:30.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:30.316+0000 7fb6a2a00640 1 -- 172.21.15.67:0/655647294 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6ac074860 con 0x7fb6ac1028b0 2024-03-28T16:08:30.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:30.316+0000 7fb6a2a00640 1 --2- 172.21.15.67:0/655647294 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6ac1028b0 0x7fb6ac073a90 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fb6940057c0 tx=0x7fb6940042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:08:30.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:30.316+0000 7fb6a8c00640 1 -- 172.21.15.67:0/655647294 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb694041030 con 0x7fb6ac1028b0 2024-03-28T16:08:30.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:30.316+0000 7fb6b2000640 1 -- 172.21.15.67:0/655647294 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6ac1a19e0 con 0x7fb6ac1028b0 2024-03-28T16:08:30.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:30.317+0000 7fb6a8c00640 1 -- 172.21.15.67:0/655647294 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb69402c9c0 con 0x7fb6ac1028b0 2024-03-28T16:08:30.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:30.317+0000 7fb6a8c00640 1 -- 172.21.15.67:0/655647294 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb694039040 con 0x7fb6ac1028b0 2024-03-28T16:08:30.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:30.317+0000 7fb6b2000640 1 -- 172.21.15.67:0/655647294 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6ac1a1f80 con 0x7fb6ac1028b0 2024-03-28T16:08:30.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:30.319+0000 7fb6b2000640 1 -- 172.21.15.67:0/655647294 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6ac108aa0 con 0x7fb6ac1028b0 2024-03-28T16:08:30.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:30.321+0000 7fb6a8c00640 1 -- 172.21.15.67:0/655647294 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fb694044070 con 0x7fb6ac1028b0 2024-03-28T16:08:30.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:30.322+0000 7fb6a8c00640 1 --2- 172.21.15.67:0/655647294 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb6900a2880 0x7fb6900a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:30.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:30.323+0000 7fb6a8c00640 1 -- 172.21.15.67:0/655647294 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fb6940f0130 con 0x7fb6ac1028b0 2024-03-28T16:08:30.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:30.323+0000 7fb6ab400640 1 --2- 172.21.15.67:0/655647294 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb6900a2880 0x7fb6900a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:08:30.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:30.324+0000 7fb6ab400640 1 --2- 172.21.15.67:0/655647294 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb6900a2880 0x7fb6900a4d40 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac102710 tx=0x7fb69c00a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:08:30.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:30.331+0000 7fb6a8c00640 1 -- 172.21.15.67:0/655647294 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb6940bd180 con 0x7fb6ac1028b0 2024-03-28T16:08:30.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:30.707+0000 7fb6b2000640 1 -- 172.21.15.67:0/655647294 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb6ac0622a0 con 0x7fb6ac1028b0 2024-03-28T16:08:30.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:30.708+0000 7fb6a8c00640 1 -- 172.21.15.67:0/655647294 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11341 (secure 0 0 0) 0x7fb69402cb90 con 0x7fb6ac1028b0 2024-03-28T16:08:30.710 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:08:30.710 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2008297659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2008297659},{"type":"v1","addr":"172.21.15.67:6835","nonce":2008297659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14532,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/320295745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":320295745},{"type":"v1","addr":"172.21.15.67:6837","nonce":320295745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6839/691821273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":691821273},{"type":"v1","addr":"172.21.15.67:6839","nonce":691821273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6841/3211254875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6840","nonce":3211254875},{"type":"v1","addr":"172.21.15.67:6841","nonce":3211254875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2129665423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2129665423},{"type":"v1","addr":"172.21.15.183:6833","nonce":2129665423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24496,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/1589784425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":1589784425},{"type":"v1","addr":"172.21.15.183:6835","nonce":1589784425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24508,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/2185863835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":2185863835},{"type":"v1","addr":"172.21.15.183:6837","nonce":2185863835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24520,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6839/1242814398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":1242814398},{"type":"v1","addr":"172.21.15.183:6839","nonce":1242814398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":34409,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1008335521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1008335521},{"type":"v1","addr":"172.21.15.150:6833","nonce":1008335521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":34445,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3269935695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3269935695},{"type":"v1","addr":"172.21.15.150:6835","nonce":3269935695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T16:01:07.696824+0000","modified":"2024-03-28T16:01:11.997773+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":34508},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34475":{"gid":34475,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/1112886277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":1112886277},{"type":"v1","addr":"172.21.15.150:6837","nonce":1112886277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_34508":{"gid":34508,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6839/1211011614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":1211011614},{"type":"v1","addr":"172.21.15.150:6839","nonce":1211011614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34508,"qdb_cluster":[34508]},"id":1}]} 2024-03-28T16:08:30.710 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 19 2024-03-28T16:08:30.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:30.714+0000 7fb6b2000640 1 -- 172.21.15.67:0/655647294 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb6900a2880 msgr2=0x7fb6900a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:08:30.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:30.714+0000 7fb6b2000640 1 --2- 172.21.15.67:0/655647294 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb6900a2880 0x7fb6900a4d40 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac102710 tx=0x7fb69c00a2d0 comp rx=0 tx=0).stop 2024-03-28T16:08:30.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:30.714+0000 7fb6b2000640 1 -- 172.21.15.67:0/655647294 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6ac1028b0 msgr2=0x7fb6ac073a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:08:30.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:30.714+0000 7fb6b2000640 1 --2- 172.21.15.67:0/655647294 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6ac1028b0 0x7fb6ac073a90 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fb6940057c0 tx=0x7fb6940042f0 comp rx=0 tx=0).stop 2024-03-28T16:08:30.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:30.714+0000 7fb6b2000640 1 -- 172.21.15.67:0/655647294 shutdown_connections 2024-03-28T16:08:30.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:30.714+0000 7fb6b2000640 1 --2- 172.21.15.67:0/655647294 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb6900a2880 0x7fb6900a4d40 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:30.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:30.714+0000 7fb6b2000640 1 --2- 172.21.15.67:0/655647294 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb6ac103270 0x7fb6ac073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:30.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:30.714+0000 7fb6b2000640 1 --2- 172.21.15.67:0/655647294 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6ac1028b0 0x7fb6ac073a90 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:30.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:30.714+0000 7fb6b2000640 1 --2- 172.21.15.67:0/655647294 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6ac1016c0 0x7fb6ac075450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:30.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:30.714+0000 7fb6b2000640 1 -- 172.21.15.67:0/655647294 >> 172.21.15.67:0/655647294 conn(0x7fb6ac0fcc50 msgr2=0x7fb6ac105c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:08:30.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:30.715+0000 7fb6b2000640 1 -- 172.21.15.67:0/655647294 shutdown_connections 2024-03-28T16:08:30.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:30.715+0000 7fb6b2000640 1 -- 172.21.15.67:0/655647294 wait complete. 2024-03-28T16:08:30.740 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:08:30 smithi067 ceph-mon[31707]: pgmap v587: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 161 MiB/s wr, 42 op/s 2024-03-28T16:08:30.743 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T16:08:30.743 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T16:08:30.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:08:30 smithi150 ceph-mon[34825]: pgmap v587: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 161 MiB/s wr, 42 op/s 2024-03-28T16:08:30.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:08:30 smithi183 ceph-mon[35580]: pgmap v587: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 161 MiB/s wr, 42 op/s 2024-03-28T16:08:31.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.041+0000 7f48cf800640 1 -- 172.21.15.67:0/841884009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48c81038e0 msgr2=0x7f48c8103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:08:31.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.041+0000 7f48cf800640 1 --2- 172.21.15.67:0/841884009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48c81038e0 0x7f48c8103d00 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f48bc009900 tx=0x7f48bc02bc30 comp rx=0 tx=0).stop 2024-03-28T16:08:31.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.042+0000 7f48cf800640 1 -- 172.21.15.67:0/841884009 shutdown_connections 2024-03-28T16:08:31.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.042+0000 7f48cf800640 1 --2- 172.21.15.67:0/841884009 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48c8105490 0x7f48c810bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:31.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.042+0000 7f48cf800640 1 --2- 172.21.15.67:0/841884009 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f48c8104ad0 0x7f48c8104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:31.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.042+0000 7f48cf800640 1 --2- 172.21.15.67:0/841884009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48c81038e0 0x7f48c8103d00 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:31.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.042+0000 7f48cf800640 1 -- 172.21.15.67:0/841884009 >> 172.21.15.67:0/841884009 conn(0x7f48c80fee50 msgr2=0x7f48c8101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:08:31.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.042+0000 7f48cf800640 1 -- 172.21.15.67:0/841884009 shutdown_connections 2024-03-28T16:08:31.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.042+0000 7f48cf800640 1 -- 172.21.15.67:0/841884009 wait complete. 2024-03-28T16:08:31.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.043+0000 7f48cf800640 1 Processor -- start 2024-03-28T16:08:31.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.044+0000 7f48cf800640 1 -- start start 2024-03-28T16:08:31.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.044+0000 7f48cf800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f48c81038e0 0x7f48c8199b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:31.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.044+0000 7f48cf800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48c8104ad0 0x7f48c819a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:31.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.044+0000 7f48cf800640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48c8105490 0x7f48c81a10b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:31.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.044+0000 7f48cf800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48c810f450 con 0x7f48c8104ad0 2024-03-28T16:08:31.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.044+0000 7f48cf800640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48c810f2d0 con 0x7f48c8105490 2024-03-28T16:08:31.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.044+0000 7f48cf800640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48c810f5d0 con 0x7f48c81038e0 2024-03-28T16:08:31.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.045+0000 7f48cce00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f48c81038e0 0x7f48c8199b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:08:31.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.045+0000 7f48cce00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f48c81038e0 0x7f48c8199b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.67:60388/0 (socket says 172.21.15.67:60388) 2024-03-28T16:08:31.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.045+0000 7f48cce00640 1 -- 172.21.15.67:0/152134950 learned_addr learned my addr 172.21.15.67:0/152134950 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:08:31.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.045+0000 7f48cd800640 1 --2- 172.21.15.67:0/152134950 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48c8105490 0x7f48c81a10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:08:31.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.045+0000 7f48cd800640 1 -- 172.21.15.67:0/152134950 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f48c81038e0 msgr2=0x7f48c8199b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:08:31.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.045+0000 7f48cd800640 1 --2- 172.21.15.67:0/152134950 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f48c81038e0 0x7f48c8199b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:31.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.045+0000 7f48cd800640 1 -- 172.21.15.67:0/152134950 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48c8104ad0 msgr2=0x7f48c819a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:08:31.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.045+0000 7f48cd800640 1 --2- 172.21.15.67:0/152134950 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48c8104ad0 0x7f48c819a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:31.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.045+0000 7f48cd800640 1 -- 172.21.15.67:0/152134950 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48c81a1720 con 0x7f48c8105490 2024-03-28T16:08:31.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.046+0000 7f48cd800640 1 --2- 172.21.15.67:0/152134950 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48c8105490 0x7f48c81a10b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f48c000cab0 tx=0x7f48c000cf70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:08:31.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.046+0000 7f48c5600640 1 -- 172.21.15.67:0/152134950 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48c000ae20 con 0x7f48c8105490 2024-03-28T16:08:31.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.047+0000 7f48cf800640 1 -- 172.21.15.67:0/152134950 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48c81a1a10 con 0x7f48c8105490 2024-03-28T16:08:31.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.047+0000 7f48c5600640 1 -- 172.21.15.67:0/152134950 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f48c000f900 con 0x7f48c8105490 2024-03-28T16:08:31.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.047+0000 7f48c5600640 1 -- 172.21.15.67:0/152134950 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48c0018a00 con 0x7f48c8105490 2024-03-28T16:08:31.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.047+0000 7f48cf800640 1 -- 172.21.15.67:0/152134950 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48c81a1f60 con 0x7f48c8105490 2024-03-28T16:08:31.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.054+0000 7f48c5600640 1 -- 172.21.15.67:0/152134950 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f48c000fab0 con 0x7f48c8105490 2024-03-28T16:08:31.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.056+0000 7f48d25ba740 1 Processor -- start 2024-03-28T16:08:31.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.056+0000 7f48d25ba740 1 -- start start 2024-03-28T16:08:31.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.057+0000 7f48c5600640 1 --2- 172.21.15.67:0/152134950 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f48b00a2880 0x7f48b00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:31.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.058+0000 7f48c5600640 1 -- 172.21.15.67:0/152134950 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f48c00cf2b0 con 0x7f48c8105490 2024-03-28T16:08:31.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.058+0000 7f48d25ba740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ec357eb020 0x55ec357eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:31.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.058+0000 7f48d25ba740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ec357ec1f0 0x55ec357ec670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:31.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.058+0000 7f48d25ba740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ec357ecbb0 0x55ec357f3440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:31.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.058+0000 7f48c7e00640 1 --2- 172.21.15.67:0/152134950 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f48b00a2880 0x7f48b00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:08:31.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.058+0000 7f48d25ba740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec357472c0 con 0x55ec357ecbb0 2024-03-28T16:08:31.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.058+0000 7f48d25ba740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec356efc90 con 0x55ec357ec1f0 2024-03-28T16:08:31.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.058+0000 7f48d25ba740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec35743360 con 0x55ec357eb020 2024-03-28T16:08:31.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.059+0000 7f48cce00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ec357eb020 0x55ec357eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:08:31.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.059+0000 7f48cd800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ec357ecbb0 0x55ec357f3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:08:31.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.059+0000 7f48cce00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ec357eb020 0x55ec357eb440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.67:60404/0 (socket says 172.21.15.67:60404) 2024-03-28T16:08:31.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.059+0000 7f48cce00640 1 -- 172.21.15.67:0/969051489 learned_addr learned my addr 172.21.15.67:0/969051489 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:08:31.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.059+0000 7f48cce00640 1 -- 172.21.15.67:0/969051489 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ec357ec1f0 msgr2=0x55ec357ec670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T16:08:31.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.059+0000 7f48cce00640 1 --2- 172.21.15.67:0/969051489 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ec357ec1f0 0x55ec357ec670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:31.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.059+0000 7f48cce00640 1 -- 172.21.15.67:0/969051489 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ec357ecbb0 msgr2=0x55ec357f3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:08:31.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.060+0000 7f48cce00640 1 --2- 172.21.15.67:0/969051489 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ec357ecbb0 0x55ec357f3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:31.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.060+0000 7f48cce00640 1 -- 172.21.15.67:0/969051489 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ec35714080 con 0x55ec357eb020 2024-03-28T16:08:31.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.060+0000 7f48cd800640 1 --2- 172.21.15.67:0/969051489 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ec357ecbb0 0x55ec357f3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T16:08:31.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.060+0000 7f48cce00640 1 --2- 172.21.15.67:0/969051489 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ec357eb020 0x55ec357eb440 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f48bc036040 tx=0x7f48bc02c8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c33a879b6259042 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:08:31.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.061+0000 7f48b6000640 1 -- 172.21.15.67:0/969051489 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48bc045020 con 0x55ec357eb020 2024-03-28T16:08:31.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.061+0000 7f48b6000640 1 -- 172.21.15.67:0/969051489 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f48bc02cbc0 con 0x55ec357eb020 2024-03-28T16:08:31.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.061+0000 7f48b6000640 1 -- 172.21.15.67:0/969051489 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48bc004920 con 0x55ec357eb020 2024-03-28T16:08:31.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.062+0000 7f48d25ba740 1 -- 172.21.15.67:0/969051489 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ec357eb020 msgr2=0x55ec357eb440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:08:31.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.062+0000 7f48c7e00640 1 --2- 172.21.15.67:0/152134950 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f48b00a2880 0x7f48b00a4d40 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f48b8005fd0 tx=0x7f48b8005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:08:31.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.062+0000 7f48d25ba740 1 --2- 172.21.15.67:0/969051489 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ec357eb020 0x55ec357eb440 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f48bc036040 tx=0x7f48bc02c8e0 comp rx=0 tx=0).stop 2024-03-28T16:08:31.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.063+0000 7f48d25ba740 1 -- 172.21.15.67:0/969051489 shutdown_connections 2024-03-28T16:08:31.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.063+0000 7f48d25ba740 1 --2- 172.21.15.67:0/969051489 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ec357ecbb0 0x55ec357f3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:31.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.063+0000 7f48d25ba740 1 --2- 172.21.15.67:0/969051489 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ec357ec1f0 0x55ec357ec670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:31.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.063+0000 7f48d25ba740 1 --2- 172.21.15.67:0/969051489 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ec357eb020 0x55ec357eb440 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:31.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.063+0000 7f48d25ba740 1 -- 172.21.15.67:0/969051489 >> 172.21.15.67:0/969051489 conn(0x55ec357e6410 msgr2=0x55ec357e6810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:08:31.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.064+0000 7f48d25ba740 1 -- 172.21.15.67:0/969051489 shutdown_connections 2024-03-28T16:08:31.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.064+0000 7f48d25ba740 1 -- 172.21.15.67:0/969051489 wait complete. 2024-03-28T16:08:31.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.064+0000 7f48d25ba740 1 -- start start 2024-03-28T16:08:31.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.065+0000 7f48d25ba740 1 Processor -- start 2024-03-28T16:08:31.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.065+0000 7f48d25ba740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ec357eb020 0x55ec357f5290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:31.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.065+0000 7f48d25ba740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ec357ec1f0 0x55ec357f57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:31.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.065+0000 7f48d25ba740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ec357ecbb0 0x55ec357fc810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:31.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.065+0000 7f48d25ba740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec357f3b60 con 0x55ec357eb020 2024-03-28T16:08:31.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.065+0000 7f48d25ba740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec357030d0 con 0x55ec357ecbb0 2024-03-28T16:08:31.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.065+0000 7f48d25ba740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec357f3ce0 con 0x55ec357ec1f0 2024-03-28T16:08:31.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.065+0000 7f48d25ba740 10 client.0 _ll_register_callbacks cb 0x55ec35707e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:08:31.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.066+0000 7f48cce00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ec357eb020 0x55ec357f5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:08:31.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.066+0000 7f48cce00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ec357eb020 0x55ec357f5290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46296/0 (socket says 172.21.15.67:46296) 2024-03-28T16:08:31.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.066+0000 7f48cce00640 1 -- 172.21.15.67:0/3750692953 learned_addr learned my addr 172.21.15.67:0/3750692953 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:08:31.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.066+0000 7f48cd800640 1 --2- 172.21.15.67:0/3750692953 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ec357ecbb0 0x55ec357fc810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:08:31.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.067+0000 7f48cce00640 1 -- 172.21.15.67:0/3750692953 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ec357ec1f0 msgr2=0x55ec357f57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:08:31.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.067+0000 7f48cce00640 1 --2- 172.21.15.67:0/3750692953 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ec357ec1f0 0x55ec357f57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:31.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.067+0000 7f48cce00640 1 -- 172.21.15.67:0/3750692953 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ec357ecbb0 msgr2=0x55ec357fc810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:08:31.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.067+0000 7f48cce00640 1 --2- 172.21.15.67:0/3750692953 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ec357ecbb0 0x55ec357fc810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:31.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.067+0000 7f48cce00640 1 -- 172.21.15.67:0/3750692953 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ec3545b3f0 con 0x55ec357eb020 2024-03-28T16:08:31.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.067+0000 7f48cce00640 1 --2- 172.21.15.67:0/3750692953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ec357eb020 0x55ec357f5290 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f48bc02c170 tx=0x7f48bc02bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28609b869b837518 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:08:31.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.067+0000 7f48aaa00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:08:31.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.068+0000 7f48aaa00640 1 -- 172.21.15.67:0/3750692953 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48bc045070 con 0x55ec357eb020 2024-03-28T16:08:31.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.068+0000 7f48aaa00640 1 -- 172.21.15.67:0/3750692953 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f48bc003a00 con 0x55ec357eb020 2024-03-28T16:08:31.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.068+0000 7f48d25ba740 1 -- 172.21.15.67:0/3750692953 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f48bc02cbc0 con 0x55ec357eb020 2024-03-28T16:08:31.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.068+0000 7f48aaa00640 1 -- 172.21.15.67:0/3750692953 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f48bc003c00 con 0x55ec357eb020 2024-03-28T16:08:31.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.068+0000 7f48d25ba740 10 client.16038 fetch_fsmap learned FSMap version 19 2024-03-28T16:08:31.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.068+0000 7f48d25ba740 1 -- 172.21.15.67:0/3750692953 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55ec355257d0 con 0x55ec357eb020 2024-03-28T16:08:31.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.069+0000 7f48aaa00640 1 -- 172.21.15.67:0/3750692953 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f48bc02cbc0 con 0x55ec357eb020 2024-03-28T16:08:31.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.069+0000 7f48aaa00640 1 -- 172.21.15.67:0/3750692953 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f48bc04a450 con 0x55ec357eb020 2024-03-28T16:08:31.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.069+0000 7f48d25ba740 10 client.16038 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:08:31.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.069+0000 7f48d25ba740 10 client.16038 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:08:31.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.069+0000 7f48d25ba740 20 client.16038 populate_metadata read hostname 'smithi067' 2024-03-28T16:08:31.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.069+0000 7f48d25ba740 1 --2- 172.21.15.67:0/3750692953 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55ec35800970 0x55ec35802d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:31.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.069+0000 7f48d25ba740 4 client.16038 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:08:31.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.069+0000 7f48d25ba740 1 -- 172.21.15.67:0/3750692953 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ec35754800 con 0x55ec35800970 2024-03-28T16:08:31.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.070+0000 7f48c7e00640 1 --2- 172.21.15.67:0/3750692953 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55ec35800970 0x55ec35802d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:08:31.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.074+0000 7f48c7e00640 1 --2- 172.21.15.67:0/3750692953 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55ec35800970 0x55ec35802d60 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1eb690004c79d9f server_cookie=b80ef4e3dba63fe5 in_seq=0 out_seq=0 2024-03-28T16:08:31.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.074+0000 7f48aaa00640 10 client.16038 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:08:31.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.077+0000 7f48aaa00640 1 -- 172.21.15.67:0/3750692953 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55ec35754800 con 0x55ec35800970 2024-03-28T16:08:31.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.077+0000 7f48aaa00640 10 client.16038 handle_command_reply: tid=0 2024-03-28T16:08:31.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.077+0000 7f48d25ba740 1 client.16038 shutdown 2024-03-28T16:08:31.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.077+0000 7f48d25ba740 1 -- 172.21.15.67:0/3750692953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ec357eb020 msgr2=0x55ec357f5290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:08:31.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.077+0000 7f48d25ba740 1 --2- 172.21.15.67:0/3750692953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ec357eb020 0x55ec357f5290 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f48bc02c170 tx=0x7f48bc02bc30 comp rx=0 tx=0).stop 2024-03-28T16:08:31.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.078+0000 7f48d25ba740 1 -- 172.21.15.67:0/3750692953 shutdown_connections 2024-03-28T16:08:31.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.078+0000 7f48d25ba740 1 --2- 172.21.15.67:0/3750692953 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55ec35800970 0x55ec35802d60 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:31.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.078+0000 7f48d25ba740 1 --2- 172.21.15.67:0/3750692953 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ec357ecbb0 0x55ec357fc810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:31.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.078+0000 7f48d25ba740 1 --2- 172.21.15.67:0/3750692953 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ec357ec1f0 0x55ec357f57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:31.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.078+0000 7f48d25ba740 1 --2- 172.21.15.67:0/3750692953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ec357eb020 0x55ec357f5290 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:31.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.078+0000 7f48d25ba740 1 -- 172.21.15.67:0/3750692953 >> 172.21.15.67:0/3750692953 conn(0x55ec357e6410 msgr2=0x55ec357dceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:08:31.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.078+0000 7f48aaa00640 0 client.16038 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:08:31.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.078+0000 7f48d25ba740 1 -- 172.21.15.67:0/3750692953 shutdown_connections 2024-03-28T16:08:31.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.079+0000 7f48d25ba740 1 -- 172.21.15.67:0/3750692953 wait complete. 2024-03-28T16:08:31.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.079+0000 7f48d25ba740 20 client.16038 trim_cache size 0 max 16384 2024-03-28T16:08:31.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.101+0000 7f48d25ba740 1 Processor -- start 2024-03-28T16:08:31.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.101+0000 7f48d25ba740 1 -- start start 2024-03-28T16:08:31.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.101+0000 7f48d25ba740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ec357eb020 0x55ec357f40b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:31.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.101+0000 7f48d25ba740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ec357ec1f0 0x55ec357e67f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:31.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.101+0000 7f48d25ba740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ec357ecbb0 0x55ec357ef290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:31.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.101+0000 7f48d25ba740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48c81a46e0 con 0x55ec357ecbb0 2024-03-28T16:08:31.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.101+0000 7f48d25ba740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48c81a6ca0 con 0x55ec357eb020 2024-03-28T16:08:31.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.101+0000 7f48d25ba740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48c81a4ce0 con 0x55ec357ec1f0 2024-03-28T16:08:31.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.102+0000 7f48cce00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ec357eb020 0x55ec357f40b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:08:31.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.102+0000 7f48cce00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ec357eb020 0x55ec357f40b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.67:36380/0 (socket says 172.21.15.67:36380) 2024-03-28T16:08:31.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.102+0000 7f48cce00640 1 -- 172.21.15.67:0/1492490344 learned_addr learned my addr 172.21.15.67:0/1492490344 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:08:31.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.102+0000 7f48cd800640 1 --2- 172.21.15.67:0/1492490344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ec357ecbb0 0x55ec357ef290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:08:31.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.102+0000 7f48c7e00640 1 --2- 172.21.15.67:0/1492490344 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ec357ec1f0 0x55ec357e67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:08:31.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.103+0000 7f48cd800640 1 -- 172.21.15.67:0/1492490344 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ec357ec1f0 msgr2=0x55ec357e67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:08:31.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.103+0000 7f48cd800640 1 --2- 172.21.15.67:0/1492490344 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ec357ec1f0 0x55ec357e67f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:31.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.103+0000 7f48cd800640 1 -- 172.21.15.67:0/1492490344 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ec357eb020 msgr2=0x55ec357f40b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:08:31.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.103+0000 7f48cd800640 1 --2- 172.21.15.67:0/1492490344 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ec357eb020 0x55ec357f40b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:31.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.103+0000 7f48cd800640 1 -- 172.21.15.67:0/1492490344 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ec357f6280 con 0x55ec357ecbb0 2024-03-28T16:08:31.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.103+0000 7f48cd800640 1 --2- 172.21.15.67:0/1492490344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ec357ecbb0 0x55ec357ef290 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f48c009b900 tx=0x7f48c009bc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6460e4a61904b931 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:08:31.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.103+0000 7f48b5600640 1 -- 172.21.15.67:0/1492490344 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48c00b9070 con 0x55ec357ecbb0 2024-03-28T16:08:31.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.104+0000 7f48b5600640 1 -- 172.21.15.67:0/1492490344 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f48c00cce70 con 0x55ec357ecbb0 2024-03-28T16:08:31.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.104+0000 7f48b5600640 1 -- 172.21.15.67:0/1492490344 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48c00a41a0 con 0x55ec357ecbb0 2024-03-28T16:08:31.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.105+0000 7f48d25ba740 1 -- 172.21.15.67:0/1492490344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ec357ecbb0 msgr2=0x55ec357ef290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:08:31.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.105+0000 7f48d25ba740 1 --2- 172.21.15.67:0/1492490344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ec357ecbb0 0x55ec357ef290 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f48c009b900 tx=0x7f48c009bc50 comp rx=0 tx=0).stop 2024-03-28T16:08:31.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.105+0000 7f48d25ba740 1 -- 172.21.15.67:0/1492490344 shutdown_connections 2024-03-28T16:08:31.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.105+0000 7f48d25ba740 1 --2- 172.21.15.67:0/1492490344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ec357ecbb0 0x55ec357ef290 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:31.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.105+0000 7f48d25ba740 1 --2- 172.21.15.67:0/1492490344 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ec357ec1f0 0x55ec357e67f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:31.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.105+0000 7f48d25ba740 1 --2- 172.21.15.67:0/1492490344 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ec357eb020 0x55ec357f40b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:31.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.105+0000 7f48d25ba740 1 -- 172.21.15.67:0/1492490344 >> 172.21.15.67:0/1492490344 conn(0x55ec357e6410 msgr2=0x55ec357e9f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:08:31.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.107+0000 7f48d25ba740 1 -- 172.21.15.67:0/1492490344 shutdown_connections 2024-03-28T16:08:31.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.107+0000 7f48d25ba740 1 -- 172.21.15.67:0/1492490344 wait complete. 2024-03-28T16:08:31.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.107+0000 7f48d25ba740 1 -- start start 2024-03-28T16:08:31.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.108+0000 7f48d25ba740 1 Processor -- start 2024-03-28T16:08:31.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.108+0000 7f48d25ba740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ec357eb020 0x55ec357f7b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:31.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.108+0000 7f48d25ba740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ec357ec1f0 0x55ec357f80d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:31.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.108+0000 7f48d25ba740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ec357ecbb0 0x55ec357fb0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:31.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.108+0000 7f48d25ba740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48c81a5320 con 0x55ec357ec1f0 2024-03-28T16:08:31.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.108+0000 7f48d25ba740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48c81a6640 con 0x55ec357ecbb0 2024-03-28T16:08:31.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.108+0000 7f48d25ba740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48c81a5980 con 0x55ec357eb020 2024-03-28T16:08:31.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.108+0000 7f48d25ba740 10 client.0 _ll_register_callbacks cb 0x7f48c0097f20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:08:31.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.109+0000 7f48c7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ec357ec1f0 0x55ec357f80d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:08:31.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.109+0000 7f48cd800640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ec357ecbb0 0x55ec357fb0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:08:31.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.109+0000 7f48c7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ec357ec1f0 0x55ec357f80d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46316/0 (socket says 172.21.15.67:46316) 2024-03-28T16:08:31.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.109+0000 7f48c7e00640 1 -- 172.21.15.67:0/516716108 learned_addr learned my addr 172.21.15.67:0/516716108 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:08:31.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.110+0000 7f48c7e00640 1 -- 172.21.15.67:0/516716108 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ec357eb020 msgr2=0x55ec357f7b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:08:31.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.110+0000 7f48c7e00640 1 --2- 172.21.15.67:0/516716108 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ec357eb020 0x55ec357f7b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:31.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.110+0000 7f48c7e00640 1 -- 172.21.15.67:0/516716108 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ec357ecbb0 msgr2=0x55ec357fb0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:08:31.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.110+0000 7f48c7e00640 1 --2- 172.21.15.67:0/516716108 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ec357ecbb0 0x55ec357fb0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:31.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.110+0000 7f48c7e00640 1 -- 172.21.15.67:0/516716108 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ec357fb6e0 con 0x55ec357ec1f0 2024-03-28T16:08:31.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.110+0000 7f48cd800640 1 --2- 172.21.15.67:0/516716108 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ec357ecbb0 0x55ec357fb0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T16:08:31.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.110+0000 7f48c7e00640 1 --2- 172.21.15.67:0/516716108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ec357ec1f0 0x55ec357f80d0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f48b8012040 tx=0x7f48b8023c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=966193d766e0f92 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:08:31.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.110+0000 7f48aaa00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:08:31.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.111+0000 7f48aaa00640 1 -- 172.21.15.67:0/516716108 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48b8023130 con 0x55ec357ec1f0 2024-03-28T16:08:31.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.111+0000 7f48aaa00640 1 -- 172.21.15.67:0/516716108 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f48b80108b0 con 0x55ec357ec1f0 2024-03-28T16:08:31.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.111+0000 7f48d25ba740 1 -- 172.21.15.67:0/516716108 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f48c00cce70 con 0x55ec357ec1f0 2024-03-28T16:08:31.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.111+0000 7f48aaa00640 1 -- 172.21.15.67:0/516716108 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f48b80232d0 con 0x55ec357ec1f0 2024-03-28T16:08:31.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.111+0000 7f48d25ba740 10 client.16044 fetch_fsmap learned FSMap version 19 2024-03-28T16:08:31.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.111+0000 7f48d25ba740 1 -- 172.21.15.67:0/516716108 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55ec3585c760 con 0x55ec357ec1f0 2024-03-28T16:08:31.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.112+0000 7f48aaa00640 1 -- 172.21.15.67:0/516716108 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f48c00cce70 con 0x55ec357ec1f0 2024-03-28T16:08:31.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.112+0000 7f48aaa00640 1 -- 172.21.15.67:0/516716108 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f48b801fb60 con 0x55ec357ec1f0 2024-03-28T16:08:31.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.112+0000 7f48d25ba740 10 client.16044 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:08:31.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.112+0000 7f48d25ba740 10 client.16044 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:08:31.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.112+0000 7f48d25ba740 20 client.16044 populate_metadata read hostname 'smithi067' 2024-03-28T16:08:31.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.112+0000 7f48d25ba740 1 --2- 172.21.15.67:0/516716108 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55ec3585d0f0 0x55ec3585f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:31.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.112+0000 7f48d25ba740 4 client.16044 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T16:08:31.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.112+0000 7f48d25ba740 1 -- 172.21.15.67:0/516716108 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55ec3574eb50 con 0x55ec3585d0f0 2024-03-28T16:08:31.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.113+0000 7f48cce00640 1 --2- 172.21.15.67:0/516716108 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55ec3585d0f0 0x55ec3585f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:08:31.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.114+0000 7f48cce00640 1 --2- 172.21.15.67:0/516716108 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55ec3585d0f0 0x55ec3585f510 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cfef306600eb9543 server_cookie=cb21a57f6f6d7635 in_seq=0 out_seq=0 2024-03-28T16:08:31.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.114+0000 7f48aaa00640 10 client.16044 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:08:31.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.115+0000 7f48aaa00640 1 -- 172.21.15.67:0/516716108 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55ec3574eb50 con 0x55ec3585d0f0 2024-03-28T16:08:31.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.115+0000 7f48aaa00640 10 client.16044 handle_command_reply: tid=0 2024-03-28T16:08:31.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.115+0000 7f48d25ba740 1 client.16044 shutdown 2024-03-28T16:08:31.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.115+0000 7f48d25ba740 1 -- 172.21.15.67:0/516716108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ec357ec1f0 msgr2=0x55ec357f80d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:08:31.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.115+0000 7f48d25ba740 1 --2- 172.21.15.67:0/516716108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ec357ec1f0 0x55ec357f80d0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f48b8012040 tx=0x7f48b8023c20 comp rx=0 tx=0).stop 2024-03-28T16:08:31.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.115+0000 7f48d25ba740 1 -- 172.21.15.67:0/516716108 shutdown_connections 2024-03-28T16:08:31.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.115+0000 7f48d25ba740 1 --2- 172.21.15.67:0/516716108 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55ec3585d0f0 0x55ec3585f510 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:31.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.115+0000 7f48d25ba740 1 --2- 172.21.15.67:0/516716108 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ec357ecbb0 0x55ec357fb0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:31.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.115+0000 7f48d25ba740 1 --2- 172.21.15.67:0/516716108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ec357ec1f0 0x55ec357f80d0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:31.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.115+0000 7f48d25ba740 1 --2- 172.21.15.67:0/516716108 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ec357eb020 0x55ec357f7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:31.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.115+0000 7f48aaa00640 0 client.16044 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:08:31.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.115+0000 7f48d25ba740 1 -- 172.21.15.67:0/516716108 >> 172.21.15.67:0/516716108 conn(0x55ec357e6410 msgr2=0x55ec357e9f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:08:31.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.116+0000 7f48d25ba740 1 -- 172.21.15.67:0/516716108 shutdown_connections 2024-03-28T16:08:31.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.116+0000 7f48d25ba740 1 -- 172.21.15.67:0/516716108 wait complete. 2024-03-28T16:08:31.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.116+0000 7f48d25ba740 20 client.16044 trim_cache size 0 max 16384 2024-03-28T16:08:31.118 INFO:teuthology.orchestra.run.smithi067.stdout:[] 2024-03-28T16:08:31.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.117+0000 7f48cf800640 1 -- 172.21.15.67:0/152134950 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f48b00a2880 msgr2=0x7f48b00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:08:31.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.117+0000 7f48cf800640 1 --2- 172.21.15.67:0/152134950 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f48b00a2880 0x7f48b00a4d40 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f48b8005fd0 tx=0x7f48b8005850 comp rx=0 tx=0).stop 2024-03-28T16:08:31.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.117+0000 7f48cf800640 1 -- 172.21.15.67:0/152134950 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48c8105490 msgr2=0x7f48c81a10b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:08:31.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.117+0000 7f48cf800640 1 --2- 172.21.15.67:0/152134950 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48c8105490 0x7f48c81a10b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f48c000cab0 tx=0x7f48c000cf70 comp rx=0 tx=0).stop 2024-03-28T16:08:31.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.117+0000 7f48cf800640 1 -- 172.21.15.67:0/152134950 shutdown_connections 2024-03-28T16:08:31.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.117+0000 7f48cf800640 1 --2- 172.21.15.67:0/152134950 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f48b00a2880 0x7f48b00a4d40 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:31.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.117+0000 7f48cf800640 1 --2- 172.21.15.67:0/152134950 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48c8105490 0x7f48c81a10b0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:31.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.117+0000 7f48cf800640 1 --2- 172.21.15.67:0/152134950 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48c8104ad0 0x7f48c819a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:31.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.117+0000 7f48cf800640 1 --2- 172.21.15.67:0/152134950 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f48c81038e0 0x7f48c8199b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:31.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.117+0000 7f48cf800640 1 -- 172.21.15.67:0/152134950 >> 172.21.15.67:0/152134950 conn(0x7f48c80fee50 msgr2=0x7f48c8107ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:08:31.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.117+0000 7f48cf800640 1 -- 172.21.15.67:0/152134950 shutdown_connections 2024-03-28T16:08:31.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:31.117+0000 7f48cf800640 1 -- 172.21.15.67:0/152134950 wait complete. 2024-03-28T16:08:31.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:08:31 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/655647294' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:08:31.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:08:31 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/655647294' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:08:31.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:08:31 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/655647294' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:08:32.142 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T16:08:32.142 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T16:08:32.143 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T16:08:32.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.424+0000 7fbb03c00640 1 -- 172.21.15.67:0/2399901571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbafc1016b0 msgr2=0x7fbafc101ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:08:32.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.424+0000 7fbb03c00640 1 --2- 172.21.15.67:0/2399901571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbafc1016b0 0x7fbafc101ad0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fbaf0009900 tx=0x7fbaf002bc30 comp rx=0 tx=0).stop 2024-03-28T16:08:32.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.425+0000 7fbb03c00640 1 -- 172.21.15.67:0/2399901571 shutdown_connections 2024-03-28T16:08:32.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.425+0000 7fbb03c00640 1 --2- 172.21.15.67:0/2399901571 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbafc103260 0x7fbafc109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:32.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.425+0000 7fbb03c00640 1 --2- 172.21.15.67:0/2399901571 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbafc1028a0 0x7fbafc102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:32.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.425+0000 7fbb03c00640 1 --2- 172.21.15.67:0/2399901571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbafc1016b0 0x7fbafc101ad0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:32.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.425+0000 7fbb03c00640 1 -- 172.21.15.67:0/2399901571 >> 172.21.15.67:0/2399901571 conn(0x7fbafc0fcc20 msgr2=0x7fbafc0ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:08:32.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.425+0000 7fbb03c00640 1 -- 172.21.15.67:0/2399901571 shutdown_connections 2024-03-28T16:08:32.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.425+0000 7fbb03c00640 1 -- 172.21.15.67:0/2399901571 wait complete. 2024-03-28T16:08:32.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.426+0000 7fbb03c00640 1 Processor -- start 2024-03-28T16:08:32.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.426+0000 7fbb03c00640 1 -- start start 2024-03-28T16:08:32.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.427+0000 7fbb03c00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbafc1016b0 0x7fbafc199aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:32.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.427+0000 7fbb03c00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbafc1028a0 0x7fbafc199fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:32.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.427+0000 7fbb03c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbafc103260 0x7fbafc1a1050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:32.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.427+0000 7fbb03c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbafc10f460 con 0x7fbafc103260 2024-03-28T16:08:32.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.427+0000 7fbb03c00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbafc10f2e0 con 0x7fbafc1016b0 2024-03-28T16:08:32.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.427+0000 7fbb03c00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbafc10f5e0 con 0x7fbafc1028a0 2024-03-28T16:08:32.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.427+0000 7fbb01c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbafc103260 0x7fbafc1a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:08:32.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.427+0000 7fbb01c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbafc103260 0x7fbafc1a1050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46348/0 (socket says 172.21.15.67:46348) 2024-03-28T16:08:32.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.427+0000 7fbb01c00640 1 -- 172.21.15.67:0/506308335 learned_addr learned my addr 172.21.15.67:0/506308335 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:08:32.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.428+0000 7fbafbe00640 1 --2- 172.21.15.67:0/506308335 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbafc1028a0 0x7fbafc199fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:08:32.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.428+0000 7fbb01200640 1 --2- 172.21.15.67:0/506308335 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbafc1016b0 0x7fbafc199aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:08:32.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.428+0000 7fbb01c00640 1 -- 172.21.15.67:0/506308335 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbafc1028a0 msgr2=0x7fbafc199fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:08:32.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.428+0000 7fbb01c00640 1 --2- 172.21.15.67:0/506308335 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbafc1028a0 0x7fbafc199fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:32.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.428+0000 7fbb01c00640 1 -- 172.21.15.67:0/506308335 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbafc1016b0 msgr2=0x7fbafc199aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:08:32.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.428+0000 7fbb01c00640 1 --2- 172.21.15.67:0/506308335 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbafc1016b0 0x7fbafc199aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:32.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.428+0000 7fbb01c00640 1 -- 172.21.15.67:0/506308335 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbafc1a16c0 con 0x7fbafc103260 2024-03-28T16:08:32.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.428+0000 7fbb01c00640 1 --2- 172.21.15.67:0/506308335 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbafc103260 0x7fbafc1a1050 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fbaf400eaa0 tx=0x7fbaf400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:08:32.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.429+0000 7fbaf9600640 1 -- 172.21.15.67:0/506308335 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf400cdb0 con 0x7fbafc103260 2024-03-28T16:08:32.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.429+0000 7fbaf9600640 1 -- 172.21.15.67:0/506308335 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbaf4005020 con 0x7fbafc103260 2024-03-28T16:08:32.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.429+0000 7fbb03c00640 1 -- 172.21.15.67:0/506308335 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbafc1a19b0 con 0x7fbafc103260 2024-03-28T16:08:32.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.429+0000 7fbaf9600640 1 -- 172.21.15.67:0/506308335 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf40116d0 con 0x7fbafc103260 2024-03-28T16:08:32.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.429+0000 7fbb03c00640 1 -- 172.21.15.67:0/506308335 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbafc1a1f00 con 0x7fbafc103260 2024-03-28T16:08:32.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.434+0000 7fbaf9600640 1 -- 172.21.15.67:0/506308335 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fbaf40051d0 con 0x7fbafc103260 2024-03-28T16:08:32.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.434+0000 7fbb06814740 1 Processor -- start 2024-03-28T16:08:32.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.435+0000 7fbb06814740 1 -- start start 2024-03-28T16:08:32.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.435+0000 7fbb06814740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559908e660a0 0x559908e664c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:32.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.435+0000 7fbb06814740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559908e66a90 0x559908e6b590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:32.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.435+0000 7fbb06814740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559908e6bc00 0x559908e6e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:32.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.435+0000 7fbb06814740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x559908dcbeb0 con 0x559908e66a90 2024-03-28T16:08:32.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.435+0000 7fbb06814740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x559908d74a60 con 0x559908e6bc00 2024-03-28T16:08:32.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.435+0000 7fbb06814740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x559908dc7f50 con 0x559908e660a0 2024-03-28T16:08:32.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.436+0000 7fbafbe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559908e660a0 0x559908e664c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:08:32.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.436+0000 7fbb01200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559908e66a90 0x559908e6b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:08:32.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.436+0000 7fbafbe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559908e660a0 0x559908e664c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.67:60442/0 (socket says 172.21.15.67:60442) 2024-03-28T16:08:32.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.436+0000 7fbafbe00640 1 -- 172.21.15.67:0/3779966543 learned_addr learned my addr 172.21.15.67:0/3779966543 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:08:32.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.436+0000 7fbafbe00640 1 --2- 172.21.15.67:0/3779966543 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559908e6bc00 0x559908e6e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:08:32.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.436+0000 7fbb01200640 1 -- 172.21.15.67:0/3779966543 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559908e660a0 msgr2=0x559908e664c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:08:32.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.436+0000 7fbb01200640 1 --2- 172.21.15.67:0/3779966543 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559908e660a0 0x559908e664c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:32.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.436+0000 7fbb01200640 1 -- 172.21.15.67:0/3779966543 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559908e6bc00 msgr2=0x559908e6e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:08:32.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.436+0000 7fbb01200640 1 --2- 172.21.15.67:0/3779966543 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559908e6bc00 0x559908e6e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:32.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.436+0000 7fbb01200640 1 -- 172.21.15.67:0/3779966543 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559908ba87e0 con 0x559908e66a90 2024-03-28T16:08:32.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.437+0000 7fbaf9600640 1 --2- 172.21.15.67:0/506308335 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fbae40a2880 0x7fbae40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:32.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.437+0000 7fbaf9600640 1 -- 172.21.15.67:0/506308335 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fbaf40ceca0 con 0x7fbafc103260 2024-03-28T16:08:32.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.437+0000 7fbb01200640 1 --2- 172.21.15.67:0/3779966543 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559908e66a90 0x559908e6b590 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fbaf0036040 tx=0x7fbaf0005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf0d73d9bc2f2e3e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:08:32.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.437+0000 7fbafbe00640 1 --2- 172.21.15.67:0/506308335 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fbae40a2880 0x7fbae40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:08:32.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.437+0000 7fbaea000640 1 -- 172.21.15.67:0/3779966543 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf0044020 con 0x559908e66a90 2024-03-28T16:08:32.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.438+0000 7fbaea000640 1 -- 172.21.15.67:0/3779966543 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbaf002cbe0 con 0x559908e66a90 2024-03-28T16:08:32.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.438+0000 7fbaea000640 1 -- 172.21.15.67:0/3779966543 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf0004920 con 0x559908e66a90 2024-03-28T16:08:32.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.439+0000 7fbafbe00640 1 --2- 172.21.15.67:0/506308335 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fbae40a2880 0x7fbae40a4d40 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x559908e672f0 tx=0x7fbaec0046c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:08:32.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.439+0000 7fbb06814740 1 -- 172.21.15.67:0/3779966543 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559908e66a90 msgr2=0x559908e6b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:08:32.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.439+0000 7fbb06814740 1 --2- 172.21.15.67:0/3779966543 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559908e66a90 0x559908e6b590 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fbaf0036040 tx=0x7fbaf0005dd0 comp rx=0 tx=0).stop 2024-03-28T16:08:32.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.440+0000 7fbb06814740 1 -- 172.21.15.67:0/3779966543 shutdown_connections 2024-03-28T16:08:32.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.440+0000 7fbb06814740 1 --2- 172.21.15.67:0/3779966543 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559908e6bc00 0x559908e6e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:32.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.440+0000 7fbb06814740 1 --2- 172.21.15.67:0/3779966543 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559908e66a90 0x559908e6b590 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:32.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.440+0000 7fbb06814740 1 --2- 172.21.15.67:0/3779966543 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559908e660a0 0x559908e664c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:32.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.440+0000 7fbb06814740 1 -- 172.21.15.67:0/3779966543 >> 172.21.15.67:0/3779966543 conn(0x559908e61610 msgr2=0x559908e61a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:08:32.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.440+0000 7fbb06814740 1 -- 172.21.15.67:0/3779966543 shutdown_connections 2024-03-28T16:08:32.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.440+0000 7fbb06814740 1 -- 172.21.15.67:0/3779966543 wait complete. 2024-03-28T16:08:32.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.440+0000 7fbb06814740 1 -- start start 2024-03-28T16:08:32.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.441+0000 7fbb06814740 1 Processor -- start 2024-03-28T16:08:32.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.441+0000 7fbb06814740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559908e660a0 0x559908e702b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:32.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.441+0000 7fbb06814740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559908e66a90 0x559908e707f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:32.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.441+0000 7fbb06814740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559908e6bc00 0x559908e77830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:32.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.441+0000 7fbb06814740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x559908e6eb50 con 0x559908e6bc00 2024-03-28T16:08:32.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.441+0000 7fbb06814740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x559908d87eb0 con 0x559908e66a90 2024-03-28T16:08:32.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.441+0000 7fbb06814740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x559908e6ecd0 con 0x559908e660a0 2024-03-28T16:08:32.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.441+0000 7fbb06814740 10 client.0 _ll_register_callbacks cb 0x559908d7c150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:08:32.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.442+0000 7fbb01c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559908e6bc00 0x559908e77830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:08:32.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.442+0000 7fbb01c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559908e6bc00 0x559908e77830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46360/0 (socket says 172.21.15.67:46360) 2024-03-28T16:08:32.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.442+0000 7fbb01c00640 1 -- 172.21.15.67:0/2788083130 learned_addr learned my addr 172.21.15.67:0/2788083130 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:08:32.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.442+0000 7fbafbe00640 1 --2- 172.21.15.67:0/2788083130 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559908e66a90 0x559908e707f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:08:32.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.443+0000 7fbb01c00640 1 -- 172.21.15.67:0/2788083130 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559908e660a0 msgr2=0x559908e702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:08:32.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.443+0000 7fbb01c00640 1 --2- 172.21.15.67:0/2788083130 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559908e660a0 0x559908e702b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:32.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.443+0000 7fbb01c00640 1 -- 172.21.15.67:0/2788083130 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559908e66a90 msgr2=0x559908e707f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:08:32.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.443+0000 7fbb01c00640 1 --2- 172.21.15.67:0/2788083130 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559908e66a90 0x559908e707f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:32.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.443+0000 7fbb01c00640 1 -- 172.21.15.67:0/2788083130 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559908c08f70 con 0x559908e6bc00 2024-03-28T16:08:32.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.443+0000 7fbb01c00640 1 --2- 172.21.15.67:0/2788083130 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559908e6bc00 0x559908e77830 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fbaf400efd0 tx=0x7fbaf409be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ae8df06d1732568 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:08:32.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.443+0000 7fbadea00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:08:32.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.444+0000 7fbadea00640 1 -- 172.21.15.67:0/2788083130 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf401f040 con 0x559908e6bc00 2024-03-28T16:08:32.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.444+0000 7fbadea00640 1 -- 172.21.15.67:0/2788083130 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fbaf40a3960 con 0x559908e6bc00 2024-03-28T16:08:32.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.444+0000 7fbb06814740 1 -- 172.21.15.67:0/2788083130 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbaf002cbe0 con 0x559908e6bc00 2024-03-28T16:08:32.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.444+0000 7fbadea00640 1 -- 172.21.15.67:0/2788083130 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbaf40a3df0 con 0x559908e6bc00 2024-03-28T16:08:32.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.444+0000 7fbb06814740 10 client.16056 fetch_fsmap learned FSMap version 19 2024-03-28T16:08:32.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.444+0000 7fbb06814740 1 -- 172.21.15.67:0/2788083130 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x559908aa0170 con 0x559908e6bc00 2024-03-28T16:08:32.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.445+0000 7fbadea00640 1 -- 172.21.15.67:0/2788083130 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbaf002cbe0 con 0x559908e6bc00 2024-03-28T16:08:32.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.445+0000 7fbadea00640 1 -- 172.21.15.67:0/2788083130 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fbaf400f040 con 0x559908e6bc00 2024-03-28T16:08:32.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.445+0000 7fbb06814740 10 client.16056 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:08:32.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.445+0000 7fbb06814740 10 client.16056 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:08:32.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.445+0000 7fbb06814740 20 client.16056 populate_metadata read hostname 'smithi067' 2024-03-28T16:08:32.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.445+0000 7fbb06814740 1 --2- 172.21.15.67:0/2788083130 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x559908e7c0d0 0x559908e7e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:32.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.445+0000 7fbb06814740 4 client.16056 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:08:32.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.445+0000 7fbb06814740 1 -- 172.21.15.67:0/2788083130 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559908dd93f0 con 0x559908e7c0d0 2024-03-28T16:08:32.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.447+0000 7fbb01200640 1 --2- 172.21.15.67:0/2788083130 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x559908e7c0d0 0x559908e7e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:08:32.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.447+0000 7fbb01200640 1 --2- 172.21.15.67:0/2788083130 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x559908e7c0d0 0x559908e7e590 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6cc16953e8d02615 server_cookie=a5ee8bd80886abd5 in_seq=0 out_seq=0 2024-03-28T16:08:32.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.447+0000 7fbadea00640 10 client.16056 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:08:32.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.449+0000 7fbadea00640 1 -- 172.21.15.67:0/2788083130 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x559908dd93f0 con 0x559908e7c0d0 2024-03-28T16:08:32.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.449+0000 7fbadea00640 10 client.16056 handle_command_reply: tid=0 2024-03-28T16:08:32.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.449+0000 7fbb06814740 1 client.16056 shutdown 2024-03-28T16:08:32.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.450+0000 7fbb06814740 1 -- 172.21.15.67:0/2788083130 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559908e6bc00 msgr2=0x559908e77830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:08:32.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.450+0000 7fbb06814740 1 --2- 172.21.15.67:0/2788083130 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559908e6bc00 0x559908e77830 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fbaf400efd0 tx=0x7fbaf409be90 comp rx=0 tx=0).stop 2024-03-28T16:08:32.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.450+0000 7fbb06814740 1 -- 172.21.15.67:0/2788083130 shutdown_connections 2024-03-28T16:08:32.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.450+0000 7fbb06814740 1 --2- 172.21.15.67:0/2788083130 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x559908e7c0d0 0x559908e7e590 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:32.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.450+0000 7fbb06814740 1 --2- 172.21.15.67:0/2788083130 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559908e6bc00 0x559908e77830 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:32.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.450+0000 7fbb06814740 1 --2- 172.21.15.67:0/2788083130 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559908e66a90 0x559908e707f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:32.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.450+0000 7fbb06814740 1 --2- 172.21.15.67:0/2788083130 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559908e660a0 0x559908e702b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:32.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.450+0000 7fbb06814740 1 -- 172.21.15.67:0/2788083130 >> 172.21.15.67:0/2788083130 conn(0x559908e61610 msgr2=0x559908e60830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:08:32.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.450+0000 7fbadea00640 0 client.16056 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:08:32.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.450+0000 7fbb06814740 1 -- 172.21.15.67:0/2788083130 shutdown_connections 2024-03-28T16:08:32.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.450+0000 7fbb06814740 1 -- 172.21.15.67:0/2788083130 wait complete. 2024-03-28T16:08:32.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.450+0000 7fbb06814740 20 client.16056 trim_cache size 0 max 16384 2024-03-28T16:08:32.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.483+0000 7fbb06814740 1 Processor -- start 2024-03-28T16:08:32.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.484+0000 7fbb06814740 1 -- start start 2024-03-28T16:08:32.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.484+0000 7fbb06814740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559908e660a0 0x559908e67530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:32.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.484+0000 7fbb06814740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559908e66a90 0x559908e777f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:32.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.484+0000 7fbb06814740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559908e6bc00 0x559908e67a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:32.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.484+0000 7fbb06814740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbafc1a8bc0 con 0x559908e6bc00 2024-03-28T16:08:32.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.484+0000 7fbb06814740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbafc1ab180 con 0x559908e660a0 2024-03-28T16:08:32.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.485+0000 7fbb06814740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbafc1a91c0 con 0x559908e66a90 2024-03-28T16:08:32.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.485+0000 7fbb01200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559908e660a0 0x559908e67530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:08:32.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.485+0000 7fbb01200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559908e660a0 0x559908e67530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.67:36424/0 (socket says 172.21.15.67:36424) 2024-03-28T16:08:32.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.485+0000 7fbb01200640 1 -- 172.21.15.67:0/416898497 learned_addr learned my addr 172.21.15.67:0/416898497 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:08:32.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.485+0000 7fbb01c00640 1 --2- 172.21.15.67:0/416898497 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559908e6bc00 0x559908e67a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:08:32.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.486+0000 7fbb01200640 1 -- 172.21.15.67:0/416898497 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559908e66a90 msgr2=0x559908e777f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:08:32.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.486+0000 7fbb01200640 1 --2- 172.21.15.67:0/416898497 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559908e66a90 0x559908e777f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:32.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.486+0000 7fbb01200640 1 -- 172.21.15.67:0/416898497 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559908e6bc00 msgr2=0x559908e67a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:08:32.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.486+0000 7fbb01200640 1 --2- 172.21.15.67:0/416898497 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559908e6bc00 0x559908e67a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:32.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.486+0000 7fbb01200640 1 -- 172.21.15.67:0/416898497 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559908e5cc20 con 0x559908e660a0 2024-03-28T16:08:32.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.486+0000 7fbb01c00640 1 --2- 172.21.15.67:0/416898497 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559908e6bc00 0x559908e67a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T16:08:32.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.487+0000 7fbb01200640 1 --2- 172.21.15.67:0/416898497 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559908e660a0 0x559908e67530 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fbaf0009900 tx=0x7fbaf0005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7cf9b96fea49c18 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:08:32.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.488+0000 7fbae9600640 1 -- 172.21.15.67:0/416898497 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf0052020 con 0x559908e660a0 2024-03-28T16:08:32.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.488+0000 7fbae9600640 1 -- 172.21.15.67:0/416898497 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbaf002cac0 con 0x559908e660a0 2024-03-28T16:08:32.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.488+0000 7fbae9600640 1 -- 172.21.15.67:0/416898497 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf0044040 con 0x559908e660a0 2024-03-28T16:08:32.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.489+0000 7fbb06814740 1 -- 172.21.15.67:0/416898497 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559908e660a0 msgr2=0x559908e67530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:08:32.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.489+0000 7fbb06814740 1 --2- 172.21.15.67:0/416898497 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559908e660a0 0x559908e67530 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fbaf0009900 tx=0x7fbaf0005fd0 comp rx=0 tx=0).stop 2024-03-28T16:08:32.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.489+0000 7fbb06814740 1 -- 172.21.15.67:0/416898497 shutdown_connections 2024-03-28T16:08:32.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.489+0000 7fbb06814740 1 --2- 172.21.15.67:0/416898497 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559908e6bc00 0x559908e67a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:32.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.489+0000 7fbb06814740 1 --2- 172.21.15.67:0/416898497 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559908e66a90 0x559908e777f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:32.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.489+0000 7fbb06814740 1 --2- 172.21.15.67:0/416898497 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559908e660a0 0x559908e67530 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:32.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.489+0000 7fbb06814740 1 -- 172.21.15.67:0/416898497 >> 172.21.15.67:0/416898497 conn(0x559908e61610 msgr2=0x559908e65100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:08:32.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.492+0000 7fbb06814740 1 -- 172.21.15.67:0/416898497 shutdown_connections 2024-03-28T16:08:32.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.492+0000 7fbb06814740 1 -- 172.21.15.67:0/416898497 wait complete. 2024-03-28T16:08:32.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.493+0000 7fbb06814740 1 -- start start 2024-03-28T16:08:32.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.493+0000 7fbb06814740 1 Processor -- start 2024-03-28T16:08:32.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.493+0000 7fbb06814740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559908e660a0 0x559908e73a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:32.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.494+0000 7fbb06814740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559908e66a90 0x559908e73f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:32.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.494+0000 7fbb06814740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559908e6bc00 0x559908e76f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:32.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.494+0000 7fbb06814740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbafc1a9800 con 0x559908e66a90 2024-03-28T16:08:32.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.494+0000 7fbb06814740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbafc1aab20 con 0x559908e6bc00 2024-03-28T16:08:32.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.494+0000 7fbb06814740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbafc1a9e60 con 0x559908e660a0 2024-03-28T16:08:32.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.494+0000 7fbb06814740 10 client.0 _ll_register_callbacks cb 0x7fbaf4005830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:08:32.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.494+0000 7fbafbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559908e66a90 0x559908e73f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:08:32.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.494+0000 7fbafbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559908e66a90 0x559908e73f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46382/0 (socket says 172.21.15.67:46382) 2024-03-28T16:08:32.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.494+0000 7fbafbe00640 1 -- 172.21.15.67:0/3342639936 learned_addr learned my addr 172.21.15.67:0/3342639936 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:08:32.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.494+0000 7fbb01200640 1 --2- 172.21.15.67:0/3342639936 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559908e660a0 0x559908e73a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:08:32.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.495+0000 7fbb01c00640 1 --2- 172.21.15.67:0/3342639936 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559908e6bc00 0x559908e76f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:08:32.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.495+0000 7fbafbe00640 1 -- 172.21.15.67:0/3342639936 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559908e660a0 msgr2=0x559908e73a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:08:32.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.495+0000 7fbafbe00640 1 --2- 172.21.15.67:0/3342639936 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559908e660a0 0x559908e73a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:32.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.495+0000 7fbafbe00640 1 -- 172.21.15.67:0/3342639936 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559908e6bc00 msgr2=0x559908e76f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:08:32.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.495+0000 7fbafbe00640 1 --2- 172.21.15.67:0/3342639936 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559908e6bc00 0x559908e76f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:32.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.495+0000 7fbafbe00640 1 -- 172.21.15.67:0/3342639936 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559908e77580 con 0x559908e66a90 2024-03-28T16:08:32.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.495+0000 7fbb01200640 1 --2- 172.21.15.67:0/3342639936 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559908e660a0 0x559908e73a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:08:32.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.495+0000 7fbb01c00640 1 --2- 172.21.15.67:0/3342639936 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559908e6bc00 0x559908e76f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:08:32.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.496+0000 7fbafbe00640 1 --2- 172.21.15.67:0/3342639936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559908e66a90 0x559908e73f70 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fbaec005f10 tx=0x7fbaec00bb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a35d91061a44b27f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:08:32.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.496+0000 7fbadea00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:08:32.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.496+0000 7fbadea00640 1 -- 172.21.15.67:0/3342639936 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaec00ee60 con 0x559908e66a90 2024-03-28T16:08:32.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.496+0000 7fbadea00640 1 -- 172.21.15.67:0/3342639936 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fbaec01f3f0 con 0x559908e66a90 2024-03-28T16:08:32.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.497+0000 7fbb06814740 1 -- 172.21.15.67:0/3342639936 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbaf002cac0 con 0x559908e66a90 2024-03-28T16:08:32.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.497+0000 7fbadea00640 1 -- 172.21.15.67:0/3342639936 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbaec01f5f0 con 0x559908e66a90 2024-03-28T16:08:32.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.497+0000 7fbb06814740 10 client.16062 fetch_fsmap learned FSMap version 19 2024-03-28T16:08:32.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.497+0000 7fbb06814740 1 -- 172.21.15.67:0/3342639936 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x559908ef1e80 con 0x559908e66a90 2024-03-28T16:08:32.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.498+0000 7fbadea00640 1 -- 172.21.15.67:0/3342639936 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbaf002cac0 con 0x559908e66a90 2024-03-28T16:08:32.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.498+0000 7fbadea00640 1 -- 172.21.15.67:0/3342639936 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fbaec0328c0 con 0x559908e66a90 2024-03-28T16:08:32.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.498+0000 7fbb06814740 10 client.16062 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:08:32.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.498+0000 7fbb06814740 10 client.16062 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:08:32.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.498+0000 7fbb06814740 20 client.16062 populate_metadata read hostname 'smithi067' 2024-03-28T16:08:32.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.498+0000 7fbb06814740 1 --2- 172.21.15.67:0/3342639936 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x559908ef2840 0x559908ef4c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:32.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.498+0000 7fbb06814740 4 client.16062 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T16:08:32.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.498+0000 7fbb06814740 1 -- 172.21.15.67:0/3342639936 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x559908dd3740 con 0x559908ef2840 2024-03-28T16:08:32.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.499+0000 7fbb01200640 1 --2- 172.21.15.67:0/3342639936 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x559908ef2840 0x559908ef4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:08:32.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.500+0000 7fbb01200640 1 --2- 172.21.15.67:0/3342639936 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x559908ef2840 0x559908ef4c60 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f0624913c7e7f0a server_cookie=24d1617ec96a9321 in_seq=0 out_seq=0 2024-03-28T16:08:32.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.500+0000 7fbadea00640 10 client.16062 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:08:32.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.503+0000 7fbadea00640 1 -- 172.21.15.67:0/3342639936 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x559908dd3740 con 0x559908ef2840 2024-03-28T16:08:32.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.503+0000 7fbadea00640 10 client.16062 handle_command_reply: tid=0 2024-03-28T16:08:32.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.503+0000 7fbb06814740 1 client.16062 shutdown 2024-03-28T16:08:32.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.504+0000 7fbb06814740 1 -- 172.21.15.67:0/3342639936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559908e66a90 msgr2=0x559908e73f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:08:32.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.504+0000 7fbb06814740 1 --2- 172.21.15.67:0/3342639936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559908e66a90 0x559908e73f70 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fbaec005f10 tx=0x7fbaec00bb80 comp rx=0 tx=0).stop 2024-03-28T16:08:32.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.504+0000 7fbb06814740 1 -- 172.21.15.67:0/3342639936 shutdown_connections 2024-03-28T16:08:32.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.504+0000 7fbb06814740 1 --2- 172.21.15.67:0/3342639936 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x559908ef2840 0x559908ef4c60 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:32.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.505+0000 7fbadea00640 0 client.16062 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:08:32.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.505+0000 7fbb06814740 1 --2- 172.21.15.67:0/3342639936 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559908e6bc00 0x559908e76f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:32.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.505+0000 7fbb06814740 1 --2- 172.21.15.67:0/3342639936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559908e66a90 0x559908e73f70 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:32.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.505+0000 7fbb06814740 1 --2- 172.21.15.67:0/3342639936 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559908e660a0 0x559908e73a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:32.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.505+0000 7fbb06814740 1 -- 172.21.15.67:0/3342639936 >> 172.21.15.67:0/3342639936 conn(0x559908e61610 msgr2=0x559908e5ffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:08:32.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.505+0000 7fbb06814740 1 -- 172.21.15.67:0/3342639936 shutdown_connections 2024-03-28T16:08:32.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.505+0000 7fbb06814740 1 -- 172.21.15.67:0/3342639936 wait complete. 2024-03-28T16:08:32.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.505+0000 7fbb06814740 20 client.16062 trim_cache size 0 max 16384 2024-03-28T16:08:32.508 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-28T16:08:32.508 INFO:teuthology.orchestra.run.smithi067.stdout: "return_code": 0, 2024-03-28T16:08:32.508 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_tag": "5ff8cfe3-c44b-46e1-b8b7-08f862da00d8", 2024-03-28T16:08:32.508 INFO:teuthology.orchestra.run.smithi067.stdout: "mode": "asynchronous" 2024-03-28T16:08:32.508 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-28T16:08:32.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.506+0000 7fbb03c00640 1 -- 172.21.15.67:0/506308335 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fbae40a2880 msgr2=0x7fbae40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:08:32.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.506+0000 7fbb03c00640 1 --2- 172.21.15.67:0/506308335 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fbae40a2880 0x7fbae40a4d40 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x559908e672f0 tx=0x7fbaec0046c0 comp rx=0 tx=0).stop 2024-03-28T16:08:32.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.506+0000 7fbb03c00640 1 -- 172.21.15.67:0/506308335 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbafc103260 msgr2=0x7fbafc1a1050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:08:32.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.506+0000 7fbb03c00640 1 --2- 172.21.15.67:0/506308335 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbafc103260 0x7fbafc1a1050 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fbaf400eaa0 tx=0x7fbaf400ef60 comp rx=0 tx=0).stop 2024-03-28T16:08:32.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.506+0000 7fbb03c00640 1 -- 172.21.15.67:0/506308335 shutdown_connections 2024-03-28T16:08:32.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.506+0000 7fbb03c00640 1 --2- 172.21.15.67:0/506308335 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fbae40a2880 0x7fbae40a4d40 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:32.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.506+0000 7fbb03c00640 1 --2- 172.21.15.67:0/506308335 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbafc103260 0x7fbafc1a1050 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:32.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.506+0000 7fbb03c00640 1 --2- 172.21.15.67:0/506308335 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbafc1028a0 0x7fbafc199fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:32.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.506+0000 7fbb03c00640 1 --2- 172.21.15.67:0/506308335 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbafc1016b0 0x7fbafc199aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:32.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.507+0000 7fbb03c00640 1 -- 172.21.15.67:0/506308335 >> 172.21.15.67:0/506308335 conn(0x7fbafc0fcc20 msgr2=0x7fbafc105ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:08:32.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.507+0000 7fbb03c00640 1 -- 172.21.15.67:0/506308335 shutdown_connections 2024-03-28T16:08:32.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.507+0000 7fbb03c00640 1 -- 172.21.15.67:0/506308335 wait complete. 2024-03-28T16:08:32.531 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:08:32 smithi067 ceph-mon[31707]: pgmap v588: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 238 MiB/s wr, 62 op/s 2024-03-28T16:08:32.534 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T16:08:32.534 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T16:08:32.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.794+0000 7f0365200640 1 -- 172.21.15.67:0/747466167 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0360101790 msgr2=0x7f0360101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:08:32.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.794+0000 7f0365200640 1 --2- 172.21.15.67:0/747466167 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0360101790 0x7f0360101c10 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f0350009900 tx=0x7f035002bc30 comp rx=0 tx=0).stop 2024-03-28T16:08:32.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.794+0000 7f0365200640 1 -- 172.21.15.67:0/747466167 shutdown_connections 2024-03-28T16:08:32.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.794+0000 7f0365200640 1 --2- 172.21.15.67:0/747466167 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0360102150 0x7f03601089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:32.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.794+0000 7f0365200640 1 --2- 172.21.15.67:0/747466167 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0360101790 0x7f0360101c10 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:32.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.794+0000 7f0365200640 1 --2- 172.21.15.67:0/747466167 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f03601005a0 0x7f03601009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:32.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.794+0000 7f0365200640 1 -- 172.21.15.67:0/747466167 >> 172.21.15.67:0/747466167 conn(0x7f03600fbb30 msgr2=0x7f03600fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:08:32.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.794+0000 7f0365200640 1 -- 172.21.15.67:0/747466167 shutdown_connections 2024-03-28T16:08:32.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.794+0000 7f0365200640 1 -- 172.21.15.67:0/747466167 wait complete. 2024-03-28T16:08:32.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.795+0000 7f0365200640 1 Processor -- start 2024-03-28T16:08:32.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.795+0000 7f0365200640 1 -- start start 2024-03-28T16:08:32.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.796+0000 7f0365200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f03601005a0 0x7f036019e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:32.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.796+0000 7f0365200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0360101790 0x7f036019eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:32.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.796+0000 7f0365200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0360102150 0x7f03601a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:32.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.796+0000 7f0365200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f036010c120 con 0x7f0360101790 2024-03-28T16:08:32.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.796+0000 7f0365200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f036010bfa0 con 0x7f0360102150 2024-03-28T16:08:32.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.796+0000 7f0365200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f036010c2a0 con 0x7f03601005a0 2024-03-28T16:08:32.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.797+0000 7f035de00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f03601005a0 0x7f036019e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:08:32.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.797+0000 7f035de00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f03601005a0 0x7f036019e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.67:60500/0 (socket says 172.21.15.67:60500) 2024-03-28T16:08:32.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.797+0000 7f035de00640 1 -- 172.21.15.67:0/360551566 learned_addr learned my addr 172.21.15.67:0/360551566 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:08:32.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.797+0000 7f035e800640 1 --2- 172.21.15.67:0/360551566 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0360102150 0x7f03601a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:08:32.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.797+0000 7f035d400640 1 --2- 172.21.15.67:0/360551566 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0360101790 0x7f036019eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:08:32.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.797+0000 7f035de00640 1 -- 172.21.15.67:0/360551566 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0360102150 msgr2=0x7f03601a5500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:08:32.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.797+0000 7f035de00640 1 --2- 172.21.15.67:0/360551566 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0360102150 0x7f03601a5500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:32.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.797+0000 7f035de00640 1 -- 172.21.15.67:0/360551566 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0360101790 msgr2=0x7f036019eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:08:32.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.797+0000 7f035de00640 1 --2- 172.21.15.67:0/360551566 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0360101790 0x7f036019eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:32.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.797+0000 7f035de00640 1 -- 172.21.15.67:0/360551566 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03601a5bd0 con 0x7f03601005a0 2024-03-28T16:08:32.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.797+0000 7f035d400640 1 --2- 172.21.15.67:0/360551566 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0360101790 0x7f036019eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:08:32.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.798+0000 7f035de00640 1 --2- 172.21.15.67:0/360551566 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f03601005a0 0x7f036019e640 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f035800ead0 tx=0x7f035800ef90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:08:32.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.799+0000 7f0356a00640 1 -- 172.21.15.67:0/360551566 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0358016070 con 0x7f03601005a0 2024-03-28T16:08:32.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.799+0000 7f0356a00640 1 -- 172.21.15.67:0/360551566 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f035800ce40 con 0x7f03601005a0 2024-03-28T16:08:32.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.799+0000 7f0356a00640 1 -- 172.21.15.67:0/360551566 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03580116d0 con 0x7f03601005a0 2024-03-28T16:08:32.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.799+0000 7f0365200640 1 -- 172.21.15.67:0/360551566 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03601a5ec0 con 0x7f03601005a0 2024-03-28T16:08:32.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.800+0000 7f0365200640 1 -- 172.21.15.67:0/360551566 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03601a62d0 con 0x7f03601005a0 2024-03-28T16:08:32.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:08:32 smithi150 ceph-mon[34825]: pgmap v588: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 238 MiB/s wr, 62 op/s 2024-03-28T16:08:32.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.804+0000 7f0367f7d740 1 Processor -- start 2024-03-28T16:08:32.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.804+0000 7f0367f7d740 1 -- start start 2024-03-28T16:08:32.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.804+0000 7f0367f7d740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efb62dbe80 0x55efb62dc2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:32.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.804+0000 7f0367f7d740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55efb62dc870 0x55efb62e1370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:32.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.804+0000 7f0367f7d740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55efb62e19e0 0x55efb62e3e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:32.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.804+0000 7f0367f7d740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efb6241250 con 0x55efb62dc870 2024-03-28T16:08:32.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.804+0000 7f0367f7d740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efb61e9a30 con 0x55efb62dbe80 2024-03-28T16:08:32.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.804+0000 7f0367f7d740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efb623d2f0 con 0x55efb62e19e0 2024-03-28T16:08:32.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.805+0000 7f035d400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55efb62e19e0 0x55efb62e3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:08:32.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.805+0000 7f035d400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55efb62e19e0 0x55efb62e3e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.67:60502/0 (socket says 172.21.15.67:60502) 2024-03-28T16:08:32.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.805+0000 7f035d400640 1 -- 172.21.15.67:0/3537156612 learned_addr learned my addr 172.21.15.67:0/3537156612 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:08:32.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.805+0000 7f035e800640 1 --2- 172.21.15.67:0/3537156612 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efb62dbe80 0x55efb62dc2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:08:32.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.806+0000 7f035e800640 1 -- 172.21.15.67:0/3537156612 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55efb62e19e0 msgr2=0x55efb62e3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:08:32.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.806+0000 7f035e800640 1 --2- 172.21.15.67:0/3537156612 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55efb62e19e0 0x55efb62e3e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:32.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.806+0000 7f035e800640 1 -- 172.21.15.67:0/3537156612 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55efb62dc870 msgr2=0x55efb62e1370 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T16:08:32.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.806+0000 7f035e800640 1 --2- 172.21.15.67:0/3537156612 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55efb62dc870 0x55efb62e1370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:32.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.806+0000 7f035e800640 1 -- 172.21.15.67:0/3537156612 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55efb61d4690 con 0x55efb62dbe80 2024-03-28T16:08:32.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.806+0000 7f035d400640 1 --2- 172.21.15.67:0/3537156612 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55efb62e19e0 0x55efb62e3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T16:08:32.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.806+0000 7f0356a00640 1 -- 172.21.15.67:0/360551566 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f0358019070 con 0x7f03601005a0 2024-03-28T16:08:32.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.806+0000 7f035e800640 1 --2- 172.21.15.67:0/3537156612 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efb62dbe80 0x55efb62dc2a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f034800db60 tx=0x7f034800f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8efc4a9a6f4a208 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:08:32.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.807+0000 7f034f400640 1 -- 172.21.15.67:0/3537156612 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f034800de50 con 0x55efb62dbe80 2024-03-28T16:08:32.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.807+0000 7f034f400640 1 -- 172.21.15.67:0/3537156612 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0348005020 con 0x55efb62dbe80 2024-03-28T16:08:32.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.807+0000 7f034f400640 1 -- 172.21.15.67:0/3537156612 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0348004b00 con 0x55efb62dbe80 2024-03-28T16:08:32.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.808+0000 7f0367f7d740 1 -- 172.21.15.67:0/3537156612 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efb62dbe80 msgr2=0x55efb62dc2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:08:32.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.808+0000 7f0367f7d740 1 --2- 172.21.15.67:0/3537156612 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efb62dbe80 0x55efb62dc2a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f034800db60 tx=0x7f034800f730 comp rx=0 tx=0).stop 2024-03-28T16:08:32.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.809+0000 7f0356a00640 1 --2- 172.21.15.67:0/360551566 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f03440a28d0 0x7f03440a4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:32.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.809+0000 7f0356a00640 1 -- 172.21.15.67:0/360551566 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f03580ce540 con 0x7f03601005a0 2024-03-28T16:08:32.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.810+0000 7f0367f7d740 1 -- 172.21.15.67:0/3537156612 shutdown_connections 2024-03-28T16:08:32.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.810+0000 7f0367f7d740 1 --2- 172.21.15.67:0/3537156612 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55efb62e19e0 0x55efb62e3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:32.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.810+0000 7f0367f7d740 1 --2- 172.21.15.67:0/3537156612 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55efb62dc870 0x55efb62e1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:32.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.810+0000 7f0367f7d740 1 --2- 172.21.15.67:0/3537156612 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efb62dbe80 0x55efb62dc2a0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:32.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.810+0000 7f0367f7d740 1 -- 172.21.15.67:0/3537156612 >> 172.21.15.67:0/3537156612 conn(0x55efb62a4200 msgr2=0x55efb62d77f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:08:32.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.811+0000 7f035e800640 1 --2- 172.21.15.67:0/360551566 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f03440a28d0 0x7f03440a4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:08:32.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.811+0000 7f0367f7d740 1 -- 172.21.15.67:0/3537156612 shutdown_connections 2024-03-28T16:08:32.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.812+0000 7f035e800640 1 --2- 172.21.15.67:0/360551566 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f03440a28d0 0x7f03440a4d90 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f034800d920 tx=0x7f0348000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:08:32.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.812+0000 7f0367f7d740 1 -- 172.21.15.67:0/3537156612 wait complete. 2024-03-28T16:08:32.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.813+0000 7f0367f7d740 1 -- start start 2024-03-28T16:08:32.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.814+0000 7f0367f7d740 1 Processor -- start 2024-03-28T16:08:32.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.814+0000 7f0367f7d740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55efb62dbe80 0x55efb62e5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:32.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.814+0000 7f0367f7d740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efb62dc870 0x55efb62e6420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:32.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.814+0000 7f0367f7d740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55efb62e19e0 0x55efb62ed460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:32.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.815+0000 7f035de00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55efb62dbe80 0x55efb62e5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:08:32.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.815+0000 7f0367f7d740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efb62e4990 con 0x55efb62e19e0 2024-03-28T16:08:32.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.815+0000 7f035de00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55efb62dbe80 0x55efb62e5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.67:60510/0 (socket says 172.21.15.67:60510) 2024-03-28T16:08:32.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.815+0000 7f035de00640 1 -- 172.21.15.67:0/2523969494 learned_addr learned my addr 172.21.15.67:0/2523969494 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:08:32.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.815+0000 7f0367f7d740 1 -- 172.21.15.67:0/2523969494 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efb61fce80 con 0x55efb62dc870 2024-03-28T16:08:32.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.815+0000 7f0367f7d740 1 -- 172.21.15.67:0/2523969494 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efb62e4b10 con 0x55efb62dbe80 2024-03-28T16:08:32.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.815+0000 7f035e800640 1 --2- 172.21.15.67:0/2523969494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55efb62e19e0 0x55efb62ed460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:08:32.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.815+0000 7f035d400640 1 --2- 172.21.15.67:0/2523969494 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efb62dc870 0x55efb62e6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:08:32.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.815+0000 7f035de00640 1 -- 172.21.15.67:0/2523969494 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efb62dc870 msgr2=0x55efb62e6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:08:32.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.815+0000 7f035de00640 1 --2- 172.21.15.67:0/2523969494 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efb62dc870 0x55efb62e6420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:32.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.815+0000 7f035de00640 1 -- 172.21.15.67:0/2523969494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55efb62e19e0 msgr2=0x55efb62ed460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:08:32.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.815+0000 7f035de00640 1 --2- 172.21.15.67:0/2523969494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55efb62e19e0 0x55efb62ed460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:32.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.815+0000 7f035de00640 1 -- 172.21.15.67:0/2523969494 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55efb601f7d0 con 0x55efb62dbe80 2024-03-28T16:08:32.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.816+0000 7f0367f7d740 10 client.0 _ll_register_callbacks cb 0x55efb620f130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:08:32.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.816+0000 7f0367f7d740 1 -- 172.21.15.67:0/2523969494 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0348005020 con 0x55efb62dbe80 2024-03-28T16:08:32.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.816+0000 7f035de00640 1 --2- 172.21.15.67:0/2523969494 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55efb62dbe80 0x55efb62e5ee0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f035800c700 tx=0x7f035809a8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=17645eca1f851e09 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:08:32.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.816+0000 7f0343e00640 10 client.25549 ms_handle_connect on v2:172.21.15.183:3300/0 2024-03-28T16:08:32.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.816+0000 7f0343e00640 1 -- 172.21.15.67:0/2523969494 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0358020040 con 0x55efb62dbe80 2024-03-28T16:08:32.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.816+0000 7f0343e00640 1 -- 172.21.15.67:0/2523969494 <== mon.1 v2:172.21.15.183:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f035809a2e0 con 0x55efb62dbe80 2024-03-28T16:08:32.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.817+0000 7f0343e00640 1 -- 172.21.15.67:0/2523969494 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f035809a520 con 0x55efb62dbe80 2024-03-28T16:08:32.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.817+0000 7f0367f7d740 10 client.25549 fetch_fsmap learned FSMap version 19 2024-03-28T16:08:32.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.817+0000 7f0367f7d740 1 -- 172.21.15.67:0/2523969494 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({fsmap=19}) v3 -- 0x55efb5f553f0 con 0x55efb62dbe80 2024-03-28T16:08:32.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.818+0000 7f0343e00640 1 -- 172.21.15.67:0/2523969494 <== mon.1 v2:172.21.15.183:3300/0 4 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f035800f040 con 0x55efb62dbe80 2024-03-28T16:08:32.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.818+0000 7f0367f7d740 10 client.25549 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:08:32.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.818+0000 7f0367f7d740 10 client.25549 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:08:32.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.818+0000 7f0367f7d740 20 client.25549 populate_metadata read hostname 'smithi067' 2024-03-28T16:08:32.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.818+0000 7f0367f7d740 1 --2- 172.21.15.67:0/2523969494 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55efb62f1ab0 0x55efb62f3f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:32.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.818+0000 7f0367f7d740 4 client.25549 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:08:32.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.818+0000 7f0367f7d740 1 -- 172.21.15.67:0/2523969494 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55efb624e790 con 0x55efb62f1ab0 2024-03-28T16:08:32.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.819+0000 7f035d400640 1 --2- 172.21.15.67:0/2523969494 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55efb62f1ab0 0x55efb62f3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:08:32.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.819+0000 7f0343e00640 10 client.25549 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:08:32.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.819+0000 7f035d400640 1 --2- 172.21.15.67:0/2523969494 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55efb62f1ab0 0x55efb62f3f70 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3875f3802db3e1dc server_cookie=40f3ad42781a244c in_seq=0 out_seq=0 2024-03-28T16:08:32.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.820+0000 7f0343e00640 1 -- 172.21.15.67:0/2523969494 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55efb624e790 con 0x55efb62f1ab0 2024-03-28T16:08:32.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.821+0000 7f0343e00640 10 client.25549 handle_command_reply: tid=0 2024-03-28T16:08:32.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.821+0000 7f0367f7d740 1 client.25549 shutdown 2024-03-28T16:08:32.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.821+0000 7f0367f7d740 1 -- 172.21.15.67:0/2523969494 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55efb62dbe80 msgr2=0x55efb62e5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:08:32.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.821+0000 7f0367f7d740 1 --2- 172.21.15.67:0/2523969494 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55efb62dbe80 0x55efb62e5ee0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f035800c700 tx=0x7f035809a8b0 comp rx=0 tx=0).stop 2024-03-28T16:08:32.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.821+0000 7f0367f7d740 1 -- 172.21.15.67:0/2523969494 shutdown_connections 2024-03-28T16:08:32.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.821+0000 7f0367f7d740 1 --2- 172.21.15.67:0/2523969494 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55efb62f1ab0 0x55efb62f3f70 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:32.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.821+0000 7f0367f7d740 1 --2- 172.21.15.67:0/2523969494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55efb62e19e0 0x55efb62ed460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:32.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.821+0000 7f0367f7d740 1 --2- 172.21.15.67:0/2523969494 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efb62dc870 0x55efb62e6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:32.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.821+0000 7f0367f7d740 1 --2- 172.21.15.67:0/2523969494 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55efb62dbe80 0x55efb62e5ee0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:32.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.821+0000 7f0343e00640 0 client.25549 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:08:32.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.821+0000 7f0367f7d740 1 -- 172.21.15.67:0/2523969494 >> 172.21.15.67:0/2523969494 conn(0x55efb62a4200 msgr2=0x55efb62d6a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:08:32.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.821+0000 7f0367f7d740 1 -- 172.21.15.67:0/2523969494 shutdown_connections 2024-03-28T16:08:32.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.821+0000 7f0367f7d740 1 -- 172.21.15.67:0/2523969494 wait complete. 2024-03-28T16:08:32.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.822+0000 7f0367f7d740 20 client.25549 trim_cache size 0 max 16384 2024-03-28T16:08:32.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:08:32 smithi183 ceph-mon[35580]: pgmap v588: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 238 MiB/s wr, 62 op/s 2024-03-28T16:08:32.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.851+0000 7f0367f7d740 1 Processor -- start 2024-03-28T16:08:32.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.851+0000 7f0367f7d740 1 -- start start 2024-03-28T16:08:32.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.851+0000 7f0367f7d740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efb62dbe80 0x55efb62e6380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:32.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.851+0000 7f0367f7d740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55efb62dc870 0x55efb62ed420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:32.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.851+0000 7f0367f7d740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55efb62e19e0 0x55efb62dd9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:32.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.851+0000 7f0367f7d740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03601ad070 con 0x55efb62e19e0 2024-03-28T16:08:32.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.851+0000 7f0367f7d740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03601af630 con 0x55efb62dbe80 2024-03-28T16:08:32.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.851+0000 7f0367f7d740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03601ad670 con 0x55efb62dc870 2024-03-28T16:08:32.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.852+0000 7f035e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55efb62e19e0 0x55efb62dd9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:08:32.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.852+0000 7f035e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55efb62e19e0 0x55efb62dd9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46432/0 (socket says 172.21.15.67:46432) 2024-03-28T16:08:32.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.852+0000 7f035e800640 1 -- 172.21.15.67:0/2460652210 learned_addr learned my addr 172.21.15.67:0/2460652210 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:08:32.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.852+0000 7f035de00640 1 --2- 172.21.15.67:0/2460652210 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efb62dbe80 0x55efb62e6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:08:32.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.852+0000 7f035e800640 1 -- 172.21.15.67:0/2460652210 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55efb62dc870 msgr2=0x55efb62ed420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:08:32.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.852+0000 7f035e800640 1 --2- 172.21.15.67:0/2460652210 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55efb62dc870 0x55efb62ed420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:32.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.852+0000 7f035e800640 1 -- 172.21.15.67:0/2460652210 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efb62dbe80 msgr2=0x55efb62e6380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:08:32.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.852+0000 7f035e800640 1 --2- 172.21.15.67:0/2460652210 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efb62dbe80 0x55efb62e6380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:32.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.852+0000 7f035e800640 1 -- 172.21.15.67:0/2460652210 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55efb5f150f0 con 0x55efb62e19e0 2024-03-28T16:08:32.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.853+0000 7f035e800640 1 --2- 172.21.15.67:0/2460652210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55efb62e19e0 0x55efb62dd9f0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f0348005930 tx=0x7f0348022000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ea6df25a06e089d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:08:32.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.853+0000 7f034ea00640 1 -- 172.21.15.67:0/2460652210 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0348037070 con 0x55efb62e19e0 2024-03-28T16:08:32.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.853+0000 7f034ea00640 1 -- 172.21.15.67:0/2460652210 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f03480055a0 con 0x55efb62e19e0 2024-03-28T16:08:32.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.853+0000 7f034ea00640 1 -- 172.21.15.67:0/2460652210 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0348011e60 con 0x55efb62e19e0 2024-03-28T16:08:32.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.854+0000 7f0367f7d740 1 -- 172.21.15.67:0/2460652210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55efb62e19e0 msgr2=0x55efb62dd9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:08:32.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.854+0000 7f0367f7d740 1 --2- 172.21.15.67:0/2460652210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55efb62e19e0 0x55efb62dd9f0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f0348005930 tx=0x7f0348022000 comp rx=0 tx=0).stop 2024-03-28T16:08:32.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.854+0000 7f0367f7d740 1 -- 172.21.15.67:0/2460652210 shutdown_connections 2024-03-28T16:08:32.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.854+0000 7f0367f7d740 1 --2- 172.21.15.67:0/2460652210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55efb62e19e0 0x55efb62dd9f0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:32.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.854+0000 7f0367f7d740 1 --2- 172.21.15.67:0/2460652210 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55efb62dc870 0x55efb62ed420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:32.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.854+0000 7f0367f7d740 1 --2- 172.21.15.67:0/2460652210 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efb62dbe80 0x55efb62e6380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:32.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.854+0000 7f0367f7d740 1 -- 172.21.15.67:0/2460652210 >> 172.21.15.67:0/2460652210 conn(0x55efb62a4200 msgr2=0x55efb62dd2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:08:32.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.854+0000 7f0367f7d740 1 -- 172.21.15.67:0/2460652210 shutdown_connections 2024-03-28T16:08:32.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.855+0000 7f0367f7d740 1 -- 172.21.15.67:0/2460652210 wait complete. 2024-03-28T16:08:32.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.855+0000 7f0367f7d740 1 -- start start 2024-03-28T16:08:32.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.855+0000 7f0367f7d740 1 Processor -- start 2024-03-28T16:08:32.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.856+0000 7f0367f7d740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55efb62dbe80 0x55efb62eb080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:32.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.856+0000 7f0367f7d740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55efb62dc870 0x55efb62eb5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:32.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.856+0000 7f0367f7d740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efb62e19e0 0x55efb62ec5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:32.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.856+0000 7f0367f7d740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03601adcb0 con 0x55efb62dc870 2024-03-28T16:08:32.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.856+0000 7f0367f7d740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03601aefd0 con 0x55efb62e19e0 2024-03-28T16:08:32.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.856+0000 7f0367f7d740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03601ae310 con 0x55efb62dbe80 2024-03-28T16:08:32.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.856+0000 7f0367f7d740 10 client.0 _ll_register_callbacks cb 0x7f03580043b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:08:32.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.856+0000 7f035d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55efb62dc870 0x55efb62eb5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:08:32.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.856+0000 7f035d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55efb62dc870 0x55efb62eb5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46434/0 (socket says 172.21.15.67:46434) 2024-03-28T16:08:32.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.856+0000 7f035d400640 1 -- 172.21.15.67:0/3559955169 learned_addr learned my addr 172.21.15.67:0/3559955169 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:08:32.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.857+0000 7f035e800640 1 --2- 172.21.15.67:0/3559955169 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efb62e19e0 0x55efb62ec5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:08:32.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.857+0000 7f035d400640 1 -- 172.21.15.67:0/3559955169 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55efb62dbe80 msgr2=0x55efb62eb080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:08:32.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.857+0000 7f035d400640 1 --2- 172.21.15.67:0/3559955169 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55efb62dbe80 0x55efb62eb080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:32.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.857+0000 7f035d400640 1 -- 172.21.15.67:0/3559955169 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efb62e19e0 msgr2=0x55efb62ec5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:08:32.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.857+0000 7f035d400640 1 --2- 172.21.15.67:0/3559955169 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efb62e19e0 0x55efb62ec5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:32.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.857+0000 7f035d400640 1 -- 172.21.15.67:0/3559955169 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55efb62ecbc0 con 0x55efb62dc870 2024-03-28T16:08:32.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.857+0000 7f035d400640 1 --2- 172.21.15.67:0/3559955169 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55efb62dc870 0x55efb62eb5c0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f03500099f0 tx=0x7f035002cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8e8577357ec2cbf server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:08:32.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.857+0000 7f0343e00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:08:32.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.858+0000 7f0343e00640 1 -- 172.21.15.67:0/3559955169 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0350004030 con 0x55efb62dc870 2024-03-28T16:08:32.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.858+0000 7f0343e00640 1 -- 172.21.15.67:0/3559955169 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f03500041d0 con 0x55efb62dc870 2024-03-28T16:08:32.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.858+0000 7f0367f7d740 1 -- 172.21.15.67:0/3559955169 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f03480055a0 con 0x55efb62dc870 2024-03-28T16:08:32.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.858+0000 7f0343e00640 1 -- 172.21.15.67:0/3559955169 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f03500043f0 con 0x55efb62dc870 2024-03-28T16:08:32.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.858+0000 7f0367f7d740 10 client.16074 fetch_fsmap learned FSMap version 19 2024-03-28T16:08:32.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.858+0000 7f0367f7d740 1 -- 172.21.15.67:0/3559955169 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55efb634d990 con 0x55efb62dc870 2024-03-28T16:08:32.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.858+0000 7f0343e00640 1 -- 172.21.15.67:0/3559955169 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f03480055a0 con 0x55efb62dc870 2024-03-28T16:08:32.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.858+0000 7f0343e00640 1 -- 172.21.15.67:0/3559955169 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f03500046c0 con 0x55efb62dc870 2024-03-28T16:08:32.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.859+0000 7f0367f7d740 10 client.16074 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:08:32.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.859+0000 7f0367f7d740 10 client.16074 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:08:32.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.859+0000 7f0367f7d740 20 client.16074 populate_metadata read hostname 'smithi067' 2024-03-28T16:08:32.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.859+0000 7f0367f7d740 1 --2- 172.21.15.67:0/3559955169 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55efb634e310 0x55efb6350730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:08:32.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.859+0000 7f0367f7d740 4 client.16074 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T16:08:32.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.859+0000 7f0367f7d740 1 -- 172.21.15.67:0/3559955169 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55efb6248ae0 con 0x55efb634e310 2024-03-28T16:08:32.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.859+0000 7f035de00640 1 --2- 172.21.15.67:0/3559955169 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55efb634e310 0x55efb6350730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:08:32.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.861+0000 7f035de00640 1 --2- 172.21.15.67:0/3559955169 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55efb634e310 0x55efb6350730 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c832bfce56d53963 server_cookie=893846f67696ff0d in_seq=0 out_seq=0 2024-03-28T16:08:32.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.861+0000 7f0343e00640 10 client.16074 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:08:32.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.862+0000 7f0343e00640 1 -- 172.21.15.67:0/3559955169 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55efb6248ae0 con 0x55efb634e310 2024-03-28T16:08:32.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.862+0000 7f0343e00640 10 client.16074 handle_command_reply: tid=0 2024-03-28T16:08:32.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.862+0000 7f0367f7d740 1 client.16074 shutdown 2024-03-28T16:08:32.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.862+0000 7f0367f7d740 1 -- 172.21.15.67:0/3559955169 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55efb62dc870 msgr2=0x55efb62eb5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:08:32.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.862+0000 7f0367f7d740 1 --2- 172.21.15.67:0/3559955169 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55efb62dc870 0x55efb62eb5c0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f03500099f0 tx=0x7f035002cc70 comp rx=0 tx=0).stop 2024-03-28T16:08:32.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.862+0000 7f0367f7d740 1 -- 172.21.15.67:0/3559955169 shutdown_connections 2024-03-28T16:08:32.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.862+0000 7f0367f7d740 1 --2- 172.21.15.67:0/3559955169 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55efb634e310 0x55efb6350730 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:32.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.862+0000 7f0367f7d740 1 --2- 172.21.15.67:0/3559955169 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efb62e19e0 0x55efb62ec5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:32.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.862+0000 7f0367f7d740 1 --2- 172.21.15.67:0/3559955169 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55efb62dc870 0x55efb62eb5c0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:32.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.862+0000 7f0367f7d740 1 --2- 172.21.15.67:0/3559955169 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55efb62dbe80 0x55efb62eb080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:32.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.862+0000 7f0343e00640 0 client.16074 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:08:32.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.862+0000 7f0367f7d740 1 -- 172.21.15.67:0/3559955169 >> 172.21.15.67:0/3559955169 conn(0x55efb62a4200 msgr2=0x55efb62d6a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:08:32.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.863+0000 7f0367f7d740 1 -- 172.21.15.67:0/3559955169 shutdown_connections 2024-03-28T16:08:32.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.863+0000 7f0367f7d740 1 -- 172.21.15.67:0/3559955169 wait complete. 2024-03-28T16:08:32.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.863+0000 7f0367f7d740 20 client.16074 trim_cache size 0 max 16384 2024-03-28T16:08:32.865 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-28T16:08:32.865 INFO:teuthology.orchestra.run.smithi067.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T16:08:32.865 INFO:teuthology.orchestra.run.smithi067.stdout: "scrubs": { 2024-03-28T16:08:32.865 INFO:teuthology.orchestra.run.smithi067.stdout: "5ff8cfe3-c44b-46e1-b8b7-08f862da00d8": { 2024-03-28T16:08:32.865 INFO:teuthology.orchestra.run.smithi067.stdout: "path": "/", 2024-03-28T16:08:32.866 INFO:teuthology.orchestra.run.smithi067.stdout: "tag": "5ff8cfe3-c44b-46e1-b8b7-08f862da00d8", 2024-03-28T16:08:32.866 INFO:teuthology.orchestra.run.smithi067.stdout: "options": "recursive,force" 2024-03-28T16:08:32.866 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:08:32.866 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:08:32.866 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-28T16:08:32.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.864+0000 7f0365200640 1 -- 172.21.15.67:0/360551566 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f03440a28d0 msgr2=0x7f03440a4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:08:32.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.864+0000 7f0365200640 1 --2- 172.21.15.67:0/360551566 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f03440a28d0 0x7f03440a4d90 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f034800d920 tx=0x7f0348000bc0 comp rx=0 tx=0).stop 2024-03-28T16:08:32.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.864+0000 7f0365200640 1 -- 172.21.15.67:0/360551566 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f03601005a0 msgr2=0x7f036019e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:08:32.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.864+0000 7f0365200640 1 --2- 172.21.15.67:0/360551566 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f03601005a0 0x7f036019e640 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f035800ead0 tx=0x7f035800ef90 comp rx=0 tx=0).stop 2024-03-28T16:08:32.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.864+0000 7f0365200640 1 -- 172.21.15.67:0/360551566 shutdown_connections 2024-03-28T16:08:32.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.864+0000 7f0365200640 1 --2- 172.21.15.67:0/360551566 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f03440a28d0 0x7f03440a4d90 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:32.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.864+0000 7f0365200640 1 --2- 172.21.15.67:0/360551566 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0360102150 0x7f03601a5500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:32.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.864+0000 7f0365200640 1 --2- 172.21.15.67:0/360551566 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0360101790 0x7f036019eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:32.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.864+0000 7f0365200640 1 --2- 172.21.15.67:0/360551566 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f03601005a0 0x7f036019e640 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:08:32.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.864+0000 7f0365200640 1 -- 172.21.15.67:0/360551566 >> 172.21.15.67:0/360551566 conn(0x7f03600fbb30 msgr2=0x7f03600688f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:08:32.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.864+0000 7f0365200640 1 -- 172.21.15.67:0/360551566 shutdown_connections 2024-03-28T16:08:32.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:08:32.865+0000 7f0365200640 1 -- 172.21.15.67:0/360551566 wait complete. 2024-03-28T16:08:32.892 INFO:tasks.cephfs.filesystem:scrub status for tag:5ff8cfe3-c44b-46e1-b8b7-08f862da00d8 - {'path': '/', 'tag': '5ff8cfe3-c44b-46e1-b8b7-08f862da00d8', 'options': 'recursive,force'} 2024-03-28T16:08:33.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:08:33 smithi150 ceph-mon[34825]: scrub queued for path: / 2024-03-28T16:08:33.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:08:33 smithi150 ceph-mon[34825]: scrub summary: idle+waiting paths [/] 2024-03-28T16:08:33.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:08:33 smithi150 ceph-mon[34825]: scrub summary: active paths [/] 2024-03-28T16:08:33.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:08:33 smithi150 ceph-mon[34825]: scrub summary: idle+waiting paths [/] 2024-03-28T16:08:33.822 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:08:33 smithi067 ceph-mon[31707]: scrub queued for path: / 2024-03-28T16:08:33.822 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:08:33 smithi067 ceph-mon[31707]: scrub summary: idle+waiting paths [/] 2024-03-28T16:08:33.822 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:08:33 smithi067 ceph-mon[31707]: scrub summary: active paths [/] 2024-03-28T16:08:33.822 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:08:33 smithi067 ceph-mon[31707]: scrub summary: idle+waiting paths [/] 2024-03-28T16:08:33.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:08:33 smithi183 ceph-mon[35580]: scrub queued for path: / 2024-03-28T16:08:33.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:08:33 smithi183 ceph-mon[35580]: scrub summary: idle+waiting paths [/] 2024-03-28T16:08:33.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:08:33 smithi183 ceph-mon[35580]: scrub summary: active paths [/] 2024-03-28T16:08:33.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:08:33 smithi183 ceph-mon[35580]: scrub summary: idle+waiting paths [/] 2024-03-28T16:08:34.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:08:34 smithi150 ceph-mon[34825]: pgmap v589: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 293 MiB/s wr, 75 op/s 2024-03-28T16:08:34.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:08:34 smithi067 ceph-mon[31707]: pgmap v589: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 293 MiB/s wr, 75 op/s 2024-03-28T16:08:34.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:08:34 smithi183 ceph-mon[35580]: pgmap v589: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 293 MiB/s wr, 75 op/s 2024-03-28T16:08:36.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:08:36 smithi150 ceph-mon[34825]: pgmap v590: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 364 MiB/s wr, 93 op/s 2024-03-28T16:08:36.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:08:36 smithi067 ceph-mon[31707]: pgmap v590: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 364 MiB/s wr, 93 op/s 2024-03-28T16:08:36.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:08:36 smithi183 ceph-mon[35580]: pgmap v590: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 364 MiB/s wr, 93 op/s 2024-03-28T16:08:37.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:08:37 smithi150 ceph-mon[34825]: scrub summary: idle 2024-03-28T16:08:37.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:08:37 smithi067 ceph-mon[31707]: scrub summary: idle 2024-03-28T16:08:37.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:08:37 smithi183 ceph-mon[35580]: scrub summary: idle 2024-03-28T16:08:38.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:08:38 smithi150 ceph-mon[34825]: pgmap v591: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 386 MiB/s wr, 98 op/s 2024-03-28T16:08:38.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:08:38 smithi067 ceph-mon[31707]: pgmap v591: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 386 MiB/s wr, 98 op/s 2024-03-28T16:08:38.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:08:38 smithi183 ceph-mon[35580]: pgmap v591: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 386 MiB/s wr, 98 op/s 2024-03-28T16:08:40.802 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:08:40 smithi150 ceph-mon[34825]: pgmap v592: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 429 MiB/s wr, 109 op/s 2024-03-28T16:08:40.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:08:40 smithi067 ceph-mon[31707]: pgmap v592: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 429 MiB/s wr, 109 op/s 2024-03-28T16:08:40.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:08:40 smithi183 ceph-mon[35580]: pgmap v592: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 429 MiB/s wr, 109 op/s 2024-03-28T16:08:42.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:08:42 smithi067 ceph-mon[31707]: pgmap v593: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 484 MiB/s wr, 123 op/s 2024-03-28T16:08:42.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:08:42 smithi183 ceph-mon[35580]: pgmap v593: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 484 MiB/s wr, 123 op/s 2024-03-28T16:08:43.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:08:42 smithi150 ceph-mon[34825]: pgmap v593: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 484 MiB/s wr, 123 op/s 2024-03-28T16:08:44.819 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:08:44 smithi067 ceph-mon[31707]: pgmap v594: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 468 MiB/s wr, 118 op/s 2024-03-28T16:08:44.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:08:44 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T16:08:44.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:08:44 smithi183 ceph-mon[35580]: pgmap v594: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 468 MiB/s wr, 118 op/s 2024-03-28T16:08:44.850 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:08:44 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T16:08:45.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:08:44 smithi150 ceph-mon[34825]: pgmap v594: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 468 MiB/s wr, 118 op/s 2024-03-28T16:08:45.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:08:44 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T16:08:46.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:08:45 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:08:46.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:08:45 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T16:08:46.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:08:45 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:08:46.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:08:45 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:08:46.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:08:45 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T16:08:46.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:08:45 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:08:46.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:08:45 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:08:46.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:08:45 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T16:08:46.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:08:45 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:08:46.248 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-03-28T16:08:46.248 INFO:tasks.workunit.client.0.smithi067.stdout: Children see throughput for 1 rewriters = 464901.28 kB/sec 2024-03-28T16:08:46.248 INFO:tasks.workunit.client.0.smithi067.stdout: Parent sees throughput for 1 rewriters = 464875.15 kB/sec 2024-03-28T16:08:46.248 INFO:tasks.workunit.client.0.smithi067.stdout: Min throughput per process = 464901.28 kB/sec 2024-03-28T16:08:46.248 INFO:tasks.workunit.client.0.smithi067.stdout: Max throughput per process = 464901.28 kB/sec 2024-03-28T16:08:46.248 INFO:tasks.workunit.client.0.smithi067.stdout: Avg throughput per process = 464901.28 kB/sec 2024-03-28T16:08:46.248 INFO:tasks.workunit.client.0.smithi067.stdout: Min xfer = 10485760.00 kB 2024-03-28T16:08:47.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:08:46 smithi150 ceph-mon[34825]: pgmap v595: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 473 MiB/s wr, 120 op/s 2024-03-28T16:08:47.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:08:46 smithi067 ceph-mon[31707]: pgmap v595: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 473 MiB/s wr, 120 op/s 2024-03-28T16:08:47.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:08:46 smithi183 ceph-mon[35580]: pgmap v595: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 473 MiB/s wr, 120 op/s 2024-03-28T16:08:49.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:08:48 smithi150 ceph-mon[34825]: pgmap v596: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 422 MiB/s wr, 107 op/s 2024-03-28T16:08:49.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:08:48 smithi067 ceph-mon[31707]: pgmap v596: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 422 MiB/s wr, 107 op/s 2024-03-28T16:08:49.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:08:48 smithi183 ceph-mon[35580]: pgmap v596: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 422 MiB/s wr, 107 op/s 2024-03-28T16:08:51.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:08:50 smithi150 ceph-mon[34825]: pgmap v597: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 433 MiB/s wr, 109 op/s 2024-03-28T16:08:51.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:08:50 smithi067 ceph-mon[31707]: pgmap v597: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 433 MiB/s wr, 109 op/s 2024-03-28T16:08:51.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:08:50 smithi183 ceph-mon[35580]: pgmap v597: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 433 MiB/s wr, 109 op/s 2024-03-28T16:08:51.673 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-03-28T16:08:51.673 INFO:tasks.workunit.client.0.smithi067.stdout: Children see throughput for 1 readers = 6003554.00 kB/sec 2024-03-28T16:08:51.673 INFO:tasks.workunit.client.0.smithi067.stdout: Parent sees throughput for 1 readers = 6000678.42 kB/sec 2024-03-28T16:08:51.673 INFO:tasks.workunit.client.0.smithi067.stdout: Min throughput per process = 6003554.00 kB/sec 2024-03-28T16:08:51.673 INFO:tasks.workunit.client.0.smithi067.stdout: Max throughput per process = 6003554.00 kB/sec 2024-03-28T16:08:51.673 INFO:tasks.workunit.client.0.smithi067.stdout: Avg throughput per process = 6003554.00 kB/sec 2024-03-28T16:08:51.673 INFO:tasks.workunit.client.0.smithi067.stdout: Min xfer = 10485760.00 kB 2024-03-28T16:08:53.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:08:52 smithi150 ceph-mon[34825]: pgmap v598: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 392 MiB/s wr, 100 op/s 2024-03-28T16:08:53.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:08:52 smithi067 ceph-mon[31707]: pgmap v598: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 392 MiB/s wr, 100 op/s 2024-03-28T16:08:53.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:08:52 smithi183 ceph-mon[35580]: pgmap v598: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 392 MiB/s wr, 100 op/s 2024-03-28T16:08:55.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:08:54 smithi150 ceph-mon[34825]: pgmap v599: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 287 MiB/s wr, 73 op/s 2024-03-28T16:08:55.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:08:54 smithi067 ceph-mon[31707]: pgmap v599: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 287 MiB/s wr, 73 op/s 2024-03-28T16:08:55.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:08:54 smithi183 ceph-mon[35580]: pgmap v599: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 287 MiB/s wr, 73 op/s 2024-03-28T16:08:56.372 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-03-28T16:08:56.372 INFO:tasks.workunit.client.0.smithi067.stdout: Children see throughput for 1 re-readers = 4014846.75 kB/sec 2024-03-28T16:08:56.372 INFO:tasks.workunit.client.0.smithi067.stdout: Parent sees throughput for 1 re-readers = 4013629.58 kB/sec 2024-03-28T16:08:56.372 INFO:tasks.workunit.client.0.smithi067.stdout: Min throughput per process = 4014846.75 kB/sec 2024-03-28T16:08:56.372 INFO:tasks.workunit.client.0.smithi067.stdout: Max throughput per process = 4014846.75 kB/sec 2024-03-28T16:08:56.372 INFO:tasks.workunit.client.0.smithi067.stdout: Avg throughput per process = 4014846.75 kB/sec 2024-03-28T16:08:56.372 INFO:tasks.workunit.client.0.smithi067.stdout: Min xfer = 10485760.00 kB 2024-03-28T16:08:57.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:08:56 smithi150 ceph-mon[34825]: pgmap v600: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 202 MiB/s wr, 53 op/s 2024-03-28T16:08:57.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:08:56 smithi067 ceph-mon[31707]: pgmap v600: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 202 MiB/s wr, 53 op/s 2024-03-28T16:08:57.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:08:56 smithi183 ceph-mon[35580]: pgmap v600: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 202 MiB/s wr, 53 op/s 2024-03-28T16:08:59.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:08:58 smithi150 ceph-mon[34825]: pgmap v601: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 114 MiB/s wr, 30 op/s 2024-03-28T16:08:59.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:08:58 smithi067 ceph-mon[31707]: pgmap v601: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 114 MiB/s wr, 30 op/s 2024-03-28T16:08:59.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:08:58 smithi183 ceph-mon[35580]: pgmap v601: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 114 MiB/s wr, 30 op/s 2024-03-28T16:09:01.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:00 smithi150 ceph-mon[34825]: pgmap v602: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 75 MiB/s wr, 20 op/s 2024-03-28T16:09:01.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:00 smithi067 ceph-mon[31707]: pgmap v602: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 75 MiB/s wr, 20 op/s 2024-03-28T16:09:01.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:00 smithi183 ceph-mon[35580]: pgmap v602: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 75 MiB/s wr, 20 op/s 2024-03-28T16:09:02.894 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T16:09:02.894 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T16:09:02.986 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:02 smithi067 ceph-mon[31707]: pgmap v603: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 22 MiB/s wr, 8 op/s 2024-03-28T16:09:03.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:02 smithi150 ceph-mon[34825]: pgmap v603: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 22 MiB/s wr, 8 op/s 2024-03-28T16:09:03.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:02 smithi183 ceph-mon[35580]: pgmap v603: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 22 MiB/s wr, 8 op/s 2024-03-28T16:09:03.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.128+0000 7f21cf200640 1 -- 172.21.15.67:0/3765993240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21d00740a0 msgr2=0x7f21d0074500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:03.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.128+0000 7f21cf200640 1 --2- 172.21.15.67:0/3765993240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21d00740a0 0x7f21d0074500 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f21b8009900 tx=0x7f21b802bcc0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.128+0000 7f21cf200640 1 -- 172.21.15.67:0/3765993240 shutdown_connections 2024-03-28T16:09:03.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.128+0000 7f21cf200640 1 --2- 172.21.15.67:0/3765993240 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f21d0109410 0x7f21d010b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.128+0000 7f21cf200640 1 --2- 172.21.15.67:0/3765993240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21d00740a0 0x7f21d0074500 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.128+0000 7f21cf200640 1 --2- 172.21.15.67:0/3765993240 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21d0075680 0x7f21d0073ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.128+0000 7f21cf200640 1 -- 172.21.15.67:0/3765993240 >> 172.21.15.67:0/3765993240 conn(0x7f21d00fcb10 msgr2=0x7f21d00fef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:03.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.128+0000 7f21cf200640 1 -- 172.21.15.67:0/3765993240 shutdown_connections 2024-03-28T16:09:03.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.129+0000 7f21cf200640 1 -- 172.21.15.67:0/3765993240 wait complete. 2024-03-28T16:09:03.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.129+0000 7f21cf200640 1 Processor -- start 2024-03-28T16:09:03.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.130+0000 7f21cf200640 1 -- start start 2024-03-28T16:09:03.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.130+0000 7f21cf200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f21d00740a0 0x7f21d01999b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:03.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.130+0000 7f21cf200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21d0075680 0x7f21d0199ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:03.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.130+0000 7f21cf200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21d0109410 0x7f21d01a0f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:03.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.130+0000 7f21cf200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21d010f2b0 con 0x7f21d0075680 2024-03-28T16:09:03.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.130+0000 7f21cf200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21d010f130 con 0x7f21d0109410 2024-03-28T16:09:03.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.130+0000 7f21cf200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21d010f430 con 0x7f21d00740a0 2024-03-28T16:09:03.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.130+0000 7f21c7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21d0075680 0x7f21d0199ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:03.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.130+0000 7f21c7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21d0075680 0x7f21d0199ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51750/0 (socket says 172.21.15.67:51750) 2024-03-28T16:09:03.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.130+0000 7f21c7e00640 1 -- 172.21.15.67:0/965668679 learned_addr learned my addr 172.21.15.67:0/965668679 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:09:03.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.131+0000 7f21cde00640 1 --2- 172.21.15.67:0/965668679 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f21d00740a0 0x7f21d01999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:03.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.131+0000 7f21ce800640 1 --2- 172.21.15.67:0/965668679 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21d0109410 0x7f21d01a0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:03.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.131+0000 7f21c7e00640 1 -- 172.21.15.67:0/965668679 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f21d00740a0 msgr2=0x7f21d01999b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:03.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.131+0000 7f21c7e00640 1 --2- 172.21.15.67:0/965668679 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f21d00740a0 0x7f21d01999b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.131+0000 7f21c7e00640 1 -- 172.21.15.67:0/965668679 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21d0109410 msgr2=0x7f21d01a0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:03.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.131+0000 7f21c7e00640 1 --2- 172.21.15.67:0/965668679 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21d0109410 0x7f21d01a0f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.131+0000 7f21c7e00640 1 -- 172.21.15.67:0/965668679 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21d01a15d0 con 0x7f21d0075680 2024-03-28T16:09:03.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.131+0000 7f21c7e00640 1 --2- 172.21.15.67:0/965668679 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21d0075680 0x7f21d0199ef0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f21b8006010 tx=0x7f21b802d980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:03.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.132+0000 7f21c6a00640 1 -- 172.21.15.67:0/965668679 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21b8043020 con 0x7f21d0075680 2024-03-28T16:09:03.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.132+0000 7f21c6a00640 1 -- 172.21.15.67:0/965668679 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f21b802dd80 con 0x7f21d0075680 2024-03-28T16:09:03.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.132+0000 7f21cf200640 1 -- 172.21.15.67:0/965668679 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21d01a1860 con 0x7f21d0075680 2024-03-28T16:09:03.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.132+0000 7f21c6a00640 1 -- 172.21.15.67:0/965668679 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21b8003bb0 con 0x7f21d0075680 2024-03-28T16:09:03.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.132+0000 7f21cf200640 1 -- 172.21.15.67:0/965668679 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21d01a1db0 con 0x7f21d0075680 2024-03-28T16:09:03.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.135+0000 7f21d7860740 1 Processor -- start 2024-03-28T16:09:03.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.135+0000 7f21c6a00640 1 -- 172.21.15.67:0/965668679 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f21b8046020 con 0x7f21d0075680 2024-03-28T16:09:03.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.135+0000 7f21d7860740 1 -- start start 2024-03-28T16:09:03.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.135+0000 7f21d7860740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c0fd43b050 0x55c0fd43b470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:03.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.135+0000 7f21d7860740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c0fd43ba40 0x55c0fd440540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:03.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.135+0000 7f21d7860740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c0fd440bb0 0x55c0fd443040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:03.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.135+0000 7f21d7860740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c0fd3a0250 con 0x55c0fd440bb0 2024-03-28T16:09:03.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.135+0000 7f21d7860740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c0fd348a30 con 0x55c0fd43ba40 2024-03-28T16:09:03.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.135+0000 7f21d7860740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c0fd39c2f0 con 0x55c0fd43b050 2024-03-28T16:09:03.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.136+0000 7f21c7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c0fd440bb0 0x55c0fd443040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:03.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.136+0000 7f21c7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c0fd440bb0 0x55c0fd443040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51758/0 (socket says 172.21.15.67:51758) 2024-03-28T16:09:03.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.136+0000 7f21c7e00640 1 -- 172.21.15.67:0/2618492501 learned_addr learned my addr 172.21.15.67:0/2618492501 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:09:03.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.136+0000 7f21ce800640 1 --2- 172.21.15.67:0/2618492501 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c0fd43b050 0x55c0fd43b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:03.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.136+0000 7f21cde00640 1 --2- 172.21.15.67:0/2618492501 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c0fd43ba40 0x55c0fd440540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:03.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.136+0000 7f21c7e00640 1 -- 172.21.15.67:0/2618492501 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c0fd43b050 msgr2=0x55c0fd43b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:09:03.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.136+0000 7f21c7e00640 1 --2- 172.21.15.67:0/2618492501 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c0fd43b050 0x55c0fd43b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.136+0000 7f21c7e00640 1 -- 172.21.15.67:0/2618492501 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c0fd43ba40 msgr2=0x55c0fd440540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:09:03.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.136+0000 7f21c7e00640 1 --2- 172.21.15.67:0/2618492501 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c0fd43ba40 0x55c0fd440540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.136+0000 7f21c7e00640 1 -- 172.21.15.67:0/2618492501 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c0fd333690 con 0x55c0fd440bb0 2024-03-28T16:09:03.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.136+0000 7f21ce800640 1 --2- 172.21.15.67:0/2618492501 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c0fd43b050 0x55c0fd43b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T16:09:03.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.137+0000 7f21c7e00640 1 --2- 172.21.15.67:0/2618492501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c0fd440bb0 0x55c0fd443040 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f21b80f4d40 tx=0x7f21b80f4e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30f9488cb82f93a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:03.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.137+0000 7f21c6a00640 1 --2- 172.21.15.67:0/965668679 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f21b40a2800 0x7f21b40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:03.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.137+0000 7f21bf000640 1 -- 172.21.15.67:0/2618492501 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21b8100070 con 0x55c0fd440bb0 2024-03-28T16:09:03.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.137+0000 7f21bf000640 1 -- 172.21.15.67:0/2618492501 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f21b80bddf0 con 0x55c0fd440bb0 2024-03-28T16:09:03.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.137+0000 7f21bf000640 1 -- 172.21.15.67:0/2618492501 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21b809f790 con 0x55c0fd440bb0 2024-03-28T16:09:03.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.137+0000 7f21c6a00640 1 -- 172.21.15.67:0/965668679 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f21b80f0da0 con 0x7f21d0075680 2024-03-28T16:09:03.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.137+0000 7f21ce800640 1 --2- 172.21.15.67:0/965668679 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f21b40a2800 0x7f21b40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:03.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.138+0000 7f21d7860740 1 -- 172.21.15.67:0/2618492501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c0fd440bb0 msgr2=0x55c0fd443040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:03.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.138+0000 7f21d7860740 1 --2- 172.21.15.67:0/2618492501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c0fd440bb0 0x55c0fd443040 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f21b80f4d40 tx=0x7f21b80f4e30 comp rx=0 tx=0).stop 2024-03-28T16:09:03.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.138+0000 7f21ce800640 1 --2- 172.21.15.67:0/965668679 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f21b40a2800 0x7f21b40a4cc0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x55c0fd43c2a0 tx=0x7f21c8009580 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:03.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.138+0000 7f21d7860740 1 -- 172.21.15.67:0/2618492501 shutdown_connections 2024-03-28T16:09:03.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.138+0000 7f21d7860740 1 --2- 172.21.15.67:0/2618492501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c0fd440bb0 0x55c0fd443040 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.138+0000 7f21d7860740 1 --2- 172.21.15.67:0/2618492501 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c0fd43ba40 0x55c0fd440540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.138+0000 7f21d7860740 1 --2- 172.21.15.67:0/2618492501 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c0fd43b050 0x55c0fd43b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.138+0000 7f21d7860740 1 -- 172.21.15.67:0/2618492501 >> 172.21.15.67:0/2618492501 conn(0x55c0fd403200 msgr2=0x55c0fd4369c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:03.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.138+0000 7f21d7860740 1 -- 172.21.15.67:0/2618492501 shutdown_connections 2024-03-28T16:09:03.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.138+0000 7f21d7860740 1 -- 172.21.15.67:0/2618492501 wait complete. 2024-03-28T16:09:03.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.138+0000 7f21d7860740 1 -- start start 2024-03-28T16:09:03.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.139+0000 7f21d7860740 1 Processor -- start 2024-03-28T16:09:03.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.139+0000 7f21d7860740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c0fd43b050 0x55c0fd444cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:03.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.139+0000 7f21d7860740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c0fd43ba40 0x55c0fd445230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:03.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.139+0000 7f21d7860740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c0fd440bb0 0x55c0fd44c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:03.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.139+0000 7f21d7860740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c0fd443920 con 0x55c0fd43ba40 2024-03-28T16:09:03.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.139+0000 7f21d7860740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c0fd35be80 con 0x55c0fd43b050 2024-03-28T16:09:03.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.139+0000 7f21d7860740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c0fd443aa0 con 0x55c0fd440bb0 2024-03-28T16:09:03.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.139+0000 7f21d7860740 10 client.0 _ll_register_callbacks cb 0x55c0fd36e130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:09:03.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.139+0000 7f21c7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c0fd43ba40 0x55c0fd445230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:03.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.139+0000 7f21c7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c0fd43ba40 0x55c0fd445230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51770/0 (socket says 172.21.15.67:51770) 2024-03-28T16:09:03.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.140+0000 7f21c7e00640 1 -- 172.21.15.67:0/2802270428 learned_addr learned my addr 172.21.15.67:0/2802270428 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:09:03.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.140+0000 7f21ce800640 1 --2- 172.21.15.67:0/2802270428 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c0fd440bb0 0x55c0fd44c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:03.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.140+0000 7f21cde00640 1 --2- 172.21.15.67:0/2802270428 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c0fd43b050 0x55c0fd444cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:03.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.140+0000 7f21c7e00640 1 -- 172.21.15.67:0/2802270428 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c0fd440bb0 msgr2=0x55c0fd44c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:09:03.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.140+0000 7f21c7e00640 1 --2- 172.21.15.67:0/2802270428 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c0fd440bb0 0x55c0fd44c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.140+0000 7f21c7e00640 1 -- 172.21.15.67:0/2802270428 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c0fd43b050 msgr2=0x55c0fd444cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:09:03.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.140+0000 7f21c7e00640 1 --2- 172.21.15.67:0/2802270428 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c0fd43b050 0x55c0fd444cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.140+0000 7f21c7e00640 1 -- 172.21.15.67:0/2802270428 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c0fd17e7d0 con 0x55c0fd43ba40 2024-03-28T16:09:03.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.140+0000 7f21ce800640 1 --2- 172.21.15.67:0/2802270428 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c0fd440bb0 0x55c0fd44c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:09:03.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.140+0000 7f21c7e00640 1 --2- 172.21.15.67:0/2802270428 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c0fd43ba40 0x55c0fd445230 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f21b80bdfa0 tx=0x7f21b80f43d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=829354cb22fcb529 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:03.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.140+0000 7f21b3400640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:09:03.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.140+0000 7f21b3400640 1 -- 172.21.15.67:0/2802270428 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21b80a3cb0 con 0x55c0fd43ba40 2024-03-28T16:09:03.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.141+0000 7f21b3400640 1 -- 172.21.15.67:0/2802270428 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f21b80ee960 con 0x55c0fd43ba40 2024-03-28T16:09:03.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.141+0000 7f21d7860740 1 -- 172.21.15.67:0/2802270428 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f21b80bddf0 con 0x55c0fd43ba40 2024-03-28T16:09:03.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.141+0000 7f21b3400640 1 -- 172.21.15.67:0/2802270428 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f21b80a1af0 con 0x55c0fd43ba40 2024-03-28T16:09:03.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.141+0000 7f21d7860740 10 client.16086 fetch_fsmap learned FSMap version 19 2024-03-28T16:09:03.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.141+0000 7f21d7860740 1 -- 172.21.15.67:0/2802270428 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55c0fd0b43f0 con 0x55c0fd43ba40 2024-03-28T16:09:03.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.141+0000 7f21b3400640 1 -- 172.21.15.67:0/2802270428 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f21b80bddf0 con 0x55c0fd43ba40 2024-03-28T16:09:03.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.141+0000 7f21b3400640 1 -- 172.21.15.67:0/2802270428 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f21b8005500 con 0x55c0fd43ba40 2024-03-28T16:09:03.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.141+0000 7f21d7860740 10 client.16086 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:09:03.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.141+0000 7f21d7860740 10 client.16086 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:09:03.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.141+0000 7f21d7860740 20 client.16086 populate_metadata read hostname 'smithi067' 2024-03-28T16:09:03.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.141+0000 7f21d7860740 1 --2- 172.21.15.67:0/2802270428 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55c0fd450850 0x55c0fd452c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:03.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.141+0000 7f21d7860740 4 client.16086 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:09:03.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.141+0000 7f21d7860740 1 -- 172.21.15.67:0/2802270428 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c0fd3ad790 con 0x55c0fd450850 2024-03-28T16:09:03.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.142+0000 7f21cde00640 1 --2- 172.21.15.67:0/2802270428 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55c0fd450850 0x55c0fd452c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:03.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.143+0000 7f21cde00640 1 --2- 172.21.15.67:0/2802270428 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55c0fd450850 0x55c0fd452c40 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c35a636439003fc7 server_cookie=43127985d787fe3 in_seq=0 out_seq=0 2024-03-28T16:09:03.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.143+0000 7f21b3400640 10 client.16086 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:09:03.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.145+0000 7f21b3400640 1 -- 172.21.15.67:0/2802270428 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c0fd3ad790 con 0x55c0fd450850 2024-03-28T16:09:03.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.145+0000 7f21b3400640 10 client.16086 handle_command_reply: tid=0 2024-03-28T16:09:03.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.145+0000 7f21d7860740 1 client.16086 shutdown 2024-03-28T16:09:03.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.145+0000 7f21d7860740 1 -- 172.21.15.67:0/2802270428 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c0fd43ba40 msgr2=0x55c0fd445230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:03.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.145+0000 7f21d7860740 1 --2- 172.21.15.67:0/2802270428 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c0fd43ba40 0x55c0fd445230 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f21b80bdfa0 tx=0x7f21b80f43d0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.145+0000 7f21d7860740 1 -- 172.21.15.67:0/2802270428 shutdown_connections 2024-03-28T16:09:03.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.145+0000 7f21d7860740 1 --2- 172.21.15.67:0/2802270428 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55c0fd450850 0x55c0fd452c40 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.145+0000 7f21d7860740 1 --2- 172.21.15.67:0/2802270428 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c0fd440bb0 0x55c0fd44c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.145+0000 7f21d7860740 1 --2- 172.21.15.67:0/2802270428 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c0fd43ba40 0x55c0fd445230 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.145+0000 7f21d7860740 1 --2- 172.21.15.67:0/2802270428 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c0fd43b050 0x55c0fd444cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.145+0000 7f21b3400640 0 client.16086 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:09:03.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.145+0000 7f21d7860740 1 -- 172.21.15.67:0/2802270428 >> 172.21.15.67:0/2802270428 conn(0x55c0fd403200 msgr2=0x55c0fd435bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:03.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.146+0000 7f21d7860740 1 -- 172.21.15.67:0/2802270428 shutdown_connections 2024-03-28T16:09:03.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.146+0000 7f21d7860740 1 -- 172.21.15.67:0/2802270428 wait complete. 2024-03-28T16:09:03.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.146+0000 7f21d7860740 20 client.16086 trim_cache size 0 max 16384 2024-03-28T16:09:03.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.163+0000 7f21d7860740 1 Processor -- start 2024-03-28T16:09:03.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.163+0000 7f21d7860740 1 -- start start 2024-03-28T16:09:03.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.164+0000 7f21d7860740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c0fd43b050 0x55c0fd436bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:03.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.164+0000 7f21d7860740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c0fd43ba40 0x55c0fd44c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:03.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.164+0000 7f21d7860740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c0fd440bb0 0x55c0fd439c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:03.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.164+0000 7f21d7860740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21d01a89f0 con 0x55c0fd43ba40 2024-03-28T16:09:03.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.164+0000 7f21d7860740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21d01aafb0 con 0x55c0fd43b050 2024-03-28T16:09:03.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.164+0000 7f21d7860740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21d01a8ff0 con 0x55c0fd440bb0 2024-03-28T16:09:03.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.164+0000 7f21c7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c0fd43ba40 0x55c0fd44c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:03.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.164+0000 7f21c7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c0fd43ba40 0x55c0fd44c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51780/0 (socket says 172.21.15.67:51780) 2024-03-28T16:09:03.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.164+0000 7f21c7e00640 1 -- 172.21.15.67:0/2920014478 learned_addr learned my addr 172.21.15.67:0/2920014478 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:09:03.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.164+0000 7f21c7e00640 1 -- 172.21.15.67:0/2920014478 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c0fd440bb0 msgr2=0x55c0fd439c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:09:03.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.164+0000 7f21c7e00640 1 --2- 172.21.15.67:0/2920014478 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c0fd440bb0 0x55c0fd439c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.164+0000 7f21cde00640 1 --2- 172.21.15.67:0/2920014478 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c0fd43b050 0x55c0fd436bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:03.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.164+0000 7f21c7e00640 1 -- 172.21.15.67:0/2920014478 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c0fd43b050 msgr2=0x55c0fd436bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:09:03.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.164+0000 7f21c7e00640 1 --2- 172.21.15.67:0/2920014478 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c0fd43b050 0x55c0fd436bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.164+0000 7f21c7e00640 1 -- 172.21.15.67:0/2920014478 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c0fd0740f0 con 0x55c0fd43ba40 2024-03-28T16:09:03.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.165+0000 7f21c7e00640 1 --2- 172.21.15.67:0/2920014478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c0fd43ba40 0x55c0fd44c230 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f21b80b9b50 tx=0x7f21b80f4e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0868a1f1f953058 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:03.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.165+0000 7f21be600640 1 -- 172.21.15.67:0/2920014478 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21b80fd020 con 0x55c0fd43ba40 2024-03-28T16:09:03.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.165+0000 7f21be600640 1 -- 172.21.15.67:0/2920014478 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f21b80bddf0 con 0x55c0fd43ba40 2024-03-28T16:09:03.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.165+0000 7f21be600640 1 -- 172.21.15.67:0/2920014478 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21b8100070 con 0x55c0fd43ba40 2024-03-28T16:09:03.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.166+0000 7f21d7860740 1 -- 172.21.15.67:0/2920014478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c0fd43ba40 msgr2=0x55c0fd44c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:03.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.166+0000 7f21d7860740 1 --2- 172.21.15.67:0/2920014478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c0fd43ba40 0x55c0fd44c230 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f21b80b9b50 tx=0x7f21b80f4e30 comp rx=0 tx=0).stop 2024-03-28T16:09:03.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.166+0000 7f21d7860740 1 -- 172.21.15.67:0/2920014478 shutdown_connections 2024-03-28T16:09:03.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.166+0000 7f21d7860740 1 --2- 172.21.15.67:0/2920014478 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c0fd440bb0 0x55c0fd439c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.166+0000 7f21d7860740 1 --2- 172.21.15.67:0/2920014478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c0fd43ba40 0x55c0fd44c230 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.166+0000 7f21d7860740 1 --2- 172.21.15.67:0/2920014478 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c0fd43b050 0x55c0fd436bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.166+0000 7f21d7860740 1 -- 172.21.15.67:0/2920014478 >> 172.21.15.67:0/2920014478 conn(0x55c0fd403200 msgr2=0x55c0fd4364e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:03.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.166+0000 7f21d7860740 1 -- 172.21.15.67:0/2920014478 shutdown_connections 2024-03-28T16:09:03.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.166+0000 7f21d7860740 1 -- 172.21.15.67:0/2920014478 wait complete. 2024-03-28T16:09:03.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.166+0000 7f21d7860740 1 -- start start 2024-03-28T16:09:03.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.166+0000 7f21d7860740 1 Processor -- start 2024-03-28T16:09:03.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.167+0000 7f21d7860740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c0fd43b050 0x55c0fd44a110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:03.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.167+0000 7f21d7860740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c0fd43ba40 0x55c0fd44a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:03.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.167+0000 7f21d7860740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c0fd440bb0 0x55c0fd44b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:03.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.167+0000 7f21d7860740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21d01a9630 con 0x55c0fd440bb0 2024-03-28T16:09:03.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.167+0000 7f21d7860740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21d01aa950 con 0x55c0fd43b050 2024-03-28T16:09:03.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.167+0000 7f21d7860740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21d01a9c90 con 0x55c0fd43ba40 2024-03-28T16:09:03.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.167+0000 7f21d7860740 10 client.0 _ll_register_callbacks cb 0x7f21c000b770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:09:03.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.167+0000 7f21ce800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c0fd440bb0 0x55c0fd44b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:03.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.167+0000 7f21ce800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c0fd440bb0 0x55c0fd44b660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51794/0 (socket says 172.21.15.67:51794) 2024-03-28T16:09:03.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.167+0000 7f21ce800640 1 -- 172.21.15.67:0/3556423655 learned_addr learned my addr 172.21.15.67:0/3556423655 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:09:03.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.167+0000 7f21cde00640 1 --2- 172.21.15.67:0/3556423655 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c0fd43b050 0x55c0fd44a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:03.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.167+0000 7f21c7e00640 1 --2- 172.21.15.67:0/3556423655 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c0fd43ba40 0x55c0fd44a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:03.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.168+0000 7f21ce800640 1 -- 172.21.15.67:0/3556423655 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c0fd43ba40 msgr2=0x55c0fd44a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:09:03.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.168+0000 7f21ce800640 1 --2- 172.21.15.67:0/3556423655 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c0fd43ba40 0x55c0fd44a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.168+0000 7f21ce800640 1 -- 172.21.15.67:0/3556423655 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c0fd43b050 msgr2=0x55c0fd44a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:09:03.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.168+0000 7f21ce800640 1 --2- 172.21.15.67:0/3556423655 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c0fd43b050 0x55c0fd44a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.168+0000 7f21ce800640 1 -- 172.21.15.67:0/3556423655 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c0fd44bc50 con 0x55c0fd440bb0 2024-03-28T16:09:03.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.168+0000 7f21ce800640 1 --2- 172.21.15.67:0/3556423655 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c0fd440bb0 0x55c0fd44b660 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f21c8012980 tx=0x7f21c8004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc36bd702ac6c665 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:03.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.168+0000 7f21b3400640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:09:03.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.168+0000 7f21b3400640 1 -- 172.21.15.67:0/3556423655 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21c801ecc0 con 0x55c0fd440bb0 2024-03-28T16:09:03.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.168+0000 7f21b3400640 1 -- 172.21.15.67:0/3556423655 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f21c8028450 con 0x55c0fd440bb0 2024-03-28T16:09:03.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.168+0000 7f21d7860740 1 -- 172.21.15.67:0/3556423655 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f21b80bddf0 con 0x55c0fd440bb0 2024-03-28T16:09:03.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.168+0000 7f21b3400640 1 -- 172.21.15.67:0/3556423655 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f21c801ee60 con 0x55c0fd440bb0 2024-03-28T16:09:03.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.168+0000 7f21d7860740 10 client.16092 fetch_fsmap learned FSMap version 19 2024-03-28T16:09:03.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.169+0000 7f21d7860740 1 -- 172.21.15.67:0/3556423655 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55c0fd4ac990 con 0x55c0fd440bb0 2024-03-28T16:09:03.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.169+0000 7f21b3400640 1 -- 172.21.15.67:0/3556423655 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f21b80bddf0 con 0x55c0fd440bb0 2024-03-28T16:09:03.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.169+0000 7f21b3400640 1 -- 172.21.15.67:0/3556423655 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f21c800f040 con 0x55c0fd440bb0 2024-03-28T16:09:03.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.169+0000 7f21d7860740 10 client.16092 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:09:03.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.169+0000 7f21d7860740 10 client.16092 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:09:03.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.169+0000 7f21d7860740 20 client.16092 populate_metadata read hostname 'smithi067' 2024-03-28T16:09:03.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.169+0000 7f21d7860740 1 --2- 172.21.15.67:0/3556423655 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55c0fd4ad3a0 0x55c0fd4af7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:03.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.169+0000 7f21d7860740 4 client.16092 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T16:09:03.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.169+0000 7f21d7860740 1 -- 172.21.15.67:0/3556423655 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55c0fd3a7ae0 con 0x55c0fd4ad3a0 2024-03-28T16:09:03.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.170+0000 7f21cde00640 1 --2- 172.21.15.67:0/3556423655 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55c0fd4ad3a0 0x55c0fd4af7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:03.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.170+0000 7f21cde00640 1 --2- 172.21.15.67:0/3556423655 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55c0fd4ad3a0 0x55c0fd4af7c0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9613ed9314574afa server_cookie=1cb05ccd0a781956 in_seq=0 out_seq=0 2024-03-28T16:09:03.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.170+0000 7f21b3400640 10 client.16092 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:09:03.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.171+0000 7f21b3400640 1 -- 172.21.15.67:0/3556423655 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55c0fd3a7ae0 con 0x55c0fd4ad3a0 2024-03-28T16:09:03.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.171+0000 7f21b3400640 10 client.16092 handle_command_reply: tid=0 2024-03-28T16:09:03.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.171+0000 7f21d7860740 1 client.16092 shutdown 2024-03-28T16:09:03.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.171+0000 7f21d7860740 1 -- 172.21.15.67:0/3556423655 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c0fd440bb0 msgr2=0x55c0fd44b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:03.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.171+0000 7f21d7860740 1 --2- 172.21.15.67:0/3556423655 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c0fd440bb0 0x55c0fd44b660 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f21c8012980 tx=0x7f21c8004ab0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.171+0000 7f21d7860740 1 -- 172.21.15.67:0/3556423655 shutdown_connections 2024-03-28T16:09:03.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.171+0000 7f21d7860740 1 --2- 172.21.15.67:0/3556423655 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55c0fd4ad3a0 0x55c0fd4af7c0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.171+0000 7f21d7860740 1 --2- 172.21.15.67:0/3556423655 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c0fd440bb0 0x55c0fd44b660 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.171+0000 7f21d7860740 1 --2- 172.21.15.67:0/3556423655 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c0fd43ba40 0x55c0fd44a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.171+0000 7f21d7860740 1 --2- 172.21.15.67:0/3556423655 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c0fd43b050 0x55c0fd44a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.171+0000 7f21d7860740 1 -- 172.21.15.67:0/3556423655 >> 172.21.15.67:0/3556423655 conn(0x55c0fd403200 msgr2=0x55c0fd435330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:03.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.171+0000 7f21b3400640 0 client.16092 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:09:03.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.171+0000 7f21d7860740 1 -- 172.21.15.67:0/3556423655 shutdown_connections 2024-03-28T16:09:03.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.171+0000 7f21d7860740 1 -- 172.21.15.67:0/3556423655 wait complete. 2024-03-28T16:09:03.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.171+0000 7f21d7860740 20 client.16092 trim_cache size 0 max 16384 2024-03-28T16:09:03.176 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-28T16:09:03.176 INFO:teuthology.orchestra.run.smithi067.stdout: "status": "no active scrubs running", 2024-03-28T16:09:03.176 INFO:teuthology.orchestra.run.smithi067.stdout: "scrubs": {} 2024-03-28T16:09:03.176 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-28T16:09:03.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.172+0000 7f21cf200640 1 -- 172.21.15.67:0/965668679 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f21b40a2800 msgr2=0x7f21b40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:03.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.172+0000 7f21cf200640 1 --2- 172.21.15.67:0/965668679 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f21b40a2800 0x7f21b40a4cc0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x55c0fd43c2a0 tx=0x7f21c8009580 comp rx=0 tx=0).stop 2024-03-28T16:09:03.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.172+0000 7f21cf200640 1 -- 172.21.15.67:0/965668679 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21d0075680 msgr2=0x7f21d0199ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:03.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.172+0000 7f21cf200640 1 --2- 172.21.15.67:0/965668679 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21d0075680 0x7f21d0199ef0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f21b8006010 tx=0x7f21b802d980 comp rx=0 tx=0).stop 2024-03-28T16:09:03.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.172+0000 7f21cf200640 1 -- 172.21.15.67:0/965668679 shutdown_connections 2024-03-28T16:09:03.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.172+0000 7f21cf200640 1 --2- 172.21.15.67:0/965668679 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f21b40a2800 0x7f21b40a4cc0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.172+0000 7f21cf200640 1 --2- 172.21.15.67:0/965668679 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21d0109410 0x7f21d01a0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.172+0000 7f21cf200640 1 --2- 172.21.15.67:0/965668679 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21d0075680 0x7f21d0199ef0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.172+0000 7f21cf200640 1 --2- 172.21.15.67:0/965668679 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f21d00740a0 0x7f21d01999b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.172+0000 7f21cf200640 1 -- 172.21.15.67:0/965668679 >> 172.21.15.67:0/965668679 conn(0x7f21d00fcb10 msgr2=0x7f21d01053a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:03.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.172+0000 7f21cf200640 1 -- 172.21.15.67:0/965668679 shutdown_connections 2024-03-28T16:09:03.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.172+0000 7f21cf200640 1 -- 172.21.15.67:0/965668679 wait complete. 2024-03-28T16:09:03.189 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T16:09:03.189 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T16:09:03.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.409+0000 7f4261c00640 1 -- 172.21.15.67:0/754478831 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f425c1026f0 msgr2=0x7f425c102b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:03.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.409+0000 7f4261c00640 1 --2- 172.21.15.67:0/754478831 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f425c1026f0 0x7f425c102b70 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f424c009ca0 tx=0x7f424c02c0f0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.409+0000 7f4261c00640 1 -- 172.21.15.67:0/754478831 shutdown_connections 2024-03-28T16:09:03.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.409+0000 7f4261c00640 1 --2- 172.21.15.67:0/754478831 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f425c1030b0 0x7f425c109940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.409+0000 7f4261c00640 1 --2- 172.21.15.67:0/754478831 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f425c1026f0 0x7f425c102b70 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.409+0000 7f4261c00640 1 --2- 172.21.15.67:0/754478831 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f425c101500 0x7f425c101920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.409+0000 7f4261c00640 1 -- 172.21.15.67:0/754478831 >> 172.21.15.67:0/754478831 conn(0x7f425c0fca70 msgr2=0x7f425c0feeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:03.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.409+0000 7f4261c00640 1 -- 172.21.15.67:0/754478831 shutdown_connections 2024-03-28T16:09:03.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.409+0000 7f4261c00640 1 -- 172.21.15.67:0/754478831 wait complete. 2024-03-28T16:09:03.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.410+0000 7f4261c00640 1 Processor -- start 2024-03-28T16:09:03.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.410+0000 7f4261c00640 1 -- start start 2024-03-28T16:09:03.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.411+0000 7f4261c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f425c101500 0x7f425c199910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:03.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.411+0000 7f4261c00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f425c1026f0 0x7f425c199e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:03.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.411+0000 7f4261c00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f425c1030b0 0x7f425c1a0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:03.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.411+0000 7f4261c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f425c10d040 con 0x7f425c101500 2024-03-28T16:09:03.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.411+0000 7f4261c00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f425c10cec0 con 0x7f425c1030b0 2024-03-28T16:09:03.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.411+0000 7f4261c00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f425c10d1c0 con 0x7f425c1026f0 2024-03-28T16:09:03.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.411+0000 7f425be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f425c101500 0x7f425c199910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:03.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.411+0000 7f425be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f425c101500 0x7f425c199910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51808/0 (socket says 172.21.15.67:51808) 2024-03-28T16:09:03.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.411+0000 7f425be00640 1 -- 172.21.15.67:0/878450555 learned_addr learned my addr 172.21.15.67:0/878450555 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:09:03.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.411+0000 7f425b400640 1 --2- 172.21.15.67:0/878450555 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f425c1026f0 0x7f425c199e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:03.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.411+0000 7f4261200640 1 --2- 172.21.15.67:0/878450555 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f425c1030b0 0x7f425c1a0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:03.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.411+0000 7f425be00640 1 -- 172.21.15.67:0/878450555 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f425c1026f0 msgr2=0x7f425c199e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:03.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.411+0000 7f425be00640 1 --2- 172.21.15.67:0/878450555 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f425c1026f0 0x7f425c199e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.411+0000 7f425be00640 1 -- 172.21.15.67:0/878450555 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f425c1030b0 msgr2=0x7f425c1a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:03.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.411+0000 7f425be00640 1 --2- 172.21.15.67:0/878450555 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f425c1030b0 0x7f425c1a0ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.411+0000 7f425be00640 1 -- 172.21.15.67:0/878450555 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f425c1a1530 con 0x7f425c101500 2024-03-28T16:09:03.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.411+0000 7f425b400640 1 --2- 172.21.15.67:0/878450555 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f425c1026f0 0x7f425c199e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:09:03.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.412+0000 7f425be00640 1 --2- 172.21.15.67:0/878450555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f425c101500 0x7f425c199910 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f424400d770 tx=0x7f424400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:03.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.412+0000 7f4258c00640 1 -- 172.21.15.67:0/878450555 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4244017070 con 0x7f425c101500 2024-03-28T16:09:03.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.412+0000 7f4258c00640 1 -- 172.21.15.67:0/878450555 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4244009c30 con 0x7f425c101500 2024-03-28T16:09:03.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.412+0000 7f4261c00640 1 -- 172.21.15.67:0/878450555 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f425c1a1820 con 0x7f425c101500 2024-03-28T16:09:03.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.412+0000 7f4258c00640 1 -- 172.21.15.67:0/878450555 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4244003d00 con 0x7f425c101500 2024-03-28T16:09:03.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.412+0000 7f4261c00640 1 -- 172.21.15.67:0/878450555 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f425c1a1dc0 con 0x7f425c101500 2024-03-28T16:09:03.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.413+0000 7f4261c00640 1 -- 172.21.15.67:0/878450555 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f425c1a1fc0 con 0x7f425c101500 2024-03-28T16:09:03.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.416+0000 7f4258c00640 1 -- 172.21.15.67:0/878450555 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f4244003ea0 con 0x7f425c101500 2024-03-28T16:09:03.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.417+0000 7f4258c00640 1 --2- 172.21.15.67:0/878450555 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f42400a2880 0x7f42400a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:03.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.418+0000 7f4258c00640 1 -- 172.21.15.67:0/878450555 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f42440ce830 con 0x7f425c101500 2024-03-28T16:09:03.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.418+0000 7f425b400640 1 --2- 172.21.15.67:0/878450555 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f42400a2880 0x7f42400a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:03.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.418+0000 7f425b400640 1 --2- 172.21.15.67:0/878450555 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f42400a2880 0x7f42400a4d40 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f424c009430 tx=0x7f424c009a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:03.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.423+0000 7f4258c00640 1 -- 172.21.15.67:0/878450555 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f424409b880 con 0x7f425c101500 2024-03-28T16:09:03.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.659+0000 7f4261c00640 1 -- 172.21.15.67:0/878450555 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f425c1a2350 con 0x7f425c101500 2024-03-28T16:09:03.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.660+0000 7f4258c00640 1 -- 172.21.15.67:0/878450555 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11341 (secure 0 0 0) 0x7f4244009de0 con 0x7f425c101500 2024-03-28T16:09:03.660 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:09:03.661 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2008297659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2008297659},{"type":"v1","addr":"172.21.15.67:6835","nonce":2008297659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14532,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/320295745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":320295745},{"type":"v1","addr":"172.21.15.67:6837","nonce":320295745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6839/691821273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":691821273},{"type":"v1","addr":"172.21.15.67:6839","nonce":691821273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6841/3211254875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6840","nonce":3211254875},{"type":"v1","addr":"172.21.15.67:6841","nonce":3211254875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2129665423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2129665423},{"type":"v1","addr":"172.21.15.183:6833","nonce":2129665423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24496,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/1589784425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":1589784425},{"type":"v1","addr":"172.21.15.183:6835","nonce":1589784425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24508,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/2185863835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":2185863835},{"type":"v1","addr":"172.21.15.183:6837","nonce":2185863835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24520,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6839/1242814398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":1242814398},{"type":"v1","addr":"172.21.15.183:6839","nonce":1242814398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":34409,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1008335521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1008335521},{"type":"v1","addr":"172.21.15.150:6833","nonce":1008335521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":34445,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3269935695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3269935695},{"type":"v1","addr":"172.21.15.150:6835","nonce":3269935695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T16:01:07.696824+0000","modified":"2024-03-28T16:01:11.997773+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":34508},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34475":{"gid":34475,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/1112886277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":1112886277},{"type":"v1","addr":"172.21.15.150:6837","nonce":1112886277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_34508":{"gid":34508,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6839/1211011614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":1211011614},{"type":"v1","addr":"172.21.15.150:6839","nonce":1211011614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34508,"qdb_cluster":[34508]},"id":1}]} 2024-03-28T16:09:03.661 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 19 2024-03-28T16:09:03.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.662+0000 7f4261c00640 1 -- 172.21.15.67:0/878450555 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f42400a2880 msgr2=0x7f42400a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:03.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.662+0000 7f4261c00640 1 --2- 172.21.15.67:0/878450555 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f42400a2880 0x7f42400a4d40 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f424c009430 tx=0x7f424c009a20 comp rx=0 tx=0).stop 2024-03-28T16:09:03.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.662+0000 7f4261c00640 1 -- 172.21.15.67:0/878450555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f425c101500 msgr2=0x7f425c199910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:03.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.662+0000 7f4261c00640 1 --2- 172.21.15.67:0/878450555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f425c101500 0x7f425c199910 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f424400d770 tx=0x7f424400dc30 comp rx=0 tx=0).stop 2024-03-28T16:09:03.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.662+0000 7f4261c00640 1 -- 172.21.15.67:0/878450555 shutdown_connections 2024-03-28T16:09:03.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.662+0000 7f4261c00640 1 --2- 172.21.15.67:0/878450555 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f42400a2880 0x7f42400a4d40 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.662+0000 7f4261c00640 1 --2- 172.21.15.67:0/878450555 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f425c1030b0 0x7f425c1a0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.662+0000 7f4261c00640 1 --2- 172.21.15.67:0/878450555 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f425c1026f0 0x7f425c199e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.662+0000 7f4261c00640 1 --2- 172.21.15.67:0/878450555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f425c101500 0x7f425c199910 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.662+0000 7f4261c00640 1 -- 172.21.15.67:0/878450555 >> 172.21.15.67:0/878450555 conn(0x7f425c0fca70 msgr2=0x7f425c0738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:03.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.663+0000 7f4261c00640 1 -- 172.21.15.67:0/878450555 shutdown_connections 2024-03-28T16:09:03.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.663+0000 7f4261c00640 1 -- 172.21.15.67:0/878450555 wait complete. 2024-03-28T16:09:03.683 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T16:09:03.684 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T16:09:03.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.904+0000 7f76f7e00640 1 -- 172.21.15.67:0/4272207134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76f8075680 msgr2=0x7f76f8073ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:03.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.904+0000 7f76f7e00640 1 --2- 172.21.15.67:0/4272207134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76f8075680 0x7f76f8073ad0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f76e8009900 tx=0x7f76e802bcd0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.904+0000 7f76f7e00640 1 -- 172.21.15.67:0/4272207134 shutdown_connections 2024-03-28T16:09:03.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.904+0000 7f76f7e00640 1 --2- 172.21.15.67:0/4272207134 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f76f8109430 0x7f76f810b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.904+0000 7f76f7e00640 1 --2- 172.21.15.67:0/4272207134 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76f80740a0 0x7f76f8074500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.904+0000 7f76f7e00640 1 --2- 172.21.15.67:0/4272207134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76f8075680 0x7f76f8073ad0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.904+0000 7f76f7e00640 1 -- 172.21.15.67:0/4272207134 >> 172.21.15.67:0/4272207134 conn(0x7f76f80fcb10 msgr2=0x7f76f80fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:03.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.904+0000 7f76f7e00640 1 -- 172.21.15.67:0/4272207134 shutdown_connections 2024-03-28T16:09:03.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.904+0000 7f76f7e00640 1 -- 172.21.15.67:0/4272207134 wait complete. 2024-03-28T16:09:03.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.905+0000 7f76f7e00640 1 Processor -- start 2024-03-28T16:09:03.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.905+0000 7f76f7e00640 1 -- start start 2024-03-28T16:09:03.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.906+0000 7f76f7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76f80740a0 0x7f76f8199980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:03.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.906+0000 7f76f7e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f76f8075680 0x7f76f8199ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:03.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.906+0000 7f76f7e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76f8109430 0x7f76f81a0f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:03.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.906+0000 7f76f7e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f810f2f0 con 0x7f76f80740a0 2024-03-28T16:09:03.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.906+0000 7f76f7e00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f810f170 con 0x7f76f8109430 2024-03-28T16:09:03.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.906+0000 7f76f7e00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f810f470 con 0x7f76f8075680 2024-03-28T16:09:03.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.906+0000 7f76f6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76f80740a0 0x7f76f8199980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:03.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.906+0000 7f76f6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76f80740a0 0x7f76f8199980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51820/0 (socket says 172.21.15.67:51820) 2024-03-28T16:09:03.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.906+0000 7f76f6a00640 1 -- 172.21.15.67:0/1890439180 learned_addr learned my addr 172.21.15.67:0/1890439180 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:09:03.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.906+0000 7f76ee000640 1 --2- 172.21.15.67:0/1890439180 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f76f8075680 0x7f76f8199ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:03.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.906+0000 7f76f7400640 1 --2- 172.21.15.67:0/1890439180 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76f8109430 0x7f76f81a0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:03.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.906+0000 7f76f6a00640 1 -- 172.21.15.67:0/1890439180 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f76f8075680 msgr2=0x7f76f8199ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:03.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.906+0000 7f76f6a00640 1 --2- 172.21.15.67:0/1890439180 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f76f8075680 0x7f76f8199ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.906+0000 7f76f6a00640 1 -- 172.21.15.67:0/1890439180 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76f8109430 msgr2=0x7f76f81a0f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:03.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.906+0000 7f76f6a00640 1 --2- 172.21.15.67:0/1890439180 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76f8109430 0x7f76f81a0f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.907+0000 7f76f6a00640 1 -- 172.21.15.67:0/1890439180 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76f81a15a0 con 0x7f76f80740a0 2024-03-28T16:09:03.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.907+0000 7f76f6a00640 1 --2- 172.21.15.67:0/1890439180 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76f80740a0 0x7f76f8199980 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f76e8000c00 tx=0x7f76e802dc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:03.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.907+0000 7f76fca00640 1 -- 172.21.15.67:0/1890439180 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76e8043020 con 0x7f76f80740a0 2024-03-28T16:09:03.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.907+0000 7f76fca00640 1 -- 172.21.15.67:0/1890439180 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f76e802de30 con 0x7f76f80740a0 2024-03-28T16:09:03.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.907+0000 7f76f7e00640 1 -- 172.21.15.67:0/1890439180 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76f81a1830 con 0x7f76f80740a0 2024-03-28T16:09:03.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.907+0000 7f76fca00640 1 -- 172.21.15.67:0/1890439180 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76e8003d70 con 0x7f76f80740a0 2024-03-28T16:09:03.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.907+0000 7f76f7e00640 1 -- 172.21.15.67:0/1890439180 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76f81a1d80 con 0x7f76f80740a0 2024-03-28T16:09:03.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.911+0000 7f7700a76740 1 Processor -- start 2024-03-28T16:09:03.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.911+0000 7f7700a76740 1 -- start start 2024-03-28T16:09:03.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.912+0000 7f76fca00640 1 -- 172.21.15.67:0/1890439180 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f76e8046020 con 0x7f76f80740a0 2024-03-28T16:09:03.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.912+0000 7f7700a76740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56206eff19f0 0x56206eff1e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:03.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.912+0000 7f7700a76740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56206eff2bc0 0x56206eff3040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:03.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.912+0000 7f7700a76740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56206eff3580 0x56206eff9e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:03.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.912+0000 7f7700a76740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x56206ef562c0 con 0x56206eff2bc0 2024-03-28T16:09:03.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.912+0000 7f7700a76740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x56206eefec90 con 0x56206eff3580 2024-03-28T16:09:03.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.912+0000 7f7700a76740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x56206ef52360 con 0x56206eff19f0 2024-03-28T16:09:03.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.912+0000 7f76f6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56206eff2bc0 0x56206eff3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:03.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.912+0000 7f76f6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56206eff2bc0 0x56206eff3040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51826/0 (socket says 172.21.15.67:51826) 2024-03-28T16:09:03.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.912+0000 7f76f6a00640 1 -- 172.21.15.67:0/2547745026 learned_addr learned my addr 172.21.15.67:0/2547745026 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:09:03.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.913+0000 7f76f7400640 1 --2- 172.21.15.67:0/2547745026 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56206eff19f0 0x56206eff1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:03.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.913+0000 7f76ee000640 1 --2- 172.21.15.67:0/2547745026 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56206eff3580 0x56206eff9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:03.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.913+0000 7f76f6a00640 1 -- 172.21.15.67:0/2547745026 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56206eff19f0 msgr2=0x56206eff1e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:09:03.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.913+0000 7f76f6a00640 1 --2- 172.21.15.67:0/2547745026 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56206eff19f0 0x56206eff1e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.913+0000 7f76f6a00640 1 -- 172.21.15.67:0/2547745026 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56206eff3580 msgr2=0x56206eff9e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:09:03.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.913+0000 7f76f6a00640 1 --2- 172.21.15.67:0/2547745026 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56206eff3580 0x56206eff9e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.913+0000 7f76f6a00640 1 -- 172.21.15.67:0/2547745026 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56206ef23080 con 0x56206eff2bc0 2024-03-28T16:09:03.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.913+0000 7f76ee000640 1 --2- 172.21.15.67:0/2547745026 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56206eff3580 0x56206eff9e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:09:03.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.913+0000 7f76f6a00640 1 --2- 172.21.15.67:0/2547745026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56206eff2bc0 0x56206eff3040 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f76e80f4df0 tx=0x7f76e80f4ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a03da2dddc2e739 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:03.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.913+0000 7f76fca00640 1 --2- 172.21.15.67:0/1890439180 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f76dc0a2800 0x7f76dc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:03.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.913+0000 7f76e7e00640 1 -- 172.21.15.67:0/2547745026 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76e80b9ac0 con 0x56206eff2bc0 2024-03-28T16:09:03.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.914+0000 7f76fca00640 1 -- 172.21.15.67:0/1890439180 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f76e80f0e70 con 0x7f76f80740a0 2024-03-28T16:09:03.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.914+0000 7f76e7e00640 1 -- 172.21.15.67:0/2547745026 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f76e80fe070 con 0x56206eff2bc0 2024-03-28T16:09:03.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.914+0000 7f76f7400640 1 --2- 172.21.15.67:0/1890439180 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f76dc0a2800 0x7f76dc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:03.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.914+0000 7f76e7e00640 1 -- 172.21.15.67:0/2547745026 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76e80bcc70 con 0x56206eff2bc0 2024-03-28T16:09:03.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.914+0000 7f7700a76740 1 -- 172.21.15.67:0/2547745026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56206eff2bc0 msgr2=0x56206eff3040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:03.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.914+0000 7f76f7400640 1 --2- 172.21.15.67:0/1890439180 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f76dc0a2800 0x7f76dc0a4cc0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x56206eff2ac0 tx=0x7f76f0008370 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:03.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.914+0000 7f7700a76740 1 --2- 172.21.15.67:0/2547745026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56206eff2bc0 0x56206eff3040 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f76e80f4df0 tx=0x7f76e80f4ee0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.915+0000 7f7700a76740 1 -- 172.21.15.67:0/2547745026 shutdown_connections 2024-03-28T16:09:03.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.915+0000 7f7700a76740 1 --2- 172.21.15.67:0/2547745026 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56206eff3580 0x56206eff9e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.915+0000 7f7700a76740 1 --2- 172.21.15.67:0/2547745026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56206eff2bc0 0x56206eff3040 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.915+0000 7f7700a76740 1 --2- 172.21.15.67:0/2547745026 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56206eff19f0 0x56206eff1e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.915+0000 7f7700a76740 1 -- 172.21.15.67:0/2547745026 >> 172.21.15.67:0/2547745026 conn(0x56206efecde0 msgr2=0x56206efed1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:03.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.915+0000 7f7700a76740 1 -- 172.21.15.67:0/2547745026 shutdown_connections 2024-03-28T16:09:03.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.915+0000 7f7700a76740 1 -- 172.21.15.67:0/2547745026 wait complete. 2024-03-28T16:09:03.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.915+0000 7f7700a76740 1 -- start start 2024-03-28T16:09:03.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.915+0000 7f7700a76740 1 Processor -- start 2024-03-28T16:09:03.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.916+0000 7f7700a76740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56206eff19f0 0x56206effb980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:03.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.916+0000 7f7700a76740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56206eff2bc0 0x56206effbec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:03.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.916+0000 7f7700a76740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56206eff3580 0x56206f002f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:03.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.916+0000 7f7700a76740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x56206effa350 con 0x56206eff3580 2024-03-28T16:09:03.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.916+0000 7f7700a76740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x56206ef120d0 con 0x56206eff19f0 2024-03-28T16:09:03.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.916+0000 7f7700a76740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x56206effa4d0 con 0x56206eff2bc0 2024-03-28T16:09:03.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.916+0000 7f7700a76740 10 client.0 _ll_register_callbacks cb 0x56206ef16e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:09:03.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.916+0000 7f76f7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56206eff3580 0x56206f002f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:03.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.916+0000 7f76f7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56206eff3580 0x56206f002f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51828/0 (socket says 172.21.15.67:51828) 2024-03-28T16:09:03.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.916+0000 7f76f7400640 1 -- 172.21.15.67:0/1472035806 learned_addr learned my addr 172.21.15.67:0/1472035806 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:09:03.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.916+0000 7f76f6a00640 1 --2- 172.21.15.67:0/1472035806 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56206eff19f0 0x56206effb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:03.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.916+0000 7f76ee000640 1 --2- 172.21.15.67:0/1472035806 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56206eff2bc0 0x56206effbec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:03.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.917+0000 7f76f7400640 1 -- 172.21.15.67:0/1472035806 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56206eff2bc0 msgr2=0x56206effbec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:09:03.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.917+0000 7f76f7400640 1 --2- 172.21.15.67:0/1472035806 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56206eff2bc0 0x56206effbec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.917+0000 7f76f7400640 1 -- 172.21.15.67:0/1472035806 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56206eff19f0 msgr2=0x56206effb980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:09:03.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.917+0000 7f76f7400640 1 --2- 172.21.15.67:0/1472035806 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56206eff19f0 0x56206effb980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.917+0000 7f76f7400640 1 -- 172.21.15.67:0/1472035806 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56206ec6a3f0 con 0x56206eff3580 2024-03-28T16:09:03.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.917+0000 7f76f6a00640 1 --2- 172.21.15.67:0/1472035806 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56206eff19f0 0x56206effb980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:09:03.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.917+0000 7f76f7400640 1 --2- 172.21.15.67:0/1472035806 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56206eff3580 0x56206f002f00 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f76f0011e40 tx=0x7f76f0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5032ff00c73c9227 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:03.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.917+0000 7f76e4c00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:09:03.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.917+0000 7f76e4c00640 1 -- 172.21.15.67:0/1472035806 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76f001e420 con 0x56206eff3580 2024-03-28T16:09:03.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.917+0000 7f76e4c00640 1 -- 172.21.15.67:0/1472035806 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f76f001e5c0 con 0x56206eff3580 2024-03-28T16:09:03.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.917+0000 7f7700a76740 1 -- 172.21.15.67:0/1472035806 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f76e80fe070 con 0x56206eff3580 2024-03-28T16:09:03.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.918+0000 7f76e4c00640 1 -- 172.21.15.67:0/1472035806 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f76f001e800 con 0x56206eff3580 2024-03-28T16:09:03.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.918+0000 7f7700a76740 10 client.16110 fetch_fsmap learned FSMap version 19 2024-03-28T16:09:03.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.918+0000 7f7700a76740 1 -- 172.21.15.67:0/1472035806 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x56206ed347d0 con 0x56206eff3580 2024-03-28T16:09:03.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.918+0000 7f76e4c00640 1 -- 172.21.15.67:0/1472035806 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f76e80fe070 con 0x56206eff3580 2024-03-28T16:09:03.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.918+0000 7f76e4c00640 1 -- 172.21.15.67:0/1472035806 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f76f001c8e0 con 0x56206eff3580 2024-03-28T16:09:03.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.918+0000 7f7700a76740 10 client.16110 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:09:03.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.918+0000 7f7700a76740 10 client.16110 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:09:03.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.918+0000 7f7700a76740 20 client.16110 populate_metadata read hostname 'smithi067' 2024-03-28T16:09:03.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.918+0000 7f7700a76740 1 --2- 172.21.15.67:0/1472035806 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x56206f007450 0x56206f009880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:03.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.918+0000 7f7700a76740 4 client.16110 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:09:03.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.918+0000 7f7700a76740 1 -- 172.21.15.67:0/1472035806 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56206ef63800 con 0x56206f007450 2024-03-28T16:09:03.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.919+0000 7f76f6a00640 1 --2- 172.21.15.67:0/1472035806 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x56206f007450 0x56206f009880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:03.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.920+0000 7f76f6a00640 1 --2- 172.21.15.67:0/1472035806 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x56206f007450 0x56206f009880 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=42a9f2133c00fd78 server_cookie=cc6caca7b3c70118 in_seq=0 out_seq=0 2024-03-28T16:09:03.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.920+0000 7f76e4c00640 10 client.16110 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:09:03.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.922+0000 7f76e4c00640 1 -- 172.21.15.67:0/1472035806 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56206ef63800 con 0x56206f007450 2024-03-28T16:09:03.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.922+0000 7f76e4c00640 10 client.16110 handle_command_reply: tid=0 2024-03-28T16:09:03.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.922+0000 7f7700a76740 1 client.16110 shutdown 2024-03-28T16:09:03.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.922+0000 7f7700a76740 1 -- 172.21.15.67:0/1472035806 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56206eff3580 msgr2=0x56206f002f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:03.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.922+0000 7f7700a76740 1 --2- 172.21.15.67:0/1472035806 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56206eff3580 0x56206f002f00 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f76f0011e40 tx=0x7f76f0004a40 comp rx=0 tx=0).stop 2024-03-28T16:09:03.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.922+0000 7f7700a76740 1 -- 172.21.15.67:0/1472035806 shutdown_connections 2024-03-28T16:09:03.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.922+0000 7f7700a76740 1 --2- 172.21.15.67:0/1472035806 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x56206f007450 0x56206f009880 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.922+0000 7f7700a76740 1 --2- 172.21.15.67:0/1472035806 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56206eff3580 0x56206f002f00 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.922+0000 7f7700a76740 1 --2- 172.21.15.67:0/1472035806 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56206eff2bc0 0x56206effbec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.922+0000 7f7700a76740 1 --2- 172.21.15.67:0/1472035806 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56206eff19f0 0x56206effb980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.922+0000 7f76e4c00640 0 client.16110 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:09:03.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.922+0000 7f7700a76740 1 -- 172.21.15.67:0/1472035806 >> 172.21.15.67:0/1472035806 conn(0x56206efecde0 msgr2=0x56206efec160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:03.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.922+0000 7f7700a76740 1 -- 172.21.15.67:0/1472035806 shutdown_connections 2024-03-28T16:09:03.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.923+0000 7f7700a76740 1 -- 172.21.15.67:0/1472035806 wait complete. 2024-03-28T16:09:03.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.923+0000 7f7700a76740 20 client.16110 trim_cache size 0 max 16384 2024-03-28T16:09:03.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.946+0000 7f7700a76740 1 Processor -- start 2024-03-28T16:09:03.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.947+0000 7f7700a76740 1 -- start start 2024-03-28T16:09:03.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.947+0000 7f7700a76740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56206eff19f0 0x56206efebfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:03.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.947+0000 7f7700a76740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56206eff2bc0 0x56206f002ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:03.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.947+0000 7f7700a76740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56206eff3580 0x56206eff5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:03.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.947+0000 7f7700a76740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f81a8a00 con 0x56206eff19f0 2024-03-28T16:09:03.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.947+0000 7f7700a76740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f81aafc0 con 0x56206eff3580 2024-03-28T16:09:03.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.947+0000 7f7700a76740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f81a9000 con 0x56206eff2bc0 2024-03-28T16:09:03.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.947+0000 7f76f6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56206eff19f0 0x56206efebfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:03.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.947+0000 7f76f6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56206eff19f0 0x56206efebfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51844/0 (socket says 172.21.15.67:51844) 2024-03-28T16:09:03.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.947+0000 7f76f6a00640 1 -- 172.21.15.67:0/1449138133 learned_addr learned my addr 172.21.15.67:0/1449138133 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:09:03.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.948+0000 7f76f7400640 1 --2- 172.21.15.67:0/1449138133 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56206eff3580 0x56206eff5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:03.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.948+0000 7f76f6a00640 1 -- 172.21.15.67:0/1449138133 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56206eff2bc0 msgr2=0x56206f002ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:09:03.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.948+0000 7f76f6a00640 1 --2- 172.21.15.67:0/1449138133 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56206eff2bc0 0x56206f002ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.948+0000 7f76f6a00640 1 -- 172.21.15.67:0/1449138133 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56206eff3580 msgr2=0x56206eff5bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:09:03.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.948+0000 7f76f6a00640 1 --2- 172.21.15.67:0/1449138133 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56206eff3580 0x56206eff5bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.948+0000 7f76f6a00640 1 -- 172.21.15.67:0/1449138133 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56206eff2640 con 0x56206eff19f0 2024-03-28T16:09:03.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.948+0000 7f76f7400640 1 --2- 172.21.15.67:0/1449138133 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56206eff3580 0x56206eff5bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T16:09:03.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.948+0000 7f76f6a00640 1 --2- 172.21.15.67:0/1449138133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56206eff19f0 0x56206efebfe0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f76e80eecf0 tx=0x7f76e80044f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cba98f56235fd241 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:03.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.949+0000 7f76e7400640 1 -- 172.21.15.67:0/1449138133 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76e80bcfa0 con 0x56206eff19f0 2024-03-28T16:09:03.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.949+0000 7f76e7400640 1 -- 172.21.15.67:0/1449138133 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f76e80fe040 con 0x56206eff19f0 2024-03-28T16:09:03.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.949+0000 7f76e7400640 1 -- 172.21.15.67:0/1449138133 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76e80bd180 con 0x56206eff19f0 2024-03-28T16:09:03.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.949+0000 7f7700a76740 1 -- 172.21.15.67:0/1449138133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56206eff19f0 msgr2=0x56206efebfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:03.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.949+0000 7f7700a76740 1 --2- 172.21.15.67:0/1449138133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56206eff19f0 0x56206efebfe0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f76e80eecf0 tx=0x7f76e80044f0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.950+0000 7f7700a76740 1 -- 172.21.15.67:0/1449138133 shutdown_connections 2024-03-28T16:09:03.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.950+0000 7f7700a76740 1 --2- 172.21.15.67:0/1449138133 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56206eff3580 0x56206eff5bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.950+0000 7f7700a76740 1 --2- 172.21.15.67:0/1449138133 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56206eff2bc0 0x56206f002ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.950+0000 7f7700a76740 1 --2- 172.21.15.67:0/1449138133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56206eff19f0 0x56206efebfe0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.950+0000 7f7700a76740 1 -- 172.21.15.67:0/1449138133 >> 172.21.15.67:0/1449138133 conn(0x56206efecde0 msgr2=0x56206efeb8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:03.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.950+0000 7f7700a76740 1 -- 172.21.15.67:0/1449138133 shutdown_connections 2024-03-28T16:09:03.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.950+0000 7f7700a76740 1 -- 172.21.15.67:0/1449138133 wait complete. 2024-03-28T16:09:03.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.950+0000 7f7700a76740 1 -- start start 2024-03-28T16:09:03.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.950+0000 7f7700a76740 1 Processor -- start 2024-03-28T16:09:03.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.951+0000 7f7700a76740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56206eff19f0 0x56206effed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:03.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.951+0000 7f7700a76740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56206eff2bc0 0x56206efff270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:03.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.951+0000 7f7700a76740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56206eff3580 0x56206f002290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:03.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.951+0000 7f7700a76740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f81a9640 con 0x56206eff19f0 2024-03-28T16:09:03.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.951+0000 7f7700a76740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f81aa960 con 0x56206eff2bc0 2024-03-28T16:09:03.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.951+0000 7f7700a76740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f81a9ca0 con 0x56206eff3580 2024-03-28T16:09:03.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.951+0000 7f7700a76740 10 client.0 _ll_register_callbacks cb 0x7f76f0005e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:09:03.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.951+0000 7f76f6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56206eff19f0 0x56206effed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:03.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.951+0000 7f76f6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56206eff19f0 0x56206effed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51856/0 (socket says 172.21.15.67:51856) 2024-03-28T16:09:03.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.951+0000 7f76f6a00640 1 -- 172.21.15.67:0/766527065 learned_addr learned my addr 172.21.15.67:0/766527065 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:09:03.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.951+0000 7f76ee000640 1 --2- 172.21.15.67:0/766527065 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56206eff2bc0 0x56206efff270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:03.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.951+0000 7f76f6a00640 1 -- 172.21.15.67:0/766527065 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56206eff3580 msgr2=0x56206f002290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:09:03.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.951+0000 7f76f6a00640 1 --2- 172.21.15.67:0/766527065 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56206eff3580 0x56206f002290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.951+0000 7f76f6a00640 1 -- 172.21.15.67:0/766527065 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56206eff2bc0 msgr2=0x56206efff270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:09:03.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.951+0000 7f76f6a00640 1 --2- 172.21.15.67:0/766527065 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56206eff2bc0 0x56206efff270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.951+0000 7f76f6a00640 1 -- 172.21.15.67:0/766527065 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56206f002880 con 0x56206eff19f0 2024-03-28T16:09:03.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.952+0000 7f76f6a00640 1 --2- 172.21.15.67:0/766527065 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56206eff19f0 0x56206effed30 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f76e80f4d40 tx=0x7f76e8009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd457c7a44cc80d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:03.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.952+0000 7f76e4c00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:09:03.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.952+0000 7f76e4c00640 1 -- 172.21.15.67:0/766527065 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76e80d5020 con 0x56206eff19f0 2024-03-28T16:09:03.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.952+0000 7f76e4c00640 1 -- 172.21.15.67:0/766527065 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f76e80bbb20 con 0x56206eff19f0 2024-03-28T16:09:03.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.952+0000 7f7700a76740 1 -- 172.21.15.67:0/766527065 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f76e80fe040 con 0x56206eff19f0 2024-03-28T16:09:03.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.952+0000 7f76e4c00640 1 -- 172.21.15.67:0/766527065 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f76e80bd360 con 0x56206eff19f0 2024-03-28T16:09:03.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.952+0000 7f7700a76740 10 client.16116 fetch_fsmap learned FSMap version 19 2024-03-28T16:09:03.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.952+0000 7f7700a76740 1 -- 172.21.15.67:0/766527065 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x56206f0634d0 con 0x56206eff19f0 2024-03-28T16:09:03.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.952+0000 7f76e4c00640 1 -- 172.21.15.67:0/766527065 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f76e80fe040 con 0x56206eff19f0 2024-03-28T16:09:03.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.952+0000 7f76e4c00640 1 -- 172.21.15.67:0/766527065 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f76e80f45e0 con 0x56206eff19f0 2024-03-28T16:09:03.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.953+0000 7f7700a76740 10 client.16116 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:09:03.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.953+0000 7f7700a76740 10 client.16116 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:09:03.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.953+0000 7f7700a76740 20 client.16116 populate_metadata read hostname 'smithi067' 2024-03-28T16:09:03.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.953+0000 7f7700a76740 1 --2- 172.21.15.67:0/766527065 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x56206f063be0 0x56206f066000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:03.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.953+0000 7f7700a76740 4 client.16116 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T16:09:03.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.953+0000 7f7700a76740 1 -- 172.21.15.67:0/766527065 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56206ef5db50 con 0x56206f063be0 2024-03-28T16:09:03.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.953+0000 7f76ee000640 1 --2- 172.21.15.67:0/766527065 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x56206f063be0 0x56206f066000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:03.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.954+0000 7f76ee000640 1 --2- 172.21.15.67:0/766527065 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x56206f063be0 0x56206f066000 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f52548c3f1072240 server_cookie=5e89d263da826b2 in_seq=0 out_seq=0 2024-03-28T16:09:03.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.954+0000 7f76e4c00640 10 client.16116 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:09:03.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.954+0000 7f76e4c00640 1 -- 172.21.15.67:0/766527065 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56206ef5db50 con 0x56206f063be0 2024-03-28T16:09:03.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.954+0000 7f76e4c00640 10 client.16116 handle_command_reply: tid=0 2024-03-28T16:09:03.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.954+0000 7f7700a76740 1 client.16116 shutdown 2024-03-28T16:09:03.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.954+0000 7f7700a76740 1 -- 172.21.15.67:0/766527065 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56206eff19f0 msgr2=0x56206effed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:03.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.954+0000 7f7700a76740 1 --2- 172.21.15.67:0/766527065 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56206eff19f0 0x56206effed30 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f76e80f4d40 tx=0x7f76e8009260 comp rx=0 tx=0).stop 2024-03-28T16:09:03.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.954+0000 7f7700a76740 1 -- 172.21.15.67:0/766527065 shutdown_connections 2024-03-28T16:09:03.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.954+0000 7f7700a76740 1 --2- 172.21.15.67:0/766527065 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x56206f063be0 0x56206f066000 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.954+0000 7f7700a76740 1 --2- 172.21.15.67:0/766527065 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56206eff3580 0x56206f002290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.954+0000 7f7700a76740 1 --2- 172.21.15.67:0/766527065 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56206eff2bc0 0x56206efff270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.954+0000 7f7700a76740 1 --2- 172.21.15.67:0/766527065 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56206eff19f0 0x56206effed30 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.954+0000 7f7700a76740 1 -- 172.21.15.67:0/766527065 >> 172.21.15.67:0/766527065 conn(0x56206efecde0 msgr2=0x56206efec160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:03.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.954+0000 7f76e4c00640 0 client.16116 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:09:03.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.954+0000 7f7700a76740 1 -- 172.21.15.67:0/766527065 shutdown_connections 2024-03-28T16:09:03.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.954+0000 7f7700a76740 1 -- 172.21.15.67:0/766527065 wait complete. 2024-03-28T16:09:03.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.955+0000 7f7700a76740 20 client.16116 trim_cache size 0 max 16384 2024-03-28T16:09:03.959 INFO:teuthology.orchestra.run.smithi067.stdout:[] 2024-03-28T16:09:03.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.955+0000 7f76f7e00640 1 -- 172.21.15.67:0/1890439180 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f76dc0a2800 msgr2=0x7f76dc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:03.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.955+0000 7f76f7e00640 1 --2- 172.21.15.67:0/1890439180 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f76dc0a2800 0x7f76dc0a4cc0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x56206eff2ac0 tx=0x7f76f0008370 comp rx=0 tx=0).stop 2024-03-28T16:09:03.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.955+0000 7f76f7e00640 1 -- 172.21.15.67:0/1890439180 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76f80740a0 msgr2=0x7f76f8199980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:03.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.955+0000 7f76f7e00640 1 --2- 172.21.15.67:0/1890439180 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76f80740a0 0x7f76f8199980 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f76e8000c00 tx=0x7f76e802dc40 comp rx=0 tx=0).stop 2024-03-28T16:09:03.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.956+0000 7f76f7e00640 1 -- 172.21.15.67:0/1890439180 shutdown_connections 2024-03-28T16:09:03.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.956+0000 7f76f7e00640 1 --2- 172.21.15.67:0/1890439180 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f76dc0a2800 0x7f76dc0a4cc0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.956+0000 7f76f7e00640 1 --2- 172.21.15.67:0/1890439180 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76f8109430 0x7f76f81a0f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.956+0000 7f76f7e00640 1 --2- 172.21.15.67:0/1890439180 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f76f8075680 0x7f76f8199ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.956+0000 7f76f7e00640 1 --2- 172.21.15.67:0/1890439180 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76f80740a0 0x7f76f8199980 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:03.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.956+0000 7f76f7e00640 1 -- 172.21.15.67:0/1890439180 >> 172.21.15.67:0/1890439180 conn(0x7f76f80fcb10 msgr2=0x7f76f81053c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:03.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.956+0000 7f76f7e00640 1 -- 172.21.15.67:0/1890439180 shutdown_connections 2024-03-28T16:09:03.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:03.956+0000 7f76f7e00640 1 -- 172.21.15.67:0/1890439180 wait complete. 2024-03-28T16:09:03.974 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:03 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/878450555' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:09:04.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:03 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/878450555' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:09:04.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:03 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/878450555' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:09:04.614 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-03-28T16:09:04.614 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-03-28T16:09:04.616 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-03-28T16:09:04.616 INFO:tasks.workunit.client.0.smithi067.stdout:iozone test complete. 2024-03-28T16:09:04.621 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-28T16:09:04.622 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-28T16:09:04.740 INFO:tasks.workunit:Stopping ['suites/iozone.sh'] on client.0... 2024-03-28T16:09:04.740 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-28T16:09:04.980 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T16:09:04.980 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T16:09:04.981 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T16:09:05.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:04 smithi150 ceph-mon[34825]: pgmap v604: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 85 B/s wr, 2 op/s 2024-03-28T16:09:05.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:04 smithi067 ceph-mon[31707]: pgmap v604: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 85 B/s wr, 2 op/s 2024-03-28T16:09:05.099 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:04 smithi183 ceph-mon[35580]: pgmap v604: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 85 B/s wr, 2 op/s 2024-03-28T16:09:05.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.273+0000 7f1f1fa00640 1 -- 172.21.15.67:0/2476479233 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1f18077c40 msgr2=0x7f1f180784e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:05.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.274+0000 7f1f1fa00640 1 --2- 172.21.15.67:0/2476479233 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1f18077c40 0x7f1f180784e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1f10009ca0 tx=0x7f1f1002c100 comp rx=0 tx=0).stop 2024-03-28T16:09:05.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.274+0000 7f1f1fa00640 1 -- 172.21.15.67:0/2476479233 shutdown_connections 2024-03-28T16:09:05.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.274+0000 7f1f1fa00640 1 --2- 172.21.15.67:0/2476479233 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1f18077c40 0x7f1f180784e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:05.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.274+0000 7f1f1fa00640 1 --2- 172.21.15.67:0/2476479233 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f180772a0 0x7f1f18077700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:05.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.274+0000 7f1f1fa00640 1 --2- 172.21.15.67:0/2476479233 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f18076060 0x7f1f18076480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:05.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.274+0000 7f1f1fa00640 1 -- 172.21.15.67:0/2476479233 >> 172.21.15.67:0/2476479233 conn(0x7f1f180fee50 msgr2=0x7f1f18101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:05.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.274+0000 7f1f1fa00640 1 -- 172.21.15.67:0/2476479233 shutdown_connections 2024-03-28T16:09:05.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.275+0000 7f1f1fa00640 1 -- 172.21.15.67:0/2476479233 wait complete. 2024-03-28T16:09:05.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.276+0000 7f1f1fa00640 1 Processor -- start 2024-03-28T16:09:05.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.276+0000 7f1f1fa00640 1 -- start start 2024-03-28T16:09:05.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.277+0000 7f1f1fa00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1f18076060 0x7f1f1819df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:05.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.277+0000 7f1f1fa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f180772a0 0x7f1f1819e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:05.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.277+0000 7f1f1fa00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f18077c40 0x7f1f181a54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:05.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.277+0000 7f1f1fa00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f18113860 con 0x7f1f180772a0 2024-03-28T16:09:05.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.277+0000 7f1f1fa00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f181136e0 con 0x7f1f18077c40 2024-03-28T16:09:05.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.277+0000 7f1f1fa00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f181139e0 con 0x7f1f18076060 2024-03-28T16:09:05.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.277+0000 7f1f17e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f180772a0 0x7f1f1819e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:05.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.277+0000 7f1f1d000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1f18076060 0x7f1f1819df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:05.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.277+0000 7f1f1d000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1f18076060 0x7f1f1819df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.67:42812/0 (socket says 172.21.15.67:42812) 2024-03-28T16:09:05.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.277+0000 7f1f1d000640 1 -- 172.21.15.67:0/1011543207 learned_addr learned my addr 172.21.15.67:0/1011543207 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:09:05.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.278+0000 7f1f1da00640 1 --2- 172.21.15.67:0/1011543207 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f18077c40 0x7f1f181a54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:05.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.278+0000 7f1f1d000640 1 -- 172.21.15.67:0/1011543207 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f18077c40 msgr2=0x7f1f181a54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:05.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.278+0000 7f1f1d000640 1 --2- 172.21.15.67:0/1011543207 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f18077c40 0x7f1f181a54f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:05.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.278+0000 7f1f1d000640 1 -- 172.21.15.67:0/1011543207 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f180772a0 msgr2=0x7f1f1819e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:05.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.278+0000 7f1f1d000640 1 --2- 172.21.15.67:0/1011543207 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f180772a0 0x7f1f1819e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:05.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.278+0000 7f1f1d000640 1 -- 172.21.15.67:0/1011543207 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f181a5b60 con 0x7f1f18076060 2024-03-28T16:09:05.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.278+0000 7f1f17e00640 1 --2- 172.21.15.67:0/1011543207 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f180772a0 0x7f1f1819e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T16:09:05.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.278+0000 7f1f1d000640 1 --2- 172.21.15.67:0/1011543207 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1f18076060 0x7f1f1819df40 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c00d950 tx=0x7f1f0c00dd50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:05.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.279+0000 7f1f15600640 1 -- 172.21.15.67:0/1011543207 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f0c017070 con 0x7f1f18076060 2024-03-28T16:09:05.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.279+0000 7f1f15600640 1 -- 172.21.15.67:0/1011543207 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1f0c00de80 con 0x7f1f18076060 2024-03-28T16:09:05.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.279+0000 7f1f15600640 1 -- 172.21.15.67:0/1011543207 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f0c00f3c0 con 0x7f1f18076060 2024-03-28T16:09:05.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.280+0000 7f1f1fa00640 1 -- 172.21.15.67:0/1011543207 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f181a5e50 con 0x7f1f18076060 2024-03-28T16:09:05.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.280+0000 7f1f1fa00640 1 -- 172.21.15.67:0/1011543207 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f181a62e0 con 0x7f1f18076060 2024-03-28T16:09:05.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.286+0000 7f1f2276f740 1 Processor -- start 2024-03-28T16:09:05.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.288+0000 7f1f15600640 1 -- 172.21.15.67:0/1011543207 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f1f0c010460 con 0x7f1f18076060 2024-03-28T16:09:05.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.288+0000 7f1f2276f740 1 -- start start 2024-03-28T16:09:05.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.288+0000 7f1f2276f740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5622bd6df0a0 0x5622bd6df4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:05.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.288+0000 7f1f2276f740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5622bd6dfa90 0x5622bd6e4590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:05.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.288+0000 7f1f2276f740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5622bd6e4c00 0x5622bd6e7090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:05.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.288+0000 7f1f2276f740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622bd644eb0 con 0x5622bd6dfa90 2024-03-28T16:09:05.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.288+0000 7f1f2276f740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622bd5eda60 con 0x5622bd6e4c00 2024-03-28T16:09:05.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.289+0000 7f1f2276f740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622bd640f50 con 0x5622bd6df0a0 2024-03-28T16:09:05.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.289+0000 7f1f1d000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5622bd6dfa90 0x5622bd6e4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:05.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.289+0000 7f1f1d000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5622bd6dfa90 0x5622bd6e4590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51886/0 (socket says 172.21.15.67:51886) 2024-03-28T16:09:05.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.289+0000 7f1f1d000640 1 -- 172.21.15.67:0/3401169486 learned_addr learned my addr 172.21.15.67:0/3401169486 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:09:05.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.289+0000 7f1f17e00640 1 --2- 172.21.15.67:0/3401169486 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5622bd6e4c00 0x5622bd6e7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:05.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.290+0000 7f1f1da00640 1 --2- 172.21.15.67:0/3401169486 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5622bd6df0a0 0x5622bd6df4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:05.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.290+0000 7f1f15600640 1 --2- 172.21.15.67:0/1011543207 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f1f000a28d0 0x7f1f000a4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:05.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.290+0000 7f1f1d000640 1 -- 172.21.15.67:0/3401169486 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5622bd6df0a0 msgr2=0x5622bd6df4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:09:05.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.290+0000 7f1f1d000640 1 --2- 172.21.15.67:0/3401169486 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5622bd6df0a0 0x5622bd6df4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:05.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.290+0000 7f1f1d000640 1 -- 172.21.15.67:0/3401169486 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5622bd6e4c00 msgr2=0x5622bd6e7090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:09:05.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.290+0000 7f1f1d000640 1 --2- 172.21.15.67:0/3401169486 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5622bd6e4c00 0x5622bd6e7090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:05.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.290+0000 7f1f1d000640 1 -- 172.21.15.67:0/3401169486 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5622bd4217e0 con 0x5622bd6dfa90 2024-03-28T16:09:05.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.291+0000 7f1f1da00640 1 --2- 172.21.15.67:0/3401169486 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5622bd6df0a0 0x5622bd6df4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:09:05.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.291+0000 7f1f1d000640 1 --2- 172.21.15.67:0/3401169486 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5622bd6dfa90 0x5622bd6e4590 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c0cc020 tx=0x7f1f0c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a4e467848b17bf9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:05.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.290+0000 7f1f15600640 1 -- 172.21.15.67:0/1011543207 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f1f0c0ce510 con 0x7f1f18076060 2024-03-28T16:09:05.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.291+0000 7f1f1da00640 1 --2- 172.21.15.67:0/1011543207 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f1f000a28d0 0x7f1f000a4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:05.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.292+0000 7f1f0a000640 1 -- 172.21.15.67:0/3401169486 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f0c020070 con 0x5622bd6dfa90 2024-03-28T16:09:05.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.292+0000 7f1f0a000640 1 -- 172.21.15.67:0/3401169486 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1f0c0a8e80 con 0x5622bd6dfa90 2024-03-28T16:09:05.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.292+0000 7f1f0a000640 1 -- 172.21.15.67:0/3401169486 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f0c09aec0 con 0x5622bd6dfa90 2024-03-28T16:09:05.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.294+0000 7f1f2276f740 1 -- 172.21.15.67:0/3401169486 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5622bd6dfa90 msgr2=0x5622bd6e4590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:05.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.294+0000 7f1f2276f740 1 --2- 172.21.15.67:0/3401169486 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5622bd6dfa90 0x5622bd6e4590 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c0cc020 tx=0x7f1f0c005880 comp rx=0 tx=0).stop 2024-03-28T16:09:05.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.294+0000 7f1f1da00640 1 --2- 172.21.15.67:0/1011543207 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f1f000a28d0 0x7f1f000a4d90 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f1f10032380 tx=0x7f1f1002c100 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:05.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.294+0000 7f1f2276f740 1 -- 172.21.15.67:0/3401169486 shutdown_connections 2024-03-28T16:09:05.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.294+0000 7f1f2276f740 1 --2- 172.21.15.67:0/3401169486 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5622bd6e4c00 0x5622bd6e7090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:05.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.294+0000 7f1f2276f740 1 --2- 172.21.15.67:0/3401169486 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5622bd6dfa90 0x5622bd6e4590 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:05.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.294+0000 7f1f2276f740 1 --2- 172.21.15.67:0/3401169486 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5622bd6df0a0 0x5622bd6df4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:05.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.294+0000 7f1f2276f740 1 -- 172.21.15.67:0/3401169486 >> 172.21.15.67:0/3401169486 conn(0x5622bd6da610 msgr2=0x5622bd6daa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:05.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.295+0000 7f1f2276f740 1 -- 172.21.15.67:0/3401169486 shutdown_connections 2024-03-28T16:09:05.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.295+0000 7f1f2276f740 1 -- 172.21.15.67:0/3401169486 wait complete. 2024-03-28T16:09:05.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.296+0000 7f1f2276f740 1 -- start start 2024-03-28T16:09:05.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.296+0000 7f1f2276f740 1 Processor -- start 2024-03-28T16:09:05.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.297+0000 7f1f2276f740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5622bd6df0a0 0x5622bd6fa180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:05.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.297+0000 7f1f2276f740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5622bd6dfa90 0x5622bd6fa6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:05.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.297+0000 7f1f2276f740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5622bd6e4c00 0x5622bd701700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:05.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.298+0000 7f1f17e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5622bd6dfa90 0x5622bd6fa6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:05.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.298+0000 7f1f17e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5622bd6dfa90 0x5622bd6fa6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51896/0 (socket says 172.21.15.67:51896) 2024-03-28T16:09:05.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.298+0000 7f1f1d000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5622bd6df0a0 0x5622bd6fa180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:05.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.298+0000 7f1f1d000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5622bd6df0a0 0x5622bd6fa180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.67:42824/0 (socket says 172.21.15.67:42824) 2024-03-28T16:09:05.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.298+0000 7f1f2276f740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622bd6f8b00 con 0x5622bd6dfa90 2024-03-28T16:09:05.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.298+0000 7f1f2276f740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622bd600eb0 con 0x5622bd6e4c00 2024-03-28T16:09:05.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.298+0000 7f1f2276f740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622bd6f8c80 con 0x5622bd6df0a0 2024-03-28T16:09:05.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.298+0000 7f1f2276f740 10 client.0 _ll_register_callbacks cb 0x5622bd5f5150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:09:05.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.299+0000 7f1f17e00640 1 -- 172.21.15.67:0/3812456978 learned_addr learned my addr 172.21.15.67:0/3812456978 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:09:05.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.299+0000 7f1f1da00640 1 --2- 172.21.15.67:0/3812456978 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5622bd6e4c00 0x5622bd701700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:05.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.299+0000 7f1f17e00640 1 -- 172.21.15.67:0/3812456978 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5622bd6df0a0 msgr2=0x5622bd6fa180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:09:05.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.299+0000 7f1f17e00640 1 --2- 172.21.15.67:0/3812456978 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5622bd6df0a0 0x5622bd6fa180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:05.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.299+0000 7f1f17e00640 1 -- 172.21.15.67:0/3812456978 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5622bd6e4c00 msgr2=0x5622bd701700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:09:05.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.299+0000 7f1f17e00640 1 --2- 172.21.15.67:0/3812456978 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5622bd6e4c00 0x5622bd701700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:05.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.300+0000 7f1f17e00640 1 -- 172.21.15.67:0/3812456978 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5622bd481f70 con 0x5622bd6dfa90 2024-03-28T16:09:05.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.300+0000 7f1f17e00640 1 --2- 172.21.15.67:0/3812456978 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5622bd6dfa90 0x5622bd6fa6c0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f1f04011620 tx=0x7f1f04011ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2cd66a62aec660 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:05.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.300+0000 7f1efea00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:09:05.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.301+0000 7f1efea00640 1 -- 172.21.15.67:0/3812456978 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f04015870 con 0x5622bd6dfa90 2024-03-28T16:09:05.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.301+0000 7f1f2276f740 1 -- 172.21.15.67:0/3812456978 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1f0c0a8e80 con 0x5622bd6dfa90 2024-03-28T16:09:05.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.301+0000 7f1efea00640 1 -- 172.21.15.67:0/3812456978 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f1f04015a10 con 0x5622bd6dfa90 2024-03-28T16:09:05.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.302+0000 7f1efea00640 1 -- 172.21.15.67:0/3812456978 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1f04015c10 con 0x5622bd6dfa90 2024-03-28T16:09:05.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.302+0000 7f1f2276f740 10 client.16125 fetch_fsmap learned FSMap version 19 2024-03-28T16:09:05.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.302+0000 7f1f2276f740 1 -- 172.21.15.67:0/3812456978 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5622bd319170 con 0x5622bd6dfa90 2024-03-28T16:09:05.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.302+0000 7f1efea00640 1 -- 172.21.15.67:0/3812456978 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1f0c0a8e80 con 0x5622bd6dfa90 2024-03-28T16:09:05.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.302+0000 7f1efea00640 1 -- 172.21.15.67:0/3812456978 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f1f040143e0 con 0x5622bd6dfa90 2024-03-28T16:09:05.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.303+0000 7f1f2276f740 10 client.16125 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:09:05.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.303+0000 7f1f2276f740 10 client.16125 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:09:05.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.303+0000 7f1f2276f740 20 client.16125 populate_metadata read hostname 'smithi067' 2024-03-28T16:09:05.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.303+0000 7f1f2276f740 1 --2- 172.21.15.67:0/3812456978 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5622bd706060 0x5622bd7084c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:05.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.303+0000 7f1f2276f740 4 client.16125 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:09:05.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.303+0000 7f1f2276f740 1 -- 172.21.15.67:0/3812456978 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5622bd6523f0 con 0x5622bd706060 2024-03-28T16:09:05.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.303+0000 7f1f1d000640 1 --2- 172.21.15.67:0/3812456978 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5622bd706060 0x5622bd7084c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:05.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.304+0000 7f1efea00640 10 client.16125 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:09:05.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.305+0000 7f1f1d000640 1 --2- 172.21.15.67:0/3812456978 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5622bd706060 0x5622bd7084c0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d9513176ddcba0cd server_cookie=16bf840a13eda1e0 in_seq=0 out_seq=0 2024-03-28T16:09:05.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.307+0000 7f1efea00640 1 -- 172.21.15.67:0/3812456978 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5622bd6523f0 con 0x5622bd706060 2024-03-28T16:09:05.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.307+0000 7f1efea00640 10 client.16125 handle_command_reply: tid=0 2024-03-28T16:09:05.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.307+0000 7f1f2276f740 1 client.16125 shutdown 2024-03-28T16:09:05.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.307+0000 7f1f2276f740 1 -- 172.21.15.67:0/3812456978 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5622bd6dfa90 msgr2=0x5622bd6fa6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:05.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.307+0000 7f1f2276f740 1 --2- 172.21.15.67:0/3812456978 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5622bd6dfa90 0x5622bd6fa6c0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f1f04011620 tx=0x7f1f04011ae0 comp rx=0 tx=0).stop 2024-03-28T16:09:05.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.309+0000 7f1f2276f740 1 -- 172.21.15.67:0/3812456978 shutdown_connections 2024-03-28T16:09:05.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.309+0000 7f1f2276f740 1 --2- 172.21.15.67:0/3812456978 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5622bd706060 0x5622bd7084c0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:05.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.309+0000 7f1f2276f740 1 --2- 172.21.15.67:0/3812456978 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5622bd6e4c00 0x5622bd701700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:05.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.309+0000 7f1f2276f740 1 --2- 172.21.15.67:0/3812456978 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5622bd6dfa90 0x5622bd6fa6c0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:05.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.309+0000 7f1f2276f740 1 --2- 172.21.15.67:0/3812456978 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5622bd6df0a0 0x5622bd6fa180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:05.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.309+0000 7f1f2276f740 1 -- 172.21.15.67:0/3812456978 >> 172.21.15.67:0/3812456978 conn(0x5622bd6da610 msgr2=0x5622bd6d9830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:05.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.309+0000 7f1efea00640 0 client.16125 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:09:05.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.309+0000 7f1f2276f740 1 -- 172.21.15.67:0/3812456978 shutdown_connections 2024-03-28T16:09:05.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.309+0000 7f1f2276f740 1 -- 172.21.15.67:0/3812456978 wait complete. 2024-03-28T16:09:05.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.309+0000 7f1f2276f740 20 client.16125 trim_cache size 0 max 16384 2024-03-28T16:09:05.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.344+0000 7f1f2276f740 1 Processor -- start 2024-03-28T16:09:05.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.344+0000 7f1f2276f740 1 -- start start 2024-03-28T16:09:05.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.345+0000 7f1f2276f740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5622bd6df0a0 0x5622bd6e0530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:05.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.345+0000 7f1f2276f740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5622bd6dfa90 0x5622bd7016c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:05.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.345+0000 7f1f2276f740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5622bd6e4c00 0x5622bd6e0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:05.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.345+0000 7f1f2276f740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f18109f70 con 0x5622bd6dfa90 2024-03-28T16:09:05.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.345+0000 7f1f2276f740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f181a6fb0 con 0x5622bd6df0a0 2024-03-28T16:09:05.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.345+0000 7f1f2276f740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f1810a5d0 con 0x5622bd6e4c00 2024-03-28T16:09:05.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.345+0000 7f1f1d000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5622bd6df0a0 0x5622bd6e0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:05.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.345+0000 7f1f1d000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5622bd6df0a0 0x5622bd6e0530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.67:32830/0 (socket says 172.21.15.67:32830) 2024-03-28T16:09:05.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.345+0000 7f1f1d000640 1 -- 172.21.15.67:0/1942901819 learned_addr learned my addr 172.21.15.67:0/1942901819 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:09:05.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.346+0000 7f1f1da00640 1 --2- 172.21.15.67:0/1942901819 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5622bd6e4c00 0x5622bd6e0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:05.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.346+0000 7f1f17e00640 1 --2- 172.21.15.67:0/1942901819 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5622bd6dfa90 0x5622bd7016c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:05.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.346+0000 7f1f1d000640 1 -- 172.21.15.67:0/1942901819 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5622bd6e4c00 msgr2=0x5622bd6e0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:09:05.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.346+0000 7f1f1d000640 1 --2- 172.21.15.67:0/1942901819 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5622bd6e4c00 0x5622bd6e0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:05.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.346+0000 7f1f1d000640 1 -- 172.21.15.67:0/1942901819 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5622bd6dfa90 msgr2=0x5622bd7016c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:09:05.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.346+0000 7f1f1d000640 1 --2- 172.21.15.67:0/1942901819 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5622bd6dfa90 0x5622bd7016c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:05.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.346+0000 7f1f1d000640 1 -- 172.21.15.67:0/1942901819 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5622bd6d5c20 con 0x5622bd6df0a0 2024-03-28T16:09:05.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.346+0000 7f1f17e00640 1 --2- 172.21.15.67:0/1942901819 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5622bd6dfa90 0x5622bd7016c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:09:05.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.347+0000 7f1f1d000640 1 --2- 172.21.15.67:0/1942901819 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5622bd6df0a0 0x5622bd6e0530 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c09abc0 tx=0x7f1f0c0cbfb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78bfde7e9652d02b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:05.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.347+0000 7f1f09600640 1 -- 172.21.15.67:0/1942901819 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f0c020040 con 0x5622bd6df0a0 2024-03-28T16:09:05.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.347+0000 7f1f09600640 1 -- 172.21.15.67:0/1942901819 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1f0c09b5c0 con 0x5622bd6df0a0 2024-03-28T16:09:05.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.347+0000 7f1f09600640 1 -- 172.21.15.67:0/1942901819 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f0c010750 con 0x5622bd6df0a0 2024-03-28T16:09:05.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.348+0000 7f1f2276f740 1 -- 172.21.15.67:0/1942901819 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5622bd6df0a0 msgr2=0x5622bd6e0530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:05.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.348+0000 7f1f2276f740 1 --2- 172.21.15.67:0/1942901819 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5622bd6df0a0 0x5622bd6e0530 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c09abc0 tx=0x7f1f0c0cbfb0 comp rx=0 tx=0).stop 2024-03-28T16:09:05.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.349+0000 7f1f2276f740 1 -- 172.21.15.67:0/1942901819 shutdown_connections 2024-03-28T16:09:05.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.349+0000 7f1f2276f740 1 --2- 172.21.15.67:0/1942901819 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5622bd6e4c00 0x5622bd6e0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:05.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.349+0000 7f1f2276f740 1 --2- 172.21.15.67:0/1942901819 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5622bd6dfa90 0x5622bd7016c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:05.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.349+0000 7f1f2276f740 1 --2- 172.21.15.67:0/1942901819 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5622bd6df0a0 0x5622bd6e0530 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:05.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.349+0000 7f1f2276f740 1 -- 172.21.15.67:0/1942901819 >> 172.21.15.67:0/1942901819 conn(0x5622bd6da610 msgr2=0x5622bd6de100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:05.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.349+0000 7f1f2276f740 1 -- 172.21.15.67:0/1942901819 shutdown_connections 2024-03-28T16:09:05.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.349+0000 7f1f2276f740 1 -- 172.21.15.67:0/1942901819 wait complete. 2024-03-28T16:09:05.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.350+0000 7f1f2276f740 1 -- start start 2024-03-28T16:09:05.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.350+0000 7f1f2276f740 1 Processor -- start 2024-03-28T16:09:05.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.351+0000 7f1f2276f740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5622bd6df0a0 0x5622bd6ff4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:05.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.351+0000 7f1f2276f740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5622bd6dfa90 0x5622bd6ffa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:05.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.351+0000 7f1f2276f740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5622bd6e4c00 0x5622bd700a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:05.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.351+0000 7f1f2276f740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f1810ac30 con 0x5622bd6dfa90 2024-03-28T16:09:05.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.351+0000 7f1f2276f740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f181a6950 con 0x5622bd6df0a0 2024-03-28T16:09:05.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.351+0000 7f1f2276f740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f1810b290 con 0x5622bd6e4c00 2024-03-28T16:09:05.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.351+0000 7f1f2276f740 10 client.0 _ll_register_callbacks cb 0x7f1f040098d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:09:05.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.351+0000 7f1f1d000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5622bd6df0a0 0x5622bd6ff4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:05.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.352+0000 7f1f1d000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5622bd6df0a0 0x5622bd6ff4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.67:32842/0 (socket says 172.21.15.67:32842) 2024-03-28T16:09:05.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.352+0000 7f1f1d000640 1 -- 172.21.15.67:0/3167763887 learned_addr learned my addr 172.21.15.67:0/3167763887 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:09:05.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.352+0000 7f1f17e00640 1 --2- 172.21.15.67:0/3167763887 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5622bd6dfa90 0x5622bd6ffa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:05.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.352+0000 7f1f1d000640 1 -- 172.21.15.67:0/3167763887 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5622bd6e4c00 msgr2=0x5622bd700a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:09:05.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.352+0000 7f1f1d000640 1 --2- 172.21.15.67:0/3167763887 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5622bd6e4c00 0x5622bd700a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:05.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.352+0000 7f1f1d000640 1 -- 172.21.15.67:0/3167763887 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5622bd6dfa90 msgr2=0x5622bd6ffa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:09:05.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.352+0000 7f1f1d000640 1 --2- 172.21.15.67:0/3167763887 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5622bd6dfa90 0x5622bd6ffa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:05.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.352+0000 7f1f1d000640 1 -- 172.21.15.67:0/3167763887 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5622bd6db080 con 0x5622bd6df0a0 2024-03-28T16:09:05.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.353+0000 7f1f1d000640 1 --2- 172.21.15.67:0/3167763887 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5622bd6df0a0 0x5622bd6ff4c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c000c00 tx=0x7f1f0c005880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66d61ed34bac93b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:05.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.353+0000 7f1efea00640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-28T16:09:05.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.353+0000 7f1efea00640 1 -- 172.21.15.67:0/3167763887 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f0c020070 con 0x5622bd6df0a0 2024-03-28T16:09:05.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.353+0000 7f1f2276f740 1 -- 172.21.15.67:0/3167763887 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1f0c09b5c0 con 0x5622bd6df0a0 2024-03-28T16:09:05.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.353+0000 7f1efea00640 1 -- 172.21.15.67:0/3167763887 <== mon.2 v2:172.21.15.150:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f1f0c097530 con 0x5622bd6df0a0 2024-03-28T16:09:05.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.354+0000 7f1efea00640 1 -- 172.21.15.67:0/3167763887 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1f0c09f010 con 0x5622bd6df0a0 2024-03-28T16:09:05.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.354+0000 7f1f2276f740 10 client.35627 fetch_fsmap learned FSMap version 19 2024-03-28T16:09:05.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.354+0000 7f1f2276f740 1 -- 172.21.15.67:0/3167763887 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5622bd761d80 con 0x5622bd6df0a0 2024-03-28T16:09:05.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.355+0000 7f1efea00640 1 -- 172.21.15.67:0/3167763887 <== mon.2 v2:172.21.15.150:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1f0c09b5c0 con 0x5622bd6df0a0 2024-03-28T16:09:05.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.355+0000 7f1efea00640 1 -- 172.21.15.67:0/3167763887 <== mon.2 v2:172.21.15.150:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f1f0c0b1030 con 0x5622bd6df0a0 2024-03-28T16:09:05.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.355+0000 7f1f2276f740 10 client.35627 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:09:05.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.355+0000 7f1f2276f740 10 client.35627 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:09:05.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.355+0000 7f1f2276f740 20 client.35627 populate_metadata read hostname 'smithi067' 2024-03-28T16:09:05.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.355+0000 7f1f2276f740 1 --2- 172.21.15.67:0/3167763887 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5622bd762170 0x5622bd764590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:05.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.355+0000 7f1f2276f740 4 client.35627 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T16:09:05.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.355+0000 7f1f2276f740 1 -- 172.21.15.67:0/3167763887 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5622bd6f9f70 con 0x5622bd762170 2024-03-28T16:09:05.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.357+0000 7f1f17e00640 1 --2- 172.21.15.67:0/3167763887 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5622bd762170 0x5622bd764590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:05.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.358+0000 7f1f17e00640 1 --2- 172.21.15.67:0/3167763887 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5622bd762170 0x5622bd764590 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ee3edf540a7850cc server_cookie=99bfc27091f02f45 in_seq=0 out_seq=0 2024-03-28T16:09:05.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.358+0000 7f1efea00640 10 client.35627 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:09:05.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.361+0000 7f1efea00640 1 -- 172.21.15.67:0/3167763887 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5622bd6f9f70 con 0x5622bd762170 2024-03-28T16:09:05.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.361+0000 7f1efea00640 10 client.35627 handle_command_reply: tid=0 2024-03-28T16:09:05.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.361+0000 7f1f2276f740 1 client.35627 shutdown 2024-03-28T16:09:05.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.361+0000 7f1f2276f740 1 -- 172.21.15.67:0/3167763887 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5622bd6df0a0 msgr2=0x5622bd6ff4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:05.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.361+0000 7f1f2276f740 1 --2- 172.21.15.67:0/3167763887 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5622bd6df0a0 0x5622bd6ff4c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c000c00 tx=0x7f1f0c005880 comp rx=0 tx=0).stop 2024-03-28T16:09:05.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.361+0000 7f1f2276f740 1 -- 172.21.15.67:0/3167763887 shutdown_connections 2024-03-28T16:09:05.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.361+0000 7f1f2276f740 1 --2- 172.21.15.67:0/3167763887 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x5622bd762170 0x5622bd764590 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:05.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.361+0000 7f1f2276f740 1 --2- 172.21.15.67:0/3167763887 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5622bd6e4c00 0x5622bd700a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:05.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.361+0000 7f1f2276f740 1 --2- 172.21.15.67:0/3167763887 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5622bd6dfa90 0x5622bd6ffa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:05.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.361+0000 7f1efea00640 0 client.35627 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:09:05.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.361+0000 7f1f2276f740 1 --2- 172.21.15.67:0/3167763887 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5622bd6df0a0 0x5622bd6ff4c0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:05.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.361+0000 7f1f2276f740 1 -- 172.21.15.67:0/3167763887 >> 172.21.15.67:0/3167763887 conn(0x5622bd6da610 msgr2=0x5622bd6d8fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:05.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.362+0000 7f1f2276f740 1 -- 172.21.15.67:0/3167763887 shutdown_connections 2024-03-28T16:09:05.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.362+0000 7f1f2276f740 1 -- 172.21.15.67:0/3167763887 wait complete. 2024-03-28T16:09:05.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.362+0000 7f1f2276f740 20 client.35627 trim_cache size 0 max 16384 2024-03-28T16:09:05.363 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-28T16:09:05.363 INFO:teuthology.orchestra.run.smithi067.stdout: "return_code": 0, 2024-03-28T16:09:05.363 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_tag": "674f0a94-e9f8-4e77-a8c3-dde93761779a", 2024-03-28T16:09:05.363 INFO:teuthology.orchestra.run.smithi067.stdout: "mode": "asynchronous" 2024-03-28T16:09:05.364 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-28T16:09:05.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.363+0000 7f1f1fa00640 1 -- 172.21.15.67:0/1011543207 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f1f000a28d0 msgr2=0x7f1f000a4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:05.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.363+0000 7f1f1fa00640 1 --2- 172.21.15.67:0/1011543207 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f1f000a28d0 0x7f1f000a4d90 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f1f10032380 tx=0x7f1f1002c100 comp rx=0 tx=0).stop 2024-03-28T16:09:05.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.363+0000 7f1f1fa00640 1 -- 172.21.15.67:0/1011543207 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1f18076060 msgr2=0x7f1f1819df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:05.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.363+0000 7f1f1fa00640 1 --2- 172.21.15.67:0/1011543207 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1f18076060 0x7f1f1819df40 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c00d950 tx=0x7f1f0c00dd50 comp rx=0 tx=0).stop 2024-03-28T16:09:05.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.363+0000 7f1f1fa00640 1 -- 172.21.15.67:0/1011543207 shutdown_connections 2024-03-28T16:09:05.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.363+0000 7f1f1fa00640 1 --2- 172.21.15.67:0/1011543207 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f1f000a28d0 0x7f1f000a4d90 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:05.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.363+0000 7f1f1fa00640 1 --2- 172.21.15.67:0/1011543207 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f18077c40 0x7f1f181a54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:05.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.363+0000 7f1f1fa00640 1 --2- 172.21.15.67:0/1011543207 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f180772a0 0x7f1f1819e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:05.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.363+0000 7f1f1fa00640 1 --2- 172.21.15.67:0/1011543207 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1f18076060 0x7f1f1819df40 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:05.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.363+0000 7f1f1fa00640 1 -- 172.21.15.67:0/1011543207 >> 172.21.15.67:0/1011543207 conn(0x7f1f180fee50 msgr2=0x7f1f181021b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:05.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.363+0000 7f1f1fa00640 1 -- 172.21.15.67:0/1011543207 shutdown_connections 2024-03-28T16:09:05.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.363+0000 7f1f1fa00640 1 -- 172.21.15.67:0/1011543207 wait complete. 2024-03-28T16:09:05.389 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T16:09:05.390 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T16:09:05.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.574+0000 7fb9a4a00640 1 -- 172.21.15.67:0/3441254204 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9a0104ad0 msgr2=0x7fb9a0104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:05.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.574+0000 7fb9a4a00640 1 --2- 172.21.15.67:0/3441254204 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9a0104ad0 0x7fb9a0104f50 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fb98c009900 tx=0x7fb98c02bc30 comp rx=0 tx=0).stop 2024-03-28T16:09:05.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.574+0000 7fb9a4a00640 1 -- 172.21.15.67:0/3441254204 shutdown_connections 2024-03-28T16:09:05.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.574+0000 7fb9a4a00640 1 --2- 172.21.15.67:0/3441254204 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9a0105490 0x7fb9a010bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:05.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.574+0000 7fb9a4a00640 1 --2- 172.21.15.67:0/3441254204 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9a0104ad0 0x7fb9a0104f50 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:05.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.574+0000 7fb9a4a00640 1 --2- 172.21.15.67:0/3441254204 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb9a01038e0 0x7fb9a0103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:05.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.574+0000 7fb9a4a00640 1 -- 172.21.15.67:0/3441254204 >> 172.21.15.67:0/3441254204 conn(0x7fb9a00fee50 msgr2=0x7fb9a0101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:05.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.574+0000 7fb9a4a00640 1 -- 172.21.15.67:0/3441254204 shutdown_connections 2024-03-28T16:09:05.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.574+0000 7fb9a4a00640 1 -- 172.21.15.67:0/3441254204 wait complete. 2024-03-28T16:09:05.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.575+0000 7fb9a4a00640 1 Processor -- start 2024-03-28T16:09:05.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.575+0000 7fb9a4a00640 1 -- start start 2024-03-28T16:09:05.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.576+0000 7fb9a4a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9a01038e0 0x7fb9a0199b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:05.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.576+0000 7fb9a4a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9a0104ad0 0x7fb9a019a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:05.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.576+0000 7fb9a4a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb9a0105490 0x7fb9a01a10e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:05.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.576+0000 7fb9a4a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a010f460 con 0x7fb9a0104ad0 2024-03-28T16:09:05.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.576+0000 7fb9a4a00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a010f2e0 con 0x7fb9a01038e0 2024-03-28T16:09:05.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.576+0000 7fb9a4a00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a010f5e0 con 0x7fb9a0105490 2024-03-28T16:09:05.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.576+0000 7fb99d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9a0104ad0 0x7fb9a019a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:05.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.576+0000 7fb99d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9a0104ad0 0x7fb9a019a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51932/0 (socket says 172.21.15.67:51932) 2024-03-28T16:09:05.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.576+0000 7fb99d400640 1 -- 172.21.15.67:0/3320034023 learned_addr learned my addr 172.21.15.67:0/3320034023 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:09:05.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.576+0000 7fb99de00640 1 --2- 172.21.15.67:0/3320034023 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9a01038e0 0x7fb9a0199b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:05.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.576+0000 7fb99e800640 1 --2- 172.21.15.67:0/3320034023 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb9a0105490 0x7fb9a01a10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:05.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.576+0000 7fb99d400640 1 -- 172.21.15.67:0/3320034023 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb9a0105490 msgr2=0x7fb9a01a10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:05.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.576+0000 7fb99d400640 1 --2- 172.21.15.67:0/3320034023 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb9a0105490 0x7fb9a01a10e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:05.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.576+0000 7fb99d400640 1 -- 172.21.15.67:0/3320034023 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9a01038e0 msgr2=0x7fb9a0199b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:05.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.576+0000 7fb99d400640 1 --2- 172.21.15.67:0/3320034023 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9a01038e0 0x7fb9a0199b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:05.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.576+0000 7fb99d400640 1 -- 172.21.15.67:0/3320034023 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9a01a1750 con 0x7fb9a0104ad0 2024-03-28T16:09:05.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.576+0000 7fb99de00640 1 --2- 172.21.15.67:0/3320034023 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9a01038e0 0x7fb9a0199b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T16:09:05.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.577+0000 7fb99d400640 1 --2- 172.21.15.67:0/3320034023 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9a0104ad0 0x7fb9a019a070 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fb98c000c00 tx=0x7fb98c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:05.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.577+0000 7fb996a00640 1 -- 172.21.15.67:0/3320034023 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb98c043020 con 0x7fb9a0104ad0 2024-03-28T16:09:05.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.577+0000 7fb9a4a00640 1 -- 172.21.15.67:0/3320034023 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9a01a19e0 con 0x7fb9a0104ad0 2024-03-28T16:09:05.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.577+0000 7fb996a00640 1 -- 172.21.15.67:0/3320034023 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb98c0054c0 con 0x7fb9a0104ad0 2024-03-28T16:09:05.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.577+0000 7fb996a00640 1 -- 172.21.15.67:0/3320034023 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb98c003d40 con 0x7fb9a0104ad0 2024-03-28T16:09:05.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.577+0000 7fb9a4a00640 1 -- 172.21.15.67:0/3320034023 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9a01a1f80 con 0x7fb9a0104ad0 2024-03-28T16:09:05.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.580+0000 7fb996a00640 1 -- 172.21.15.67:0/3320034023 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fb98c046020 con 0x7fb9a0104ad0 2024-03-28T16:09:05.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.581+0000 7fb996a00640 1 --2- 172.21.15.67:0/3320034023 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb9840a2800 0x7fb9840a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:05.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.582+0000 7fb9a7794740 1 Processor -- start 2024-03-28T16:09:05.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.582+0000 7fb996a00640 1 -- 172.21.15.67:0/3320034023 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fb98c0f0f50 con 0x7fb9a0104ad0 2024-03-28T16:09:05.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.582+0000 7fb99de00640 1 --2- 172.21.15.67:0/3320034023 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb9840a2800 0x7fb9840a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:05.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.582+0000 7fb9a7794740 1 -- start start 2024-03-28T16:09:05.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.582+0000 7fb9a7794740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557f11eeae80 0x557f11eeb2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:05.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.582+0000 7fb9a7794740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557f11eeb870 0x557f11ef0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:05.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.582+0000 7fb9a7794740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557f11ef09e0 0x557f11ef2e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:05.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.582+0000 7fb9a7794740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f11e50250 con 0x557f11eeae80 2024-03-28T16:09:05.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.582+0000 7fb9a7794740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f11df8a30 con 0x557f11eeb870 2024-03-28T16:09:05.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.582+0000 7fb9a7794740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f11e4c2f0 con 0x557f11ef09e0 2024-03-28T16:09:05.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.583+0000 7fb99de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557f11eeae80 0x557f11eeb2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:05.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.583+0000 7fb99de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557f11eeae80 0x557f11eeb2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51942/0 (socket says 172.21.15.67:51942) 2024-03-28T16:09:05.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.583+0000 7fb99de00640 1 -- 172.21.15.67:0/2217239467 learned_addr learned my addr 172.21.15.67:0/2217239467 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:09:05.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.583+0000 7fb99de00640 1 --2- 172.21.15.67:0/3320034023 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb9840a2800 0x7fb9840a4cc0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fb9900058c0 tx=0x7fb990005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:05.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.583+0000 7fb99e800640 1 --2- 172.21.15.67:0/2217239467 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557f11ef09e0 0x557f11ef2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:05.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.583+0000 7fb99d400640 1 --2- 172.21.15.67:0/2217239467 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557f11eeb870 0x557f11ef0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:05.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.583+0000 7fb99de00640 1 -- 172.21.15.67:0/2217239467 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557f11ef09e0 msgr2=0x557f11ef2e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:09:05.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.583+0000 7fb99de00640 1 --2- 172.21.15.67:0/2217239467 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557f11ef09e0 0x557f11ef2e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:05.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.583+0000 7fb99de00640 1 -- 172.21.15.67:0/2217239467 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557f11eeb870 msgr2=0x557f11ef0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:09:05.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.583+0000 7fb99de00640 1 --2- 172.21.15.67:0/2217239467 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557f11eeb870 0x557f11ef0370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:05.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.583+0000 7fb99de00640 1 -- 172.21.15.67:0/2217239467 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557f11de3690 con 0x557f11eeae80 2024-03-28T16:09:05.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.583+0000 7fb99d400640 1 --2- 172.21.15.67:0/2217239467 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557f11eeb870 0x557f11ef0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:09:05.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.583+0000 7fb99de00640 1 --2- 172.21.15.67:0/2217239467 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557f11eeae80 0x557f11eeb2a0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fb99000bff0 tx=0x7fb990012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f978401ca595316a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:05.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.584+0000 7fb98b000640 1 -- 172.21.15.67:0/2217239467 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb990022070 con 0x557f11eeae80 2024-03-28T16:09:05.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.584+0000 7fb98b000640 1 -- 172.21.15.67:0/2217239467 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb990011e50 con 0x557f11eeae80 2024-03-28T16:09:05.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.584+0000 7fb98b000640 1 -- 172.21.15.67:0/2217239467 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb99001da80 con 0x557f11eeae80 2024-03-28T16:09:05.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.585+0000 7fb9a7794740 1 -- 172.21.15.67:0/2217239467 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557f11eeae80 msgr2=0x557f11eeb2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:05.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.585+0000 7fb9a7794740 1 --2- 172.21.15.67:0/2217239467 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557f11eeae80 0x557f11eeb2a0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fb99000bff0 tx=0x7fb990012000 comp rx=0 tx=0).stop 2024-03-28T16:09:05.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.585+0000 7fb9a7794740 1 -- 172.21.15.67:0/2217239467 shutdown_connections 2024-03-28T16:09:05.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.585+0000 7fb9a7794740 1 --2- 172.21.15.67:0/2217239467 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557f11ef09e0 0x557f11ef2e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:05.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.585+0000 7fb9a7794740 1 --2- 172.21.15.67:0/2217239467 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557f11eeb870 0x557f11ef0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:05.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.585+0000 7fb9a7794740 1 --2- 172.21.15.67:0/2217239467 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557f11eeae80 0x557f11eeb2a0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:05.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.585+0000 7fb9a7794740 1 -- 172.21.15.67:0/2217239467 >> 172.21.15.67:0/2217239467 conn(0x557f11eb3200 msgr2=0x557f11ee67f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:05.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.585+0000 7fb9a7794740 1 -- 172.21.15.67:0/2217239467 shutdown_connections 2024-03-28T16:09:05.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.585+0000 7fb9a7794740 1 -- 172.21.15.67:0/2217239467 wait complete. 2024-03-28T16:09:05.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.585+0000 7fb9a7794740 1 -- start start 2024-03-28T16:09:05.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.586+0000 7fb9a7794740 1 Processor -- start 2024-03-28T16:09:05.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.586+0000 7fb9a7794740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557f11eeae80 0x557f11ef4e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:05.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.586+0000 7fb9a7794740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557f11eeb870 0x557f11ef5390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:05.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.586+0000 7fb9a7794740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557f11ef09e0 0x557f11efc3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:05.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.586+0000 7fb9a7794740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f11ef3990 con 0x557f11eeae80 2024-03-28T16:09:05.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.586+0000 7fb9a7794740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f11e0be80 con 0x557f11eeb870 2024-03-28T16:09:05.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.586+0000 7fb9a7794740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f11ef3b10 con 0x557f11ef09e0 2024-03-28T16:09:05.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.586+0000 7fb9a7794740 10 client.0 _ll_register_callbacks cb 0x557f11e1e130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:09:05.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.586+0000 7fb99de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557f11eeae80 0x557f11ef4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:05.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.586+0000 7fb99de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557f11eeae80 0x557f11ef4e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51948/0 (socket says 172.21.15.67:51948) 2024-03-28T16:09:05.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.586+0000 7fb99de00640 1 -- 172.21.15.67:0/1147658643 learned_addr learned my addr 172.21.15.67:0/1147658643 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:09:05.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.587+0000 7fb99e800640 1 --2- 172.21.15.67:0/1147658643 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557f11ef09e0 0x557f11efc3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:05.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.587+0000 7fb99d400640 1 --2- 172.21.15.67:0/1147658643 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557f11eeb870 0x557f11ef5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:05.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.587+0000 7fb99de00640 1 -- 172.21.15.67:0/1147658643 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557f11ef09e0 msgr2=0x557f11efc3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:09:05.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.587+0000 7fb99de00640 1 --2- 172.21.15.67:0/1147658643 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557f11ef09e0 0x557f11efc3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:05.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.587+0000 7fb99de00640 1 -- 172.21.15.67:0/1147658643 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557f11eeb870 msgr2=0x557f11ef5390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:09:05.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.587+0000 7fb99de00640 1 --2- 172.21.15.67:0/1147658643 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557f11eeb870 0x557f11ef5390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:05.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.587+0000 7fb99de00640 1 -- 172.21.15.67:0/1147658643 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557f11c2e7d0 con 0x557f11eeae80 2024-03-28T16:09:05.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.587+0000 7fb99de00640 1 --2- 172.21.15.67:0/1147658643 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557f11eeae80 0x557f11ef4e50 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fb9900057a0 tx=0x7fb99001da80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ec8886f03c89e50 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:05.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.587+0000 7fb97f400640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:09:05.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.587+0000 7fb97f400640 1 -- 172.21.15.67:0/1147658643 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb990035060 con 0x557f11eeae80 2024-03-28T16:09:05.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.587+0000 7fb97f400640 1 -- 172.21.15.67:0/1147658643 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fb990005030 con 0x557f11eeae80 2024-03-28T16:09:05.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.587+0000 7fb9a7794740 1 -- 172.21.15.67:0/1147658643 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb990011e50 con 0x557f11eeae80 2024-03-28T16:09:05.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.588+0000 7fb97f400640 1 -- 172.21.15.67:0/1147658643 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb990022050 con 0x557f11eeae80 2024-03-28T16:09:05.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.588+0000 7fb9a7794740 10 client.16140 fetch_fsmap learned FSMap version 19 2024-03-28T16:09:05.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.588+0000 7fb9a7794740 1 -- 172.21.15.67:0/1147658643 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x557f11b643f0 con 0x557f11eeae80 2024-03-28T16:09:05.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.588+0000 7fb97f400640 1 -- 172.21.15.67:0/1147658643 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb990011e50 con 0x557f11eeae80 2024-03-28T16:09:05.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.588+0000 7fb97f400640 1 -- 172.21.15.67:0/1147658643 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fb990027450 con 0x557f11eeae80 2024-03-28T16:09:05.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.588+0000 7fb9a7794740 10 client.16140 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:09:05.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.588+0000 7fb9a7794740 10 client.16140 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:09:05.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.588+0000 7fb9a7794740 20 client.16140 populate_metadata read hostname 'smithi067' 2024-03-28T16:09:05.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.588+0000 7fb9a7794740 1 --2- 172.21.15.67:0/1147658643 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x557f11f00930 0x557f11f02d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:05.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.588+0000 7fb9a7794740 4 client.16140 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:09:05.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.588+0000 7fb9a7794740 1 -- 172.21.15.67:0/1147658643 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557f11e5d790 con 0x557f11f00930 2024-03-28T16:09:05.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.589+0000 7fb99d400640 1 --2- 172.21.15.67:0/1147658643 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x557f11f00930 0x557f11f02d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:05.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.589+0000 7fb99d400640 1 --2- 172.21.15.67:0/1147658643 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x557f11f00930 0x557f11f02d20 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c8a2a140b789fc6f server_cookie=5f4a35ef050e8e13 in_seq=0 out_seq=0 2024-03-28T16:09:05.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.589+0000 7fb97f400640 10 client.16140 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:09:05.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.590+0000 7fb97f400640 1 -- 172.21.15.67:0/1147658643 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x557f11e5d790 con 0x557f11f00930 2024-03-28T16:09:05.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.590+0000 7fb97f400640 10 client.16140 handle_command_reply: tid=0 2024-03-28T16:09:05.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.591+0000 7fb9a7794740 1 client.16140 shutdown 2024-03-28T16:09:05.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.591+0000 7fb9a7794740 1 -- 172.21.15.67:0/1147658643 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557f11eeae80 msgr2=0x557f11ef4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:05.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.591+0000 7fb9a7794740 1 --2- 172.21.15.67:0/1147658643 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557f11eeae80 0x557f11ef4e50 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fb9900057a0 tx=0x7fb99001da80 comp rx=0 tx=0).stop 2024-03-28T16:09:05.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.591+0000 7fb9a7794740 1 -- 172.21.15.67:0/1147658643 shutdown_connections 2024-03-28T16:09:05.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.591+0000 7fb9a7794740 1 --2- 172.21.15.67:0/1147658643 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x557f11f00930 0x557f11f02d20 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:05.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.591+0000 7fb97f400640 0 client.16140 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:09:05.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.591+0000 7fb9a7794740 1 --2- 172.21.15.67:0/1147658643 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557f11ef09e0 0x557f11efc3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:05.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.591+0000 7fb9a7794740 1 --2- 172.21.15.67:0/1147658643 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557f11eeb870 0x557f11ef5390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:05.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.591+0000 7fb9a7794740 1 --2- 172.21.15.67:0/1147658643 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557f11eeae80 0x557f11ef4e50 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:05.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.591+0000 7fb9a7794740 1 -- 172.21.15.67:0/1147658643 >> 172.21.15.67:0/1147658643 conn(0x557f11eb3200 msgr2=0x557f11ee5a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:05.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.591+0000 7fb9a7794740 1 -- 172.21.15.67:0/1147658643 shutdown_connections 2024-03-28T16:09:05.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.591+0000 7fb9a7794740 1 -- 172.21.15.67:0/1147658643 wait complete. 2024-03-28T16:09:05.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.591+0000 7fb9a7794740 20 client.16140 trim_cache size 0 max 16384 2024-03-28T16:09:05.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.616+0000 7fb9a7794740 1 Processor -- start 2024-03-28T16:09:05.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.617+0000 7fb9a7794740 1 -- start start 2024-03-28T16:09:05.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.617+0000 7fb9a7794740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557f11eeae80 0x557f11f07590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:05.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.617+0000 7fb9a7794740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557f11eeb870 0x557f11efc390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:05.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.617+0000 7fb9a7794740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557f11ef09e0 0x557f11f07ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:05.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.617+0000 7fb9a7794740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a01a8b40 con 0x557f11ef09e0 2024-03-28T16:09:05.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.617+0000 7fb9a7794740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a01ab100 con 0x557f11eeae80 2024-03-28T16:09:05.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.617+0000 7fb9a7794740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a01a9140 con 0x557f11eeb870 2024-03-28T16:09:05.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.617+0000 7fb99e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557f11ef09e0 0x557f11f07ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:05.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.617+0000 7fb99e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557f11ef09e0 0x557f11f07ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51952/0 (socket says 172.21.15.67:51952) 2024-03-28T16:09:05.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.617+0000 7fb99e800640 1 -- 172.21.15.67:0/2534669972 learned_addr learned my addr 172.21.15.67:0/2534669972 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:09:05.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.617+0000 7fb99de00640 1 --2- 172.21.15.67:0/2534669972 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557f11eeae80 0x557f11f07590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:05.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.618+0000 7fb99d400640 1 --2- 172.21.15.67:0/2534669972 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557f11eeb870 0x557f11efc390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:05.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.618+0000 7fb99e800640 1 -- 172.21.15.67:0/2534669972 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557f11eeb870 msgr2=0x557f11efc390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:09:05.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.618+0000 7fb99e800640 1 --2- 172.21.15.67:0/2534669972 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557f11eeb870 0x557f11efc390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:05.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.618+0000 7fb99e800640 1 -- 172.21.15.67:0/2534669972 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557f11eeae80 msgr2=0x557f11f07590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:09:05.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.618+0000 7fb99e800640 1 --2- 172.21.15.67:0/2534669972 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557f11eeae80 0x557f11f07590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:05.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.618+0000 7fb99e800640 1 -- 172.21.15.67:0/2534669972 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557f11b240f0 con 0x557f11ef09e0 2024-03-28T16:09:05.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.618+0000 7fb99e800640 1 --2- 172.21.15.67:0/2534669972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557f11ef09e0 0x557f11f07ad0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fb998009d70 tx=0x7fb9980127c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b29e86d8c0df823 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:05.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.618+0000 7fb98a600640 1 -- 172.21.15.67:0/2534669972 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb998004d20 con 0x557f11ef09e0 2024-03-28T16:09:05.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.618+0000 7fb98a600640 1 -- 172.21.15.67:0/2534669972 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb9980054d0 con 0x557f11ef09e0 2024-03-28T16:09:05.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.618+0000 7fb98a600640 1 -- 172.21.15.67:0/2534669972 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb99801a9c0 con 0x557f11ef09e0 2024-03-28T16:09:05.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.619+0000 7fb9a7794740 1 -- 172.21.15.67:0/2534669972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557f11ef09e0 msgr2=0x557f11f07ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:05.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.619+0000 7fb9a7794740 1 --2- 172.21.15.67:0/2534669972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557f11ef09e0 0x557f11f07ad0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fb998009d70 tx=0x7fb9980127c0 comp rx=0 tx=0).stop 2024-03-28T16:09:05.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.619+0000 7fb9a7794740 1 -- 172.21.15.67:0/2534669972 shutdown_connections 2024-03-28T16:09:05.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.619+0000 7fb9a7794740 1 --2- 172.21.15.67:0/2534669972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557f11ef09e0 0x557f11f07ad0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:05.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.619+0000 7fb9a7794740 1 --2- 172.21.15.67:0/2534669972 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557f11eeb870 0x557f11efc390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:05.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.619+0000 7fb9a7794740 1 --2- 172.21.15.67:0/2534669972 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557f11eeae80 0x557f11f07590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:05.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.619+0000 7fb9a7794740 1 -- 172.21.15.67:0/2534669972 >> 172.21.15.67:0/2534669972 conn(0x557f11eb3200 msgr2=0x557f11ee5a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:05.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.619+0000 7fb9a7794740 1 -- 172.21.15.67:0/2534669972 shutdown_connections 2024-03-28T16:09:05.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.619+0000 7fb9a7794740 1 -- 172.21.15.67:0/2534669972 wait complete. 2024-03-28T16:09:05.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.620+0000 7fb9a7794740 1 -- start start 2024-03-28T16:09:05.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.620+0000 7fb9a7794740 1 Processor -- start 2024-03-28T16:09:05.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.620+0000 7fb9a7794740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557f11eeae80 0x557f11ef99c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:05.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.620+0000 7fb9a7794740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557f11eeb870 0x557f11ef9f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:05.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.620+0000 7fb9a7794740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557f11ef09e0 0x557f11efaf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:05.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.621+0000 7fb9a7794740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a01a9780 con 0x557f11eeae80 2024-03-28T16:09:05.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.621+0000 7fb9a7794740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a01aaaa0 con 0x557f11eeb870 2024-03-28T16:09:05.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.621+0000 7fb9a7794740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a01a9de0 con 0x557f11ef09e0 2024-03-28T16:09:05.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.621+0000 7fb9a7794740 10 client.0 _ll_register_callbacks cb 0x7fb99000bec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:09:05.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.621+0000 7fb99d400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557f11eeb870 0x557f11ef9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:05.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.621+0000 7fb99de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557f11eeae80 0x557f11ef99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:05.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.621+0000 7fb99d400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557f11eeb870 0x557f11ef9f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.67:32902/0 (socket says 172.21.15.67:32902) 2024-03-28T16:09:05.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.621+0000 7fb99d400640 1 -- 172.21.15.67:0/729983830 learned_addr learned my addr 172.21.15.67:0/729983830 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:09:05.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.621+0000 7fb99e800640 1 --2- 172.21.15.67:0/729983830 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557f11ef09e0 0x557f11efaf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:05.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.622+0000 7fb99de00640 1 -- 172.21.15.67:0/729983830 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557f11ef09e0 msgr2=0x557f11efaf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:09:05.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.622+0000 7fb99de00640 1 --2- 172.21.15.67:0/729983830 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557f11ef09e0 0x557f11efaf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:05.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.622+0000 7fb99de00640 1 -- 172.21.15.67:0/729983830 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557f11eeb870 msgr2=0x557f11ef9f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:09:05.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.622+0000 7fb99de00640 1 --2- 172.21.15.67:0/729983830 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557f11eeb870 0x557f11ef9f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:05.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.622+0000 7fb99de00640 1 -- 172.21.15.67:0/729983830 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557f11efb500 con 0x557f11eeae80 2024-03-28T16:09:05.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.622+0000 7fb99de00640 1 --2- 172.21.15.67:0/729983830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557f11eeae80 0x557f11ef99c0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fb990027940 tx=0x7fb99001de80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c340df70a322f73 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:05.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.622+0000 7fb97f400640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:09:05.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.622+0000 7fb97f400640 1 -- 172.21.15.67:0/729983830 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb990040070 con 0x557f11eeae80 2024-03-28T16:09:05.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.622+0000 7fb97f400640 1 -- 172.21.15.67:0/729983830 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fb990027ab0 con 0x557f11eeae80 2024-03-28T16:09:05.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.622+0000 7fb9a7794740 1 -- 172.21.15.67:0/729983830 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb9980054d0 con 0x557f11eeae80 2024-03-28T16:09:05.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.622+0000 7fb97f400640 1 -- 172.21.15.67:0/729983830 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb990015040 con 0x557f11eeae80 2024-03-28T16:09:05.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.622+0000 7fb9a7794740 10 client.16146 fetch_fsmap learned FSMap version 19 2024-03-28T16:09:05.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.622+0000 7fb9a7794740 1 -- 172.21.15.67:0/729983830 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x557f11f5c160 con 0x557f11eeae80 2024-03-28T16:09:05.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.623+0000 7fb97f400640 1 -- 172.21.15.67:0/729983830 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb9980054d0 con 0x557f11eeae80 2024-03-28T16:09:05.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.623+0000 7fb97f400640 1 -- 172.21.15.67:0/729983830 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fb99003c450 con 0x557f11eeae80 2024-03-28T16:09:05.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.623+0000 7fb9a7794740 10 client.16146 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:09:05.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.623+0000 7fb9a7794740 10 client.16146 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:09:05.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.623+0000 7fb9a7794740 20 client.16146 populate_metadata read hostname 'smithi067' 2024-03-28T16:09:05.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.623+0000 7fb9a7794740 1 --2- 172.21.15.67:0/729983830 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x557f11f00930 0x557f11f5ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:05.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.623+0000 7fb9a7794740 4 client.16146 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T16:09:05.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.623+0000 7fb9a7794740 1 -- 172.21.15.67:0/729983830 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x557f11e57ae0 con 0x557f11f00930 2024-03-28T16:09:05.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.623+0000 7fb99d400640 1 --2- 172.21.15.67:0/729983830 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x557f11f00930 0x557f11f5ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:05.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.624+0000 7fb99d400640 1 --2- 172.21.15.67:0/729983830 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x557f11f00930 0x557f11f5ea30 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d673b6b7340e80a server_cookie=9c8e0f11f2ca45a in_seq=0 out_seq=0 2024-03-28T16:09:05.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.624+0000 7fb97f400640 10 client.16146 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:09:05.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.624+0000 7fb97f400640 1 -- 172.21.15.67:0/729983830 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x557f11e57ae0 con 0x557f11f00930 2024-03-28T16:09:05.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.624+0000 7fb97f400640 10 client.16146 handle_command_reply: tid=0 2024-03-28T16:09:05.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.624+0000 7fb9a7794740 1 client.16146 shutdown 2024-03-28T16:09:05.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.625+0000 7fb9a7794740 1 -- 172.21.15.67:0/729983830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557f11eeae80 msgr2=0x557f11ef99c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:05.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.625+0000 7fb9a7794740 1 --2- 172.21.15.67:0/729983830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557f11eeae80 0x557f11ef99c0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fb990027940 tx=0x7fb99001de80 comp rx=0 tx=0).stop 2024-03-28T16:09:05.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.625+0000 7fb9a7794740 1 -- 172.21.15.67:0/729983830 shutdown_connections 2024-03-28T16:09:05.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.625+0000 7fb9a7794740 1 --2- 172.21.15.67:0/729983830 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x557f11f00930 0x557f11f5ea30 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:05.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.625+0000 7fb9a7794740 1 --2- 172.21.15.67:0/729983830 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557f11ef09e0 0x557f11efaf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:05.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.625+0000 7fb9a7794740 1 --2- 172.21.15.67:0/729983830 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557f11eeb870 0x557f11ef9f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:05.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.625+0000 7fb9a7794740 1 --2- 172.21.15.67:0/729983830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557f11eeae80 0x557f11ef99c0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:05.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.625+0000 7fb97f400640 0 client.16146 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:09:05.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.625+0000 7fb9a7794740 1 -- 172.21.15.67:0/729983830 >> 172.21.15.67:0/729983830 conn(0x557f11eb3200 msgr2=0x557f11ee9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:05.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.625+0000 7fb9a7794740 1 -- 172.21.15.67:0/729983830 shutdown_connections 2024-03-28T16:09:05.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.625+0000 7fb9a7794740 1 -- 172.21.15.67:0/729983830 wait complete. 2024-03-28T16:09:05.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.625+0000 7fb9a7794740 20 client.16146 trim_cache size 0 max 16384 2024-03-28T16:09:05.629 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-28T16:09:05.629 INFO:teuthology.orchestra.run.smithi067.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T16:09:05.629 INFO:teuthology.orchestra.run.smithi067.stdout: "scrubs": { 2024-03-28T16:09:05.629 INFO:teuthology.orchestra.run.smithi067.stdout: "674f0a94-e9f8-4e77-a8c3-dde93761779a": { 2024-03-28T16:09:05.629 INFO:teuthology.orchestra.run.smithi067.stdout: "path": "/", 2024-03-28T16:09:05.629 INFO:teuthology.orchestra.run.smithi067.stdout: "tag": "674f0a94-e9f8-4e77-a8c3-dde93761779a", 2024-03-28T16:09:05.630 INFO:teuthology.orchestra.run.smithi067.stdout: "options": "recursive,force" 2024-03-28T16:09:05.630 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:05.630 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:05.630 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-28T16:09:05.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.625+0000 7fb9a4a00640 1 -- 172.21.15.67:0/3320034023 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb9840a2800 msgr2=0x7fb9840a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:05.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.625+0000 7fb9a4a00640 1 --2- 172.21.15.67:0/3320034023 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb9840a2800 0x7fb9840a4cc0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fb9900058c0 tx=0x7fb990005850 comp rx=0 tx=0).stop 2024-03-28T16:09:05.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.625+0000 7fb9a4a00640 1 -- 172.21.15.67:0/3320034023 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9a0104ad0 msgr2=0x7fb9a019a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:05.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.626+0000 7fb9a4a00640 1 --2- 172.21.15.67:0/3320034023 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9a0104ad0 0x7fb9a019a070 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fb98c000c00 tx=0x7fb98c0042f0 comp rx=0 tx=0).stop 2024-03-28T16:09:05.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.626+0000 7fb9a4a00640 1 -- 172.21.15.67:0/3320034023 shutdown_connections 2024-03-28T16:09:05.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.626+0000 7fb9a4a00640 1 --2- 172.21.15.67:0/3320034023 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb9840a2800 0x7fb9840a4cc0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:05.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.626+0000 7fb9a4a00640 1 --2- 172.21.15.67:0/3320034023 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb9a0105490 0x7fb9a01a10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:05.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.626+0000 7fb9a4a00640 1 --2- 172.21.15.67:0/3320034023 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9a0104ad0 0x7fb9a019a070 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:05.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.626+0000 7fb9a4a00640 1 --2- 172.21.15.67:0/3320034023 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9a01038e0 0x7fb9a0199b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:05.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.626+0000 7fb9a4a00640 1 -- 172.21.15.67:0/3320034023 >> 172.21.15.67:0/3320034023 conn(0x7fb9a00fee50 msgr2=0x7fb9a0107e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:05.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.626+0000 7fb9a4a00640 1 -- 172.21.15.67:0/3320034023 shutdown_connections 2024-03-28T16:09:05.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:05.626+0000 7fb9a4a00640 1 -- 172.21.15.67:0/3320034023 wait complete. 2024-03-28T16:09:05.642 INFO:tasks.cephfs.filesystem:scrub status for tag:674f0a94-e9f8-4e77-a8c3-dde93761779a - {'path': '/', 'tag': '674f0a94-e9f8-4e77-a8c3-dde93761779a', 'options': 'recursive,force'} 2024-03-28T16:09:06.047 DEBUG:teuthology.parallel:result is None 2024-03-28T16:09:06.048 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-03-28T16:09:06.159 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-03-28T16:09:06.159 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-03-28T16:09:06.170 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-03-28T16:09:07.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:06 smithi150 ceph-mon[34825]: pgmap v605: 97 pgs: 97 active+clean; 7.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.1 KiB/s wr, 57 op/s 2024-03-28T16:09:07.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:06 smithi150 ceph-mon[34825]: scrub queued for path: / 2024-03-28T16:09:07.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:06 smithi150 ceph-mon[34825]: scrub summary: idle+waiting paths [/] 2024-03-28T16:09:07.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:06 smithi150 ceph-mon[34825]: scrub summary: active paths [/] 2024-03-28T16:09:07.053 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:06 smithi150 ceph-mon[34825]: scrub summary: idle+waiting paths [/] 2024-03-28T16:09:07.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:06 smithi067 ceph-mon[31707]: pgmap v605: 97 pgs: 97 active+clean; 7.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.1 KiB/s wr, 57 op/s 2024-03-28T16:09:07.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:06 smithi067 ceph-mon[31707]: scrub queued for path: / 2024-03-28T16:09:07.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:06 smithi067 ceph-mon[31707]: scrub summary: idle+waiting paths [/] 2024-03-28T16:09:07.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:06 smithi067 ceph-mon[31707]: scrub summary: active paths [/] 2024-03-28T16:09:07.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:06 smithi067 ceph-mon[31707]: scrub summary: idle+waiting paths [/] 2024-03-28T16:09:07.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:06 smithi183 ceph-mon[35580]: pgmap v605: 97 pgs: 97 active+clean; 7.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.1 KiB/s wr, 57 op/s 2024-03-28T16:09:07.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:06 smithi183 ceph-mon[35580]: scrub queued for path: / 2024-03-28T16:09:07.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:06 smithi183 ceph-mon[35580]: scrub summary: idle+waiting paths [/] 2024-03-28T16:09:07.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:06 smithi183 ceph-mon[35580]: scrub summary: active paths [/] 2024-03-28T16:09:07.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:06 smithi183 ceph-mon[35580]: scrub summary: idle+waiting paths [/] 2024-03-28T16:09:08.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:07 smithi150 ceph-mon[34825]: scrub summary: idle 2024-03-28T16:09:08.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:07 smithi067 ceph-mon[31707]: scrub summary: idle 2024-03-28T16:09:08.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:07 smithi183 ceph-mon[35580]: scrub summary: idle 2024-03-28T16:09:09.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:08 smithi150 ceph-mon[34825]: pgmap v606: 97 pgs: 97 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 1.1 KiB/s wr, 103 op/s 2024-03-28T16:09:09.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:08 smithi067 ceph-mon[31707]: pgmap v606: 97 pgs: 97 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 1.1 KiB/s wr, 103 op/s 2024-03-28T16:09:09.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:08 smithi183 ceph-mon[35580]: pgmap v606: 97 pgs: 97 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 1.1 KiB/s wr, 103 op/s 2024-03-28T16:09:11.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:10 smithi150 ceph-mon[34825]: pgmap v607: 97 pgs: 97 active+clean; 1.7 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 1.7 KiB/s wr, 181 op/s 2024-03-28T16:09:11.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:10 smithi067 ceph-mon[31707]: pgmap v607: 97 pgs: 97 active+clean; 1.7 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 1.7 KiB/s wr, 181 op/s 2024-03-28T16:09:11.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:10 smithi183 ceph-mon[35580]: pgmap v607: 97 pgs: 97 active+clean; 1.7 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 1.7 KiB/s wr, 181 op/s 2024-03-28T16:09:13.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:12 smithi150 ceph-mon[34825]: pgmap v608: 97 pgs: 97 active+clean; 979 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 3.2 KiB/s wr, 218 op/s 2024-03-28T16:09:13.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:12 smithi067 ceph-mon[31707]: pgmap v608: 97 pgs: 97 active+clean; 979 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 3.2 KiB/s wr, 218 op/s 2024-03-28T16:09:13.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:12 smithi183 ceph-mon[35580]: pgmap v608: 97 pgs: 97 active+clean; 979 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 3.2 KiB/s wr, 218 op/s 2024-03-28T16:09:15.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:14 smithi150 ceph-mon[34825]: pgmap v609: 97 pgs: 97 active+clean; 979 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 3.2 KiB/s wr, 217 op/s 2024-03-28T16:09:15.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:14 smithi067 ceph-mon[31707]: pgmap v609: 97 pgs: 97 active+clean; 979 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 3.2 KiB/s wr, 217 op/s 2024-03-28T16:09:15.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:14 smithi183 ceph-mon[35580]: pgmap v609: 97 pgs: 97 active+clean; 979 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 3.2 KiB/s wr, 217 op/s 2024-03-28T16:09:17.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:16 smithi150 ceph-mon[34825]: pgmap v610: 97 pgs: 97 active+clean; 980 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 3.2 KiB/s wr, 218 op/s 2024-03-28T16:09:17.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:16 smithi067 ceph-mon[31707]: pgmap v610: 97 pgs: 97 active+clean; 980 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 3.2 KiB/s wr, 218 op/s 2024-03-28T16:09:17.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:16 smithi183 ceph-mon[35580]: pgmap v610: 97 pgs: 97 active+clean; 980 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 3.2 KiB/s wr, 218 op/s 2024-03-28T16:09:19.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:18 smithi150 ceph-mon[34825]: pgmap v611: 97 pgs: 97 active+clean; 980 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 2.2 KiB/s wr, 162 op/s 2024-03-28T16:09:19.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:18 smithi067 ceph-mon[31707]: pgmap v611: 97 pgs: 97 active+clean; 980 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 2.2 KiB/s wr, 162 op/s 2024-03-28T16:09:19.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:18 smithi183 ceph-mon[35580]: pgmap v611: 97 pgs: 97 active+clean; 980 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 2.2 KiB/s wr, 162 op/s 2024-03-28T16:09:21.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:20 smithi150 ceph-mon[34825]: pgmap v612: 97 pgs: 97 active+clean; 980 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 2.2 KiB/s wr, 115 op/s 2024-03-28T16:09:21.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:20 smithi067 ceph-mon[31707]: pgmap v612: 97 pgs: 97 active+clean; 980 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 2.2 KiB/s wr, 115 op/s 2024-03-28T16:09:21.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:20 smithi183 ceph-mon[35580]: pgmap v612: 97 pgs: 97 active+clean; 980 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 2.2 KiB/s wr, 115 op/s 2024-03-28T16:09:23.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:22 smithi150 ceph-mon[34825]: pgmap v613: 97 pgs: 97 active+clean; 980 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 1.7 KiB/s wr, 38 op/s 2024-03-28T16:09:23.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:22 smithi067 ceph-mon[31707]: pgmap v613: 97 pgs: 97 active+clean; 980 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 1.7 KiB/s wr, 38 op/s 2024-03-28T16:09:23.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:22 smithi183 ceph-mon[35580]: pgmap v613: 97 pgs: 97 active+clean; 980 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 1.7 KiB/s wr, 38 op/s 2024-03-28T16:09:25.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:24 smithi150 ceph-mon[34825]: pgmap v614: 97 pgs: 97 active+clean; 980 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2024-03-28T16:09:25.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:24 smithi067 ceph-mon[31707]: pgmap v614: 97 pgs: 97 active+clean; 980 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2024-03-28T16:09:25.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:24 smithi183 ceph-mon[35580]: pgmap v614: 97 pgs: 97 active+clean; 980 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2024-03-28T16:09:27.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:26 smithi150 ceph-mon[34825]: pgmap v615: 97 pgs: 97 active+clean; 980 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 170 B/s wr, 2 op/s 2024-03-28T16:09:27.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:26 smithi067 ceph-mon[31707]: pgmap v615: 97 pgs: 97 active+clean; 980 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 170 B/s wr, 2 op/s 2024-03-28T16:09:27.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:26 smithi183 ceph-mon[35580]: pgmap v615: 97 pgs: 97 active+clean; 980 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 170 B/s wr, 2 op/s 2024-03-28T16:09:29.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:28 smithi150 ceph-mon[34825]: pgmap v616: 97 pgs: 97 active+clean; 980 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:09:29.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:28 smithi067 ceph-mon[31707]: pgmap v616: 97 pgs: 97 active+clean; 980 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:09:29.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:28 smithi183 ceph-mon[35580]: pgmap v616: 97 pgs: 97 active+clean; 980 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:09:31.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:30 smithi150 ceph-mon[34825]: pgmap v617: 97 pgs: 97 active+clean; 980 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:09:31.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:30 smithi067 ceph-mon[31707]: pgmap v617: 97 pgs: 97 active+clean; 980 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:09:31.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:30 smithi183 ceph-mon[35580]: pgmap v617: 97 pgs: 97 active+clean; 980 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:09:33.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:32 smithi150 ceph-mon[34825]: pgmap v618: 97 pgs: 97 active+clean; 980 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:09:33.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:32 smithi067 ceph-mon[31707]: pgmap v618: 97 pgs: 97 active+clean; 980 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:09:33.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:32 smithi183 ceph-mon[35580]: pgmap v618: 97 pgs: 97 active+clean; 980 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:09:35.052 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:34 smithi150 ceph-mon[34825]: pgmap v619: 97 pgs: 97 active+clean; 980 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:09:35.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:34 smithi067 ceph-mon[31707]: pgmap v619: 97 pgs: 97 active+clean; 980 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:09:35.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:34 smithi183 ceph-mon[35580]: pgmap v619: 97 pgs: 97 active+clean; 980 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:09:35.643 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T16:09:35.643 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T16:09:35.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.884+0000 7f5c9fe00640 1 -- 172.21.15.67:0/3000756782 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ca0107bc0 msgr2=0x7f5ca0107fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:35.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.884+0000 7f5c9fe00640 1 --2- 172.21.15.67:0/3000756782 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ca0107bc0 0x7f5ca0107fe0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f5c90009900 tx=0x7f5c9002bc30 comp rx=0 tx=0).stop 2024-03-28T16:09:35.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.884+0000 7f5c9fe00640 1 -- 172.21.15.67:0/3000756782 shutdown_connections 2024-03-28T16:09:35.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.884+0000 7f5c9fe00640 1 --2- 172.21.15.67:0/3000756782 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5ca0109770 0x7f5ca0110000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:35.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.884+0000 7f5c9fe00640 1 --2- 172.21.15.67:0/3000756782 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ca0108db0 0x7f5ca0109230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:35.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.884+0000 7f5c9fe00640 1 --2- 172.21.15.67:0/3000756782 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ca0107bc0 0x7f5ca0107fe0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:35.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.884+0000 7f5c9fe00640 1 -- 172.21.15.67:0/3000756782 >> 172.21.15.67:0/3000756782 conn(0x7f5ca0076b70 msgr2=0x7f5ca0078f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:35.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.884+0000 7f5c9fe00640 1 -- 172.21.15.67:0/3000756782 shutdown_connections 2024-03-28T16:09:35.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.884+0000 7f5c9fe00640 1 -- 172.21.15.67:0/3000756782 wait complete. 2024-03-28T16:09:35.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.885+0000 7f5c9fe00640 1 Processor -- start 2024-03-28T16:09:35.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.885+0000 7f5c9fe00640 1 -- start start 2024-03-28T16:09:35.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.885+0000 7f5c9fe00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ca0107bc0 0x7f5ca019de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:35.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.885+0000 7f5c9fe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5ca0108db0 0x7f5ca019e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:35.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.885+0000 7f5c9fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ca0109770 0x7f5ca01a5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:35.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.885+0000 7f5c9fe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca0113740 con 0x7f5ca0109770 2024-03-28T16:09:35.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.885+0000 7f5c9fe00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca01135c0 con 0x7f5ca0107bc0 2024-03-28T16:09:35.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.885+0000 7f5c9fe00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca01138c0 con 0x7f5ca0108db0 2024-03-28T16:09:35.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.886+0000 7f5c9f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ca0109770 0x7f5ca01a5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:35.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.886+0000 7f5c9f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ca0109770 0x7f5ca01a5390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33752/0 (socket says 172.21.15.67:33752) 2024-03-28T16:09:35.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.886+0000 7f5c9f400640 1 -- 172.21.15.67:0/3823342404 learned_addr learned my addr 172.21.15.67:0/3823342404 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:09:35.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.886+0000 7f5c9ea00640 1 --2- 172.21.15.67:0/3823342404 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ca0107bc0 0x7f5ca019de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:35.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.886+0000 7f5c9e000640 1 --2- 172.21.15.67:0/3823342404 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5ca0108db0 0x7f5ca019e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:35.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.886+0000 7f5c9f400640 1 -- 172.21.15.67:0/3823342404 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5ca0108db0 msgr2=0x7f5ca019e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:35.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.886+0000 7f5c9f400640 1 --2- 172.21.15.67:0/3823342404 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5ca0108db0 0x7f5ca019e370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:35.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.886+0000 7f5c9f400640 1 -- 172.21.15.67:0/3823342404 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ca0107bc0 msgr2=0x7f5ca019de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:35.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.886+0000 7f5c9f400640 1 --2- 172.21.15.67:0/3823342404 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ca0107bc0 0x7f5ca019de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:35.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.886+0000 7f5c9f400640 1 -- 172.21.15.67:0/3823342404 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ca01a5a00 con 0x7f5ca0109770 2024-03-28T16:09:35.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.886+0000 7f5c9ea00640 1 --2- 172.21.15.67:0/3823342404 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ca0107bc0 0x7f5ca019de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T16:09:35.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.887+0000 7f5c9f400640 1 --2- 172.21.15.67:0/3823342404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ca0109770 0x7f5ca01a5390 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f5c9800eaa0 tx=0x7f5c9800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:35.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.887+0000 7f5c97400640 1 -- 172.21.15.67:0/3823342404 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c9800cdb0 con 0x7f5ca0109770 2024-03-28T16:09:35.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.887+0000 7f5c9fe00640 1 -- 172.21.15.67:0/3823342404 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ca01a5cf0 con 0x7f5ca0109770 2024-03-28T16:09:35.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.887+0000 7f5c97400640 1 -- 172.21.15.67:0/3823342404 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5c98005020 con 0x7f5ca0109770 2024-03-28T16:09:35.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.887+0000 7f5c97400640 1 -- 172.21.15.67:0/3823342404 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c980116d0 con 0x7f5ca0109770 2024-03-28T16:09:35.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.887+0000 7f5c9fe00640 1 -- 172.21.15.67:0/3823342404 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ca01a61e0 con 0x7f5ca0109770 2024-03-28T16:09:35.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.891+0000 7f5ca85ef740 1 Processor -- start 2024-03-28T16:09:35.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.892+0000 7f5c97400640 1 -- 172.21.15.67:0/3823342404 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f5c980051d0 con 0x7f5ca0109770 2024-03-28T16:09:35.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.892+0000 7f5ca85ef740 1 -- start start 2024-03-28T16:09:35.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.892+0000 7f5ca85ef740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a54d958050 0x55a54d958470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:35.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.892+0000 7f5ca85ef740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a54d958a40 0x55a54d95d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:35.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.892+0000 7f5ca85ef740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a54d95dbb0 0x55a54d960040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:35.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.892+0000 7f5ca85ef740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a54d8bd250 con 0x55a54d95dbb0 2024-03-28T16:09:35.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.892+0000 7f5ca85ef740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a54d865a30 con 0x55a54d958a40 2024-03-28T16:09:35.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.892+0000 7f5ca85ef740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a54d8b92f0 con 0x55a54d958050 2024-03-28T16:09:35.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.893+0000 7f5c9e000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a54d95dbb0 0x55a54d960040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:35.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.893+0000 7f5c9e000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a54d95dbb0 0x55a54d960040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33762/0 (socket says 172.21.15.67:33762) 2024-03-28T16:09:35.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.893+0000 7f5c9ea00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a54d958a40 0x55a54d95d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:35.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.893+0000 7f5c9e000640 1 -- 172.21.15.67:0/1347099484 learned_addr learned my addr 172.21.15.67:0/1347099484 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:09:35.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.893+0000 7f5c9e000640 1 --2- 172.21.15.67:0/1347099484 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a54d958050 0x55a54d958470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:35.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.893+0000 7f5c9e000640 1 -- 172.21.15.67:0/1347099484 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a54d958050 msgr2=0x55a54d958470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:09:35.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.893+0000 7f5c9e000640 1 --2- 172.21.15.67:0/1347099484 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a54d958050 0x55a54d958470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:35.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.893+0000 7f5c9e000640 1 -- 172.21.15.67:0/1347099484 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a54d958a40 msgr2=0x55a54d95d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:09:35.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.893+0000 7f5c9e000640 1 --2- 172.21.15.67:0/1347099484 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a54d958a40 0x55a54d95d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:35.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.893+0000 7f5c9e000640 1 -- 172.21.15.67:0/1347099484 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a54d850690 con 0x55a54d95dbb0 2024-03-28T16:09:35.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.893+0000 7f5c9ea00640 1 --2- 172.21.15.67:0/1347099484 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a54d958a40 0x55a54d95d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T16:09:35.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.894+0000 7f5c97400640 1 --2- 172.21.15.67:0/3823342404 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f5c840a2800 0x7f5c840a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:35.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.894+0000 7f5c9e000640 1 --2- 172.21.15.67:0/1347099484 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a54d95dbb0 0x55a54d960040 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f5c880058c0 tx=0x7f5c8800c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55ac3d56672e5cfb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:35.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.894+0000 7f5c97400640 1 -- 172.21.15.67:0/3823342404 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f5c980cecd0 con 0x7f5ca0109770 2024-03-28T16:09:35.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.894+0000 7f5c9ea00640 1 --2- 172.21.15.67:0/3823342404 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f5c840a2800 0x7f5c840a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:35.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.894+0000 7f5c8fe00640 1 -- 172.21.15.67:0/1347099484 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c880142d0 con 0x55a54d95dbb0 2024-03-28T16:09:35.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.894+0000 7f5c8fe00640 1 -- 172.21.15.67:0/1347099484 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5c8800fc10 con 0x55a54d95dbb0 2024-03-28T16:09:35.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.894+0000 7f5c8fe00640 1 -- 172.21.15.67:0/1347099484 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c880129d0 con 0x55a54d95dbb0 2024-03-28T16:09:35.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.895+0000 7f5c9ea00640 1 --2- 172.21.15.67:0/3823342404 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f5c840a2800 0x7f5c840a4cc0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f5c9002c240 tx=0x7f5c90005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:35.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.895+0000 7f5ca85ef740 1 -- 172.21.15.67:0/1347099484 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a54d95dbb0 msgr2=0x55a54d960040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:35.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.895+0000 7f5ca85ef740 1 --2- 172.21.15.67:0/1347099484 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a54d95dbb0 0x55a54d960040 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f5c880058c0 tx=0x7f5c8800c4d0 comp rx=0 tx=0).stop 2024-03-28T16:09:35.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.895+0000 7f5ca85ef740 1 -- 172.21.15.67:0/1347099484 shutdown_connections 2024-03-28T16:09:35.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.895+0000 7f5ca85ef740 1 --2- 172.21.15.67:0/1347099484 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a54d95dbb0 0x55a54d960040 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:35.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.895+0000 7f5ca85ef740 1 --2- 172.21.15.67:0/1347099484 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a54d958a40 0x55a54d95d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:35.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.895+0000 7f5ca85ef740 1 --2- 172.21.15.67:0/1347099484 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a54d958050 0x55a54d958470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:35.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.895+0000 7f5ca85ef740 1 -- 172.21.15.67:0/1347099484 >> 172.21.15.67:0/1347099484 conn(0x55a54d920200 msgr2=0x55a54d9539c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:35.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.895+0000 7f5ca85ef740 1 -- 172.21.15.67:0/1347099484 shutdown_connections 2024-03-28T16:09:35.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.895+0000 7f5ca85ef740 1 -- 172.21.15.67:0/1347099484 wait complete. 2024-03-28T16:09:35.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.896+0000 7f5ca85ef740 1 -- start start 2024-03-28T16:09:35.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.896+0000 7f5ca85ef740 1 Processor -- start 2024-03-28T16:09:35.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.896+0000 7f5ca85ef740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a54d958050 0x55a54d962020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:35.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.896+0000 7f5ca85ef740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a54d958a40 0x55a54d962560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:35.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.896+0000 7f5ca85ef740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a54d95dbb0 0x55a54d9695a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:35.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.896+0000 7f5ca85ef740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a54d960b60 con 0x55a54d95dbb0 2024-03-28T16:09:35.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.896+0000 7f5ca85ef740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a54d878e80 con 0x55a54d958050 2024-03-28T16:09:35.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.896+0000 7f5ca85ef740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a54d960ce0 con 0x55a54d958a40 2024-03-28T16:09:35.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.896+0000 7f5ca85ef740 10 client.0 _ll_register_callbacks cb 0x55a54d88b130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:09:35.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.897+0000 7f5c9f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a54d95dbb0 0x55a54d9695a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:35.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.897+0000 7f5c9f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a54d95dbb0 0x55a54d9695a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33770/0 (socket says 172.21.15.67:33770) 2024-03-28T16:09:35.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.897+0000 7f5c9f400640 1 -- 172.21.15.67:0/2416718749 learned_addr learned my addr 172.21.15.67:0/2416718749 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:09:35.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.897+0000 7f5c9ea00640 1 --2- 172.21.15.67:0/2416718749 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a54d958050 0x55a54d962020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:35.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.897+0000 7f5c9e000640 1 --2- 172.21.15.67:0/2416718749 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a54d958a40 0x55a54d962560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:35.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.897+0000 7f5c9f400640 1 -- 172.21.15.67:0/2416718749 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a54d958a40 msgr2=0x55a54d962560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:09:35.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.897+0000 7f5c9f400640 1 --2- 172.21.15.67:0/2416718749 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a54d958a40 0x55a54d962560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:35.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.897+0000 7f5c9f400640 1 -- 172.21.15.67:0/2416718749 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a54d958050 msgr2=0x55a54d962020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:09:35.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.897+0000 7f5c9f400640 1 --2- 172.21.15.67:0/2416718749 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a54d958050 0x55a54d962020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:35.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.897+0000 7f5c9f400640 1 -- 172.21.15.67:0/2416718749 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a54d69b7d0 con 0x55a54d95dbb0 2024-03-28T16:09:35.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.897+0000 7f5c9f400640 1 --2- 172.21.15.67:0/2416718749 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a54d95dbb0 0x55a54d9695a0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f5c98097ff0 tx=0x7f5c98086110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cc1b189edd74f0a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:35.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.897+0000 7f5c8cc00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:09:35.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.898+0000 7f5c8cc00640 1 -- 172.21.15.67:0/2416718749 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c9801f040 con 0x55a54d95dbb0 2024-03-28T16:09:35.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.898+0000 7f5c8cc00640 1 -- 172.21.15.67:0/2416718749 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f5c98084bb0 con 0x55a54d95dbb0 2024-03-28T16:09:35.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.898+0000 7f5ca85ef740 1 -- 172.21.15.67:0/2416718749 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5c8800fc10 con 0x55a54d95dbb0 2024-03-28T16:09:35.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.898+0000 7f5c8cc00640 1 -- 172.21.15.67:0/2416718749 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5c9809f770 con 0x55a54d95dbb0 2024-03-28T16:09:35.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.898+0000 7f5ca85ef740 10 client.16158 fetch_fsmap learned FSMap version 19 2024-03-28T16:09:35.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.898+0000 7f5ca85ef740 1 -- 172.21.15.67:0/2416718749 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55a54d5d13f0 con 0x55a54d95dbb0 2024-03-28T16:09:35.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.898+0000 7f5c8cc00640 1 -- 172.21.15.67:0/2416718749 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5c8800fc10 con 0x55a54d95dbb0 2024-03-28T16:09:35.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.898+0000 7f5c8cc00640 1 -- 172.21.15.67:0/2416718749 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f5c9800f040 con 0x55a54d95dbb0 2024-03-28T16:09:35.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.898+0000 7f5ca85ef740 10 client.16158 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:09:35.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.898+0000 7f5ca85ef740 10 client.16158 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:09:35.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.898+0000 7f5ca85ef740 20 client.16158 populate_metadata read hostname 'smithi067' 2024-03-28T16:09:35.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.899+0000 7f5ca85ef740 1 --2- 172.21.15.67:0/2416718749 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55a54d96df10 0x55a54d970390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:35.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.899+0000 7f5ca85ef740 4 client.16158 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:09:35.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.899+0000 7f5ca85ef740 1 -- 172.21.15.67:0/2416718749 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a54d8ca790 con 0x55a54d96df10 2024-03-28T16:09:35.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.899+0000 7f5c9ea00640 1 --2- 172.21.15.67:0/2416718749 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55a54d96df10 0x55a54d970390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:35.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.900+0000 7f5c9ea00640 1 --2- 172.21.15.67:0/2416718749 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55a54d96df10 0x55a54d970390 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c0a64845470f890b server_cookie=a67021cd7db8a5d1 in_seq=0 out_seq=0 2024-03-28T16:09:35.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.900+0000 7f5c8cc00640 10 client.16158 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:09:35.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.902+0000 7f5c8cc00640 1 -- 172.21.15.67:0/2416718749 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55a54d8ca790 con 0x55a54d96df10 2024-03-28T16:09:35.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.902+0000 7f5c8cc00640 10 client.16158 handle_command_reply: tid=0 2024-03-28T16:09:35.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.902+0000 7f5ca85ef740 1 client.16158 shutdown 2024-03-28T16:09:35.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.902+0000 7f5ca85ef740 1 -- 172.21.15.67:0/2416718749 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a54d95dbb0 msgr2=0x55a54d9695a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:35.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.902+0000 7f5ca85ef740 1 --2- 172.21.15.67:0/2416718749 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a54d95dbb0 0x55a54d9695a0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f5c98097ff0 tx=0x7f5c98086110 comp rx=0 tx=0).stop 2024-03-28T16:09:35.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.903+0000 7f5ca85ef740 1 -- 172.21.15.67:0/2416718749 shutdown_connections 2024-03-28T16:09:35.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.903+0000 7f5ca85ef740 1 --2- 172.21.15.67:0/2416718749 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55a54d96df10 0x55a54d970390 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:35.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.903+0000 7f5ca85ef740 1 --2- 172.21.15.67:0/2416718749 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a54d95dbb0 0x55a54d9695a0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:35.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.903+0000 7f5ca85ef740 1 --2- 172.21.15.67:0/2416718749 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a54d958a40 0x55a54d962560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:35.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.903+0000 7f5ca85ef740 1 --2- 172.21.15.67:0/2416718749 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a54d958050 0x55a54d962020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:35.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.903+0000 7f5c8cc00640 0 client.16158 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:09:35.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.903+0000 7f5ca85ef740 1 -- 172.21.15.67:0/2416718749 >> 172.21.15.67:0/2416718749 conn(0x55a54d920200 msgr2=0x55a54d952bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:35.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.903+0000 7f5ca85ef740 1 -- 172.21.15.67:0/2416718749 shutdown_connections 2024-03-28T16:09:35.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.903+0000 7f5ca85ef740 1 -- 172.21.15.67:0/2416718749 wait complete. 2024-03-28T16:09:35.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.903+0000 7f5ca85ef740 20 client.16158 trim_cache size 0 max 16384 2024-03-28T16:09:35.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.921+0000 7f5ca85ef740 1 Processor -- start 2024-03-28T16:09:35.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.921+0000 7f5ca85ef740 1 -- start start 2024-03-28T16:09:35.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.921+0000 7f5ca85ef740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a54d958050 0x55a54d94e920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:35.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.921+0000 7f5ca85ef740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a54d958a40 0x55a54d969560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:35.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.921+0000 7f5ca85ef740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a54d95dbb0 0x55a54d956c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:35.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.921+0000 7f5ca85ef740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca01acee0 con 0x55a54d958050 2024-03-28T16:09:35.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.921+0000 7f5ca85ef740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca01af4a0 con 0x55a54d958a40 2024-03-28T16:09:35.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.921+0000 7f5ca85ef740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca01ad4e0 con 0x55a54d95dbb0 2024-03-28T16:09:35.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.922+0000 7f5c9ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a54d958050 0x55a54d94e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:35.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.922+0000 7f5c9ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a54d958050 0x55a54d94e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33784/0 (socket says 172.21.15.67:33784) 2024-03-28T16:09:35.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.922+0000 7f5c9ea00640 1 -- 172.21.15.67:0/3181952285 learned_addr learned my addr 172.21.15.67:0/3181952285 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:09:35.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.922+0000 7f5c9f400640 1 --2- 172.21.15.67:0/3181952285 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a54d95dbb0 0x55a54d956c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:35.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.922+0000 7f5c9e000640 1 --2- 172.21.15.67:0/3181952285 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a54d958a40 0x55a54d969560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:35.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.922+0000 7f5c9ea00640 1 -- 172.21.15.67:0/3181952285 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a54d95dbb0 msgr2=0x55a54d956c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:09:35.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.922+0000 7f5c9ea00640 1 --2- 172.21.15.67:0/3181952285 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a54d95dbb0 0x55a54d956c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:35.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.922+0000 7f5c9ea00640 1 -- 172.21.15.67:0/3181952285 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a54d958a40 msgr2=0x55a54d969560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:09:35.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.922+0000 7f5c9ea00640 1 --2- 172.21.15.67:0/3181952285 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a54d958a40 0x55a54d969560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:35.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.922+0000 7f5c9ea00640 1 -- 172.21.15.67:0/3181952285 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a54d5910f0 con 0x55a54d958050 2024-03-28T16:09:35.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.922+0000 7f5c9ea00640 1 --2- 172.21.15.67:0/3181952285 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a54d958050 0x55a54d94e920 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f5c90009900 tx=0x7f5c90044000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f622eb1b047abd5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:35.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.923+0000 7f5c8f400640 1 -- 172.21.15.67:0/3181952285 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c90004390 con 0x55a54d958050 2024-03-28T16:09:35.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.923+0000 7f5c8f400640 1 -- 172.21.15.67:0/3181952285 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5c900054f0 con 0x55a54d958050 2024-03-28T16:09:35.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.923+0000 7f5c8f400640 1 -- 172.21.15.67:0/3181952285 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c90004d10 con 0x55a54d958050 2024-03-28T16:09:35.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.923+0000 7f5ca85ef740 1 -- 172.21.15.67:0/3181952285 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a54d958050 msgr2=0x55a54d94e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:35.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.923+0000 7f5ca85ef740 1 --2- 172.21.15.67:0/3181952285 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a54d958050 0x55a54d94e920 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f5c90009900 tx=0x7f5c90044000 comp rx=0 tx=0).stop 2024-03-28T16:09:35.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.923+0000 7f5ca85ef740 1 -- 172.21.15.67:0/3181952285 shutdown_connections 2024-03-28T16:09:35.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.923+0000 7f5ca85ef740 1 --2- 172.21.15.67:0/3181952285 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a54d95dbb0 0x55a54d956c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:35.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.923+0000 7f5ca85ef740 1 --2- 172.21.15.67:0/3181952285 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a54d958a40 0x55a54d969560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:35.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.923+0000 7f5ca85ef740 1 --2- 172.21.15.67:0/3181952285 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a54d958050 0x55a54d94e920 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:35.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.923+0000 7f5ca85ef740 1 -- 172.21.15.67:0/3181952285 >> 172.21.15.67:0/3181952285 conn(0x55a54d920200 msgr2=0x55a54d952bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:35.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.924+0000 7f5ca85ef740 1 -- 172.21.15.67:0/3181952285 shutdown_connections 2024-03-28T16:09:35.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.924+0000 7f5ca85ef740 1 -- 172.21.15.67:0/3181952285 wait complete. 2024-03-28T16:09:35.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.924+0000 7f5ca85ef740 1 -- start start 2024-03-28T16:09:35.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.924+0000 7f5ca85ef740 1 Processor -- start 2024-03-28T16:09:35.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.925+0000 7f5ca85ef740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a54d958050 0x55a54d968e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:35.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.925+0000 7f5ca85ef740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a54d958a40 0x55a54d969370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:35.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.925+0000 7f5ca85ef740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a54d95dbb0 0x55a54d967440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:35.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.925+0000 7f5ca85ef740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca01adb20 con 0x55a54d958050 2024-03-28T16:09:35.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.925+0000 7f5ca85ef740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca01aee40 con 0x55a54d958a40 2024-03-28T16:09:35.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.925+0000 7f5ca85ef740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca01ae180 con 0x55a54d95dbb0 2024-03-28T16:09:35.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.925+0000 7f5ca85ef740 10 client.0 _ll_register_callbacks cb 0x7f5c980057e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:09:35.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.925+0000 7f5c9ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a54d958050 0x55a54d968e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:35.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.925+0000 7f5c9ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a54d958050 0x55a54d968e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33798/0 (socket says 172.21.15.67:33798) 2024-03-28T16:09:35.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.925+0000 7f5c9ea00640 1 -- 172.21.15.67:0/3064377652 learned_addr learned my addr 172.21.15.67:0/3064377652 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:09:35.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.925+0000 7f5c9e000640 1 --2- 172.21.15.67:0/3064377652 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a54d958a40 0x55a54d969370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:35.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.925+0000 7f5c9f400640 1 --2- 172.21.15.67:0/3064377652 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a54d95dbb0 0x55a54d967440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:35.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.925+0000 7f5c9ea00640 1 -- 172.21.15.67:0/3064377652 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a54d95dbb0 msgr2=0x55a54d967440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:09:35.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.925+0000 7f5c9ea00640 1 --2- 172.21.15.67:0/3064377652 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a54d95dbb0 0x55a54d967440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:35.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.925+0000 7f5c9ea00640 1 -- 172.21.15.67:0/3064377652 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a54d958a40 msgr2=0x55a54d969370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:09:35.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.925+0000 7f5c9ea00640 1 --2- 172.21.15.67:0/3064377652 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a54d958a40 0x55a54d969370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:35.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.925+0000 7f5c9ea00640 1 -- 172.21.15.67:0/3064377652 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a54d967b80 con 0x55a54d958050 2024-03-28T16:09:35.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.925+0000 7f5c9e000640 1 --2- 172.21.15.67:0/3064377652 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a54d958a40 0x55a54d969370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:09:35.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.926+0000 7f5c9ea00640 1 --2- 172.21.15.67:0/3064377652 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a54d958050 0x55a54d968e30 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f5c9002c840 tx=0x7f5c900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c1011bb75b8c332 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:35.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.926+0000 7f5c8cc00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:09:35.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.926+0000 7f5c8cc00640 1 -- 172.21.15.67:0/3064377652 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c90057070 con 0x55a54d958050 2024-03-28T16:09:35.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.926+0000 7f5c8cc00640 1 -- 172.21.15.67:0/3064377652 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f5c90005250 con 0x55a54d958050 2024-03-28T16:09:35.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.926+0000 7f5ca85ef740 1 -- 172.21.15.67:0/3064377652 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5c900054f0 con 0x55a54d958050 2024-03-28T16:09:35.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.926+0000 7f5c8cc00640 1 -- 172.21.15.67:0/3064377652 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5c900044d0 con 0x55a54d958050 2024-03-28T16:09:35.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.926+0000 7f5ca85ef740 10 client.16164 fetch_fsmap learned FSMap version 19 2024-03-28T16:09:35.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.926+0000 7f5ca85ef740 1 -- 172.21.15.67:0/3064377652 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55a54d9ca270 con 0x55a54d958050 2024-03-28T16:09:35.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.927+0000 7f5c8cc00640 1 -- 172.21.15.67:0/3064377652 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5c900054f0 con 0x55a54d958050 2024-03-28T16:09:35.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.927+0000 7f5c8cc00640 1 -- 172.21.15.67:0/3064377652 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f5c90034440 con 0x55a54d958050 2024-03-28T16:09:35.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.927+0000 7f5ca85ef740 10 client.16164 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:09:35.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.927+0000 7f5ca85ef740 10 client.16164 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:09:35.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.927+0000 7f5ca85ef740 20 client.16164 populate_metadata read hostname 'smithi067' 2024-03-28T16:09:35.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.927+0000 7f5ca85ef740 1 --2- 172.21.15.67:0/3064377652 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55a54d9cac00 0x55a54d9cd020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:35.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.927+0000 7f5ca85ef740 4 client.16164 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T16:09:35.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.927+0000 7f5ca85ef740 1 -- 172.21.15.67:0/3064377652 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55a54d8c4ae0 con 0x55a54d9cac00 2024-03-28T16:09:35.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.928+0000 7f5c9e000640 1 --2- 172.21.15.67:0/3064377652 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55a54d9cac00 0x55a54d9cd020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:35.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.929+0000 7f5c9e000640 1 --2- 172.21.15.67:0/3064377652 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55a54d9cac00 0x55a54d9cd020 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=233713f6dd23902c server_cookie=264adb03dfb6c416 in_seq=0 out_seq=0 2024-03-28T16:09:35.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.929+0000 7f5c8cc00640 10 client.16164 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:09:35.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.929+0000 7f5c8cc00640 1 -- 172.21.15.67:0/3064377652 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55a54d8c4ae0 con 0x55a54d9cac00 2024-03-28T16:09:35.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.929+0000 7f5c8cc00640 10 client.16164 handle_command_reply: tid=0 2024-03-28T16:09:35.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.929+0000 7f5ca85ef740 1 client.16164 shutdown 2024-03-28T16:09:35.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.929+0000 7f5ca85ef740 1 -- 172.21.15.67:0/3064377652 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a54d958050 msgr2=0x55a54d968e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:35.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.929+0000 7f5ca85ef740 1 --2- 172.21.15.67:0/3064377652 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a54d958050 0x55a54d968e30 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f5c9002c840 tx=0x7f5c900046c0 comp rx=0 tx=0).stop 2024-03-28T16:09:35.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.929+0000 7f5ca85ef740 1 -- 172.21.15.67:0/3064377652 shutdown_connections 2024-03-28T16:09:35.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.929+0000 7f5ca85ef740 1 --2- 172.21.15.67:0/3064377652 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55a54d9cac00 0x55a54d9cd020 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:35.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.929+0000 7f5ca85ef740 1 --2- 172.21.15.67:0/3064377652 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a54d95dbb0 0x55a54d967440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:35.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.929+0000 7f5ca85ef740 1 --2- 172.21.15.67:0/3064377652 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a54d958a40 0x55a54d969370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:35.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.929+0000 7f5ca85ef740 1 --2- 172.21.15.67:0/3064377652 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a54d958050 0x55a54d968e30 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:35.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.929+0000 7f5c8cc00640 0 client.16164 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:09:35.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.929+0000 7f5ca85ef740 1 -- 172.21.15.67:0/3064377652 >> 172.21.15.67:0/3064377652 conn(0x55a54d920200 msgr2=0x55a54d956c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:35.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.930+0000 7f5ca85ef740 1 -- 172.21.15.67:0/3064377652 shutdown_connections 2024-03-28T16:09:35.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.930+0000 7f5ca85ef740 1 -- 172.21.15.67:0/3064377652 wait complete. 2024-03-28T16:09:35.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.930+0000 7f5ca85ef740 20 client.16164 trim_cache size 0 max 16384 2024-03-28T16:09:35.933 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-28T16:09:35.933 INFO:teuthology.orchestra.run.smithi067.stdout: "status": "no active scrubs running", 2024-03-28T16:09:35.933 INFO:teuthology.orchestra.run.smithi067.stdout: "scrubs": {} 2024-03-28T16:09:35.933 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-28T16:09:35.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.930+0000 7f5c9fe00640 1 -- 172.21.15.67:0/3823342404 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f5c840a2800 msgr2=0x7f5c840a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:35.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.930+0000 7f5c9fe00640 1 --2- 172.21.15.67:0/3823342404 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f5c840a2800 0x7f5c840a4cc0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f5c9002c240 tx=0x7f5c90005a20 comp rx=0 tx=0).stop 2024-03-28T16:09:35.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.930+0000 7f5c9fe00640 1 -- 172.21.15.67:0/3823342404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ca0109770 msgr2=0x7f5ca01a5390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:35.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.930+0000 7f5c9fe00640 1 --2- 172.21.15.67:0/3823342404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ca0109770 0x7f5ca01a5390 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f5c9800eaa0 tx=0x7f5c9800ef60 comp rx=0 tx=0).stop 2024-03-28T16:09:35.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.931+0000 7f5c9fe00640 1 -- 172.21.15.67:0/3823342404 shutdown_connections 2024-03-28T16:09:35.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.931+0000 7f5c9fe00640 1 --2- 172.21.15.67:0/3823342404 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f5c840a2800 0x7f5c840a4cc0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:35.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.931+0000 7f5c9fe00640 1 --2- 172.21.15.67:0/3823342404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ca0109770 0x7f5ca01a5390 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:35.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.931+0000 7f5c9fe00640 1 --2- 172.21.15.67:0/3823342404 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5ca0108db0 0x7f5ca019e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:35.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.931+0000 7f5c9fe00640 1 --2- 172.21.15.67:0/3823342404 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ca0107bc0 0x7f5ca019de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:35.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.931+0000 7f5c9fe00640 1 -- 172.21.15.67:0/3823342404 >> 172.21.15.67:0/3823342404 conn(0x7f5ca0076b70 msgr2=0x7f5ca010c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:35.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.931+0000 7f5c9fe00640 1 -- 172.21.15.67:0/3823342404 shutdown_connections 2024-03-28T16:09:35.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:35.931+0000 7f5c9fe00640 1 -- 172.21.15.67:0/3823342404 wait complete. 2024-03-28T16:09:35.950 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T16:09:35.951 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T16:09:36.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.170+0000 7f5994000640 1 -- 172.21.15.67:0/58060739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f598c105490 msgr2=0x7f598c10bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:36.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.170+0000 7f5994000640 1 --2- 172.21.15.67:0/58060739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f598c105490 0x7f598c10bd20 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f5984009ca0 tx=0x7f598402c100 comp rx=0 tx=0).stop 2024-03-28T16:09:36.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.170+0000 7f5994000640 1 -- 172.21.15.67:0/58060739 shutdown_connections 2024-03-28T16:09:36.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.170+0000 7f5994000640 1 --2- 172.21.15.67:0/58060739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f598c105490 0x7f598c10bd20 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:36.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.170+0000 7f5994000640 1 --2- 172.21.15.67:0/58060739 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f598c104ad0 0x7f598c104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:36.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.170+0000 7f5994000640 1 --2- 172.21.15.67:0/58060739 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f598c1038e0 0x7f598c103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:36.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.170+0000 7f5994000640 1 -- 172.21.15.67:0/58060739 >> 172.21.15.67:0/58060739 conn(0x7f598c0fee50 msgr2=0x7f598c101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:36.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.170+0000 7f5994000640 1 -- 172.21.15.67:0/58060739 shutdown_connections 2024-03-28T16:09:36.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.170+0000 7f5994000640 1 -- 172.21.15.67:0/58060739 wait complete. 2024-03-28T16:09:36.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.171+0000 7f5994000640 1 Processor -- start 2024-03-28T16:09:36.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.171+0000 7f5994000640 1 -- start start 2024-03-28T16:09:36.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.172+0000 7f5994000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f598c1038e0 0x7f598c199ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:36.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.172+0000 7f5994000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f598c104ad0 0x7f598c199ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:36.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.172+0000 7f5994000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f598c105490 0x7f598c1a1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:36.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.172+0000 7f5994000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f598c10f410 con 0x7f598c104ad0 2024-03-28T16:09:36.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.172+0000 7f5994000640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f598c10f290 con 0x7f598c1038e0 2024-03-28T16:09:36.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.172+0000 7f5994000640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f598c10f590 con 0x7f598c105490 2024-03-28T16:09:36.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.172+0000 7f5990c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f598c104ad0 0x7f598c199ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:36.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.172+0000 7f5990c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f598c104ad0 0x7f598c199ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33810/0 (socket says 172.21.15.67:33810) 2024-03-28T16:09:36.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.172+0000 7f5990c00640 1 -- 172.21.15.67:0/1237181537 learned_addr learned my addr 172.21.15.67:0/1237181537 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:09:36.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.172+0000 7f5991600640 1 --2- 172.21.15.67:0/1237181537 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f598c1038e0 0x7f598c199ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:36.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.172+0000 7f5990c00640 1 -- 172.21.15.67:0/1237181537 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f598c105490 msgr2=0x7f598c1a1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:36.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.172+0000 7f5990c00640 1 --2- 172.21.15.67:0/1237181537 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f598c105490 0x7f598c1a1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:36.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.172+0000 7f5990c00640 1 -- 172.21.15.67:0/1237181537 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f598c1038e0 msgr2=0x7f598c199ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:36.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.172+0000 7f5990c00640 1 --2- 172.21.15.67:0/1237181537 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f598c1038e0 0x7f598c199ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:36.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.172+0000 7f5990c00640 1 -- 172.21.15.67:0/1237181537 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f598c1a16d0 con 0x7f598c104ad0 2024-03-28T16:09:36.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.173+0000 7f5990c00640 1 --2- 172.21.15.67:0/1237181537 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f598c104ad0 0x7f598c199ff0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f597800d950 tx=0x7f597800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:36.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.173+0000 7f598a000640 1 -- 172.21.15.67:0/1237181537 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5978017030 con 0x7f598c104ad0 2024-03-28T16:09:36.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.173+0000 7f598a000640 1 -- 172.21.15.67:0/1237181537 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5978004020 con 0x7f598c104ad0 2024-03-28T16:09:36.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.173+0000 7f5994000640 1 -- 172.21.15.67:0/1237181537 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f598c1a19c0 con 0x7f598c104ad0 2024-03-28T16:09:36.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.173+0000 7f598a000640 1 -- 172.21.15.67:0/1237181537 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59780047a0 con 0x7f598c104ad0 2024-03-28T16:09:36.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.173+0000 7f5994000640 1 -- 172.21.15.67:0/1237181537 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f598c1a1f10 con 0x7f598c104ad0 2024-03-28T16:09:36.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.175+0000 7f5994000640 1 -- 172.21.15.67:0/1237181537 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f598c10ac80 con 0x7f598c104ad0 2024-03-28T16:09:36.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.176+0000 7f598a000640 1 -- 172.21.15.67:0/1237181537 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f59780041d0 con 0x7f598c104ad0 2024-03-28T16:09:36.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.178+0000 7f598a000640 1 --2- 172.21.15.67:0/1237181537 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f59740a2800 0x7f59740a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:36.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.178+0000 7f598a000640 1 -- 172.21.15.67:0/1237181537 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f59780ce640 con 0x7f598c104ad0 2024-03-28T16:09:36.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.178+0000 7f5991600640 1 --2- 172.21.15.67:0/1237181537 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f59740a2800 0x7f59740a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:36.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.179+0000 7f5991600640 1 --2- 172.21.15.67:0/1237181537 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f59740a2800 0x7f59740a4cc0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f5980005fd0 tx=0x7f5980005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:36.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.183+0000 7f598a000640 1 -- 172.21.15.67:0/1237181537 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f597809b690 con 0x7f598c104ad0 2024-03-28T16:09:36.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.427+0000 7f5994000640 1 -- 172.21.15.67:0/1237181537 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f598c0622a0 con 0x7f598c104ad0 2024-03-28T16:09:36.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.428+0000 7f598a000640 1 -- 172.21.15.67:0/1237181537 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11341 (secure 0 0 0) 0x7f597809af90 con 0x7f598c104ad0 2024-03-28T16:09:36.429 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:09:36.429 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2008297659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2008297659},{"type":"v1","addr":"172.21.15.67:6835","nonce":2008297659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14532,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/320295745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":320295745},{"type":"v1","addr":"172.21.15.67:6837","nonce":320295745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6839/691821273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":691821273},{"type":"v1","addr":"172.21.15.67:6839","nonce":691821273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6841/3211254875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6840","nonce":3211254875},{"type":"v1","addr":"172.21.15.67:6841","nonce":3211254875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2129665423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2129665423},{"type":"v1","addr":"172.21.15.183:6833","nonce":2129665423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24496,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/1589784425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":1589784425},{"type":"v1","addr":"172.21.15.183:6835","nonce":1589784425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24508,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/2185863835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":2185863835},{"type":"v1","addr":"172.21.15.183:6837","nonce":2185863835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24520,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6839/1242814398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":1242814398},{"type":"v1","addr":"172.21.15.183:6839","nonce":1242814398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":34409,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1008335521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1008335521},{"type":"v1","addr":"172.21.15.150:6833","nonce":1008335521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":34445,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3269935695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3269935695},{"type":"v1","addr":"172.21.15.150:6835","nonce":3269935695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T16:01:07.696824+0000","modified":"2024-03-28T16:01:11.997773+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":34508},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34475":{"gid":34475,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/1112886277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":1112886277},{"type":"v1","addr":"172.21.15.150:6837","nonce":1112886277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_34508":{"gid":34508,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6839/1211011614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":1211011614},{"type":"v1","addr":"172.21.15.150:6839","nonce":1211011614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34508,"qdb_cluster":[34508]},"id":1}]} 2024-03-28T16:09:36.429 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 19 2024-03-28T16:09:36.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.431+0000 7f5994000640 1 -- 172.21.15.67:0/1237181537 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f59740a2800 msgr2=0x7f59740a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:36.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.431+0000 7f5994000640 1 --2- 172.21.15.67:0/1237181537 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f59740a2800 0x7f59740a4cc0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f5980005fd0 tx=0x7f5980005850 comp rx=0 tx=0).stop 2024-03-28T16:09:36.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.431+0000 7f5994000640 1 -- 172.21.15.67:0/1237181537 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f598c104ad0 msgr2=0x7f598c199ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:36.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.431+0000 7f5994000640 1 --2- 172.21.15.67:0/1237181537 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f598c104ad0 0x7f598c199ff0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f597800d950 tx=0x7f597800de10 comp rx=0 tx=0).stop 2024-03-28T16:09:36.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.431+0000 7f5994000640 1 -- 172.21.15.67:0/1237181537 shutdown_connections 2024-03-28T16:09:36.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.431+0000 7f5994000640 1 --2- 172.21.15.67:0/1237181537 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f59740a2800 0x7f59740a4cc0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:36.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.431+0000 7f5994000640 1 --2- 172.21.15.67:0/1237181537 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f598c105490 0x7f598c1a1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:36.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.431+0000 7f5994000640 1 --2- 172.21.15.67:0/1237181537 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f598c104ad0 0x7f598c199ff0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:36.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.431+0000 7f5994000640 1 --2- 172.21.15.67:0/1237181537 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f598c1038e0 0x7f598c199ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:36.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.431+0000 7f5994000640 1 -- 172.21.15.67:0/1237181537 >> 172.21.15.67:0/1237181537 conn(0x7f598c0fee50 msgr2=0x7f598c107e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:36.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.431+0000 7f5994000640 1 -- 172.21.15.67:0/1237181537 shutdown_connections 2024-03-28T16:09:36.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.431+0000 7f5994000640 1 -- 172.21.15.67:0/1237181537 wait complete. 2024-03-28T16:09:36.451 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T16:09:36.451 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T16:09:36.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.674+0000 7f7e14e00640 1 -- 172.21.15.67:0/2260839366 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e10101580 msgr2=0x7f7e101019a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:36.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.674+0000 7f7e14e00640 1 --2- 172.21.15.67:0/2260839366 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e10101580 0x7f7e101019a0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f7e00009ca0 tx=0x7f7e0002c0f0 comp rx=0 tx=0).stop 2024-03-28T16:09:36.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.674+0000 7f7e14e00640 1 -- 172.21.15.67:0/2260839366 shutdown_connections 2024-03-28T16:09:36.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.674+0000 7f7e14e00640 1 --2- 172.21.15.67:0/2260839366 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7e10103130 0x7f7e101099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:36.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.674+0000 7f7e14e00640 1 --2- 172.21.15.67:0/2260839366 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e10102770 0x7f7e10102bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:36.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.674+0000 7f7e14e00640 1 --2- 172.21.15.67:0/2260839366 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e10101580 0x7f7e101019a0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:36.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.674+0000 7f7e14e00640 1 -- 172.21.15.67:0/2260839366 >> 172.21.15.67:0/2260839366 conn(0x7f7e100fcb10 msgr2=0x7f7e100fef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:36.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.674+0000 7f7e14e00640 1 -- 172.21.15.67:0/2260839366 shutdown_connections 2024-03-28T16:09:36.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.674+0000 7f7e14e00640 1 -- 172.21.15.67:0/2260839366 wait complete. 2024-03-28T16:09:36.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.675+0000 7f7e14e00640 1 Processor -- start 2024-03-28T16:09:36.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.675+0000 7f7e14e00640 1 -- start start 2024-03-28T16:09:36.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.675+0000 7f7e14e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7e10101580 0x7f7e101977a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:36.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.675+0000 7f7e14e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e10102770 0x7f7e10197ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:36.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.675+0000 7f7e14e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e10103130 0x7f7e1019ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:36.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.675+0000 7f7e14e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1010d0a0 con 0x7f7e10102770 2024-03-28T16:09:36.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.675+0000 7f7e14e00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1010cf20 con 0x7f7e10103130 2024-03-28T16:09:36.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.675+0000 7f7e14e00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1010d220 con 0x7f7e10101580 2024-03-28T16:09:36.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.676+0000 7f7e0ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e10102770 0x7f7e10197ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:36.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.676+0000 7f7e0ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e10102770 0x7f7e10197ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33840/0 (socket says 172.21.15.67:33840) 2024-03-28T16:09:36.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.676+0000 7f7e0ea00640 1 -- 172.21.15.67:0/2766833816 learned_addr learned my addr 172.21.15.67:0/2766833816 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:09:36.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.676+0000 7f7e0f400640 1 --2- 172.21.15.67:0/2766833816 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7e10101580 0x7f7e101977a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:36.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.676+0000 7f7e0fe00640 1 --2- 172.21.15.67:0/2766833816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e10103130 0x7f7e1019ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:36.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.676+0000 7f7e0ea00640 1 -- 172.21.15.67:0/2766833816 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7e10101580 msgr2=0x7f7e101977a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:36.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.676+0000 7f7e0ea00640 1 --2- 172.21.15.67:0/2766833816 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7e10101580 0x7f7e101977a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:36.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.676+0000 7f7e0ea00640 1 -- 172.21.15.67:0/2766833816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e10103130 msgr2=0x7f7e1019ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:36.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.676+0000 7f7e0ea00640 1 --2- 172.21.15.67:0/2766833816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e10103130 0x7f7e1019ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:36.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.676+0000 7f7e0ea00640 1 -- 172.21.15.67:0/2766833816 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e1019f3c0 con 0x7f7e10102770 2024-03-28T16:09:36.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.676+0000 7f7e0ea00640 1 --2- 172.21.15.67:0/2766833816 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e10102770 0x7f7e10197ce0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f7df800d950 tx=0x7f7df800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:36.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.677+0000 7f7e07e00640 1 -- 172.21.15.67:0/2766833816 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7df8017030 con 0x7f7e10102770 2024-03-28T16:09:36.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.677+0000 7f7e07e00640 1 -- 172.21.15.67:0/2766833816 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7df8004020 con 0x7f7e10102770 2024-03-28T16:09:36.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.677+0000 7f7e14e00640 1 -- 172.21.15.67:0/2766833816 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e1019f6b0 con 0x7f7e10102770 2024-03-28T16:09:36.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.677+0000 7f7e07e00640 1 -- 172.21.15.67:0/2766833816 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7df80047a0 con 0x7f7e10102770 2024-03-28T16:09:36.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.677+0000 7f7e14e00640 1 -- 172.21.15.67:0/2766833816 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e1019fc00 con 0x7f7e10102770 2024-03-28T16:09:36.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.680+0000 7f7e191a3740 1 Processor -- start 2024-03-28T16:09:36.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.680+0000 7f7e191a3740 1 -- start start 2024-03-28T16:09:36.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.680+0000 7f7e07e00640 1 -- 172.21.15.67:0/2766833816 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f7df80041d0 con 0x7f7e10102770 2024-03-28T16:09:36.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.680+0000 7f7e191a3740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b3f4769e00 0x55b3f476a220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:36.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.680+0000 7f7e191a3740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b3f476afd0 0x55b3f476b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:36.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.681+0000 7f7e191a3740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b3f476b990 0x55b3f4772220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:36.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.681+0000 7f7e191a3740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3f46ca2c0 con 0x55b3f476b990 2024-03-28T16:09:36.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.681+0000 7f7e191a3740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3f4672c90 con 0x55b3f4769e00 2024-03-28T16:09:36.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.681+0000 7f7e191a3740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3f46c6360 con 0x55b3f476afd0 2024-03-28T16:09:36.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.681+0000 7f7e0ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b3f476b990 0x55b3f4772220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:36.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.681+0000 7f7e0fe00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b3f4769e00 0x55b3f476a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:36.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.681+0000 7f7e0ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b3f476b990 0x55b3f4772220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33854/0 (socket says 172.21.15.67:33854) 2024-03-28T16:09:36.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.681+0000 7f7e0ea00640 1 -- 172.21.15.67:0/2042911712 learned_addr learned my addr 172.21.15.67:0/2042911712 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:09:36.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.681+0000 7f7e0f400640 1 --2- 172.21.15.67:0/2042911712 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b3f476afd0 0x55b3f476b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:36.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.681+0000 7f7e0ea00640 1 -- 172.21.15.67:0/2042911712 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b3f476afd0 msgr2=0x55b3f476b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:09:36.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.681+0000 7f7e0ea00640 1 --2- 172.21.15.67:0/2042911712 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b3f476afd0 0x55b3f476b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:36.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.681+0000 7f7e0ea00640 1 -- 172.21.15.67:0/2042911712 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b3f4769e00 msgr2=0x55b3f476a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:09:36.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.681+0000 7f7e0ea00640 1 --2- 172.21.15.67:0/2042911712 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b3f4769e00 0x55b3f476a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:36.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.681+0000 7f7e0ea00640 1 -- 172.21.15.67:0/2042911712 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b3f4697080 con 0x55b3f476b990 2024-03-28T16:09:36.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.682+0000 7f7e0ea00640 1 --2- 172.21.15.67:0/2042911712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b3f476b990 0x55b3f4772220 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f7df80d45d0 tx=0x7f7df80d4a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e4b1b009088668d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:36.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.682+0000 7f7e07e00640 1 --2- 172.21.15.67:0/2766833816 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7df40a2800 0x7f7df40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:36.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.682+0000 7f7e04c00640 1 -- 172.21.15.67:0/2042911712 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7df80e1070 con 0x55b3f476b990 2024-03-28T16:09:36.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.682+0000 7f7e04c00640 1 -- 172.21.15.67:0/2042911712 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7df809b6b0 con 0x55b3f476b990 2024-03-28T16:09:36.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.682+0000 7f7e04c00640 1 -- 172.21.15.67:0/2042911712 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7df80826f0 con 0x55b3f476b990 2024-03-28T16:09:36.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.682+0000 7f7e07e00640 1 -- 172.21.15.67:0/2766833816 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f7df80ce660 con 0x7f7e10102770 2024-03-28T16:09:36.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.682+0000 7f7e0fe00640 1 --2- 172.21.15.67:0/2766833816 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7df40a2800 0x7f7df40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:36.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.683+0000 7f7e191a3740 1 -- 172.21.15.67:0/2042911712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b3f476b990 msgr2=0x55b3f4772220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:36.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.683+0000 7f7e191a3740 1 --2- 172.21.15.67:0/2042911712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b3f476b990 0x55b3f4772220 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f7df80d45d0 tx=0x7f7df80d4a90 comp rx=0 tx=0).stop 2024-03-28T16:09:36.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.683+0000 7f7e191a3740 1 -- 172.21.15.67:0/2042911712 shutdown_connections 2024-03-28T16:09:36.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.683+0000 7f7e191a3740 1 --2- 172.21.15.67:0/2042911712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b3f476b990 0x55b3f4772220 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:36.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.683+0000 7f7e191a3740 1 --2- 172.21.15.67:0/2042911712 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b3f476afd0 0x55b3f476b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:36.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.683+0000 7f7e191a3740 1 --2- 172.21.15.67:0/2042911712 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b3f4769e00 0x55b3f476a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:36.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.683+0000 7f7e191a3740 1 -- 172.21.15.67:0/2042911712 >> 172.21.15.67:0/2042911712 conn(0x55b3f47651f0 msgr2=0x55b3f47655f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:36.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.683+0000 7f7e0fe00640 1 --2- 172.21.15.67:0/2766833816 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7df40a2800 0x7f7df40a4cc0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x55b3f476aed0 tx=0x7f7e0800b420 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:36.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.683+0000 7f7e191a3740 1 -- 172.21.15.67:0/2042911712 shutdown_connections 2024-03-28T16:09:36.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.683+0000 7f7e191a3740 1 -- 172.21.15.67:0/2042911712 wait complete. 2024-03-28T16:09:36.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.684+0000 7f7e191a3740 1 -- start start 2024-03-28T16:09:36.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.684+0000 7f7e191a3740 1 Processor -- start 2024-03-28T16:09:36.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.684+0000 7f7e191a3740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b3f4769e00 0x55b3f4773fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:36.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.684+0000 7f7e191a3740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b3f476afd0 0x55b3f4774520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:36.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.684+0000 7f7e191a3740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b3f476b990 0x55b3f477b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:36.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.684+0000 7f7e191a3740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3f47729a0 con 0x55b3f4769e00 2024-03-28T16:09:36.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.684+0000 7f7e191a3740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3f46860d0 con 0x55b3f476afd0 2024-03-28T16:09:36.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.684+0000 7f7e191a3740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3f4772b20 con 0x55b3f476b990 2024-03-28T16:09:36.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.684+0000 7f7e191a3740 10 client.0 _ll_register_callbacks cb 0x55b3f468ae80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:09:36.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.685+0000 7f7e0f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b3f4769e00 0x55b3f4773fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:36.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.685+0000 7f7e0f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b3f4769e00 0x55b3f4773fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33868/0 (socket says 172.21.15.67:33868) 2024-03-28T16:09:36.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.685+0000 7f7e0f400640 1 -- 172.21.15.67:0/3974039372 learned_addr learned my addr 172.21.15.67:0/3974039372 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:09:36.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.685+0000 7f7e0ea00640 1 --2- 172.21.15.67:0/3974039372 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b3f476afd0 0x55b3f4774520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:36.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.685+0000 7f7e0fe00640 1 --2- 172.21.15.67:0/3974039372 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b3f476b990 0x55b3f477b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:36.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.685+0000 7f7e0f400640 1 -- 172.21.15.67:0/3974039372 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b3f476b990 msgr2=0x55b3f477b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:09:36.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.685+0000 7f7e0f400640 1 --2- 172.21.15.67:0/3974039372 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b3f476b990 0x55b3f477b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:36.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.685+0000 7f7e0f400640 1 -- 172.21.15.67:0/3974039372 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b3f476afd0 msgr2=0x55b3f4774520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:09:36.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.685+0000 7f7e0f400640 1 --2- 172.21.15.67:0/3974039372 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b3f476afd0 0x55b3f4774520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:36.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.685+0000 7f7e0f400640 1 -- 172.21.15.67:0/3974039372 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b3f43de3f0 con 0x55b3f4769e00 2024-03-28T16:09:36.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.685+0000 7f7e0ea00640 1 --2- 172.21.15.67:0/3974039372 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b3f476afd0 0x55b3f4774520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:09:36.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.685+0000 7f7e0fe00640 1 --2- 172.21.15.67:0/3974039372 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b3f476b990 0x55b3f477b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:09:36.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.685+0000 7f7e0f400640 1 --2- 172.21.15.67:0/3974039372 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b3f4769e00 0x55b3f4773fe0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f7e00032380 tx=0x7f7e00032670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c8105a5032fd20e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:36.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.685+0000 7f7dfd600640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:09:36.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.686+0000 7f7dfd600640 1 -- 172.21.15.67:0/3974039372 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e00043070 con 0x55b3f4769e00 2024-03-28T16:09:36.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.686+0000 7f7dfd600640 1 -- 172.21.15.67:0/3974039372 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f7e00003970 con 0x55b3f4769e00 2024-03-28T16:09:36.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.686+0000 7f7e191a3740 1 -- 172.21.15.67:0/3974039372 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7df809b6b0 con 0x55b3f4769e00 2024-03-28T16:09:36.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.686+0000 7f7dfd600640 1 -- 172.21.15.67:0/3974039372 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7e0004f050 con 0x55b3f4769e00 2024-03-28T16:09:36.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.686+0000 7f7e191a3740 10 client.16182 fetch_fsmap learned FSMap version 19 2024-03-28T16:09:36.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.686+0000 7f7e191a3740 1 -- 172.21.15.67:0/3974039372 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55b3f44a87d0 con 0x55b3f4769e00 2024-03-28T16:09:36.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.686+0000 7f7dfd600640 1 -- 172.21.15.67:0/3974039372 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7df809b6b0 con 0x55b3f4769e00 2024-03-28T16:09:36.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.686+0000 7f7dfd600640 1 -- 172.21.15.67:0/3974039372 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f7e00034440 con 0x55b3f4769e00 2024-03-28T16:09:36.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.686+0000 7f7e191a3740 10 client.16182 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:09:36.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.687+0000 7f7e191a3740 10 client.16182 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:09:36.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.687+0000 7f7e191a3740 20 client.16182 populate_metadata read hostname 'smithi067' 2024-03-28T16:09:36.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.687+0000 7f7e191a3740 1 --2- 172.21.15.67:0/3974039372 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55b3f477fa00 0x55b3f4781e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:36.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.687+0000 7f7e191a3740 4 client.16182 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:09:36.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.687+0000 7f7e191a3740 1 -- 172.21.15.67:0/3974039372 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b3f46d7800 con 0x55b3f477fa00 2024-03-28T16:09:36.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.687+0000 7f7e0ea00640 1 --2- 172.21.15.67:0/3974039372 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55b3f477fa00 0x55b3f4781e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:36.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.688+0000 7f7e0ea00640 1 --2- 172.21.15.67:0/3974039372 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55b3f477fa00 0x55b3f4781e70 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f1a350166b347400 server_cookie=9dc143c9ff9063c0 in_seq=0 out_seq=0 2024-03-28T16:09:36.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.688+0000 7f7dfd600640 10 client.16182 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:09:36.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.690+0000 7f7dfd600640 1 -- 172.21.15.67:0/3974039372 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55b3f46d7800 con 0x55b3f477fa00 2024-03-28T16:09:36.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.690+0000 7f7dfd600640 10 client.16182 handle_command_reply: tid=0 2024-03-28T16:09:36.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.690+0000 7f7e191a3740 1 client.16182 shutdown 2024-03-28T16:09:36.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.690+0000 7f7e191a3740 1 -- 172.21.15.67:0/3974039372 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b3f4769e00 msgr2=0x55b3f4773fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:36.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.690+0000 7f7e191a3740 1 --2- 172.21.15.67:0/3974039372 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b3f4769e00 0x55b3f4773fe0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f7e00032380 tx=0x7f7e00032670 comp rx=0 tx=0).stop 2024-03-28T16:09:36.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.690+0000 7f7e191a3740 1 -- 172.21.15.67:0/3974039372 shutdown_connections 2024-03-28T16:09:36.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.690+0000 7f7e191a3740 1 --2- 172.21.15.67:0/3974039372 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55b3f477fa00 0x55b3f4781e70 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:36.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.690+0000 7f7e191a3740 1 --2- 172.21.15.67:0/3974039372 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b3f476b990 0x55b3f477b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:36.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.690+0000 7f7e191a3740 1 --2- 172.21.15.67:0/3974039372 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b3f476afd0 0x55b3f4774520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:36.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.690+0000 7f7e191a3740 1 --2- 172.21.15.67:0/3974039372 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b3f4769e00 0x55b3f4773fe0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:36.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.690+0000 7f7e191a3740 1 -- 172.21.15.67:0/3974039372 >> 172.21.15.67:0/3974039372 conn(0x55b3f47651f0 msgr2=0x55b3f4768450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:36.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.690+0000 7f7dfd600640 0 client.16182 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:09:36.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.691+0000 7f7e191a3740 1 -- 172.21.15.67:0/3974039372 shutdown_connections 2024-03-28T16:09:36.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.691+0000 7f7e191a3740 1 -- 172.21.15.67:0/3974039372 wait complete. 2024-03-28T16:09:36.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.691+0000 7f7e191a3740 20 client.16182 trim_cache size 0 max 16384 2024-03-28T16:09:36.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.704+0000 7f7e191a3740 1 Processor -- start 2024-03-28T16:09:36.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.704+0000 7f7e191a3740 1 -- start start 2024-03-28T16:09:36.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.704+0000 7f7e191a3740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b3f4769e00 0x55b3f47732b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:36.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.705+0000 7f7e191a3740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b3f476afd0 0x55b3f477b520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:36.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.705+0000 7f7e191a3740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b3f476b990 0x55b3f47737f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:36.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.705+0000 7f7e191a3740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e101a46a0 con 0x55b3f476b990 2024-03-28T16:09:36.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.705+0000 7f7e191a3740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e101a6c60 con 0x55b3f4769e00 2024-03-28T16:09:36.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.705+0000 7f7e191a3740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e101a4ca0 con 0x55b3f476afd0 2024-03-28T16:09:36.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.705+0000 7f7e0fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b3f476b990 0x55b3f47737f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:36.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.705+0000 7f7e0fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b3f476b990 0x55b3f47737f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33874/0 (socket says 172.21.15.67:33874) 2024-03-28T16:09:36.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.705+0000 7f7e0fe00640 1 -- 172.21.15.67:0/1635797734 learned_addr learned my addr 172.21.15.67:0/1635797734 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:09:36.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.705+0000 7f7e0f400640 1 --2- 172.21.15.67:0/1635797734 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b3f4769e00 0x55b3f47732b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:36.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.705+0000 7f7e0ea00640 1 --2- 172.21.15.67:0/1635797734 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b3f476afd0 0x55b3f477b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:36.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.705+0000 7f7e0fe00640 1 -- 172.21.15.67:0/1635797734 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b3f476afd0 msgr2=0x55b3f477b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:09:36.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.705+0000 7f7e0fe00640 1 --2- 172.21.15.67:0/1635797734 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b3f476afd0 0x55b3f477b520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:36.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.705+0000 7f7e0fe00640 1 -- 172.21.15.67:0/1635797734 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b3f4769e00 msgr2=0x55b3f47732b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:09:36.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.705+0000 7f7e0fe00640 1 --2- 172.21.15.67:0/1635797734 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b3f4769e00 0x55b3f47732b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:36.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.705+0000 7f7e0fe00640 1 -- 172.21.15.67:0/1635797734 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b3f475c9c0 con 0x55b3f476b990 2024-03-28T16:09:36.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.706+0000 7f7e0fe00640 1 --2- 172.21.15.67:0/1635797734 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b3f476b990 0x55b3f47737f0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f7e080104d0 tx=0x7f7e08015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6ba0fb887d7c40c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:36.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.706+0000 7f7dffe00640 1 -- 172.21.15.67:0/1635797734 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e08028670 con 0x55b3f476b990 2024-03-28T16:09:36.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.706+0000 7f7dffe00640 1 -- 172.21.15.67:0/1635797734 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7e08023070 con 0x55b3f476b990 2024-03-28T16:09:36.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.706+0000 7f7dffe00640 1 -- 172.21.15.67:0/1635797734 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e0801c6d0 con 0x55b3f476b990 2024-03-28T16:09:36.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.706+0000 7f7e191a3740 1 -- 172.21.15.67:0/1635797734 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b3f476b990 msgr2=0x55b3f47737f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:36.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.706+0000 7f7e191a3740 1 --2- 172.21.15.67:0/1635797734 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b3f476b990 0x55b3f47737f0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f7e080104d0 tx=0x7f7e08015000 comp rx=0 tx=0).stop 2024-03-28T16:09:36.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.707+0000 7f7e191a3740 1 -- 172.21.15.67:0/1635797734 shutdown_connections 2024-03-28T16:09:36.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.707+0000 7f7e191a3740 1 --2- 172.21.15.67:0/1635797734 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b3f476b990 0x55b3f47737f0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:36.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.707+0000 7f7e191a3740 1 --2- 172.21.15.67:0/1635797734 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b3f476afd0 0x55b3f477b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:36.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.707+0000 7f7e191a3740 1 --2- 172.21.15.67:0/1635797734 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b3f4769e00 0x55b3f47732b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:36.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.707+0000 7f7e191a3740 1 -- 172.21.15.67:0/1635797734 >> 172.21.15.67:0/1635797734 conn(0x55b3f47651f0 msgr2=0x55b3f476ea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:36.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.707+0000 7f7e191a3740 1 -- 172.21.15.67:0/1635797734 shutdown_connections 2024-03-28T16:09:36.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.707+0000 7f7e191a3740 1 -- 172.21.15.67:0/1635797734 wait complete. 2024-03-28T16:09:36.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.707+0000 7f7e191a3740 1 -- start start 2024-03-28T16:09:36.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.707+0000 7f7e191a3740 1 Processor -- start 2024-03-28T16:09:36.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.708+0000 7f7e191a3740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b3f4769e00 0x55b3f4779cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:36.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.708+0000 7f7e191a3740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b3f476afd0 0x55b3f477a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:36.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.708+0000 7f7e191a3740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b3f476b990 0x55b3f477a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:36.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.708+0000 7f7e191a3740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e101a52e0 con 0x55b3f4769e00 2024-03-28T16:09:36.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.708+0000 7f7e191a3740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e101a6600 con 0x55b3f476afd0 2024-03-28T16:09:36.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.708+0000 7f7e191a3740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e101a5940 con 0x55b3f476b990 2024-03-28T16:09:36.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.708+0000 7f7e191a3740 10 client.0 _ll_register_callbacks cb 0x7f7e00037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T16:09:36.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.708+0000 7f7e0f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b3f4769e00 0x55b3f4779cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:36.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.708+0000 7f7e0f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b3f4769e00 0x55b3f4779cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33888/0 (socket says 172.21.15.67:33888) 2024-03-28T16:09:36.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.708+0000 7f7e0f400640 1 -- 172.21.15.67:0/2243105426 learned_addr learned my addr 172.21.15.67:0/2243105426 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:09:36.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.708+0000 7f7e0f400640 1 -- 172.21.15.67:0/2243105426 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b3f476b990 msgr2=0x55b3f477a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:09:36.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.708+0000 7f7e0f400640 1 --2- 172.21.15.67:0/2243105426 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b3f476b990 0x55b3f477a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:36.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.708+0000 7f7e0ea00640 1 --2- 172.21.15.67:0/2243105426 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b3f476afd0 0x55b3f477a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:36.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.708+0000 7f7e0f400640 1 -- 172.21.15.67:0/2243105426 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b3f476afd0 msgr2=0x55b3f477a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:09:36.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.708+0000 7f7e0f400640 1 --2- 172.21.15.67:0/2243105426 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b3f476afd0 0x55b3f477a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:36.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.708+0000 7f7e0f400640 1 -- 172.21.15.67:0/2243105426 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b3f47db300 con 0x55b3f4769e00 2024-03-28T16:09:36.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.708+0000 7f7e0ea00640 1 --2- 172.21.15.67:0/2243105426 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b3f476afd0 0x55b3f477a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:09:36.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.709+0000 7f7e0f400640 1 --2- 172.21.15.67:0/2243105426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b3f4769e00 0x55b3f4779cb0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f7e0002c700 tx=0x7f7e00034c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3abf3c61572e183e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:36.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.709+0000 7f7dfd600640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-03-28T16:09:36.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.709+0000 7f7dfd600640 1 -- 172.21.15.67:0/2243105426 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e00058060 con 0x55b3f4769e00 2024-03-28T16:09:36.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.709+0000 7f7dfd600640 1 -- 172.21.15.67:0/2243105426 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f7e000328e0 con 0x55b3f4769e00 2024-03-28T16:09:36.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.709+0000 7f7e191a3740 1 -- 172.21.15.67:0/2243105426 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7e08023070 con 0x55b3f4769e00 2024-03-28T16:09:36.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.709+0000 7f7dfd600640 1 -- 172.21.15.67:0/2243105426 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7e00043070 con 0x55b3f4769e00 2024-03-28T16:09:36.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.709+0000 7f7e191a3740 10 client.16188 fetch_fsmap learned FSMap version 19 2024-03-28T16:09:36.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.709+0000 7f7e191a3740 1 -- 172.21.15.67:0/2243105426 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55b3f47dc5a0 con 0x55b3f4769e00 2024-03-28T16:09:36.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.709+0000 7f7dfd600640 1 -- 172.21.15.67:0/2243105426 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7e08023070 con 0x55b3f4769e00 2024-03-28T16:09:36.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.710+0000 7f7dfd600640 1 -- 172.21.15.67:0/2243105426 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f7e00004640 con 0x55b3f4769e00 2024-03-28T16:09:36.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.710+0000 7f7e191a3740 10 client.16188 fetch_fsmap finished waiting for FSMap version 19 2024-03-28T16:09:36.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.710+0000 7f7e191a3740 10 client.16188 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T16:09:36.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.710+0000 7f7e191a3740 20 client.16188 populate_metadata read hostname 'smithi067' 2024-03-28T16:09:36.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.710+0000 7f7e191a3740 1 --2- 172.21.15.67:0/2243105426 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55b3f47dce70 0x55b3f47df290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:36.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.710+0000 7f7e191a3740 4 client.16188 mds_command: new command op to 34508 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T16:09:36.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.710+0000 7f7e191a3740 1 -- 172.21.15.67:0/2243105426 --> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55b3f46d1b50 con 0x55b3f47dce70 2024-03-28T16:09:36.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.710+0000 7f7e0ea00640 1 --2- 172.21.15.67:0/2243105426 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55b3f47dce70 0x55b3f47df290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:36.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.712+0000 7f7e0ea00640 1 --2- 172.21.15.67:0/2243105426 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55b3f47dce70 0x55b3f47df290 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=47daf199ceecb32c server_cookie=d17a5d5c2b2877f4 in_seq=0 out_seq=0 2024-03-28T16:09:36.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.712+0000 7f7dfd600640 10 client.16188 ms_handle_connect on v2:172.21.15.150:6838/1211011614 2024-03-28T16:09:36.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.712+0000 7f7dfd600640 1 -- 172.21.15.67:0/2243105426 <== mds.0 v2:172.21.15.150:6838/1211011614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55b3f46d1b50 con 0x55b3f47dce70 2024-03-28T16:09:36.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.712+0000 7f7dfd600640 10 client.16188 handle_command_reply: tid=0 2024-03-28T16:09:36.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.712+0000 7f7e191a3740 1 client.16188 shutdown 2024-03-28T16:09:36.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.712+0000 7f7e191a3740 1 -- 172.21.15.67:0/2243105426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b3f4769e00 msgr2=0x55b3f4779cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:36.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.712+0000 7f7e191a3740 1 --2- 172.21.15.67:0/2243105426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b3f4769e00 0x55b3f4779cb0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f7e0002c700 tx=0x7f7e00034c30 comp rx=0 tx=0).stop 2024-03-28T16:09:36.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.712+0000 7f7e191a3740 1 -- 172.21.15.67:0/2243105426 shutdown_connections 2024-03-28T16:09:36.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.712+0000 7f7e191a3740 1 --2- 172.21.15.67:0/2243105426 >> [v2:172.21.15.150:6838/1211011614,v1:172.21.15.150:6839/1211011614] conn(0x55b3f47dce70 0x55b3f47df290 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:36.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.712+0000 7f7e191a3740 1 --2- 172.21.15.67:0/2243105426 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b3f476b990 0x55b3f477a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:36.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.712+0000 7f7e191a3740 1 --2- 172.21.15.67:0/2243105426 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b3f476afd0 0x55b3f477a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:36.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.712+0000 7f7e191a3740 1 --2- 172.21.15.67:0/2243105426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b3f4769e00 0x55b3f4779cb0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:36.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.712+0000 7f7e191a3740 1 -- 172.21.15.67:0/2243105426 >> 172.21.15.67:0/2243105426 conn(0x55b3f47651f0 msgr2=0x55b3f4786730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:36.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.712+0000 7f7dfd600640 0 client.16188 ms_handle_reset on v2:172.21.15.150:6838/1211011614 2024-03-28T16:09:36.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.712+0000 7f7e191a3740 1 -- 172.21.15.67:0/2243105426 shutdown_connections 2024-03-28T16:09:36.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.713+0000 7f7e191a3740 1 -- 172.21.15.67:0/2243105426 wait complete. 2024-03-28T16:09:36.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.713+0000 7f7e191a3740 20 client.16188 trim_cache size 0 max 16384 2024-03-28T16:09:36.717 INFO:teuthology.orchestra.run.smithi067.stdout:[] 2024-03-28T16:09:36.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.713+0000 7f7e14e00640 1 -- 172.21.15.67:0/2766833816 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7df40a2800 msgr2=0x7f7df40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:36.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.713+0000 7f7e14e00640 1 --2- 172.21.15.67:0/2766833816 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7df40a2800 0x7f7df40a4cc0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x55b3f476aed0 tx=0x7f7e0800b420 comp rx=0 tx=0).stop 2024-03-28T16:09:36.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.713+0000 7f7e14e00640 1 -- 172.21.15.67:0/2766833816 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e10102770 msgr2=0x7f7e10197ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:36.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.713+0000 7f7e14e00640 1 --2- 172.21.15.67:0/2766833816 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e10102770 0x7f7e10197ce0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f7df800d950 tx=0x7f7df800de10 comp rx=0 tx=0).stop 2024-03-28T16:09:36.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.713+0000 7f7e14e00640 1 -- 172.21.15.67:0/2766833816 shutdown_connections 2024-03-28T16:09:36.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.713+0000 7f7e14e00640 1 --2- 172.21.15.67:0/2766833816 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7df40a2800 0x7f7df40a4cc0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:36.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.713+0000 7f7e14e00640 1 --2- 172.21.15.67:0/2766833816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e10103130 0x7f7e1019ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:36.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.713+0000 7f7e14e00640 1 --2- 172.21.15.67:0/2766833816 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e10102770 0x7f7e10197ce0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:36.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.713+0000 7f7e14e00640 1 --2- 172.21.15.67:0/2766833816 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7e10101580 0x7f7e101977a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:36.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.713+0000 7f7e14e00640 1 -- 172.21.15.67:0/2766833816 >> 172.21.15.67:0/2766833816 conn(0x7f7e100fcb10 msgr2=0x7f7e10105b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:36.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.713+0000 7f7e14e00640 1 -- 172.21.15.67:0/2766833816 shutdown_connections 2024-03-28T16:09:36.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:36.714+0000 7f7e14e00640 1 -- 172.21.15.67:0/2766833816 wait complete. 2024-03-28T16:09:37.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:36 smithi067 ceph-mon[31707]: pgmap v620: 97 pgs: 97 active+clean; 980 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:09:37.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:36 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/1237181537' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:09:37.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:36 smithi183 ceph-mon[35580]: pgmap v620: 97 pgs: 97 active+clean; 980 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:09:37.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:36 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/1237181537' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:09:37.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:36 smithi150 ceph-mon[34825]: pgmap v620: 97 pgs: 97 active+clean; 980 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:09:37.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:36 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/1237181537' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:09:37.732 INFO:tasks.fwd_scrub.fs.[cephfs]:end scrubbing fs: cephfs 2024-03-28T16:09:37.732 INFO:tasks.fwd_scrub:done joining 2024-03-28T16:09:37.733 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-03-28T16:09:37.743 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T16:09:37.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:37.975+0000 7f0685a00640 1 -- 172.21.15.67:0/904469409 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0680101f70 msgr2=0x7f0680104360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:37.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:37.975+0000 7f0685a00640 1 --2- 172.21.15.67:0/904469409 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0680101f70 0x7f0680104360 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f0670009900 tx=0x7f067002bc40 comp rx=0 tx=0).stop 2024-03-28T16:09:37.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:37.975+0000 7f0685a00640 1 -- 172.21.15.67:0/904469409 shutdown_connections 2024-03-28T16:09:37.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:37.975+0000 7f0685a00640 1 --2- 172.21.15.67:0/904469409 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06801071d0 0x7f0680109620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:37.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:37.975+0000 7f0685a00640 1 --2- 172.21.15.67:0/904469409 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f06801048a0 0x7f0680106c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:37.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:37.975+0000 7f0685a00640 1 --2- 172.21.15.67:0/904469409 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0680101f70 0x7f0680104360 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:37.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:37.975+0000 7f0685a00640 1 -- 172.21.15.67:0/904469409 >> 172.21.15.67:0/904469409 conn(0x7f06800fb960 msgr2=0x7f06800fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:37.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:37.975+0000 7f0685a00640 1 -- 172.21.15.67:0/904469409 shutdown_connections 2024-03-28T16:09:37.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:37.976+0000 7f0685a00640 1 -- 172.21.15.67:0/904469409 wait complete. 2024-03-28T16:09:37.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:37.976+0000 7f0685a00640 1 Processor -- start 2024-03-28T16:09:37.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:37.977+0000 7f0685a00640 1 -- start start 2024-03-28T16:09:37.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:37.977+0000 7f0685a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0680101f70 0x7f06801998b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:37.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:37.977+0000 7f0685a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06801048a0 0x7f0680199df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:37.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:37.977+0000 7f0685a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06801071d0 0x7f06801a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:37.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:37.977+0000 7f0685a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f068010d080 con 0x7f06801071d0 2024-03-28T16:09:37.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:37.977+0000 7f0685a00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f068010cf00 con 0x7f06801048a0 2024-03-28T16:09:37.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:37.977+0000 7f0685a00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f068010d200 con 0x7f0680101f70 2024-03-28T16:09:37.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:37.977+0000 7f0685000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06801071d0 0x7f06801a0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:37.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:37.977+0000 7f0685000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06801071d0 0x7f06801a0e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33908/0 (socket says 172.21.15.67:33908) 2024-03-28T16:09:37.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:37.977+0000 7f0685000640 1 -- 172.21.15.67:0/3488633950 learned_addr learned my addr 172.21.15.67:0/3488633950 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:09:37.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:37.978+0000 7f067fe00640 1 --2- 172.21.15.67:0/3488633950 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0680101f70 0x7f06801998b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:37.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:37.978+0000 7f067f400640 1 --2- 172.21.15.67:0/3488633950 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06801048a0 0x7f0680199df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:37.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:37.978+0000 7f0685000640 1 -- 172.21.15.67:0/3488633950 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0680101f70 msgr2=0x7f06801998b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:37.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:37.978+0000 7f0685000640 1 --2- 172.21.15.67:0/3488633950 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0680101f70 0x7f06801998b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:37.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:37.978+0000 7f0685000640 1 -- 172.21.15.67:0/3488633950 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06801048a0 msgr2=0x7f0680199df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:37.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:37.978+0000 7f0685000640 1 --2- 172.21.15.67:0/3488633950 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06801048a0 0x7f0680199df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:37.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:37.978+0000 7f0685000640 1 -- 172.21.15.67:0/3488633950 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06801a14d0 con 0x7f06801071d0 2024-03-28T16:09:37.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:37.978+0000 7f067f400640 1 --2- 172.21.15.67:0/3488633950 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06801048a0 0x7f0680199df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T16:09:37.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:37.978+0000 7f0685000640 1 --2- 172.21.15.67:0/3488633950 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06801071d0 0x7f06801a0e60 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f067800cce0 tx=0x7f067800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:37.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:37.979+0000 7f067cc00640 1 -- 172.21.15.67:0/3488633950 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f067800f450 con 0x7f06801071d0 2024-03-28T16:09:37.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:37.979+0000 7f0685a00640 1 -- 172.21.15.67:0/3488633950 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06801a17c0 con 0x7f06801071d0 2024-03-28T16:09:37.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:37.979+0000 7f067cc00640 1 -- 172.21.15.67:0/3488633950 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f067800faa0 con 0x7f06801071d0 2024-03-28T16:09:37.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:37.979+0000 7f067cc00640 1 -- 172.21.15.67:0/3488633950 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0678018b70 con 0x7f06801071d0 2024-03-28T16:09:37.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:37.979+0000 7f0685a00640 1 -- 172.21.15.67:0/3488633950 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06801a1cc0 con 0x7f06801071d0 2024-03-28T16:09:37.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:37.980+0000 7f0685a00640 1 -- 172.21.15.67:0/3488633950 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0680108d20 con 0x7f06801071d0 2024-03-28T16:09:37.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:37.982+0000 7f067cc00640 1 -- 172.21.15.67:0/3488633950 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f0678018d10 con 0x7f06801071d0 2024-03-28T16:09:37.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:37.983+0000 7f067cc00640 1 --2- 172.21.15.67:0/3488633950 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f06640a2800 0x7f06640a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:37.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:37.983+0000 7f067cc00640 1 -- 172.21.15.67:0/3488633950 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f06780cef00 con 0x7f06801071d0 2024-03-28T16:09:37.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:37.984+0000 7f067fe00640 1 --2- 172.21.15.67:0/3488633950 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f06640a2800 0x7f06640a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:37.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:37.984+0000 7f067fe00640 1 --2- 172.21.15.67:0/3488633950 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f06640a2800 0x7f06640a4cc0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f0670006010 tx=0x7f0670000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:37.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:37.988+0000 7f067cc00640 1 -- 172.21.15.67:0/3488633950 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0678014030 con 0x7f06801071d0 2024-03-28T16:09:38.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.194+0000 7f0685a00640 1 -- 172.21.15.67:0/3488633950 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f06800622a0 con 0x7f06801071d0 2024-03-28T16:09:38.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.199+0000 7f067cc00640 1 -- 172.21.15.67:0/3488633950 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) v1 ==== 74+0+22673 (secure 0 0 0) 0x7f067809bf50 con 0x7f06801071d0 2024-03-28T16:09:38.200 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:09:38.200 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":92,"fsid":"b5f8225a-ed1a-11ee-95d0-87774f69a715","created":"2024-03-28T15:50:21.471924+0000","modified":"2024-03-28T16:01:40.643215+0000","last_up_change":"2024-03-28T16:00:01.369123+0000","last_in_change":"2024-03-28T15:59:37.515483+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T15:54:55.588007+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T16:01:05.719801+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T16:01:07.206303+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7c1791b6-9c0e-4512-a327-cad166bb5581","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6803","nonce":1894681701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6805","nonce":1894681701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6809","nonce":1894681701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6807","nonce":1894681701}]},"public_addr":"172.21.15.67:6803/1894681701","cluster_addr":"172.21.15.67:6805/1894681701","heartbeat_back_addr":"172.21.15.67:6809/1894681701","heartbeat_front_addr":"172.21.15.67:6807/1894681701","state":["exists","up"]},{"osd":1,"uuid":"0cdda7f3-d00c-4436-bcc1-49fb8072b2e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6801","nonce":1734465762}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6803","nonce":1734465762}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6807","nonce":1734465762}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6805","nonce":1734465762}]},"public_addr":"172.21.15.150:6801/1734465762","cluster_addr":"172.21.15.150:6803/1734465762","heartbeat_back_addr":"172.21.15.150:6807/1734465762","heartbeat_front_addr":"172.21.15.150:6805/1734465762","state":["exists","up"]},{"osd":2,"uuid":"a55b9a02-c3fd-4061-95c0-53a24a17ade0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6801","nonce":3398095931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6803","nonce":3398095931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6807","nonce":3398095931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6805","nonce":3398095931}]},"public_addr":"172.21.15.183:6801/3398095931","cluster_addr":"172.21.15.183:6803/3398095931","heartbeat_back_addr":"172.21.15.183:6807/3398095931","heartbeat_front_addr":"172.21.15.183:6805/3398095931","state":["exists","up"]},{"osd":3,"uuid":"d34940c0-5ce9-4b8b-aa36-4872414a043f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6811","nonce":1933541875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6813","nonce":1933541875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6817","nonce":1933541875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6815","nonce":1933541875}]},"public_addr":"172.21.15.67:6811/1933541875","cluster_addr":"172.21.15.67:6813/1933541875","heartbeat_back_addr":"172.21.15.67:6817/1933541875","heartbeat_front_addr":"172.21.15.67:6815/1933541875","state":["exists","up"]},{"osd":4,"uuid":"5cc3e260-83a3-4a4d-aa6a-34a3b12b2b5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6809","nonce":3775269993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6811","nonce":3775269993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6815","nonce":3775269993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6813","nonce":3775269993}]},"public_addr":"172.21.15.150:6809/3775269993","cluster_addr":"172.21.15.150:6811/3775269993","heartbeat_back_addr":"172.21.15.150:6815/3775269993","heartbeat_front_addr":"172.21.15.150:6813/3775269993","state":["exists","up"]},{"osd":5,"uuid":"a2542b21-d6ac-413e-8b9b-4a6548b20840","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6809","nonce":164284137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6811","nonce":164284137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6815","nonce":164284137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6813","nonce":164284137}]},"public_addr":"172.21.15.183:6809/164284137","cluster_addr":"172.21.15.183:6811/164284137","heartbeat_back_addr":"172.21.15.183:6815/164284137","heartbeat_front_addr":"172.21.15.183:6813/164284137","state":["exists","up"]},{"osd":6,"uuid":"8905a2d3-0f1b-4853-a218-22da2635150a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6819","nonce":3565528013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6821","nonce":3565528013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6825","nonce":3565528013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6823","nonce":3565528013}]},"public_addr":"172.21.15.67:6819/3565528013","cluster_addr":"172.21.15.67:6821/3565528013","heartbeat_back_addr":"172.21.15.67:6825/3565528013","heartbeat_front_addr":"172.21.15.67:6823/3565528013","state":["exists","up"]},{"osd":7,"uuid":"b275055b-b451-4180-91b1-e6e61a828c48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6817","nonce":3735115340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6819","nonce":3735115340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6823","nonce":3735115340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6821","nonce":3735115340}]},"public_addr":"172.21.15.150:6817/3735115340","cluster_addr":"172.21.15.150:6819/3735115340","heartbeat_back_addr":"172.21.15.150:6823/3735115340","heartbeat_front_addr":"172.21.15.150:6821/3735115340","state":["exists","up"]},{"osd":8,"uuid":"b6abf7ad-eb34-4f60-a4eb-348a92d7195b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6817","nonce":1025400095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6819","nonce":1025400095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6823","nonce":1025400095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6821","nonce":1025400095}]},"public_addr":"172.21.15.183:6817/1025400095","cluster_addr":"172.21.15.183:6819/1025400095","heartbeat_back_addr":"172.21.15.183:6823/1025400095","heartbeat_front_addr":"172.21.15.183:6821/1025400095","state":["exists","up"]},{"osd":9,"uuid":"28a78e3a-0657-40aa-816f-16a6fadadcbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6827","nonce":76885409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6829","nonce":76885409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6833","nonce":76885409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6831","nonce":76885409}]},"public_addr":"172.21.15.67:6827/76885409","cluster_addr":"172.21.15.67:6829/76885409","heartbeat_back_addr":"172.21.15.67:6833/76885409","heartbeat_front_addr":"172.21.15.67:6831/76885409","state":["exists","up"]},{"osd":10,"uuid":"d287907e-424f-4b49-a06e-4613c6de0474","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6825","nonce":3489982080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6827","nonce":3489982080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6831","nonce":3489982080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6829","nonce":3489982080}]},"public_addr":"172.21.15.150:6825/3489982080","cluster_addr":"172.21.15.150:6827/3489982080","heartbeat_back_addr":"172.21.15.150:6831/3489982080","heartbeat_front_addr":"172.21.15.150:6829/3489982080","state":["exists","up"]},{"osd":11,"uuid":"a3c29c47-a228-4f42-a23c-fbdb0c093699","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":85,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6825","nonce":1892639849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6826","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6827","nonce":1892639849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6831","nonce":1892639849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6829","nonce":1892639849}]},"public_addr":"172.21.15.183:6825/1892639849","cluster_addr":"172.21.15.183:6827/1892639849","heartbeat_back_addr":"172.21.15.183:6831/1892639849","heartbeat_front_addr":"172.21.15.183:6829/1892639849","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:53:45.646211+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:54:18.956590+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:54:52.161646+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:55:27.502822+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:56:01.162932+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:56:34.674452+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:57:09.298679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:57:43.230406+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:58:17.003821+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:58:51.085863+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:59:25.665065+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T16:00:00.263457+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":1}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6801/1631001968":"2024-03-29T15:51:38.875334+0000","172.21.15.67:6800/1631001968":"2024-03-29T15:51:38.875334+0000","172.21.15.67:0/1533056310":"2024-03-29T15:51:38.875334+0000","172.21.15.67:6801/3580491071":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/2030412458":"2024-03-29T15:51:03.859740+0000","172.21.15.67:6800/3580491071":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/2905307533":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/3763759988":"2024-03-29T15:51:38.875334+0000","172.21.15.67:0/4089399880":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/4239108661":"2024-03-29T15:51:38.875334+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T16:09:38.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.201+0000 7f0685a00640 1 -- 172.21.15.67:0/3488633950 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f06640a2800 msgr2=0x7f06640a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:38.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.201+0000 7f0685a00640 1 --2- 172.21.15.67:0/3488633950 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f06640a2800 0x7f06640a4cc0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f0670006010 tx=0x7f0670000bc0 comp rx=0 tx=0).stop 2024-03-28T16:09:38.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.201+0000 7f0685a00640 1 -- 172.21.15.67:0/3488633950 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06801071d0 msgr2=0x7f06801a0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:38.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.201+0000 7f0685a00640 1 --2- 172.21.15.67:0/3488633950 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06801071d0 0x7f06801a0e60 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f067800cce0 tx=0x7f067800a620 comp rx=0 tx=0).stop 2024-03-28T16:09:38.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.202+0000 7f0685a00640 1 -- 172.21.15.67:0/3488633950 shutdown_connections 2024-03-28T16:09:38.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.202+0000 7f0685a00640 1 --2- 172.21.15.67:0/3488633950 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f06640a2800 0x7f06640a4cc0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:38.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.202+0000 7f0685a00640 1 --2- 172.21.15.67:0/3488633950 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06801071d0 0x7f06801a0e60 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:38.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.202+0000 7f0685a00640 1 --2- 172.21.15.67:0/3488633950 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06801048a0 0x7f0680199df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:38.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.202+0000 7f0685a00640 1 --2- 172.21.15.67:0/3488633950 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0680101f70 0x7f06801998b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:38.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.202+0000 7f0685a00640 1 -- 172.21.15.67:0/3488633950 >> 172.21.15.67:0/3488633950 conn(0x7f06800fb960 msgr2=0x7f0680101610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:38.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.202+0000 7f0685a00640 1 -- 172.21.15.67:0/3488633950 shutdown_connections 2024-03-28T16:09:38.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.202+0000 7f0685a00640 1 -- 172.21.15.67:0/3488633950 wait complete. 2024-03-28T16:09:38.223 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T15:54:55.588007+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T16:01:05.719801+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T16:01:07.206303+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T16:09:38.223 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T16:09:38.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.453+0000 7f4a30a00640 1 -- 172.21.15.67:0/2092969274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a2c1037c0 msgr2=0x7f4a2c103be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:38.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.453+0000 7f4a30a00640 1 --2- 172.21.15.67:0/2092969274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a2c1037c0 0x7f4a2c103be0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f4a1c009900 tx=0x7f4a1c02bc30 comp rx=0 tx=0).stop 2024-03-28T16:09:38.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.454+0000 7f4a30a00640 1 -- 172.21.15.67:0/2092969274 shutdown_connections 2024-03-28T16:09:38.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.454+0000 7f4a30a00640 1 --2- 172.21.15.67:0/2092969274 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4a2c105370 0x7f4a2c10bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:38.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.454+0000 7f4a30a00640 1 --2- 172.21.15.67:0/2092969274 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a2c1049b0 0x7f4a2c104e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:38.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.454+0000 7f4a30a00640 1 --2- 172.21.15.67:0/2092969274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a2c1037c0 0x7f4a2c103be0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:38.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.454+0000 7f4a30a00640 1 -- 172.21.15.67:0/2092969274 >> 172.21.15.67:0/2092969274 conn(0x7f4a2c0fed50 msgr2=0x7f4a2c101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:38.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.454+0000 7f4a30a00640 1 -- 172.21.15.67:0/2092969274 shutdown_connections 2024-03-28T16:09:38.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.454+0000 7f4a30a00640 1 -- 172.21.15.67:0/2092969274 wait complete. 2024-03-28T16:09:38.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.455+0000 7f4a30a00640 1 Processor -- start 2024-03-28T16:09:38.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.455+0000 7f4a30a00640 1 -- start start 2024-03-28T16:09:38.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.455+0000 7f4a30a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4a2c1037c0 0x7f4a2c1999e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:38.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.455+0000 7f4a30a00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a2c1049b0 0x7f4a2c199f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:38.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.455+0000 7f4a30a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a2c105370 0x7f4a2c1a0f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:38.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.455+0000 7f4a30a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a2c10f340 con 0x7f4a2c105370 2024-03-28T16:09:38.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.455+0000 7f4a30a00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a2c10f1c0 con 0x7f4a2c1049b0 2024-03-28T16:09:38.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.455+0000 7f4a30a00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a2c10f4c0 con 0x7f4a2c1037c0 2024-03-28T16:09:38.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.456+0000 7f4a2be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a2c105370 0x7f4a2c1a0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:38.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.456+0000 7f4a2be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a2c105370 0x7f4a2c1a0f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33914/0 (socket says 172.21.15.67:33914) 2024-03-28T16:09:38.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.456+0000 7f4a2be00640 1 -- 172.21.15.67:0/2392005020 learned_addr learned my addr 172.21.15.67:0/2392005020 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:09:38.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.456+0000 7f4a2b400640 1 --2- 172.21.15.67:0/2392005020 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4a2c1037c0 0x7f4a2c1999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:38.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.456+0000 7f4a2aa00640 1 --2- 172.21.15.67:0/2392005020 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a2c1049b0 0x7f4a2c199f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:38.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.456+0000 7f4a2be00640 1 -- 172.21.15.67:0/2392005020 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4a2c1037c0 msgr2=0x7f4a2c1999e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:38.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.456+0000 7f4a2be00640 1 --2- 172.21.15.67:0/2392005020 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4a2c1037c0 0x7f4a2c1999e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:38.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.456+0000 7f4a2be00640 1 -- 172.21.15.67:0/2392005020 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a2c1049b0 msgr2=0x7f4a2c199f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:38.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.456+0000 7f4a2be00640 1 --2- 172.21.15.67:0/2392005020 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a2c1049b0 0x7f4a2c199f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:38.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.456+0000 7f4a2be00640 1 -- 172.21.15.67:0/2392005020 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a2c1a1600 con 0x7f4a2c105370 2024-03-28T16:09:38.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.456+0000 7f4a2be00640 1 --2- 172.21.15.67:0/2392005020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a2c105370 0x7f4a2c1a0f90 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f4a2400eb10 tx=0x7f4a2400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:38.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.457+0000 7f4a23e00640 1 -- 172.21.15.67:0/2392005020 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a24004b20 con 0x7f4a2c105370 2024-03-28T16:09:38.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.457+0000 7f4a23e00640 1 -- 172.21.15.67:0/2392005020 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4a24004cc0 con 0x7f4a2c105370 2024-03-28T16:09:38.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.457+0000 7f4a23e00640 1 -- 172.21.15.67:0/2392005020 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a24010690 con 0x7f4a2c105370 2024-03-28T16:09:38.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.457+0000 7f4a30a00640 1 -- 172.21.15.67:0/2392005020 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a2c1a18f0 con 0x7f4a2c105370 2024-03-28T16:09:38.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.457+0000 7f4a30a00640 1 -- 172.21.15.67:0/2392005020 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a2c1a1e90 con 0x7f4a2c105370 2024-03-28T16:09:38.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.458+0000 7f4a30a00640 1 -- 172.21.15.67:0/2392005020 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a2c10ab60 con 0x7f4a2c105370 2024-03-28T16:09:38.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.460+0000 7f4a23e00640 1 -- 172.21.15.67:0/2392005020 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f4a24005020 con 0x7f4a2c105370 2024-03-28T16:09:38.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.461+0000 7f4a23e00640 1 --2- 172.21.15.67:0/2392005020 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f4a100a2880 0x7f4a100a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:38.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.462+0000 7f4a23e00640 1 -- 172.21.15.67:0/2392005020 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f4a240cf100 con 0x7f4a2c105370 2024-03-28T16:09:38.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.462+0000 7f4a2b400640 1 --2- 172.21.15.67:0/2392005020 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f4a100a2880 0x7f4a100a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:38.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.463+0000 7f4a2b400640 1 --2- 172.21.15.67:0/2392005020 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f4a100a2880 0x7f4a100a4d40 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f4a1c02c240 tx=0x7f4a1c02c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:38.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.466+0000 7f4a23e00640 1 -- 172.21.15.67:0/2392005020 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4a2409c150 con 0x7f4a2c105370 2024-03-28T16:09:38.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.670+0000 7f4a30a00640 1 -- 172.21.15.67:0/2392005020 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4a2c0622a0 con 0x7f4a2c105370 2024-03-28T16:09:38.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.671+0000 7f4a23e00640 1 -- 172.21.15.67:0/2392005020 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v92) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4a2409ba50 con 0x7f4a2c105370 2024-03-28T16:09:38.671 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-03-28T16:09:38.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.674+0000 7f4a30a00640 1 -- 172.21.15.67:0/2392005020 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f4a100a2880 msgr2=0x7f4a100a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:38.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.674+0000 7f4a30a00640 1 --2- 172.21.15.67:0/2392005020 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f4a100a2880 0x7f4a100a4d40 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f4a1c02c240 tx=0x7f4a1c02c7e0 comp rx=0 tx=0).stop 2024-03-28T16:09:38.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.674+0000 7f4a30a00640 1 -- 172.21.15.67:0/2392005020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a2c105370 msgr2=0x7f4a2c1a0f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:38.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.674+0000 7f4a30a00640 1 --2- 172.21.15.67:0/2392005020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a2c105370 0x7f4a2c1a0f90 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f4a2400eb10 tx=0x7f4a2400c2d0 comp rx=0 tx=0).stop 2024-03-28T16:09:38.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.674+0000 7f4a30a00640 1 -- 172.21.15.67:0/2392005020 shutdown_connections 2024-03-28T16:09:38.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.674+0000 7f4a30a00640 1 --2- 172.21.15.67:0/2392005020 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f4a100a2880 0x7f4a100a4d40 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:38.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.674+0000 7f4a30a00640 1 --2- 172.21.15.67:0/2392005020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a2c105370 0x7f4a2c1a0f90 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:38.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.674+0000 7f4a30a00640 1 --2- 172.21.15.67:0/2392005020 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a2c1049b0 0x7f4a2c199f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:38.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.674+0000 7f4a30a00640 1 --2- 172.21.15.67:0/2392005020 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4a2c1037c0 0x7f4a2c1999e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:38.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.674+0000 7f4a30a00640 1 -- 172.21.15.67:0/2392005020 >> 172.21.15.67:0/2392005020 conn(0x7f4a2c0fed50 msgr2=0x7f4a2c107d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:38.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.674+0000 7f4a30a00640 1 -- 172.21.15.67:0/2392005020 shutdown_connections 2024-03-28T16:09:38.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.674+0000 7f4a30a00640 1 -- 172.21.15.67:0/2392005020 wait complete. 2024-03-28T16:09:38.696 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T16:09:38.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.926+0000 7f8933e00640 1 -- 172.21.15.67:0/1733556335 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8934101580 msgr2=0x7f89341019a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:38.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.926+0000 7f8933e00640 1 --2- 172.21.15.67:0/1733556335 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8934101580 0x7f89341019a0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f8924009ca0 tx=0x7f892402c100 comp rx=0 tx=0).stop 2024-03-28T16:09:38.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.927+0000 7f8933e00640 1 -- 172.21.15.67:0/1733556335 shutdown_connections 2024-03-28T16:09:38.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.927+0000 7f8933e00640 1 --2- 172.21.15.67:0/1733556335 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8934103130 0x7f89341099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:38.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.927+0000 7f8933e00640 1 --2- 172.21.15.67:0/1733556335 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8934102770 0x7f8934102bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:38.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.927+0000 7f8933e00640 1 --2- 172.21.15.67:0/1733556335 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8934101580 0x7f89341019a0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:38.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.927+0000 7f8933e00640 1 -- 172.21.15.67:0/1733556335 >> 172.21.15.67:0/1733556335 conn(0x7f89340fcb10 msgr2=0x7f89340fef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:38.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.927+0000 7f8933e00640 1 -- 172.21.15.67:0/1733556335 shutdown_connections 2024-03-28T16:09:38.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.927+0000 7f8933e00640 1 -- 172.21.15.67:0/1733556335 wait complete. 2024-03-28T16:09:38.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.928+0000 7f8933e00640 1 Processor -- start 2024-03-28T16:09:38.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.928+0000 7f8933e00640 1 -- start start 2024-03-28T16:09:38.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.928+0000 7f8933e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8934101580 0x7f8934197750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:38.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.928+0000 7f8933e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8934102770 0x7f8934197c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:38.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.928+0000 7f8933e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8934103130 0x7f893419ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:38.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.928+0000 7f8933e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f893410d0c0 con 0x7f8934103130 2024-03-28T16:09:38.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.928+0000 7f8933e00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f893410cf40 con 0x7f8934101580 2024-03-28T16:09:38.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.928+0000 7f8933e00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f893410d240 con 0x7f8934102770 2024-03-28T16:09:38.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.928+0000 7f8933400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8934103130 0x7f893419ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:38.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.929+0000 7f8933400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8934103130 0x7f893419ed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33940/0 (socket says 172.21.15.67:33940) 2024-03-28T16:09:38.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.929+0000 7f8933400640 1 -- 172.21.15.67:0/1587543219 learned_addr learned my addr 172.21.15.67:0/1587543219 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:09:38.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.929+0000 7f892a000640 1 --2- 172.21.15.67:0/1587543219 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8934102770 0x7f8934197c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:38.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.929+0000 7f8932a00640 1 --2- 172.21.15.67:0/1587543219 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8934101580 0x7f8934197750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:38.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.929+0000 7f8933400640 1 -- 172.21.15.67:0/1587543219 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8934102770 msgr2=0x7f8934197c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:38.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.929+0000 7f8933400640 1 --2- 172.21.15.67:0/1587543219 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8934102770 0x7f8934197c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:38.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.929+0000 7f8933400640 1 -- 172.21.15.67:0/1587543219 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8934101580 msgr2=0x7f8934197750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:38.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.929+0000 7f8933400640 1 --2- 172.21.15.67:0/1587543219 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8934101580 0x7f8934197750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:38.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.929+0000 7f8933400640 1 -- 172.21.15.67:0/1587543219 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f893419f370 con 0x7f8934103130 2024-03-28T16:09:38.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.929+0000 7f8932a00640 1 --2- 172.21.15.67:0/1587543219 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8934101580 0x7f8934197750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:09:38.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.929+0000 7f8933400640 1 --2- 172.21.15.67:0/1587543219 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8934103130 0x7f893419ed00 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f892c00c970 tx=0x7f892c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:38.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.930+0000 7f892be00640 1 -- 172.21.15.67:0/1587543219 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f892c007bb0 con 0x7f8934103130 2024-03-28T16:09:38.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.930+0000 7f892be00640 1 -- 172.21.15.67:0/1587543219 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f892c007d50 con 0x7f8934103130 2024-03-28T16:09:38.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.930+0000 7f892be00640 1 -- 172.21.15.67:0/1587543219 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f892c018700 con 0x7f8934103130 2024-03-28T16:09:38.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.930+0000 7f8933e00640 1 -- 172.21.15.67:0/1587543219 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f893419f660 con 0x7f8934103130 2024-03-28T16:09:38.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.930+0000 7f8933e00640 1 -- 172.21.15.67:0/1587543219 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f893419fb60 con 0x7f8934103130 2024-03-28T16:09:38.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.931+0000 7f8933e00640 1 -- 172.21.15.67:0/1587543219 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89341088a0 con 0x7f8934103130 2024-03-28T16:09:38.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.934+0000 7f892be00640 1 -- 172.21.15.67:0/1587543219 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f892c00fa20 con 0x7f8934103130 2024-03-28T16:09:38.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.936+0000 7f892be00640 1 --2- 172.21.15.67:0/1587543219 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f89180a2800 0x7f89180a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:38.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.936+0000 7f892be00640 1 -- 172.21.15.67:0/1587543219 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f892c0cecb0 con 0x7f8934103130 2024-03-28T16:09:38.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.936+0000 7f8932a00640 1 --2- 172.21.15.67:0/1587543219 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f89180a2800 0x7f89180a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:38.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.937+0000 7f8932a00640 1 --2- 172.21.15.67:0/1587543219 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f89180a2800 0x7f89180a4cc0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f8924006010 tx=0x7f8924032340 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:38.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:38.939+0000 7f892be00640 1 -- 172.21.15.67:0/1587543219 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f892c014030 con 0x7f8934103130 2024-03-28T16:09:39.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:38 smithi067 ceph-mon[31707]: pgmap v621: 97 pgs: 97 active+clean; 980 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:09:39.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:38 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/3488633950' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:09:39.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:38 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/2392005020' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T16:09:39.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:38 smithi183 ceph-mon[35580]: pgmap v621: 97 pgs: 97 active+clean; 980 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:09:39.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:38 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/3488633950' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:09:39.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:38 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/2392005020' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T16:09:39.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.141+0000 7f8933e00640 1 -- 172.21.15.67:0/1587543219 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f893402d070 con 0x7f8934103130 2024-03-28T16:09:39.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.142+0000 7f892be00640 1 -- 172.21.15.67:0/1587543219 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v92) v1 ==== 104+0+11 (secure 0 0 0) 0x7f892c09bdf0 con 0x7f8934103130 2024-03-28T16:09:39.142 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-03-28T16:09:39.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.145+0000 7f8933e00640 1 -- 172.21.15.67:0/1587543219 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f89180a2800 msgr2=0x7f89180a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:39.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.145+0000 7f8933e00640 1 --2- 172.21.15.67:0/1587543219 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f89180a2800 0x7f89180a4cc0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f8924006010 tx=0x7f8924032340 comp rx=0 tx=0).stop 2024-03-28T16:09:39.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.145+0000 7f8933e00640 1 -- 172.21.15.67:0/1587543219 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8934103130 msgr2=0x7f893419ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:39.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.145+0000 7f8933e00640 1 --2- 172.21.15.67:0/1587543219 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8934103130 0x7f893419ed00 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f892c00c970 tx=0x7f892c00ce30 comp rx=0 tx=0).stop 2024-03-28T16:09:39.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.145+0000 7f8933e00640 1 -- 172.21.15.67:0/1587543219 shutdown_connections 2024-03-28T16:09:39.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.145+0000 7f8933e00640 1 --2- 172.21.15.67:0/1587543219 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f89180a2800 0x7f89180a4cc0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:39.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.145+0000 7f8933e00640 1 --2- 172.21.15.67:0/1587543219 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8934103130 0x7f893419ed00 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:39.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.145+0000 7f8933e00640 1 --2- 172.21.15.67:0/1587543219 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8934102770 0x7f8934197c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:39.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.145+0000 7f8933e00640 1 --2- 172.21.15.67:0/1587543219 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8934101580 0x7f8934197750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:39.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.145+0000 7f8933e00640 1 -- 172.21.15.67:0/1587543219 >> 172.21.15.67:0/1587543219 conn(0x7f89340fcb10 msgr2=0x7f8934105a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:39.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.145+0000 7f8933e00640 1 -- 172.21.15.67:0/1587543219 shutdown_connections 2024-03-28T16:09:39.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.145+0000 7f8933e00640 1 -- 172.21.15.67:0/1587543219 wait complete. 2024-03-28T16:09:39.163 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T16:09:39.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:38 smithi150 ceph-mon[34825]: pgmap v621: 97 pgs: 97 active+clean; 980 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:09:39.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:38 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/3488633950' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:09:39.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:38 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/2392005020' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T16:09:39.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.387+0000 7f5d95e00640 1 -- 172.21.15.67:0/1628093870 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d90105470 msgr2=0x7f5d9010bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:39.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.387+0000 7f5d95e00640 1 --2- 172.21.15.67:0/1628093870 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d90105470 0x7f5d9010bd00 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f5d88009ca0 tx=0x7f5d8802c100 comp rx=0 tx=0).stop 2024-03-28T16:09:39.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.387+0000 7f5d95e00640 1 -- 172.21.15.67:0/1628093870 shutdown_connections 2024-03-28T16:09:39.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.387+0000 7f5d95e00640 1 --2- 172.21.15.67:0/1628093870 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d90105470 0x7f5d9010bd00 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:39.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.387+0000 7f5d95e00640 1 --2- 172.21.15.67:0/1628093870 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5d90104ab0 0x7f5d90104f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:39.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.387+0000 7f5d95e00640 1 --2- 172.21.15.67:0/1628093870 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5d901038c0 0x7f5d90103ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:39.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.387+0000 7f5d95e00640 1 -- 172.21.15.67:0/1628093870 >> 172.21.15.67:0/1628093870 conn(0x7f5d900fee50 msgr2=0x7f5d90101270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:39.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.387+0000 7f5d95e00640 1 -- 172.21.15.67:0/1628093870 shutdown_connections 2024-03-28T16:09:39.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.387+0000 7f5d95e00640 1 -- 172.21.15.67:0/1628093870 wait complete. 2024-03-28T16:09:39.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.388+0000 7f5d95e00640 1 Processor -- start 2024-03-28T16:09:39.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.388+0000 7f5d95e00640 1 -- start start 2024-03-28T16:09:39.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.388+0000 7f5d95e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5d901038c0 0x7f5d90079450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:39.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.388+0000 7f5d95e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d90104ab0 0x7f5d90079990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:39.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.389+0000 7f5d95e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5d90105470 0x7f5d90075fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:39.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.389+0000 7f5d95e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d9010f3d0 con 0x7f5d90104ab0 2024-03-28T16:09:39.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.389+0000 7f5d95e00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d9010f250 con 0x7f5d901038c0 2024-03-28T16:09:39.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.389+0000 7f5d95e00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d9010f550 con 0x7f5d90105470 2024-03-28T16:09:39.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.389+0000 7f5d8e000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d90104ab0 0x7f5d90079990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:39.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.389+0000 7f5d8e000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d90104ab0 0x7f5d90079990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35304/0 (socket says 172.21.15.67:35304) 2024-03-28T16:09:39.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.389+0000 7f5d8e000640 1 -- 172.21.15.67:0/1370559961 learned_addr learned my addr 172.21.15.67:0/1370559961 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:09:39.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.389+0000 7f5d8ea00640 1 --2- 172.21.15.67:0/1370559961 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5d901038c0 0x7f5d90079450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:39.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.389+0000 7f5d8f400640 1 --2- 172.21.15.67:0/1370559961 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5d90105470 0x7f5d90075fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:39.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.389+0000 7f5d8e000640 1 -- 172.21.15.67:0/1370559961 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5d90105470 msgr2=0x7f5d90075fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:39.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.389+0000 7f5d8e000640 1 --2- 172.21.15.67:0/1370559961 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5d90105470 0x7f5d90075fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:39.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.389+0000 7f5d8e000640 1 -- 172.21.15.67:0/1370559961 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5d901038c0 msgr2=0x7f5d90079450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:39.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.389+0000 7f5d8e000640 1 --2- 172.21.15.67:0/1370559961 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5d901038c0 0x7f5d90079450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:39.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.389+0000 7f5d8e000640 1 -- 172.21.15.67:0/1370559961 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d90076880 con 0x7f5d90104ab0 2024-03-28T16:09:39.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.390+0000 7f5d8f400640 1 --2- 172.21.15.67:0/1370559961 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5d90105470 0x7f5d90075fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T16:09:39.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.390+0000 7f5d8e000640 1 --2- 172.21.15.67:0/1370559961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d90104ab0 0x7f5d90079990 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f5d7800e9f0 tx=0x7f5d7800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:39.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.390+0000 7f5d94a00640 1 -- 172.21.15.67:0/1370559961 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d7800cd80 con 0x7f5d90104ab0 2024-03-28T16:09:39.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.390+0000 7f5d94a00640 1 -- 172.21.15.67:0/1370559961 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5d780108e0 con 0x7f5d90104ab0 2024-03-28T16:09:39.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.390+0000 7f5d95e00640 1 -- 172.21.15.67:0/1370559961 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d901a5ec0 con 0x7f5d90104ab0 2024-03-28T16:09:39.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.390+0000 7f5d94a00640 1 -- 172.21.15.67:0/1370559961 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d78018920 con 0x7f5d90104ab0 2024-03-28T16:09:39.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.390+0000 7f5d95e00640 1 -- 172.21.15.67:0/1370559961 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d901a6270 con 0x7f5d90104ab0 2024-03-28T16:09:39.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.392+0000 7f5d95e00640 1 -- 172.21.15.67:0/1370559961 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d900688e0 con 0x7f5d90104ab0 2024-03-28T16:09:39.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.394+0000 7f5d94a00640 1 -- 172.21.15.67:0/1370559961 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f5d78010a90 con 0x7f5d90104ab0 2024-03-28T16:09:39.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.395+0000 7f5d94a00640 1 --2- 172.21.15.67:0/1370559961 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f5d740a2800 0x7f5d740a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:39.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.395+0000 7f5d94a00640 1 -- 172.21.15.67:0/1370559961 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f5d780cf300 con 0x7f5d90104ab0 2024-03-28T16:09:39.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.395+0000 7f5d8ea00640 1 --2- 172.21.15.67:0/1370559961 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f5d740a2800 0x7f5d740a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:39.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.396+0000 7f5d8ea00640 1 --2- 172.21.15.67:0/1370559961 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f5d740a2800 0x7f5d740a4cc0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f5d90104910 tx=0x7f5d8000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:39.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.398+0000 7f5d94a00640 1 -- 172.21.15.67:0/1370559961 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5d7809c350 con 0x7f5d90104ab0 2024-03-28T16:09:39.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.604+0000 7f5d95e00640 1 -- 172.21.15.67:0/1370559961 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5d9002d2a0 con 0x7f5d90104ab0 2024-03-28T16:09:39.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.605+0000 7f5d94a00640 1 -- 172.21.15.67:0/1370559961 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5d7809bc50 con 0x7f5d90104ab0 2024-03-28T16:09:39.605 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-03-28T16:09:39.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.608+0000 7f5d95e00640 1 -- 172.21.15.67:0/1370559961 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f5d740a2800 msgr2=0x7f5d740a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:39.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.608+0000 7f5d95e00640 1 --2- 172.21.15.67:0/1370559961 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f5d740a2800 0x7f5d740a4cc0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f5d90104910 tx=0x7f5d8000b390 comp rx=0 tx=0).stop 2024-03-28T16:09:39.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.608+0000 7f5d95e00640 1 -- 172.21.15.67:0/1370559961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d90104ab0 msgr2=0x7f5d90079990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:39.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.608+0000 7f5d95e00640 1 --2- 172.21.15.67:0/1370559961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d90104ab0 0x7f5d90079990 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f5d7800e9f0 tx=0x7f5d7800eeb0 comp rx=0 tx=0).stop 2024-03-28T16:09:39.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.608+0000 7f5d95e00640 1 -- 172.21.15.67:0/1370559961 shutdown_connections 2024-03-28T16:09:39.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.608+0000 7f5d95e00640 1 --2- 172.21.15.67:0/1370559961 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f5d740a2800 0x7f5d740a4cc0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:39.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.608+0000 7f5d95e00640 1 --2- 172.21.15.67:0/1370559961 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5d90105470 0x7f5d90075fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:39.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.608+0000 7f5d95e00640 1 --2- 172.21.15.67:0/1370559961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d90104ab0 0x7f5d90079990 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:39.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.608+0000 7f5d95e00640 1 --2- 172.21.15.67:0/1370559961 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5d901038c0 0x7f5d90079450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:39.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.608+0000 7f5d95e00640 1 -- 172.21.15.67:0/1370559961 >> 172.21.15.67:0/1370559961 conn(0x7f5d900fee50 msgr2=0x7f5d90107e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:39.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.608+0000 7f5d95e00640 1 -- 172.21.15.67:0/1370559961 shutdown_connections 2024-03-28T16:09:39.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.608+0000 7f5d95e00640 1 -- 172.21.15.67:0/1370559961 wait complete. 2024-03-28T16:09:39.627 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-03-28T16:09:39.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.844+0000 7f156c000640 1 -- 172.21.15.67:0/760501659 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1564104ad0 msgr2=0x7f1564104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:39.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.844+0000 7f156c000640 1 --2- 172.21.15.67:0/760501659 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1564104ad0 0x7f1564104f50 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f1550009900 tx=0x7f155002bc30 comp rx=0 tx=0).stop 2024-03-28T16:09:39.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.844+0000 7f156c000640 1 -- 172.21.15.67:0/760501659 shutdown_connections 2024-03-28T16:09:39.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.844+0000 7f156c000640 1 --2- 172.21.15.67:0/760501659 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1564105490 0x7f156410bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:39.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.844+0000 7f156c000640 1 --2- 172.21.15.67:0/760501659 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1564104ad0 0x7f1564104f50 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:39.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.844+0000 7f156c000640 1 --2- 172.21.15.67:0/760501659 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15641038e0 0x7f1564103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:39.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.844+0000 7f156c000640 1 -- 172.21.15.67:0/760501659 >> 172.21.15.67:0/760501659 conn(0x7f15640fee50 msgr2=0x7f1564101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:39.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.844+0000 7f156c000640 1 -- 172.21.15.67:0/760501659 shutdown_connections 2024-03-28T16:09:39.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.844+0000 7f156c000640 1 -- 172.21.15.67:0/760501659 wait complete. 2024-03-28T16:09:39.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.845+0000 7f156c000640 1 Processor -- start 2024-03-28T16:09:39.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.845+0000 7f156c000640 1 -- start start 2024-03-28T16:09:39.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.846+0000 7f156c000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f15641038e0 0x7f1564199b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:39.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.846+0000 7f156c000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1564104ad0 0x7f156419a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:39.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.846+0000 7f156c000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1564105490 0x7f15641a1110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:39.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.846+0000 7f156c000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f156410f460 con 0x7f1564104ad0 2024-03-28T16:09:39.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.846+0000 7f156c000640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f156410f2e0 con 0x7f1564105490 2024-03-28T16:09:39.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.846+0000 7f156c000640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f156410f5e0 con 0x7f15641038e0 2024-03-28T16:09:39.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.846+0000 7f1568c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1564104ad0 0x7f156419a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:39.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.846+0000 7f1568c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1564104ad0 0x7f156419a0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35324/0 (socket says 172.21.15.67:35324) 2024-03-28T16:09:39.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.846+0000 7f1568c00640 1 -- 172.21.15.67:0/1170207559 learned_addr learned my addr 172.21.15.67:0/1170207559 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:09:39.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.846+0000 7f1569600640 1 --2- 172.21.15.67:0/1170207559 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f15641038e0 0x7f1564199b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:39.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.846+0000 7f156a000640 1 --2- 172.21.15.67:0/1170207559 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1564105490 0x7f15641a1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:39.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.847+0000 7f1568c00640 1 -- 172.21.15.67:0/1170207559 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f15641038e0 msgr2=0x7f1564199b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:39.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.847+0000 7f1568c00640 1 --2- 172.21.15.67:0/1170207559 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f15641038e0 0x7f1564199b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:39.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.847+0000 7f1568c00640 1 -- 172.21.15.67:0/1170207559 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1564105490 msgr2=0x7f15641a1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:39.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.847+0000 7f1568c00640 1 --2- 172.21.15.67:0/1170207559 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1564105490 0x7f15641a1110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:39.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.847+0000 7f1568c00640 1 -- 172.21.15.67:0/1170207559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15641a1780 con 0x7f1564104ad0 2024-03-28T16:09:39.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.847+0000 7f156a000640 1 --2- 172.21.15.67:0/1170207559 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1564105490 0x7f15641a1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:09:39.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.847+0000 7f1568c00640 1 --2- 172.21.15.67:0/1170207559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1564104ad0 0x7f156419a0a0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f1550000c00 tx=0x7f15500042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:39.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.847+0000 7f1562000640 1 -- 172.21.15.67:0/1170207559 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1550043020 con 0x7f1564104ad0 2024-03-28T16:09:39.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.847+0000 7f1562000640 1 -- 172.21.15.67:0/1170207559 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f15500054c0 con 0x7f1564104ad0 2024-03-28T16:09:39.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.847+0000 7f156c000640 1 -- 172.21.15.67:0/1170207559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15641a1a10 con 0x7f1564104ad0 2024-03-28T16:09:39.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.847+0000 7f1562000640 1 -- 172.21.15.67:0/1170207559 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1550003d40 con 0x7f1564104ad0 2024-03-28T16:09:39.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.847+0000 7f156c000640 1 -- 172.21.15.67:0/1170207559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15641a1f60 con 0x7f1564104ad0 2024-03-28T16:09:39.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.849+0000 7f156c000640 1 -- 172.21.15.67:0/1170207559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f156410aca0 con 0x7f1564104ad0 2024-03-28T16:09:39.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.850+0000 7f1562000640 1 -- 172.21.15.67:0/1170207559 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f1550046020 con 0x7f1564104ad0 2024-03-28T16:09:39.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.852+0000 7f1562000640 1 --2- 172.21.15.67:0/1170207559 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f154c0a2800 0x7f154c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:39.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.852+0000 7f1562000640 1 -- 172.21.15.67:0/1170207559 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f15500f0fe0 con 0x7f1564104ad0 2024-03-28T16:09:39.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.852+0000 7f1569600640 1 --2- 172.21.15.67:0/1170207559 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f154c0a2800 0x7f154c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:39.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.853+0000 7f1569600640 1 --2- 172.21.15.67:0/1170207559 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f154c0a2800 0x7f154c0a4cc0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f1558005fd0 tx=0x7f1558005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:39.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:39.856+0000 7f1562000640 1 -- 172.21.15.67:0/1170207559 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f155004d020 con 0x7f1564104ad0 2024-03-28T16:09:40.069 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:39 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/1587543219' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T16:09:40.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:39 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/1370559961' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T16:09:40.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:39 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/1587543219' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T16:09:40.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:39 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/1370559961' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T16:09:40.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:40.124+0000 7f156c000640 1 -- 172.21.15.67:0/1170207559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) v1 -- 0x7f15640622a0 con 0x7f1564104ad0 2024-03-28T16:09:40.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:40.144+0000 7f1562000640 1 -- 172.21.15.67:0/1170207559 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v18) v1 ==== 81+0+739827 (secure 0 0 0) 0x7f15500be1b0 con 0x7f1564104ad0 2024-03-28T16:09:40.151 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:09:40.151 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-28T16:09:40.151 INFO:teuthology.orchestra.run.smithi067.stdout: "epoch": 18, 2024-03-28T16:09:40.151 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.152 INFO:teuthology.orchestra.run.smithi067.stdout: "active_gid": 14152, 2024-03-28T16:09:40.152 INFO:teuthology.orchestra.run.smithi067.stdout: "active_name": "x", 2024-03-28T16:09:40.152 INFO:teuthology.orchestra.run.smithi067.stdout: "active_addrs": { 2024-03-28T16:09:40.152 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-03-28T16:09:40.152 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.152 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-03-28T16:09:40.152 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6800", 2024-03-28T16:09:40.152 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 208726654 2024-03-28T16:09:40.152 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.152 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.152 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-03-28T16:09:40.152 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6801", 2024-03-28T16:09:40.152 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 208726654 2024-03-28T16:09:40.152 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.152 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-28T16:09:40.153 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.153 INFO:teuthology.orchestra.run.smithi067.stdout: "active_addr": "172.21.15.67:6801/208726654", 2024-03-28T16:09:40.153 INFO:teuthology.orchestra.run.smithi067.stdout: "active_change": "2024-03-28T15:51:38.875557+0000", 2024-03-28T16:09:40.153 INFO:teuthology.orchestra.run.smithi067.stdout: "active_mgr_features": 4540701547738038271, 2024-03-28T16:09:40.153 INFO:teuthology.orchestra.run.smithi067.stdout: "available": true, 2024-03-28T16:09:40.153 INFO:teuthology.orchestra.run.smithi067.stdout: "standbys": [ 2024-03-28T16:09:40.153 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.153 INFO:teuthology.orchestra.run.smithi067.stdout: "gid": 34103, 2024-03-28T16:09:40.153 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "z", 2024-03-28T16:09:40.153 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_features": 4540701547738038271, 2024-03-28T16:09:40.153 INFO:teuthology.orchestra.run.smithi067.stdout: "available_modules": [ 2024-03-28T16:09:40.153 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.153 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "alerts", 2024-03-28T16:09:40.153 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:40.153 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:40.154 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.154 INFO:teuthology.orchestra.run.smithi067.stdout: "interval": { 2024-03-28T16:09:40.154 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "interval", 2024-03-28T16:09:40.154 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "secs", 2024-03-28T16:09:40.154 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.154 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.154 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "60", 2024-03-28T16:09:40.154 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.154 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.154 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.154 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "How frequently to reexamine health status", 2024-03-28T16:09:40.154 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.155 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.155 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.155 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.155 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.155 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.155 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.155 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.155 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.155 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.155 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.155 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.155 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.155 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.155 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.155 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.156 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.156 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.156 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.156 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.156 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.156 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.156 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.156 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.156 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.156 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.156 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.156 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.156 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.156 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.157 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.157 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.157 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.157 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.157 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.157 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.157 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.157 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.157 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.157 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:40.157 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:40.157 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.157 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.157 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.157 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:40.158 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.158 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.158 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.158 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.158 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.158 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.158 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.158 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.158 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.158 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.158 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.158 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.158 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.158 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.158 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.159 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:40.159 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:40.159 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.159 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.159 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.159 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.159 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.159 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.159 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.159 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.159 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.159 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.159 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.159 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.159 INFO:teuthology.orchestra.run.smithi067.stdout: "smtp_destination": { 2024-03-28T16:09:40.160 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "smtp_destination", 2024-03-28T16:09:40.160 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.160 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.160 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.160 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.160 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.160 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.160 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.160 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Email address to send alerts to", 2024-03-28T16:09:40.160 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.160 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.160 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.160 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.160 INFO:teuthology.orchestra.run.smithi067.stdout: "smtp_from_name": { 2024-03-28T16:09:40.160 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "smtp_from_name", 2024-03-28T16:09:40.161 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.161 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.161 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.161 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "Ceph", 2024-03-28T16:09:40.161 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.161 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.161 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.161 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Email From: name", 2024-03-28T16:09:40.161 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.161 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.161 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.161 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.161 INFO:teuthology.orchestra.run.smithi067.stdout: "smtp_host": { 2024-03-28T16:09:40.161 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "smtp_host", 2024-03-28T16:09:40.161 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.161 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.162 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.162 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.162 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.162 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.162 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.162 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "SMTP server", 2024-03-28T16:09:40.162 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.162 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.162 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.162 INFO:teuthology.orchestra.run.smithi067.stdout: "smtp_password": { 2024-03-28T16:09:40.162 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "smtp_password", 2024-03-28T16:09:40.162 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.162 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.162 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.163 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.163 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.163 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.163 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.163 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Password to authenticate with", 2024-03-28T16:09:40.163 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.163 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.163 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.163 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.163 INFO:teuthology.orchestra.run.smithi067.stdout: "smtp_port": { 2024-03-28T16:09:40.163 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "smtp_port", 2024-03-28T16:09:40.163 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.163 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.163 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.163 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "465", 2024-03-28T16:09:40.164 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.164 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.164 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.164 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "SMTP port", 2024-03-28T16:09:40.164 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.164 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.164 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.164 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.164 INFO:teuthology.orchestra.run.smithi067.stdout: "smtp_sender": { 2024-03-28T16:09:40.164 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "smtp_sender", 2024-03-28T16:09:40.164 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.164 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.164 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.164 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.164 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.165 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.165 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.165 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "SMTP envelope sender", 2024-03-28T16:09:40.165 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.165 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.165 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.165 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.165 INFO:teuthology.orchestra.run.smithi067.stdout: "smtp_ssl": { 2024-03-28T16:09:40.165 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "smtp_ssl", 2024-03-28T16:09:40.165 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.165 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.165 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.165 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.165 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.166 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.166 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.166 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-28T16:09:40.166 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.166 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.166 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.166 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.166 INFO:teuthology.orchestra.run.smithi067.stdout: "smtp_user": { 2024-03-28T16:09:40.166 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "smtp_user", 2024-03-28T16:09:40.166 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.166 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.166 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.166 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.166 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.166 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.167 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.167 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "User to authenticate as", 2024-03-28T16:09:40.167 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.167 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.167 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.167 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.167 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.167 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.167 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.167 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "balancer", 2024-03-28T16:09:40.167 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:40.167 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:40.167 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.167 INFO:teuthology.orchestra.run.smithi067.stdout: "active": { 2024-03-28T16:09:40.167 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "active", 2024-03-28T16:09:40.168 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.168 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.168 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.168 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.168 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.168 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.168 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.168 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "automatically balance PGs across cluster", 2024-03-28T16:09:40.168 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.168 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.168 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.168 INFO:teuthology.orchestra.run.smithi067.stdout: "begin_time": { 2024-03-28T16:09:40.168 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "begin_time", 2024-03-28T16:09:40.168 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.169 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.169 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.169 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "0000", 2024-03-28T16:09:40.169 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.169 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.169 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.169 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "beginning time of day to automatically balance", 2024-03-28T16:09:40.169 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-28T16:09:40.169 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.169 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.169 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.169 INFO:teuthology.orchestra.run.smithi067.stdout: "begin_weekday": { 2024-03-28T16:09:40.169 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "begin_weekday", 2024-03-28T16:09:40.169 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "uint", 2024-03-28T16:09:40.170 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.170 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.170 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "0", 2024-03-28T16:09:40.170 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "0", 2024-03-28T16:09:40.170 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "6", 2024-03-28T16:09:40.170 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.170 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-28T16:09:40.170 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-28T16:09:40.170 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.170 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.170 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.170 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_compat_max_iterations": { 2024-03-28T16:09:40.170 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "crush_compat_max_iterations", 2024-03-28T16:09:40.170 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "uint", 2024-03-28T16:09:40.170 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.171 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.171 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "25", 2024-03-28T16:09:40.171 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "1", 2024-03-28T16:09:40.171 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "250", 2024-03-28T16:09:40.171 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.171 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-28T16:09:40.171 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.171 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.171 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.171 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.171 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_compat_metrics": { 2024-03-28T16:09:40.171 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "crush_compat_metrics", 2024-03-28T16:09:40.171 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.171 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.171 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.172 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "pgs,objects,bytes", 2024-03-28T16:09:40.172 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.172 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.172 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.172 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-28T16:09:40.172 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-28T16:09:40.172 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.172 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.172 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.172 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_compat_step": { 2024-03-28T16:09:40.172 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "crush_compat_step", 2024-03-28T16:09:40.172 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "float", 2024-03-28T16:09:40.172 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.172 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.172 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "0.5", 2024-03-28T16:09:40.172 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "0.001", 2024-03-28T16:09:40.173 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "0.999", 2024-03-28T16:09:40.173 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.173 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "aggressiveness of optimization", 2024-03-28T16:09:40.173 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-28T16:09:40.173 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.173 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.173 INFO:teuthology.orchestra.run.smithi067.stdout: "end_time": { 2024-03-28T16:09:40.173 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "end_time", 2024-03-28T16:09:40.173 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.173 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.173 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.173 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "2359", 2024-03-28T16:09:40.173 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.173 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.173 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.174 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "ending time of day to automatically balance", 2024-03-28T16:09:40.174 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-28T16:09:40.174 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.174 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.174 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.174 INFO:teuthology.orchestra.run.smithi067.stdout: "end_weekday": { 2024-03-28T16:09:40.174 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "end_weekday", 2024-03-28T16:09:40.174 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "uint", 2024-03-28T16:09:40.174 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.174 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.174 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "0", 2024-03-28T16:09:40.174 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "0", 2024-03-28T16:09:40.174 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "6", 2024-03-28T16:09:40.174 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.174 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-28T16:09:40.175 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-28T16:09:40.175 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.175 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.175 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.175 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.175 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.175 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.175 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.175 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.175 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.175 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.175 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.175 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.175 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.175 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.176 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.176 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.176 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.176 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.176 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.176 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.176 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.176 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.176 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.176 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.176 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.176 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.176 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.176 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.176 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.176 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.177 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.177 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.177 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.177 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.177 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.177 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.177 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.177 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.177 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:40.177 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:40.177 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.177 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.177 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.177 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:40.178 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.178 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.178 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.178 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.178 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.178 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.178 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.178 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.178 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.178 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.178 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.178 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.178 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.178 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.178 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.179 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:40.179 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:40.179 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.179 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.179 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.179 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.179 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.179 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.179 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.179 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.179 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.179 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.179 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.179 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.179 INFO:teuthology.orchestra.run.smithi067.stdout: "min_score": { 2024-03-28T16:09:40.180 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "min_score", 2024-03-28T16:09:40.180 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "float", 2024-03-28T16:09:40.180 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.180 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.180 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "0", 2024-03-28T16:09:40.180 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.180 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.180 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.180 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-28T16:09:40.180 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.180 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.180 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.180 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.180 INFO:teuthology.orchestra.run.smithi067.stdout: "mode": { 2024-03-28T16:09:40.180 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "mode", 2024-03-28T16:09:40.181 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.181 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.181 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.181 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "upmap", 2024-03-28T16:09:40.181 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.181 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.181 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.181 INFO:teuthology.orchestra.run.smithi067.stdout: "crush-compat", 2024-03-28T16:09:40.181 INFO:teuthology.orchestra.run.smithi067.stdout: "none", 2024-03-28T16:09:40.181 INFO:teuthology.orchestra.run.smithi067.stdout: "read", 2024-03-28T16:09:40.181 INFO:teuthology.orchestra.run.smithi067.stdout: "upmap", 2024-03-28T16:09:40.181 INFO:teuthology.orchestra.run.smithi067.stdout: "upmap-read" 2024-03-28T16:09:40.181 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.181 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Balancer mode", 2024-03-28T16:09:40.181 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.181 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.182 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.182 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.182 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_ids": { 2024-03-28T16:09:40.182 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "pool_ids", 2024-03-28T16:09:40.182 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.182 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.182 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.182 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.182 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.182 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.182 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.182 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-28T16:09:40.182 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.182 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.183 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.183 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.183 INFO:teuthology.orchestra.run.smithi067.stdout: "sleep_interval": { 2024-03-28T16:09:40.183 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "sleep_interval", 2024-03-28T16:09:40.183 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "secs", 2024-03-28T16:09:40.183 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.183 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.183 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "60", 2024-03-28T16:09:40.183 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.183 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.183 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.183 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-28T16:09:40.183 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.183 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.183 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.184 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.184 INFO:teuthology.orchestra.run.smithi067.stdout: "upmap_max_deviation": { 2024-03-28T16:09:40.184 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "upmap_max_deviation", 2024-03-28T16:09:40.184 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.184 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.184 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.184 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "5", 2024-03-28T16:09:40.184 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "1", 2024-03-28T16:09:40.184 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.184 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.184 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-28T16:09:40.184 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-28T16:09:40.184 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.184 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.184 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.184 INFO:teuthology.orchestra.run.smithi067.stdout: "upmap_max_optimizations": { 2024-03-28T16:09:40.185 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "upmap_max_optimizations", 2024-03-28T16:09:40.185 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "uint", 2024-03-28T16:09:40.185 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.185 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.185 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "10", 2024-03-28T16:09:40.185 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.185 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.185 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.185 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-28T16:09:40.185 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.185 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.185 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.185 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.185 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.185 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.186 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.186 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "cephadm", 2024-03-28T16:09:40.186 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:40.186 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:40.186 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.186 INFO:teuthology.orchestra.run.smithi067.stdout: "agent_down_multiplier": { 2024-03-28T16:09:40.186 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "agent_down_multiplier", 2024-03-28T16:09:40.186 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "float", 2024-03-28T16:09:40.186 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.186 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.186 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "3.0", 2024-03-28T16:09:40.186 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.186 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.186 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.186 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-28T16:09:40.187 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.187 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.187 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.187 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.187 INFO:teuthology.orchestra.run.smithi067.stdout: "agent_refresh_rate": { 2024-03-28T16:09:40.187 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "agent_refresh_rate", 2024-03-28T16:09:40.187 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "secs", 2024-03-28T16:09:40.187 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.187 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.187 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "20", 2024-03-28T16:09:40.187 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.187 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.187 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.187 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-28T16:09:40.187 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.187 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.188 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.188 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.188 INFO:teuthology.orchestra.run.smithi067.stdout: "agent_starting_port": { 2024-03-28T16:09:40.188 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "agent_starting_port", 2024-03-28T16:09:40.188 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.188 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.188 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.188 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "4721", 2024-03-28T16:09:40.188 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.188 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.188 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.188 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-28T16:09:40.188 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.188 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.188 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.189 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.189 INFO:teuthology.orchestra.run.smithi067.stdout: "allow_ptrace": { 2024-03-28T16:09:40.189 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "allow_ptrace", 2024-03-28T16:09:40.189 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.189 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.189 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.189 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.189 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.189 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.189 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.189 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-28T16:09:40.189 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-28T16:09:40.189 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.189 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.189 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.190 INFO:teuthology.orchestra.run.smithi067.stdout: "autotune_interval": { 2024-03-28T16:09:40.190 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "autotune_interval", 2024-03-28T16:09:40.190 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "secs", 2024-03-28T16:09:40.190 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.190 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.190 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "600", 2024-03-28T16:09:40.190 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.190 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.190 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.190 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-28T16:09:40.190 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.190 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.190 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.190 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.190 INFO:teuthology.orchestra.run.smithi067.stdout: "autotune_memory_target_ratio": { 2024-03-28T16:09:40.191 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "autotune_memory_target_ratio", 2024-03-28T16:09:40.191 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "float", 2024-03-28T16:09:40.191 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.191 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.191 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "0.7", 2024-03-28T16:09:40.191 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.191 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.191 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.191 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-28T16:09:40.191 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.191 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.191 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.191 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.191 INFO:teuthology.orchestra.run.smithi067.stdout: "cephadm_log_destination": { 2024-03-28T16:09:40.192 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "cephadm_log_destination", 2024-03-28T16:09:40.192 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.192 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.192 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.192 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.192 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.192 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.192 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.192 INFO:teuthology.orchestra.run.smithi067.stdout: "file", 2024-03-28T16:09:40.192 INFO:teuthology.orchestra.run.smithi067.stdout: "file,syslog", 2024-03-28T16:09:40.192 INFO:teuthology.orchestra.run.smithi067.stdout: "syslog" 2024-03-28T16:09:40.192 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.192 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-28T16:09:40.192 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.192 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.193 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.193 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.193 INFO:teuthology.orchestra.run.smithi067.stdout: "cgroups_split": { 2024-03-28T16:09:40.193 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "cgroups_split", 2024-03-28T16:09:40.193 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.193 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.193 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.193 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.193 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.193 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.193 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.193 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-28T16:09:40.193 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.193 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.193 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.193 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.194 INFO:teuthology.orchestra.run.smithi067.stdout: "config_checks_enabled": { 2024-03-28T16:09:40.194 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "config_checks_enabled", 2024-03-28T16:09:40.194 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.194 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.194 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.194 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.194 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.194 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.194 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.194 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-28T16:09:40.194 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.194 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.194 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.194 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.194 INFO:teuthology.orchestra.run.smithi067.stdout: "config_dashboard": { 2024-03-28T16:09:40.195 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "config_dashboard", 2024-03-28T16:09:40.195 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.195 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.195 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.195 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.195 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.195 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.195 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.195 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-28T16:09:40.195 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.195 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.195 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.195 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.195 INFO:teuthology.orchestra.run.smithi067.stdout: "container_image_alertmanager": { 2024-03-28T16:09:40.195 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "container_image_alertmanager", 2024-03-28T16:09:40.195 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.196 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.196 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.196 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-28T16:09:40.196 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.196 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.196 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.196 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Prometheus container image", 2024-03-28T16:09:40.196 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.196 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.196 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.196 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.196 INFO:teuthology.orchestra.run.smithi067.stdout: "container_image_base": { 2024-03-28T16:09:40.196 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "container_image_base", 2024-03-28T16:09:40.196 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.196 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.197 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.197 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-28T16:09:40.197 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.197 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.197 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.197 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Container image name, without the tag", 2024-03-28T16:09:40.197 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.197 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.197 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.197 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.197 INFO:teuthology.orchestra.run.smithi067.stdout: "container_image_elasticsearch": { 2024-03-28T16:09:40.197 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "container_image_elasticsearch", 2024-03-28T16:09:40.197 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.197 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.197 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.197 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-28T16:09:40.198 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.198 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.198 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.198 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "elasticsearch container image", 2024-03-28T16:09:40.198 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.198 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.198 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.198 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.198 INFO:teuthology.orchestra.run.smithi067.stdout: "container_image_grafana": { 2024-03-28T16:09:40.198 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "container_image_grafana", 2024-03-28T16:09:40.198 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.198 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.198 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.198 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-28T16:09:40.198 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.199 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.199 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.199 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Prometheus container image", 2024-03-28T16:09:40.199 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.199 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.199 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.199 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.199 INFO:teuthology.orchestra.run.smithi067.stdout: "container_image_haproxy": { 2024-03-28T16:09:40.199 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "container_image_haproxy", 2024-03-28T16:09:40.199 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.199 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.199 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.199 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-28T16:09:40.199 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.199 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.200 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.200 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "HAproxy container image", 2024-03-28T16:09:40.200 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.200 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.200 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.200 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.200 INFO:teuthology.orchestra.run.smithi067.stdout: "container_image_jaeger_agent": { 2024-03-28T16:09:40.200 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "container_image_jaeger_agent", 2024-03-28T16:09:40.200 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.200 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.200 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.200 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-28T16:09:40.200 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.200 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.201 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.201 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Jaeger agent container image", 2024-03-28T16:09:40.201 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.201 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.201 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.201 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.201 INFO:teuthology.orchestra.run.smithi067.stdout: "container_image_jaeger_collector": { 2024-03-28T16:09:40.201 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "container_image_jaeger_collector", 2024-03-28T16:09:40.201 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.201 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.201 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.201 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-28T16:09:40.201 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.201 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.201 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.202 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Jaeger collector container image", 2024-03-28T16:09:40.202 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.202 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.202 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.202 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.202 INFO:teuthology.orchestra.run.smithi067.stdout: "container_image_jaeger_query": { 2024-03-28T16:09:40.202 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "container_image_jaeger_query", 2024-03-28T16:09:40.202 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.202 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.202 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.202 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-28T16:09:40.202 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.202 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.202 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.203 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Jaeger query container image", 2024-03-28T16:09:40.203 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.203 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.203 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.203 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.203 INFO:teuthology.orchestra.run.smithi067.stdout: "container_image_keepalived": { 2024-03-28T16:09:40.203 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "container_image_keepalived", 2024-03-28T16:09:40.203 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.203 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.203 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.203 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-28T16:09:40.203 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.203 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.203 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.203 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Keepalived container image", 2024-03-28T16:09:40.203 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.204 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.204 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.204 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.204 INFO:teuthology.orchestra.run.smithi067.stdout: "container_image_loki": { 2024-03-28T16:09:40.204 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "container_image_loki", 2024-03-28T16:09:40.204 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.204 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.204 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.204 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-28T16:09:40.204 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.204 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.204 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.204 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Loki container image", 2024-03-28T16:09:40.204 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.204 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.205 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.205 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.205 INFO:teuthology.orchestra.run.smithi067.stdout: "container_image_node_exporter": { 2024-03-28T16:09:40.205 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "container_image_node_exporter", 2024-03-28T16:09:40.205 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.205 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.205 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.205 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-28T16:09:40.205 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.205 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.205 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.205 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Prometheus container image", 2024-03-28T16:09:40.205 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.205 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.205 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.205 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.206 INFO:teuthology.orchestra.run.smithi067.stdout: "container_image_nvmeof": { 2024-03-28T16:09:40.206 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "container_image_nvmeof", 2024-03-28T16:09:40.206 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.206 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.206 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.206 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-28T16:09:40.206 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.206 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.206 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.206 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Nvme-of container image", 2024-03-28T16:09:40.206 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.206 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.206 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.206 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.206 INFO:teuthology.orchestra.run.smithi067.stdout: "container_image_prometheus": { 2024-03-28T16:09:40.207 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "container_image_prometheus", 2024-03-28T16:09:40.207 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.207 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.207 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.207 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-28T16:09:40.207 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.207 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.207 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.207 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Prometheus container image", 2024-03-28T16:09:40.207 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.207 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.207 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.207 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.207 INFO:teuthology.orchestra.run.smithi067.stdout: "container_image_promtail": { 2024-03-28T16:09:40.207 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "container_image_promtail", 2024-03-28T16:09:40.208 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.208 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.208 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.208 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-28T16:09:40.208 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.208 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.208 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.208 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Promtail container image", 2024-03-28T16:09:40.208 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.208 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.208 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.208 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.208 INFO:teuthology.orchestra.run.smithi067.stdout: "container_image_snmp_gateway": { 2024-03-28T16:09:40.208 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "container_image_snmp_gateway", 2024-03-28T16:09:40.208 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.208 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.209 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.209 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-28T16:09:40.209 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.209 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.209 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.209 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "SNMP Gateway container image", 2024-03-28T16:09:40.209 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.209 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.209 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.209 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.209 INFO:teuthology.orchestra.run.smithi067.stdout: "container_init": { 2024-03-28T16:09:40.209 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "container_init", 2024-03-28T16:09:40.209 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.209 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.209 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.209 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.210 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.210 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.210 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.210 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Run podman/docker with `--init`", 2024-03-28T16:09:40.210 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.210 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.210 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.210 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.210 INFO:teuthology.orchestra.run.smithi067.stdout: "daemon_cache_timeout": { 2024-03-28T16:09:40.210 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "daemon_cache_timeout", 2024-03-28T16:09:40.210 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "secs", 2024-03-28T16:09:40.210 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.210 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.210 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "600", 2024-03-28T16:09:40.211 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.211 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.211 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.211 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-28T16:09:40.211 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.211 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.211 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.211 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.211 INFO:teuthology.orchestra.run.smithi067.stdout: "default_cephadm_command_timeout": { 2024-03-28T16:09:40.211 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "default_cephadm_command_timeout", 2024-03-28T16:09:40.211 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.211 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.211 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.211 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "900", 2024-03-28T16:09:40.211 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.212 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.212 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.212 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-28T16:09:40.212 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.212 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.212 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.212 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.212 INFO:teuthology.orchestra.run.smithi067.stdout: "default_registry": { 2024-03-28T16:09:40.212 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "default_registry", 2024-03-28T16:09:40.212 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.212 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.212 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.212 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "docker.io", 2024-03-28T16:09:40.212 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.212 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.213 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.213 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-28T16:09:40.213 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.213 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.213 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.213 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.213 INFO:teuthology.orchestra.run.smithi067.stdout: "device_cache_timeout": { 2024-03-28T16:09:40.213 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "device_cache_timeout", 2024-03-28T16:09:40.213 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "secs", 2024-03-28T16:09:40.213 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.213 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.213 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "1800", 2024-03-28T16:09:40.213 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.213 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.213 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.213 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "seconds to cache device inventory", 2024-03-28T16:09:40.214 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.214 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.214 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.214 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.214 INFO:teuthology.orchestra.run.smithi067.stdout: "device_enhanced_scan": { 2024-03-28T16:09:40.214 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "device_enhanced_scan", 2024-03-28T16:09:40.214 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.214 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.214 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.214 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.214 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.214 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.214 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.214 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-28T16:09:40.214 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.215 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.215 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.215 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.215 INFO:teuthology.orchestra.run.smithi067.stdout: "facts_cache_timeout": { 2024-03-28T16:09:40.215 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "facts_cache_timeout", 2024-03-28T16:09:40.215 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "secs", 2024-03-28T16:09:40.215 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.215 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.215 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "60", 2024-03-28T16:09:40.215 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.215 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.215 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.215 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "seconds to cache host facts data", 2024-03-28T16:09:40.215 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.215 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.216 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.216 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.216 INFO:teuthology.orchestra.run.smithi067.stdout: "host_check_interval": { 2024-03-28T16:09:40.216 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "host_check_interval", 2024-03-28T16:09:40.216 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "secs", 2024-03-28T16:09:40.216 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.216 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.216 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "600", 2024-03-28T16:09:40.216 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.216 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.216 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.216 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "how frequently to perform a host check", 2024-03-28T16:09:40.216 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.216 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.216 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.217 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.217 INFO:teuthology.orchestra.run.smithi067.stdout: "hw_monitoring": { 2024-03-28T16:09:40.217 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "hw_monitoring", 2024-03-28T16:09:40.217 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.217 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.217 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.217 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.217 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.217 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.217 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.217 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-28T16:09:40.217 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.217 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.217 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.217 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.218 INFO:teuthology.orchestra.run.smithi067.stdout: "inventory_list_all": { 2024-03-28T16:09:40.218 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "inventory_list_all", 2024-03-28T16:09:40.218 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.218 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.218 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.218 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.218 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.218 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.218 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.218 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-28T16:09:40.218 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.218 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.218 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.218 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.218 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.219 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.219 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.219 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.219 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.219 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.219 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.219 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.219 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.219 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.219 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.219 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.219 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.219 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.219 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.219 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.219 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.220 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.220 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.220 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.220 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.220 INFO:teuthology.orchestra.run.smithi067.stdout: "log_refresh_metadata": { 2024-03-28T16:09:40.220 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_refresh_metadata", 2024-03-28T16:09:40.220 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.220 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.220 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.220 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.220 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.220 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.220 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.220 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-28T16:09:40.220 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.221 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.221 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.221 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.221 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.221 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.221 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.221 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.221 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.221 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.221 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.221 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.221 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.221 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-28T16:09:40.221 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.221 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.221 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.222 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.222 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:40.222 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:40.222 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.222 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.222 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.222 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:40.222 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.222 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.222 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.222 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.222 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.222 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.222 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.223 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.223 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.223 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.223 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.223 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.223 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.223 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.223 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.223 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:40.223 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:40.223 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.223 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.223 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.223 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.223 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.224 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.224 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.224 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.224 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.224 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.224 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.224 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.224 INFO:teuthology.orchestra.run.smithi067.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-28T16:09:40.224 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-28T16:09:40.224 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.224 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.224 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.224 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.224 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.224 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.225 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.225 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-28T16:09:40.225 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.225 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.225 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.225 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.225 INFO:teuthology.orchestra.run.smithi067.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-28T16:09:40.225 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-28T16:09:40.225 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.225 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.225 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.225 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "*", 2024-03-28T16:09:40.225 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.225 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.225 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.226 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-28T16:09:40.226 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.226 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.226 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.226 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.226 INFO:teuthology.orchestra.run.smithi067.stdout: "max_count_per_host": { 2024-03-28T16:09:40.226 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "max_count_per_host", 2024-03-28T16:09:40.226 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.226 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.226 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.226 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "10", 2024-03-28T16:09:40.226 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.226 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.226 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.226 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "max number of daemons per service per host", 2024-03-28T16:09:40.227 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.227 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.227 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.227 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.227 INFO:teuthology.orchestra.run.smithi067.stdout: "max_osd_draining_count": { 2024-03-28T16:09:40.227 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "max_osd_draining_count", 2024-03-28T16:09:40.227 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.227 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.227 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.227 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "10", 2024-03-28T16:09:40.227 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.227 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.227 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.227 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-28T16:09:40.227 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.227 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.228 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.228 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.228 INFO:teuthology.orchestra.run.smithi067.stdout: "migration_current": { 2024-03-28T16:09:40.228 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "migration_current", 2024-03-28T16:09:40.228 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.228 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.228 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.228 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.228 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.228 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.228 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.228 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "internal - do not modify", 2024-03-28T16:09:40.228 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.228 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.228 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.229 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.229 INFO:teuthology.orchestra.run.smithi067.stdout: "mode": { 2024-03-28T16:09:40.229 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "mode", 2024-03-28T16:09:40.229 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.229 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.229 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.229 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "root", 2024-03-28T16:09:40.229 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.229 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.229 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.229 INFO:teuthology.orchestra.run.smithi067.stdout: "cephadm-package", 2024-03-28T16:09:40.229 INFO:teuthology.orchestra.run.smithi067.stdout: "root" 2024-03-28T16:09:40.229 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.229 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "mode for remote execution of cephadm", 2024-03-28T16:09:40.229 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.229 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.230 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.230 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.230 INFO:teuthology.orchestra.run.smithi067.stdout: "oob_default_addr": { 2024-03-28T16:09:40.230 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "oob_default_addr", 2024-03-28T16:09:40.230 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.230 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.230 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.230 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "169.254.1.1", 2024-03-28T16:09:40.230 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.230 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.230 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.230 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-28T16:09:40.230 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.230 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.230 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.231 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.231 INFO:teuthology.orchestra.run.smithi067.stdout: "prometheus_alerts_path": { 2024-03-28T16:09:40.231 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "prometheus_alerts_path", 2024-03-28T16:09:40.231 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.231 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.231 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.231 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-28T16:09:40.231 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.231 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.231 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.231 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-28T16:09:40.231 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.231 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.231 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.231 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.231 INFO:teuthology.orchestra.run.smithi067.stdout: "registry_insecure": { 2024-03-28T16:09:40.232 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "registry_insecure", 2024-03-28T16:09:40.232 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.232 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.232 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.232 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.232 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.232 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.232 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.232 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-28T16:09:40.232 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.232 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.232 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.232 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.232 INFO:teuthology.orchestra.run.smithi067.stdout: "registry_password": { 2024-03-28T16:09:40.232 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "registry_password", 2024-03-28T16:09:40.232 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.233 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.233 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.233 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.233 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.233 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.233 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.233 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-28T16:09:40.233 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.233 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.233 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.233 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.233 INFO:teuthology.orchestra.run.smithi067.stdout: "registry_url": { 2024-03-28T16:09:40.233 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "registry_url", 2024-03-28T16:09:40.233 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.233 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.234 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.234 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.234 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.234 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.234 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.234 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-28T16:09:40.234 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.234 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.234 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.234 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.234 INFO:teuthology.orchestra.run.smithi067.stdout: "registry_username": { 2024-03-28T16:09:40.234 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "registry_username", 2024-03-28T16:09:40.234 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.234 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.234 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.235 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.235 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.235 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.235 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.235 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-28T16:09:40.235 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.235 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.235 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.235 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.235 INFO:teuthology.orchestra.run.smithi067.stdout: "secure_monitoring_stack": { 2024-03-28T16:09:40.235 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "secure_monitoring_stack", 2024-03-28T16:09:40.235 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.235 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.235 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.236 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.236 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.236 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.236 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.236 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-28T16:09:40.236 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.236 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.236 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.236 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.236 INFO:teuthology.orchestra.run.smithi067.stdout: "service_discovery_port": { 2024-03-28T16:09:40.236 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "service_discovery_port", 2024-03-28T16:09:40.236 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.236 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.236 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.236 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "8765", 2024-03-28T16:09:40.237 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.237 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.237 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.237 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "cephadm service discovery port", 2024-03-28T16:09:40.237 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.237 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.237 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.237 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.237 INFO:teuthology.orchestra.run.smithi067.stdout: "ssh_config_file": { 2024-03-28T16:09:40.237 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "ssh_config_file", 2024-03-28T16:09:40.237 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.237 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.237 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.237 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.237 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.238 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.238 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.238 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-28T16:09:40.238 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.238 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.238 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.238 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.238 INFO:teuthology.orchestra.run.smithi067.stdout: "use_agent": { 2024-03-28T16:09:40.238 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "use_agent", 2024-03-28T16:09:40.238 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.238 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.238 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.238 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.238 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.238 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.238 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.239 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-28T16:09:40.239 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.239 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.239 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.239 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.239 INFO:teuthology.orchestra.run.smithi067.stdout: "use_repo_digest": { 2024-03-28T16:09:40.239 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "use_repo_digest", 2024-03-28T16:09:40.239 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.239 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.239 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.239 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.239 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.239 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.239 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.239 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-28T16:09:40.240 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.240 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.240 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.240 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.240 INFO:teuthology.orchestra.run.smithi067.stdout: "warn_on_failed_host_check": { 2024-03-28T16:09:40.240 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "warn_on_failed_host_check", 2024-03-28T16:09:40.240 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.240 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.240 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.240 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.240 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.240 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.240 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.240 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "raise a health warning if the host check fails", 2024-03-28T16:09:40.240 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.240 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.241 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.241 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.241 INFO:teuthology.orchestra.run.smithi067.stdout: "warn_on_stray_daemons": { 2024-03-28T16:09:40.241 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "warn_on_stray_daemons", 2024-03-28T16:09:40.241 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.241 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.241 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.241 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.241 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.241 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.241 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.241 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-28T16:09:40.241 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.241 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.241 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.241 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.242 INFO:teuthology.orchestra.run.smithi067.stdout: "warn_on_stray_hosts": { 2024-03-28T16:09:40.242 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "warn_on_stray_hosts", 2024-03-28T16:09:40.242 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.242 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.242 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.242 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.242 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.242 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.242 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.242 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-28T16:09:40.242 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.242 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.242 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.242 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.242 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.243 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.243 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.243 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "crash", 2024-03-28T16:09:40.243 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:40.243 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:40.243 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.243 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.243 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.243 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.243 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.243 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.243 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.243 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.243 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.243 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.243 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.244 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.244 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.244 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.244 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.244 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.244 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.244 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.244 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.244 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.244 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.244 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.244 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.244 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.244 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.244 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.244 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.245 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.245 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.245 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.245 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.245 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.245 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.245 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.245 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.245 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.245 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:40.245 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:40.245 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.245 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.245 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.246 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:40.246 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.246 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.246 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.246 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.246 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.246 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.246 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.246 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.246 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.246 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.246 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.246 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.246 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.246 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.247 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.247 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:40.247 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:40.247 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.247 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.247 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.247 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.247 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.247 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.247 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.247 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.247 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.247 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.247 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.247 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.248 INFO:teuthology.orchestra.run.smithi067.stdout: "retain_interval": { 2024-03-28T16:09:40.248 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "retain_interval", 2024-03-28T16:09:40.248 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "secs", 2024-03-28T16:09:40.248 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.248 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.248 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "31536000", 2024-03-28T16:09:40.248 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.248 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.248 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.248 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-28T16:09:40.248 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.248 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.248 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.248 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.248 INFO:teuthology.orchestra.run.smithi067.stdout: "warn_recent_interval": { 2024-03-28T16:09:40.249 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "warn_recent_interval", 2024-03-28T16:09:40.249 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "secs", 2024-03-28T16:09:40.249 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.249 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.249 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "1209600", 2024-03-28T16:09:40.249 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.249 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.249 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.249 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-28T16:09:40.249 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.249 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.249 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.249 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.249 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.249 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.250 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.250 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "dashboard", 2024-03-28T16:09:40.250 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:40.250 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:40.250 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.250 INFO:teuthology.orchestra.run.smithi067.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-28T16:09:40.250 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-28T16:09:40.250 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.250 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.250 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.250 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "10", 2024-03-28T16:09:40.250 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.250 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.250 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.250 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.250 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.251 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.251 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.251 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.251 INFO:teuthology.orchestra.run.smithi067.stdout: "ALERTMANAGER_API_HOST": { 2024-03-28T16:09:40.251 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-28T16:09:40.251 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.251 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.251 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.251 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.251 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.251 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.251 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.251 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.251 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.252 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.252 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.252 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.252 INFO:teuthology.orchestra.run.smithi067.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-28T16:09:40.252 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-28T16:09:40.252 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.252 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.252 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.252 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.252 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.252 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.252 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.252 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.252 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.252 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.252 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.253 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.253 INFO:teuthology.orchestra.run.smithi067.stdout: "AUDIT_API_ENABLED": { 2024-03-28T16:09:40.253 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "AUDIT_API_ENABLED", 2024-03-28T16:09:40.253 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.253 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.253 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.253 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.253 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.253 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.253 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.253 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.253 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.253 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.253 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.253 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.253 INFO:teuthology.orchestra.run.smithi067.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-28T16:09:40.254 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-28T16:09:40.254 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.254 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.254 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.254 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.254 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.254 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.254 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.254 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.254 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.254 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.254 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.254 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.254 INFO:teuthology.orchestra.run.smithi067.stdout: "ENABLE_BROWSABLE_API": { 2024-03-28T16:09:40.255 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-28T16:09:40.255 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.255 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.255 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.255 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.255 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.255 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.255 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.255 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.255 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.255 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.255 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.255 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.255 INFO:teuthology.orchestra.run.smithi067.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-28T16:09:40.255 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-28T16:09:40.255 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.256 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.256 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.256 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.256 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.256 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.256 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.256 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.256 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.256 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.256 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.256 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.256 INFO:teuthology.orchestra.run.smithi067.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-28T16:09:40.256 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-28T16:09:40.256 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.256 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.256 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.257 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.257 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.257 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.257 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.257 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.257 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.257 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.257 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.257 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.257 INFO:teuthology.orchestra.run.smithi067.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-28T16:09:40.257 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-28T16:09:40.257 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.257 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.257 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.258 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.258 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.258 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.258 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.258 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.258 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.258 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.258 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.258 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.258 INFO:teuthology.orchestra.run.smithi067.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-28T16:09:40.258 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-28T16:09:40.258 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.258 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.258 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.259 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.259 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.259 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.259 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.259 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.259 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.259 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.259 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.259 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.259 INFO:teuthology.orchestra.run.smithi067.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-28T16:09:40.259 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-28T16:09:40.259 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.259 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.259 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.259 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.259 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.260 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.260 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.260 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.260 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.260 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.260 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.260 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.260 INFO:teuthology.orchestra.run.smithi067.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-28T16:09:40.260 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-28T16:09:40.260 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.260 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.260 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.260 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.260 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.260 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.261 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.261 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.261 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.261 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.261 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.261 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.261 INFO:teuthology.orchestra.run.smithi067.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-28T16:09:40.261 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-28T16:09:40.261 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.261 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.261 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.261 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.261 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.261 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.261 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.262 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.262 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.262 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.262 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.262 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.262 INFO:teuthology.orchestra.run.smithi067.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-28T16:09:40.262 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-28T16:09:40.262 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.262 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.262 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.262 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.262 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.262 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.262 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.262 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.263 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.263 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.263 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.263 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.263 INFO:teuthology.orchestra.run.smithi067.stdout: "GRAFANA_API_PASSWORD": { 2024-03-28T16:09:40.263 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-28T16:09:40.263 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.263 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.263 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.263 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "admin", 2024-03-28T16:09:40.263 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.263 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.263 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.263 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.263 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.264 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.264 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.264 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.264 INFO:teuthology.orchestra.run.smithi067.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-28T16:09:40.264 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-28T16:09:40.264 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.264 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.264 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.264 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.264 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.264 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.264 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.264 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.264 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.264 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.264 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.265 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.265 INFO:teuthology.orchestra.run.smithi067.stdout: "GRAFANA_API_URL": { 2024-03-28T16:09:40.265 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "GRAFANA_API_URL", 2024-03-28T16:09:40.265 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.265 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.265 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.265 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.265 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.265 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.265 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.265 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.265 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.265 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.265 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.265 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.265 INFO:teuthology.orchestra.run.smithi067.stdout: "GRAFANA_API_USERNAME": { 2024-03-28T16:09:40.266 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-28T16:09:40.266 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.266 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.266 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.266 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "admin", 2024-03-28T16:09:40.266 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.266 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.266 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.266 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.266 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.266 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.266 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.266 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.266 INFO:teuthology.orchestra.run.smithi067.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-28T16:09:40.266 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-28T16:09:40.267 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.267 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.267 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.267 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.267 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.267 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.267 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.267 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.267 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.267 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.267 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.267 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.267 INFO:teuthology.orchestra.run.smithi067.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-28T16:09:40.267 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-28T16:09:40.267 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.267 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.268 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.268 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.268 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.268 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.268 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.268 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.268 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.268 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.268 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.268 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.268 INFO:teuthology.orchestra.run.smithi067.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-28T16:09:40.268 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-28T16:09:40.268 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.268 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.268 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.268 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.269 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.269 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.269 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.269 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.269 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.269 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.269 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.269 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.269 INFO:teuthology.orchestra.run.smithi067.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-28T16:09:40.269 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-28T16:09:40.269 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.269 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.269 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.269 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.269 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.270 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.270 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.270 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.270 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.270 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.270 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.270 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.270 INFO:teuthology.orchestra.run.smithi067.stdout: "MULTICLUSTER_CONFIG": { 2024-03-28T16:09:40.270 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-28T16:09:40.270 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.270 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.270 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.270 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "{}", 2024-03-28T16:09:40.270 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.270 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.271 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.271 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.271 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.271 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.271 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.271 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.271 INFO:teuthology.orchestra.run.smithi067.stdout: "PROMETHEUS_API_HOST": { 2024-03-28T16:09:40.271 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-28T16:09:40.271 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.271 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.271 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.271 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.271 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.271 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.271 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.271 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.272 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.272 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.272 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.272 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.272 INFO:teuthology.orchestra.run.smithi067.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-28T16:09:40.272 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-28T16:09:40.272 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.272 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.272 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.272 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.272 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.272 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.272 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.272 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.272 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.273 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.273 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.273 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.273 INFO:teuthology.orchestra.run.smithi067.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-28T16:09:40.273 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-28T16:09:40.273 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.273 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.273 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.273 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.273 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.273 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.273 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.273 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.273 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.273 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.274 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.274 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.274 INFO:teuthology.orchestra.run.smithi067.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-28T16:09:40.274 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-28T16:09:40.274 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.274 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.274 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.274 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.274 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.274 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.274 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.274 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.274 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.274 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.274 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.275 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.275 INFO:teuthology.orchestra.run.smithi067.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-28T16:09:40.275 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-28T16:09:40.275 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.275 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.275 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.275 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.275 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.275 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.275 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.275 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.275 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.275 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.275 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.275 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.276 INFO:teuthology.orchestra.run.smithi067.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-28T16:09:40.276 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-28T16:09:40.276 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.276 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.276 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.276 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.276 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.276 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.276 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.276 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.276 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.276 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.276 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.276 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.276 INFO:teuthology.orchestra.run.smithi067.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-28T16:09:40.276 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-28T16:09:40.277 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.277 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.277 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.277 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.277 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.277 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.277 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.277 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.277 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.277 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.277 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.277 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.277 INFO:teuthology.orchestra.run.smithi067.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-28T16:09:40.277 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-28T16:09:40.277 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.278 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.278 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.278 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.278 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.278 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.278 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.278 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.278 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.278 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.278 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.278 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.278 INFO:teuthology.orchestra.run.smithi067.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-28T16:09:40.278 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-28T16:09:40.278 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.278 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.279 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.279 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.279 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.279 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.279 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.279 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.279 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.279 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.279 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.279 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.279 INFO:teuthology.orchestra.run.smithi067.stdout: "PWD_POLICY_ENABLED": { 2024-03-28T16:09:40.279 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-28T16:09:40.279 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.279 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.279 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.279 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.280 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.280 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.280 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.280 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.280 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.280 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.280 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.280 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.280 INFO:teuthology.orchestra.run.smithi067.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-28T16:09:40.280 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-28T16:09:40.280 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.280 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.280 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.280 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-28T16:09:40.280 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.280 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.281 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.281 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.281 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.281 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.281 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.281 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.281 INFO:teuthology.orchestra.run.smithi067.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-28T16:09:40.281 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-28T16:09:40.281 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.281 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.281 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.281 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "10", 2024-03-28T16:09:40.281 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.281 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.281 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.282 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.282 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.282 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.282 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.282 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.282 INFO:teuthology.orchestra.run.smithi067.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-28T16:09:40.282 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-28T16:09:40.282 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.282 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.282 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.282 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "8", 2024-03-28T16:09:40.282 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.282 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.283 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.283 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.283 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.283 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.283 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.283 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.283 INFO:teuthology.orchestra.run.smithi067.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-28T16:09:40.283 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-28T16:09:40.283 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.283 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.283 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.283 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "45", 2024-03-28T16:09:40.283 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.283 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.283 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.284 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.284 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.284 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.284 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.284 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.284 INFO:teuthology.orchestra.run.smithi067.stdout: "RGW_API_ACCESS_KEY": { 2024-03-28T16:09:40.284 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-28T16:09:40.284 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.284 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.284 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.284 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.284 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.284 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.284 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.284 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.285 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.285 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.285 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.285 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.285 INFO:teuthology.orchestra.run.smithi067.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-28T16:09:40.285 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-28T16:09:40.285 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.285 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.285 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.285 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "admin", 2024-03-28T16:09:40.285 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.285 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.285 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.285 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.285 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.286 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.286 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.286 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.286 INFO:teuthology.orchestra.run.smithi067.stdout: "RGW_API_SECRET_KEY": { 2024-03-28T16:09:40.286 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-28T16:09:40.286 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.286 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.286 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.286 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.286 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.286 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.286 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.286 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.286 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.287 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.287 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.287 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.287 INFO:teuthology.orchestra.run.smithi067.stdout: "RGW_API_SSL_VERIFY": { 2024-03-28T16:09:40.287 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-28T16:09:40.287 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.287 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.287 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.287 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.287 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.287 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.287 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.287 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.287 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.287 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.288 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.288 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.288 INFO:teuthology.orchestra.run.smithi067.stdout: "UNSAFE_TLS_v1_2": { 2024-03-28T16:09:40.288 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-28T16:09:40.288 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.288 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.288 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.288 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.288 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.288 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.288 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.288 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.288 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.288 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.288 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.289 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.289 INFO:teuthology.orchestra.run.smithi067.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-28T16:09:40.289 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-28T16:09:40.289 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.289 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.289 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.289 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "0", 2024-03-28T16:09:40.289 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.289 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.289 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.289 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.289 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.289 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.289 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.289 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.289 INFO:teuthology.orchestra.run.smithi067.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-28T16:09:40.290 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-28T16:09:40.290 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.290 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.290 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.290 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "10", 2024-03-28T16:09:40.290 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.290 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.290 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.290 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.290 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.290 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.290 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.290 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.290 INFO:teuthology.orchestra.run.smithi067.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-28T16:09:40.290 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-28T16:09:40.291 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.291 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.291 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.291 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "5", 2024-03-28T16:09:40.291 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.291 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.291 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.291 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.291 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.291 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.291 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.291 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.291 INFO:teuthology.orchestra.run.smithi067.stdout: "cross_origin_url": { 2024-03-28T16:09:40.291 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "cross_origin_url", 2024-03-28T16:09:40.291 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.291 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.292 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.292 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.292 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.292 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.292 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.292 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.292 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.292 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.292 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.292 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.292 INFO:teuthology.orchestra.run.smithi067.stdout: "crt_file": { 2024-03-28T16:09:40.292 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "crt_file", 2024-03-28T16:09:40.292 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.292 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.292 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.292 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.293 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.293 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.293 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.293 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.293 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.293 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.293 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.293 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.293 INFO:teuthology.orchestra.run.smithi067.stdout: "debug": { 2024-03-28T16:09:40.293 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "debug", 2024-03-28T16:09:40.293 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.293 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.293 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.293 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.294 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.294 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.294 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.294 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Enable/disable debug options", 2024-03-28T16:09:40.294 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.294 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.294 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.294 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.294 INFO:teuthology.orchestra.run.smithi067.stdout: "jwt_token_ttl": { 2024-03-28T16:09:40.294 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "jwt_token_ttl", 2024-03-28T16:09:40.294 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.294 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.294 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.294 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "28800", 2024-03-28T16:09:40.294 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.295 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.295 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.295 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.295 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.295 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.295 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.295 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.295 INFO:teuthology.orchestra.run.smithi067.stdout: "key_file": { 2024-03-28T16:09:40.295 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "key_file", 2024-03-28T16:09:40.295 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.295 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.295 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.295 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.295 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.295 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.296 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.296 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.296 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.296 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.296 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.296 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.296 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.296 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.296 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.296 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.296 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.296 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.296 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.296 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.296 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.297 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.297 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.297 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.297 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.297 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.297 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.297 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.297 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.297 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.297 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.297 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.297 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.297 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.297 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.297 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.297 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.298 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.298 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.298 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.298 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.298 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.298 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.298 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.298 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.298 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.298 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.298 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:40.298 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:40.298 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.298 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.299 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.299 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:40.299 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.299 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.299 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.299 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.299 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.299 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.299 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.299 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.299 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.299 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.299 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.299 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.299 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.299 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.300 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.300 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:40.300 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:40.300 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.300 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.300 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.300 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.300 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.300 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.300 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.300 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.300 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.300 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.300 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.300 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.300 INFO:teuthology.orchestra.run.smithi067.stdout: "motd": { 2024-03-28T16:09:40.301 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "motd", 2024-03-28T16:09:40.301 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.301 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.301 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.301 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.301 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.301 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.301 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.301 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "The message of the day", 2024-03-28T16:09:40.301 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.301 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.301 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.301 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.301 INFO:teuthology.orchestra.run.smithi067.stdout: "redirect_resolve_ip_addr": { 2024-03-28T16:09:40.301 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "redirect_resolve_ip_addr", 2024-03-28T16:09:40.302 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.302 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.302 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.302 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.302 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.302 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.302 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.302 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.302 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.302 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.302 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.302 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.302 INFO:teuthology.orchestra.run.smithi067.stdout: "server_addr": { 2024-03-28T16:09:40.302 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "server_addr", 2024-03-28T16:09:40.302 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.303 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.303 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.303 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "::", 2024-03-28T16:09:40.303 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.303 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.303 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.303 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.303 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.303 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.303 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.303 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.303 INFO:teuthology.orchestra.run.smithi067.stdout: "server_port": { 2024-03-28T16:09:40.303 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "server_port", 2024-03-28T16:09:40.303 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.303 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.304 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.304 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "8080", 2024-03-28T16:09:40.304 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.304 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.304 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.304 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.304 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.304 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.304 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.304 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.304 INFO:teuthology.orchestra.run.smithi067.stdout: "ssl": { 2024-03-28T16:09:40.304 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "ssl", 2024-03-28T16:09:40.304 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.304 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.304 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.305 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.305 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.305 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.305 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.305 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.305 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.305 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.305 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.305 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.305 INFO:teuthology.orchestra.run.smithi067.stdout: "ssl_server_port": { 2024-03-28T16:09:40.305 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "ssl_server_port", 2024-03-28T16:09:40.305 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.305 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.305 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.305 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "8443", 2024-03-28T16:09:40.305 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.306 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.306 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.306 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.306 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.306 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.306 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.306 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.306 INFO:teuthology.orchestra.run.smithi067.stdout: "standby_behaviour": { 2024-03-28T16:09:40.306 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "standby_behaviour", 2024-03-28T16:09:40.306 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.306 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.306 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.306 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "redirect", 2024-03-28T16:09:40.306 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.307 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.307 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.307 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.307 INFO:teuthology.orchestra.run.smithi067.stdout: "redirect" 2024-03-28T16:09:40.307 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.307 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.307 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.307 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.307 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.307 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.307 INFO:teuthology.orchestra.run.smithi067.stdout: "standby_error_status_code": { 2024-03-28T16:09:40.307 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "standby_error_status_code", 2024-03-28T16:09:40.307 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.307 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.307 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.308 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "500", 2024-03-28T16:09:40.308 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "400", 2024-03-28T16:09:40.308 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "599", 2024-03-28T16:09:40.308 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.308 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.308 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.308 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.308 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.308 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.308 INFO:teuthology.orchestra.run.smithi067.stdout: "url_prefix": { 2024-03-28T16:09:40.308 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "url_prefix", 2024-03-28T16:09:40.308 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.308 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.308 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.308 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.309 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.309 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.309 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.309 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.309 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.309 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.309 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.309 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.309 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.309 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.309 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "devicehealth", 2024-03-28T16:09:40.309 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:40.309 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:40.310 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.310 INFO:teuthology.orchestra.run.smithi067.stdout: "enable_monitoring": { 2024-03-28T16:09:40.310 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "enable_monitoring", 2024-03-28T16:09:40.310 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.310 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.310 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.310 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.310 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.310 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.310 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.310 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "monitor device health metrics", 2024-03-28T16:09:40.310 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.310 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.310 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.311 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.311 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.311 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.311 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.311 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.311 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.311 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.311 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.311 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.311 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.311 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.311 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.311 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.311 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.311 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.312 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.312 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.312 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.312 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.312 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.312 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.312 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.312 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.312 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.312 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.312 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.312 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.312 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.312 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.312 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.312 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.313 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.313 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.313 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.313 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.313 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:40.313 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:40.313 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.313 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.313 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.313 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:40.313 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.313 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.313 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.313 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.313 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.314 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.314 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.314 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.314 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.314 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.314 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.314 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.314 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.314 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.314 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:40.314 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:40.314 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.314 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.314 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.314 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.315 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.315 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.315 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.315 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.315 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.315 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.315 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.315 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.315 INFO:teuthology.orchestra.run.smithi067.stdout: "mark_out_threshold": { 2024-03-28T16:09:40.315 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "mark_out_threshold", 2024-03-28T16:09:40.315 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "secs", 2024-03-28T16:09:40.315 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.315 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.315 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "2419200", 2024-03-28T16:09:40.315 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.316 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.316 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.316 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-28T16:09:40.316 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.316 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.316 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.316 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.316 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_name": { 2024-03-28T16:09:40.316 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "pool_name", 2024-03-28T16:09:40.316 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.316 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.316 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.316 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "device_health_metrics", 2024-03-28T16:09:40.316 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.316 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.316 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.317 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-28T16:09:40.317 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.317 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.317 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.317 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.317 INFO:teuthology.orchestra.run.smithi067.stdout: "retention_period": { 2024-03-28T16:09:40.317 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "retention_period", 2024-03-28T16:09:40.317 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "secs", 2024-03-28T16:09:40.317 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.317 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.317 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "15552000", 2024-03-28T16:09:40.317 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.317 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.317 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.317 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "how long to retain device health metrics", 2024-03-28T16:09:40.318 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.318 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.318 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.318 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.318 INFO:teuthology.orchestra.run.smithi067.stdout: "scrape_frequency": { 2024-03-28T16:09:40.318 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "scrape_frequency", 2024-03-28T16:09:40.318 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "secs", 2024-03-28T16:09:40.318 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.318 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.318 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "86400", 2024-03-28T16:09:40.318 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.318 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.318 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.318 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-28T16:09:40.319 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.319 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.319 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.319 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.319 INFO:teuthology.orchestra.run.smithi067.stdout: "self_heal": { 2024-03-28T16:09:40.319 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "self_heal", 2024-03-28T16:09:40.319 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.319 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.319 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.319 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.319 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.319 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.319 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.319 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-28T16:09:40.319 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.320 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.320 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.320 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.320 INFO:teuthology.orchestra.run.smithi067.stdout: "sleep_interval": { 2024-03-28T16:09:40.320 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "sleep_interval", 2024-03-28T16:09:40.320 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "secs", 2024-03-28T16:09:40.320 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.320 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.320 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "600", 2024-03-28T16:09:40.320 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.320 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.320 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.320 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "how frequently to wake up and check device health", 2024-03-28T16:09:40.320 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.320 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.321 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.321 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.321 INFO:teuthology.orchestra.run.smithi067.stdout: "warn_threshold": { 2024-03-28T16:09:40.321 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "warn_threshold", 2024-03-28T16:09:40.321 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "secs", 2024-03-28T16:09:40.321 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.321 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.321 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "7257600", 2024-03-28T16:09:40.321 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.321 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.321 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.321 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-28T16:09:40.321 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.321 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.321 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.322 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.322 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.322 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.322 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.322 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "diskprediction_local", 2024-03-28T16:09:40.322 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:40.322 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:40.322 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.322 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.322 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.322 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.322 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.322 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.322 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.322 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.323 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.323 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.323 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.323 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.323 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.323 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.323 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.323 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.323 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.323 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.323 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.323 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.323 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.323 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.323 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.324 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.324 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.324 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.324 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.324 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.324 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.324 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.324 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.324 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.324 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.324 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.324 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.324 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.324 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:40.324 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:40.325 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.325 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.325 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.325 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:40.325 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.325 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.325 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.325 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.325 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.325 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.325 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.325 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.325 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.325 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.325 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.326 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.326 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.326 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.326 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.326 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:40.326 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:40.326 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.326 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.326 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.326 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.326 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.326 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.326 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.326 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.326 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.327 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.327 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.327 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.327 INFO:teuthology.orchestra.run.smithi067.stdout: "predict_interval": { 2024-03-28T16:09:40.327 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "predict_interval", 2024-03-28T16:09:40.327 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.327 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.327 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.327 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "86400", 2024-03-28T16:09:40.327 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.327 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.327 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.327 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.327 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.327 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.328 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.328 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.328 INFO:teuthology.orchestra.run.smithi067.stdout: "predictor_model": { 2024-03-28T16:09:40.328 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "predictor_model", 2024-03-28T16:09:40.328 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.328 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.328 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.328 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "prophetstor", 2024-03-28T16:09:40.328 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.328 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.328 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.328 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.328 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.328 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.328 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.329 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.329 INFO:teuthology.orchestra.run.smithi067.stdout: "sleep_interval": { 2024-03-28T16:09:40.329 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "sleep_interval", 2024-03-28T16:09:40.329 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.329 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.329 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.329 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "600", 2024-03-28T16:09:40.329 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.329 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.329 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.329 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.329 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.329 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.329 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.329 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.330 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.330 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.330 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.330 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "influx", 2024-03-28T16:09:40.330 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": false, 2024-03-28T16:09:40.330 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "influxdb python module not found", 2024-03-28T16:09:40.330 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.330 INFO:teuthology.orchestra.run.smithi067.stdout: "batch_size": { 2024-03-28T16:09:40.330 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "batch_size", 2024-03-28T16:09:40.330 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.330 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.330 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.330 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "5000", 2024-03-28T16:09:40.330 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.330 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.331 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.331 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-28T16:09:40.331 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.331 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.331 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.331 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.331 INFO:teuthology.orchestra.run.smithi067.stdout: "database": { 2024-03-28T16:09:40.331 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "database", 2024-03-28T16:09:40.331 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.331 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.331 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.331 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "ceph", 2024-03-28T16:09:40.331 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.331 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.332 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.332 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-28T16:09:40.332 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.332 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.332 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.332 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.332 INFO:teuthology.orchestra.run.smithi067.stdout: "hostname": { 2024-03-28T16:09:40.332 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "hostname", 2024-03-28T16:09:40.332 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.332 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.332 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.332 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.332 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.332 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.333 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.333 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "InfluxDB server hostname", 2024-03-28T16:09:40.333 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.333 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.333 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.333 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.333 INFO:teuthology.orchestra.run.smithi067.stdout: "interval": { 2024-03-28T16:09:40.333 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "interval", 2024-03-28T16:09:40.333 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "secs", 2024-03-28T16:09:40.333 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.333 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.333 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "30", 2024-03-28T16:09:40.333 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "5", 2024-03-28T16:09:40.333 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.333 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.334 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-28T16:09:40.334 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.334 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.334 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.334 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.334 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.334 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.334 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.334 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.334 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.334 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.334 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.334 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.334 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.335 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.335 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.335 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.335 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.335 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.335 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.335 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.335 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.335 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.335 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.335 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.335 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.335 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.335 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.336 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.336 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.336 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.336 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.336 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.336 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.336 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.336 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.336 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.336 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.336 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.336 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.336 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:40.336 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:40.336 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.337 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.337 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.337 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:40.337 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.337 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.337 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.337 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.337 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.337 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.337 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.337 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.337 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.337 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.337 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.338 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.338 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.338 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.338 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.338 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:40.338 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:40.338 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.338 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.338 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.338 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.338 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.338 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.338 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.338 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.338 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.339 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.339 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.339 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.339 INFO:teuthology.orchestra.run.smithi067.stdout: "password": { 2024-03-28T16:09:40.339 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "password", 2024-03-28T16:09:40.339 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.339 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.339 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.339 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.339 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.339 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.339 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.339 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "password of InfluxDB server user", 2024-03-28T16:09:40.339 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.339 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.339 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.340 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.340 INFO:teuthology.orchestra.run.smithi067.stdout: "port": { 2024-03-28T16:09:40.340 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "port", 2024-03-28T16:09:40.340 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.340 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.340 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.340 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "8086", 2024-03-28T16:09:40.340 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.340 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.340 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.340 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "InfluxDB server port", 2024-03-28T16:09:40.340 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.340 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.340 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.340 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.340 INFO:teuthology.orchestra.run.smithi067.stdout: "ssl": { 2024-03-28T16:09:40.341 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "ssl", 2024-03-28T16:09:40.341 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.341 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.341 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.341 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "false", 2024-03-28T16:09:40.341 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.341 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.341 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.341 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-28T16:09:40.341 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.341 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.341 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.341 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.341 INFO:teuthology.orchestra.run.smithi067.stdout: "threads": { 2024-03-28T16:09:40.341 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "threads", 2024-03-28T16:09:40.342 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.342 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.342 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.342 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "5", 2024-03-28T16:09:40.342 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "1", 2024-03-28T16:09:40.342 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "32", 2024-03-28T16:09:40.342 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.342 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-28T16:09:40.342 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.342 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.342 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.342 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.342 INFO:teuthology.orchestra.run.smithi067.stdout: "username": { 2024-03-28T16:09:40.342 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "username", 2024-03-28T16:09:40.342 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.342 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.343 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.343 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.343 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.343 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.343 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.343 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "username of InfluxDB server user", 2024-03-28T16:09:40.343 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.343 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.343 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.343 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.343 INFO:teuthology.orchestra.run.smithi067.stdout: "verify_ssl": { 2024-03-28T16:09:40.343 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "verify_ssl", 2024-03-28T16:09:40.343 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.343 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.343 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.344 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "true", 2024-03-28T16:09:40.344 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.344 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.344 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.344 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-28T16:09:40.344 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.344 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.344 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.344 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.344 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.344 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.344 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.344 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "insights", 2024-03-28T16:09:40.344 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:40.344 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:40.345 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.345 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.345 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.345 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.345 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.345 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.345 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.345 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.345 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.345 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.345 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.345 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.345 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.345 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.346 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.346 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.346 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.346 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.346 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.346 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.346 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.346 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.346 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.346 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.346 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.346 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.346 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.346 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.346 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.347 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.347 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.347 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.347 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.347 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.347 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.347 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.347 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:40.347 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:40.347 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.347 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.347 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.347 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:40.348 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.348 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.348 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.348 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.348 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.348 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.348 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.348 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.348 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.348 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.348 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.348 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.348 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.348 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.348 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.349 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:40.349 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:40.349 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.349 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.349 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.349 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.349 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.349 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.349 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.349 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.349 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.349 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.349 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.349 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.350 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.350 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.350 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.350 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "iostat", 2024-03-28T16:09:40.350 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:40.350 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:40.350 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.350 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.350 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.350 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.350 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.350 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.350 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.350 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.350 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.351 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.351 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.351 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.351 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.351 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.351 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.351 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.351 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.351 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.351 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.351 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.351 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.351 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.351 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.351 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.351 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.352 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.352 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.352 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.352 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.352 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.352 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.352 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.352 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.352 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.352 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.352 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.353 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:40.353 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:40.353 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.353 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.354 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.354 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:40.354 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.354 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.354 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.354 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.354 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.354 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.354 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.354 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.354 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.354 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.354 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.354 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.355 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.355 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.355 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.355 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:40.355 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:40.355 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.355 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.355 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.355 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.355 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.355 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.355 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.355 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.355 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.355 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.355 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.356 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.356 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.356 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.356 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.356 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "k8sevents", 2024-03-28T16:09:40.356 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:40.356 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:40.356 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.356 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph_event_retention_days": { 2024-03-28T16:09:40.356 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "ceph_event_retention_days", 2024-03-28T16:09:40.356 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.356 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.356 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.356 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "7", 2024-03-28T16:09:40.357 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.357 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.357 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.357 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-28T16:09:40.357 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.357 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.357 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.357 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.357 INFO:teuthology.orchestra.run.smithi067.stdout: "config_check_secs": { 2024-03-28T16:09:40.357 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "config_check_secs", 2024-03-28T16:09:40.357 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.357 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.357 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.357 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "10", 2024-03-28T16:09:40.357 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "10", 2024-03-28T16:09:40.358 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.358 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.358 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-28T16:09:40.358 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.358 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.358 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.358 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.358 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.358 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.358 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.358 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.358 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.358 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.359 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.359 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.359 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.359 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.359 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.359 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.359 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.359 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.359 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.359 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.359 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.359 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.359 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.359 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.360 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.360 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.360 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.360 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.360 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.360 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.360 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.360 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.360 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.360 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.360 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.360 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.360 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.360 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.360 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.361 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:40.361 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:40.361 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.361 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.361 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.361 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:40.361 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.361 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.361 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.361 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.361 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.361 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.361 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.361 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.361 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.362 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.362 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.362 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.362 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.362 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.362 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.362 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:40.362 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:40.362 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.362 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.362 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.362 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.362 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.362 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.362 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.363 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.363 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.363 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.363 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.363 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.363 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.363 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.363 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.363 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "localpool", 2024-03-28T16:09:40.363 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:40.363 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:40.363 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.363 INFO:teuthology.orchestra.run.smithi067.stdout: "failure_domain": { 2024-03-28T16:09:40.363 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "failure_domain", 2024-03-28T16:09:40.363 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.364 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.364 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.364 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "host", 2024-03-28T16:09:40.364 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.364 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.364 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.364 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "failure domain for any created local pool", 2024-03-28T16:09:40.364 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-28T16:09:40.364 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.364 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.364 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.364 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.364 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.364 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.365 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.365 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.365 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.365 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.365 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.365 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.365 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.365 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.365 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.365 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.365 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.365 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.365 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.365 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.365 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.365 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.366 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.366 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.366 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.366 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.366 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.366 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.366 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.366 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.366 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.366 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.366 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.366 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.366 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.366 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.366 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.367 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.367 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:40.367 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:40.367 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.367 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.367 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.367 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:40.367 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.367 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.367 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.367 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.367 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.367 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.367 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.367 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.368 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.368 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.368 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.368 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.368 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.368 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.368 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.368 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:40.368 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:40.368 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.368 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.368 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.368 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.368 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.368 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.369 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.369 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.369 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.369 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.369 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.369 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.369 INFO:teuthology.orchestra.run.smithi067.stdout: "min_size": { 2024-03-28T16:09:40.369 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "min_size", 2024-03-28T16:09:40.369 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.369 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.369 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.369 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.369 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.369 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.369 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.369 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "default min_size for any created local pool", 2024-03-28T16:09:40.370 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-28T16:09:40.370 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.370 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.370 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.370 INFO:teuthology.orchestra.run.smithi067.stdout: "num_rep": { 2024-03-28T16:09:40.370 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "num_rep", 2024-03-28T16:09:40.370 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.370 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.370 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.370 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "3", 2024-03-28T16:09:40.370 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.370 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.370 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.370 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "default replica count for any created local pool", 2024-03-28T16:09:40.371 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.371 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.371 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.371 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.371 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num": { 2024-03-28T16:09:40.371 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "pg_num", 2024-03-28T16:09:40.371 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.371 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.371 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.371 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "128", 2024-03-28T16:09:40.371 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.371 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.371 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.371 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "default pg_num for any created local pool", 2024-03-28T16:09:40.371 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.372 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.372 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.372 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.372 INFO:teuthology.orchestra.run.smithi067.stdout: "prefix": { 2024-03-28T16:09:40.372 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "prefix", 2024-03-28T16:09:40.372 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.372 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.372 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.372 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.372 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.372 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.372 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.372 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "name prefix for any created local pool", 2024-03-28T16:09:40.372 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.372 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.373 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.373 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.373 INFO:teuthology.orchestra.run.smithi067.stdout: "subtree": { 2024-03-28T16:09:40.373 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "subtree", 2024-03-28T16:09:40.373 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.373 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.373 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.373 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "rack", 2024-03-28T16:09:40.373 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.373 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.373 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.373 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-28T16:09:40.373 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-28T16:09:40.373 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.374 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.374 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.374 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.374 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.374 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.374 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "mds_autoscaler", 2024-03-28T16:09:40.374 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:40.374 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:40.374 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.374 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.374 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.374 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.374 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.374 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.374 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.375 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.375 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.375 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.375 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.375 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.375 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.375 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.375 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.375 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.375 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.375 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.375 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.375 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.375 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.375 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.376 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.376 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.376 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.376 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.376 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.376 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.376 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.376 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.376 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.376 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.376 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.376 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.376 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.376 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.376 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:40.377 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:40.377 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.377 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.377 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.377 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:40.377 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.377 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.377 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.377 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.377 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.377 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.377 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.377 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.377 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.377 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.377 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.378 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.378 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.378 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.378 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.378 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:40.378 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:40.378 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.378 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.378 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.378 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.378 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.378 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.378 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.378 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.378 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.379 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.379 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.379 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.379 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.379 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.379 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.379 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "mirroring", 2024-03-28T16:09:40.379 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:40.379 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:40.379 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.379 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.379 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.379 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.379 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.379 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.380 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.380 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.380 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.380 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.380 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.380 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.380 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.380 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.380 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.380 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.380 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.380 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.380 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.380 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.380 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.380 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.381 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.381 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.381 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.381 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.381 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.381 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.381 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.381 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.381 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.381 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.381 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.381 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.381 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.381 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.381 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:40.382 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:40.382 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.382 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.382 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.382 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:40.382 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.382 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.382 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.382 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.382 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.382 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.382 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.382 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.382 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.382 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.383 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.383 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.383 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.383 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.383 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.383 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:40.383 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:40.383 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.383 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.383 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.383 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.383 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.383 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.383 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.384 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.384 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.384 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.384 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.384 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.384 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.384 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.384 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.384 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "nfs", 2024-03-28T16:09:40.384 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:40.384 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:40.384 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.384 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.384 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.384 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.385 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.385 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.385 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.385 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.385 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.385 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.385 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.385 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.385 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.385 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.385 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.385 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.385 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.385 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.385 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.386 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.386 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.386 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.386 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.386 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.386 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.386 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.386 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.386 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.386 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.386 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.386 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.386 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.386 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.387 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.387 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.387 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.387 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:40.387 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:40.387 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.387 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.387 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.387 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:40.387 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.387 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.387 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.387 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.387 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.387 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.388 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.388 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.388 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.388 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.388 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.388 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.388 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.388 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.388 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.388 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:40.388 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:40.388 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.388 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.388 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.389 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.389 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.389 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.389 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.389 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.389 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.389 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.389 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.389 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.389 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.389 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.389 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.389 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "orchestrator", 2024-03-28T16:09:40.389 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:40.389 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:40.390 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.390 INFO:teuthology.orchestra.run.smithi067.stdout: "fail_fs": { 2024-03-28T16:09:40.390 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "fail_fs", 2024-03-28T16:09:40.390 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.390 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.390 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.390 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.390 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.390 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.390 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.390 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-28T16:09:40.390 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.390 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.390 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.390 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.391 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.391 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.391 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.391 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.391 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.391 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.391 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.391 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.391 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.391 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.391 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.391 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.391 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.391 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.391 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.392 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.392 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.392 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.392 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.392 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.392 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.392 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.392 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.392 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.392 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.392 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.392 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.392 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.392 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.392 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.393 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.393 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.393 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.393 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.393 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.393 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:40.393 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:40.393 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.393 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.393 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.393 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:40.393 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.393 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.393 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.393 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.393 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.394 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.394 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.394 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.394 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.394 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.394 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.394 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.394 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.394 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.394 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.394 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:40.394 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:40.394 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.394 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.395 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.395 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.395 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.395 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.395 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.395 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.395 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.395 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.395 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.395 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.395 INFO:teuthology.orchestra.run.smithi067.stdout: "orchestrator": { 2024-03-28T16:09:40.395 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "orchestrator", 2024-03-28T16:09:40.395 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.395 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.395 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.396 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.396 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.396 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.396 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.396 INFO:teuthology.orchestra.run.smithi067.stdout: "cephadm", 2024-03-28T16:09:40.396 INFO:teuthology.orchestra.run.smithi067.stdout: "rook", 2024-03-28T16:09:40.396 INFO:teuthology.orchestra.run.smithi067.stdout: "test_orchestrator" 2024-03-28T16:09:40.396 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.396 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Orchestrator backend", 2024-03-28T16:09:40.396 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.396 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.396 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.396 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.396 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.396 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.397 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.397 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "osd_perf_query", 2024-03-28T16:09:40.397 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:40.397 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:40.397 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.397 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.397 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.397 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.397 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.397 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.397 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.397 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.397 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.397 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.398 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.398 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.398 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.398 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.398 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.398 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.398 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.398 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.398 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.398 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.398 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.398 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.398 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.398 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.399 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.399 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.399 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.399 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.399 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.399 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.399 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.399 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.399 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.399 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.399 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.399 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.399 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:40.399 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:40.399 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.400 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.400 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.400 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:40.400 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.400 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.400 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.400 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.400 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.400 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.400 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.400 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.400 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.400 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.400 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.401 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.401 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.401 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.401 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.401 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:40.401 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:40.401 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.401 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.401 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.401 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.401 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.401 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.401 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.401 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.401 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.402 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.402 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.402 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.402 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.402 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.402 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.402 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "osd_support", 2024-03-28T16:09:40.402 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:40.402 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:40.402 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.402 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.402 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.402 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.402 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.403 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.403 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.403 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.403 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.403 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.403 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.403 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.403 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.403 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.403 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.403 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.403 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.403 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.403 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.403 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.404 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.404 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.404 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.404 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.404 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.404 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.404 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.404 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.404 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.404 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.404 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.404 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.404 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.404 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.404 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.404 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.405 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:40.405 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:40.405 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.405 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.405 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.405 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:40.405 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.405 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.405 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.405 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.405 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.405 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.405 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.405 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.406 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.406 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.406 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.406 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.406 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.406 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.406 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.406 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:40.406 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:40.406 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.406 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.406 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.406 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.406 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.406 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.407 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.407 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.407 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.407 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.407 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.407 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.407 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.407 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.407 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.407 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "pg_autoscaler", 2024-03-28T16:09:40.407 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:40.407 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:40.407 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.407 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.407 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.407 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.408 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.408 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.408 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.408 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.408 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.408 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.408 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.408 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.408 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.408 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.408 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.408 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.408 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.408 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.408 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.409 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.409 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.409 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.409 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.409 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.409 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.409 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.409 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.409 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.409 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.409 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.409 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.409 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.409 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.409 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.410 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.410 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.410 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:40.410 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:40.410 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.410 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.410 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.410 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:40.410 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.410 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.410 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.410 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.410 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.411 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.411 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.411 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.411 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.411 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.411 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.411 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.411 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.411 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.411 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.411 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:40.411 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:40.411 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.411 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.411 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.412 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.412 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.412 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.412 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.412 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.412 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.412 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.412 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.412 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.412 INFO:teuthology.orchestra.run.smithi067.stdout: "sleep_interval": { 2024-03-28T16:09:40.412 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "sleep_interval", 2024-03-28T16:09:40.412 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "secs", 2024-03-28T16:09:40.412 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.412 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.413 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "60", 2024-03-28T16:09:40.413 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.413 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.413 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.413 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.413 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.413 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.413 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.413 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.413 INFO:teuthology.orchestra.run.smithi067.stdout: "threshold": { 2024-03-28T16:09:40.413 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "threshold", 2024-03-28T16:09:40.413 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "float", 2024-03-28T16:09:40.413 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.413 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.413 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "3.0", 2024-03-28T16:09:40.414 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "1.0", 2024-03-28T16:09:40.414 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.414 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.414 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "scaling threshold", 2024-03-28T16:09:40.414 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-28T16:09:40.414 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.414 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.414 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.414 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.414 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.414 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.414 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "progress", 2024-03-28T16:09:40.414 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:40.415 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:40.415 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.415 INFO:teuthology.orchestra.run.smithi067.stdout: "allow_pg_recovery_event": { 2024-03-28T16:09:40.415 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "allow_pg_recovery_event", 2024-03-28T16:09:40.415 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.415 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.415 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.415 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.415 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.415 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.415 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.415 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-28T16:09:40.415 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.415 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.416 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.416 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.416 INFO:teuthology.orchestra.run.smithi067.stdout: "enabled": { 2024-03-28T16:09:40.416 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "enabled", 2024-03-28T16:09:40.416 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.416 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.416 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.416 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.416 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.416 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.416 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.416 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.416 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.416 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.416 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.417 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.417 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.417 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.417 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.417 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.417 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.417 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.417 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.417 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.417 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.417 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.417 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.417 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.417 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.417 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.418 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.418 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.418 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.418 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.418 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.418 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.418 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.418 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.418 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.418 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.418 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.418 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.418 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.418 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.419 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.419 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.419 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.419 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.419 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.419 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.419 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.419 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:40.419 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:40.419 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.419 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.419 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.419 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:40.419 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.419 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.420 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.420 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.420 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.420 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.420 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.420 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.420 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.420 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.420 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.420 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.420 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.420 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.420 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.420 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:40.421 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:40.421 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.421 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.421 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.421 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.421 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.421 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.421 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.421 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.421 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.421 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.421 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.421 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.421 INFO:teuthology.orchestra.run.smithi067.stdout: "max_completed_events": { 2024-03-28T16:09:40.422 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "max_completed_events", 2024-03-28T16:09:40.422 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.422 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.422 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.422 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "50", 2024-03-28T16:09:40.422 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.422 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.422 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.422 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "number of past completed events to remember", 2024-03-28T16:09:40.422 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.422 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.422 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.422 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.422 INFO:teuthology.orchestra.run.smithi067.stdout: "sleep_interval": { 2024-03-28T16:09:40.423 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "sleep_interval", 2024-03-28T16:09:40.423 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "secs", 2024-03-28T16:09:40.423 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.423 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.423 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "5", 2024-03-28T16:09:40.423 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.423 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.423 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.423 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "how long the module is going to sleep", 2024-03-28T16:09:40.423 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.423 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.423 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.423 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.423 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.424 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.424 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.424 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "prometheus", 2024-03-28T16:09:40.424 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:40.424 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:40.424 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.424 INFO:teuthology.orchestra.run.smithi067.stdout: "cache": { 2024-03-28T16:09:40.424 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "cache", 2024-03-28T16:09:40.424 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.424 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.424 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.424 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.424 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.424 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.424 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.425 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.425 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.425 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.425 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.425 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.425 INFO:teuthology.orchestra.run.smithi067.stdout: "exclude_perf_counters": { 2024-03-28T16:09:40.425 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "exclude_perf_counters", 2024-03-28T16:09:40.425 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.425 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.425 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.425 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.425 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.425 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.425 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.425 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-28T16:09:40.426 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-28T16:09:40.426 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.426 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.426 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.426 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.426 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.426 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.426 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.426 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.426 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.426 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.426 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.426 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.426 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.427 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.427 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.427 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.427 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.427 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.427 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.427 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.427 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.427 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.427 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.427 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.427 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.427 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.427 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.428 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.428 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.428 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.428 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.428 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.428 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.428 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.428 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.428 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.428 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.428 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.428 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:40.428 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:40.428 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.429 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.429 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.429 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:40.429 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.429 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.429 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.429 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.429 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.429 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.429 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.429 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.429 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.429 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.429 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.429 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.430 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.430 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.430 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.430 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:40.430 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:40.430 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.430 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.430 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.430 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.430 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.430 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.430 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.430 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.430 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.430 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.431 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.431 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.431 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_stats_pools": { 2024-03-28T16:09:40.431 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "rbd_stats_pools", 2024-03-28T16:09:40.431 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.431 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.431 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.431 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.431 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.431 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.431 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.431 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.431 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.431 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.431 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.432 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.432 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-28T16:09:40.432 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-28T16:09:40.432 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.432 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.432 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.432 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "300", 2024-03-28T16:09:40.432 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.432 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.432 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.432 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.432 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.432 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.432 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.432 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.433 INFO:teuthology.orchestra.run.smithi067.stdout: "scrape_interval": { 2024-03-28T16:09:40.433 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "scrape_interval", 2024-03-28T16:09:40.433 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "float", 2024-03-28T16:09:40.433 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.433 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.433 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "15.0", 2024-03-28T16:09:40.433 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.433 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.433 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.433 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.433 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.433 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.433 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.433 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.433 INFO:teuthology.orchestra.run.smithi067.stdout: "server_addr": { 2024-03-28T16:09:40.434 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "server_addr", 2024-03-28T16:09:40.434 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.434 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.434 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.434 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "::", 2024-03-28T16:09:40.434 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.434 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.434 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.434 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-28T16:09:40.434 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.434 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.434 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.434 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.434 INFO:teuthology.orchestra.run.smithi067.stdout: "server_port": { 2024-03-28T16:09:40.434 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "server_port", 2024-03-28T16:09:40.435 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.435 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.435 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.435 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "9283", 2024-03-28T16:09:40.435 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.435 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.435 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.435 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-28T16:09:40.435 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.435 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.435 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.435 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.435 INFO:teuthology.orchestra.run.smithi067.stdout: "stale_cache_strategy": { 2024-03-28T16:09:40.435 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "stale_cache_strategy", 2024-03-28T16:09:40.435 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.436 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.436 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.436 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "log", 2024-03-28T16:09:40.436 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.436 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.436 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.436 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.436 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.436 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.436 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.436 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.436 INFO:teuthology.orchestra.run.smithi067.stdout: "standby_behaviour": { 2024-03-28T16:09:40.436 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "standby_behaviour", 2024-03-28T16:09:40.436 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.437 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.437 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.437 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "default", 2024-03-28T16:09:40.437 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.437 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.437 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.437 INFO:teuthology.orchestra.run.smithi067.stdout: "default", 2024-03-28T16:09:40.437 INFO:teuthology.orchestra.run.smithi067.stdout: "error" 2024-03-28T16:09:40.437 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.437 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.437 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.437 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.437 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.437 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.437 INFO:teuthology.orchestra.run.smithi067.stdout: "standby_error_status_code": { 2024-03-28T16:09:40.438 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "standby_error_status_code", 2024-03-28T16:09:40.438 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.438 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.438 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.438 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "500", 2024-03-28T16:09:40.438 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "400", 2024-03-28T16:09:40.438 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "599", 2024-03-28T16:09:40.438 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.438 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.438 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.438 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.438 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.438 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.438 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.438 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.439 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.439 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "rbd_support", 2024-03-28T16:09:40.439 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:40.439 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:40.439 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.439 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.439 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.439 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.439 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.439 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.439 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.439 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.439 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.439 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.439 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.440 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.440 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.440 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.440 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.440 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.440 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.440 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.440 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.440 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.440 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.440 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.440 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.440 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.440 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.440 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.440 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.441 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.441 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.441 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.441 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.441 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.441 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.441 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.441 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.441 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.441 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:40.441 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:40.441 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.441 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.441 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.441 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:40.442 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.442 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.442 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.442 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.442 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.442 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.442 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.442 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.442 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.442 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.442 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.442 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.442 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.442 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.442 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.443 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:40.443 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:40.443 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.443 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.443 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.443 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.443 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.443 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.443 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.443 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.443 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.443 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.443 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.443 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.443 INFO:teuthology.orchestra.run.smithi067.stdout: "max_concurrent_snap_create": { 2024-03-28T16:09:40.443 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "max_concurrent_snap_create", 2024-03-28T16:09:40.444 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.444 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.444 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.444 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "10", 2024-03-28T16:09:40.444 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.444 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.444 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.444 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.444 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.444 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.444 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.444 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.444 INFO:teuthology.orchestra.run.smithi067.stdout: "mirror_snapshot_schedule": { 2024-03-28T16:09:40.444 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "mirror_snapshot_schedule", 2024-03-28T16:09:40.444 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.445 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.445 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.445 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.445 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.445 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.445 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.445 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.445 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.445 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.445 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.445 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.445 INFO:teuthology.orchestra.run.smithi067.stdout: "trash_purge_schedule": { 2024-03-28T16:09:40.445 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "trash_purge_schedule", 2024-03-28T16:09:40.445 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.445 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.445 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.446 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.446 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.446 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.446 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.446 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.446 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.446 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.446 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.446 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.446 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.446 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.446 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.446 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "restful", 2024-03-28T16:09:40.446 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:40.446 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:40.447 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.447 INFO:teuthology.orchestra.run.smithi067.stdout: "enable_auth": { 2024-03-28T16:09:40.447 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "enable_auth", 2024-03-28T16:09:40.447 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.447 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.447 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.447 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.447 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.447 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.447 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.447 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.447 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.447 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.447 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.447 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.448 INFO:teuthology.orchestra.run.smithi067.stdout: "key_file": { 2024-03-28T16:09:40.448 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "key_file", 2024-03-28T16:09:40.448 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.448 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.448 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.448 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.448 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.448 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.448 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.448 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.448 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.448 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.448 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.448 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.448 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.449 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.449 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.449 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.449 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.449 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.449 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.449 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.449 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.449 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.449 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.449 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.450 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.450 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.450 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.450 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.450 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.450 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.450 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.450 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.450 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.450 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.450 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.450 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.450 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.450 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.450 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.451 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.451 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.451 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.451 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.451 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.451 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.451 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.451 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.451 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:40.451 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:40.451 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.451 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.451 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.451 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:40.452 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.452 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.452 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.452 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.452 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.452 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.452 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.452 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.452 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.452 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.452 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.452 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.452 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.452 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.452 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.453 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:40.453 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:40.453 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.453 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.453 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.453 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.453 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.453 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.453 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.453 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.453 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.453 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.453 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.453 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.453 INFO:teuthology.orchestra.run.smithi067.stdout: "server_addr": { 2024-03-28T16:09:40.454 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "server_addr", 2024-03-28T16:09:40.454 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.454 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.454 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.454 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.454 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.454 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.454 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.454 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.454 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.454 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.454 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.454 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.454 INFO:teuthology.orchestra.run.smithi067.stdout: "server_port": { 2024-03-28T16:09:40.455 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "server_port", 2024-03-28T16:09:40.455 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.455 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.455 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.455 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.455 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.455 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.455 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.455 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.455 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.455 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.455 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.455 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.456 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.456 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.456 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.456 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "rgw", 2024-03-28T16:09:40.456 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:40.456 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:40.456 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.456 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.456 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.456 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.456 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.456 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.456 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.456 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.456 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.457 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.457 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.457 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.457 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.457 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.457 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.457 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.457 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.457 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.457 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.457 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.457 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.457 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.457 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.457 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.458 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.458 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.458 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.458 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.458 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.458 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.458 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.458 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.458 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.458 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.458 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.458 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.458 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:40.458 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:40.458 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.459 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.459 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.459 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:40.459 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.459 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.459 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.459 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.459 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.459 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.459 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.459 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.459 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.459 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.459 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.459 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.459 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.460 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.460 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.460 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:40.460 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:40.460 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.460 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.460 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.460 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.460 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.460 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.460 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.460 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.460 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.460 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.460 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.461 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.461 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.461 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.461 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.461 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "rook", 2024-03-28T16:09:40.461 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:40.461 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:40.461 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.461 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.461 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.461 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.461 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.461 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.461 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.461 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.462 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.462 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.462 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.462 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.462 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.462 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.462 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.462 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.462 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.462 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.462 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.462 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.462 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.463 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.463 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.463 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.463 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.463 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.463 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.463 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.463 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.463 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.463 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.463 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.463 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.463 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.463 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.463 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.464 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:40.464 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:40.464 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.464 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.464 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.464 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:40.464 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.464 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.464 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.464 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.464 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.464 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.464 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.464 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.464 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.465 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.465 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.465 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.465 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.465 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.465 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.465 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:40.465 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:40.465 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.465 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.465 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.465 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.465 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.466 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.466 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.466 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.466 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.466 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.466 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.466 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.466 INFO:teuthology.orchestra.run.smithi067.stdout: "storage_class": { 2024-03-28T16:09:40.466 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "storage_class", 2024-03-28T16:09:40.466 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.466 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.466 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.466 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "local", 2024-03-28T16:09:40.466 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.467 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.467 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.467 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-28T16:09:40.467 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.467 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.467 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.467 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.467 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.467 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.467 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.467 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "selftest", 2024-03-28T16:09:40.467 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:40.467 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:40.467 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.468 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.468 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.468 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.468 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.468 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.468 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.468 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.468 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.468 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.468 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.468 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.468 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.468 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.468 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.468 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.469 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.469 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.469 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.469 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.469 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.469 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.469 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.469 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.469 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.469 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.469 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.469 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.469 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.469 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.469 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.470 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.470 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.470 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.470 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.470 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.470 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:40.470 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:40.470 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.470 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.470 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.470 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:40.470 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.470 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.470 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.471 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.471 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.471 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.471 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.471 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.471 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.471 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.471 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.471 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.471 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.471 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.471 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.471 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:40.471 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:40.472 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.472 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.472 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.472 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.472 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.472 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.472 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.472 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.472 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.472 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.472 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.472 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.472 INFO:teuthology.orchestra.run.smithi067.stdout: "roption1": { 2024-03-28T16:09:40.472 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "roption1", 2024-03-28T16:09:40.472 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.473 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.473 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.473 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.473 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.473 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.473 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.473 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.473 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.473 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.473 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.473 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.473 INFO:teuthology.orchestra.run.smithi067.stdout: "roption2": { 2024-03-28T16:09:40.473 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "roption2", 2024-03-28T16:09:40.473 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.473 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.474 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.474 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "xyz", 2024-03-28T16:09:40.474 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.474 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.474 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.474 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.474 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.474 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.474 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.474 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.474 INFO:teuthology.orchestra.run.smithi067.stdout: "rwoption1": { 2024-03-28T16:09:40.474 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "rwoption1", 2024-03-28T16:09:40.474 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.474 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.475 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.475 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.475 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.475 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.475 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.475 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.475 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.475 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.475 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.475 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.475 INFO:teuthology.orchestra.run.smithi067.stdout: "rwoption2": { 2024-03-28T16:09:40.475 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "rwoption2", 2024-03-28T16:09:40.475 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.475 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.476 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.476 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.476 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.476 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.476 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.476 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.476 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.476 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.476 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.476 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.476 INFO:teuthology.orchestra.run.smithi067.stdout: "rwoption3": { 2024-03-28T16:09:40.476 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "rwoption3", 2024-03-28T16:09:40.476 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "float", 2024-03-28T16:09:40.476 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.477 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.477 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.477 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.477 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.477 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.477 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.477 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.477 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.477 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.477 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.477 INFO:teuthology.orchestra.run.smithi067.stdout: "rwoption4": { 2024-03-28T16:09:40.477 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "rwoption4", 2024-03-28T16:09:40.477 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.477 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.477 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.478 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.478 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.478 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.478 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.478 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.478 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.478 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.478 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.478 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.478 INFO:teuthology.orchestra.run.smithi067.stdout: "rwoption5": { 2024-03-28T16:09:40.478 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "rwoption5", 2024-03-28T16:09:40.478 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.478 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.478 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.478 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.479 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.479 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.479 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.479 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.479 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.479 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.479 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.479 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.479 INFO:teuthology.orchestra.run.smithi067.stdout: "rwoption6": { 2024-03-28T16:09:40.479 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "rwoption6", 2024-03-28T16:09:40.479 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.479 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.479 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.479 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.480 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.480 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.480 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.480 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.480 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.480 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.480 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.480 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.480 INFO:teuthology.orchestra.run.smithi067.stdout: "rwoption7": { 2024-03-28T16:09:40.480 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "rwoption7", 2024-03-28T16:09:40.480 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.480 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.480 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.480 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.480 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "1", 2024-03-28T16:09:40.481 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "42", 2024-03-28T16:09:40.481 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.481 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.481 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.481 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.481 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.481 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.481 INFO:teuthology.orchestra.run.smithi067.stdout: "testkey": { 2024-03-28T16:09:40.481 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "testkey", 2024-03-28T16:09:40.481 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.481 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.481 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.481 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.481 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.481 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.482 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.482 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.482 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.482 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.482 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.482 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.482 INFO:teuthology.orchestra.run.smithi067.stdout: "testlkey": { 2024-03-28T16:09:40.482 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "testlkey", 2024-03-28T16:09:40.482 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.482 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.482 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.482 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.482 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.482 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.483 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.483 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.483 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.483 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.483 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.483 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.483 INFO:teuthology.orchestra.run.smithi067.stdout: "testnewline": { 2024-03-28T16:09:40.483 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "testnewline", 2024-03-28T16:09:40.483 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.483 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.483 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.483 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.483 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.483 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.483 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.483 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.484 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.484 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.484 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.484 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.484 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.484 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.484 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.484 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "snap_schedule", 2024-03-28T16:09:40.484 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:40.484 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:40.484 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.484 INFO:teuthology.orchestra.run.smithi067.stdout: "allow_m_granularity": { 2024-03-28T16:09:40.484 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "allow_m_granularity", 2024-03-28T16:09:40.484 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.485 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.485 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.485 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.485 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.485 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.485 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.485 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "allow minute scheduled snapshots", 2024-03-28T16:09:40.485 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.485 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.485 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.485 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.485 INFO:teuthology.orchestra.run.smithi067.stdout: "dump_on_update": { 2024-03-28T16:09:40.485 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "dump_on_update", 2024-03-28T16:09:40.485 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.485 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.485 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.486 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.486 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.486 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.486 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.486 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "dump database to debug log on update", 2024-03-28T16:09:40.486 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.486 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.486 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.486 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.486 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.486 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.486 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.486 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.486 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.487 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.487 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.487 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.487 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.487 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.487 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.487 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.487 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.487 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.487 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.487 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.487 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.487 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.487 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.487 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.488 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.488 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.488 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.488 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.488 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.488 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.488 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.488 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.488 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.488 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.488 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.488 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.488 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.488 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.488 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.489 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:40.489 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:40.489 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.489 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.489 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.489 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:40.489 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.489 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.489 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.489 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.489 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.489 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.489 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.489 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.489 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.490 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.490 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.490 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.490 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.490 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.490 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.490 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:40.490 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:40.490 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.490 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.490 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.490 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.490 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.490 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.490 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.491 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.491 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.491 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.491 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.491 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.491 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.491 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.491 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.491 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "stats", 2024-03-28T16:09:40.491 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:40.491 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:40.491 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.491 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.491 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.491 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.492 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.492 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.492 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.492 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.492 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.492 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.492 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.492 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.492 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.492 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.492 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.492 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.492 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.492 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.492 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.493 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.493 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.493 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.493 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.493 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.493 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.493 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.493 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.493 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.493 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.493 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.493 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.493 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.493 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.494 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.494 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.494 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.494 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:40.494 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:40.494 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.494 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.494 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.494 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:40.494 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.494 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.494 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.494 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.494 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.495 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.495 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.495 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.495 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.495 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.495 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.495 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.495 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.495 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.495 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.495 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:40.495 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:40.495 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.495 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.495 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.496 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.496 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.496 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.496 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.496 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.496 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.496 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.496 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.496 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.496 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.496 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.496 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.496 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "status", 2024-03-28T16:09:40.496 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:40.496 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:40.497 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.497 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.497 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.497 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.497 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.497 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.497 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.497 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.497 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.497 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.497 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.497 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.497 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.497 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.497 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.497 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.498 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.498 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.498 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.498 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.498 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.498 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.498 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.498 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.498 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.498 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.498 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.498 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.498 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.498 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.498 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.499 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.499 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.499 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.499 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.499 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.499 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:40.499 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:40.499 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.499 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.499 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.499 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:40.499 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.499 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.499 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.499 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.500 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.500 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.500 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.500 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.500 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.500 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.500 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.500 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.500 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.500 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.500 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.500 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:40.500 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:40.500 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.500 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.500 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.501 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.501 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.501 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.501 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.501 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.501 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.501 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.501 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.501 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.501 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.501 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.501 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.501 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "telegraf", 2024-03-28T16:09:40.501 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:40.501 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:40.502 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.502 INFO:teuthology.orchestra.run.smithi067.stdout: "address": { 2024-03-28T16:09:40.502 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "address", 2024-03-28T16:09:40.502 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.502 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.502 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.502 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-28T16:09:40.502 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.502 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.502 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.502 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.502 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.502 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.502 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.503 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.503 INFO:teuthology.orchestra.run.smithi067.stdout: "interval": { 2024-03-28T16:09:40.503 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "interval", 2024-03-28T16:09:40.503 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "secs", 2024-03-28T16:09:40.503 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.503 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.503 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "15", 2024-03-28T16:09:40.503 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.503 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.503 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.503 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.503 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.503 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.503 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.503 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.504 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.504 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.504 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.504 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.504 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.504 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.504 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.504 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.504 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.504 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.504 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.504 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.504 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.504 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.504 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.505 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.505 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.505 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.505 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.505 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.505 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.505 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.505 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.505 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.505 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.505 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.505 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.505 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.505 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.505 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.506 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.506 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.506 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.506 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.506 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.506 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:40.506 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:40.506 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.506 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.506 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.506 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:40.506 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.506 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.506 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.507 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.507 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.507 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.507 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.507 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.507 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.507 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.507 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.507 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.507 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.507 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.507 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.507 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:40.507 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:40.507 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.508 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.508 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.508 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.508 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.508 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.508 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.508 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.508 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.508 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.508 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.508 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.508 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.508 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.508 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.509 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "telemetry", 2024-03-28T16:09:40.509 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:40.509 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:40.509 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.509 INFO:teuthology.orchestra.run.smithi067.stdout: "channel_basic": { 2024-03-28T16:09:40.509 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "channel_basic", 2024-03-28T16:09:40.509 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.509 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.509 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.509 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.509 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.509 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.509 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.509 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-28T16:09:40.509 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.510 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.510 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.510 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.510 INFO:teuthology.orchestra.run.smithi067.stdout: "channel_crash": { 2024-03-28T16:09:40.510 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "channel_crash", 2024-03-28T16:09:40.510 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.510 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.510 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.510 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.510 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.510 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.510 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.510 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-28T16:09:40.510 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.511 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.511 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.511 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.511 INFO:teuthology.orchestra.run.smithi067.stdout: "channel_device": { 2024-03-28T16:09:40.511 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "channel_device", 2024-03-28T16:09:40.511 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.511 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.511 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.511 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.511 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.511 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.511 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.511 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-28T16:09:40.511 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.511 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.511 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.512 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.512 INFO:teuthology.orchestra.run.smithi067.stdout: "channel_ident": { 2024-03-28T16:09:40.512 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "channel_ident", 2024-03-28T16:09:40.512 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.512 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.512 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.512 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.512 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.512 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.512 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.512 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-28T16:09:40.512 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.512 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.512 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.512 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.513 INFO:teuthology.orchestra.run.smithi067.stdout: "channel_perf": { 2024-03-28T16:09:40.513 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "channel_perf", 2024-03-28T16:09:40.513 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.513 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.513 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.513 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.513 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.513 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.513 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.513 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-28T16:09:40.513 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.513 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.513 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.513 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.513 INFO:teuthology.orchestra.run.smithi067.stdout: "contact": { 2024-03-28T16:09:40.514 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "contact", 2024-03-28T16:09:40.514 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.514 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.514 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.514 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.514 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.514 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.514 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.514 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.514 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.514 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.514 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.514 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.514 INFO:teuthology.orchestra.run.smithi067.stdout: "description": { 2024-03-28T16:09:40.514 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "description", 2024-03-28T16:09:40.515 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.515 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.515 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.515 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.515 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.515 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.515 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.515 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.515 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.515 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.515 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.515 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.515 INFO:teuthology.orchestra.run.smithi067.stdout: "device_url": { 2024-03-28T16:09:40.515 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "device_url", 2024-03-28T16:09:40.516 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.516 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.516 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.516 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-28T16:09:40.516 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.516 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.516 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.516 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.516 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.516 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.516 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.516 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.516 INFO:teuthology.orchestra.run.smithi067.stdout: "enabled": { 2024-03-28T16:09:40.516 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "enabled", 2024-03-28T16:09:40.516 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.517 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.517 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.517 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.517 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.517 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.517 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.517 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.517 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.517 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.517 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.517 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.517 INFO:teuthology.orchestra.run.smithi067.stdout: "interval": { 2024-03-28T16:09:40.517 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "interval", 2024-03-28T16:09:40.517 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.517 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.518 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.518 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "24", 2024-03-28T16:09:40.518 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "8", 2024-03-28T16:09:40.518 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.518 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.518 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.518 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.518 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.518 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.518 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.518 INFO:teuthology.orchestra.run.smithi067.stdout: "last_opt_revision": { 2024-03-28T16:09:40.518 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "last_opt_revision", 2024-03-28T16:09:40.518 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.518 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.518 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.519 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "1", 2024-03-28T16:09:40.519 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.519 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.519 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.519 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.519 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.519 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.519 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.519 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.519 INFO:teuthology.orchestra.run.smithi067.stdout: "leaderboard": { 2024-03-28T16:09:40.519 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "leaderboard", 2024-03-28T16:09:40.519 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.519 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.519 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.519 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.519 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.520 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.520 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.520 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.520 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.520 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.520 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.520 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.520 INFO:teuthology.orchestra.run.smithi067.stdout: "leaderboard_description": { 2024-03-28T16:09:40.520 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "leaderboard_description", 2024-03-28T16:09:40.520 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.520 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.520 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.520 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.520 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.520 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.521 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.521 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.521 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.521 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.521 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.521 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.521 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.521 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.521 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.521 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.521 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.521 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.521 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.521 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.521 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.522 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.522 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.522 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.522 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.522 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.522 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.522 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.522 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.522 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.522 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.522 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.522 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.522 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.522 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.522 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.523 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.523 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.523 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.523 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.523 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.523 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.523 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.523 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.523 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.523 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.523 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.523 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:40.523 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:40.523 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.523 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.523 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.524 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:40.524 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.524 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.524 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.524 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.524 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.524 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.524 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.524 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.524 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.524 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.524 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.524 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.524 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.525 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.525 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.525 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:40.525 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:40.525 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.525 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.525 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.525 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.525 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.525 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.525 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.525 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.525 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.525 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.525 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.526 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.526 INFO:teuthology.orchestra.run.smithi067.stdout: "organization": { 2024-03-28T16:09:40.526 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "organization", 2024-03-28T16:09:40.526 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.526 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.526 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.526 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.526 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.526 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.526 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.526 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.526 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.526 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.526 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.526 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.526 INFO:teuthology.orchestra.run.smithi067.stdout: "proxy": { 2024-03-28T16:09:40.527 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "proxy", 2024-03-28T16:09:40.527 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.527 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.527 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.527 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.527 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.527 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.527 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.527 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.527 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.527 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.527 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.527 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.527 INFO:teuthology.orchestra.run.smithi067.stdout: "url": { 2024-03-28T16:09:40.527 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "url", 2024-03-28T16:09:40.528 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.528 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.528 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.528 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-28T16:09:40.528 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.528 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.528 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.528 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.528 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.528 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.528 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.528 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.528 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.528 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.528 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.529 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "test_orchestrator", 2024-03-28T16:09:40.529 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:40.529 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:40.529 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.529 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.529 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.529 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.529 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.529 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.529 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.529 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.529 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.529 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.529 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.529 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.530 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.530 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.530 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.530 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.530 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.530 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.530 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.530 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.530 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.530 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.530 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.530 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.530 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.530 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.531 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.531 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.531 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.531 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.531 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.531 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.531 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.531 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.531 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.531 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.531 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:40.531 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:40.531 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.531 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.531 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.532 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:40.532 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.532 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.532 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.532 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.532 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.532 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.532 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.532 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.532 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.532 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.532 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.532 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.532 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.532 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.533 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.533 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:40.533 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:40.533 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.533 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.533 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.533 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.533 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.533 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.533 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.533 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.533 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.533 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.533 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.534 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.534 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.534 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.534 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.534 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "volumes", 2024-03-28T16:09:40.534 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:40.534 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:40.534 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.534 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.534 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.534 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.534 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.534 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.534 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.535 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.535 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.535 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.535 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.535 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.535 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.535 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.535 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.535 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.535 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.535 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.535 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.535 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.535 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.535 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.536 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.536 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.536 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.536 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.536 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.536 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.536 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.536 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.536 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.536 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.536 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.536 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.536 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.536 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.536 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:40.537 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:40.537 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.537 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.537 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.537 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:40.537 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.537 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.537 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.537 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.537 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.537 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.537 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.537 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.537 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.537 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.538 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.538 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.538 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.538 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.538 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.538 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:40.538 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:40.538 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.538 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.538 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.538 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.538 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.538 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.538 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.538 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.539 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.539 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.539 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.539 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.539 INFO:teuthology.orchestra.run.smithi067.stdout: "max_concurrent_clones": { 2024-03-28T16:09:40.539 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "max_concurrent_clones", 2024-03-28T16:09:40.539 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.539 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.539 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.539 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "4", 2024-03-28T16:09:40.539 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.539 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.539 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.539 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-28T16:09:40.539 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.540 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.540 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.540 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.540 INFO:teuthology.orchestra.run.smithi067.stdout: "periodic_async_work": { 2024-03-28T16:09:40.540 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "periodic_async_work", 2024-03-28T16:09:40.540 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.540 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.540 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.540 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.540 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.540 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.540 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.540 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Periodically check for async work", 2024-03-28T16:09:40.540 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.540 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.541 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.541 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.541 INFO:teuthology.orchestra.run.smithi067.stdout: "snapshot_clone_delay": { 2024-03-28T16:09:40.541 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "snapshot_clone_delay", 2024-03-28T16:09:40.541 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.541 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.541 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.541 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "0", 2024-03-28T16:09:40.541 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.541 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.541 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.541 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-28T16:09:40.541 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.541 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.541 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.542 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.542 INFO:teuthology.orchestra.run.smithi067.stdout: "snapshot_clone_no_wait": { 2024-03-28T16:09:40.542 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "snapshot_clone_no_wait", 2024-03-28T16:09:40.542 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.542 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.542 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.542 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.542 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.542 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.542 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.542 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-28T16:09:40.542 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.542 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.543 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.543 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.543 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.543 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.543 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.543 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "zabbix", 2024-03-28T16:09:40.543 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:40.543 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:40.543 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.543 INFO:teuthology.orchestra.run.smithi067.stdout: "discovery_interval": { 2024-03-28T16:09:40.543 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "discovery_interval", 2024-03-28T16:09:40.543 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "uint", 2024-03-28T16:09:40.543 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.543 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.543 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "100", 2024-03-28T16:09:40.544 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.544 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.544 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.544 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.544 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.544 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.544 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.544 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.544 INFO:teuthology.orchestra.run.smithi067.stdout: "identifier": { 2024-03-28T16:09:40.544 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "identifier", 2024-03-28T16:09:40.544 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.544 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.544 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.544 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.544 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.545 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.545 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.545 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.545 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.545 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.545 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.545 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.545 INFO:teuthology.orchestra.run.smithi067.stdout: "interval": { 2024-03-28T16:09:40.545 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "interval", 2024-03-28T16:09:40.545 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "secs", 2024-03-28T16:09:40.545 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.545 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.545 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "60", 2024-03-28T16:09:40.545 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.545 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.546 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.546 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.546 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.546 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.546 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.546 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.546 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.546 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.546 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.546 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.546 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.546 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.546 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.546 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.547 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.547 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.547 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.547 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.547 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.547 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.547 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.547 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.547 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.547 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.547 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.547 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.547 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.547 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.547 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.548 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.548 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.548 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.548 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.548 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.548 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.548 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.548 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.548 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.548 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.548 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.548 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.548 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:40.548 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:40.548 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.549 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.549 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.549 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:40.549 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.549 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.549 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.549 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.549 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.549 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.549 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.549 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.549 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.549 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.549 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.550 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.550 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.550 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.550 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.550 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:40.550 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:40.550 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.550 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.550 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.550 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.550 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.550 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.550 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.550 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.550 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.550 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.551 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.551 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.551 INFO:teuthology.orchestra.run.smithi067.stdout: "zabbix_host": { 2024-03-28T16:09:40.551 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "zabbix_host", 2024-03-28T16:09:40.551 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.551 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.551 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.551 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.551 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.551 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.551 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.551 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.551 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.551 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.551 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.552 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.552 INFO:teuthology.orchestra.run.smithi067.stdout: "zabbix_port": { 2024-03-28T16:09:40.552 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "zabbix_port", 2024-03-28T16:09:40.552 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.552 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.552 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.552 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "10051", 2024-03-28T16:09:40.552 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.552 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.552 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.552 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.552 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.552 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.552 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.552 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.553 INFO:teuthology.orchestra.run.smithi067.stdout: "zabbix_sender": { 2024-03-28T16:09:40.553 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "zabbix_sender", 2024-03-28T16:09:40.553 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.553 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.553 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.553 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-28T16:09:40.553 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.553 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.553 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.553 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.553 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.553 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.553 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.553 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.553 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.554 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.554 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-28T16:09:40.554 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.554 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.554 INFO:teuthology.orchestra.run.smithi067.stdout: "gid": 34109, 2024-03-28T16:09:40.554 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "y", 2024-03-28T16:09:40.554 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_features": 4540701547738038271, 2024-03-28T16:09:40.554 INFO:teuthology.orchestra.run.smithi067.stdout: "available_modules": [ 2024-03-28T16:09:40.554 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.554 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "alerts", 2024-03-28T16:09:40.554 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:40.554 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:40.554 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.554 INFO:teuthology.orchestra.run.smithi067.stdout: "interval": { 2024-03-28T16:09:40.554 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "interval", 2024-03-28T16:09:40.555 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "secs", 2024-03-28T16:09:40.555 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.555 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.555 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "60", 2024-03-28T16:09:40.555 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.555 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.555 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.555 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "How frequently to reexamine health status", 2024-03-28T16:09:40.555 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.555 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.555 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.555 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.555 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.555 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.556 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.556 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.556 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.556 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.556 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.556 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.556 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.556 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.556 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.556 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.556 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.556 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.556 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.556 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.556 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.557 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.557 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.557 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.557 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.557 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.557 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.557 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.557 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.557 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.557 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.557 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.557 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.557 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.557 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.558 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.558 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.558 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.558 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.558 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:40.558 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:40.558 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.558 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.558 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.558 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:40.558 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.558 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.558 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.558 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.558 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.559 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.559 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.559 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.559 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.559 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.559 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.559 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.559 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.559 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.559 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.559 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:40.559 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:40.561 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.561 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.561 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.561 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.561 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.561 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.561 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.562 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.562 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.562 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.562 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.562 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.562 INFO:teuthology.orchestra.run.smithi067.stdout: "smtp_destination": { 2024-03-28T16:09:40.562 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "smtp_destination", 2024-03-28T16:09:40.562 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.562 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.562 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.562 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.562 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.562 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.562 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.562 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Email address to send alerts to", 2024-03-28T16:09:40.563 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.563 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.563 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.563 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.563 INFO:teuthology.orchestra.run.smithi067.stdout: "smtp_from_name": { 2024-03-28T16:09:40.563 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "smtp_from_name", 2024-03-28T16:09:40.563 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.563 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.563 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.563 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "Ceph", 2024-03-28T16:09:40.563 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.563 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.563 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.563 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Email From: name", 2024-03-28T16:09:40.563 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.564 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.564 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.564 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.564 INFO:teuthology.orchestra.run.smithi067.stdout: "smtp_host": { 2024-03-28T16:09:40.564 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "smtp_host", 2024-03-28T16:09:40.564 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.564 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.564 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.564 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.564 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.564 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.564 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.564 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "SMTP server", 2024-03-28T16:09:40.564 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.565 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.565 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.565 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.565 INFO:teuthology.orchestra.run.smithi067.stdout: "smtp_password": { 2024-03-28T16:09:40.565 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "smtp_password", 2024-03-28T16:09:40.565 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.565 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.565 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.565 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.565 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.565 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.565 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.565 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Password to authenticate with", 2024-03-28T16:09:40.565 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.565 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.565 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.566 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.566 INFO:teuthology.orchestra.run.smithi067.stdout: "smtp_port": { 2024-03-28T16:09:40.566 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "smtp_port", 2024-03-28T16:09:40.566 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.566 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.566 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.566 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "465", 2024-03-28T16:09:40.566 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.566 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.566 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.566 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "SMTP port", 2024-03-28T16:09:40.566 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.566 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.566 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.567 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.567 INFO:teuthology.orchestra.run.smithi067.stdout: "smtp_sender": { 2024-03-28T16:09:40.567 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "smtp_sender", 2024-03-28T16:09:40.567 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.567 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.567 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.567 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.567 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.567 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.567 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.567 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "SMTP envelope sender", 2024-03-28T16:09:40.567 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.567 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.567 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.568 INFO:teuthology.orchestra.run.smithi067.stdout: "smtp_ssl": { 2024-03-28T16:09:40.568 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "smtp_ssl", 2024-03-28T16:09:40.568 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.568 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.568 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.568 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.568 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.568 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.568 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.568 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-28T16:09:40.568 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.568 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.568 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.568 INFO:teuthology.orchestra.run.smithi067.stdout: "smtp_user": { 2024-03-28T16:09:40.569 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "smtp_user", 2024-03-28T16:09:40.569 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.569 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.569 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.569 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.569 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.569 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.569 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.569 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "User to authenticate as", 2024-03-28T16:09:40.569 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.569 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.569 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.569 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.569 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.570 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.570 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.570 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "balancer", 2024-03-28T16:09:40.570 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:40.570 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:40.570 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.570 INFO:teuthology.orchestra.run.smithi067.stdout: "active": { 2024-03-28T16:09:40.570 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "active", 2024-03-28T16:09:40.570 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.570 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.570 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.570 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.570 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.570 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.571 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.571 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "automatically balance PGs across cluster", 2024-03-28T16:09:40.571 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.571 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.571 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.571 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.571 INFO:teuthology.orchestra.run.smithi067.stdout: "begin_time": { 2024-03-28T16:09:40.571 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "begin_time", 2024-03-28T16:09:40.571 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.571 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.571 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.571 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "0000", 2024-03-28T16:09:40.571 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.571 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.572 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.572 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "beginning time of day to automatically balance", 2024-03-28T16:09:40.572 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-28T16:09:40.572 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.572 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.572 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.572 INFO:teuthology.orchestra.run.smithi067.stdout: "begin_weekday": { 2024-03-28T16:09:40.572 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "begin_weekday", 2024-03-28T16:09:40.572 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "uint", 2024-03-28T16:09:40.572 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.572 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.572 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "0", 2024-03-28T16:09:40.572 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "0", 2024-03-28T16:09:40.572 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "6", 2024-03-28T16:09:40.573 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.573 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-28T16:09:40.573 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-28T16:09:40.573 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.573 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.573 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.573 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_compat_max_iterations": { 2024-03-28T16:09:40.573 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "crush_compat_max_iterations", 2024-03-28T16:09:40.573 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "uint", 2024-03-28T16:09:40.573 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.573 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.573 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "25", 2024-03-28T16:09:40.573 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "1", 2024-03-28T16:09:40.573 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "250", 2024-03-28T16:09:40.574 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.574 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-28T16:09:40.574 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.574 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.574 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.574 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.574 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_compat_metrics": { 2024-03-28T16:09:40.574 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "crush_compat_metrics", 2024-03-28T16:09:40.574 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.574 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.574 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.574 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "pgs,objects,bytes", 2024-03-28T16:09:40.574 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.574 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.575 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.575 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-28T16:09:40.575 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-28T16:09:40.575 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.575 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.575 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.575 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_compat_step": { 2024-03-28T16:09:40.575 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "crush_compat_step", 2024-03-28T16:09:40.575 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "float", 2024-03-28T16:09:40.575 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.575 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.575 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "0.5", 2024-03-28T16:09:40.575 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "0.001", 2024-03-28T16:09:40.575 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "0.999", 2024-03-28T16:09:40.576 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.576 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "aggressiveness of optimization", 2024-03-28T16:09:40.576 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-28T16:09:40.576 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.576 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.576 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.576 INFO:teuthology.orchestra.run.smithi067.stdout: "end_time": { 2024-03-28T16:09:40.576 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "end_time", 2024-03-28T16:09:40.576 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.576 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.576 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.576 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "2359", 2024-03-28T16:09:40.576 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.576 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.576 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.577 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "ending time of day to automatically balance", 2024-03-28T16:09:40.577 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-28T16:09:40.577 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.577 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.577 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.577 INFO:teuthology.orchestra.run.smithi067.stdout: "end_weekday": { 2024-03-28T16:09:40.577 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "end_weekday", 2024-03-28T16:09:40.577 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "uint", 2024-03-28T16:09:40.577 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.577 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.577 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "0", 2024-03-28T16:09:40.577 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "0", 2024-03-28T16:09:40.577 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "6", 2024-03-28T16:09:40.577 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.578 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-28T16:09:40.578 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-28T16:09:40.578 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.578 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.578 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.578 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.578 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.578 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.578 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.578 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.578 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.578 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.578 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.578 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.579 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.579 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.579 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.579 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.579 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.579 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.579 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.579 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.579 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.579 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.579 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.579 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.579 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.579 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.579 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.580 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.580 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.580 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.580 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.580 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.580 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.580 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.580 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.580 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.580 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.580 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.580 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:40.580 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:40.580 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.581 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.581 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.581 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:40.581 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.581 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.581 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.581 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.581 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.581 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.581 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.581 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.581 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.581 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.581 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.582 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.582 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.582 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.582 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.582 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:40.582 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:40.582 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.582 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.582 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.582 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.582 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.582 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.582 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.582 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.582 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.583 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.583 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.583 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.583 INFO:teuthology.orchestra.run.smithi067.stdout: "min_score": { 2024-03-28T16:09:40.583 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "min_score", 2024-03-28T16:09:40.583 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "float", 2024-03-28T16:09:40.583 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.583 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.583 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "0", 2024-03-28T16:09:40.583 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.583 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.583 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.583 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-28T16:09:40.583 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.583 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.584 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.584 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.584 INFO:teuthology.orchestra.run.smithi067.stdout: "mode": { 2024-03-28T16:09:40.584 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "mode", 2024-03-28T16:09:40.584 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.584 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.584 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.584 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "upmap", 2024-03-28T16:09:40.584 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.584 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.584 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.584 INFO:teuthology.orchestra.run.smithi067.stdout: "crush-compat", 2024-03-28T16:09:40.584 INFO:teuthology.orchestra.run.smithi067.stdout: "none", 2024-03-28T16:09:40.584 INFO:teuthology.orchestra.run.smithi067.stdout: "read", 2024-03-28T16:09:40.585 INFO:teuthology.orchestra.run.smithi067.stdout: "upmap", 2024-03-28T16:09:40.585 INFO:teuthology.orchestra.run.smithi067.stdout: "upmap-read" 2024-03-28T16:09:40.585 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.585 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Balancer mode", 2024-03-28T16:09:40.585 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.585 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.585 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.585 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.585 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_ids": { 2024-03-28T16:09:40.585 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "pool_ids", 2024-03-28T16:09:40.585 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.585 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.585 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.585 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.585 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.586 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.586 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.586 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-28T16:09:40.586 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.586 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.586 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.586 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.586 INFO:teuthology.orchestra.run.smithi067.stdout: "sleep_interval": { 2024-03-28T16:09:40.586 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "sleep_interval", 2024-03-28T16:09:40.586 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "secs", 2024-03-28T16:09:40.586 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.586 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.586 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "60", 2024-03-28T16:09:40.587 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.587 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.587 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.587 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-28T16:09:40.587 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.587 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.587 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.587 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.587 INFO:teuthology.orchestra.run.smithi067.stdout: "upmap_max_deviation": { 2024-03-28T16:09:40.587 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "upmap_max_deviation", 2024-03-28T16:09:40.587 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.587 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.587 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.587 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "5", 2024-03-28T16:09:40.587 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "1", 2024-03-28T16:09:40.588 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.588 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.588 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-28T16:09:40.588 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-28T16:09:40.588 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.588 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.588 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.588 INFO:teuthology.orchestra.run.smithi067.stdout: "upmap_max_optimizations": { 2024-03-28T16:09:40.588 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "upmap_max_optimizations", 2024-03-28T16:09:40.588 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "uint", 2024-03-28T16:09:40.588 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.588 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.588 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "10", 2024-03-28T16:09:40.588 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.589 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.589 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.589 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-28T16:09:40.589 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.589 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.589 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.589 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.589 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.589 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.589 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.589 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "cephadm", 2024-03-28T16:09:40.589 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:40.589 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:40.589 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.589 INFO:teuthology.orchestra.run.smithi067.stdout: "agent_down_multiplier": { 2024-03-28T16:09:40.590 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "agent_down_multiplier", 2024-03-28T16:09:40.590 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "float", 2024-03-28T16:09:40.590 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.590 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.590 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "3.0", 2024-03-28T16:09:40.590 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.590 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.590 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.590 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-28T16:09:40.590 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.590 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.590 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.590 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.590 INFO:teuthology.orchestra.run.smithi067.stdout: "agent_refresh_rate": { 2024-03-28T16:09:40.590 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "agent_refresh_rate", 2024-03-28T16:09:40.591 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "secs", 2024-03-28T16:09:40.591 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.591 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.591 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "20", 2024-03-28T16:09:40.591 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.591 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.591 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.591 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-28T16:09:40.591 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.591 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.591 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.591 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.591 INFO:teuthology.orchestra.run.smithi067.stdout: "agent_starting_port": { 2024-03-28T16:09:40.591 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "agent_starting_port", 2024-03-28T16:09:40.591 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.592 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.592 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.592 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "4721", 2024-03-28T16:09:40.592 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.592 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.592 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.592 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-28T16:09:40.592 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.592 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.592 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.592 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.592 INFO:teuthology.orchestra.run.smithi067.stdout: "allow_ptrace": { 2024-03-28T16:09:40.592 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "allow_ptrace", 2024-03-28T16:09:40.592 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.592 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.593 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.593 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.593 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.593 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.593 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.593 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-28T16:09:40.593 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-28T16:09:40.593 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.593 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.593 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.593 INFO:teuthology.orchestra.run.smithi067.stdout: "autotune_interval": { 2024-03-28T16:09:40.593 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "autotune_interval", 2024-03-28T16:09:40.593 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "secs", 2024-03-28T16:09:40.593 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.594 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.594 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "600", 2024-03-28T16:09:40.594 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.594 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.594 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.594 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-28T16:09:40.594 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.594 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.594 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.594 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.594 INFO:teuthology.orchestra.run.smithi067.stdout: "autotune_memory_target_ratio": { 2024-03-28T16:09:40.594 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "autotune_memory_target_ratio", 2024-03-28T16:09:40.594 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "float", 2024-03-28T16:09:40.594 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.595 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.595 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "0.7", 2024-03-28T16:09:40.595 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.595 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.595 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.595 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-28T16:09:40.595 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.595 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.595 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.595 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.595 INFO:teuthology.orchestra.run.smithi067.stdout: "cephadm_log_destination": { 2024-03-28T16:09:40.595 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "cephadm_log_destination", 2024-03-28T16:09:40.595 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.595 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.595 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.596 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.596 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.596 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.596 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.596 INFO:teuthology.orchestra.run.smithi067.stdout: "file", 2024-03-28T16:09:40.596 INFO:teuthology.orchestra.run.smithi067.stdout: "file,syslog", 2024-03-28T16:09:40.596 INFO:teuthology.orchestra.run.smithi067.stdout: "syslog" 2024-03-28T16:09:40.596 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.596 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-28T16:09:40.596 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.596 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.596 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.596 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.596 INFO:teuthology.orchestra.run.smithi067.stdout: "cgroups_split": { 2024-03-28T16:09:40.597 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "cgroups_split", 2024-03-28T16:09:40.597 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.597 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.597 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.597 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.597 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.597 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.597 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.597 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-28T16:09:40.597 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.597 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.597 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.597 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.597 INFO:teuthology.orchestra.run.smithi067.stdout: "config_checks_enabled": { 2024-03-28T16:09:40.598 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "config_checks_enabled", 2024-03-28T16:09:40.598 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.598 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.598 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.598 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.598 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.598 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.598 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.598 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-28T16:09:40.598 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.598 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.598 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.598 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.598 INFO:teuthology.orchestra.run.smithi067.stdout: "config_dashboard": { 2024-03-28T16:09:40.599 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "config_dashboard", 2024-03-28T16:09:40.599 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.599 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.599 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.599 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.599 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.599 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.599 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.599 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-28T16:09:40.599 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.599 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.599 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.599 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.599 INFO:teuthology.orchestra.run.smithi067.stdout: "container_image_alertmanager": { 2024-03-28T16:09:40.599 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "container_image_alertmanager", 2024-03-28T16:09:40.600 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.600 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.600 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.600 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-28T16:09:40.600 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.600 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.600 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.600 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Prometheus container image", 2024-03-28T16:09:40.600 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.600 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.600 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.600 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.600 INFO:teuthology.orchestra.run.smithi067.stdout: "container_image_base": { 2024-03-28T16:09:40.600 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "container_image_base", 2024-03-28T16:09:40.600 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.601 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.601 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.601 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-28T16:09:40.601 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.601 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.601 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.601 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Container image name, without the tag", 2024-03-28T16:09:40.601 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.601 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.601 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.601 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.601 INFO:teuthology.orchestra.run.smithi067.stdout: "container_image_elasticsearch": { 2024-03-28T16:09:40.601 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "container_image_elasticsearch", 2024-03-28T16:09:40.601 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.601 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.601 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.602 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-28T16:09:40.602 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.602 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.602 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.602 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "elasticsearch container image", 2024-03-28T16:09:40.602 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.602 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.602 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.602 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.602 INFO:teuthology.orchestra.run.smithi067.stdout: "container_image_grafana": { 2024-03-28T16:09:40.602 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "container_image_grafana", 2024-03-28T16:09:40.602 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.602 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.602 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.602 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-28T16:09:40.603 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.603 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.603 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.603 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Prometheus container image", 2024-03-28T16:09:40.603 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.603 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.603 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.603 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.603 INFO:teuthology.orchestra.run.smithi067.stdout: "container_image_haproxy": { 2024-03-28T16:09:40.603 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "container_image_haproxy", 2024-03-28T16:09:40.603 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.603 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.603 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.603 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-28T16:09:40.603 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.604 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.604 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.604 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "HAproxy container image", 2024-03-28T16:09:40.604 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.604 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.604 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.604 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.604 INFO:teuthology.orchestra.run.smithi067.stdout: "container_image_jaeger_agent": { 2024-03-28T16:09:40.604 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "container_image_jaeger_agent", 2024-03-28T16:09:40.604 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.604 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.604 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.604 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-28T16:09:40.604 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.605 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.605 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.605 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Jaeger agent container image", 2024-03-28T16:09:40.605 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.605 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.605 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.605 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.605 INFO:teuthology.orchestra.run.smithi067.stdout: "container_image_jaeger_collector": { 2024-03-28T16:09:40.605 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "container_image_jaeger_collector", 2024-03-28T16:09:40.605 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.605 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.605 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.605 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-28T16:09:40.605 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.606 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.606 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.606 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Jaeger collector container image", 2024-03-28T16:09:40.606 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.606 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.606 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.606 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.606 INFO:teuthology.orchestra.run.smithi067.stdout: "container_image_jaeger_query": { 2024-03-28T16:09:40.606 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "container_image_jaeger_query", 2024-03-28T16:09:40.606 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.606 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.606 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.606 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-28T16:09:40.606 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.607 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.607 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.607 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Jaeger query container image", 2024-03-28T16:09:40.607 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.607 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.607 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.607 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.607 INFO:teuthology.orchestra.run.smithi067.stdout: "container_image_keepalived": { 2024-03-28T16:09:40.607 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "container_image_keepalived", 2024-03-28T16:09:40.607 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.607 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.607 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.607 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-28T16:09:40.607 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.608 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.608 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.608 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Keepalived container image", 2024-03-28T16:09:40.608 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.608 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.608 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.608 INFO:teuthology.orchestra.run.smithi067.stdout: "container_image_loki": { 2024-03-28T16:09:40.608 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "container_image_loki", 2024-03-28T16:09:40.608 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.608 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.608 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.608 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-28T16:09:40.608 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.608 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.609 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.609 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Loki container image", 2024-03-28T16:09:40.609 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.609 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.609 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.609 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.609 INFO:teuthology.orchestra.run.smithi067.stdout: "container_image_node_exporter": { 2024-03-28T16:09:40.609 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "container_image_node_exporter", 2024-03-28T16:09:40.609 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.609 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.609 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.609 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-28T16:09:40.609 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.609 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.609 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.610 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Prometheus container image", 2024-03-28T16:09:40.610 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.610 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.610 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.610 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.610 INFO:teuthology.orchestra.run.smithi067.stdout: "container_image_nvmeof": { 2024-03-28T16:09:40.610 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "container_image_nvmeof", 2024-03-28T16:09:40.610 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.610 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.610 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.610 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-28T16:09:40.610 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.610 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.610 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.611 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Nvme-of container image", 2024-03-28T16:09:40.611 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.611 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.611 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.611 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.611 INFO:teuthology.orchestra.run.smithi067.stdout: "container_image_prometheus": { 2024-03-28T16:09:40.611 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "container_image_prometheus", 2024-03-28T16:09:40.611 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.611 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.611 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.611 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-28T16:09:40.611 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.611 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.611 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.611 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Prometheus container image", 2024-03-28T16:09:40.612 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.612 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.612 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.612 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.612 INFO:teuthology.orchestra.run.smithi067.stdout: "container_image_promtail": { 2024-03-28T16:09:40.612 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "container_image_promtail", 2024-03-28T16:09:40.612 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.612 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.612 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.612 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-28T16:09:40.612 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.612 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.612 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.612 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Promtail container image", 2024-03-28T16:09:40.613 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.613 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.613 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.613 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.613 INFO:teuthology.orchestra.run.smithi067.stdout: "container_image_snmp_gateway": { 2024-03-28T16:09:40.613 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "container_image_snmp_gateway", 2024-03-28T16:09:40.613 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.613 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.613 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.613 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-28T16:09:40.613 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.613 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.613 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.613 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "SNMP Gateway container image", 2024-03-28T16:09:40.613 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.614 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.614 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.614 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.614 INFO:teuthology.orchestra.run.smithi067.stdout: "container_init": { 2024-03-28T16:09:40.614 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "container_init", 2024-03-28T16:09:40.614 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.614 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.614 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.614 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.614 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.614 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.614 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.614 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Run podman/docker with `--init`", 2024-03-28T16:09:40.614 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.614 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.615 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.615 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.615 INFO:teuthology.orchestra.run.smithi067.stdout: "daemon_cache_timeout": { 2024-03-28T16:09:40.615 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "daemon_cache_timeout", 2024-03-28T16:09:40.615 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "secs", 2024-03-28T16:09:40.615 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.615 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.615 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "600", 2024-03-28T16:09:40.615 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.615 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.615 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.615 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-28T16:09:40.615 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.615 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.615 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.615 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.616 INFO:teuthology.orchestra.run.smithi067.stdout: "default_cephadm_command_timeout": { 2024-03-28T16:09:40.616 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "default_cephadm_command_timeout", 2024-03-28T16:09:40.616 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.616 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.616 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.616 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "900", 2024-03-28T16:09:40.616 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.616 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.616 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.616 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-28T16:09:40.616 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.616 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.616 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.616 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.617 INFO:teuthology.orchestra.run.smithi067.stdout: "default_registry": { 2024-03-28T16:09:40.617 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "default_registry", 2024-03-28T16:09:40.617 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.617 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.617 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.617 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "docker.io", 2024-03-28T16:09:40.617 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.617 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.617 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.617 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-28T16:09:40.617 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.617 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.617 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.617 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.617 INFO:teuthology.orchestra.run.smithi067.stdout: "device_cache_timeout": { 2024-03-28T16:09:40.618 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "device_cache_timeout", 2024-03-28T16:09:40.618 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "secs", 2024-03-28T16:09:40.618 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.618 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.618 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "1800", 2024-03-28T16:09:40.618 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.618 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.618 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.618 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "seconds to cache device inventory", 2024-03-28T16:09:40.618 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.618 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.618 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.618 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.618 INFO:teuthology.orchestra.run.smithi067.stdout: "device_enhanced_scan": { 2024-03-28T16:09:40.619 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "device_enhanced_scan", 2024-03-28T16:09:40.619 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.619 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.619 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.619 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.619 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.619 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.619 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.619 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-28T16:09:40.619 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.619 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.619 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.619 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.619 INFO:teuthology.orchestra.run.smithi067.stdout: "facts_cache_timeout": { 2024-03-28T16:09:40.619 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "facts_cache_timeout", 2024-03-28T16:09:40.620 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "secs", 2024-03-28T16:09:40.620 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.620 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.620 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "60", 2024-03-28T16:09:40.620 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.620 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.620 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.620 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "seconds to cache host facts data", 2024-03-28T16:09:40.620 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.620 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.620 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.620 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.620 INFO:teuthology.orchestra.run.smithi067.stdout: "host_check_interval": { 2024-03-28T16:09:40.620 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "host_check_interval", 2024-03-28T16:09:40.620 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "secs", 2024-03-28T16:09:40.621 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.621 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.621 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "600", 2024-03-28T16:09:40.621 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.621 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.621 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.621 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "how frequently to perform a host check", 2024-03-28T16:09:40.621 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.621 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.621 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.621 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.621 INFO:teuthology.orchestra.run.smithi067.stdout: "hw_monitoring": { 2024-03-28T16:09:40.621 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "hw_monitoring", 2024-03-28T16:09:40.621 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.621 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.622 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.622 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.622 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.622 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.622 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.622 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-28T16:09:40.622 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.622 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.622 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.622 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.622 INFO:teuthology.orchestra.run.smithi067.stdout: "inventory_list_all": { 2024-03-28T16:09:40.622 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "inventory_list_all", 2024-03-28T16:09:40.622 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.622 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.623 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.623 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.623 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.623 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.623 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.623 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-28T16:09:40.623 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.623 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.623 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.623 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.623 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.623 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.623 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.623 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.623 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.623 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.624 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.624 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.624 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.624 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.624 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.624 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.624 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.624 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.624 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.624 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.624 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.624 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.624 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.624 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.625 INFO:teuthology.orchestra.run.smithi067.stdout: "log_refresh_metadata": { 2024-03-28T16:09:40.625 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_refresh_metadata", 2024-03-28T16:09:40.625 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.625 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.625 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.625 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.625 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.625 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.625 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.625 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-28T16:09:40.625 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.625 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.625 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.626 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.626 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.626 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.626 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.626 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.626 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.626 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.626 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.626 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.626 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-28T16:09:40.626 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.626 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.626 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.626 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.627 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:40.627 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:40.627 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.627 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.627 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.627 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:40.627 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.627 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.627 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.627 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.627 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.627 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.627 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.627 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.627 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.628 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.628 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.628 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.628 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.628 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.628 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.628 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:40.628 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:40.628 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.628 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.628 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.628 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.628 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.628 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.628 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.629 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.629 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.629 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.629 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.629 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.629 INFO:teuthology.orchestra.run.smithi067.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-28T16:09:40.629 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-28T16:09:40.629 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.629 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.629 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.629 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.629 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.629 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.629 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.629 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-28T16:09:40.630 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.630 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.630 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.630 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.630 INFO:teuthology.orchestra.run.smithi067.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-28T16:09:40.630 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-28T16:09:40.630 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.630 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.630 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.630 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "*", 2024-03-28T16:09:40.630 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.630 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.630 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.630 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-28T16:09:40.631 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.631 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.631 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.631 INFO:teuthology.orchestra.run.smithi067.stdout: "max_count_per_host": { 2024-03-28T16:09:40.631 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "max_count_per_host", 2024-03-28T16:09:40.631 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.631 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.631 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.631 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "10", 2024-03-28T16:09:40.631 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.631 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.631 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.631 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "max number of daemons per service per host", 2024-03-28T16:09:40.632 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.632 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.632 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.632 INFO:teuthology.orchestra.run.smithi067.stdout: "max_osd_draining_count": { 2024-03-28T16:09:40.632 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "max_osd_draining_count", 2024-03-28T16:09:40.632 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.632 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.632 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.632 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "10", 2024-03-28T16:09:40.632 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.632 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.632 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.632 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-28T16:09:40.633 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.633 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.633 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.633 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.633 INFO:teuthology.orchestra.run.smithi067.stdout: "migration_current": { 2024-03-28T16:09:40.633 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "migration_current", 2024-03-28T16:09:40.633 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.633 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.633 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.633 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.633 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.633 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.633 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.633 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "internal - do not modify", 2024-03-28T16:09:40.634 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.634 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.634 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.634 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.634 INFO:teuthology.orchestra.run.smithi067.stdout: "mode": { 2024-03-28T16:09:40.634 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "mode", 2024-03-28T16:09:40.634 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.634 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.634 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.634 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "root", 2024-03-28T16:09:40.634 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.634 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.634 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.634 INFO:teuthology.orchestra.run.smithi067.stdout: "cephadm-package", 2024-03-28T16:09:40.635 INFO:teuthology.orchestra.run.smithi067.stdout: "root" 2024-03-28T16:09:40.635 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.635 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "mode for remote execution of cephadm", 2024-03-28T16:09:40.635 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.635 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.635 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.635 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.635 INFO:teuthology.orchestra.run.smithi067.stdout: "oob_default_addr": { 2024-03-28T16:09:40.635 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "oob_default_addr", 2024-03-28T16:09:40.635 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.635 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.635 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.635 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "169.254.1.1", 2024-03-28T16:09:40.635 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.635 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.635 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.636 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-28T16:09:40.636 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.636 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.636 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.636 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.636 INFO:teuthology.orchestra.run.smithi067.stdout: "prometheus_alerts_path": { 2024-03-28T16:09:40.636 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "prometheus_alerts_path", 2024-03-28T16:09:40.636 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.636 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.636 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.636 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-28T16:09:40.636 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.636 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.636 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.636 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-28T16:09:40.637 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.637 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.637 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.637 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.637 INFO:teuthology.orchestra.run.smithi067.stdout: "registry_insecure": { 2024-03-28T16:09:40.637 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "registry_insecure", 2024-03-28T16:09:40.637 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.637 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.637 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.637 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.637 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.637 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.637 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.637 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-28T16:09:40.637 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.638 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.638 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.638 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.638 INFO:teuthology.orchestra.run.smithi067.stdout: "registry_password": { 2024-03-28T16:09:40.638 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "registry_password", 2024-03-28T16:09:40.638 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.638 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.638 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.638 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.638 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.638 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.638 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.638 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-28T16:09:40.638 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.638 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.639 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.639 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.639 INFO:teuthology.orchestra.run.smithi067.stdout: "registry_url": { 2024-03-28T16:09:40.639 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "registry_url", 2024-03-28T16:09:40.639 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.639 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.639 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.639 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.639 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.639 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.639 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.639 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-28T16:09:40.639 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.639 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.640 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.640 INFO:teuthology.orchestra.run.smithi067.stdout: "registry_username": { 2024-03-28T16:09:40.640 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "registry_username", 2024-03-28T16:09:40.640 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.640 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.640 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.640 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.640 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.640 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.640 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.640 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-28T16:09:40.640 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.640 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.640 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.641 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.641 INFO:teuthology.orchestra.run.smithi067.stdout: "secure_monitoring_stack": { 2024-03-28T16:09:40.641 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "secure_monitoring_stack", 2024-03-28T16:09:40.641 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.641 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.641 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.641 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.641 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.641 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.641 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.641 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-28T16:09:40.641 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.641 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.641 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.642 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.642 INFO:teuthology.orchestra.run.smithi067.stdout: "service_discovery_port": { 2024-03-28T16:09:40.642 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "service_discovery_port", 2024-03-28T16:09:40.642 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.642 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.642 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.642 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "8765", 2024-03-28T16:09:40.642 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.642 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.642 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.642 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "cephadm service discovery port", 2024-03-28T16:09:40.642 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.642 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.642 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.643 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.643 INFO:teuthology.orchestra.run.smithi067.stdout: "ssh_config_file": { 2024-03-28T16:09:40.643 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "ssh_config_file", 2024-03-28T16:09:40.643 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.643 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.643 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.643 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.643 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.643 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.643 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.643 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-28T16:09:40.643 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.643 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.643 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.643 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.644 INFO:teuthology.orchestra.run.smithi067.stdout: "use_agent": { 2024-03-28T16:09:40.644 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "use_agent", 2024-03-28T16:09:40.644 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.644 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.644 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.644 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.644 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.644 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.644 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.644 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-28T16:09:40.644 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.644 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.644 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.644 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.644 INFO:teuthology.orchestra.run.smithi067.stdout: "use_repo_digest": { 2024-03-28T16:09:40.645 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "use_repo_digest", 2024-03-28T16:09:40.645 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.645 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.645 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.645 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.645 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.645 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.645 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.645 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-28T16:09:40.645 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.645 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.645 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.645 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.645 INFO:teuthology.orchestra.run.smithi067.stdout: "warn_on_failed_host_check": { 2024-03-28T16:09:40.646 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "warn_on_failed_host_check", 2024-03-28T16:09:40.646 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.646 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.646 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.646 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.646 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.646 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.646 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.646 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "raise a health warning if the host check fails", 2024-03-28T16:09:40.646 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.646 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.646 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.646 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.646 INFO:teuthology.orchestra.run.smithi067.stdout: "warn_on_stray_daemons": { 2024-03-28T16:09:40.647 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "warn_on_stray_daemons", 2024-03-28T16:09:40.647 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.647 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.647 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.647 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.647 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.647 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.647 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.647 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-28T16:09:40.647 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.647 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.647 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.647 INFO:teuthology.orchestra.run.smithi067.stdout: "warn_on_stray_hosts": { 2024-03-28T16:09:40.647 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "warn_on_stray_hosts", 2024-03-28T16:09:40.648 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.648 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.648 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.648 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.648 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.648 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.648 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.648 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-28T16:09:40.648 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.648 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.648 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.648 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.648 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.648 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.648 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.649 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "crash", 2024-03-28T16:09:40.649 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:40.649 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:40.649 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.649 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.649 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.649 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.649 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.649 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.649 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.649 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.649 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.649 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.649 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.649 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.650 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.650 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.650 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.650 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.650 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.650 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.650 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.650 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.650 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.650 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.650 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.650 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.650 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.650 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.650 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.651 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.651 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.651 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.651 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.651 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.651 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.651 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.651 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.651 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.651 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:40.651 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:40.651 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.651 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.651 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.651 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:40.652 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.652 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.652 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.652 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.652 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.652 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.652 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.653 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:39 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/1587543219' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T16:09:40.653 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:39 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/1370559961' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T16:09:40.656 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.656 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.656 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.656 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.656 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.656 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.656 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.656 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.656 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:40.657 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:40.657 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.657 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.657 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.657 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.657 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.657 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.657 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.657 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.657 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.657 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.657 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.657 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.657 INFO:teuthology.orchestra.run.smithi067.stdout: "retain_interval": { 2024-03-28T16:09:40.657 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "retain_interval", 2024-03-28T16:09:40.658 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "secs", 2024-03-28T16:09:40.658 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.658 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.658 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "31536000", 2024-03-28T16:09:40.658 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.658 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.658 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.658 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-28T16:09:40.658 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.658 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.658 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.658 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.658 INFO:teuthology.orchestra.run.smithi067.stdout: "warn_recent_interval": { 2024-03-28T16:09:40.658 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "warn_recent_interval", 2024-03-28T16:09:40.659 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "secs", 2024-03-28T16:09:40.659 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.659 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.659 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "1209600", 2024-03-28T16:09:40.659 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.659 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.659 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.659 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-28T16:09:40.659 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.659 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.659 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.659 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.659 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.660 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "dashboard", 2024-03-28T16:09:40.660 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:40.660 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:40.660 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.660 INFO:teuthology.orchestra.run.smithi067.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-28T16:09:40.660 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-28T16:09:40.660 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.660 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.660 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.660 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "10", 2024-03-28T16:09:40.660 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.660 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.660 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.660 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.661 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.661 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.661 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.661 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.661 INFO:teuthology.orchestra.run.smithi067.stdout: "ALERTMANAGER_API_HOST": { 2024-03-28T16:09:40.661 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-28T16:09:40.661 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.661 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.661 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.661 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.661 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.661 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.661 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.661 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.661 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.662 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.662 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.662 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.662 INFO:teuthology.orchestra.run.smithi067.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-28T16:09:40.662 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-28T16:09:40.662 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.662 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.662 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.662 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.662 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.662 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.662 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.662 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.662 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.663 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.663 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.663 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.663 INFO:teuthology.orchestra.run.smithi067.stdout: "AUDIT_API_ENABLED": { 2024-03-28T16:09:40.663 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "AUDIT_API_ENABLED", 2024-03-28T16:09:40.663 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.663 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.663 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.663 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.663 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.663 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.663 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.663 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.663 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.664 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.664 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.664 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.664 INFO:teuthology.orchestra.run.smithi067.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-28T16:09:40.664 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-28T16:09:40.664 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.664 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.664 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.664 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.664 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.664 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.664 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.664 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.664 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.664 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.664 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.665 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.665 INFO:teuthology.orchestra.run.smithi067.stdout: "ENABLE_BROWSABLE_API": { 2024-03-28T16:09:40.665 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-28T16:09:40.665 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.665 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.665 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.665 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.665 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.665 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.665 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.665 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.665 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.665 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.665 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.665 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.666 INFO:teuthology.orchestra.run.smithi067.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-28T16:09:40.666 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-28T16:09:40.666 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.666 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.666 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.666 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.666 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.666 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.666 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.666 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.666 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.666 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.666 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.666 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.667 INFO:teuthology.orchestra.run.smithi067.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-28T16:09:40.667 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-28T16:09:40.667 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.667 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.667 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.667 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.667 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.667 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.667 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.667 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.667 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.667 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.667 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.667 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.668 INFO:teuthology.orchestra.run.smithi067.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-28T16:09:40.668 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-28T16:09:40.668 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.668 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.668 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.668 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.668 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.668 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.668 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.668 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.668 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.668 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.668 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.668 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.668 INFO:teuthology.orchestra.run.smithi067.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-28T16:09:40.669 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-28T16:09:40.669 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.669 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.669 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.669 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.669 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.669 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.669 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.669 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.669 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.669 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.669 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.669 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.669 INFO:teuthology.orchestra.run.smithi067.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-28T16:09:40.670 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-28T16:09:40.670 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.670 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.670 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.670 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.670 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.670 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.670 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.670 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.670 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.670 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.670 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.670 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.671 INFO:teuthology.orchestra.run.smithi067.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-28T16:09:40.671 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-28T16:09:40.671 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.671 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.671 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.671 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.671 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.671 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.671 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.671 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.671 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.671 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.671 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.671 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.672 INFO:teuthology.orchestra.run.smithi067.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-28T16:09:40.672 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-28T16:09:40.672 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.672 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.672 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.672 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.672 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.672 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.672 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.672 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.672 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.672 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.672 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.672 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.673 INFO:teuthology.orchestra.run.smithi067.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-28T16:09:40.673 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-28T16:09:40.673 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.673 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.673 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.673 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.673 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.673 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.673 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.673 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.673 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.673 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.673 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.673 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.673 INFO:teuthology.orchestra.run.smithi067.stdout: "GRAFANA_API_PASSWORD": { 2024-03-28T16:09:40.674 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-28T16:09:40.674 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.674 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.674 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.674 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "admin", 2024-03-28T16:09:40.674 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.674 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.674 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.674 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.674 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.674 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.674 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.674 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.675 INFO:teuthology.orchestra.run.smithi067.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-28T16:09:40.675 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-28T16:09:40.675 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.675 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.675 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.675 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.675 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.675 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.675 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.675 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.675 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.675 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.675 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.675 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.675 INFO:teuthology.orchestra.run.smithi067.stdout: "GRAFANA_API_URL": { 2024-03-28T16:09:40.676 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "GRAFANA_API_URL", 2024-03-28T16:09:40.676 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.676 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.676 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.676 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.676 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.676 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.676 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.676 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.676 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.676 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.676 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.676 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.676 INFO:teuthology.orchestra.run.smithi067.stdout: "GRAFANA_API_USERNAME": { 2024-03-28T16:09:40.676 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-28T16:09:40.677 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.677 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.677 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.677 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "admin", 2024-03-28T16:09:40.677 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.677 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.677 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.677 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.677 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.677 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.677 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.677 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.677 INFO:teuthology.orchestra.run.smithi067.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-28T16:09:40.677 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-28T16:09:40.678 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.678 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.678 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.678 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.678 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.678 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.678 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.678 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.678 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.678 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.678 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.678 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.678 INFO:teuthology.orchestra.run.smithi067.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-28T16:09:40.678 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-28T16:09:40.679 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.679 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.679 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.679 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.679 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.679 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.679 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.679 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.679 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.679 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.679 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.679 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.679 INFO:teuthology.orchestra.run.smithi067.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-28T16:09:40.679 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-28T16:09:40.679 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.680 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.680 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.680 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.680 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.680 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.680 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.680 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.680 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.680 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.680 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.680 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.680 INFO:teuthology.orchestra.run.smithi067.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-28T16:09:40.680 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-28T16:09:40.680 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.680 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.681 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.681 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.681 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.681 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.681 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.681 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.681 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.681 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.681 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.681 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.681 INFO:teuthology.orchestra.run.smithi067.stdout: "MULTICLUSTER_CONFIG": { 2024-03-28T16:09:40.681 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-28T16:09:40.681 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.681 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.681 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.681 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "{}", 2024-03-28T16:09:40.682 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.682 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.682 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.682 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.682 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.682 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.682 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.682 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.682 INFO:teuthology.orchestra.run.smithi067.stdout: "PROMETHEUS_API_HOST": { 2024-03-28T16:09:40.682 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-28T16:09:40.682 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.682 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.682 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.683 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.683 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.683 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.683 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.683 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.683 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.683 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.683 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.683 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.683 INFO:teuthology.orchestra.run.smithi067.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-28T16:09:40.683 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-28T16:09:40.683 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.683 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.683 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.683 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.684 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.684 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.684 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.684 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.684 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.684 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.684 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.684 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.684 INFO:teuthology.orchestra.run.smithi067.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-28T16:09:40.684 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-28T16:09:40.684 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.684 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.684 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.684 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.684 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.685 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.685 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.685 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.685 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.685 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.685 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.685 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.685 INFO:teuthology.orchestra.run.smithi067.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-28T16:09:40.685 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-28T16:09:40.685 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.685 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.685 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.685 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.685 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.685 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.686 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.686 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.686 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.686 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.686 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.686 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.686 INFO:teuthology.orchestra.run.smithi067.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-28T16:09:40.686 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-28T16:09:40.686 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.686 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.686 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.686 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.686 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.686 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.687 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.687 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.687 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.687 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.687 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.687 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.687 INFO:teuthology.orchestra.run.smithi067.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-28T16:09:40.687 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-28T16:09:40.687 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.687 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.687 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.687 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.687 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.687 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.687 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.688 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.688 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.688 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.688 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.688 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.688 INFO:teuthology.orchestra.run.smithi067.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-28T16:09:40.688 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-28T16:09:40.688 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.688 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.688 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.688 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.688 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.688 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.688 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.688 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.689 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.689 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.689 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.689 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.689 INFO:teuthology.orchestra.run.smithi067.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-28T16:09:40.689 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-28T16:09:40.689 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.689 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.689 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.689 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.689 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.689 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.689 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.689 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.689 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.690 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.690 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.690 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.690 INFO:teuthology.orchestra.run.smithi067.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-28T16:09:40.690 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-28T16:09:40.690 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.690 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.690 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.690 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.690 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.690 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.690 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.690 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.690 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.690 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.691 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.691 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.691 INFO:teuthology.orchestra.run.smithi067.stdout: "PWD_POLICY_ENABLED": { 2024-03-28T16:09:40.691 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-28T16:09:40.691 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.691 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.691 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.691 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.691 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.691 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.691 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.691 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.691 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.691 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.691 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.692 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.692 INFO:teuthology.orchestra.run.smithi067.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-28T16:09:40.692 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-28T16:09:40.692 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.692 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.692 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.692 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-28T16:09:40.692 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.692 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.692 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.692 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.692 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.693 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.693 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.693 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.693 INFO:teuthology.orchestra.run.smithi067.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-28T16:09:40.693 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-28T16:09:40.693 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.693 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.693 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.693 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "10", 2024-03-28T16:09:40.693 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.693 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.693 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.693 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.693 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.693 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.694 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.694 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.694 INFO:teuthology.orchestra.run.smithi067.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-28T16:09:40.694 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-28T16:09:40.694 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.694 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.694 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.694 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "8", 2024-03-28T16:09:40.694 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.694 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.694 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.694 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.694 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.694 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.695 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.695 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.695 INFO:teuthology.orchestra.run.smithi067.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-28T16:09:40.695 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-28T16:09:40.695 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.695 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.695 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.695 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "45", 2024-03-28T16:09:40.695 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.695 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.695 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.695 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.695 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.695 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.696 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.696 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.696 INFO:teuthology.orchestra.run.smithi067.stdout: "RGW_API_ACCESS_KEY": { 2024-03-28T16:09:40.696 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-28T16:09:40.696 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.696 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.696 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.696 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.696 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.696 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.696 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.696 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.696 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.696 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.696 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.697 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.697 INFO:teuthology.orchestra.run.smithi067.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-28T16:09:40.697 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-28T16:09:40.697 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.697 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.697 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.697 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "admin", 2024-03-28T16:09:40.697 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.697 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.697 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.697 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.697 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.697 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.697 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.698 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.698 INFO:teuthology.orchestra.run.smithi067.stdout: "RGW_API_SECRET_KEY": { 2024-03-28T16:09:40.698 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-28T16:09:40.698 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.698 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.698 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.698 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.698 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.698 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.698 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.698 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.698 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.698 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.698 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.699 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.699 INFO:teuthology.orchestra.run.smithi067.stdout: "RGW_API_SSL_VERIFY": { 2024-03-28T16:09:40.699 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-28T16:09:40.699 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.699 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.699 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.699 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.699 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.699 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.699 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.699 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.699 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.699 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.699 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.699 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.700 INFO:teuthology.orchestra.run.smithi067.stdout: "UNSAFE_TLS_v1_2": { 2024-03-28T16:09:40.700 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-28T16:09:40.700 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.700 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.700 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.700 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.700 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.700 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.700 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.700 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.700 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.700 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.700 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.700 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.700 INFO:teuthology.orchestra.run.smithi067.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-28T16:09:40.701 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-28T16:09:40.701 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.701 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.701 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.701 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "0", 2024-03-28T16:09:40.701 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.701 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.701 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.701 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.701 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.701 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.701 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.701 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.701 INFO:teuthology.orchestra.run.smithi067.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-28T16:09:40.702 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-28T16:09:40.702 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.702 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.702 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.702 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "10", 2024-03-28T16:09:40.702 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.702 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.702 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.702 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.702 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.702 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.702 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.702 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.702 INFO:teuthology.orchestra.run.smithi067.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-28T16:09:40.703 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-28T16:09:40.703 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.703 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.703 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.703 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "5", 2024-03-28T16:09:40.703 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.703 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.703 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.703 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.703 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.703 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.703 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.703 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.703 INFO:teuthology.orchestra.run.smithi067.stdout: "cross_origin_url": { 2024-03-28T16:09:40.703 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "cross_origin_url", 2024-03-28T16:09:40.704 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.704 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.704 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.704 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.704 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.704 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.704 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.704 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.704 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.704 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.704 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.704 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.704 INFO:teuthology.orchestra.run.smithi067.stdout: "crt_file": { 2024-03-28T16:09:40.704 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "crt_file", 2024-03-28T16:09:40.705 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.705 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.705 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.705 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.705 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.705 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.705 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.705 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.705 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.705 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.705 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.705 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.705 INFO:teuthology.orchestra.run.smithi067.stdout: "debug": { 2024-03-28T16:09:40.705 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "debug", 2024-03-28T16:09:40.705 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.706 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.706 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.706 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.706 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.706 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.706 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.706 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Enable/disable debug options", 2024-03-28T16:09:40.706 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.706 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.706 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.706 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.706 INFO:teuthology.orchestra.run.smithi067.stdout: "jwt_token_ttl": { 2024-03-28T16:09:40.706 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "jwt_token_ttl", 2024-03-28T16:09:40.706 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.706 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.707 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.707 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "28800", 2024-03-28T16:09:40.707 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.707 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.707 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.707 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.707 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.707 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.707 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.707 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.707 INFO:teuthology.orchestra.run.smithi067.stdout: "key_file": { 2024-03-28T16:09:40.707 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "key_file", 2024-03-28T16:09:40.707 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.707 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.708 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.708 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.708 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.708 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.708 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.708 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.708 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.708 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.708 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.708 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.708 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.708 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.708 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.708 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.708 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.709 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.709 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.709 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.709 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.709 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.709 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.709 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.709 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.709 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.709 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.709 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.709 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.709 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.709 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.709 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.710 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.710 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.710 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.710 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.710 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.710 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.710 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.710 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.710 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.710 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.710 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.710 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.710 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.710 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.711 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.711 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:40.711 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:40.711 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.711 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.711 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.711 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:40.711 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.711 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.711 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.711 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.711 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.711 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.711 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.711 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.712 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.712 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.712 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.712 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.712 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.712 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.712 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.712 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:40.712 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:40.712 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.712 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.712 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.712 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.712 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.712 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.713 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.713 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.713 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.713 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.713 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.713 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.713 INFO:teuthology.orchestra.run.smithi067.stdout: "motd": { 2024-03-28T16:09:40.713 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "motd", 2024-03-28T16:09:40.713 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.713 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.713 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.713 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.713 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.713 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.714 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.714 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "The message of the day", 2024-03-28T16:09:40.714 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.714 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.714 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.714 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.714 INFO:teuthology.orchestra.run.smithi067.stdout: "redirect_resolve_ip_addr": { 2024-03-28T16:09:40.714 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "redirect_resolve_ip_addr", 2024-03-28T16:09:40.714 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.714 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.714 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.714 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.714 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.714 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.714 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.715 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.715 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.715 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.715 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.715 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.715 INFO:teuthology.orchestra.run.smithi067.stdout: "server_addr": { 2024-03-28T16:09:40.715 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "server_addr", 2024-03-28T16:09:40.715 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.715 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.715 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.715 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "::", 2024-03-28T16:09:40.715 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.715 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.715 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.715 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.716 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.716 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.716 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.716 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.716 INFO:teuthology.orchestra.run.smithi067.stdout: "server_port": { 2024-03-28T16:09:40.716 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "server_port", 2024-03-28T16:09:40.716 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.716 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.716 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.716 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "8080", 2024-03-28T16:09:40.716 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.716 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.716 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.716 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.717 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.717 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.717 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.717 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.717 INFO:teuthology.orchestra.run.smithi067.stdout: "ssl": { 2024-03-28T16:09:40.717 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "ssl", 2024-03-28T16:09:40.717 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.717 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.717 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.717 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.717 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.717 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.717 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.717 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.717 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.718 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.718 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.718 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.718 INFO:teuthology.orchestra.run.smithi067.stdout: "ssl_server_port": { 2024-03-28T16:09:40.718 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "ssl_server_port", 2024-03-28T16:09:40.718 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.718 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.718 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.718 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "8443", 2024-03-28T16:09:40.718 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.718 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.718 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.718 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.718 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.718 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.719 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.719 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.719 INFO:teuthology.orchestra.run.smithi067.stdout: "standby_behaviour": { 2024-03-28T16:09:40.719 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "standby_behaviour", 2024-03-28T16:09:40.719 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.719 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.719 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.719 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "redirect", 2024-03-28T16:09:40.719 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.719 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.719 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.719 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.719 INFO:teuthology.orchestra.run.smithi067.stdout: "redirect" 2024-03-28T16:09:40.719 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.720 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.720 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.720 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.720 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.720 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.720 INFO:teuthology.orchestra.run.smithi067.stdout: "standby_error_status_code": { 2024-03-28T16:09:40.720 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "standby_error_status_code", 2024-03-28T16:09:40.720 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.720 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.720 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.720 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "500", 2024-03-28T16:09:40.720 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "400", 2024-03-28T16:09:40.720 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "599", 2024-03-28T16:09:40.720 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.720 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.721 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.721 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.721 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.721 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.721 INFO:teuthology.orchestra.run.smithi067.stdout: "url_prefix": { 2024-03-28T16:09:40.721 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "url_prefix", 2024-03-28T16:09:40.721 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.721 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.721 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.721 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.721 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.721 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.721 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.721 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.721 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.722 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.722 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.722 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.722 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.722 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.722 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.722 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "devicehealth", 2024-03-28T16:09:40.722 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:40.722 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:40.722 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.722 INFO:teuthology.orchestra.run.smithi067.stdout: "enable_monitoring": { 2024-03-28T16:09:40.722 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "enable_monitoring", 2024-03-28T16:09:40.722 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.722 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.723 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.723 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.723 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.723 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.723 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.723 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "monitor device health metrics", 2024-03-28T16:09:40.723 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.723 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.723 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.723 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.723 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.723 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.723 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.723 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.723 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.724 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.724 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.724 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.724 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.724 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.724 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.724 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.724 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.724 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.724 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.724 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.724 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.724 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.724 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.725 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.725 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.725 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.725 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.725 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.725 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.725 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.725 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.725 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.725 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.725 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.725 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.725 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.725 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.725 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.726 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.726 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:40.726 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:40.726 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.726 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.726 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.726 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:40.726 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.726 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.726 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.726 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.726 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.726 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.726 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.726 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.727 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.727 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.727 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.727 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.727 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.727 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.727 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.727 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:40.727 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:40.727 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.727 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.727 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.727 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.727 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.727 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.728 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.728 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.728 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.728 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.728 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.728 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.728 INFO:teuthology.orchestra.run.smithi067.stdout: "mark_out_threshold": { 2024-03-28T16:09:40.728 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "mark_out_threshold", 2024-03-28T16:09:40.728 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "secs", 2024-03-28T16:09:40.728 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.728 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.728 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "2419200", 2024-03-28T16:09:40.728 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.728 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.728 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.729 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-28T16:09:40.729 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.729 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.729 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.729 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.729 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_name": { 2024-03-28T16:09:40.729 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "pool_name", 2024-03-28T16:09:40.729 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.729 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.729 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.729 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "device_health_metrics", 2024-03-28T16:09:40.729 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.729 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.729 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.729 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-28T16:09:40.730 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.730 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.730 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.730 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.730 INFO:teuthology.orchestra.run.smithi067.stdout: "retention_period": { 2024-03-28T16:09:40.730 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "retention_period", 2024-03-28T16:09:40.730 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "secs", 2024-03-28T16:09:40.730 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.730 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.730 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "15552000", 2024-03-28T16:09:40.730 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.730 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.730 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.730 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "how long to retain device health metrics", 2024-03-28T16:09:40.730 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.731 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.731 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.731 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.731 INFO:teuthology.orchestra.run.smithi067.stdout: "scrape_frequency": { 2024-03-28T16:09:40.731 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "scrape_frequency", 2024-03-28T16:09:40.731 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "secs", 2024-03-28T16:09:40.731 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.731 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.731 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "86400", 2024-03-28T16:09:40.731 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.731 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.731 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.731 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-28T16:09:40.731 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.732 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.732 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.732 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.732 INFO:teuthology.orchestra.run.smithi067.stdout: "self_heal": { 2024-03-28T16:09:40.732 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "self_heal", 2024-03-28T16:09:40.732 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.732 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.732 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.732 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.732 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.732 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.732 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.732 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-28T16:09:40.732 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.732 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.733 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.733 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.733 INFO:teuthology.orchestra.run.smithi067.stdout: "sleep_interval": { 2024-03-28T16:09:40.733 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "sleep_interval", 2024-03-28T16:09:40.733 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "secs", 2024-03-28T16:09:40.733 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.733 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.733 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "600", 2024-03-28T16:09:40.733 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.733 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.733 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.733 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "how frequently to wake up and check device health", 2024-03-28T16:09:40.733 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.733 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.734 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.734 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.734 INFO:teuthology.orchestra.run.smithi067.stdout: "warn_threshold": { 2024-03-28T16:09:40.734 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "warn_threshold", 2024-03-28T16:09:40.734 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "secs", 2024-03-28T16:09:40.734 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.734 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.734 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "7257600", 2024-03-28T16:09:40.734 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.734 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.734 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.734 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-28T16:09:40.734 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.734 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.734 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.735 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.735 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.735 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.735 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.735 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "diskprediction_local", 2024-03-28T16:09:40.735 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:40.735 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:40.735 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.735 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.735 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.735 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.735 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.735 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.735 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.735 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.736 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.736 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.736 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.736 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.736 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.736 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.736 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.736 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.736 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.736 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.736 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.736 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.736 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.736 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.736 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.737 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.737 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.737 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.737 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.737 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.737 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.737 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.737 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.737 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.737 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.737 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.737 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.737 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.737 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:40.737 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:40.738 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.738 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.738 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.738 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:40.738 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.738 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.738 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.738 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.738 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.738 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.738 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.738 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.738 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.739 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.739 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.739 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.739 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.739 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.739 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.739 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:40.739 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:40.739 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.739 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.739 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.739 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.739 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.739 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.739 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.740 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.740 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.740 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.740 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.740 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.740 INFO:teuthology.orchestra.run.smithi067.stdout: "predict_interval": { 2024-03-28T16:09:40.740 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "predict_interval", 2024-03-28T16:09:40.740 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.740 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.740 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.740 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "86400", 2024-03-28T16:09:40.740 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.740 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.740 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.741 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.741 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.741 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.741 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.741 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.741 INFO:teuthology.orchestra.run.smithi067.stdout: "predictor_model": { 2024-03-28T16:09:40.741 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "predictor_model", 2024-03-28T16:09:40.741 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.741 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.741 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.741 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "prophetstor", 2024-03-28T16:09:40.741 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.741 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.742 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.742 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.742 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.742 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.742 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.742 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.742 INFO:teuthology.orchestra.run.smithi067.stdout: "sleep_interval": { 2024-03-28T16:09:40.742 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "sleep_interval", 2024-03-28T16:09:40.742 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.742 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.742 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.742 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "600", 2024-03-28T16:09:40.742 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.742 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.742 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.743 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.743 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.743 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.743 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.743 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.743 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.743 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.743 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.743 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "influx", 2024-03-28T16:09:40.743 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": false, 2024-03-28T16:09:40.743 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "influxdb python module not found", 2024-03-28T16:09:40.743 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.743 INFO:teuthology.orchestra.run.smithi067.stdout: "batch_size": { 2024-03-28T16:09:40.743 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "batch_size", 2024-03-28T16:09:40.744 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.744 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.744 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.744 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "5000", 2024-03-28T16:09:40.744 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.744 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.744 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.744 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-28T16:09:40.744 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.744 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.744 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.744 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.744 INFO:teuthology.orchestra.run.smithi067.stdout: "database": { 2024-03-28T16:09:40.744 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "database", 2024-03-28T16:09:40.744 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.745 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.745 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.745 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "ceph", 2024-03-28T16:09:40.745 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.745 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.745 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.745 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-28T16:09:40.745 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.745 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.745 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.745 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.745 INFO:teuthology.orchestra.run.smithi067.stdout: "hostname": { 2024-03-28T16:09:40.745 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "hostname", 2024-03-28T16:09:40.745 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.745 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.746 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.746 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.746 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.746 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.746 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.746 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "InfluxDB server hostname", 2024-03-28T16:09:40.746 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.746 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.746 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.746 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.746 INFO:teuthology.orchestra.run.smithi067.stdout: "interval": { 2024-03-28T16:09:40.746 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "interval", 2024-03-28T16:09:40.746 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "secs", 2024-03-28T16:09:40.746 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.747 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.747 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "30", 2024-03-28T16:09:40.747 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "5", 2024-03-28T16:09:40.747 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.747 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.747 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-28T16:09:40.747 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.747 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.747 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.747 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.747 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.747 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.747 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.747 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.747 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.748 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.748 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.748 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.748 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.748 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.748 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.748 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.748 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.748 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.748 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.748 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.748 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.748 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.748 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.748 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.749 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.749 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.749 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.749 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.749 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.749 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.749 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.749 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.749 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.749 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.749 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.749 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.749 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.749 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.749 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.750 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:40.750 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:40.750 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.750 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.750 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.750 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:40.750 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.750 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.750 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.750 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.750 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.750 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.750 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.751 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.751 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.751 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.751 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.751 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.751 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.751 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.751 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.751 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:40.751 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:40.751 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.751 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.751 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.751 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.751 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.752 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.752 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.752 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.752 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.752 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.752 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.752 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.752 INFO:teuthology.orchestra.run.smithi067.stdout: "password": { 2024-03-28T16:09:40.752 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "password", 2024-03-28T16:09:40.752 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.752 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.752 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.752 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.752 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.752 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.753 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.753 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "password of InfluxDB server user", 2024-03-28T16:09:40.753 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.753 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.753 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.753 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.753 INFO:teuthology.orchestra.run.smithi067.stdout: "port": { 2024-03-28T16:09:40.753 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "port", 2024-03-28T16:09:40.753 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.753 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.753 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.753 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "8086", 2024-03-28T16:09:40.753 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.753 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.753 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.754 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "InfluxDB server port", 2024-03-28T16:09:40.754 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.754 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.754 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.754 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.754 INFO:teuthology.orchestra.run.smithi067.stdout: "ssl": { 2024-03-28T16:09:40.754 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "ssl", 2024-03-28T16:09:40.754 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.754 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.754 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.754 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "false", 2024-03-28T16:09:40.754 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.754 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.754 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.755 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-28T16:09:40.755 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.755 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.755 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.755 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.755 INFO:teuthology.orchestra.run.smithi067.stdout: "threads": { 2024-03-28T16:09:40.755 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "threads", 2024-03-28T16:09:40.755 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.755 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.755 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.755 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "5", 2024-03-28T16:09:40.755 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "1", 2024-03-28T16:09:40.755 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "32", 2024-03-28T16:09:40.755 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.755 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-28T16:09:40.756 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.756 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.756 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.756 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.756 INFO:teuthology.orchestra.run.smithi067.stdout: "username": { 2024-03-28T16:09:40.756 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "username", 2024-03-28T16:09:40.756 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.756 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.756 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.756 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.756 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.756 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.756 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.756 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "username of InfluxDB server user", 2024-03-28T16:09:40.756 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.757 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.757 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.757 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.757 INFO:teuthology.orchestra.run.smithi067.stdout: "verify_ssl": { 2024-03-28T16:09:40.757 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "verify_ssl", 2024-03-28T16:09:40.757 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.757 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.757 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.757 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "true", 2024-03-28T16:09:40.757 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.757 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.757 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.757 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-28T16:09:40.757 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.757 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.758 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.758 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.758 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.758 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.758 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.758 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "insights", 2024-03-28T16:09:40.758 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:40.758 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:40.758 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.758 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.758 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.758 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.758 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.759 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.759 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.759 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.759 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.759 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.759 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.759 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.759 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.759 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.759 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.759 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.759 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.759 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.759 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.759 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.760 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.760 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.760 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.760 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.760 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.760 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.760 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.760 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.760 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.760 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.760 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.760 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.760 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.760 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.760 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.761 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.761 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:40.761 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:40.761 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.761 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.761 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.761 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:40.761 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.761 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.761 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.761 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.761 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.761 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.761 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.761 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.761 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.762 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.762 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.762 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.762 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.762 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.762 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.762 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:40.762 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:40.762 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.762 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.762 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.762 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.762 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.762 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.763 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.763 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.763 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.763 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.763 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.763 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.763 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.763 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.763 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.763 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "iostat", 2024-03-28T16:09:40.763 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:40.763 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:40.763 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.763 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.763 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.764 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.764 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.765 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.765 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.765 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.765 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.765 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.765 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.765 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.765 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.765 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.766 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.766 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.766 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.766 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.766 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.766 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.766 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.766 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.766 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.766 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.766 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.766 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.766 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.767 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.767 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.767 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.767 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.767 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.767 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.767 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.767 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.767 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.767 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:40.767 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:40.767 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.767 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.767 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.767 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:40.768 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.768 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.768 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.768 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.768 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.768 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.768 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.768 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.768 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.768 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.768 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.768 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.768 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.768 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.769 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.769 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:40.769 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:40.769 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.769 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.769 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.769 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.769 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.769 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.769 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.769 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.769 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.769 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.769 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.769 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.770 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.770 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.770 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.770 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "k8sevents", 2024-03-28T16:09:40.770 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:40.770 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:40.770 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.770 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph_event_retention_days": { 2024-03-28T16:09:40.770 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "ceph_event_retention_days", 2024-03-28T16:09:40.770 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.770 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.770 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.770 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "7", 2024-03-28T16:09:40.770 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.771 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.771 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.771 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-28T16:09:40.771 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.771 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.771 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.771 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.771 INFO:teuthology.orchestra.run.smithi067.stdout: "config_check_secs": { 2024-03-28T16:09:40.771 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "config_check_secs", 2024-03-28T16:09:40.771 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.771 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.771 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.771 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "10", 2024-03-28T16:09:40.771 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "10", 2024-03-28T16:09:40.771 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.772 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.772 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-28T16:09:40.772 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.772 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.772 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.772 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.772 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.772 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.772 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.772 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.772 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.772 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.772 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.772 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.772 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.773 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.773 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.773 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.773 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.773 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.773 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.773 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.773 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.773 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.773 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.773 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.773 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.773 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.773 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.774 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.774 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.774 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.774 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.774 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.774 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.774 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.774 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.774 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.774 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.774 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.774 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.774 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:40.774 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:40.775 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.775 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.775 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.775 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:40.775 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.775 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.775 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.775 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.775 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.775 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.775 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.775 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.775 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.775 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.775 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.776 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.776 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.776 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.776 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.776 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:40.776 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:40.776 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.776 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.776 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.776 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.776 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.776 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.776 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.777 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.777 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.777 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.777 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.777 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.777 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.777 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.777 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.777 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "localpool", 2024-03-28T16:09:40.777 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:40.777 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:40.777 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.777 INFO:teuthology.orchestra.run.smithi067.stdout: "failure_domain": { 2024-03-28T16:09:40.777 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "failure_domain", 2024-03-28T16:09:40.778 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.778 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.778 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.778 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "host", 2024-03-28T16:09:40.778 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.778 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.778 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.778 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "failure domain for any created local pool", 2024-03-28T16:09:40.778 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-28T16:09:40.778 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.778 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.778 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.778 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.778 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.779 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.779 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.779 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.779 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.779 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.779 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.779 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.779 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.779 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.779 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.779 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.779 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.779 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.779 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.779 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.780 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.780 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.780 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.780 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.780 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.780 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.780 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.780 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.780 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.780 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.780 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.780 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.780 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.780 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.781 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.781 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.781 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.781 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.781 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:40.781 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:40.781 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.781 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.781 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.781 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:40.781 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.781 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.781 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.781 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.781 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.782 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.782 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.782 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.782 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.782 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.782 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.782 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.782 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.782 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.782 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.782 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:40.782 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:40.782 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.782 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.782 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.783 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.783 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.783 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.783 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.783 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.783 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.783 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.783 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.783 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.783 INFO:teuthology.orchestra.run.smithi067.stdout: "min_size": { 2024-03-28T16:09:40.783 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "min_size", 2024-03-28T16:09:40.783 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.783 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.783 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.783 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.784 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.784 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.784 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.784 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "default min_size for any created local pool", 2024-03-28T16:09:40.784 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-28T16:09:40.784 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.784 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.784 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.784 INFO:teuthology.orchestra.run.smithi067.stdout: "num_rep": { 2024-03-28T16:09:40.784 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "num_rep", 2024-03-28T16:09:40.784 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.784 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.784 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.784 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "3", 2024-03-28T16:09:40.785 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.785 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.785 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.785 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "default replica count for any created local pool", 2024-03-28T16:09:40.785 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.785 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.785 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.785 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.785 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num": { 2024-03-28T16:09:40.785 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "pg_num", 2024-03-28T16:09:40.785 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.785 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.785 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.785 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "128", 2024-03-28T16:09:40.785 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.785 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.786 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.786 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "default pg_num for any created local pool", 2024-03-28T16:09:40.786 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.786 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.786 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.786 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.786 INFO:teuthology.orchestra.run.smithi067.stdout: "prefix": { 2024-03-28T16:09:40.786 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "prefix", 2024-03-28T16:09:40.786 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.786 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.787 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.787 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.787 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.787 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.787 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.787 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "name prefix for any created local pool", 2024-03-28T16:09:40.787 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.787 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.787 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.787 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.787 INFO:teuthology.orchestra.run.smithi067.stdout: "subtree": { 2024-03-28T16:09:40.787 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "subtree", 2024-03-28T16:09:40.787 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.787 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.788 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.788 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "rack", 2024-03-28T16:09:40.788 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.788 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.788 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.788 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-28T16:09:40.788 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-28T16:09:40.788 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.788 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.788 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.788 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.788 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.788 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.788 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "mds_autoscaler", 2024-03-28T16:09:40.788 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:40.789 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:40.789 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.789 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.789 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.789 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.789 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.789 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.789 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.789 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.789 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.789 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.789 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.789 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.789 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.790 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.790 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.790 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.790 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.790 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.790 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.790 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.790 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.790 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.790 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.790 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.790 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.790 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.790 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.790 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.791 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.791 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.791 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.791 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.791 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.791 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.791 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.791 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.791 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:40.791 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:40.791 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.791 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.791 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.792 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:40.792 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.792 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.792 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.792 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.792 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.792 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.792 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.792 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.792 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.792 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.792 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.792 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.792 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.793 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.793 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.793 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:40.793 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:40.793 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.793 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.793 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.793 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.793 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.793 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.793 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.793 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.793 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.793 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.794 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.794 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.794 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.794 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.794 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.794 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "mirroring", 2024-03-28T16:09:40.794 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:40.794 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:40.794 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.794 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.794 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.794 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.794 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.794 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.795 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.795 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.795 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.795 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.795 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.795 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.795 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.795 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.795 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.795 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.795 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.795 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.795 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.795 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.796 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.796 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.796 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.796 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.796 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.796 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.796 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.796 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.796 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.796 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.796 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.796 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.796 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.796 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.796 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.797 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.797 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:40.797 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:40.797 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.797 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.797 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.797 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:40.797 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.797 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.797 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.797 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.797 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.798 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.798 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.798 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.798 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.798 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.798 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.798 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.798 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.798 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.798 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.798 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:40.798 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:40.798 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.798 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.799 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.799 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.799 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.799 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.799 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.799 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.799 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.799 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.799 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.799 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.799 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.799 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.799 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.799 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "nfs", 2024-03-28T16:09:40.800 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:40.800 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:40.800 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.800 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.800 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.800 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.800 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.800 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.800 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.800 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.800 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.800 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.800 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.801 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.801 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.801 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.801 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.801 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.801 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.801 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.801 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.801 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.801 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.801 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.801 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.801 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.801 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.802 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.802 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.802 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.802 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.802 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.802 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.802 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.802 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.802 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.802 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.802 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.802 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:40.802 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:40.802 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.803 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.803 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.803 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:40.803 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.803 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.803 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.803 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.803 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.803 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.803 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.803 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.803 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.803 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.803 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.803 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.804 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.804 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.804 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.804 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:40.804 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:40.804 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.804 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.804 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.804 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.804 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.804 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.804 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.804 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.804 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.804 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.805 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.805 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.805 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.805 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.805 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.805 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "orchestrator", 2024-03-28T16:09:40.805 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:40.805 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:40.805 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.805 INFO:teuthology.orchestra.run.smithi067.stdout: "fail_fs": { 2024-03-28T16:09:40.805 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "fail_fs", 2024-03-28T16:09:40.805 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.805 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.805 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.806 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.806 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.806 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.806 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.806 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-28T16:09:40.806 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.806 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.806 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.806 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.806 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.806 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.806 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.806 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.806 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.807 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.807 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.807 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.807 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.807 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.807 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.807 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.807 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.807 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.807 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.807 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.807 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.807 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.807 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.807 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.808 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.808 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.808 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.808 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.808 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.808 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.808 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.808 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.808 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.808 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.808 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.808 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.808 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.808 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.808 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.809 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:40.809 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:40.809 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.809 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.809 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.809 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:40.809 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.809 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.809 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.809 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.809 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.809 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.809 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.809 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.810 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.810 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.810 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.810 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.810 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.810 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.810 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.810 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:40.810 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:40.810 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.810 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.810 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.810 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.810 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.811 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.811 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.811 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.811 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.811 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.811 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.811 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.811 INFO:teuthology.orchestra.run.smithi067.stdout: "orchestrator": { 2024-03-28T16:09:40.811 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "orchestrator", 2024-03-28T16:09:40.811 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.811 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.811 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.811 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.811 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.811 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.812 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.812 INFO:teuthology.orchestra.run.smithi067.stdout: "cephadm", 2024-03-28T16:09:40.812 INFO:teuthology.orchestra.run.smithi067.stdout: "rook", 2024-03-28T16:09:40.812 INFO:teuthology.orchestra.run.smithi067.stdout: "test_orchestrator" 2024-03-28T16:09:40.812 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.812 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Orchestrator backend", 2024-03-28T16:09:40.812 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.812 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.812 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.812 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.812 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.812 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.812 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.812 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "osd_perf_query", 2024-03-28T16:09:40.812 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:40.813 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:40.813 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.813 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.813 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.813 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.813 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.813 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.813 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.813 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.813 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.813 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.813 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.813 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.813 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.813 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.814 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.814 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.814 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.814 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.814 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.814 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.814 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.814 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.814 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.814 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.814 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.814 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.814 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.814 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.815 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.815 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.815 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.815 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.815 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.815 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.815 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.815 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.815 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:40.815 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:40.815 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.815 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.815 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.816 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:40.816 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.816 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.816 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.816 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.816 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.816 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.816 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.816 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.816 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.816 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.816 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.816 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.816 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.816 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.816 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.817 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:40.817 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:40.817 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.817 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.817 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.817 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.817 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.817 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.817 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.817 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.817 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.817 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.817 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.818 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.818 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.818 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.818 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.818 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "osd_support", 2024-03-28T16:09:40.818 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:40.818 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:40.818 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.818 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.818 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.818 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.818 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.818 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.818 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.819 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.819 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.819 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.819 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.819 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.819 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.819 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.819 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.819 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.819 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.819 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.819 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.819 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.819 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.819 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.820 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.820 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.820 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.820 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.820 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.820 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.820 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.820 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.820 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.820 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.820 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.820 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.820 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.820 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.820 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:40.821 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:40.821 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.821 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.821 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.821 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:40.821 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.821 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.821 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.821 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.821 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.821 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.821 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.821 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.821 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.821 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.822 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.822 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.822 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.822 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.822 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.822 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:40.822 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:40.822 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.822 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.822 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.822 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.822 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.822 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.823 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.823 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.823 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.823 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.823 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.823 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.823 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.823 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.823 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.823 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "pg_autoscaler", 2024-03-28T16:09:40.823 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:40.823 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:40.823 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.823 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.823 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.824 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.824 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.824 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.824 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.824 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.824 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.824 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.824 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.824 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.824 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.824 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.824 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.824 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.824 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.824 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.824 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.825 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.825 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.825 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.825 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.825 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.825 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.825 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.825 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.825 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.825 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.825 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.825 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.825 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.825 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.825 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.826 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.826 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.826 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:40.826 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:40.826 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.826 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.826 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.826 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:40.826 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.826 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.826 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.826 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.826 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.826 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.826 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.827 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.827 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.827 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.827 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.827 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.827 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.827 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.827 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.827 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:40.827 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:40.827 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.827 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.827 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.827 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.828 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.828 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.828 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.828 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.828 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.828 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.828 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.828 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.828 INFO:teuthology.orchestra.run.smithi067.stdout: "sleep_interval": { 2024-03-28T16:09:40.828 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "sleep_interval", 2024-03-28T16:09:40.828 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "secs", 2024-03-28T16:09:40.828 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.828 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.828 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "60", 2024-03-28T16:09:40.828 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.829 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.829 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.829 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.829 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.829 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.829 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.829 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.829 INFO:teuthology.orchestra.run.smithi067.stdout: "threshold": { 2024-03-28T16:09:40.829 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "threshold", 2024-03-28T16:09:40.829 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "float", 2024-03-28T16:09:40.829 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.829 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.829 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "3.0", 2024-03-28T16:09:40.829 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "1.0", 2024-03-28T16:09:40.829 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.830 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.830 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "scaling threshold", 2024-03-28T16:09:40.830 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-28T16:09:40.830 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.830 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.830 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.830 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.830 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.830 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.830 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "progress", 2024-03-28T16:09:40.830 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:40.830 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:40.830 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.831 INFO:teuthology.orchestra.run.smithi067.stdout: "allow_pg_recovery_event": { 2024-03-28T16:09:40.831 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "allow_pg_recovery_event", 2024-03-28T16:09:40.831 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.831 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.831 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.831 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.831 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.831 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.831 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.831 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-28T16:09:40.831 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.831 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.831 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.831 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.832 INFO:teuthology.orchestra.run.smithi067.stdout: "enabled": { 2024-03-28T16:09:40.832 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "enabled", 2024-03-28T16:09:40.832 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.832 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.832 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.832 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.832 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.832 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.832 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.832 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.832 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.832 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.832 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.832 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.833 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.833 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.833 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.833 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.833 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.833 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.833 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.833 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.833 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.833 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.833 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.833 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.833 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.833 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.833 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.834 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.834 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.834 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.834 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.834 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.834 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.834 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.834 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.834 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.834 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.834 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.834 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.834 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.834 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.835 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.835 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.835 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.835 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.835 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.835 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.835 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:40.835 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:40.835 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.835 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.835 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.835 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:40.835 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.835 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.835 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.836 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.836 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.836 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.836 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.836 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.836 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.836 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.836 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.836 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.836 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.836 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.836 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.836 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:40.836 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:40.836 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.837 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.837 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.837 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.837 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.837 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.837 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.837 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.837 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.837 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.837 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.837 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.837 INFO:teuthology.orchestra.run.smithi067.stdout: "max_completed_events": { 2024-03-28T16:09:40.837 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "max_completed_events", 2024-03-28T16:09:40.837 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.838 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.838 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.838 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "50", 2024-03-28T16:09:40.838 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.838 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.838 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.838 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "number of past completed events to remember", 2024-03-28T16:09:40.838 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.838 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.838 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.838 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.838 INFO:teuthology.orchestra.run.smithi067.stdout: "sleep_interval": { 2024-03-28T16:09:40.838 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "sleep_interval", 2024-03-28T16:09:40.838 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "secs", 2024-03-28T16:09:40.839 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.839 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.839 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "5", 2024-03-28T16:09:40.839 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.839 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.839 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.839 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "how long the module is going to sleep", 2024-03-28T16:09:40.839 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.839 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.839 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.839 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.839 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.839 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.839 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.839 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "prometheus", 2024-03-28T16:09:40.840 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:40.840 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:40.840 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.840 INFO:teuthology.orchestra.run.smithi067.stdout: "cache": { 2024-03-28T16:09:40.840 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "cache", 2024-03-28T16:09:40.840 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.840 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.840 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.840 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.840 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.840 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.840 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.840 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.840 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.840 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.841 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.841 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.841 INFO:teuthology.orchestra.run.smithi067.stdout: "exclude_perf_counters": { 2024-03-28T16:09:40.841 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "exclude_perf_counters", 2024-03-28T16:09:40.841 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.841 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.841 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.841 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.841 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.841 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.841 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.841 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-28T16:09:40.841 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-28T16:09:40.841 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.841 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.842 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.842 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.842 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.842 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.842 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.842 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.842 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.842 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.842 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.842 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.842 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.842 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.843 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.843 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.843 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.843 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.843 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.843 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.843 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.843 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.843 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.843 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.843 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.843 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.843 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.843 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.844 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.844 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.844 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.844 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.844 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.844 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.844 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.844 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.844 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.844 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.844 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:40.844 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:40.844 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.844 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.844 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.845 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:40.845 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.845 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.845 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.845 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.845 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.845 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.845 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.845 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.845 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.845 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.845 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.845 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.845 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.846 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.846 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.846 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:40.846 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:40.846 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.846 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.846 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.846 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.846 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.846 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.846 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.846 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.846 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.846 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.847 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.847 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.847 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_stats_pools": { 2024-03-28T16:09:40.847 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "rbd_stats_pools", 2024-03-28T16:09:40.847 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.847 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.847 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.847 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.847 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.847 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.847 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.847 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.847 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.847 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.847 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.848 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.848 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-28T16:09:40.848 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-28T16:09:40.848 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.848 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.848 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.848 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "300", 2024-03-28T16:09:40.848 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.848 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.848 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.848 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.848 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.848 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.848 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.849 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.849 INFO:teuthology.orchestra.run.smithi067.stdout: "scrape_interval": { 2024-03-28T16:09:40.849 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "scrape_interval", 2024-03-28T16:09:40.849 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "float", 2024-03-28T16:09:40.849 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.849 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.849 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "15.0", 2024-03-28T16:09:40.849 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.849 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.849 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.849 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.849 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.849 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.849 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.850 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.850 INFO:teuthology.orchestra.run.smithi067.stdout: "server_addr": { 2024-03-28T16:09:40.850 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "server_addr", 2024-03-28T16:09:40.850 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.850 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.850 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.850 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "::", 2024-03-28T16:09:40.850 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.850 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.850 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.850 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-28T16:09:40.850 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.850 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.850 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.851 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.851 INFO:teuthology.orchestra.run.smithi067.stdout: "server_port": { 2024-03-28T16:09:40.851 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "server_port", 2024-03-28T16:09:40.851 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.851 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.851 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.851 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "9283", 2024-03-28T16:09:40.851 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.851 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.851 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.851 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-28T16:09:40.851 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.851 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.851 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.851 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.852 INFO:teuthology.orchestra.run.smithi067.stdout: "stale_cache_strategy": { 2024-03-28T16:09:40.852 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "stale_cache_strategy", 2024-03-28T16:09:40.852 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.852 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.852 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.852 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "log", 2024-03-28T16:09:40.852 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.852 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.852 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.852 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.852 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.852 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.852 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.852 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.852 INFO:teuthology.orchestra.run.smithi067.stdout: "standby_behaviour": { 2024-03-28T16:09:40.853 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "standby_behaviour", 2024-03-28T16:09:40.853 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.853 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.853 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.853 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "default", 2024-03-28T16:09:40.853 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.853 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.853 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.853 INFO:teuthology.orchestra.run.smithi067.stdout: "default", 2024-03-28T16:09:40.853 INFO:teuthology.orchestra.run.smithi067.stdout: "error" 2024-03-28T16:09:40.853 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.853 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.853 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.853 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.854 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.854 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.854 INFO:teuthology.orchestra.run.smithi067.stdout: "standby_error_status_code": { 2024-03-28T16:09:40.854 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "standby_error_status_code", 2024-03-28T16:09:40.854 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.854 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.854 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.854 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "500", 2024-03-28T16:09:40.854 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "400", 2024-03-28T16:09:40.854 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "599", 2024-03-28T16:09:40.854 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.854 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.854 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.854 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.854 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.855 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.855 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.855 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.855 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.855 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "rbd_support", 2024-03-28T16:09:40.855 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:40.855 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:40.855 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.855 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.855 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.855 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.855 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.855 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.855 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.856 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.856 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.856 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.856 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.856 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.856 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.856 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.856 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.856 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.856 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.856 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.856 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.856 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.856 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.856 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.857 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.857 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.857 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.857 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.857 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.857 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.857 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.857 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.857 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.857 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.857 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.857 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.857 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.857 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.857 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:40.858 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:40.858 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.858 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.858 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.858 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:40.858 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.858 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.858 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.858 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.858 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.858 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.858 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.858 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.858 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.859 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.859 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.859 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.859 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.859 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.859 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.859 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:40.859 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:40.859 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.859 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.859 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.859 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.859 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.859 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.859 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.860 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.860 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.860 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.860 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.860 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.860 INFO:teuthology.orchestra.run.smithi067.stdout: "max_concurrent_snap_create": { 2024-03-28T16:09:40.860 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "max_concurrent_snap_create", 2024-03-28T16:09:40.860 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.860 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.860 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.860 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "10", 2024-03-28T16:09:40.860 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.860 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.860 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.860 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.861 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.861 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.861 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.861 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.861 INFO:teuthology.orchestra.run.smithi067.stdout: "mirror_snapshot_schedule": { 2024-03-28T16:09:40.861 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "mirror_snapshot_schedule", 2024-03-28T16:09:40.861 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.861 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.861 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.861 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.861 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.861 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.861 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.861 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.861 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.862 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.862 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.862 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.862 INFO:teuthology.orchestra.run.smithi067.stdout: "trash_purge_schedule": { 2024-03-28T16:09:40.862 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "trash_purge_schedule", 2024-03-28T16:09:40.862 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.862 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.862 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.862 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.862 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.862 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.862 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.862 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.862 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.862 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.863 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.863 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.863 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.863 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.863 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.863 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "restful", 2024-03-28T16:09:40.863 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:40.863 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:40.863 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.863 INFO:teuthology.orchestra.run.smithi067.stdout: "enable_auth": { 2024-03-28T16:09:40.863 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "enable_auth", 2024-03-28T16:09:40.863 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.863 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.863 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.863 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.864 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.864 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.864 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.864 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.864 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.864 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.864 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.864 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.864 INFO:teuthology.orchestra.run.smithi067.stdout: "key_file": { 2024-03-28T16:09:40.864 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "key_file", 2024-03-28T16:09:40.864 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.864 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.864 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.864 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.864 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.865 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.865 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.865 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.865 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.865 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.865 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.865 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.865 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.865 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.865 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.865 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.865 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.865 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.865 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.865 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.866 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.866 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.866 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.866 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.866 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.866 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.866 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.866 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.866 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.866 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.866 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.866 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.866 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.866 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.867 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.867 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.867 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.867 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.867 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.867 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.867 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.867 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.867 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.867 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.867 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.867 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.867 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.867 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:40.867 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:40.868 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.868 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.868 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.868 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:40.868 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.868 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.868 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.868 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.868 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.868 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.868 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.868 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.868 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.868 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.869 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.869 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.869 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.869 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.869 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.869 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:40.869 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:40.869 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.869 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.869 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.869 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.869 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.869 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.869 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.869 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.870 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.870 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.870 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.870 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.870 INFO:teuthology.orchestra.run.smithi067.stdout: "server_addr": { 2024-03-28T16:09:40.870 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "server_addr", 2024-03-28T16:09:40.870 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.870 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.870 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.870 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.870 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.870 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.870 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.870 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.871 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.871 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.871 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.871 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.871 INFO:teuthology.orchestra.run.smithi067.stdout: "server_port": { 2024-03-28T16:09:40.871 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "server_port", 2024-03-28T16:09:40.871 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.871 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.871 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.871 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.871 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.871 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.871 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.871 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.871 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.872 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.872 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.872 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.872 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.872 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.872 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.872 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "rgw", 2024-03-28T16:09:40.872 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:40.872 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:40.872 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.872 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.872 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.872 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.872 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.872 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.873 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.873 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.873 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.873 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.873 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.873 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.873 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.873 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.873 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.873 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.873 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.873 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.873 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.873 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.873 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.874 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.874 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.874 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.874 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.874 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.874 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.874 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.874 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.874 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.874 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.874 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.874 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.874 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.874 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.875 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.875 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:40.875 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:40.875 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.875 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.875 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.875 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:40.875 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.875 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.875 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.875 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.875 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.875 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.876 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.876 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.876 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.876 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.876 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.876 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.876 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.876 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.876 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.876 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:40.876 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:40.876 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.876 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.876 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.877 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.877 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.877 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.877 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.877 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.877 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.877 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.877 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.877 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.877 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.877 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.877 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.877 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "rook", 2024-03-28T16:09:40.877 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:40.877 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:40.878 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.878 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.878 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.878 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.878 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.878 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.878 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.878 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.878 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.878 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.878 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.878 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.878 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.878 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.878 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.879 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.879 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.879 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.879 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.879 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.879 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.879 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.879 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.879 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.879 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.879 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.879 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.879 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.879 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.880 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.880 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.880 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.880 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.880 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.880 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.880 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.880 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:40.880 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:40.880 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.880 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.880 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.880 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:40.880 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.881 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.881 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.881 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.881 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.881 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.881 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.881 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.881 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.881 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.881 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.881 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.881 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.881 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.881 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.881 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:40.882 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:40.882 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.882 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.882 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.882 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.882 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.882 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.882 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.882 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.882 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.882 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.882 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.882 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.882 INFO:teuthology.orchestra.run.smithi067.stdout: "storage_class": { 2024-03-28T16:09:40.883 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "storage_class", 2024-03-28T16:09:40.883 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.883 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.883 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.883 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "local", 2024-03-28T16:09:40.883 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.883 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.883 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.883 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-28T16:09:40.883 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.883 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.883 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.883 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.883 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.884 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.884 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.884 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "selftest", 2024-03-28T16:09:40.884 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:40.884 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:40.884 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.884 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.884 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.884 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.884 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.884 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.884 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.884 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.884 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.884 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.885 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.885 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.885 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.885 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.885 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.885 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.885 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.885 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.885 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.885 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.885 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.885 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.885 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.885 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.885 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.886 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.886 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.886 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.886 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.886 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.886 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.886 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.886 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.886 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.886 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.886 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.886 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:40.886 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:40.886 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.886 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.887 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.887 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:40.887 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.887 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.887 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.887 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.887 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.887 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.887 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.887 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.887 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.887 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.887 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.887 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.888 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.888 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.888 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.888 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:40.888 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:40.888 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.888 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.888 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.888 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.888 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.888 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.888 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.888 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.888 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.888 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.888 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.889 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.889 INFO:teuthology.orchestra.run.smithi067.stdout: "roption1": { 2024-03-28T16:09:40.889 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "roption1", 2024-03-28T16:09:40.889 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.889 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.889 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.889 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.889 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.889 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.889 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.889 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.889 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.889 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.889 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.889 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.890 INFO:teuthology.orchestra.run.smithi067.stdout: "roption2": { 2024-03-28T16:09:40.890 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "roption2", 2024-03-28T16:09:40.890 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.890 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.890 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.890 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "xyz", 2024-03-28T16:09:40.890 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.890 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.890 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.890 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.890 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.890 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.890 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.890 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.891 INFO:teuthology.orchestra.run.smithi067.stdout: "rwoption1": { 2024-03-28T16:09:40.891 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "rwoption1", 2024-03-28T16:09:40.891 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.891 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.891 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.891 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.891 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.891 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.891 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.891 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.891 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.891 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.891 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.891 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.891 INFO:teuthology.orchestra.run.smithi067.stdout: "rwoption2": { 2024-03-28T16:09:40.892 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "rwoption2", 2024-03-28T16:09:40.892 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.892 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.892 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.892 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.892 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.892 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.892 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.892 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.892 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.892 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.892 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.892 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.892 INFO:teuthology.orchestra.run.smithi067.stdout: "rwoption3": { 2024-03-28T16:09:40.892 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "rwoption3", 2024-03-28T16:09:40.893 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "float", 2024-03-28T16:09:40.893 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.893 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.893 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.893 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.893 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.893 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.893 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.893 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.893 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.893 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.893 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.893 INFO:teuthology.orchestra.run.smithi067.stdout: "rwoption4": { 2024-03-28T16:09:40.893 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "rwoption4", 2024-03-28T16:09:40.893 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.894 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.894 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.894 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.894 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.894 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.894 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.894 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.894 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.894 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.894 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.894 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.894 INFO:teuthology.orchestra.run.smithi067.stdout: "rwoption5": { 2024-03-28T16:09:40.894 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "rwoption5", 2024-03-28T16:09:40.894 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.895 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.895 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.895 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.895 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.895 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.895 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.895 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.895 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.895 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.895 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.895 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.895 INFO:teuthology.orchestra.run.smithi067.stdout: "rwoption6": { 2024-03-28T16:09:40.895 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "rwoption6", 2024-03-28T16:09:40.895 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.895 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.896 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.896 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.896 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.896 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.896 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.896 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.896 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.896 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.896 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.896 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.896 INFO:teuthology.orchestra.run.smithi067.stdout: "rwoption7": { 2024-03-28T16:09:40.896 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "rwoption7", 2024-03-28T16:09:40.896 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.896 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.896 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.897 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.897 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "1", 2024-03-28T16:09:40.897 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "42", 2024-03-28T16:09:40.897 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.897 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.897 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.897 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.897 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.897 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.897 INFO:teuthology.orchestra.run.smithi067.stdout: "testkey": { 2024-03-28T16:09:40.897 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "testkey", 2024-03-28T16:09:40.897 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.897 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.897 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.898 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.898 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.898 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.898 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.898 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.898 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.898 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.898 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.898 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.898 INFO:teuthology.orchestra.run.smithi067.stdout: "testlkey": { 2024-03-28T16:09:40.898 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "testlkey", 2024-03-28T16:09:40.898 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.898 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.898 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.898 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.899 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.899 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.899 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.899 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.899 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.899 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.899 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.899 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.899 INFO:teuthology.orchestra.run.smithi067.stdout: "testnewline": { 2024-03-28T16:09:40.899 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "testnewline", 2024-03-28T16:09:40.899 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.899 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.899 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.899 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.899 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.900 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.900 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.900 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.900 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.900 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.900 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.900 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.900 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.900 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.900 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.900 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "snap_schedule", 2024-03-28T16:09:40.900 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:40.900 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:40.900 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.900 INFO:teuthology.orchestra.run.smithi067.stdout: "allow_m_granularity": { 2024-03-28T16:09:40.901 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "allow_m_granularity", 2024-03-28T16:09:40.901 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.901 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.901 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.901 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.901 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.901 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.901 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.901 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "allow minute scheduled snapshots", 2024-03-28T16:09:40.901 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.901 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.901 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.901 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.901 INFO:teuthology.orchestra.run.smithi067.stdout: "dump_on_update": { 2024-03-28T16:09:40.901 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "dump_on_update", 2024-03-28T16:09:40.902 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.902 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.902 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.902 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.902 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.902 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.902 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.902 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "dump database to debug log on update", 2024-03-28T16:09:40.902 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.902 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.902 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.902 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.902 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.902 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.903 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.903 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.903 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.903 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.903 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.903 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.903 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.903 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.903 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.903 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.903 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.903 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.903 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.903 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.903 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.904 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.904 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.904 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.904 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.904 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.904 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.904 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.904 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.904 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.904 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.904 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.904 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.904 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.904 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.904 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.905 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.905 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.905 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.905 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:40.905 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:40.905 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.905 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.905 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.905 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:40.905 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.905 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.905 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.905 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.905 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.905 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.906 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.906 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.906 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.906 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.906 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.906 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.906 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.906 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.906 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.906 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:40.906 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:40.906 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.906 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.906 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.907 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.907 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.907 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.907 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.907 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.907 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.907 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.907 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.907 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.907 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.907 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.907 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.907 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "stats", 2024-03-28T16:09:40.907 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:40.907 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:40.908 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.908 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.908 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.908 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.908 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.908 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.908 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.908 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.908 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.908 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.908 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.908 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.908 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.908 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.908 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.909 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.909 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.909 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.909 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.909 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.909 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.909 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.909 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.909 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.909 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.909 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.909 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.909 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.909 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.910 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.910 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.910 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.910 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.910 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.910 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.910 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.910 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:40.910 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:40.910 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.910 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.910 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.910 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:40.910 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.911 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.911 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.911 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.911 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.911 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.911 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.911 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.911 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.911 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.911 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.911 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.911 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.911 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.911 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.911 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:40.912 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:40.912 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.912 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.912 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.912 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.912 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.912 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.912 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.912 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.912 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.912 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.912 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.912 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.912 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.912 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.913 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.913 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "status", 2024-03-28T16:09:40.913 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:40.913 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:40.913 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.913 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.913 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.913 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.913 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.913 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.913 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.913 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.913 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.913 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.914 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.914 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.914 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.914 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.914 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.914 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.914 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.914 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.914 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.914 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.914 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.914 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.914 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.914 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.915 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.915 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.915 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.915 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.915 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.915 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.915 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.915 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.915 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.915 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.915 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.915 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.915 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:40.915 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:40.915 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.916 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.916 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.916 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:40.916 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.916 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.916 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.916 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.916 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.916 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.916 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.916 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.916 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.916 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.916 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.916 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.917 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.917 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.917 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.917 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:40.917 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:40.917 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.917 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.917 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.917 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.917 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.917 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.917 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.917 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.917 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.918 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.918 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.918 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.918 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.918 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.918 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.918 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "telegraf", 2024-03-28T16:09:40.918 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:40.918 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:40.918 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.918 INFO:teuthology.orchestra.run.smithi067.stdout: "address": { 2024-03-28T16:09:40.918 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "address", 2024-03-28T16:09:40.918 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.918 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.919 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.919 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-28T16:09:40.919 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.919 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.919 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.919 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.919 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.919 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.919 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.919 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.919 INFO:teuthology.orchestra.run.smithi067.stdout: "interval": { 2024-03-28T16:09:40.919 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "interval", 2024-03-28T16:09:40.919 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "secs", 2024-03-28T16:09:40.919 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.919 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.920 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "15", 2024-03-28T16:09:40.920 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.920 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.920 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.920 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.920 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.920 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.920 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.920 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.920 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.920 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.920 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.920 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.920 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.921 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.921 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.921 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.921 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.921 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.921 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.921 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.921 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.921 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.921 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.921 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.921 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.921 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.921 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.921 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.922 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.922 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.922 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.922 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.922 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.922 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.922 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.922 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.922 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.922 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.922 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.922 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.922 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.922 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.923 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.923 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:40.923 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:40.923 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.923 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.923 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.923 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:40.923 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.923 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.923 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.923 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.923 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.923 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.923 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.923 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.924 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.924 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.924 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.924 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.924 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.924 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.924 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.924 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:40.924 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:40.924 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.924 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.924 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.924 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.924 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.925 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.925 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.925 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.925 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.925 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.925 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.925 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.925 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.925 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.925 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.925 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "telemetry", 2024-03-28T16:09:40.925 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:40.925 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:40.925 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.925 INFO:teuthology.orchestra.run.smithi067.stdout: "channel_basic": { 2024-03-28T16:09:40.926 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "channel_basic", 2024-03-28T16:09:40.926 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.926 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.926 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.926 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.926 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.926 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.926 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.926 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-28T16:09:40.926 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.926 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.926 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.926 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.927 INFO:teuthology.orchestra.run.smithi067.stdout: "channel_crash": { 2024-03-28T16:09:40.927 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "channel_crash", 2024-03-28T16:09:40.927 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.927 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.927 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.927 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.927 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.927 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.927 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.927 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-28T16:09:40.927 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.927 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.927 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.927 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.927 INFO:teuthology.orchestra.run.smithi067.stdout: "channel_device": { 2024-03-28T16:09:40.928 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "channel_device", 2024-03-28T16:09:40.928 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.928 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.928 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.928 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.928 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.928 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.928 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.928 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-28T16:09:40.928 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.928 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.928 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.928 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.928 INFO:teuthology.orchestra.run.smithi067.stdout: "channel_ident": { 2024-03-28T16:09:40.928 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "channel_ident", 2024-03-28T16:09:40.929 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.929 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.929 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.929 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.929 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.929 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.929 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.929 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-28T16:09:40.929 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.929 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.929 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.929 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.929 INFO:teuthology.orchestra.run.smithi067.stdout: "channel_perf": { 2024-03-28T16:09:40.929 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "channel_perf", 2024-03-28T16:09:40.929 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.930 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.930 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.930 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.930 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.930 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.930 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.930 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-28T16:09:40.930 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.930 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.930 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.930 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.930 INFO:teuthology.orchestra.run.smithi067.stdout: "contact": { 2024-03-28T16:09:40.930 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "contact", 2024-03-28T16:09:40.930 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.931 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.931 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.931 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.931 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.931 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.931 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.931 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.931 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.931 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.931 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.931 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.931 INFO:teuthology.orchestra.run.smithi067.stdout: "description": { 2024-03-28T16:09:40.931 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "description", 2024-03-28T16:09:40.931 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.931 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.932 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.932 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.932 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.932 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.932 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.932 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.932 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.932 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.932 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.932 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.932 INFO:teuthology.orchestra.run.smithi067.stdout: "device_url": { 2024-03-28T16:09:40.932 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "device_url", 2024-03-28T16:09:40.932 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.932 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.932 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.933 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-28T16:09:40.933 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.933 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.933 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.933 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.933 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.933 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.933 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.933 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.933 INFO:teuthology.orchestra.run.smithi067.stdout: "enabled": { 2024-03-28T16:09:40.933 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "enabled", 2024-03-28T16:09:40.933 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.933 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.933 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.933 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.934 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.934 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.934 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.934 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.934 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.934 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.934 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.934 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.934 INFO:teuthology.orchestra.run.smithi067.stdout: "interval": { 2024-03-28T16:09:40.934 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "interval", 2024-03-28T16:09:40.934 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.934 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.934 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.934 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "24", 2024-03-28T16:09:40.935 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "8", 2024-03-28T16:09:40.935 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.935 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.935 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.935 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.935 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.935 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.935 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.935 INFO:teuthology.orchestra.run.smithi067.stdout: "last_opt_revision": { 2024-03-28T16:09:40.935 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "last_opt_revision", 2024-03-28T16:09:40.935 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.935 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.935 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.935 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "1", 2024-03-28T16:09:40.936 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.936 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.936 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.936 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.936 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.936 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.936 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.936 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.936 INFO:teuthology.orchestra.run.smithi067.stdout: "leaderboard": { 2024-03-28T16:09:40.936 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "leaderboard", 2024-03-28T16:09:40.936 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.936 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.936 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.936 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.936 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.937 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.937 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.937 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.937 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.937 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.937 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.937 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.937 INFO:teuthology.orchestra.run.smithi067.stdout: "leaderboard_description": { 2024-03-28T16:09:40.937 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "leaderboard_description", 2024-03-28T16:09:40.937 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.937 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.937 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.937 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.937 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.937 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.938 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.938 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.938 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.938 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.938 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.938 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.938 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.938 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.938 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.938 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.938 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.938 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.938 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.938 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.938 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.939 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.939 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.939 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.939 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.939 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.939 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.939 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.939 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.939 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.939 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.939 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.939 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.939 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.939 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.940 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.940 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.940 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.940 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.940 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.940 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.940 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.940 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.940 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.940 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.940 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.940 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.940 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:40.940 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:40.940 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.941 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.941 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.941 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:40.941 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.941 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.941 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.941 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.941 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.941 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.941 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.941 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.941 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.941 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.941 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.941 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.942 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.942 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.942 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.942 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:40.942 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:40.942 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.942 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.942 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.942 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.942 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.942 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.942 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.942 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.942 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.942 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.943 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.943 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.943 INFO:teuthology.orchestra.run.smithi067.stdout: "organization": { 2024-03-28T16:09:40.943 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "organization", 2024-03-28T16:09:40.943 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.943 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.943 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.943 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.943 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.943 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.943 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.943 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.943 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.943 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.944 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.944 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.944 INFO:teuthology.orchestra.run.smithi067.stdout: "proxy": { 2024-03-28T16:09:40.944 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "proxy", 2024-03-28T16:09:40.944 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.944 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.944 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.944 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.944 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.944 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.944 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.944 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.944 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.944 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.944 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.944 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.945 INFO:teuthology.orchestra.run.smithi067.stdout: "url": { 2024-03-28T16:09:40.945 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "url", 2024-03-28T16:09:40.945 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.945 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.945 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.945 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-28T16:09:40.945 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.945 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.945 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.945 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.945 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.945 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.945 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.945 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.946 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.946 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.946 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.946 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "test_orchestrator", 2024-03-28T16:09:40.946 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:40.946 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:40.946 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.946 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.946 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.946 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.946 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.946 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.946 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.947 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.947 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.947 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.947 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.947 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.947 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.947 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.947 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.947 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.947 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.947 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.947 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.947 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.947 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.948 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.948 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.948 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.948 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.948 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.948 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.948 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.948 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.948 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.948 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.948 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.948 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.948 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.948 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.948 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.949 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:40.949 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:40.949 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.949 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.949 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.949 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:40.949 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.949 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.949 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.949 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.949 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.949 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.949 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.949 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.950 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.950 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.950 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.950 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.950 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.950 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.950 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.950 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:40.950 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:40.950 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.950 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.950 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.950 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.950 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.951 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.951 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.951 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.951 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.951 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.951 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.951 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.951 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.951 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.951 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.951 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "volumes", 2024-03-28T16:09:40.951 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:40.951 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:40.952 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.952 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.952 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.952 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.952 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.952 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.952 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.952 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.952 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.952 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.952 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.952 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.952 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.952 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.953 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.953 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.953 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.953 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.953 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.953 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.953 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.953 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.953 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.953 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.953 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.953 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.953 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.953 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.953 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.954 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.954 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.954 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.954 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.954 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.954 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.954 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.954 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:40.954 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:40.954 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.954 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.954 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.954 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:40.955 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.955 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.955 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.955 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.955 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.955 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.955 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.955 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.955 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.955 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.955 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.955 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.955 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.955 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.955 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.956 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:40.956 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:40.956 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.956 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.956 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.956 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.956 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.956 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.956 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.956 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.956 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.956 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.956 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.956 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.957 INFO:teuthology.orchestra.run.smithi067.stdout: "max_concurrent_clones": { 2024-03-28T16:09:40.957 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "max_concurrent_clones", 2024-03-28T16:09:40.957 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.957 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.957 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.957 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "4", 2024-03-28T16:09:40.957 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.957 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.957 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.957 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-28T16:09:40.957 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.957 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.957 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.957 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.957 INFO:teuthology.orchestra.run.smithi067.stdout: "periodic_async_work": { 2024-03-28T16:09:40.958 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "periodic_async_work", 2024-03-28T16:09:40.958 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.958 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.958 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.958 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.958 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.958 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.958 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.958 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Periodically check for async work", 2024-03-28T16:09:40.958 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.958 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.958 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.958 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.958 INFO:teuthology.orchestra.run.smithi067.stdout: "snapshot_clone_delay": { 2024-03-28T16:09:40.959 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "snapshot_clone_delay", 2024-03-28T16:09:40.959 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.959 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.959 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.959 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "0", 2024-03-28T16:09:40.959 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.959 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.959 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.959 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-28T16:09:40.959 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.959 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.959 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.959 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.959 INFO:teuthology.orchestra.run.smithi067.stdout: "snapshot_clone_no_wait": { 2024-03-28T16:09:40.959 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "snapshot_clone_no_wait", 2024-03-28T16:09:40.960 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.960 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.960 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.960 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.960 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.960 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.960 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.960 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-28T16:09:40.960 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.960 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.960 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.960 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.960 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.960 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.960 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.961 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "zabbix", 2024-03-28T16:09:40.961 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:40.961 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:40.961 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.961 INFO:teuthology.orchestra.run.smithi067.stdout: "discovery_interval": { 2024-03-28T16:09:40.961 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "discovery_interval", 2024-03-28T16:09:40.961 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "uint", 2024-03-28T16:09:40.961 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.961 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.961 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "100", 2024-03-28T16:09:40.961 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.961 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.961 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.961 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.962 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.962 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.962 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.962 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.962 INFO:teuthology.orchestra.run.smithi067.stdout: "identifier": { 2024-03-28T16:09:40.962 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "identifier", 2024-03-28T16:09:40.962 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.962 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.962 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.962 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.962 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.962 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.962 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.962 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.963 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.963 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.963 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.963 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.963 INFO:teuthology.orchestra.run.smithi067.stdout: "interval": { 2024-03-28T16:09:40.963 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "interval", 2024-03-28T16:09:40.963 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "secs", 2024-03-28T16:09:40.963 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.963 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.963 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "60", 2024-03-28T16:09:40.963 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.963 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.963 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.963 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.964 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.964 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.964 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.964 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.964 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.964 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.964 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.964 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.964 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.964 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.964 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.964 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.964 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.964 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.964 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.965 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.965 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.965 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.965 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.965 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.965 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.965 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.965 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.965 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.965 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.965 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.965 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.965 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.965 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.965 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.966 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.966 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.966 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.966 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.966 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.966 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.966 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.969 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.969 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.969 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:40.969 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:40.969 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.970 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.970 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.970 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:40.970 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.970 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.970 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.970 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.970 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.970 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.970 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.970 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.970 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.970 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.970 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.971 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.971 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.971 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.971 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.971 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:40.971 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:40.971 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.971 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.971 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.971 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.971 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.971 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.971 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.971 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.971 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.972 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.972 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.972 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.972 INFO:teuthology.orchestra.run.smithi067.stdout: "zabbix_host": { 2024-03-28T16:09:40.972 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "zabbix_host", 2024-03-28T16:09:40.972 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.972 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.972 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.972 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.972 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.972 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.972 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.972 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.972 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.972 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.973 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.973 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.973 INFO:teuthology.orchestra.run.smithi067.stdout: "zabbix_port": { 2024-03-28T16:09:40.973 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "zabbix_port", 2024-03-28T16:09:40.973 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.973 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.973 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.973 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "10051", 2024-03-28T16:09:40.973 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.973 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.973 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.973 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.973 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.973 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.973 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.974 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.974 INFO:teuthology.orchestra.run.smithi067.stdout: "zabbix_sender": { 2024-03-28T16:09:40.974 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "zabbix_sender", 2024-03-28T16:09:40.974 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.974 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.974 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:40.974 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-28T16:09:40.974 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.974 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.974 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.974 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.974 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.974 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.974 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.974 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.975 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.975 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.975 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-28T16:09:40.975 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.975 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.975 INFO:teuthology.orchestra.run.smithi067.stdout: "modules": [ 2024-03-28T16:09:40.975 INFO:teuthology.orchestra.run.smithi067.stdout: "cephadm", 2024-03-28T16:09:40.975 INFO:teuthology.orchestra.run.smithi067.stdout: "dashboard", 2024-03-28T16:09:40.975 INFO:teuthology.orchestra.run.smithi067.stdout: "iostat", 2024-03-28T16:09:40.975 INFO:teuthology.orchestra.run.smithi067.stdout: "nfs", 2024-03-28T16:09:40.975 INFO:teuthology.orchestra.run.smithi067.stdout: "restful" 2024-03-28T16:09:40.975 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.975 INFO:teuthology.orchestra.run.smithi067.stdout: "available_modules": [ 2024-03-28T16:09:40.975 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.975 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "alerts", 2024-03-28T16:09:40.976 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:40.976 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:40.976 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.976 INFO:teuthology.orchestra.run.smithi067.stdout: "interval": { 2024-03-28T16:09:40.976 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "interval", 2024-03-28T16:09:40.976 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "secs", 2024-03-28T16:09:40.976 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.976 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.976 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "60", 2024-03-28T16:09:40.976 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.976 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.976 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.976 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "How frequently to reexamine health status", 2024-03-28T16:09:40.976 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.976 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.977 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.977 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.977 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.977 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.977 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.977 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.977 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.977 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.977 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.977 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.977 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.977 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.977 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.977 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.978 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.978 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.978 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.978 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.978 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.978 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.978 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.978 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.978 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.978 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.978 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.978 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.978 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.978 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.979 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.979 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.979 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.979 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.979 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.979 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.979 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.979 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.979 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.979 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:40.979 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:40.979 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.979 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.979 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.979 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:40.980 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.980 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.980 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.980 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.980 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.980 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.980 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.980 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.980 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.980 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.980 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.980 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.980 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.980 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.981 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.981 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:40.981 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:40.981 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.981 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.981 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.981 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.981 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.981 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.981 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.981 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.981 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.981 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.981 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.981 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.982 INFO:teuthology.orchestra.run.smithi067.stdout: "smtp_destination": { 2024-03-28T16:09:40.982 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "smtp_destination", 2024-03-28T16:09:40.982 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.982 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.982 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.982 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.982 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.982 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.982 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.982 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Email address to send alerts to", 2024-03-28T16:09:40.982 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.982 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.982 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.982 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.982 INFO:teuthology.orchestra.run.smithi067.stdout: "smtp_from_name": { 2024-03-28T16:09:40.983 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "smtp_from_name", 2024-03-28T16:09:40.983 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.983 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.983 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.983 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "Ceph", 2024-03-28T16:09:40.983 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.983 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.983 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.983 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Email From: name", 2024-03-28T16:09:40.983 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.983 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.983 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.983 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.983 INFO:teuthology.orchestra.run.smithi067.stdout: "smtp_host": { 2024-03-28T16:09:40.984 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "smtp_host", 2024-03-28T16:09:40.984 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.984 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.984 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.984 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.984 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.984 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.984 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.984 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "SMTP server", 2024-03-28T16:09:40.984 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.984 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.984 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.984 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.984 INFO:teuthology.orchestra.run.smithi067.stdout: "smtp_password": { 2024-03-28T16:09:40.985 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "smtp_password", 2024-03-28T16:09:40.985 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.985 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.985 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.985 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.985 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.985 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.985 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.985 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Password to authenticate with", 2024-03-28T16:09:40.985 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.985 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.985 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.985 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.985 INFO:teuthology.orchestra.run.smithi067.stdout: "smtp_port": { 2024-03-28T16:09:40.985 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "smtp_port", 2024-03-28T16:09:40.986 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:40.986 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.986 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.986 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "465", 2024-03-28T16:09:40.986 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.986 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.986 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.986 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "SMTP port", 2024-03-28T16:09:40.986 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.986 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.986 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.986 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.986 INFO:teuthology.orchestra.run.smithi067.stdout: "smtp_sender": { 2024-03-28T16:09:40.986 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "smtp_sender", 2024-03-28T16:09:40.987 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.987 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.987 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.987 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.987 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.987 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.987 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.987 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "SMTP envelope sender", 2024-03-28T16:09:40.987 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.987 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.987 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.987 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.987 INFO:teuthology.orchestra.run.smithi067.stdout: "smtp_ssl": { 2024-03-28T16:09:40.987 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "smtp_ssl", 2024-03-28T16:09:40.987 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.988 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.988 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.988 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.988 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.988 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.988 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.988 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-28T16:09:40.988 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.988 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.988 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.988 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.988 INFO:teuthology.orchestra.run.smithi067.stdout: "smtp_user": { 2024-03-28T16:09:40.988 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "smtp_user", 2024-03-28T16:09:40.988 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.988 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.989 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.989 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.989 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.989 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.989 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.989 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "User to authenticate as", 2024-03-28T16:09:40.989 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.989 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.989 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.989 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.989 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:40.989 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.989 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:40.989 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "balancer", 2024-03-28T16:09:40.989 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:40.990 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:40.990 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:40.990 INFO:teuthology.orchestra.run.smithi067.stdout: "active": { 2024-03-28T16:09:40.990 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "active", 2024-03-28T16:09:40.990 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.990 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.990 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.990 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:40.990 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.990 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.990 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.990 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "automatically balance PGs across cluster", 2024-03-28T16:09:40.990 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.990 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.991 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.991 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.991 INFO:teuthology.orchestra.run.smithi067.stdout: "begin_time": { 2024-03-28T16:09:40.991 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "begin_time", 2024-03-28T16:09:40.991 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.991 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.991 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.991 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "0000", 2024-03-28T16:09:40.991 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.991 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.991 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.991 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "beginning time of day to automatically balance", 2024-03-28T16:09:40.991 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-28T16:09:40.991 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.991 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.992 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.992 INFO:teuthology.orchestra.run.smithi067.stdout: "begin_weekday": { 2024-03-28T16:09:40.992 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "begin_weekday", 2024-03-28T16:09:40.992 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "uint", 2024-03-28T16:09:40.992 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.992 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.992 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "0", 2024-03-28T16:09:40.992 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "0", 2024-03-28T16:09:40.992 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "6", 2024-03-28T16:09:40.992 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.992 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-28T16:09:40.992 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-28T16:09:40.992 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.992 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.992 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.993 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_compat_max_iterations": { 2024-03-28T16:09:40.993 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "crush_compat_max_iterations", 2024-03-28T16:09:40.993 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "uint", 2024-03-28T16:09:40.993 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.993 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.993 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "25", 2024-03-28T16:09:40.993 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "1", 2024-03-28T16:09:40.993 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "250", 2024-03-28T16:09:40.993 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.993 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-28T16:09:40.993 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.993 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.993 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.993 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.994 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_compat_metrics": { 2024-03-28T16:09:40.994 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "crush_compat_metrics", 2024-03-28T16:09:40.994 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.994 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.994 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.994 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "pgs,objects,bytes", 2024-03-28T16:09:40.994 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.994 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.994 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.994 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-28T16:09:40.994 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-28T16:09:40.994 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.994 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.994 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.995 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_compat_step": { 2024-03-28T16:09:40.995 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "crush_compat_step", 2024-03-28T16:09:40.995 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "float", 2024-03-28T16:09:40.995 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.995 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.995 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "0.5", 2024-03-28T16:09:40.995 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "0.001", 2024-03-28T16:09:40.995 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "0.999", 2024-03-28T16:09:40.995 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.995 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "aggressiveness of optimization", 2024-03-28T16:09:40.995 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-28T16:09:40.995 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.995 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.995 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.996 INFO:teuthology.orchestra.run.smithi067.stdout: "end_time": { 2024-03-28T16:09:40.996 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "end_time", 2024-03-28T16:09:40.996 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.996 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.996 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.996 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "2359", 2024-03-28T16:09:40.996 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.996 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.996 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.996 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "ending time of day to automatically balance", 2024-03-28T16:09:40.996 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-28T16:09:40.996 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.996 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.996 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.997 INFO:teuthology.orchestra.run.smithi067.stdout: "end_weekday": { 2024-03-28T16:09:40.997 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "end_weekday", 2024-03-28T16:09:40.997 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "uint", 2024-03-28T16:09:40.997 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.997 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.997 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "0", 2024-03-28T16:09:40.997 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "0", 2024-03-28T16:09:40.997 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "6", 2024-03-28T16:09:40.997 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.997 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-28T16:09:40.997 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-28T16:09:40.997 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.997 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.997 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.997 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:40.998 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:40.998 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:40.998 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.998 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.998 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:40.998 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.998 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.998 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:40.998 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:40.998 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:40.998 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:40.998 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:40.998 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:40.998 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:40.998 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:40.999 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:40.999 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:40.999 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:40.999 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:40.999 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:40.999 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:40.999 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:40.999 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:40.999 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:40.999 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:40.999 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:40.999 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:40.999 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:40.999 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:40.999 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.000 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.000 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.000 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.000 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.000 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:41.000 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:41.000 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.000 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.000 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.000 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:41.000 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.000 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.000 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.000 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:41.001 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:41.001 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:41.001 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:41.001 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:41.001 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:41.001 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.001 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.001 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.001 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.001 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.001 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.001 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:41.001 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:41.001 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.002 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.002 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.002 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.002 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.002 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.002 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.002 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.002 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.002 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.002 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.002 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.002 INFO:teuthology.orchestra.run.smithi067.stdout: "min_score": { 2024-03-28T16:09:41.002 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "min_score", 2024-03-28T16:09:41.002 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "float", 2024-03-28T16:09:41.002 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.003 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.003 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "0", 2024-03-28T16:09:41.003 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.003 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.003 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.003 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-28T16:09:41.003 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.003 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.003 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.003 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.003 INFO:teuthology.orchestra.run.smithi067.stdout: "mode": { 2024-03-28T16:09:41.003 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "mode", 2024-03-28T16:09:41.003 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.003 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.003 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.004 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "upmap", 2024-03-28T16:09:41.004 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.004 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.004 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.004 INFO:teuthology.orchestra.run.smithi067.stdout: "crush-compat", 2024-03-28T16:09:41.004 INFO:teuthology.orchestra.run.smithi067.stdout: "none", 2024-03-28T16:09:41.004 INFO:teuthology.orchestra.run.smithi067.stdout: "read", 2024-03-28T16:09:41.004 INFO:teuthology.orchestra.run.smithi067.stdout: "upmap", 2024-03-28T16:09:41.004 INFO:teuthology.orchestra.run.smithi067.stdout: "upmap-read" 2024-03-28T16:09:41.004 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.004 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Balancer mode", 2024-03-28T16:09:41.004 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.004 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.004 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.004 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.005 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_ids": { 2024-03-28T16:09:41.005 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "pool_ids", 2024-03-28T16:09:41.005 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.005 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.005 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.005 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.005 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.005 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.005 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.005 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-28T16:09:41.005 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.005 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.005 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.005 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.005 INFO:teuthology.orchestra.run.smithi067.stdout: "sleep_interval": { 2024-03-28T16:09:41.006 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "sleep_interval", 2024-03-28T16:09:41.006 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "secs", 2024-03-28T16:09:41.006 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.006 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.006 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "60", 2024-03-28T16:09:41.006 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.006 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.006 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.006 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-28T16:09:41.006 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.006 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.006 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.006 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.006 INFO:teuthology.orchestra.run.smithi067.stdout: "upmap_max_deviation": { 2024-03-28T16:09:41.007 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "upmap_max_deviation", 2024-03-28T16:09:41.007 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:41.007 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.007 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.007 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "5", 2024-03-28T16:09:41.007 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "1", 2024-03-28T16:09:41.007 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.007 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.007 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-28T16:09:41.007 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-28T16:09:41.007 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.007 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.007 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.007 INFO:teuthology.orchestra.run.smithi067.stdout: "upmap_max_optimizations": { 2024-03-28T16:09:41.007 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "upmap_max_optimizations", 2024-03-28T16:09:41.008 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "uint", 2024-03-28T16:09:41.008 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.008 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.008 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "10", 2024-03-28T16:09:41.008 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.008 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.008 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.008 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-28T16:09:41.008 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.008 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.008 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.008 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.008 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.008 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.008 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:41.009 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "cephadm", 2024-03-28T16:09:41.009 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:41.009 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:41.009 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:41.009 INFO:teuthology.orchestra.run.smithi067.stdout: "agent_down_multiplier": { 2024-03-28T16:09:41.009 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "agent_down_multiplier", 2024-03-28T16:09:41.009 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "float", 2024-03-28T16:09:41.009 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.009 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.009 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "3.0", 2024-03-28T16:09:41.009 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.009 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.009 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.009 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-28T16:09:41.010 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.010 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.010 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.010 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.010 INFO:teuthology.orchestra.run.smithi067.stdout: "agent_refresh_rate": { 2024-03-28T16:09:41.010 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "agent_refresh_rate", 2024-03-28T16:09:41.010 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "secs", 2024-03-28T16:09:41.010 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.010 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.010 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "20", 2024-03-28T16:09:41.010 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.010 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.010 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.010 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-28T16:09:41.010 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.011 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.011 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.011 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.011 INFO:teuthology.orchestra.run.smithi067.stdout: "agent_starting_port": { 2024-03-28T16:09:41.011 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "agent_starting_port", 2024-03-28T16:09:41.011 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:41.011 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.011 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.011 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "4721", 2024-03-28T16:09:41.011 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.011 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.011 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.011 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-28T16:09:41.011 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.011 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.012 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.012 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.012 INFO:teuthology.orchestra.run.smithi067.stdout: "allow_ptrace": { 2024-03-28T16:09:41.012 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "allow_ptrace", 2024-03-28T16:09:41.012 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.012 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.012 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.012 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.012 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.012 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.012 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.012 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-28T16:09:41.012 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-28T16:09:41.012 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.013 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.013 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.013 INFO:teuthology.orchestra.run.smithi067.stdout: "autotune_interval": { 2024-03-28T16:09:41.013 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "autotune_interval", 2024-03-28T16:09:41.013 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "secs", 2024-03-28T16:09:41.013 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.013 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.013 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "600", 2024-03-28T16:09:41.013 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.013 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.013 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.013 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-28T16:09:41.013 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.013 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.013 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.014 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.014 INFO:teuthology.orchestra.run.smithi067.stdout: "autotune_memory_target_ratio": { 2024-03-28T16:09:41.014 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "autotune_memory_target_ratio", 2024-03-28T16:09:41.014 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "float", 2024-03-28T16:09:41.014 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.014 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.014 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "0.7", 2024-03-28T16:09:41.014 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.014 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.014 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.014 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-28T16:09:41.014 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.014 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.014 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.015 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.015 INFO:teuthology.orchestra.run.smithi067.stdout: "cephadm_log_destination": { 2024-03-28T16:09:41.015 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "cephadm_log_destination", 2024-03-28T16:09:41.015 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.015 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.015 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.015 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.015 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.015 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.015 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.015 INFO:teuthology.orchestra.run.smithi067.stdout: "file", 2024-03-28T16:09:41.015 INFO:teuthology.orchestra.run.smithi067.stdout: "file,syslog", 2024-03-28T16:09:41.015 INFO:teuthology.orchestra.run.smithi067.stdout: "syslog" 2024-03-28T16:09:41.015 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.015 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-28T16:09:41.016 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.016 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.016 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.016 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.016 INFO:teuthology.orchestra.run.smithi067.stdout: "cgroups_split": { 2024-03-28T16:09:41.016 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "cgroups_split", 2024-03-28T16:09:41.016 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.016 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.016 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.016 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:41.016 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.016 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.016 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.016 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-28T16:09:41.016 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.017 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.017 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.017 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.017 INFO:teuthology.orchestra.run.smithi067.stdout: "config_checks_enabled": { 2024-03-28T16:09:41.017 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "config_checks_enabled", 2024-03-28T16:09:41.017 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.017 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.017 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.017 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.017 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.017 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.017 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.017 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-28T16:09:41.017 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.017 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.018 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.018 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.018 INFO:teuthology.orchestra.run.smithi067.stdout: "config_dashboard": { 2024-03-28T16:09:41.018 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "config_dashboard", 2024-03-28T16:09:41.018 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.018 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.018 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.018 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:41.018 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.018 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.018 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.018 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-28T16:09:41.018 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.018 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.019 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.019 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.019 INFO:teuthology.orchestra.run.smithi067.stdout: "container_image_alertmanager": { 2024-03-28T16:09:41.019 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "container_image_alertmanager", 2024-03-28T16:09:41.019 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.019 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.019 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.019 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-28T16:09:41.019 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.019 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.019 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.019 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Prometheus container image", 2024-03-28T16:09:41.019 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.019 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.019 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.020 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.020 INFO:teuthology.orchestra.run.smithi067.stdout: "container_image_base": { 2024-03-28T16:09:41.020 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "container_image_base", 2024-03-28T16:09:41.020 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.020 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.020 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.020 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-28T16:09:41.020 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.020 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.020 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.020 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Container image name, without the tag", 2024-03-28T16:09:41.020 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.020 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.020 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.021 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.021 INFO:teuthology.orchestra.run.smithi067.stdout: "container_image_elasticsearch": { 2024-03-28T16:09:41.021 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "container_image_elasticsearch", 2024-03-28T16:09:41.021 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.021 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.021 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.021 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-28T16:09:41.021 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.021 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.021 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.021 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "elasticsearch container image", 2024-03-28T16:09:41.021 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.021 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.021 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.022 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.022 INFO:teuthology.orchestra.run.smithi067.stdout: "container_image_grafana": { 2024-03-28T16:09:41.022 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "container_image_grafana", 2024-03-28T16:09:41.022 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.022 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.022 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.022 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-28T16:09:41.022 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.022 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.022 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.022 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Prometheus container image", 2024-03-28T16:09:41.022 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.022 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.022 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.023 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.023 INFO:teuthology.orchestra.run.smithi067.stdout: "container_image_haproxy": { 2024-03-28T16:09:41.023 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "container_image_haproxy", 2024-03-28T16:09:41.023 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.023 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.023 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.023 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-28T16:09:41.023 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.023 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.023 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.023 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "HAproxy container image", 2024-03-28T16:09:41.023 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.023 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.024 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.024 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.024 INFO:teuthology.orchestra.run.smithi067.stdout: "container_image_jaeger_agent": { 2024-03-28T16:09:41.024 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "container_image_jaeger_agent", 2024-03-28T16:09:41.024 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.024 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.024 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.024 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-28T16:09:41.024 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.024 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.024 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.024 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Jaeger agent container image", 2024-03-28T16:09:41.024 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.024 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.025 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.025 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.025 INFO:teuthology.orchestra.run.smithi067.stdout: "container_image_jaeger_collector": { 2024-03-28T16:09:41.025 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "container_image_jaeger_collector", 2024-03-28T16:09:41.025 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.025 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.025 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.025 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-28T16:09:41.025 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.025 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.025 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.025 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Jaeger collector container image", 2024-03-28T16:09:41.025 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.025 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.025 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.026 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.026 INFO:teuthology.orchestra.run.smithi067.stdout: "container_image_jaeger_query": { 2024-03-28T16:09:41.026 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "container_image_jaeger_query", 2024-03-28T16:09:41.026 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.026 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.026 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.026 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-28T16:09:41.026 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.026 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.026 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.026 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Jaeger query container image", 2024-03-28T16:09:41.026 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.026 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.027 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.027 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.027 INFO:teuthology.orchestra.run.smithi067.stdout: "container_image_keepalived": { 2024-03-28T16:09:41.027 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "container_image_keepalived", 2024-03-28T16:09:41.027 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.027 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.027 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.027 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-28T16:09:41.027 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.027 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.027 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.027 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Keepalived container image", 2024-03-28T16:09:41.027 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.027 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.028 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.028 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.028 INFO:teuthology.orchestra.run.smithi067.stdout: "container_image_loki": { 2024-03-28T16:09:41.028 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "container_image_loki", 2024-03-28T16:09:41.028 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.028 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.028 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.028 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-28T16:09:41.028 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.028 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.028 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.028 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Loki container image", 2024-03-28T16:09:41.028 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.028 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.028 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.029 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.029 INFO:teuthology.orchestra.run.smithi067.stdout: "container_image_node_exporter": { 2024-03-28T16:09:41.029 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "container_image_node_exporter", 2024-03-28T16:09:41.029 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.029 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.029 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.029 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-28T16:09:41.029 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.029 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.029 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.029 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Prometheus container image", 2024-03-28T16:09:41.029 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.029 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.029 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.029 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.030 INFO:teuthology.orchestra.run.smithi067.stdout: "container_image_nvmeof": { 2024-03-28T16:09:41.030 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "container_image_nvmeof", 2024-03-28T16:09:41.030 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.030 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.030 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.030 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-28T16:09:41.030 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.030 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.030 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.030 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Nvme-of container image", 2024-03-28T16:09:41.030 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.030 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.030 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.030 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.031 INFO:teuthology.orchestra.run.smithi067.stdout: "container_image_prometheus": { 2024-03-28T16:09:41.031 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "container_image_prometheus", 2024-03-28T16:09:41.031 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.031 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.031 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.031 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-28T16:09:41.031 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.031 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.031 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.031 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Prometheus container image", 2024-03-28T16:09:41.031 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.031 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.031 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.031 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.031 INFO:teuthology.orchestra.run.smithi067.stdout: "container_image_promtail": { 2024-03-28T16:09:41.031 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "container_image_promtail", 2024-03-28T16:09:41.032 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.032 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.032 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.032 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-28T16:09:41.032 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.032 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.032 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.032 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Promtail container image", 2024-03-28T16:09:41.032 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.032 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.032 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.032 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.032 INFO:teuthology.orchestra.run.smithi067.stdout: "container_image_snmp_gateway": { 2024-03-28T16:09:41.032 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "container_image_snmp_gateway", 2024-03-28T16:09:41.033 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.033 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.033 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.033 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-28T16:09:41.033 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.033 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.033 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.033 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "SNMP Gateway container image", 2024-03-28T16:09:41.033 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.033 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.033 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.033 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.033 INFO:teuthology.orchestra.run.smithi067.stdout: "container_init": { 2024-03-28T16:09:41.033 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "container_init", 2024-03-28T16:09:41.033 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.034 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.034 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.034 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:41.034 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.034 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.034 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.034 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Run podman/docker with `--init`", 2024-03-28T16:09:41.034 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.034 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.034 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.034 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.034 INFO:teuthology.orchestra.run.smithi067.stdout: "daemon_cache_timeout": { 2024-03-28T16:09:41.034 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "daemon_cache_timeout", 2024-03-28T16:09:41.034 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "secs", 2024-03-28T16:09:41.034 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.035 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.035 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "600", 2024-03-28T16:09:41.035 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.035 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.035 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.035 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-28T16:09:41.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:40.159+0000 7f156c000640 1 -- 172.21.15.67:0/1170207559 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f154c0a2800 msgr2=0x7f154c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:41.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:40.159+0000 7f156c000640 1 --2- 172.21.15.67:0/1170207559 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f154c0a2800 0x7f154c0a4cc0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f1558005fd0 tx=0x7f1558005850 comp rx=0 tx=0).stop 2024-03-28T16:09:41.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:40.159+0000 7f156c000640 1 -- 172.21.15.67:0/1170207559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1564104ad0 msgr2=0x7f156419a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:41.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:40.159+0000 7f156c000640 1 --2- 172.21.15.67:0/1170207559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1564104ad0 0x7f156419a0a0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f1550000c00 tx=0x7f15500042f0 comp rx=0 tx=0).stop 2024-03-28T16:09:41.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:40.159+0000 7f156c000640 1 -- 172.21.15.67:0/1170207559 shutdown_connections 2024-03-28T16:09:41.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:40.159+0000 7f156c000640 1 --2- 172.21.15.67:0/1170207559 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f154c0a2800 0x7f154c0a4cc0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:41.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:40.159+0000 7f156c000640 1 --2- 172.21.15.67:0/1170207559 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1564105490 0x7f15641a1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:41.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:40.159+0000 7f156c000640 1 --2- 172.21.15.67:0/1170207559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1564104ad0 0x7f156419a0a0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:41.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:40.159+0000 7f156c000640 1 --2- 172.21.15.67:0/1170207559 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f15641038e0 0x7f1564199b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:41.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:40.159+0000 7f156c000640 1 -- 172.21.15.67:0/1170207559 >> 172.21.15.67:0/1170207559 conn(0x7f15640fee50 msgr2=0x7f1564107e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:41.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:40.160+0000 7f156c000640 1 -- 172.21.15.67:0/1170207559 shutdown_connections 2024-03-28T16:09:41.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:40.160+0000 7f156c000640 1 -- 172.21.15.67:0/1170207559 wait complete. 2024-03-28T16:09:41.039 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.039 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.039 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.039 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.039 INFO:teuthology.orchestra.run.smithi067.stdout: "default_cephadm_command_timeout": { 2024-03-28T16:09:41.039 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "default_cephadm_command_timeout", 2024-03-28T16:09:41.039 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:41.039 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.039 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.039 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "900", 2024-03-28T16:09:41.039 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.039 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.040 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.040 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-28T16:09:41.040 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.040 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.040 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.040 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.040 INFO:teuthology.orchestra.run.smithi067.stdout: "default_registry": { 2024-03-28T16:09:41.040 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "default_registry", 2024-03-28T16:09:41.040 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.040 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.040 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.040 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "docker.io", 2024-03-28T16:09:41.040 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.040 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.041 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.041 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-28T16:09:41.041 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.041 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.041 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.041 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.041 INFO:teuthology.orchestra.run.smithi067.stdout: "device_cache_timeout": { 2024-03-28T16:09:41.041 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "device_cache_timeout", 2024-03-28T16:09:41.041 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "secs", 2024-03-28T16:09:41.041 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.041 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.041 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "1800", 2024-03-28T16:09:41.041 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.041 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.042 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.042 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "seconds to cache device inventory", 2024-03-28T16:09:41.042 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.042 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.042 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.042 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.042 INFO:teuthology.orchestra.run.smithi067.stdout: "device_enhanced_scan": { 2024-03-28T16:09:41.042 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "device_enhanced_scan", 2024-03-28T16:09:41.042 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.042 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.042 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.042 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.042 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.042 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.043 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.043 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-28T16:09:41.043 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.043 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.043 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.043 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.043 INFO:teuthology.orchestra.run.smithi067.stdout: "facts_cache_timeout": { 2024-03-28T16:09:41.043 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "facts_cache_timeout", 2024-03-28T16:09:41.043 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "secs", 2024-03-28T16:09:41.043 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.043 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.043 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "60", 2024-03-28T16:09:41.043 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.043 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.044 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.044 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "seconds to cache host facts data", 2024-03-28T16:09:41.044 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.044 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.044 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.044 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.044 INFO:teuthology.orchestra.run.smithi067.stdout: "host_check_interval": { 2024-03-28T16:09:41.044 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "host_check_interval", 2024-03-28T16:09:41.044 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "secs", 2024-03-28T16:09:41.044 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.044 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.044 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "600", 2024-03-28T16:09:41.044 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.044 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.045 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.045 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "how frequently to perform a host check", 2024-03-28T16:09:41.045 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.045 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.045 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.045 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.045 INFO:teuthology.orchestra.run.smithi067.stdout: "hw_monitoring": { 2024-03-28T16:09:41.045 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "hw_monitoring", 2024-03-28T16:09:41.045 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.045 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.045 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.045 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.045 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.045 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.045 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.046 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-28T16:09:41.046 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.046 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.046 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.046 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.046 INFO:teuthology.orchestra.run.smithi067.stdout: "inventory_list_all": { 2024-03-28T16:09:41.046 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "inventory_list_all", 2024-03-28T16:09:41.046 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.046 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.046 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.046 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.046 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.046 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.046 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.047 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-28T16:09:41.047 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.047 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.047 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.047 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.047 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:41.047 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:41.047 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.047 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.047 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.047 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.047 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.047 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.047 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.047 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:41.048 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:41.048 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:41.048 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:41.048 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:41.048 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:41.048 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.048 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.048 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.048 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.048 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.048 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.048 INFO:teuthology.orchestra.run.smithi067.stdout: "log_refresh_metadata": { 2024-03-28T16:09:41.048 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_refresh_metadata", 2024-03-28T16:09:41.048 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.048 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.049 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.049 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.049 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.049 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.049 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.049 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-28T16:09:41.049 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.049 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.049 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.049 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.049 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:41.049 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:41.049 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.049 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.050 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.050 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:41.050 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.050 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.050 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.050 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-28T16:09:41.050 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.050 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.050 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.050 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.050 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:41.050 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:41.050 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.050 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.051 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.051 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:41.051 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.051 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.051 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.051 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:41.051 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:41.051 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:41.051 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:41.051 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:41.051 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:41.051 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.051 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.051 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.051 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.052 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.052 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.052 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:41.052 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:41.052 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.052 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.052 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.052 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.052 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.052 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.052 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.052 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.052 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.052 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.052 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.052 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.053 INFO:teuthology.orchestra.run.smithi067.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-28T16:09:41.053 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-28T16:09:41.053 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.053 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.053 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.053 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.053 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.053 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.053 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.053 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-28T16:09:41.053 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.053 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.053 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.053 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.054 INFO:teuthology.orchestra.run.smithi067.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-28T16:09:41.054 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-28T16:09:41.054 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.054 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.054 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.054 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "*", 2024-03-28T16:09:41.054 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.054 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.054 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.054 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-28T16:09:41.054 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.054 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.054 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.054 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.054 INFO:teuthology.orchestra.run.smithi067.stdout: "max_count_per_host": { 2024-03-28T16:09:41.055 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "max_count_per_host", 2024-03-28T16:09:41.055 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:41.055 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.055 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.055 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "10", 2024-03-28T16:09:41.055 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.055 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.055 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.055 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "max number of daemons per service per host", 2024-03-28T16:09:41.055 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.055 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.055 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.055 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.055 INFO:teuthology.orchestra.run.smithi067.stdout: "max_osd_draining_count": { 2024-03-28T16:09:41.056 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "max_osd_draining_count", 2024-03-28T16:09:41.056 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:41.056 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.056 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.056 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "10", 2024-03-28T16:09:41.056 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.056 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.056 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.056 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-28T16:09:41.056 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.056 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.056 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.056 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.056 INFO:teuthology.orchestra.run.smithi067.stdout: "migration_current": { 2024-03-28T16:09:41.057 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "migration_current", 2024-03-28T16:09:41.057 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:41.057 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.057 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.057 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.057 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.057 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.057 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.057 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "internal - do not modify", 2024-03-28T16:09:41.057 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.057 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.057 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.057 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.057 INFO:teuthology.orchestra.run.smithi067.stdout: "mode": { 2024-03-28T16:09:41.058 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "mode", 2024-03-28T16:09:41.058 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.058 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.058 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.058 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "root", 2024-03-28T16:09:41.058 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.058 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.058 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.058 INFO:teuthology.orchestra.run.smithi067.stdout: "cephadm-package", 2024-03-28T16:09:41.058 INFO:teuthology.orchestra.run.smithi067.stdout: "root" 2024-03-28T16:09:41.058 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.058 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "mode for remote execution of cephadm", 2024-03-28T16:09:41.058 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.058 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.058 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.059 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.059 INFO:teuthology.orchestra.run.smithi067.stdout: "oob_default_addr": { 2024-03-28T16:09:41.059 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "oob_default_addr", 2024-03-28T16:09:41.059 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.059 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.059 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.059 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "169.254.1.1", 2024-03-28T16:09:41.059 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.059 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.059 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.059 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-28T16:09:41.059 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.059 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.060 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.060 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.060 INFO:teuthology.orchestra.run.smithi067.stdout: "prometheus_alerts_path": { 2024-03-28T16:09:41.060 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "prometheus_alerts_path", 2024-03-28T16:09:41.060 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.060 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.060 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.060 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-28T16:09:41.060 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.060 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.061 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.061 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-28T16:09:41.061 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.061 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.061 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.061 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.061 INFO:teuthology.orchestra.run.smithi067.stdout: "registry_insecure": { 2024-03-28T16:09:41.062 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "registry_insecure", 2024-03-28T16:09:41.062 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.062 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.062 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.062 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.062 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.062 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.063 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.063 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-28T16:09:41.063 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.063 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.063 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.063 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.063 INFO:teuthology.orchestra.run.smithi067.stdout: "registry_password": { 2024-03-28T16:09:41.064 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "registry_password", 2024-03-28T16:09:41.064 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.064 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.064 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.064 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.064 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.064 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.064 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.065 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-28T16:09:41.065 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.065 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.065 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.065 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.065 INFO:teuthology.orchestra.run.smithi067.stdout: "registry_url": { 2024-03-28T16:09:41.065 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "registry_url", 2024-03-28T16:09:41.066 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.066 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.066 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.066 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.066 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.066 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.066 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.067 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-28T16:09:41.067 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.067 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.067 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.067 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.067 INFO:teuthology.orchestra.run.smithi067.stdout: "registry_username": { 2024-03-28T16:09:41.067 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "registry_username", 2024-03-28T16:09:41.067 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.068 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.068 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.068 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.068 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.068 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.068 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.068 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-28T16:09:41.069 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.069 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.069 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.069 INFO:teuthology.orchestra.run.smithi067.stdout: "secure_monitoring_stack": { 2024-03-28T16:09:41.069 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "secure_monitoring_stack", 2024-03-28T16:09:41.069 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.069 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.069 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.069 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.070 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.070 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.070 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.070 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-28T16:09:41.070 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.070 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.070 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.070 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.070 INFO:teuthology.orchestra.run.smithi067.stdout: "service_discovery_port": { 2024-03-28T16:09:41.070 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "service_discovery_port", 2024-03-28T16:09:41.070 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:41.070 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.070 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.070 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "8765", 2024-03-28T16:09:41.071 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.071 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.071 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.071 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "cephadm service discovery port", 2024-03-28T16:09:41.071 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.071 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.071 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.071 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.071 INFO:teuthology.orchestra.run.smithi067.stdout: "ssh_config_file": { 2024-03-28T16:09:41.071 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "ssh_config_file", 2024-03-28T16:09:41.071 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.071 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.071 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.071 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.071 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.072 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.072 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.072 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-28T16:09:41.072 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.072 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.072 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.072 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.072 INFO:teuthology.orchestra.run.smithi067.stdout: "use_agent": { 2024-03-28T16:09:41.072 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "use_agent", 2024-03-28T16:09:41.072 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.072 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.072 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.072 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.072 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.073 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.073 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.073 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-28T16:09:41.073 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.073 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.073 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.073 INFO:teuthology.orchestra.run.smithi067.stdout: "use_repo_digest": { 2024-03-28T16:09:41.073 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "use_repo_digest", 2024-03-28T16:09:41.073 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.073 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.073 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.073 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:41.073 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.073 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.074 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.074 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-28T16:09:41.074 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.074 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.074 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.074 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.074 INFO:teuthology.orchestra.run.smithi067.stdout: "warn_on_failed_host_check": { 2024-03-28T16:09:41.074 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "warn_on_failed_host_check", 2024-03-28T16:09:41.074 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.074 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.074 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.074 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:41.074 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.075 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.075 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.075 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "raise a health warning if the host check fails", 2024-03-28T16:09:41.075 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.075 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.075 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.075 INFO:teuthology.orchestra.run.smithi067.stdout: "warn_on_stray_daemons": { 2024-03-28T16:09:41.075 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "warn_on_stray_daemons", 2024-03-28T16:09:41.075 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.075 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.075 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.075 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:41.075 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.075 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.076 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.076 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-28T16:09:41.076 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.076 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.076 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.076 INFO:teuthology.orchestra.run.smithi067.stdout: "warn_on_stray_hosts": { 2024-03-28T16:09:41.076 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "warn_on_stray_hosts", 2024-03-28T16:09:41.076 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.076 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.076 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.076 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:41.076 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.076 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.077 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.077 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-28T16:09:41.077 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.077 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.077 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.077 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.077 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:41.077 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "crash", 2024-03-28T16:09:41.077 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:41.077 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:41.077 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:41.077 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:41.077 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:41.078 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.078 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.078 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.078 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.078 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.078 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.078 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.078 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:41.078 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:41.078 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:41.078 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:41.078 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:41.078 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:41.078 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.079 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.079 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.079 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.079 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.079 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.079 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:41.079 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:41.079 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.079 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.079 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.079 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.079 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.079 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.079 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.079 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.080 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.080 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.080 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.080 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:41.080 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:41.080 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.080 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.080 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.080 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:41.080 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.080 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.080 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.080 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:41.080 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:41.080 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:41.081 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:41.081 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:41.081 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:41.081 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.081 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.081 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.081 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.081 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.081 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:41.081 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:41.081 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.081 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.081 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.082 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.082 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.082 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.082 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.082 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.082 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.082 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.082 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.082 INFO:teuthology.orchestra.run.smithi067.stdout: "retain_interval": { 2024-03-28T16:09:41.082 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "retain_interval", 2024-03-28T16:09:41.082 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "secs", 2024-03-28T16:09:41.082 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.082 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.082 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "31536000", 2024-03-28T16:09:41.083 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.083 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.083 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.083 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-28T16:09:41.083 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.083 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.083 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.083 INFO:teuthology.orchestra.run.smithi067.stdout: "warn_recent_interval": { 2024-03-28T16:09:41.083 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "warn_recent_interval", 2024-03-28T16:09:41.083 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "secs", 2024-03-28T16:09:41.083 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.083 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.083 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "1209600", 2024-03-28T16:09:41.083 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.084 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.084 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.084 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-28T16:09:41.084 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.084 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.084 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.084 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.084 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.084 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:41.084 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "dashboard", 2024-03-28T16:09:41.084 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:41.084 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:41.084 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:41.084 INFO:teuthology.orchestra.run.smithi067.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-28T16:09:41.085 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-28T16:09:41.085 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:41.085 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.085 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.085 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "10", 2024-03-28T16:09:41.085 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.085 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.085 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.085 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.085 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.085 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.085 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.085 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.085 INFO:teuthology.orchestra.run.smithi067.stdout: "ALERTMANAGER_API_HOST": { 2024-03-28T16:09:41.085 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-28T16:09:41.086 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.086 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.086 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.086 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.086 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.086 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.086 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.086 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.086 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.086 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.086 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.086 INFO:teuthology.orchestra.run.smithi067.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-28T16:09:41.086 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-28T16:09:41.087 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.087 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.087 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.087 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:41.087 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.087 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.087 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.087 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.087 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.087 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.087 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.087 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.087 INFO:teuthology.orchestra.run.smithi067.stdout: "AUDIT_API_ENABLED": { 2024-03-28T16:09:41.087 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "AUDIT_API_ENABLED", 2024-03-28T16:09:41.088 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.088 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.088 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.088 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.088 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.088 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.088 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.088 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.088 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.088 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.088 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.088 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.088 INFO:teuthology.orchestra.run.smithi067.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-28T16:09:41.088 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-28T16:09:41.089 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.089 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.089 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.089 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:41.089 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.089 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.089 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.089 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.089 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.089 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.089 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.089 INFO:teuthology.orchestra.run.smithi067.stdout: "ENABLE_BROWSABLE_API": { 2024-03-28T16:09:41.089 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-28T16:09:41.090 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.090 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.090 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.090 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:41.090 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.090 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.090 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.090 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.090 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.090 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.090 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.090 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.090 INFO:teuthology.orchestra.run.smithi067.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-28T16:09:41.090 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-28T16:09:41.091 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.091 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.091 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.091 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:41.091 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.091 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.091 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.091 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.091 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.091 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.091 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.091 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.091 INFO:teuthology.orchestra.run.smithi067.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-28T16:09:41.092 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-28T16:09:41.092 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.092 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.092 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.092 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:41.092 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.092 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.092 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.092 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.092 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.092 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.092 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.092 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.092 INFO:teuthology.orchestra.run.smithi067.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-28T16:09:41.093 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-28T16:09:41.093 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.093 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.093 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.093 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:41.093 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.093 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.093 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.093 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.093 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.093 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.093 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.093 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.093 INFO:teuthology.orchestra.run.smithi067.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-28T16:09:41.093 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-28T16:09:41.094 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.094 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.094 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.094 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:41.094 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.094 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.094 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.094 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.094 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.094 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.094 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.094 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.094 INFO:teuthology.orchestra.run.smithi067.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-28T16:09:41.094 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-28T16:09:41.095 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.095 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.095 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.095 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:41.095 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.095 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.095 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.095 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.095 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.095 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.095 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.095 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.095 INFO:teuthology.orchestra.run.smithi067.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-28T16:09:41.095 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-28T16:09:41.095 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.096 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.096 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.096 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:41.096 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.096 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.096 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.096 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.096 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.096 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.096 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.096 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.096 INFO:teuthology.orchestra.run.smithi067.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-28T16:09:41.096 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-28T16:09:41.096 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.096 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.097 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.097 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:41.097 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.097 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.097 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.097 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.097 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.097 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.097 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.097 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.097 INFO:teuthology.orchestra.run.smithi067.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-28T16:09:41.097 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-28T16:09:41.097 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.097 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.097 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.098 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.098 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.098 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.098 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.098 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.098 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.098 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.098 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.098 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.098 INFO:teuthology.orchestra.run.smithi067.stdout: "GRAFANA_API_PASSWORD": { 2024-03-28T16:09:41.098 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-28T16:09:41.098 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.098 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.098 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.098 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "admin", 2024-03-28T16:09:41.099 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.099 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.099 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.099 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.099 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.099 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.099 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.099 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.099 INFO:teuthology.orchestra.run.smithi067.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-28T16:09:41.099 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-28T16:09:41.099 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.099 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.099 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.099 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:41.099 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.100 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.100 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.100 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.100 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.100 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.100 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.100 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.100 INFO:teuthology.orchestra.run.smithi067.stdout: "GRAFANA_API_URL": { 2024-03-28T16:09:41.100 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "GRAFANA_API_URL", 2024-03-28T16:09:41.100 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.100 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.100 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.100 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.100 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.100 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.101 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.101 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.101 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.101 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.101 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.101 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.101 INFO:teuthology.orchestra.run.smithi067.stdout: "GRAFANA_API_USERNAME": { 2024-03-28T16:09:41.101 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-28T16:09:41.101 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.101 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.101 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.101 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "admin", 2024-03-28T16:09:41.101 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.101 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.101 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.102 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.102 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.102 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.102 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.102 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.102 INFO:teuthology.orchestra.run.smithi067.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-28T16:09:41.102 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-28T16:09:41.102 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.102 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.102 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.102 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.102 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.102 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.102 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.102 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.103 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.103 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.103 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.103 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.103 INFO:teuthology.orchestra.run.smithi067.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-28T16:09:41.103 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-28T16:09:41.103 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.103 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.103 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.103 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.103 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.103 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.103 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.103 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.103 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.104 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.104 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.104 INFO:teuthology.orchestra.run.smithi067.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-28T16:09:41.104 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-28T16:09:41.104 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.104 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.104 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.104 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:41.104 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.104 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.104 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.104 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.104 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.105 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.105 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.105 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.105 INFO:teuthology.orchestra.run.smithi067.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-28T16:09:41.105 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-28T16:09:41.105 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.105 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.105 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.105 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.105 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.105 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.105 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.105 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.105 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.105 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.106 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.106 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.106 INFO:teuthology.orchestra.run.smithi067.stdout: "MULTICLUSTER_CONFIG": { 2024-03-28T16:09:41.106 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-28T16:09:41.106 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.106 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.106 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.106 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "{}", 2024-03-28T16:09:41.106 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.106 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.106 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.106 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.106 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.106 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.107 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.107 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.107 INFO:teuthology.orchestra.run.smithi067.stdout: "PROMETHEUS_API_HOST": { 2024-03-28T16:09:41.107 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-28T16:09:41.107 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.107 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.107 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.107 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.107 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.107 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.107 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.107 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.107 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.107 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.107 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.108 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.108 INFO:teuthology.orchestra.run.smithi067.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-28T16:09:41.108 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-28T16:09:41.108 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.108 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.108 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.108 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:41.108 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.108 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.108 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.108 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.108 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.108 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.108 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.108 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.109 INFO:teuthology.orchestra.run.smithi067.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-28T16:09:41.109 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-28T16:09:41.109 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.109 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.109 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.109 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.109 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.109 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.109 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.109 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.109 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.109 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.109 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.109 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.109 INFO:teuthology.orchestra.run.smithi067.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-28T16:09:41.110 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-28T16:09:41.110 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.110 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.110 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.110 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.110 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.110 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.110 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.110 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.110 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.110 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.110 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.110 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.110 INFO:teuthology.orchestra.run.smithi067.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-28T16:09:41.110 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-28T16:09:41.111 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.111 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.111 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.111 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:41.111 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.111 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.111 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.111 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.111 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.111 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.111 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.111 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.111 INFO:teuthology.orchestra.run.smithi067.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-28T16:09:41.111 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-28T16:09:41.111 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.112 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.112 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.112 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:41.112 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.112 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.112 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.112 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.112 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.112 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.112 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.112 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.112 INFO:teuthology.orchestra.run.smithi067.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-28T16:09:41.112 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-28T16:09:41.112 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.112 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.113 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.113 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.113 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.113 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.113 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.113 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.113 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.113 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.113 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.113 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.113 INFO:teuthology.orchestra.run.smithi067.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-28T16:09:41.113 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-28T16:09:41.113 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.113 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.113 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.114 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.114 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.114 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.114 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.114 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.114 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.114 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.114 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.114 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.114 INFO:teuthology.orchestra.run.smithi067.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-28T16:09:41.114 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-28T16:09:41.114 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.114 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.114 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.114 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.115 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.115 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.115 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.115 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.115 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.115 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.115 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.115 INFO:teuthology.orchestra.run.smithi067.stdout: "PWD_POLICY_ENABLED": { 2024-03-28T16:09:41.115 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-28T16:09:41.115 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.115 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.115 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.115 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:41.115 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.116 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.116 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.116 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.116 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.116 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.116 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.116 INFO:teuthology.orchestra.run.smithi067.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-28T16:09:41.116 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-28T16:09:41.116 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.116 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.116 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.116 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-28T16:09:41.116 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.116 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.117 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.117 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.117 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.117 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.117 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.117 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.117 INFO:teuthology.orchestra.run.smithi067.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-28T16:09:41.117 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-28T16:09:41.117 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:41.117 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.117 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.117 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "10", 2024-03-28T16:09:41.117 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.117 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.117 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.117 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.118 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.118 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.118 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.118 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.118 INFO:teuthology.orchestra.run.smithi067.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-28T16:09:41.118 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-28T16:09:41.118 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:41.118 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.118 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.118 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "8", 2024-03-28T16:09:41.118 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.118 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.118 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.119 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.119 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.119 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.119 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.119 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.119 INFO:teuthology.orchestra.run.smithi067.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-28T16:09:41.119 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-28T16:09:41.119 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:41.119 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.119 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.119 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "45", 2024-03-28T16:09:41.119 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.119 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.119 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.119 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.120 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.120 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.120 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.120 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.120 INFO:teuthology.orchestra.run.smithi067.stdout: "RGW_API_ACCESS_KEY": { 2024-03-28T16:09:41.120 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-28T16:09:41.120 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.120 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.120 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.120 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.120 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.120 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.120 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.120 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.120 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.121 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.121 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.121 INFO:teuthology.orchestra.run.smithi067.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-28T16:09:41.121 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-28T16:09:41.121 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.121 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.121 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.121 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "admin", 2024-03-28T16:09:41.121 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.121 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.121 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.121 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.121 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.121 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.122 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.122 INFO:teuthology.orchestra.run.smithi067.stdout: "RGW_API_SECRET_KEY": { 2024-03-28T16:09:41.122 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-28T16:09:41.122 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.122 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.122 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.122 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.122 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.122 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.122 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.122 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.122 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.122 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.123 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.123 INFO:teuthology.orchestra.run.smithi067.stdout: "RGW_API_SSL_VERIFY": { 2024-03-28T16:09:41.123 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-28T16:09:41.123 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.123 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.123 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.123 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:41.123 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.123 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.123 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.123 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.123 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.123 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.124 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.124 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.124 INFO:teuthology.orchestra.run.smithi067.stdout: "UNSAFE_TLS_v1_2": { 2024-03-28T16:09:41.124 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-28T16:09:41.124 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.124 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.124 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.124 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.124 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.124 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.124 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.124 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.124 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.124 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.124 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.125 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.125 INFO:teuthology.orchestra.run.smithi067.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-28T16:09:41.125 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-28T16:09:41.125 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:41.125 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.125 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.125 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "0", 2024-03-28T16:09:41.125 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.125 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.125 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.125 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.125 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.125 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.125 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.125 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.126 INFO:teuthology.orchestra.run.smithi067.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-28T16:09:41.126 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-28T16:09:41.126 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:41.126 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.126 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.126 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "10", 2024-03-28T16:09:41.126 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.126 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.126 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.126 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.126 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.126 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.126 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.126 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.127 INFO:teuthology.orchestra.run.smithi067.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-28T16:09:41.127 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-28T16:09:41.127 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:41.127 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.127 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.127 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "5", 2024-03-28T16:09:41.127 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.127 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.127 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.127 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.127 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.127 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.127 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.127 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.127 INFO:teuthology.orchestra.run.smithi067.stdout: "cross_origin_url": { 2024-03-28T16:09:41.128 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "cross_origin_url", 2024-03-28T16:09:41.128 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.128 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.128 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.128 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.128 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.128 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.128 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.128 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.128 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.128 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.128 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.128 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.128 INFO:teuthology.orchestra.run.smithi067.stdout: "crt_file": { 2024-03-28T16:09:41.129 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "crt_file", 2024-03-28T16:09:41.129 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.129 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.129 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.129 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.129 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.129 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.129 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.129 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.129 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.129 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.129 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.129 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.129 INFO:teuthology.orchestra.run.smithi067.stdout: "debug": { 2024-03-28T16:09:41.129 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "debug", 2024-03-28T16:09:41.130 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.130 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.130 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.130 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.130 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.130 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.130 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.130 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Enable/disable debug options", 2024-03-28T16:09:41.130 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.130 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.130 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.130 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.130 INFO:teuthology.orchestra.run.smithi067.stdout: "jwt_token_ttl": { 2024-03-28T16:09:41.130 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "jwt_token_ttl", 2024-03-28T16:09:41.130 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:41.131 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.131 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.131 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "28800", 2024-03-28T16:09:41.131 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.131 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.131 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.131 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.131 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.131 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.131 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.131 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.131 INFO:teuthology.orchestra.run.smithi067.stdout: "key_file": { 2024-03-28T16:09:41.131 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "key_file", 2024-03-28T16:09:41.131 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.132 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.132 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.132 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.132 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.132 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.132 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.132 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.132 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.132 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.132 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.132 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:41.132 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:41.132 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.133 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.133 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.133 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.133 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.133 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.133 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.133 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:41.133 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:41.133 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:41.133 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:41.133 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:41.133 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:41.133 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.133 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.133 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.134 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.134 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.134 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.134 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:41.134 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:41.134 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.134 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.134 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.134 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.134 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.134 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.134 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.134 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.134 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.134 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.135 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.135 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.135 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:41.135 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:41.135 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.135 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.135 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.135 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:41.135 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.135 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.135 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.135 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:41.135 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:41.135 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:41.135 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:41.136 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:41.136 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:41.136 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.136 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.136 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.136 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.136 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.136 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.136 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:41.136 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:41.136 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.136 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.136 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.136 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.137 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.137 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.137 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.137 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.137 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.137 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.137 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.137 INFO:teuthology.orchestra.run.smithi067.stdout: "motd": { 2024-03-28T16:09:41.137 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "motd", 2024-03-28T16:09:41.137 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.137 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.137 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.137 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.137 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.138 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.138 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.138 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "The message of the day", 2024-03-28T16:09:41.138 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.138 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.138 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.138 INFO:teuthology.orchestra.run.smithi067.stdout: "redirect_resolve_ip_addr": { 2024-03-28T16:09:41.138 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "redirect_resolve_ip_addr", 2024-03-28T16:09:41.138 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.138 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.138 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.138 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.138 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.139 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.139 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.139 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.139 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.139 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.139 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.139 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.139 INFO:teuthology.orchestra.run.smithi067.stdout: "server_addr": { 2024-03-28T16:09:41.139 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "server_addr", 2024-03-28T16:09:41.139 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.139 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.139 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.139 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "::", 2024-03-28T16:09:41.139 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.140 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.140 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.140 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.140 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.140 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.140 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.140 INFO:teuthology.orchestra.run.smithi067.stdout: "server_port": { 2024-03-28T16:09:41.140 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "server_port", 2024-03-28T16:09:41.140 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:41.140 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.140 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.140 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "8080", 2024-03-28T16:09:41.140 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.140 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.141 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.141 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.141 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.141 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.141 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.141 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.141 INFO:teuthology.orchestra.run.smithi067.stdout: "ssl": { 2024-03-28T16:09:41.141 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "ssl", 2024-03-28T16:09:41.141 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.141 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.141 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.141 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:41.141 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.141 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.142 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.142 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.142 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.142 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.142 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.142 INFO:teuthology.orchestra.run.smithi067.stdout: "ssl_server_port": { 2024-03-28T16:09:41.142 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "ssl_server_port", 2024-03-28T16:09:41.142 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:41.142 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.142 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.142 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "8443", 2024-03-28T16:09:41.142 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.142 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.142 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.143 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.143 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.143 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.143 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.143 INFO:teuthology.orchestra.run.smithi067.stdout: "standby_behaviour": { 2024-03-28T16:09:41.143 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "standby_behaviour", 2024-03-28T16:09:41.143 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.143 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.143 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.143 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "redirect", 2024-03-28T16:09:41.143 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.143 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.143 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.143 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:41.144 INFO:teuthology.orchestra.run.smithi067.stdout: "redirect" 2024-03-28T16:09:41.144 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.144 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.144 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.144 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.144 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.144 INFO:teuthology.orchestra.run.smithi067.stdout: "standby_error_status_code": { 2024-03-28T16:09:41.144 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "standby_error_status_code", 2024-03-28T16:09:41.144 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:41.144 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.144 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.144 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "500", 2024-03-28T16:09:41.144 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "400", 2024-03-28T16:09:41.144 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "599", 2024-03-28T16:09:41.145 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.145 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.145 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.145 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.145 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.145 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.145 INFO:teuthology.orchestra.run.smithi067.stdout: "url_prefix": { 2024-03-28T16:09:41.145 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "url_prefix", 2024-03-28T16:09:41.145 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.145 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.145 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.145 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.145 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.145 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.145 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.146 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.146 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.146 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.146 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.146 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.146 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.146 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.146 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:41.146 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "devicehealth", 2024-03-28T16:09:41.146 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:41.146 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:41.146 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:41.146 INFO:teuthology.orchestra.run.smithi067.stdout: "enable_monitoring": { 2024-03-28T16:09:41.146 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "enable_monitoring", 2024-03-28T16:09:41.146 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.147 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.147 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.147 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:41.147 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.147 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.147 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.147 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "monitor device health metrics", 2024-03-28T16:09:41.147 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.147 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.147 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.147 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.147 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:41.147 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:41.147 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.147 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.148 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.148 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.148 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.148 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.148 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.148 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:41.148 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:41.148 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:41.148 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:41.148 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:41.148 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:41.148 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.148 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.148 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.148 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.149 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.149 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.149 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:41.149 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:41.149 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.149 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.149 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.149 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.149 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.149 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.149 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.149 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.149 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.149 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.149 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.150 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.150 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:41.150 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:41.150 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.150 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.150 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.150 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:41.150 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.150 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.150 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.150 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:41.150 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:41.150 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:41.150 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:41.150 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:41.151 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:41.151 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.151 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.151 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.151 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.151 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.151 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:41.151 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:41.151 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.151 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.151 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.151 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.151 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.151 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.152 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.152 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.152 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.152 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.152 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.152 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.152 INFO:teuthology.orchestra.run.smithi067.stdout: "mark_out_threshold": { 2024-03-28T16:09:41.152 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "mark_out_threshold", 2024-03-28T16:09:41.152 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "secs", 2024-03-28T16:09:41.152 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.152 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.152 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "2419200", 2024-03-28T16:09:41.152 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.152 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.153 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.153 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-28T16:09:41.153 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.153 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.153 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.153 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.153 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_name": { 2024-03-28T16:09:41.153 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "pool_name", 2024-03-28T16:09:41.153 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.153 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.153 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.153 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "device_health_metrics", 2024-03-28T16:09:41.153 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.153 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.153 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.154 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-28T16:09:41.154 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.154 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.154 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.154 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.154 INFO:teuthology.orchestra.run.smithi067.stdout: "retention_period": { 2024-03-28T16:09:41.154 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "retention_period", 2024-03-28T16:09:41.154 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "secs", 2024-03-28T16:09:41.154 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.154 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.154 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "15552000", 2024-03-28T16:09:41.154 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.154 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.154 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.154 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "how long to retain device health metrics", 2024-03-28T16:09:41.155 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.155 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.155 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.155 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.155 INFO:teuthology.orchestra.run.smithi067.stdout: "scrape_frequency": { 2024-03-28T16:09:41.155 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "scrape_frequency", 2024-03-28T16:09:41.155 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "secs", 2024-03-28T16:09:41.155 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.155 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.155 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "86400", 2024-03-28T16:09:41.155 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.155 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.155 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.155 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-28T16:09:41.155 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.156 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.156 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.156 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.156 INFO:teuthology.orchestra.run.smithi067.stdout: "self_heal": { 2024-03-28T16:09:41.156 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "self_heal", 2024-03-28T16:09:41.156 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.156 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.156 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.156 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:41.156 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.156 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.156 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.156 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-28T16:09:41.156 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.156 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.157 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.157 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.157 INFO:teuthology.orchestra.run.smithi067.stdout: "sleep_interval": { 2024-03-28T16:09:41.157 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "sleep_interval", 2024-03-28T16:09:41.157 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "secs", 2024-03-28T16:09:41.157 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.157 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.157 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "600", 2024-03-28T16:09:41.157 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.157 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.157 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.157 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "how frequently to wake up and check device health", 2024-03-28T16:09:41.157 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.157 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.157 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.158 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.158 INFO:teuthology.orchestra.run.smithi067.stdout: "warn_threshold": { 2024-03-28T16:09:41.158 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "warn_threshold", 2024-03-28T16:09:41.158 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "secs", 2024-03-28T16:09:41.158 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.158 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.158 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "7257600", 2024-03-28T16:09:41.158 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.158 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.158 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.158 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-28T16:09:41.158 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.158 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.158 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.159 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.159 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.159 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.159 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:41.159 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "diskprediction_local", 2024-03-28T16:09:41.159 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:41.159 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:41.159 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:41.159 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:41.159 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:41.159 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.159 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.159 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.159 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.160 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.160 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.160 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.160 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:41.160 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:41.160 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:41.160 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:41.160 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:41.160 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:41.160 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.160 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.160 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.160 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.160 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.160 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.161 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:41.161 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:41.161 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.161 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.161 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.161 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.161 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.161 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.161 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.161 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.161 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.161 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.161 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.161 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.161 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:41.161 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:41.162 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.162 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.162 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.162 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:41.162 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.162 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.162 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.162 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:41.162 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:41.162 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:41.162 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:41.162 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:41.162 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:41.162 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.163 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.163 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.163 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.163 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.163 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.163 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:41.163 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:41.163 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.163 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.163 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.163 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.163 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.163 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.163 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.163 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.163 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.164 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.164 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.164 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.164 INFO:teuthology.orchestra.run.smithi067.stdout: "predict_interval": { 2024-03-28T16:09:41.164 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "predict_interval", 2024-03-28T16:09:41.164 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.164 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.164 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.164 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "86400", 2024-03-28T16:09:41.164 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.164 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.164 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.164 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.164 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.164 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.165 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.165 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.165 INFO:teuthology.orchestra.run.smithi067.stdout: "predictor_model": { 2024-03-28T16:09:41.165 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "predictor_model", 2024-03-28T16:09:41.165 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.165 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.165 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.165 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "prophetstor", 2024-03-28T16:09:41.165 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.165 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.165 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.165 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.165 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.165 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.166 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.166 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.166 INFO:teuthology.orchestra.run.smithi067.stdout: "sleep_interval": { 2024-03-28T16:09:41.166 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "sleep_interval", 2024-03-28T16:09:41.166 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.166 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.166 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.166 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "600", 2024-03-28T16:09:41.166 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.166 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.166 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.166 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.166 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.166 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.166 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.167 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.167 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.167 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.167 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:41.167 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "influx", 2024-03-28T16:09:41.167 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": false, 2024-03-28T16:09:41.167 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "influxdb python module not found", 2024-03-28T16:09:41.167 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:41.167 INFO:teuthology.orchestra.run.smithi067.stdout: "batch_size": { 2024-03-28T16:09:41.167 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "batch_size", 2024-03-28T16:09:41.167 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:41.167 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.167 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.167 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "5000", 2024-03-28T16:09:41.168 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.168 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.168 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.168 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-28T16:09:41.168 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.168 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.168 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.168 INFO:teuthology.orchestra.run.smithi067.stdout: "database": { 2024-03-28T16:09:41.168 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "database", 2024-03-28T16:09:41.168 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.168 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.168 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.168 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "ceph", 2024-03-28T16:09:41.169 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.169 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.169 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.169 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-28T16:09:41.169 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.169 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.169 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.169 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.169 INFO:teuthology.orchestra.run.smithi067.stdout: "hostname": { 2024-03-28T16:09:41.169 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "hostname", 2024-03-28T16:09:41.169 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.169 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.169 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.169 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.169 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.170 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.170 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.170 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "InfluxDB server hostname", 2024-03-28T16:09:41.170 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.170 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.170 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.170 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.170 INFO:teuthology.orchestra.run.smithi067.stdout: "interval": { 2024-03-28T16:09:41.170 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "interval", 2024-03-28T16:09:41.170 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "secs", 2024-03-28T16:09:41.170 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.170 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.170 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "30", 2024-03-28T16:09:41.170 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "5", 2024-03-28T16:09:41.171 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.171 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.171 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-28T16:09:41.171 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.171 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.171 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.171 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.171 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:41.171 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:41.171 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.171 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.171 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.171 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.171 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.172 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.172 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.172 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:41.172 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:41.172 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:41.172 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:41.172 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:41.172 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:41.172 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.172 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.172 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.172 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.172 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.173 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:41.173 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:41.173 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.173 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.173 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.174 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.174 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.174 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.174 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.174 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.174 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.174 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.174 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.174 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.174 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:41.174 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:41.174 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.174 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.174 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.174 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:41.175 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.175 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.175 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.175 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:41.175 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:41.175 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:41.175 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:41.175 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:41.175 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:41.175 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.175 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.175 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.175 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.175 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.176 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.176 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:41.176 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:41.176 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.176 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.176 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.176 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.176 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.176 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.176 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.176 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.176 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.176 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.176 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.176 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.177 INFO:teuthology.orchestra.run.smithi067.stdout: "password": { 2024-03-28T16:09:41.177 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "password", 2024-03-28T16:09:41.177 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.177 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.177 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.177 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.177 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.177 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.177 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.177 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "password of InfluxDB server user", 2024-03-28T16:09:41.177 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.177 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.177 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.177 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.177 INFO:teuthology.orchestra.run.smithi067.stdout: "port": { 2024-03-28T16:09:41.178 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "port", 2024-03-28T16:09:41.178 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:41.178 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.178 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.178 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "8086", 2024-03-28T16:09:41.178 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.178 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.178 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.178 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "InfluxDB server port", 2024-03-28T16:09:41.178 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.178 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.178 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.178 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.178 INFO:teuthology.orchestra.run.smithi067.stdout: "ssl": { 2024-03-28T16:09:41.179 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "ssl", 2024-03-28T16:09:41.179 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.179 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.179 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.179 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "false", 2024-03-28T16:09:41.179 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.179 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.179 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.179 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-28T16:09:41.179 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.179 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.179 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.179 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.179 INFO:teuthology.orchestra.run.smithi067.stdout: "threads": { 2024-03-28T16:09:41.179 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "threads", 2024-03-28T16:09:41.180 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:41.180 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.180 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.180 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "5", 2024-03-28T16:09:41.180 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "1", 2024-03-28T16:09:41.180 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "32", 2024-03-28T16:09:41.180 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.180 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-28T16:09:41.180 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.180 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.180 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.180 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.180 INFO:teuthology.orchestra.run.smithi067.stdout: "username": { 2024-03-28T16:09:41.180 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "username", 2024-03-28T16:09:41.180 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.181 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.181 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.181 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.181 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.181 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.181 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.181 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "username of InfluxDB server user", 2024-03-28T16:09:41.181 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.181 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.181 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.181 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.181 INFO:teuthology.orchestra.run.smithi067.stdout: "verify_ssl": { 2024-03-28T16:09:41.181 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "verify_ssl", 2024-03-28T16:09:41.181 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.182 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.182 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.182 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "true", 2024-03-28T16:09:41.182 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.182 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.182 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.182 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-28T16:09:41.182 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.182 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.182 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.182 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.182 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.182 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.182 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:41.182 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "insights", 2024-03-28T16:09:41.183 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:41.183 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:41.183 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:41.183 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:41.183 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:41.183 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.183 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.183 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.183 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.183 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.183 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.183 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.183 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:41.183 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:41.183 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:41.184 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:41.184 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:41.184 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:41.184 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.184 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.184 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.184 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.184 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.184 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.184 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:41.184 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:41.184 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.184 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.184 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.184 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.185 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.185 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.185 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.185 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.185 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.185 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.185 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.185 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.185 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:41.185 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:41.185 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.185 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.185 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.185 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:41.186 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.186 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.186 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.186 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:41.186 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:41.186 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:41.186 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:41.186 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:41.186 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:41.186 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.186 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.186 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.186 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.186 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.187 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.187 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:41.187 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:41.187 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.187 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.187 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.187 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.187 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.187 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.187 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.187 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.187 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.187 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.187 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.187 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.188 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.188 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.188 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:41.188 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "iostat", 2024-03-28T16:09:41.188 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:41.188 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:41.188 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:41.188 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:41.188 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:41.188 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.188 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.188 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.188 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.188 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.188 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.189 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.189 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:41.189 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:41.189 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:41.189 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:41.189 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:41.189 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:41.189 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.189 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.189 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.189 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.189 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.189 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.189 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:41.189 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:41.190 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.190 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.190 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.190 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.190 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.190 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.190 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.190 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.190 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.190 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.190 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.190 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.190 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:41.190 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:41.190 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.191 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.191 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.191 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:41.191 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.191 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.191 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.191 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:41.191 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:41.191 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:41.191 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:41.191 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:41.191 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:41.191 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.191 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.191 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.192 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.192 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.192 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.192 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:41.192 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:41.192 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.192 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.192 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.192 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.192 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.192 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.192 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.192 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.192 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.192 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.193 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.193 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.193 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.193 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.193 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:41.193 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "k8sevents", 2024-03-28T16:09:41.193 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:41.193 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:41.193 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:41.193 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph_event_retention_days": { 2024-03-28T16:09:41.193 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "ceph_event_retention_days", 2024-03-28T16:09:41.193 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:41.193 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.193 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.193 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "7", 2024-03-28T16:09:41.194 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.194 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.194 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.194 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-28T16:09:41.194 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.194 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.194 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.194 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.194 INFO:teuthology.orchestra.run.smithi067.stdout: "config_check_secs": { 2024-03-28T16:09:41.194 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "config_check_secs", 2024-03-28T16:09:41.194 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:41.194 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.194 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.194 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "10", 2024-03-28T16:09:41.195 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "10", 2024-03-28T16:09:41.195 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.195 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.195 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-28T16:09:41.195 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.195 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.195 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.195 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.195 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:41.195 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:41.195 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.195 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.195 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.195 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.195 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.196 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.196 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.196 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:41.196 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:41.196 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:41.196 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:41.196 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:41.196 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:41.196 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.196 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.196 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.196 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.196 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.196 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.196 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:41.196 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:41.197 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.197 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.197 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.197 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.197 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.197 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.197 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.197 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.197 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.197 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.197 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.197 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.197 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:41.197 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:41.198 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.198 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.198 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.198 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:41.198 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.198 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.198 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.198 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:41.198 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:41.198 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:41.198 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:41.198 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:41.198 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:41.198 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.199 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.199 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.199 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.199 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.199 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.199 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:41.199 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:41.199 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.199 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.199 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.199 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.199 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.199 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.199 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.199 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.200 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.200 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.200 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.200 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.200 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.200 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.200 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:41.200 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "localpool", 2024-03-28T16:09:41.200 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:41.200 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:41.200 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:41.200 INFO:teuthology.orchestra.run.smithi067.stdout: "failure_domain": { 2024-03-28T16:09:41.200 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "failure_domain", 2024-03-28T16:09:41.200 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.201 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.201 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.201 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "host", 2024-03-28T16:09:41.201 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.201 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.201 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.201 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "failure domain for any created local pool", 2024-03-28T16:09:41.201 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-28T16:09:41.201 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.201 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.201 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.201 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:41.201 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:41.201 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.202 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.202 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.202 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.202 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.202 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.202 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.202 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:41.202 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:41.202 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:41.202 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:41.202 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:41.202 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:41.202 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.202 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.202 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.203 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.203 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.203 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.203 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:41.203 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:41.203 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.203 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.203 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.203 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.203 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.203 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.203 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.203 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.203 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.203 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.204 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.204 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.204 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:41.204 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:41.204 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.204 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.204 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.204 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:41.204 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.204 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.204 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.204 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:41.204 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:41.204 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:41.204 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:41.205 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:41.205 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:41.205 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.205 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.205 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.205 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.205 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.205 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.205 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:41.205 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:41.205 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.205 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.205 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.205 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.206 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.206 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.206 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.206 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.206 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.206 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.206 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.206 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.206 INFO:teuthology.orchestra.run.smithi067.stdout: "min_size": { 2024-03-28T16:09:41.206 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "min_size", 2024-03-28T16:09:41.206 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:41.206 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.206 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.206 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.206 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.207 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.207 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.207 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "default min_size for any created local pool", 2024-03-28T16:09:41.207 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-28T16:09:41.207 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.207 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.207 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.207 INFO:teuthology.orchestra.run.smithi067.stdout: "num_rep": { 2024-03-28T16:09:41.207 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "num_rep", 2024-03-28T16:09:41.207 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:41.207 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.207 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.207 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "3", 2024-03-28T16:09:41.207 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.208 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.208 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.208 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "default replica count for any created local pool", 2024-03-28T16:09:41.208 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.208 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.208 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.208 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.208 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num": { 2024-03-28T16:09:41.208 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "pg_num", 2024-03-28T16:09:41.208 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:41.208 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.208 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.208 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "128", 2024-03-28T16:09:41.208 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.208 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.209 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.209 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "default pg_num for any created local pool", 2024-03-28T16:09:41.209 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.209 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.209 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.209 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.209 INFO:teuthology.orchestra.run.smithi067.stdout: "prefix": { 2024-03-28T16:09:41.209 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "prefix", 2024-03-28T16:09:41.209 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.209 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.209 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.209 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.209 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.209 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.209 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.210 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "name prefix for any created local pool", 2024-03-28T16:09:41.210 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.210 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.210 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.210 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.210 INFO:teuthology.orchestra.run.smithi067.stdout: "subtree": { 2024-03-28T16:09:41.210 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "subtree", 2024-03-28T16:09:41.210 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.210 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.210 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.210 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "rack", 2024-03-28T16:09:41.210 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.210 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.210 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.211 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-28T16:09:41.211 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-28T16:09:41.211 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.211 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.211 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.211 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.211 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.211 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:41.211 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "mds_autoscaler", 2024-03-28T16:09:41.211 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:41.211 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:41.211 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:41.211 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:41.211 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:41.211 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.212 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.212 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.212 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.212 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.212 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.212 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.212 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:41.212 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:41.212 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:41.212 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:41.212 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:41.212 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:41.212 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.212 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.212 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.213 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.213 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.213 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.213 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:41.213 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:41.213 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.213 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.213 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.213 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.213 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.213 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.213 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.213 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.213 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.214 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.214 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.214 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.214 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:41.214 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:41.214 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.214 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.214 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.214 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:41.214 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.214 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.214 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.214 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:41.214 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:41.215 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:41.215 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:41.215 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:41.215 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:41.215 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.215 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.215 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.215 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.215 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.215 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.215 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:41.215 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:41.215 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.215 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.216 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.216 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.216 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.216 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.216 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.216 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.216 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.216 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.216 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.216 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.216 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.216 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.216 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:41.216 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "mirroring", 2024-03-28T16:09:41.216 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:41.217 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:41.217 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:41.217 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:41.217 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:41.217 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.217 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.217 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.217 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.217 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.217 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.217 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.217 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:41.217 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:41.217 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:41.218 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:41.218 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:41.218 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:41.218 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.218 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.218 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.218 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.218 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.218 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.218 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:41.218 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:41.218 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.218 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.218 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.218 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.219 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.219 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.219 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.219 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.219 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.219 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.219 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.219 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.219 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:41.219 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:41.219 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.219 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.219 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.219 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:41.220 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.220 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.220 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.220 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:41.220 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:41.220 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:41.220 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:41.220 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:41.220 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:41.220 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.220 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.220 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.220 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.220 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.220 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.221 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:41.221 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:41.221 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.221 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.221 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.221 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.221 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.221 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.221 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.221 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.221 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.221 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.221 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.221 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.221 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.221 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.222 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:41.222 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "nfs", 2024-03-28T16:09:41.222 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:41.222 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:41.222 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:41.222 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:41.222 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:41.222 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.222 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.222 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.222 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.222 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.222 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.222 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.222 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:41.223 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:41.223 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:41.223 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:41.223 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:41.223 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:41.223 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.223 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.223 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.223 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.223 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.223 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.223 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:41.223 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:41.223 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.223 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.224 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.224 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.224 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.224 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.224 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.224 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.224 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.224 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.224 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.224 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.224 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:41.224 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:41.224 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.224 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.224 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.225 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:41.225 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.225 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.225 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.225 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:41.225 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:41.225 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:41.225 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:41.225 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:41.225 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:41.225 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.225 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.225 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.225 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.225 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.226 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.226 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:41.226 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:41.226 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.226 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.226 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.226 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.226 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.226 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.226 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.226 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.226 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.226 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.226 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.226 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.227 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.227 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.227 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:41.227 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "orchestrator", 2024-03-28T16:09:41.227 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:41.227 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:41.227 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:41.227 INFO:teuthology.orchestra.run.smithi067.stdout: "fail_fs": { 2024-03-28T16:09:41.227 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "fail_fs", 2024-03-28T16:09:41.227 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.227 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.227 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.227 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.227 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.227 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.228 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.228 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-28T16:09:41.228 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.228 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.228 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.228 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.228 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:41.228 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:41.228 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.228 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.228 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.228 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.228 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.228 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.228 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.228 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:41.229 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:41.229 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:41.229 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:41.229 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:41.229 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:41.229 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.229 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.229 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.229 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.229 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.229 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.229 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:41.229 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:41.229 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.229 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.230 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.230 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.230 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.230 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.230 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.230 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.230 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.230 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.230 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.230 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.230 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:41.230 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:41.230 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.230 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.231 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.231 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:41.231 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.231 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.231 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.231 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:41.231 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:41.231 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:41.231 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:41.231 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:41.231 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:41.231 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.231 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.231 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.231 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.232 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.232 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.232 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:41.232 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:41.232 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.232 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.232 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.232 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.232 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.232 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.232 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.232 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.232 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.232 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.232 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.233 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.233 INFO:teuthology.orchestra.run.smithi067.stdout: "orchestrator": { 2024-03-28T16:09:41.233 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "orchestrator", 2024-03-28T16:09:41.233 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.233 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.233 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.233 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.233 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.233 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.233 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.233 INFO:teuthology.orchestra.run.smithi067.stdout: "cephadm", 2024-03-28T16:09:41.233 INFO:teuthology.orchestra.run.smithi067.stdout: "rook", 2024-03-28T16:09:41.233 INFO:teuthology.orchestra.run.smithi067.stdout: "test_orchestrator" 2024-03-28T16:09:41.233 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.233 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Orchestrator backend", 2024-03-28T16:09:41.234 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.234 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.234 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.234 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.234 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.234 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.234 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:41.234 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "osd_perf_query", 2024-03-28T16:09:41.234 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:41.234 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:41.234 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:41.234 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:41.234 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:41.235 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.235 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.235 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.235 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.235 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.235 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.235 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.235 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:41.235 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:41.235 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:41.235 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:41.235 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:41.235 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:41.235 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.235 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.236 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.236 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.236 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.236 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.236 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:41.236 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:41.236 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.236 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.236 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.236 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.236 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.236 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.236 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.236 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.237 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.237 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.237 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.237 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.237 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:41.237 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:41.237 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.237 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.237 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.237 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:41.237 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.237 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.237 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.237 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:41.237 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:41.238 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:41.238 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:41.238 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:41.238 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:41.238 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.238 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.238 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.238 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.238 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.238 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.238 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:41.238 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:41.238 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.239 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.239 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.239 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.239 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.239 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.239 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.239 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.239 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.239 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.239 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.239 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.239 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.239 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.239 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:41.239 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "osd_support", 2024-03-28T16:09:41.240 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:41.240 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:41.240 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:41.240 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:41.240 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:41.240 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.240 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.240 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.240 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.240 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.240 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.240 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.240 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:41.240 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:41.240 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:41.241 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:41.241 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:41.241 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:41.241 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.241 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.241 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.241 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.241 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.241 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.241 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:41.241 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:41.241 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.241 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.241 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.242 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.242 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.242 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.242 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.242 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.242 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.242 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.242 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.242 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.242 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:41.242 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:41.242 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.242 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.242 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.242 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:41.243 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.243 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.243 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.243 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:41.243 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:41.243 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:41.243 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:41.243 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:41.243 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:41.243 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.243 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.243 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.243 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.243 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.244 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.244 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:41.244 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:41.244 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.244 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.244 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.244 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.244 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.244 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.244 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.244 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.244 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.244 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.244 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.244 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.245 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.245 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.245 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:41.245 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "pg_autoscaler", 2024-03-28T16:09:41.245 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:41.245 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:41.245 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:41.245 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:41.245 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:41.245 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.245 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.245 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.245 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.245 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.245 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.246 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.246 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:41.246 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:41.246 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:41.246 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:41.246 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:41.246 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:41.246 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.246 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.246 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.246 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.246 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.246 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.246 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:41.247 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:41.247 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.247 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.247 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.247 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.247 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.247 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.247 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.247 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.247 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.247 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.247 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.247 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.247 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:41.248 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:41.248 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.248 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.248 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.248 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:41.248 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.248 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.248 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.248 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:41.248 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:41.248 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:41.248 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:41.248 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:41.248 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:41.248 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.249 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.249 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.249 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.249 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.249 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.249 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:41.249 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:41.249 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.249 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.249 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.249 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.249 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.249 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.249 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.249 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.250 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.250 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.250 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.250 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.250 INFO:teuthology.orchestra.run.smithi067.stdout: "sleep_interval": { 2024-03-28T16:09:41.250 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "sleep_interval", 2024-03-28T16:09:41.250 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "secs", 2024-03-28T16:09:41.250 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.250 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.250 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "60", 2024-03-28T16:09:41.250 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.250 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.250 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.250 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.250 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.251 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.251 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.251 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.251 INFO:teuthology.orchestra.run.smithi067.stdout: "threshold": { 2024-03-28T16:09:41.251 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "threshold", 2024-03-28T16:09:41.251 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "float", 2024-03-28T16:09:41.251 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.251 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.251 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "3.0", 2024-03-28T16:09:41.251 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "1.0", 2024-03-28T16:09:41.251 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.251 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.251 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "scaling threshold", 2024-03-28T16:09:41.251 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-28T16:09:41.252 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.252 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.252 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.252 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.252 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.252 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:41.252 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "progress", 2024-03-28T16:09:41.252 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:41.252 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:41.252 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:41.252 INFO:teuthology.orchestra.run.smithi067.stdout: "allow_pg_recovery_event": { 2024-03-28T16:09:41.252 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "allow_pg_recovery_event", 2024-03-28T16:09:41.252 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.252 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.252 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.252 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.253 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.253 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.253 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.253 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-28T16:09:41.253 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.253 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.253 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.253 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.253 INFO:teuthology.orchestra.run.smithi067.stdout: "enabled": { 2024-03-28T16:09:41.253 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "enabled", 2024-03-28T16:09:41.253 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.253 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.253 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.253 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:41.254 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.254 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.254 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.254 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.254 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.254 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.254 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.254 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.254 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:41.254 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:41.254 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.254 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.254 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.254 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.254 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.255 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.255 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.255 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:41.255 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:41.255 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:41.255 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:41.255 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:41.255 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:41.255 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.255 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.255 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.255 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.255 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.255 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.256 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:41.256 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:41.256 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.256 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.256 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.256 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.256 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.256 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.256 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.256 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.256 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.256 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.256 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.256 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.256 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:41.257 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:41.257 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.257 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.257 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.257 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:41.257 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.257 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.257 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.257 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:41.257 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:41.257 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:41.257 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:41.257 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:41.257 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:41.257 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.258 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.258 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.258 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.258 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.258 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.258 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:41.258 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:41.258 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.258 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.258 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.258 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.258 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.258 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.258 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.258 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.259 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.259 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.259 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.259 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.259 INFO:teuthology.orchestra.run.smithi067.stdout: "max_completed_events": { 2024-03-28T16:09:41.259 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "max_completed_events", 2024-03-28T16:09:41.259 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:41.259 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.259 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.259 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "50", 2024-03-28T16:09:41.259 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.259 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.259 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.259 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "number of past completed events to remember", 2024-03-28T16:09:41.259 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.260 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.260 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.260 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.260 INFO:teuthology.orchestra.run.smithi067.stdout: "sleep_interval": { 2024-03-28T16:09:41.260 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "sleep_interval", 2024-03-28T16:09:41.260 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "secs", 2024-03-28T16:09:41.260 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.260 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.260 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "5", 2024-03-28T16:09:41.260 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.260 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.260 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.260 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "how long the module is going to sleep", 2024-03-28T16:09:41.260 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.260 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.260 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.261 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.261 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.261 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.261 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:41.261 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "prometheus", 2024-03-28T16:09:41.261 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:41.261 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:41.261 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:41.261 INFO:teuthology.orchestra.run.smithi067.stdout: "cache": { 2024-03-28T16:09:41.261 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "cache", 2024-03-28T16:09:41.261 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.261 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.261 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.261 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:41.261 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.262 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.262 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.262 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.262 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.262 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.262 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.262 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.262 INFO:teuthology.orchestra.run.smithi067.stdout: "exclude_perf_counters": { 2024-03-28T16:09:41.262 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "exclude_perf_counters", 2024-03-28T16:09:41.262 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.262 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.262 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.262 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:41.262 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.262 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.263 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.263 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-28T16:09:41.263 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-28T16:09:41.263 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.263 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.263 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.263 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:41.263 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:41.263 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.263 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.263 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.263 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.263 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.263 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.264 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.264 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:41.264 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:41.264 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:41.264 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:41.264 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:41.264 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:41.264 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.264 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.264 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.264 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.264 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.264 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.264 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:41.264 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:41.264 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.265 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.265 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.265 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.265 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.265 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.265 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.265 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.265 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.265 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.265 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.265 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.265 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:41.265 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:41.265 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.266 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.266 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.266 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:41.266 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.266 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.266 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.266 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:41.266 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:41.266 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:41.266 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:41.266 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:41.266 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:41.266 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.266 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.267 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.267 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.267 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.267 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.267 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:41.267 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:41.267 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.267 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.267 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.267 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.267 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.267 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.267 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.267 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.268 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.268 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.268 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.268 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.268 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_stats_pools": { 2024-03-28T16:09:41.268 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "rbd_stats_pools", 2024-03-28T16:09:41.268 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.268 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.268 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.268 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.268 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.268 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.268 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.268 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.269 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.269 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.269 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.269 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.269 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-28T16:09:41.269 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-28T16:09:41.269 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:41.269 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.269 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.269 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "300", 2024-03-28T16:09:41.269 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.269 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.269 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.269 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.269 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.270 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.270 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.270 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.270 INFO:teuthology.orchestra.run.smithi067.stdout: "scrape_interval": { 2024-03-28T16:09:41.270 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "scrape_interval", 2024-03-28T16:09:41.270 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "float", 2024-03-28T16:09:41.270 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.270 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.270 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "15.0", 2024-03-28T16:09:41.270 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.270 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.270 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.270 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.270 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.270 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.271 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.271 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.271 INFO:teuthology.orchestra.run.smithi067.stdout: "server_addr": { 2024-03-28T16:09:41.271 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "server_addr", 2024-03-28T16:09:41.271 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.271 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.271 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.271 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "::", 2024-03-28T16:09:41.271 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.271 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.271 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.271 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-28T16:09:41.271 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.271 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.271 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.272 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.272 INFO:teuthology.orchestra.run.smithi067.stdout: "server_port": { 2024-03-28T16:09:41.272 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "server_port", 2024-03-28T16:09:41.272 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:41.272 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.272 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.272 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "9283", 2024-03-28T16:09:41.272 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.272 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.272 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.272 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-28T16:09:41.272 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.272 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.272 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.272 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.273 INFO:teuthology.orchestra.run.smithi067.stdout: "stale_cache_strategy": { 2024-03-28T16:09:41.273 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "stale_cache_strategy", 2024-03-28T16:09:41.273 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.273 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.273 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.273 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "log", 2024-03-28T16:09:41.273 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.273 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.273 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.273 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.273 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.273 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.273 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.273 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.273 INFO:teuthology.orchestra.run.smithi067.stdout: "standby_behaviour": { 2024-03-28T16:09:41.274 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "standby_behaviour", 2024-03-28T16:09:41.274 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.274 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.274 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.274 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "default", 2024-03-28T16:09:41.274 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.274 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.274 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.274 INFO:teuthology.orchestra.run.smithi067.stdout: "default", 2024-03-28T16:09:41.274 INFO:teuthology.orchestra.run.smithi067.stdout: "error" 2024-03-28T16:09:41.274 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.274 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.274 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.274 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.274 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.275 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.275 INFO:teuthology.orchestra.run.smithi067.stdout: "standby_error_status_code": { 2024-03-28T16:09:41.275 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "standby_error_status_code", 2024-03-28T16:09:41.275 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:41.275 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.275 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.275 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "500", 2024-03-28T16:09:41.275 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "400", 2024-03-28T16:09:41.275 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "599", 2024-03-28T16:09:41.275 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.275 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.275 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.275 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.275 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.275 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.276 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.276 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.276 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:41.276 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "rbd_support", 2024-03-28T16:09:41.276 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:41.276 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:41.276 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:41.276 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:41.276 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:41.276 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.276 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.276 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.276 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.276 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.276 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.277 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.277 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:41.277 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:41.277 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:41.277 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:41.277 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:41.277 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:41.277 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.277 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.277 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.277 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.277 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.277 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.277 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:41.278 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:41.278 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.278 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.278 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.278 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.278 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.278 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.278 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.278 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.278 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.278 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.278 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.278 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.278 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:41.279 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:41.279 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.279 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.279 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.279 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:41.279 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.279 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.279 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.279 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:41.279 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:41.279 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:41.279 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:41.279 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:41.280 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:41.280 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.280 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.280 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.280 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.280 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.280 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.280 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:41.280 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:41.280 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.280 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.280 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.280 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.280 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.280 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.281 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.281 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.281 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.281 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.281 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.281 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.281 INFO:teuthology.orchestra.run.smithi067.stdout: "max_concurrent_snap_create": { 2024-03-28T16:09:41.281 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "max_concurrent_snap_create", 2024-03-28T16:09:41.281 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:41.281 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.281 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.281 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "10", 2024-03-28T16:09:41.281 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.281 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.281 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.282 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.282 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.282 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.282 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.282 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.282 INFO:teuthology.orchestra.run.smithi067.stdout: "mirror_snapshot_schedule": { 2024-03-28T16:09:41.282 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "mirror_snapshot_schedule", 2024-03-28T16:09:41.282 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.282 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.282 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.282 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.282 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.282 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.282 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.283 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.283 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.283 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.283 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.283 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.283 INFO:teuthology.orchestra.run.smithi067.stdout: "trash_purge_schedule": { 2024-03-28T16:09:41.283 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "trash_purge_schedule", 2024-03-28T16:09:41.283 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.283 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.283 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.283 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.283 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.283 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.283 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.283 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.284 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.284 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.284 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.284 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.284 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.284 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.284 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:41.284 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "restful", 2024-03-28T16:09:41.284 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:41.284 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:41.284 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:41.284 INFO:teuthology.orchestra.run.smithi067.stdout: "enable_auth": { 2024-03-28T16:09:41.284 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "enable_auth", 2024-03-28T16:09:41.284 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.284 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.285 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.285 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:41.285 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.285 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.285 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.285 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.285 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.285 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.285 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.285 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.285 INFO:teuthology.orchestra.run.smithi067.stdout: "key_file": { 2024-03-28T16:09:41.285 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "key_file", 2024-03-28T16:09:41.285 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.286 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.286 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.286 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.286 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.286 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.286 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.286 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.286 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.286 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.286 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.286 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.286 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:41.286 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:41.286 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.286 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.287 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.287 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.287 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.287 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.287 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.287 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:41.287 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:41.287 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:41.287 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:41.287 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:41.287 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:41.287 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.287 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.287 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.287 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.288 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.288 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.288 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:41.288 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:41.288 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.288 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.288 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.288 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.288 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.288 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.288 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.288 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.288 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.288 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.288 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.289 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.289 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:41.289 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:41.289 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.289 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.289 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.289 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:41.289 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.289 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.289 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.289 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:41.289 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:41.289 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:41.289 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:41.289 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:41.289 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:41.290 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.290 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.290 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.290 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.290 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.290 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.290 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:41.290 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:41.290 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.290 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.290 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.290 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.290 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.290 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.290 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.291 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.291 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.291 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.291 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.291 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.291 INFO:teuthology.orchestra.run.smithi067.stdout: "server_addr": { 2024-03-28T16:09:41.291 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "server_addr", 2024-03-28T16:09:41.291 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.291 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.291 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.291 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.291 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.291 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.291 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.291 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.292 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.292 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.292 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.292 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.292 INFO:teuthology.orchestra.run.smithi067.stdout: "server_port": { 2024-03-28T16:09:41.292 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "server_port", 2024-03-28T16:09:41.292 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.292 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.292 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.292 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.292 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.292 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.292 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.292 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.292 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.293 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.293 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.293 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.293 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.293 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.293 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:41.293 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "rgw", 2024-03-28T16:09:41.293 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:41.293 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:41.293 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:41.293 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:41.293 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:41.293 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.293 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.293 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.294 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.294 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.294 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.294 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.294 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:41.294 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:41.294 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:41.294 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:41.294 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:41.294 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:41.294 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.294 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.294 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.294 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.294 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.295 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.295 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:41.295 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:41.295 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.295 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.295 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.295 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.295 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.295 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.295 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.295 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.295 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.295 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.295 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.295 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.296 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:41.296 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:41.296 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.296 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.296 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.296 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:41.296 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.296 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.296 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.296 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:41.296 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:41.296 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:41.296 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:41.296 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:41.296 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:41.297 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.297 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.297 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.297 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.297 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.297 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.297 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:41.297 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:41.297 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.297 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.297 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.297 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.297 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.297 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.298 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.298 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.298 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.298 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.298 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.298 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.298 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.298 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.298 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:41.298 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "rook", 2024-03-28T16:09:41.298 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:41.298 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:41.298 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:41.298 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:41.298 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:41.299 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.299 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.299 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.299 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.299 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.299 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.299 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.299 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:41.299 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:41.299 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:41.299 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:41.299 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:41.299 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:41.299 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.300 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.300 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.300 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.300 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.300 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.300 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:41.300 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:41.300 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.300 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.300 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.300 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.300 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.300 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.300 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.300 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.301 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.301 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.301 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.301 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.301 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:41.301 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:41.301 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.301 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.301 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.301 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:41.301 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.301 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.301 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.301 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:41.301 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:41.302 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:41.302 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:41.302 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:41.302 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:41.302 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.302 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.302 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.302 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.302 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.302 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.302 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:41.302 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:41.302 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.302 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.303 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.303 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.303 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.303 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.303 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.303 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.303 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.303 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.303 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.303 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.303 INFO:teuthology.orchestra.run.smithi067.stdout: "storage_class": { 2024-03-28T16:09:41.303 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "storage_class", 2024-03-28T16:09:41.303 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.303 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.303 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.304 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "local", 2024-03-28T16:09:41.304 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.304 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.304 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.304 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-28T16:09:41.304 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.304 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.304 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.304 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.304 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.304 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.304 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:41.304 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "selftest", 2024-03-28T16:09:41.304 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:41.304 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:41.304 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:41.305 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:41.305 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:41.305 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.305 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.305 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.305 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.305 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.305 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.305 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.305 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:41.305 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:41.305 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:41.305 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:41.305 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:41.305 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:41.305 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.306 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.306 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.306 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.306 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.306 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.306 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:41.306 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:41.306 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.306 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.306 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.306 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.306 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.306 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.306 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.307 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.307 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.307 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.307 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.307 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.307 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:41.307 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:41.307 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.307 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.307 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.307 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:41.307 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.307 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.307 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.307 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:41.308 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:41.308 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:41.308 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:41.308 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:41.308 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:41.308 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.308 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.308 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.308 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.308 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.308 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.308 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:41.308 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:41.308 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.308 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.309 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.309 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.309 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.309 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.309 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.309 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.309 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.309 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.309 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.309 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.309 INFO:teuthology.orchestra.run.smithi067.stdout: "roption1": { 2024-03-28T16:09:41.309 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "roption1", 2024-03-28T16:09:41.309 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.309 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.309 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.309 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.310 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.310 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.310 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.310 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.310 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.310 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.310 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.310 INFO:teuthology.orchestra.run.smithi067.stdout: "roption2": { 2024-03-28T16:09:41.310 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "roption2", 2024-03-28T16:09:41.310 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.310 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.310 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.310 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "xyz", 2024-03-28T16:09:41.311 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.311 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.311 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.311 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.311 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.311 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.311 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.311 INFO:teuthology.orchestra.run.smithi067.stdout: "rwoption1": { 2024-03-28T16:09:41.311 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "rwoption1", 2024-03-28T16:09:41.311 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.311 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.311 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.311 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.311 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.312 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.312 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.312 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.312 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.312 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.312 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.312 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.312 INFO:teuthology.orchestra.run.smithi067.stdout: "rwoption2": { 2024-03-28T16:09:41.312 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "rwoption2", 2024-03-28T16:09:41.312 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:41.312 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.312 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.312 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.312 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.312 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.313 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.313 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.313 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.313 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.313 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.313 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.313 INFO:teuthology.orchestra.run.smithi067.stdout: "rwoption3": { 2024-03-28T16:09:41.313 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "rwoption3", 2024-03-28T16:09:41.313 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "float", 2024-03-28T16:09:41.313 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.313 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.313 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.313 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.313 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.313 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.314 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.314 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.314 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.314 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.314 INFO:teuthology.orchestra.run.smithi067.stdout: "rwoption4": { 2024-03-28T16:09:41.314 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "rwoption4", 2024-03-28T16:09:41.314 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.314 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.314 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.314 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.314 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.314 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.314 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.315 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.315 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.315 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.315 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.315 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.315 INFO:teuthology.orchestra.run.smithi067.stdout: "rwoption5": { 2024-03-28T16:09:41.315 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "rwoption5", 2024-03-28T16:09:41.315 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.315 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.315 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.315 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.315 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.315 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.315 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.315 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.316 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.316 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.316 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.316 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.316 INFO:teuthology.orchestra.run.smithi067.stdout: "rwoption6": { 2024-03-28T16:09:41.316 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "rwoption6", 2024-03-28T16:09:41.316 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.316 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.316 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.316 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:41.316 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.316 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.316 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.316 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.316 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.317 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.317 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.317 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.317 INFO:teuthology.orchestra.run.smithi067.stdout: "rwoption7": { 2024-03-28T16:09:41.317 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "rwoption7", 2024-03-28T16:09:41.317 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:41.317 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.317 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.317 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.317 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "1", 2024-03-28T16:09:41.317 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "42", 2024-03-28T16:09:41.317 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.317 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.317 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.317 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.317 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.318 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.318 INFO:teuthology.orchestra.run.smithi067.stdout: "testkey": { 2024-03-28T16:09:41.318 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "testkey", 2024-03-28T16:09:41.318 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.318 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.318 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.318 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.318 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.318 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.318 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.318 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.318 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.318 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.318 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.319 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.319 INFO:teuthology.orchestra.run.smithi067.stdout: "testlkey": { 2024-03-28T16:09:41.319 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "testlkey", 2024-03-28T16:09:41.319 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.319 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.319 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.319 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.319 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.319 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.319 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.319 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.319 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.319 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.319 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.319 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.320 INFO:teuthology.orchestra.run.smithi067.stdout: "testnewline": { 2024-03-28T16:09:41.320 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "testnewline", 2024-03-28T16:09:41.320 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.320 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.320 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.320 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.320 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.320 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.320 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.320 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.320 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.320 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.320 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.320 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.320 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.320 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.321 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:41.321 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "snap_schedule", 2024-03-28T16:09:41.321 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:41.321 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:41.321 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:41.321 INFO:teuthology.orchestra.run.smithi067.stdout: "allow_m_granularity": { 2024-03-28T16:09:41.321 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "allow_m_granularity", 2024-03-28T16:09:41.321 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.321 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.321 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.321 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.321 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.321 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.321 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.321 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "allow minute scheduled snapshots", 2024-03-28T16:09:41.322 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.322 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.322 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.322 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.322 INFO:teuthology.orchestra.run.smithi067.stdout: "dump_on_update": { 2024-03-28T16:09:41.322 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "dump_on_update", 2024-03-28T16:09:41.322 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.322 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.322 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.322 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.322 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.322 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.322 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.322 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "dump database to debug log on update", 2024-03-28T16:09:41.323 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.323 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.323 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.323 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.323 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:41.323 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:41.323 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.323 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.323 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.323 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.323 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.323 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.323 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.323 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:41.323 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:41.324 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:41.324 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:41.324 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:41.324 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:41.324 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.324 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.324 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.324 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.324 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.324 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.324 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:41.324 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:41.324 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.324 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.324 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.325 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.325 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.325 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.325 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.325 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.325 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.325 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.325 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.325 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.325 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:41.325 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:41.325 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.325 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.325 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.325 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:41.325 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.326 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.326 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.326 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:41.326 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:41.326 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:41.326 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:41.326 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:41.326 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:41.326 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.326 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.326 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.326 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.326 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.326 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.327 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:41.327 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:41.327 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.327 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.327 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.327 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.327 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.327 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.327 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.327 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.327 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.327 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.327 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.327 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.327 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.328 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.328 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:41.328 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "stats", 2024-03-28T16:09:41.328 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:41.328 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:41.328 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:41.328 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:41.328 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:41.328 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.328 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.328 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.328 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.328 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.328 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.329 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.329 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:41.329 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:41.329 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:41.329 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:41.329 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:41.329 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:41.329 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.329 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.329 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.329 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.329 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.329 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.329 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:41.329 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:41.330 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.330 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.330 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.330 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.330 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.330 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.330 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.330 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.330 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.330 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.330 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.330 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.330 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:41.330 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:41.331 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.331 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.331 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.331 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:41.331 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.331 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.331 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.331 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:41.331 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:41.331 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:41.331 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:41.331 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:41.331 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:41.331 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.331 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.332 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.332 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.332 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.332 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.332 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:41.332 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:41.332 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.332 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.332 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.332 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.332 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.332 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.332 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.332 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.332 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.333 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.333 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.333 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.333 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.333 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.333 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:41.333 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "status", 2024-03-28T16:09:41.333 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:41.333 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:41.333 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:41.333 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:41.333 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:41.333 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.333 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.333 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.334 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.334 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.334 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.334 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.334 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:41.334 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:41.334 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:41.334 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:41.334 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:41.334 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:41.334 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.334 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.334 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.334 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.335 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.335 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.335 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:41.335 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:41.335 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.335 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.335 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.335 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.335 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.335 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.335 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.335 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.335 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.335 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.335 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.336 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.336 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:41.336 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:41.336 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.336 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.336 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.336 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:41.336 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.336 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.336 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.336 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:41.336 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:41.336 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:41.336 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:41.336 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:41.336 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:41.337 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.337 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.337 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.337 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.337 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.337 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.337 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:41.337 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:41.337 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.337 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.337 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.337 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.337 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.337 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.337 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.338 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.338 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.338 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.338 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.338 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.338 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.338 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.338 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:41.338 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "telegraf", 2024-03-28T16:09:41.338 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:41.338 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:41.338 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:41.338 INFO:teuthology.orchestra.run.smithi067.stdout: "address": { 2024-03-28T16:09:41.338 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "address", 2024-03-28T16:09:41.339 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.339 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.339 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.339 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-28T16:09:41.339 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.339 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.339 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.339 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.339 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.339 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.339 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.339 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.339 INFO:teuthology.orchestra.run.smithi067.stdout: "interval": { 2024-03-28T16:09:41.339 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "interval", 2024-03-28T16:09:41.339 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "secs", 2024-03-28T16:09:41.340 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.340 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.340 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "15", 2024-03-28T16:09:41.340 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.340 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.340 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.340 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.340 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.340 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.340 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.340 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.340 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:41.340 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:41.340 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.340 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.341 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.341 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.341 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.341 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.341 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.341 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:41.341 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:41.341 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:41.341 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:41.341 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:41.341 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:41.341 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.341 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.341 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.341 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.341 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.342 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.342 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:41.342 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:41.342 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.342 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.342 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.342 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.342 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.342 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.342 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.342 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.342 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.342 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.342 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.343 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.343 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:41.343 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:41.343 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.343 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.343 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.343 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:41.343 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.343 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.343 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.343 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:41.343 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:41.343 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:41.343 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:41.343 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:41.344 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:41.344 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.344 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.344 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.344 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.344 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.344 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.344 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:41.344 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:41.344 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.344 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.344 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.344 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.344 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.344 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.345 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.345 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.345 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.345 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.345 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.345 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.345 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.345 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.345 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:41.345 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "telemetry", 2024-03-28T16:09:41.345 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:41.345 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:41.345 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:41.345 INFO:teuthology.orchestra.run.smithi067.stdout: "channel_basic": { 2024-03-28T16:09:41.345 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "channel_basic", 2024-03-28T16:09:41.346 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.346 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.346 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.346 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:41.346 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.346 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.346 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.346 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-28T16:09:41.346 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.346 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.346 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.346 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.346 INFO:teuthology.orchestra.run.smithi067.stdout: "channel_crash": { 2024-03-28T16:09:41.346 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "channel_crash", 2024-03-28T16:09:41.347 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.347 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.347 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.347 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:41.347 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.347 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.347 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.347 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-28T16:09:41.347 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.347 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.347 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.347 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.347 INFO:teuthology.orchestra.run.smithi067.stdout: "channel_device": { 2024-03-28T16:09:41.347 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "channel_device", 2024-03-28T16:09:41.348 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.348 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.348 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.348 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:41.348 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.348 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.348 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.348 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-28T16:09:41.348 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.348 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.348 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.348 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.348 INFO:teuthology.orchestra.run.smithi067.stdout: "channel_ident": { 2024-03-28T16:09:41.348 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "channel_ident", 2024-03-28T16:09:41.349 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.349 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.349 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.349 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.349 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.349 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.349 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.349 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-28T16:09:41.349 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.349 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.349 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.349 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.349 INFO:teuthology.orchestra.run.smithi067.stdout: "channel_perf": { 2024-03-28T16:09:41.349 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "channel_perf", 2024-03-28T16:09:41.349 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.350 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.350 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.350 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.350 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.350 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.350 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.350 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-28T16:09:41.350 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.350 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.350 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.350 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.350 INFO:teuthology.orchestra.run.smithi067.stdout: "contact": { 2024-03-28T16:09:41.350 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "contact", 2024-03-28T16:09:41.350 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.351 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.351 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.351 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.351 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.351 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.351 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.351 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.351 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.351 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.351 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.351 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.351 INFO:teuthology.orchestra.run.smithi067.stdout: "description": { 2024-03-28T16:09:41.351 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "description", 2024-03-28T16:09:41.351 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.351 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.352 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.352 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.352 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.352 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.352 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.352 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.352 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.352 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.352 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.352 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.352 INFO:teuthology.orchestra.run.smithi067.stdout: "device_url": { 2024-03-28T16:09:41.352 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "device_url", 2024-03-28T16:09:41.352 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.352 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.353 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.353 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-28T16:09:41.353 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.353 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.353 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.353 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.353 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.353 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.353 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.353 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.353 INFO:teuthology.orchestra.run.smithi067.stdout: "enabled": { 2024-03-28T16:09:41.353 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "enabled", 2024-03-28T16:09:41.353 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.353 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.354 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.354 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.354 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.354 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.354 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.354 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.354 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.354 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.354 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.354 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.354 INFO:teuthology.orchestra.run.smithi067.stdout: "interval": { 2024-03-28T16:09:41.354 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "interval", 2024-03-28T16:09:41.354 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:41.354 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.354 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.355 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "24", 2024-03-28T16:09:41.355 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "8", 2024-03-28T16:09:41.355 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.355 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.355 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.355 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.355 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.355 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.355 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.355 INFO:teuthology.orchestra.run.smithi067.stdout: "last_opt_revision": { 2024-03-28T16:09:41.355 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "last_opt_revision", 2024-03-28T16:09:41.355 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:41.355 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.355 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.355 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "1", 2024-03-28T16:09:41.356 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.356 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.356 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.356 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.356 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.356 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.356 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.356 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.356 INFO:teuthology.orchestra.run.smithi067.stdout: "leaderboard": { 2024-03-28T16:09:41.356 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "leaderboard", 2024-03-28T16:09:41.356 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.356 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.356 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.356 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.356 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.357 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.357 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.357 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.357 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.357 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.357 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.357 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.357 INFO:teuthology.orchestra.run.smithi067.stdout: "leaderboard_description": { 2024-03-28T16:09:41.357 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "leaderboard_description", 2024-03-28T16:09:41.357 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.357 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.357 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.357 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.357 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.357 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.358 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.358 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.358 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.358 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.358 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.358 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.358 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:41.358 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:41.358 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.358 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.358 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.358 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.358 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.358 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.359 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.359 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:41.359 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:41.359 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:41.359 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:41.359 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:41.359 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:41.359 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.359 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.359 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.359 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.359 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.359 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.359 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:41.359 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:41.360 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.360 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.360 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.360 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.360 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.360 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.360 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.360 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.360 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.360 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.360 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.360 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.360 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:41.360 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:41.360 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.361 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.361 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.361 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:41.361 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.361 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.361 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.361 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:41.361 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:41.361 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:41.361 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:41.361 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:41.361 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:41.361 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.361 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.362 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.362 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.362 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.362 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.362 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:41.362 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:41.362 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.362 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.362 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.362 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.362 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.362 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.362 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.362 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.363 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.363 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.363 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.363 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.363 INFO:teuthology.orchestra.run.smithi067.stdout: "organization": { 2024-03-28T16:09:41.363 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "organization", 2024-03-28T16:09:41.363 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.363 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.363 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.363 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.363 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.363 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.363 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.363 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.364 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.364 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.364 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.364 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.364 INFO:teuthology.orchestra.run.smithi067.stdout: "proxy": { 2024-03-28T16:09:41.364 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "proxy", 2024-03-28T16:09:41.364 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.364 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.364 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.364 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.364 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.364 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.364 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.364 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.364 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.365 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.365 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.365 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.365 INFO:teuthology.orchestra.run.smithi067.stdout: "url": { 2024-03-28T16:09:41.365 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "url", 2024-03-28T16:09:41.365 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.365 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.365 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.365 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-28T16:09:41.365 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.365 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.365 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.365 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.365 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.365 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.366 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.366 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.366 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.366 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.366 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:41.366 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "test_orchestrator", 2024-03-28T16:09:41.366 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:41.366 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:41.366 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:41.366 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:41.366 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:41.366 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.366 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.366 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.366 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.367 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.367 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.367 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.367 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:41.367 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:41.367 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:41.367 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:41.367 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:41.367 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:41.367 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.367 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.367 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.367 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.367 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.367 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.368 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:41.368 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:41.368 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.368 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.368 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.368 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.368 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.368 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.368 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.368 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.368 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.368 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.368 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.368 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.368 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:41.369 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:41.369 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.369 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.369 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.369 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:41.369 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.369 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.369 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.369 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:41.369 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:41.369 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:41.369 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:41.369 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:41.369 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:41.369 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.370 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.370 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.370 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.370 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.370 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.370 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:41.370 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:41.370 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.370 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.370 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.370 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.370 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.370 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.370 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.370 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.371 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.371 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.371 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.371 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.371 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.371 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.371 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:41.371 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "volumes", 2024-03-28T16:09:41.371 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:41.371 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:41.371 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:41.371 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:41.371 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:41.371 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.371 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.372 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.372 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.372 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.372 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.372 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.372 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:41.372 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:41.372 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:41.372 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:41.372 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:41.372 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:41.372 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.372 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.372 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.372 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.373 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.373 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.373 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:41.373 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:41.373 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.373 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.373 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.373 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.373 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.373 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.373 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.373 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.373 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.373 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.373 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.373 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.374 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:41.374 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:41.374 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.374 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.374 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.374 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:41.374 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.374 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.374 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.374 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:41.374 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:41.374 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:41.374 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:41.374 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:41.375 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:41.375 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.375 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.375 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.375 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.377 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.377 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.377 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:41.377 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:41.377 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.377 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.378 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.378 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.378 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.378 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.378 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.378 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.378 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.378 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.378 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.378 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.378 INFO:teuthology.orchestra.run.smithi067.stdout: "max_concurrent_clones": { 2024-03-28T16:09:41.378 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "max_concurrent_clones", 2024-03-28T16:09:41.378 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:41.378 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.378 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.379 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "4", 2024-03-28T16:09:41.379 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.379 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.379 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.379 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-28T16:09:41.379 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.379 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.379 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.379 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.379 INFO:teuthology.orchestra.run.smithi067.stdout: "periodic_async_work": { 2024-03-28T16:09:41.379 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "periodic_async_work", 2024-03-28T16:09:41.379 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.379 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.379 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.380 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.380 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.380 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.380 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.380 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Periodically check for async work", 2024-03-28T16:09:41.380 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.380 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.380 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.380 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.380 INFO:teuthology.orchestra.run.smithi067.stdout: "snapshot_clone_delay": { 2024-03-28T16:09:41.380 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "snapshot_clone_delay", 2024-03-28T16:09:41.380 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:41.380 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.380 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.380 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "0", 2024-03-28T16:09:41.381 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.381 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.381 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.381 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-28T16:09:41.381 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.381 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.381 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.381 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.381 INFO:teuthology.orchestra.run.smithi067.stdout: "snapshot_clone_no_wait": { 2024-03-28T16:09:41.381 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "snapshot_clone_no_wait", 2024-03-28T16:09:41.381 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.381 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.381 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.381 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "True", 2024-03-28T16:09:41.382 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.382 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.382 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.382 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-28T16:09:41.382 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.382 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.382 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.382 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.382 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.382 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.382 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:41.382 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "zabbix", 2024-03-28T16:09:41.382 INFO:teuthology.orchestra.run.smithi067.stdout: "can_run": true, 2024-03-28T16:09:41.382 INFO:teuthology.orchestra.run.smithi067.stdout: "error_string": "", 2024-03-28T16:09:41.383 INFO:teuthology.orchestra.run.smithi067.stdout: "module_options": { 2024-03-28T16:09:41.383 INFO:teuthology.orchestra.run.smithi067.stdout: "discovery_interval": { 2024-03-28T16:09:41.383 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "discovery_interval", 2024-03-28T16:09:41.383 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "uint", 2024-03-28T16:09:41.383 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.383 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.383 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "100", 2024-03-28T16:09:41.383 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.383 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.383 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.383 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.383 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.383 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.383 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.383 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.384 INFO:teuthology.orchestra.run.smithi067.stdout: "identifier": { 2024-03-28T16:09:41.384 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "identifier", 2024-03-28T16:09:41.384 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.384 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.384 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.384 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.384 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.384 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.384 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.384 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.384 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.384 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.384 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.384 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.385 INFO:teuthology.orchestra.run.smithi067.stdout: "interval": { 2024-03-28T16:09:41.385 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "interval", 2024-03-28T16:09:41.385 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "secs", 2024-03-28T16:09:41.385 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.385 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.385 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "60", 2024-03-28T16:09:41.385 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.385 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.385 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.385 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.385 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.385 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.385 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.385 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.386 INFO:teuthology.orchestra.run.smithi067.stdout: "log_level": { 2024-03-28T16:09:41.386 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_level", 2024-03-28T16:09:41.386 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.386 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.386 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.386 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.386 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.386 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.386 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.386 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:41.386 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:41.386 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:41.386 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:41.386 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:41.387 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:41.387 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.387 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.387 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.387 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.387 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.387 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.387 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster": { 2024-03-28T16:09:41.387 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster", 2024-03-28T16:09:41.387 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.387 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.387 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.387 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.387 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.387 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.388 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.388 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.388 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.388 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.388 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.388 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.388 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_cluster_level": { 2024-03-28T16:09:41.388 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_cluster_level", 2024-03-28T16:09:41.388 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.388 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.388 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.388 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "info", 2024-03-28T16:09:41.388 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.388 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.389 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [ 2024-03-28T16:09:41.389 INFO:teuthology.orchestra.run.smithi067.stdout: "", 2024-03-28T16:09:41.389 INFO:teuthology.orchestra.run.smithi067.stdout: "critical", 2024-03-28T16:09:41.389 INFO:teuthology.orchestra.run.smithi067.stdout: "debug", 2024-03-28T16:09:41.389 INFO:teuthology.orchestra.run.smithi067.stdout: "error", 2024-03-28T16:09:41.389 INFO:teuthology.orchestra.run.smithi067.stdout: "info", 2024-03-28T16:09:41.389 INFO:teuthology.orchestra.run.smithi067.stdout: "warning" 2024-03-28T16:09:41.389 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.389 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.389 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.389 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.389 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.389 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.389 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": { 2024-03-28T16:09:41.389 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "log_to_file", 2024-03-28T16:09:41.390 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "bool", 2024-03-28T16:09:41.390 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.390 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-03-28T16:09:41.390 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "False", 2024-03-28T16:09:41.390 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.390 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.390 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.390 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.390 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.390 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.390 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.390 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.390 INFO:teuthology.orchestra.run.smithi067.stdout: "zabbix_host": { 2024-03-28T16:09:41.390 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "zabbix_host", 2024-03-28T16:09:41.391 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.391 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.391 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.391 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "", 2024-03-28T16:09:41.391 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.391 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.391 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.391 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.391 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.391 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.391 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.391 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.391 INFO:teuthology.orchestra.run.smithi067.stdout: "zabbix_port": { 2024-03-28T16:09:41.391 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "zabbix_port", 2024-03-28T16:09:41.391 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "int", 2024-03-28T16:09:41.392 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.392 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.392 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "10051", 2024-03-28T16:09:41.392 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.392 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.392 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.392 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.392 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.392 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.392 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.392 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.392 INFO:teuthology.orchestra.run.smithi067.stdout: "zabbix_sender": { 2024-03-28T16:09:41.392 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "zabbix_sender", 2024-03-28T16:09:41.392 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "str", 2024-03-28T16:09:41.392 INFO:teuthology.orchestra.run.smithi067.stdout: "level": "advanced", 2024-03-28T16:09:41.393 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 0, 2024-03-28T16:09:41.393 INFO:teuthology.orchestra.run.smithi067.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-28T16:09:41.393 INFO:teuthology.orchestra.run.smithi067.stdout: "min": "", 2024-03-28T16:09:41.393 INFO:teuthology.orchestra.run.smithi067.stdout: "max": "", 2024-03-28T16:09:41.393 INFO:teuthology.orchestra.run.smithi067.stdout: "enum_allowed": [], 2024-03-28T16:09:41.393 INFO:teuthology.orchestra.run.smithi067.stdout: "desc": "", 2024-03-28T16:09:41.393 INFO:teuthology.orchestra.run.smithi067.stdout: "long_desc": "", 2024-03-28T16:09:41.393 INFO:teuthology.orchestra.run.smithi067.stdout: "tags": [], 2024-03-28T16:09:41.393 INFO:teuthology.orchestra.run.smithi067.stdout: "see_also": [] 2024-03-28T16:09:41.393 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.393 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.393 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.393 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.393 INFO:teuthology.orchestra.run.smithi067.stdout: "services": { 2024-03-28T16:09:41.394 INFO:teuthology.orchestra.run.smithi067.stdout: "dashboard": "https://172.21.15.67:8443/" 2024-03-28T16:09:41.394 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.394 INFO:teuthology.orchestra.run.smithi067.stdout: "always_on_modules": { 2024-03-28T16:09:41.394 INFO:teuthology.orchestra.run.smithi067.stdout: "octopus": [ 2024-03-28T16:09:41.394 INFO:teuthology.orchestra.run.smithi067.stdout: "balancer", 2024-03-28T16:09:41.394 INFO:teuthology.orchestra.run.smithi067.stdout: "crash", 2024-03-28T16:09:41.394 INFO:teuthology.orchestra.run.smithi067.stdout: "devicehealth", 2024-03-28T16:09:41.394 INFO:teuthology.orchestra.run.smithi067.stdout: "orchestrator", 2024-03-28T16:09:41.394 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscaler", 2024-03-28T16:09:41.394 INFO:teuthology.orchestra.run.smithi067.stdout: "progress", 2024-03-28T16:09:41.394 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_support", 2024-03-28T16:09:41.394 INFO:teuthology.orchestra.run.smithi067.stdout: "status", 2024-03-28T16:09:41.394 INFO:teuthology.orchestra.run.smithi067.stdout: "telemetry", 2024-03-28T16:09:41.394 INFO:teuthology.orchestra.run.smithi067.stdout: "volumes" 2024-03-28T16:09:41.395 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.395 INFO:teuthology.orchestra.run.smithi067.stdout: "pacific": [ 2024-03-28T16:09:41.395 INFO:teuthology.orchestra.run.smithi067.stdout: "balancer", 2024-03-28T16:09:41.395 INFO:teuthology.orchestra.run.smithi067.stdout: "crash", 2024-03-28T16:09:41.395 INFO:teuthology.orchestra.run.smithi067.stdout: "devicehealth", 2024-03-28T16:09:41.395 INFO:teuthology.orchestra.run.smithi067.stdout: "orchestrator", 2024-03-28T16:09:41.395 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscaler", 2024-03-28T16:09:41.395 INFO:teuthology.orchestra.run.smithi067.stdout: "progress", 2024-03-28T16:09:41.395 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_support", 2024-03-28T16:09:41.395 INFO:teuthology.orchestra.run.smithi067.stdout: "status", 2024-03-28T16:09:41.395 INFO:teuthology.orchestra.run.smithi067.stdout: "telemetry", 2024-03-28T16:09:41.395 INFO:teuthology.orchestra.run.smithi067.stdout: "volumes" 2024-03-28T16:09:41.395 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.395 INFO:teuthology.orchestra.run.smithi067.stdout: "quincy": [ 2024-03-28T16:09:41.396 INFO:teuthology.orchestra.run.smithi067.stdout: "balancer", 2024-03-28T16:09:41.396 INFO:teuthology.orchestra.run.smithi067.stdout: "crash", 2024-03-28T16:09:41.396 INFO:teuthology.orchestra.run.smithi067.stdout: "devicehealth", 2024-03-28T16:09:41.396 INFO:teuthology.orchestra.run.smithi067.stdout: "orchestrator", 2024-03-28T16:09:41.396 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscaler", 2024-03-28T16:09:41.396 INFO:teuthology.orchestra.run.smithi067.stdout: "progress", 2024-03-28T16:09:41.396 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_support", 2024-03-28T16:09:41.396 INFO:teuthology.orchestra.run.smithi067.stdout: "status", 2024-03-28T16:09:41.396 INFO:teuthology.orchestra.run.smithi067.stdout: "telemetry", 2024-03-28T16:09:41.396 INFO:teuthology.orchestra.run.smithi067.stdout: "volumes" 2024-03-28T16:09:41.396 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.396 INFO:teuthology.orchestra.run.smithi067.stdout: "reef": [ 2024-03-28T16:09:41.396 INFO:teuthology.orchestra.run.smithi067.stdout: "balancer", 2024-03-28T16:09:41.396 INFO:teuthology.orchestra.run.smithi067.stdout: "crash", 2024-03-28T16:09:41.396 INFO:teuthology.orchestra.run.smithi067.stdout: "devicehealth", 2024-03-28T16:09:41.397 INFO:teuthology.orchestra.run.smithi067.stdout: "orchestrator", 2024-03-28T16:09:41.397 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscaler", 2024-03-28T16:09:41.397 INFO:teuthology.orchestra.run.smithi067.stdout: "progress", 2024-03-28T16:09:41.397 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_support", 2024-03-28T16:09:41.397 INFO:teuthology.orchestra.run.smithi067.stdout: "status", 2024-03-28T16:09:41.397 INFO:teuthology.orchestra.run.smithi067.stdout: "telemetry", 2024-03-28T16:09:41.397 INFO:teuthology.orchestra.run.smithi067.stdout: "volumes" 2024-03-28T16:09:41.397 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-28T16:09:41.397 INFO:teuthology.orchestra.run.smithi067.stdout: "squid": [ 2024-03-28T16:09:41.397 INFO:teuthology.orchestra.run.smithi067.stdout: "balancer", 2024-03-28T16:09:41.397 INFO:teuthology.orchestra.run.smithi067.stdout: "crash", 2024-03-28T16:09:41.397 INFO:teuthology.orchestra.run.smithi067.stdout: "devicehealth", 2024-03-28T16:09:41.397 INFO:teuthology.orchestra.run.smithi067.stdout: "orchestrator", 2024-03-28T16:09:41.397 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscaler", 2024-03-28T16:09:41.397 INFO:teuthology.orchestra.run.smithi067.stdout: "progress", 2024-03-28T16:09:41.397 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_support", 2024-03-28T16:09:41.398 INFO:teuthology.orchestra.run.smithi067.stdout: "status", 2024-03-28T16:09:41.398 INFO:teuthology.orchestra.run.smithi067.stdout: "telemetry", 2024-03-28T16:09:41.398 INFO:teuthology.orchestra.run.smithi067.stdout: "volumes" 2024-03-28T16:09:41.398 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-28T16:09:41.398 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.398 INFO:teuthology.orchestra.run.smithi067.stdout: "last_failure_osd_epoch": 3, 2024-03-28T16:09:41.398 INFO:teuthology.orchestra.run.smithi067.stdout: "active_clients": [ 2024-03-28T16:09:41.398 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:41.398 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "devicehealth", 2024-03-28T16:09:41.398 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-03-28T16:09:41.398 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:41.398 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-03-28T16:09:41.398 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:0", 2024-03-28T16:09:41.398 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 457469350 2024-03-28T16:09:41.399 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.399 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-28T16:09:41.399 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.399 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:41.399 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "libcephsqlite", 2024-03-28T16:09:41.399 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-03-28T16:09:41.399 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:41.399 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-03-28T16:09:41.399 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:0", 2024-03-28T16:09:41.399 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1633803673 2024-03-28T16:09:41.399 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.399 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-28T16:09:41.399 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.399 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:41.399 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "rbd_support", 2024-03-28T16:09:41.400 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-03-28T16:09:41.400 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:41.400 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-03-28T16:09:41.400 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:0", 2024-03-28T16:09:41.400 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3523948111 2024-03-28T16:09:41.400 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.400 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-28T16:09:41.400 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-28T16:09:41.400 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:41.400 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "volumes", 2024-03-28T16:09:41.400 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-03-28T16:09:41.400 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-28T16:09:41.400 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-03-28T16:09:41.400 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:0", 2024-03-28T16:09:41.400 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2423556346 2024-03-28T16:09:41.401 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.401 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-28T16:09:41.401 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-28T16:09:41.401 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-28T16:09:41.401 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-28T16:09:41.407 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-03-28T16:09:41.417 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-03-28T16:09:41.418 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-28T16:09:41.418 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T16:09:41.419 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:40 smithi067 ceph-mon[31707]: pgmap v622: 97 pgs: 97 active+clean; 980 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:09:41.419 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:40 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/1170207559' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-28T16:09:41.419 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:40 smithi150 ceph-mon[34825]: pgmap v622: 97 pgs: 97 active+clean; 980 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:09:41.419 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:40 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/1170207559' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-28T16:09:41.419 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:40 smithi183 ceph-mon[35580]: pgmap v622: 97 pgs: 97 active+clean; 980 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:09:41.420 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:40 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/1170207559' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-28T16:09:41.479 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-28T16:09:41.479 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T16:09:41.530 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T16:09:41.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:41.802+0000 7f7d11000640 1 -- 172.21.15.67:0/2563823356 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d0c074100 msgr2=0x7f7d0c074580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:41.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:41.802+0000 7f7d11000640 1 --2- 172.21.15.67:0/2563823356 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d0c074100 0x7f7d0c074580 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f7cf4009930 tx=0x7f7cf402bc30 comp rx=0 tx=0).stop 2024-03-28T16:09:41.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:41.802+0000 7f7d11000640 1 -- 172.21.15.67:0/2563823356 shutdown_connections 2024-03-28T16:09:41.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:41.802+0000 7f7d11000640 1 --2- 172.21.15.67:0/2563823356 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7d0c10dab0 0x7f7d0c10fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:41.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:41.802+0000 7f7d11000640 1 --2- 172.21.15.67:0/2563823356 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d0c074100 0x7f7d0c074580 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:41.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:41.802+0000 7f7d11000640 1 --2- 172.21.15.67:0/2563823356 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d0c0756e0 0x7f7d0c073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:41.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:41.802+0000 7f7d11000640 1 -- 172.21.15.67:0/2563823356 >> 172.21.15.67:0/2563823356 conn(0x7f7d0c0fca30 msgr2=0x7f7d0c0fee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:41.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:41.802+0000 7f7d11000640 1 -- 172.21.15.67:0/2563823356 shutdown_connections 2024-03-28T16:09:41.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:41.802+0000 7f7d11000640 1 -- 172.21.15.67:0/2563823356 wait complete. 2024-03-28T16:09:41.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:41.803+0000 7f7d11000640 1 Processor -- start 2024-03-28T16:09:41.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:41.804+0000 7f7d11000640 1 -- start start 2024-03-28T16:09:41.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:41.804+0000 7f7d11000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7d0c074100 0x7f7d0c19dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:41.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:41.804+0000 7f7d11000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d0c0756e0 0x7f7d0c19e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:41.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:41.804+0000 7f7d11000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d0c10dab0 0x7f7d0c1a5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:41.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:41.804+0000 7f7d11000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d0c1135e0 con 0x7f7d0c0756e0 2024-03-28T16:09:41.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:41.804+0000 7f7d11000640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d0c113460 con 0x7f7d0c10dab0 2024-03-28T16:09:41.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:41.804+0000 7f7d11000640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d0c113760 con 0x7f7d0c074100 2024-03-28T16:09:41.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:41.804+0000 7f7d03e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d0c0756e0 0x7f7d0c19e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:41.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:41.804+0000 7f7d03e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d0c0756e0 0x7f7d0c19e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35334/0 (socket says 172.21.15.67:35334) 2024-03-28T16:09:41.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:41.804+0000 7f7d03e00640 1 -- 172.21.15.67:0/822608362 learned_addr learned my addr 172.21.15.67:0/822608362 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:09:41.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:41.804+0000 7f7d0b400640 1 --2- 172.21.15.67:0/822608362 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7d0c074100 0x7f7d0c19dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:41.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:41.805+0000 7f7d0be00640 1 --2- 172.21.15.67:0/822608362 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d0c10dab0 0x7f7d0c1a5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:41.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:41.805+0000 7f7d03e00640 1 -- 172.21.15.67:0/822608362 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7d0c074100 msgr2=0x7f7d0c19dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:41.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:41.805+0000 7f7d03e00640 1 --2- 172.21.15.67:0/822608362 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7d0c074100 0x7f7d0c19dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:41.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:41.805+0000 7f7d03e00640 1 -- 172.21.15.67:0/822608362 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d0c10dab0 msgr2=0x7f7d0c1a5290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:41.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:41.805+0000 7f7d03e00640 1 --2- 172.21.15.67:0/822608362 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d0c10dab0 0x7f7d0c1a5290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:41.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:41.805+0000 7f7d03e00640 1 -- 172.21.15.67:0/822608362 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d0c1a5930 con 0x7f7d0c0756e0 2024-03-28T16:09:41.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:41.805+0000 7f7d0b400640 1 --2- 172.21.15.67:0/822608362 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7d0c074100 0x7f7d0c19dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:09:41.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:41.805+0000 7f7d0be00640 1 --2- 172.21.15.67:0/822608362 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d0c10dab0 0x7f7d0c1a5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:09:41.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:41.805+0000 7f7d03e00640 1 --2- 172.21.15.67:0/822608362 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d0c0756e0 0x7f7d0c19e260 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f7cf402c840 tx=0x7f7cf40042e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:41.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:41.805+0000 7f7d08c00640 1 -- 172.21.15.67:0/822608362 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cf4043020 con 0x7f7d0c0756e0 2024-03-28T16:09:41.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:41.805+0000 7f7d08c00640 1 -- 172.21.15.67:0/822608362 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7cf40044d0 con 0x7f7d0c0756e0 2024-03-28T16:09:41.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:41.805+0000 7f7d11000640 1 -- 172.21.15.67:0/822608362 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d0c1a5bc0 con 0x7f7d0c0756e0 2024-03-28T16:09:41.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:41.805+0000 7f7d08c00640 1 -- 172.21.15.67:0/822608362 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cf4003e20 con 0x7f7d0c0756e0 2024-03-28T16:09:41.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:41.806+0000 7f7d11000640 1 -- 172.21.15.67:0/822608362 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d0c1a6130 con 0x7f7d0c0756e0 2024-03-28T16:09:41.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:41.807+0000 7f7d11000640 1 -- 172.21.15.67:0/822608362 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d0c103c80 con 0x7f7d0c0756e0 2024-03-28T16:09:41.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:41.809+0000 7f7d08c00640 1 -- 172.21.15.67:0/822608362 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f7cf4046020 con 0x7f7d0c0756e0 2024-03-28T16:09:41.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:41.810+0000 7f7d08c00640 1 --2- 172.21.15.67:0/822608362 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7cf00a2880 0x7f7cf00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:41.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:41.810+0000 7f7d08c00640 1 -- 172.21.15.67:0/822608362 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f7cf40f0fe0 con 0x7f7d0c0756e0 2024-03-28T16:09:41.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:41.810+0000 7f7d0b400640 1 --2- 172.21.15.67:0/822608362 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7cf00a2880 0x7f7cf00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:41.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:41.811+0000 7f7d0b400640 1 --2- 172.21.15.67:0/822608362 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7cf00a2880 0x7f7cf00a4d40 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c19f1f0 tx=0x7f7cfc005db0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:41.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:41.815+0000 7f7d08c00640 1 -- 172.21.15.67:0/822608362 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7cf404d020 con 0x7f7d0c0756e0 2024-03-28T16:09:42.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.028+0000 7f7d11000640 1 -- 172.21.15.67:0/822608362 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7d0c02d2d0 con 0x7f7d0c0756e0 2024-03-28T16:09:42.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.031+0000 7f7d08c00640 1 -- 172.21.15.67:0/822608362 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) v1 ==== 74+0+22673 (secure 0 0 0) 0x7f7cf40be030 con 0x7f7d0c0756e0 2024-03-28T16:09:42.032 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:09:42.032 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":92,"fsid":"b5f8225a-ed1a-11ee-95d0-87774f69a715","created":"2024-03-28T15:50:21.471924+0000","modified":"2024-03-28T16:01:40.643215+0000","last_up_change":"2024-03-28T16:00:01.369123+0000","last_in_change":"2024-03-28T15:59:37.515483+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T15:54:55.588007+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T16:01:05.719801+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T16:01:07.206303+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7c1791b6-9c0e-4512-a327-cad166bb5581","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6803","nonce":1894681701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6805","nonce":1894681701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6809","nonce":1894681701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6807","nonce":1894681701}]},"public_addr":"172.21.15.67:6803/1894681701","cluster_addr":"172.21.15.67:6805/1894681701","heartbeat_back_addr":"172.21.15.67:6809/1894681701","heartbeat_front_addr":"172.21.15.67:6807/1894681701","state":["exists","up"]},{"osd":1,"uuid":"0cdda7f3-d00c-4436-bcc1-49fb8072b2e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6801","nonce":1734465762}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6803","nonce":1734465762}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6807","nonce":1734465762}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6805","nonce":1734465762}]},"public_addr":"172.21.15.150:6801/1734465762","cluster_addr":"172.21.15.150:6803/1734465762","heartbeat_back_addr":"172.21.15.150:6807/1734465762","heartbeat_front_addr":"172.21.15.150:6805/1734465762","state":["exists","up"]},{"osd":2,"uuid":"a55b9a02-c3fd-4061-95c0-53a24a17ade0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6801","nonce":3398095931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6803","nonce":3398095931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6807","nonce":3398095931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6805","nonce":3398095931}]},"public_addr":"172.21.15.183:6801/3398095931","cluster_addr":"172.21.15.183:6803/3398095931","heartbeat_back_addr":"172.21.15.183:6807/3398095931","heartbeat_front_addr":"172.21.15.183:6805/3398095931","state":["exists","up"]},{"osd":3,"uuid":"d34940c0-5ce9-4b8b-aa36-4872414a043f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6811","nonce":1933541875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6813","nonce":1933541875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6817","nonce":1933541875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6815","nonce":1933541875}]},"public_addr":"172.21.15.67:6811/1933541875","cluster_addr":"172.21.15.67:6813/1933541875","heartbeat_back_addr":"172.21.15.67:6817/1933541875","heartbeat_front_addr":"172.21.15.67:6815/1933541875","state":["exists","up"]},{"osd":4,"uuid":"5cc3e260-83a3-4a4d-aa6a-34a3b12b2b5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6809","nonce":3775269993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6811","nonce":3775269993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6815","nonce":3775269993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6813","nonce":3775269993}]},"public_addr":"172.21.15.150:6809/3775269993","cluster_addr":"172.21.15.150:6811/3775269993","heartbeat_back_addr":"172.21.15.150:6815/3775269993","heartbeat_front_addr":"172.21.15.150:6813/3775269993","state":["exists","up"]},{"osd":5,"uuid":"a2542b21-d6ac-413e-8b9b-4a6548b20840","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6809","nonce":164284137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6811","nonce":164284137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6815","nonce":164284137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6813","nonce":164284137}]},"public_addr":"172.21.15.183:6809/164284137","cluster_addr":"172.21.15.183:6811/164284137","heartbeat_back_addr":"172.21.15.183:6815/164284137","heartbeat_front_addr":"172.21.15.183:6813/164284137","state":["exists","up"]},{"osd":6,"uuid":"8905a2d3-0f1b-4853-a218-22da2635150a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6819","nonce":3565528013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6821","nonce":3565528013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6825","nonce":3565528013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6823","nonce":3565528013}]},"public_addr":"172.21.15.67:6819/3565528013","cluster_addr":"172.21.15.67:6821/3565528013","heartbeat_back_addr":"172.21.15.67:6825/3565528013","heartbeat_front_addr":"172.21.15.67:6823/3565528013","state":["exists","up"]},{"osd":7,"uuid":"b275055b-b451-4180-91b1-e6e61a828c48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6817","nonce":3735115340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6819","nonce":3735115340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6823","nonce":3735115340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6821","nonce":3735115340}]},"public_addr":"172.21.15.150:6817/3735115340","cluster_addr":"172.21.15.150:6819/3735115340","heartbeat_back_addr":"172.21.15.150:6823/3735115340","heartbeat_front_addr":"172.21.15.150:6821/3735115340","state":["exists","up"]},{"osd":8,"uuid":"b6abf7ad-eb34-4f60-a4eb-348a92d7195b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6817","nonce":1025400095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6819","nonce":1025400095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6823","nonce":1025400095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6821","nonce":1025400095}]},"public_addr":"172.21.15.183:6817/1025400095","cluster_addr":"172.21.15.183:6819/1025400095","heartbeat_back_addr":"172.21.15.183:6823/1025400095","heartbeat_front_addr":"172.21.15.183:6821/1025400095","state":["exists","up"]},{"osd":9,"uuid":"28a78e3a-0657-40aa-816f-16a6fadadcbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6827","nonce":76885409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6829","nonce":76885409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6833","nonce":76885409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6831","nonce":76885409}]},"public_addr":"172.21.15.67:6827/76885409","cluster_addr":"172.21.15.67:6829/76885409","heartbeat_back_addr":"172.21.15.67:6833/76885409","heartbeat_front_addr":"172.21.15.67:6831/76885409","state":["exists","up"]},{"osd":10,"uuid":"d287907e-424f-4b49-a06e-4613c6de0474","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6825","nonce":3489982080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6827","nonce":3489982080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6831","nonce":3489982080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6829","nonce":3489982080}]},"public_addr":"172.21.15.150:6825/3489982080","cluster_addr":"172.21.15.150:6827/3489982080","heartbeat_back_addr":"172.21.15.150:6831/3489982080","heartbeat_front_addr":"172.21.15.150:6829/3489982080","state":["exists","up"]},{"osd":11,"uuid":"a3c29c47-a228-4f42-a23c-fbdb0c093699","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":85,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6825","nonce":1892639849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6826","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6827","nonce":1892639849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6831","nonce":1892639849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6829","nonce":1892639849}]},"public_addr":"172.21.15.183:6825/1892639849","cluster_addr":"172.21.15.183:6827/1892639849","heartbeat_back_addr":"172.21.15.183:6831/1892639849","heartbeat_front_addr":"172.21.15.183:6829/1892639849","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:53:45.646211+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:54:18.956590+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:54:52.161646+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:55:27.502822+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:56:01.162932+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:56:34.674452+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:57:09.298679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:57:43.230406+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:58:17.003821+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:58:51.085863+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:59:25.665065+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T16:00:00.263457+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":1}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6801/1631001968":"2024-03-29T15:51:38.875334+0000","172.21.15.67:6800/1631001968":"2024-03-29T15:51:38.875334+0000","172.21.15.67:0/1533056310":"2024-03-29T15:51:38.875334+0000","172.21.15.67:6801/3580491071":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/2030412458":"2024-03-29T15:51:03.859740+0000","172.21.15.67:6800/3580491071":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/2905307533":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/3763759988":"2024-03-29T15:51:38.875334+0000","172.21.15.67:0/4089399880":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/4239108661":"2024-03-29T15:51:38.875334+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T16:09:42.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.033+0000 7f7d11000640 1 -- 172.21.15.67:0/822608362 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7cf00a2880 msgr2=0x7f7cf00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:42.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.033+0000 7f7d11000640 1 --2- 172.21.15.67:0/822608362 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7cf00a2880 0x7f7cf00a4d40 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c19f1f0 tx=0x7f7cfc005db0 comp rx=0 tx=0).stop 2024-03-28T16:09:42.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.033+0000 7f7d11000640 1 -- 172.21.15.67:0/822608362 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d0c0756e0 msgr2=0x7f7d0c19e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:42.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.033+0000 7f7d11000640 1 --2- 172.21.15.67:0/822608362 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d0c0756e0 0x7f7d0c19e260 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f7cf402c840 tx=0x7f7cf40042e0 comp rx=0 tx=0).stop 2024-03-28T16:09:42.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.034+0000 7f7d11000640 1 -- 172.21.15.67:0/822608362 shutdown_connections 2024-03-28T16:09:42.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.034+0000 7f7d11000640 1 --2- 172.21.15.67:0/822608362 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f7cf00a2880 0x7f7cf00a4d40 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:42.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.034+0000 7f7d11000640 1 --2- 172.21.15.67:0/822608362 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d0c10dab0 0x7f7d0c1a5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:42.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.034+0000 7f7d11000640 1 --2- 172.21.15.67:0/822608362 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d0c0756e0 0x7f7d0c19e260 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:42.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.034+0000 7f7d11000640 1 --2- 172.21.15.67:0/822608362 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7d0c074100 0x7f7d0c19dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:42.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.034+0000 7f7d11000640 1 -- 172.21.15.67:0/822608362 >> 172.21.15.67:0/822608362 conn(0x7f7d0c0fca30 msgr2=0x7f7d0c100e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:42.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.034+0000 7f7d11000640 1 -- 172.21.15.67:0/822608362 shutdown_connections 2024-03-28T16:09:42.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.034+0000 7f7d11000640 1 -- 172.21.15.67:0/822608362 wait complete. 2024-03-28T16:09:42.053 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T15:54:55.588007+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T16:01:05.719801+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T16:01:07.206303+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T16:09:42.054 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T16:09:42.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.282+0000 7f0a3f200640 1 -- 172.21.15.67:0/2690573685 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a400740a0 msgr2=0x7f0a40074500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:42.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.282+0000 7f0a3f200640 1 --2- 172.21.15.67:0/2690573685 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a400740a0 0x7f0a40074500 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c009900 tx=0x7f0a2c02bcd0 comp rx=0 tx=0).stop 2024-03-28T16:09:42.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.282+0000 7f0a3f200640 1 -- 172.21.15.67:0/2690573685 shutdown_connections 2024-03-28T16:09:42.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.282+0000 7f0a3f200640 1 --2- 172.21.15.67:0/2690573685 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a40109430 0x7f0a4010b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:42.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.282+0000 7f0a3f200640 1 --2- 172.21.15.67:0/2690573685 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a400740a0 0x7f0a40074500 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:42.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.282+0000 7f0a3f200640 1 --2- 172.21.15.67:0/2690573685 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0a40075680 0x7f0a40073ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:42.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.282+0000 7f0a3f200640 1 -- 172.21.15.67:0/2690573685 >> 172.21.15.67:0/2690573685 conn(0x7f0a400fcb10 msgr2=0x7f0a400fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:42.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.282+0000 7f0a3f200640 1 -- 172.21.15.67:0/2690573685 shutdown_connections 2024-03-28T16:09:42.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.282+0000 7f0a3f200640 1 -- 172.21.15.67:0/2690573685 wait complete. 2024-03-28T16:09:42.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.283+0000 7f0a3f200640 1 Processor -- start 2024-03-28T16:09:42.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.283+0000 7f0a3f200640 1 -- start start 2024-03-28T16:09:42.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.284+0000 7f0a3f200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0a400740a0 0x7f0a401999a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:42.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.284+0000 7f0a3f200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a40075680 0x7f0a40199ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:42.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.284+0000 7f0a3f200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a40109430 0x7f0a401a0f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:42.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.284+0000 7f0a3f200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a4010f2b0 con 0x7f0a40109430 2024-03-28T16:09:42.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.284+0000 7f0a3f200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a4010f130 con 0x7f0a40075680 2024-03-28T16:09:42.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.284+0000 7f0a3f200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a4010f430 con 0x7f0a400740a0 2024-03-28T16:09:42.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.284+0000 7f0a3e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a40109430 0x7f0a401a0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:42.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.284+0000 7f0a3e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a40109430 0x7f0a401a0f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35344/0 (socket says 172.21.15.67:35344) 2024-03-28T16:09:42.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.284+0000 7f0a3e800640 1 -- 172.21.15.67:0/1316468240 learned_addr learned my addr 172.21.15.67:0/1316468240 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:09:42.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.284+0000 7f0a3d400640 1 --2- 172.21.15.67:0/1316468240 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a40075680 0x7f0a40199ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:42.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.284+0000 7f0a3de00640 1 --2- 172.21.15.67:0/1316468240 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0a400740a0 0x7f0a401999a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:42.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.285+0000 7f0a3e800640 1 -- 172.21.15.67:0/1316468240 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0a400740a0 msgr2=0x7f0a401999a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:42.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.285+0000 7f0a3e800640 1 --2- 172.21.15.67:0/1316468240 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0a400740a0 0x7f0a401999a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:42.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.285+0000 7f0a3e800640 1 -- 172.21.15.67:0/1316468240 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a40075680 msgr2=0x7f0a40199ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:42.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.285+0000 7f0a3e800640 1 --2- 172.21.15.67:0/1316468240 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a40075680 0x7f0a40199ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:42.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.285+0000 7f0a3e800640 1 -- 172.21.15.67:0/1316468240 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a401a15c0 con 0x7f0a40109430 2024-03-28T16:09:42.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.285+0000 7f0a3e800640 1 --2- 172.21.15.67:0/1316468240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a40109430 0x7f0a401a0f50 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f0a3800eaa0 tx=0x7f0a3800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:42.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.285+0000 7f0a36a00640 1 -- 172.21.15.67:0/1316468240 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a3800cdb0 con 0x7f0a40109430 2024-03-28T16:09:42.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.285+0000 7f0a3f200640 1 -- 172.21.15.67:0/1316468240 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a401a18b0 con 0x7f0a40109430 2024-03-28T16:09:42.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.285+0000 7f0a36a00640 1 -- 172.21.15.67:0/1316468240 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0a38005020 con 0x7f0a40109430 2024-03-28T16:09:42.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.285+0000 7f0a36a00640 1 -- 172.21.15.67:0/1316468240 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a3801f060 con 0x7f0a40109430 2024-03-28T16:09:42.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.285+0000 7f0a3f200640 1 -- 172.21.15.67:0/1316468240 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a401a1db0 con 0x7f0a40109430 2024-03-28T16:09:42.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.287+0000 7f0a3f200640 1 -- 172.21.15.67:0/1316468240 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a401081f0 con 0x7f0a40109430 2024-03-28T16:09:42.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.289+0000 7f0a36a00640 1 -- 172.21.15.67:0/1316468240 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f0a380051d0 con 0x7f0a40109430 2024-03-28T16:09:42.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.291+0000 7f0a36a00640 1 --2- 172.21.15.67:0/1316468240 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f0a240a2800 0x7f0a240a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:42.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.291+0000 7f0a36a00640 1 -- 172.21.15.67:0/1316468240 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f0a380ced00 con 0x7f0a40109430 2024-03-28T16:09:42.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.291+0000 7f0a3de00640 1 --2- 172.21.15.67:0/1316468240 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f0a240a2800 0x7f0a240a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:42.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.292+0000 7f0a3de00640 1 --2- 172.21.15.67:0/1316468240 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f0a240a2800 0x7f0a240a4cc0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f0a300058c0 tx=0x7f0a3000a4c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:42.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.295+0000 7f0a36a00640 1 -- 172.21.15.67:0/1316468240 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0a38016030 con 0x7f0a40109430 2024-03-28T16:09:42.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.502+0000 7f0a3f200640 1 -- 172.21.15.67:0/1316468240 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0a400622a0 con 0x7f0a40109430 2024-03-28T16:09:42.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.503+0000 7f0a36a00640 1 -- 172.21.15.67:0/1316468240 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v92) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0a3809bd50 con 0x7f0a40109430 2024-03-28T16:09:42.504 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-03-28T16:09:42.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.506+0000 7f0a3f200640 1 -- 172.21.15.67:0/1316468240 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f0a240a2800 msgr2=0x7f0a240a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:42.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.506+0000 7f0a3f200640 1 --2- 172.21.15.67:0/1316468240 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f0a240a2800 0x7f0a240a4cc0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f0a300058c0 tx=0x7f0a3000a4c0 comp rx=0 tx=0).stop 2024-03-28T16:09:42.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.506+0000 7f0a3f200640 1 -- 172.21.15.67:0/1316468240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a40109430 msgr2=0x7f0a401a0f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:42.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.506+0000 7f0a3f200640 1 --2- 172.21.15.67:0/1316468240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a40109430 0x7f0a401a0f50 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f0a3800eaa0 tx=0x7f0a3800ef60 comp rx=0 tx=0).stop 2024-03-28T16:09:42.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.506+0000 7f0a3f200640 1 -- 172.21.15.67:0/1316468240 shutdown_connections 2024-03-28T16:09:42.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.506+0000 7f0a3f200640 1 --2- 172.21.15.67:0/1316468240 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f0a240a2800 0x7f0a240a4cc0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:42.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.506+0000 7f0a3f200640 1 --2- 172.21.15.67:0/1316468240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a40109430 0x7f0a401a0f50 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:42.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.506+0000 7f0a3f200640 1 --2- 172.21.15.67:0/1316468240 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a40075680 0x7f0a40199ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:42.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.506+0000 7f0a3f200640 1 --2- 172.21.15.67:0/1316468240 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0a400740a0 0x7f0a401999a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:42.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.506+0000 7f0a3f200640 1 -- 172.21.15.67:0/1316468240 >> 172.21.15.67:0/1316468240 conn(0x7f0a400fcb10 msgr2=0x7f0a401053c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:42.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.506+0000 7f0a3f200640 1 -- 172.21.15.67:0/1316468240 shutdown_connections 2024-03-28T16:09:42.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.506+0000 7f0a3f200640 1 -- 172.21.15.67:0/1316468240 wait complete. 2024-03-28T16:09:42.525 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T16:09:42.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.750+0000 7f22ed000640 1 -- 172.21.15.67:0/388472495 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22e8104ab0 msgr2=0x7f22e8104f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:42.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.750+0000 7f22ed000640 1 --2- 172.21.15.67:0/388472495 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22e8104ab0 0x7f22e8104f30 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f22d0009900 tx=0x7f22d002bc30 comp rx=0 tx=0).stop 2024-03-28T16:09:42.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.751+0000 7f22ed000640 1 -- 172.21.15.67:0/388472495 shutdown_connections 2024-03-28T16:09:42.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.751+0000 7f22ed000640 1 --2- 172.21.15.67:0/388472495 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22e8105470 0x7f22e810bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:42.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.751+0000 7f22ed000640 1 --2- 172.21.15.67:0/388472495 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22e8104ab0 0x7f22e8104f30 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:42.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.751+0000 7f22ed000640 1 --2- 172.21.15.67:0/388472495 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f22e81038c0 0x7f22e8103ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:42.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.751+0000 7f22ed000640 1 -- 172.21.15.67:0/388472495 >> 172.21.15.67:0/388472495 conn(0x7f22e80fee50 msgr2=0x7f22e8101270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:42.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.751+0000 7f22ed000640 1 -- 172.21.15.67:0/388472495 shutdown_connections 2024-03-28T16:09:42.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.751+0000 7f22ed000640 1 -- 172.21.15.67:0/388472495 wait complete. 2024-03-28T16:09:42.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.752+0000 7f22ed000640 1 Processor -- start 2024-03-28T16:09:42.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.752+0000 7f22ed000640 1 -- start start 2024-03-28T16:09:42.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.752+0000 7f22ed000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22e81038c0 0x7f22e8199b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:42.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.752+0000 7f22ed000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22e8104ab0 0x7f22e819a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:42.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.752+0000 7f22ed000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f22e8105470 0x7f22e81a10e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:42.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.752+0000 7f22ed000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22e810f440 con 0x7f22e8104ab0 2024-03-28T16:09:42.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.752+0000 7f22ed000640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22e810f2c0 con 0x7f22e81038c0 2024-03-28T16:09:42.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.752+0000 7f22ed000640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22e810f5c0 con 0x7f22e8105470 2024-03-28T16:09:42.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.753+0000 7f22e5400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22e8104ab0 0x7f22e819a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:42.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.753+0000 7f22e5400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22e8104ab0 0x7f22e819a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35372/0 (socket says 172.21.15.67:35372) 2024-03-28T16:09:42.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.753+0000 7f22e5400640 1 -- 172.21.15.67:0/461175338 learned_addr learned my addr 172.21.15.67:0/461175338 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:09:42.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.753+0000 7f22e5e00640 1 --2- 172.21.15.67:0/461175338 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22e81038c0 0x7f22e8199b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:42.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.753+0000 7f22e6800640 1 --2- 172.21.15.67:0/461175338 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f22e8105470 0x7f22e81a10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:42.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.753+0000 7f22e5400640 1 -- 172.21.15.67:0/461175338 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f22e8105470 msgr2=0x7f22e81a10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:42.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.753+0000 7f22e5400640 1 --2- 172.21.15.67:0/461175338 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f22e8105470 0x7f22e81a10e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:42.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.753+0000 7f22e5400640 1 -- 172.21.15.67:0/461175338 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22e81038c0 msgr2=0x7f22e8199b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:42.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.753+0000 7f22e5400640 1 --2- 172.21.15.67:0/461175338 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22e81038c0 0x7f22e8199b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:42.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.753+0000 7f22e5400640 1 -- 172.21.15.67:0/461175338 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22e81a1750 con 0x7f22e8104ab0 2024-03-28T16:09:42.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.753+0000 7f22e5400640 1 --2- 172.21.15.67:0/461175338 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22e8104ab0 0x7f22e819a070 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f22d0000c00 tx=0x7f22d00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:42.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.754+0000 7f22dea00640 1 -- 172.21.15.67:0/461175338 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22d0043020 con 0x7f22e8104ab0 2024-03-28T16:09:42.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.754+0000 7f22dea00640 1 -- 172.21.15.67:0/461175338 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f22d00054c0 con 0x7f22e8104ab0 2024-03-28T16:09:42.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.754+0000 7f22dea00640 1 -- 172.21.15.67:0/461175338 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22d0003d70 con 0x7f22e8104ab0 2024-03-28T16:09:42.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.754+0000 7f22ed000640 1 -- 172.21.15.67:0/461175338 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22e81a19e0 con 0x7f22e8104ab0 2024-03-28T16:09:42.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.754+0000 7f22ed000640 1 -- 172.21.15.67:0/461175338 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22e81a1f80 con 0x7f22e8104ab0 2024-03-28T16:09:42.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.755+0000 7f22ed000640 1 -- 172.21.15.67:0/461175338 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22e810acd0 con 0x7f22e8104ab0 2024-03-28T16:09:42.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.757+0000 7f22dea00640 1 -- 172.21.15.67:0/461175338 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f22d0046020 con 0x7f22e8104ab0 2024-03-28T16:09:42.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.759+0000 7f22dea00640 1 --2- 172.21.15.67:0/461175338 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f22cc0a2880 0x7f22cc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:42.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.760+0000 7f22dea00640 1 -- 172.21.15.67:0/461175338 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f22d00f1060 con 0x7f22e8104ab0 2024-03-28T16:09:42.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.760+0000 7f22e5e00640 1 --2- 172.21.15.67:0/461175338 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f22cc0a2880 0x7f22cc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:42.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.760+0000 7f22e5e00640 1 --2- 172.21.15.67:0/461175338 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f22cc0a2880 0x7f22cc0a4d40 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f22e8104910 tx=0x7f22d8009870 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:42.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.764+0000 7f22dea00640 1 -- 172.21.15.67:0/461175338 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f22d004d020 con 0x7f22e8104ab0 2024-03-28T16:09:42.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.961+0000 7f22ed000640 1 -- 172.21.15.67:0/461175338 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f22e80622a0 con 0x7f22e8104ab0 2024-03-28T16:09:42.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.961+0000 7f22dea00640 1 -- 172.21.15.67:0/461175338 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v92) v1 ==== 104+0+11 (secure 0 0 0) 0x7f22d00be1d0 con 0x7f22e8104ab0 2024-03-28T16:09:42.963 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-03-28T16:09:42.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.964+0000 7f22ed000640 1 -- 172.21.15.67:0/461175338 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f22cc0a2880 msgr2=0x7f22cc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:42.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.964+0000 7f22ed000640 1 --2- 172.21.15.67:0/461175338 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f22cc0a2880 0x7f22cc0a4d40 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f22e8104910 tx=0x7f22d8009870 comp rx=0 tx=0).stop 2024-03-28T16:09:42.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.964+0000 7f22ed000640 1 -- 172.21.15.67:0/461175338 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22e8104ab0 msgr2=0x7f22e819a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:42.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.964+0000 7f22ed000640 1 --2- 172.21.15.67:0/461175338 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22e8104ab0 0x7f22e819a070 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f22d0000c00 tx=0x7f22d00042f0 comp rx=0 tx=0).stop 2024-03-28T16:09:42.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.965+0000 7f22ed000640 1 -- 172.21.15.67:0/461175338 shutdown_connections 2024-03-28T16:09:42.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.965+0000 7f22ed000640 1 --2- 172.21.15.67:0/461175338 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f22cc0a2880 0x7f22cc0a4d40 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:42.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.965+0000 7f22ed000640 1 --2- 172.21.15.67:0/461175338 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f22e8105470 0x7f22e81a10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:42.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.965+0000 7f22ed000640 1 --2- 172.21.15.67:0/461175338 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22e8104ab0 0x7f22e819a070 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:42.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.965+0000 7f22ed000640 1 --2- 172.21.15.67:0/461175338 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22e81038c0 0x7f22e8199b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:42.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.965+0000 7f22ed000640 1 -- 172.21.15.67:0/461175338 >> 172.21.15.67:0/461175338 conn(0x7f22e80fee50 msgr2=0x7f22e8107ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:42.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.965+0000 7f22ed000640 1 -- 172.21.15.67:0/461175338 shutdown_connections 2024-03-28T16:09:42.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:42.965+0000 7f22ed000640 1 -- 172.21.15.67:0/461175338 wait complete. 2024-03-28T16:09:42.985 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T16:09:43.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:42 smithi183 ceph-mon[35580]: pgmap v623: 97 pgs: 97 active+clean; 980 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:09:43.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:42 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/822608362' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:09:43.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:42 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/1316468240' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T16:09:43.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.212+0000 7f2b58c00640 1 -- 172.21.15.67:0/3140504014 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b54108db0 msgr2=0x7f2b54109230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:43.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.212+0000 7f2b58c00640 1 --2- 172.21.15.67:0/3140504014 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b54108db0 0x7f2b54109230 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f2b40009900 tx=0x7f2b4002bc30 comp rx=0 tx=0).stop 2024-03-28T16:09:43.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.212+0000 7f2b58c00640 1 -- 172.21.15.67:0/3140504014 shutdown_connections 2024-03-28T16:09:43.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.212+0000 7f2b58c00640 1 --2- 172.21.15.67:0/3140504014 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2b54109770 0x7f2b54110000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:43.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.212+0000 7f2b58c00640 1 --2- 172.21.15.67:0/3140504014 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b54108db0 0x7f2b54109230 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:43.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.212+0000 7f2b58c00640 1 --2- 172.21.15.67:0/3140504014 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b54107bc0 0x7f2b54107fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:43.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.212+0000 7f2b58c00640 1 -- 172.21.15.67:0/3140504014 >> 172.21.15.67:0/3140504014 conn(0x7f2b54076b70 msgr2=0x7f2b54078f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:43.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.212+0000 7f2b58c00640 1 -- 172.21.15.67:0/3140504014 shutdown_connections 2024-03-28T16:09:43.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.212+0000 7f2b58c00640 1 -- 172.21.15.67:0/3140504014 wait complete. 2024-03-28T16:09:43.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.213+0000 7f2b58c00640 1 Processor -- start 2024-03-28T16:09:43.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.213+0000 7f2b58c00640 1 -- start start 2024-03-28T16:09:43.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.213+0000 7f2b58c00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b54107bc0 0x7f2b5419de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:43.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.213+0000 7f2b58c00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2b54108db0 0x7f2b5419e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:43.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.213+0000 7f2b58c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b54109770 0x7f2b541a5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:43.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.213+0000 7f2b58c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b54113740 con 0x7f2b54109770 2024-03-28T16:09:43.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.213+0000 7f2b58c00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b541135c0 con 0x7f2b54107bc0 2024-03-28T16:09:43.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.213+0000 7f2b58c00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b541138c0 con 0x7f2b54108db0 2024-03-28T16:09:43.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.213+0000 7f2b53e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b54109770 0x7f2b541a5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:43.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.213+0000 7f2b53e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b54109770 0x7f2b541a5390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35404/0 (socket says 172.21.15.67:35404) 2024-03-28T16:09:43.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.213+0000 7f2b53e00640 1 -- 172.21.15.67:0/3811333857 learned_addr learned my addr 172.21.15.67:0/3811333857 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:09:43.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.214+0000 7f2b52a00640 1 --2- 172.21.15.67:0/3811333857 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2b54108db0 0x7f2b5419e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:43.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.214+0000 7f2b53400640 1 --2- 172.21.15.67:0/3811333857 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b54107bc0 0x7f2b5419de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:43.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.214+0000 7f2b53e00640 1 -- 172.21.15.67:0/3811333857 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2b54108db0 msgr2=0x7f2b5419e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:43.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.214+0000 7f2b53e00640 1 --2- 172.21.15.67:0/3811333857 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2b54108db0 0x7f2b5419e370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:43.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.214+0000 7f2b53e00640 1 -- 172.21.15.67:0/3811333857 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b54107bc0 msgr2=0x7f2b5419de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:43.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.214+0000 7f2b53e00640 1 --2- 172.21.15.67:0/3811333857 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b54107bc0 0x7f2b5419de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:43.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.214+0000 7f2b53e00640 1 -- 172.21.15.67:0/3811333857 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b541a5a00 con 0x7f2b54109770 2024-03-28T16:09:43.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.214+0000 7f2b53400640 1 --2- 172.21.15.67:0/3811333857 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b54107bc0 0x7f2b5419de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:09:43.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.214+0000 7f2b53e00640 1 --2- 172.21.15.67:0/3811333857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b54109770 0x7f2b541a5390 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c00eaa0 tx=0x7f2b4c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:43.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.214+0000 7f2b4be00640 1 -- 172.21.15.67:0/3811333857 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b4c00cdb0 con 0x7f2b54109770 2024-03-28T16:09:43.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.214+0000 7f2b4be00640 1 -- 172.21.15.67:0/3811333857 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f2b4c005020 con 0x7f2b54109770 2024-03-28T16:09:43.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.214+0000 7f2b58c00640 1 -- 172.21.15.67:0/3811333857 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b541a5cf0 con 0x7f2b54109770 2024-03-28T16:09:43.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.215+0000 7f2b4be00640 1 -- 172.21.15.67:0/3811333857 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b4c0116d0 con 0x7f2b54109770 2024-03-28T16:09:43.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.215+0000 7f2b58c00640 1 -- 172.21.15.67:0/3811333857 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b541a6200 con 0x7f2b54109770 2024-03-28T16:09:43.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.216+0000 7f2b58c00640 1 -- 172.21.15.67:0/3811333857 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b5410eee0 con 0x7f2b54109770 2024-03-28T16:09:43.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.218+0000 7f2b4be00640 1 -- 172.21.15.67:0/3811333857 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f2b4c0051d0 con 0x7f2b54109770 2024-03-28T16:09:43.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.219+0000 7f2b4be00640 1 --2- 172.21.15.67:0/3811333857 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f2b380a2880 0x7f2b380a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:43.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.219+0000 7f2b4be00640 1 -- 172.21.15.67:0/3811333857 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f2b4c0ceca0 con 0x7f2b54109770 2024-03-28T16:09:43.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.219+0000 7f2b53400640 1 --2- 172.21.15.67:0/3811333857 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f2b380a2880 0x7f2b380a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:43.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.220+0000 7f2b53400640 1 --2- 172.21.15.67:0/3811333857 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f2b380a2880 0x7f2b380a4d40 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f2b54108c10 tx=0x7f2b44005e70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:43.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.224+0000 7f2b4be00640 1 -- 172.21.15.67:0/3811333857 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2b4c016030 con 0x7f2b54109770 2024-03-28T16:09:43.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:42 smithi150 ceph-mon[34825]: pgmap v623: 97 pgs: 97 active+clean; 980 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:09:43.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:42 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/822608362' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:09:43.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:42 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/1316468240' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T16:09:43.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:42 smithi067 ceph-mon[31707]: pgmap v623: 97 pgs: 97 active+clean; 980 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:09:43.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:42 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/822608362' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:09:43.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:42 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/1316468240' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T16:09:43.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.416+0000 7f2b58c00640 1 -- 172.21.15.67:0/3811333857 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2b540622a0 con 0x7f2b54109770 2024-03-28T16:09:43.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.417+0000 7f2b4be00640 1 -- 172.21.15.67:0/3811333857 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2b4c09bcf0 con 0x7f2b54109770 2024-03-28T16:09:43.417 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-03-28T16:09:43.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.419+0000 7f2b58c00640 1 -- 172.21.15.67:0/3811333857 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f2b380a2880 msgr2=0x7f2b380a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:43.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.419+0000 7f2b58c00640 1 --2- 172.21.15.67:0/3811333857 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f2b380a2880 0x7f2b380a4d40 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f2b54108c10 tx=0x7f2b44005e70 comp rx=0 tx=0).stop 2024-03-28T16:09:43.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.420+0000 7f2b58c00640 1 -- 172.21.15.67:0/3811333857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b54109770 msgr2=0x7f2b541a5390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:43.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.420+0000 7f2b58c00640 1 --2- 172.21.15.67:0/3811333857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b54109770 0x7f2b541a5390 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c00eaa0 tx=0x7f2b4c00ef60 comp rx=0 tx=0).stop 2024-03-28T16:09:43.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.420+0000 7f2b58c00640 1 -- 172.21.15.67:0/3811333857 shutdown_connections 2024-03-28T16:09:43.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.420+0000 7f2b58c00640 1 --2- 172.21.15.67:0/3811333857 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f2b380a2880 0x7f2b380a4d40 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:43.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.420+0000 7f2b58c00640 1 --2- 172.21.15.67:0/3811333857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b54109770 0x7f2b541a5390 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:43.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.420+0000 7f2b58c00640 1 --2- 172.21.15.67:0/3811333857 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2b54108db0 0x7f2b5419e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:43.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.420+0000 7f2b58c00640 1 --2- 172.21.15.67:0/3811333857 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b54107bc0 0x7f2b5419de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:43.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.420+0000 7f2b58c00640 1 -- 172.21.15.67:0/3811333857 >> 172.21.15.67:0/3811333857 conn(0x7f2b54076b70 msgr2=0x7f2b5410c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:43.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.420+0000 7f2b58c00640 1 -- 172.21.15.67:0/3811333857 shutdown_connections 2024-03-28T16:09:43.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.420+0000 7f2b58c00640 1 -- 172.21.15.67:0/3811333857 wait complete. 2024-03-28T16:09:43.438 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-28T16:09:43.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.658+0000 7fa94be00640 1 -- 172.21.15.67:0/3656762923 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa94c1037a0 msgr2=0x7fa94c103bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:43.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.658+0000 7fa94be00640 1 --2- 172.21.15.67:0/3656762923 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa94c1037a0 0x7fa94c103bc0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fa93c009900 tx=0x7fa93c02bc30 comp rx=0 tx=0).stop 2024-03-28T16:09:43.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.658+0000 7fa94be00640 1 -- 172.21.15.67:0/3656762923 shutdown_connections 2024-03-28T16:09:43.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.658+0000 7fa94be00640 1 --2- 172.21.15.67:0/3656762923 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa94c105350 0x7fa94c10bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:43.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.658+0000 7fa94be00640 1 --2- 172.21.15.67:0/3656762923 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa94c104990 0x7fa94c104e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:43.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.658+0000 7fa94be00640 1 --2- 172.21.15.67:0/3656762923 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa94c1037a0 0x7fa94c103bc0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:43.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.658+0000 7fa94be00640 1 -- 172.21.15.67:0/3656762923 >> 172.21.15.67:0/3656762923 conn(0x7fa94c0fed10 msgr2=0x7fa94c101150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:43.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.658+0000 7fa94be00640 1 -- 172.21.15.67:0/3656762923 shutdown_connections 2024-03-28T16:09:43.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.658+0000 7fa94be00640 1 -- 172.21.15.67:0/3656762923 wait complete. 2024-03-28T16:09:43.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.659+0000 7fa94be00640 1 Processor -- start 2024-03-28T16:09:43.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.659+0000 7fa94be00640 1 -- start start 2024-03-28T16:09:43.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.660+0000 7fa94be00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa94c1037a0 0x7fa94c199a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:43.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.660+0000 7fa94be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa94c104990 0x7fa94c199f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:43.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.660+0000 7fa94be00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa94c105350 0x7fa94c1a0f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:43.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.660+0000 7fa94be00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa94c10f320 con 0x7fa94c104990 2024-03-28T16:09:43.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.660+0000 7fa94be00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa94c10f1a0 con 0x7fa94c105350 2024-03-28T16:09:43.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.660+0000 7fa94be00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa94c10f4a0 con 0x7fa94c1037a0 2024-03-28T16:09:43.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.660+0000 7fa94a000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa94c104990 0x7fa94c199f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:43.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.660+0000 7fa94a000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa94c104990 0x7fa94c199f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35428/0 (socket says 172.21.15.67:35428) 2024-03-28T16:09:43.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.660+0000 7fa94a000640 1 -- 172.21.15.67:0/786946058 learned_addr learned my addr 172.21.15.67:0/786946058 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:09:43.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.660+0000 7fa94b400640 1 --2- 172.21.15.67:0/786946058 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa94c105350 0x7fa94c1a0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:43.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.660+0000 7fa94aa00640 1 --2- 172.21.15.67:0/786946058 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa94c1037a0 0x7fa94c199a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:43.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.661+0000 7fa94a000640 1 -- 172.21.15.67:0/786946058 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa94c1037a0 msgr2=0x7fa94c199a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:43.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.661+0000 7fa94a000640 1 --2- 172.21.15.67:0/786946058 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa94c1037a0 0x7fa94c199a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:43.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.661+0000 7fa94a000640 1 -- 172.21.15.67:0/786946058 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa94c105350 msgr2=0x7fa94c1a0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:43.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.661+0000 7fa94a000640 1 --2- 172.21.15.67:0/786946058 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa94c105350 0x7fa94c1a0f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:43.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.661+0000 7fa94a000640 1 -- 172.21.15.67:0/786946058 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa94c1a15e0 con 0x7fa94c104990 2024-03-28T16:09:43.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.661+0000 7fa94aa00640 1 --2- 172.21.15.67:0/786946058 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa94c1037a0 0x7fa94c199a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:09:43.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.661+0000 7fa94a000640 1 --2- 172.21.15.67:0/786946058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa94c104990 0x7fa94c199f50 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fa93800e9f0 tx=0x7fa93800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:43.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.661+0000 7fa943400640 1 -- 172.21.15.67:0/786946058 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa93800cd80 con 0x7fa94c104990 2024-03-28T16:09:43.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.661+0000 7fa943400640 1 -- 172.21.15.67:0/786946058 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa9380108e0 con 0x7fa94c104990 2024-03-28T16:09:43.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.661+0000 7fa94be00640 1 -- 172.21.15.67:0/786946058 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa94c1a18d0 con 0x7fa94c104990 2024-03-28T16:09:43.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.661+0000 7fa943400640 1 -- 172.21.15.67:0/786946058 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa938018920 con 0x7fa94c104990 2024-03-28T16:09:43.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.661+0000 7fa94be00640 1 -- 172.21.15.67:0/786946058 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa94c1a1e70 con 0x7fa94c104990 2024-03-28T16:09:43.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.663+0000 7fa94be00640 1 -- 172.21.15.67:0/786946058 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa94c10ab90 con 0x7fa94c104990 2024-03-28T16:09:43.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.665+0000 7fa943400640 1 -- 172.21.15.67:0/786946058 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fa938010a90 con 0x7fa94c104990 2024-03-28T16:09:43.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.667+0000 7fa943400640 1 --2- 172.21.15.67:0/786946058 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fa9300a2880 0x7fa9300a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:43.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.667+0000 7fa943400640 1 -- 172.21.15.67:0/786946058 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fa9380ceeb0 con 0x7fa94c104990 2024-03-28T16:09:43.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.667+0000 7fa94aa00640 1 --2- 172.21.15.67:0/786946058 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fa9300a2880 0x7fa9300a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:43.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.668+0000 7fa94aa00640 1 --2- 172.21.15.67:0/786946058 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fa9300a2880 0x7fa9300a4d40 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fa93c036040 tx=0x7fa93c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:43.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.670+0000 7fa943400640 1 -- 172.21.15.67:0/786946058 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa93809bf00 con 0x7fa94c104990 2024-03-28T16:09:43.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.891+0000 7fa94be00640 1 -- 172.21.15.67:0/786946058 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fa94c0622a0 con 0x7fa94c104990 2024-03-28T16:09:43.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.892+0000 7fa943400640 1 -- 172.21.15.67:0/786946058 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v92) v1 ==== 81+0+592 (secure 0 0 0) 0x7fa93809b800 con 0x7fa94c104990 2024-03-28T16:09:43.892 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:6801/1631001968 2024-03-29T15:51:38.875334+0000 2024-03-28T16:09:43.892 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:6800/1631001968 2024-03-29T15:51:38.875334+0000 2024-03-28T16:09:43.892 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:0/1533056310 2024-03-29T15:51:38.875334+0000 2024-03-28T16:09:43.892 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:6801/3580491071 2024-03-29T15:51:03.859740+0000 2024-03-28T16:09:43.892 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:0/2030412458 2024-03-29T15:51:03.859740+0000 2024-03-28T16:09:43.893 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:6800/3580491071 2024-03-29T15:51:03.859740+0000 2024-03-28T16:09:43.893 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:0/2905307533 2024-03-29T15:51:03.859740+0000 2024-03-28T16:09:43.893 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:0/3763759988 2024-03-29T15:51:38.875334+0000 2024-03-28T16:09:43.893 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:0/4089399880 2024-03-29T15:51:03.859740+0000 2024-03-28T16:09:43.893 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:0/4239108661 2024-03-29T15:51:38.875334+0000 2024-03-28T16:09:43.893 INFO:teuthology.orchestra.run.smithi067.stderr:listed 10 entries 2024-03-28T16:09:43.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.895+0000 7fa94be00640 1 -- 172.21.15.67:0/786946058 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fa9300a2880 msgr2=0x7fa9300a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:43.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.895+0000 7fa94be00640 1 --2- 172.21.15.67:0/786946058 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fa9300a2880 0x7fa9300a4d40 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fa93c036040 tx=0x7fa93c005fd0 comp rx=0 tx=0).stop 2024-03-28T16:09:43.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.895+0000 7fa94be00640 1 -- 172.21.15.67:0/786946058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa94c104990 msgr2=0x7fa94c199f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:43.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.895+0000 7fa94be00640 1 --2- 172.21.15.67:0/786946058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa94c104990 0x7fa94c199f50 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fa93800e9f0 tx=0x7fa93800eeb0 comp rx=0 tx=0).stop 2024-03-28T16:09:43.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.895+0000 7fa94be00640 1 -- 172.21.15.67:0/786946058 shutdown_connections 2024-03-28T16:09:43.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.895+0000 7fa94be00640 1 --2- 172.21.15.67:0/786946058 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fa9300a2880 0x7fa9300a4d40 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:43.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.895+0000 7fa94be00640 1 --2- 172.21.15.67:0/786946058 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa94c105350 0x7fa94c1a0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:43.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.895+0000 7fa94be00640 1 --2- 172.21.15.67:0/786946058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa94c104990 0x7fa94c199f50 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:43.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.895+0000 7fa94be00640 1 --2- 172.21.15.67:0/786946058 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa94c1037a0 0x7fa94c199a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:43.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.895+0000 7fa94be00640 1 -- 172.21.15.67:0/786946058 >> 172.21.15.67:0/786946058 conn(0x7fa94c0fed10 msgr2=0x7fa94c107d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:43.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.895+0000 7fa94be00640 1 -- 172.21.15.67:0/786946058 shutdown_connections 2024-03-28T16:09:43.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:43.895+0000 7fa94be00640 1 -- 172.21.15.67:0/786946058 wait complete. 2024-03-28T16:09:43.915 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-28T16:09:43.916 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T16:09:43.969 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T16:09:44.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:43 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/461175338' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T16:09:44.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:43 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/3811333857' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T16:09:44.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.191+0000 7f1160c00640 1 -- 172.21.15.67:0/1997564554 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f115c1048e0 msgr2=0x7f115c104d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:44.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.191+0000 7f1160c00640 1 --2- 172.21.15.67:0/1997564554 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f115c1048e0 0x7f115c104d60 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f1148009900 tx=0x7f114802bc40 comp rx=0 tx=0).stop 2024-03-28T16:09:44.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.191+0000 7f1160c00640 1 -- 172.21.15.67:0/1997564554 shutdown_connections 2024-03-28T16:09:44.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.191+0000 7f1160c00640 1 --2- 172.21.15.67:0/1997564554 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f115c1052a0 0x7f115c10bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:44.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.191+0000 7f1160c00640 1 --2- 172.21.15.67:0/1997564554 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f115c1048e0 0x7f115c104d60 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:44.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.191+0000 7f1160c00640 1 --2- 172.21.15.67:0/1997564554 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f115c1036f0 0x7f115c103b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:44.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.191+0000 7f1160c00640 1 -- 172.21.15.67:0/1997564554 >> 172.21.15.67:0/1997564554 conn(0x7f115c0fec80 msgr2=0x7f115c1010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:44.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.191+0000 7f1160c00640 1 -- 172.21.15.67:0/1997564554 shutdown_connections 2024-03-28T16:09:44.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.191+0000 7f1160c00640 1 -- 172.21.15.67:0/1997564554 wait complete. 2024-03-28T16:09:44.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.192+0000 7f1160c00640 1 Processor -- start 2024-03-28T16:09:44.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.192+0000 7f1160c00640 1 -- start start 2024-03-28T16:09:44.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.192+0000 7f1160c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f115c1036f0 0x7f115c199900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:44.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.192+0000 7f1160c00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f115c1048e0 0x7f115c199e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:44.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.192+0000 7f1160c00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f115c1052a0 0x7f115c1a0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:44.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.192+0000 7f1160c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f115c10f250 con 0x7f115c1036f0 2024-03-28T16:09:44.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.192+0000 7f1160c00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f115c10f0d0 con 0x7f115c1048e0 2024-03-28T16:09:44.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.192+0000 7f1160c00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f115c10f3d0 con 0x7f115c1052a0 2024-03-28T16:09:44.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.193+0000 7f115b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f115c1036f0 0x7f115c199900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:44.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.193+0000 7f115b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f115c1036f0 0x7f115c199900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35444/0 (socket says 172.21.15.67:35444) 2024-03-28T16:09:44.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.193+0000 7f115b400640 1 -- 172.21.15.67:0/1829813571 learned_addr learned my addr 172.21.15.67:0/1829813571 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:09:44.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.193+0000 7f115aa00640 1 --2- 172.21.15.67:0/1829813571 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f115c1048e0 0x7f115c199e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:44.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.193+0000 7f115be00640 1 --2- 172.21.15.67:0/1829813571 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f115c1052a0 0x7f115c1a0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:44.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.193+0000 7f115b400640 1 -- 172.21.15.67:0/1829813571 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f115c1052a0 msgr2=0x7f115c1a0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:44.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.193+0000 7f115b400640 1 --2- 172.21.15.67:0/1829813571 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f115c1052a0 0x7f115c1a0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:44.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.193+0000 7f115b400640 1 -- 172.21.15.67:0/1829813571 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f115c1048e0 msgr2=0x7f115c199e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:44.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.193+0000 7f115b400640 1 --2- 172.21.15.67:0/1829813571 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f115c1048e0 0x7f115c199e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:44.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.193+0000 7f115b400640 1 -- 172.21.15.67:0/1829813571 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f115c1a1520 con 0x7f115c1036f0 2024-03-28T16:09:44.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.193+0000 7f115aa00640 1 --2- 172.21.15.67:0/1829813571 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f115c1048e0 0x7f115c199e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:09:44.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.193+0000 7f115be00640 1 --2- 172.21.15.67:0/1829813571 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f115c1052a0 0x7f115c1a0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:09:44.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.193+0000 7f115b400640 1 --2- 172.21.15.67:0/1829813571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f115c1036f0 0x7f115c199900 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f114c00d770 tx=0x7f114c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:44.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.194+0000 7f1153e00640 1 -- 172.21.15.67:0/1829813571 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f114c017070 con 0x7f115c1036f0 2024-03-28T16:09:44.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.194+0000 7f1153e00640 1 -- 172.21.15.67:0/1829813571 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f114c009c30 con 0x7f115c1036f0 2024-03-28T16:09:44.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.194+0000 7f1160c00640 1 -- 172.21.15.67:0/1829813571 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f115c1a1810 con 0x7f115c1036f0 2024-03-28T16:09:44.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.194+0000 7f1153e00640 1 -- 172.21.15.67:0/1829813571 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f114c003d00 con 0x7f115c1036f0 2024-03-28T16:09:44.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.194+0000 7f1160c00640 1 -- 172.21.15.67:0/1829813571 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f115c1a1db0 con 0x7f115c1036f0 2024-03-28T16:09:44.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.195+0000 7f1160c00640 1 -- 172.21.15.67:0/1829813571 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f115c10aa90 con 0x7f115c1036f0 2024-03-28T16:09:44.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.197+0000 7f1153e00640 1 -- 172.21.15.67:0/1829813571 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f114c003ea0 con 0x7f115c1036f0 2024-03-28T16:09:44.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.198+0000 7f1153e00640 1 --2- 172.21.15.67:0/1829813571 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f11400a2880 0x7f11400a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:44.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.199+0000 7f1153e00640 1 -- 172.21.15.67:0/1829813571 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f114c0ce830 con 0x7f115c1036f0 2024-03-28T16:09:44.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.199+0000 7f115aa00640 1 --2- 172.21.15.67:0/1829813571 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f11400a2880 0x7f11400a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:44.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.200+0000 7f115aa00640 1 --2- 172.21.15.67:0/1829813571 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f11400a2880 0x7f11400a4d40 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f1148036040 tx=0x7f1148005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:44.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.204+0000 7f1153e00640 1 -- 172.21.15.67:0/1829813571 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f114c09b880 con 0x7f115c1036f0 2024-03-28T16:09:44.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:43 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/461175338' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T16:09:44.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:43 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/3811333857' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T16:09:44.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:43 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/461175338' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T16:09:44.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:43 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/3811333857' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T16:09:44.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.411+0000 7f1160c00640 1 -- 172.21.15.67:0/1829813571 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f115c062340 con 0x7f115c1036f0 2024-03-28T16:09:44.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.415+0000 7f1153e00640 1 -- 172.21.15.67:0/1829813571 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) v1 ==== 74+0+22673 (secure 0 0 0) 0x7f114c009de0 con 0x7f115c1036f0 2024-03-28T16:09:44.416 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:09:44.416 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":92,"fsid":"b5f8225a-ed1a-11ee-95d0-87774f69a715","created":"2024-03-28T15:50:21.471924+0000","modified":"2024-03-28T16:01:40.643215+0000","last_up_change":"2024-03-28T16:00:01.369123+0000","last_in_change":"2024-03-28T15:59:37.515483+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T15:54:55.588007+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T16:01:05.719801+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T16:01:07.206303+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7c1791b6-9c0e-4512-a327-cad166bb5581","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6803","nonce":1894681701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6805","nonce":1894681701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6809","nonce":1894681701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6807","nonce":1894681701}]},"public_addr":"172.21.15.67:6803/1894681701","cluster_addr":"172.21.15.67:6805/1894681701","heartbeat_back_addr":"172.21.15.67:6809/1894681701","heartbeat_front_addr":"172.21.15.67:6807/1894681701","state":["exists","up"]},{"osd":1,"uuid":"0cdda7f3-d00c-4436-bcc1-49fb8072b2e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6801","nonce":1734465762}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6803","nonce":1734465762}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6807","nonce":1734465762}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6805","nonce":1734465762}]},"public_addr":"172.21.15.150:6801/1734465762","cluster_addr":"172.21.15.150:6803/1734465762","heartbeat_back_addr":"172.21.15.150:6807/1734465762","heartbeat_front_addr":"172.21.15.150:6805/1734465762","state":["exists","up"]},{"osd":2,"uuid":"a55b9a02-c3fd-4061-95c0-53a24a17ade0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6801","nonce":3398095931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6803","nonce":3398095931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6807","nonce":3398095931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6805","nonce":3398095931}]},"public_addr":"172.21.15.183:6801/3398095931","cluster_addr":"172.21.15.183:6803/3398095931","heartbeat_back_addr":"172.21.15.183:6807/3398095931","heartbeat_front_addr":"172.21.15.183:6805/3398095931","state":["exists","up"]},{"osd":3,"uuid":"d34940c0-5ce9-4b8b-aa36-4872414a043f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6811","nonce":1933541875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6813","nonce":1933541875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6817","nonce":1933541875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6815","nonce":1933541875}]},"public_addr":"172.21.15.67:6811/1933541875","cluster_addr":"172.21.15.67:6813/1933541875","heartbeat_back_addr":"172.21.15.67:6817/1933541875","heartbeat_front_addr":"172.21.15.67:6815/1933541875","state":["exists","up"]},{"osd":4,"uuid":"5cc3e260-83a3-4a4d-aa6a-34a3b12b2b5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6809","nonce":3775269993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6811","nonce":3775269993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6815","nonce":3775269993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6813","nonce":3775269993}]},"public_addr":"172.21.15.150:6809/3775269993","cluster_addr":"172.21.15.150:6811/3775269993","heartbeat_back_addr":"172.21.15.150:6815/3775269993","heartbeat_front_addr":"172.21.15.150:6813/3775269993","state":["exists","up"]},{"osd":5,"uuid":"a2542b21-d6ac-413e-8b9b-4a6548b20840","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6809","nonce":164284137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6811","nonce":164284137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6815","nonce":164284137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6813","nonce":164284137}]},"public_addr":"172.21.15.183:6809/164284137","cluster_addr":"172.21.15.183:6811/164284137","heartbeat_back_addr":"172.21.15.183:6815/164284137","heartbeat_front_addr":"172.21.15.183:6813/164284137","state":["exists","up"]},{"osd":6,"uuid":"8905a2d3-0f1b-4853-a218-22da2635150a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6819","nonce":3565528013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6821","nonce":3565528013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6825","nonce":3565528013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6823","nonce":3565528013}]},"public_addr":"172.21.15.67:6819/3565528013","cluster_addr":"172.21.15.67:6821/3565528013","heartbeat_back_addr":"172.21.15.67:6825/3565528013","heartbeat_front_addr":"172.21.15.67:6823/3565528013","state":["exists","up"]},{"osd":7,"uuid":"b275055b-b451-4180-91b1-e6e61a828c48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6817","nonce":3735115340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6819","nonce":3735115340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6823","nonce":3735115340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6821","nonce":3735115340}]},"public_addr":"172.21.15.150:6817/3735115340","cluster_addr":"172.21.15.150:6819/3735115340","heartbeat_back_addr":"172.21.15.150:6823/3735115340","heartbeat_front_addr":"172.21.15.150:6821/3735115340","state":["exists","up"]},{"osd":8,"uuid":"b6abf7ad-eb34-4f60-a4eb-348a92d7195b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6817","nonce":1025400095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6819","nonce":1025400095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6823","nonce":1025400095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6821","nonce":1025400095}]},"public_addr":"172.21.15.183:6817/1025400095","cluster_addr":"172.21.15.183:6819/1025400095","heartbeat_back_addr":"172.21.15.183:6823/1025400095","heartbeat_front_addr":"172.21.15.183:6821/1025400095","state":["exists","up"]},{"osd":9,"uuid":"28a78e3a-0657-40aa-816f-16a6fadadcbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6827","nonce":76885409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6829","nonce":76885409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6833","nonce":76885409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6831","nonce":76885409}]},"public_addr":"172.21.15.67:6827/76885409","cluster_addr":"172.21.15.67:6829/76885409","heartbeat_back_addr":"172.21.15.67:6833/76885409","heartbeat_front_addr":"172.21.15.67:6831/76885409","state":["exists","up"]},{"osd":10,"uuid":"d287907e-424f-4b49-a06e-4613c6de0474","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6825","nonce":3489982080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6827","nonce":3489982080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6831","nonce":3489982080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6829","nonce":3489982080}]},"public_addr":"172.21.15.150:6825/3489982080","cluster_addr":"172.21.15.150:6827/3489982080","heartbeat_back_addr":"172.21.15.150:6831/3489982080","heartbeat_front_addr":"172.21.15.150:6829/3489982080","state":["exists","up"]},{"osd":11,"uuid":"a3c29c47-a228-4f42-a23c-fbdb0c093699","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":85,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6825","nonce":1892639849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6826","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6827","nonce":1892639849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6831","nonce":1892639849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6829","nonce":1892639849}]},"public_addr":"172.21.15.183:6825/1892639849","cluster_addr":"172.21.15.183:6827/1892639849","heartbeat_back_addr":"172.21.15.183:6831/1892639849","heartbeat_front_addr":"172.21.15.183:6829/1892639849","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:53:45.646211+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:54:18.956590+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:54:52.161646+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:55:27.502822+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:56:01.162932+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:56:34.674452+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:57:09.298679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:57:43.230406+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:58:17.003821+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:58:51.085863+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:59:25.665065+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T16:00:00.263457+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":1}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6801/1631001968":"2024-03-29T15:51:38.875334+0000","172.21.15.67:6800/1631001968":"2024-03-29T15:51:38.875334+0000","172.21.15.67:0/1533056310":"2024-03-29T15:51:38.875334+0000","172.21.15.67:6801/3580491071":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/2030412458":"2024-03-29T15:51:03.859740+0000","172.21.15.67:6800/3580491071":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/2905307533":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/3763759988":"2024-03-29T15:51:38.875334+0000","172.21.15.67:0/4089399880":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/4239108661":"2024-03-29T15:51:38.875334+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T16:09:44.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.418+0000 7f1160c00640 1 -- 172.21.15.67:0/1829813571 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f11400a2880 msgr2=0x7f11400a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:44.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.418+0000 7f1160c00640 1 --2- 172.21.15.67:0/1829813571 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f11400a2880 0x7f11400a4d40 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f1148036040 tx=0x7f1148005fd0 comp rx=0 tx=0).stop 2024-03-28T16:09:44.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.418+0000 7f1160c00640 1 -- 172.21.15.67:0/1829813571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f115c1036f0 msgr2=0x7f115c199900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:44.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.418+0000 7f1160c00640 1 --2- 172.21.15.67:0/1829813571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f115c1036f0 0x7f115c199900 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f114c00d770 tx=0x7f114c00dc30 comp rx=0 tx=0).stop 2024-03-28T16:09:44.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.419+0000 7f1160c00640 1 -- 172.21.15.67:0/1829813571 shutdown_connections 2024-03-28T16:09:44.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.419+0000 7f1160c00640 1 --2- 172.21.15.67:0/1829813571 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f11400a2880 0x7f11400a4d40 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:44.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.419+0000 7f1160c00640 1 --2- 172.21.15.67:0/1829813571 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f115c1052a0 0x7f115c1a0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:44.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.419+0000 7f1160c00640 1 --2- 172.21.15.67:0/1829813571 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f115c1048e0 0x7f115c199e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:44.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.419+0000 7f1160c00640 1 --2- 172.21.15.67:0/1829813571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f115c1036f0 0x7f115c199900 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:44.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.419+0000 7f1160c00640 1 -- 172.21.15.67:0/1829813571 >> 172.21.15.67:0/1829813571 conn(0x7f115c0fec80 msgr2=0x7f115c107c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:44.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.419+0000 7f1160c00640 1 -- 172.21.15.67:0/1829813571 shutdown_connections 2024-03-28T16:09:44.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.419+0000 7f1160c00640 1 -- 172.21.15.67:0/1829813571 wait complete. 2024-03-28T16:09:44.439 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T15:54:55.588007+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T16:01:05.719801+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T16:01:07.206303+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T16:09:44.440 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T16:09:44.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.670+0000 7f95ce400640 1 -- 172.21.15.67:0/3641379496 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95c8102000 msgr2=0x7f95c81043f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:44.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.670+0000 7f95ce400640 1 --2- 172.21.15.67:0/3641379496 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95c8102000 0x7f95c81043f0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f95b8009ca0 tx=0x7f95b802c0f0 comp rx=0 tx=0).stop 2024-03-28T16:09:44.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.670+0000 7f95ce400640 1 -- 172.21.15.67:0/3641379496 shutdown_connections 2024-03-28T16:09:44.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.670+0000 7f95ce400640 1 --2- 172.21.15.67:0/3641379496 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95c8107260 0x7f95c81096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:44.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.670+0000 7f95ce400640 1 --2- 172.21.15.67:0/3641379496 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f95c8104930 0x7f95c8106d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:44.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.670+0000 7f95ce400640 1 --2- 172.21.15.67:0/3641379496 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95c8102000 0x7f95c81043f0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:44.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.670+0000 7f95ce400640 1 -- 172.21.15.67:0/3641379496 >> 172.21.15.67:0/3641379496 conn(0x7f95c80fba10 msgr2=0x7f95c80fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:44.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.670+0000 7f95ce400640 1 -- 172.21.15.67:0/3641379496 shutdown_connections 2024-03-28T16:09:44.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.670+0000 7f95ce400640 1 -- 172.21.15.67:0/3641379496 wait complete. 2024-03-28T16:09:44.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.671+0000 7f95ce400640 1 Processor -- start 2024-03-28T16:09:44.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.671+0000 7f95ce400640 1 -- start start 2024-03-28T16:09:44.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.672+0000 7f95ce400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95c8102000 0x7f95c8101970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:44.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.672+0000 7f95ce400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95c8104930 0x7f95c80fffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:44.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.672+0000 7f95ce400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f95c8107260 0x7f95c81004f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:44.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.672+0000 7f95ce400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95c810d0c0 con 0x7f95c8102000 2024-03-28T16:09:44.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.672+0000 7f95ce400640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95c810cf40 con 0x7f95c8104930 2024-03-28T16:09:44.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.672+0000 7f95ce400640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95c810d240 con 0x7f95c8107260 2024-03-28T16:09:44.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.672+0000 7f95cd000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95c8102000 0x7f95c8101970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:44.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.672+0000 7f95cd000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95c8102000 0x7f95c8101970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35462/0 (socket says 172.21.15.67:35462) 2024-03-28T16:09:44.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.672+0000 7f95cd000640 1 -- 172.21.15.67:0/1714244002 learned_addr learned my addr 172.21.15.67:0/1714244002 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:09:44.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.672+0000 7f95c7e00640 1 --2- 172.21.15.67:0/1714244002 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95c8104930 0x7f95c80fffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:44.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.672+0000 7f95cda00640 1 --2- 172.21.15.67:0/1714244002 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f95c8107260 0x7f95c81004f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:44.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.672+0000 7f95cd000640 1 -- 172.21.15.67:0/1714244002 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f95c8107260 msgr2=0x7f95c81004f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:44.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.672+0000 7f95cd000640 1 --2- 172.21.15.67:0/1714244002 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f95c8107260 0x7f95c81004f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:44.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.672+0000 7f95cd000640 1 -- 172.21.15.67:0/1714244002 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95c8104930 msgr2=0x7f95c80fffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:44.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.672+0000 7f95cd000640 1 --2- 172.21.15.67:0/1714244002 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95c8104930 0x7f95c80fffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:44.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.672+0000 7f95cd000640 1 -- 172.21.15.67:0/1714244002 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95c8100d80 con 0x7f95c8102000 2024-03-28T16:09:44.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.673+0000 7f95cd000640 1 --2- 172.21.15.67:0/1714244002 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95c8102000 0x7f95c8101970 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f95b8009dc0 tx=0x7f95b80328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:44.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.673+0000 7f95c5600640 1 -- 172.21.15.67:0/1714244002 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b8041020 con 0x7f95c8102000 2024-03-28T16:09:44.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.673+0000 7f95c5600640 1 -- 172.21.15.67:0/1714244002 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f95b8003e20 con 0x7f95c8102000 2024-03-28T16:09:44.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.673+0000 7f95ce400640 1 -- 172.21.15.67:0/1714244002 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95c81a1890 con 0x7f95c8102000 2024-03-28T16:09:44.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.673+0000 7f95c5600640 1 -- 172.21.15.67:0/1714244002 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b8039040 con 0x7f95c8102000 2024-03-28T16:09:44.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.673+0000 7f95ce400640 1 -- 172.21.15.67:0/1714244002 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95c81a1de0 con 0x7f95c8102000 2024-03-28T16:09:44.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.675+0000 7f95ce400640 1 -- 172.21.15.67:0/1714244002 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95c80fd500 con 0x7f95c8102000 2024-03-28T16:09:44.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.677+0000 7f95c5600640 1 -- 172.21.15.67:0/1714244002 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f95b8044020 con 0x7f95c8102000 2024-03-28T16:09:44.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.678+0000 7f95c5600640 1 --2- 172.21.15.67:0/1714244002 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f95ac0a2800 0x7f95ac0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:44.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.678+0000 7f95c5600640 1 -- 172.21.15.67:0/1714244002 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f95b80f0e30 con 0x7f95c8102000 2024-03-28T16:09:44.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.678+0000 7f95c7e00640 1 --2- 172.21.15.67:0/1714244002 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f95ac0a2800 0x7f95ac0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:44.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.679+0000 7f95c7e00640 1 --2- 172.21.15.67:0/1714244002 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f95ac0a2800 0x7f95ac0a4cc0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f95b4005fd0 tx=0x7f95b4005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:44.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.684+0000 7f95c5600640 1 -- 172.21.15.67:0/1714244002 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f95b804b020 con 0x7f95c8102000 2024-03-28T16:09:44.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.894+0000 7f95ce400640 1 -- 172.21.15.67:0/1714244002 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f95c80622a0 con 0x7f95c8102000 2024-03-28T16:09:44.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.895+0000 7f95c5600640 1 -- 172.21.15.67:0/1714244002 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v92) v1 ==== 93+0+10 (secure 0 0 0) 0x7f95b80bde80 con 0x7f95c8102000 2024-03-28T16:09:44.895 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-03-28T16:09:44.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.898+0000 7f95ce400640 1 -- 172.21.15.67:0/1714244002 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f95ac0a2800 msgr2=0x7f95ac0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:44.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.898+0000 7f95ce400640 1 --2- 172.21.15.67:0/1714244002 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f95ac0a2800 0x7f95ac0a4cc0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f95b4005fd0 tx=0x7f95b4005850 comp rx=0 tx=0).stop 2024-03-28T16:09:44.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.898+0000 7f95ce400640 1 -- 172.21.15.67:0/1714244002 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95c8102000 msgr2=0x7f95c8101970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:44.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.898+0000 7f95ce400640 1 --2- 172.21.15.67:0/1714244002 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95c8102000 0x7f95c8101970 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f95b8009dc0 tx=0x7f95b80328a0 comp rx=0 tx=0).stop 2024-03-28T16:09:44.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.898+0000 7f95ce400640 1 -- 172.21.15.67:0/1714244002 shutdown_connections 2024-03-28T16:09:44.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.898+0000 7f95ce400640 1 --2- 172.21.15.67:0/1714244002 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f95ac0a2800 0x7f95ac0a4cc0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:44.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.898+0000 7f95ce400640 1 --2- 172.21.15.67:0/1714244002 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f95c8107260 0x7f95c81004f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:44.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.898+0000 7f95ce400640 1 --2- 172.21.15.67:0/1714244002 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95c8104930 0x7f95c80fffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:44.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.898+0000 7f95ce400640 1 --2- 172.21.15.67:0/1714244002 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95c8102000 0x7f95c8101970 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:44.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.898+0000 7f95ce400640 1 -- 172.21.15.67:0/1714244002 >> 172.21.15.67:0/1714244002 conn(0x7f95c80fba10 msgr2=0x7f95c8102b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:44.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.898+0000 7f95ce400640 1 -- 172.21.15.67:0/1714244002 shutdown_connections 2024-03-28T16:09:44.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:44.898+0000 7f95ce400640 1 -- 172.21.15.67:0/1714244002 wait complete. 2024-03-28T16:09:44.917 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T16:09:45.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.138+0000 7f1491c00640 1 -- 172.21.15.67:0/3480747951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f148c102770 msgr2=0x7f148c102bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:45.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.138+0000 7f1491c00640 1 --2- 172.21.15.67:0/3480747951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f148c102770 0x7f148c102bf0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f1478009900 tx=0x7f147802bc30 comp rx=0 tx=0).stop 2024-03-28T16:09:45.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.138+0000 7f1491c00640 1 -- 172.21.15.67:0/3480747951 shutdown_connections 2024-03-28T16:09:45.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.138+0000 7f1491c00640 1 --2- 172.21.15.67:0/3480747951 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f148c103130 0x7f148c1099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:45.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.138+0000 7f1491c00640 1 --2- 172.21.15.67:0/3480747951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f148c102770 0x7f148c102bf0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:45.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.138+0000 7f1491c00640 1 --2- 172.21.15.67:0/3480747951 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f148c101580 0x7f148c1019a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:45.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.138+0000 7f1491c00640 1 -- 172.21.15.67:0/3480747951 >> 172.21.15.67:0/3480747951 conn(0x7f148c0fcb10 msgr2=0x7f148c0fef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:45.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.138+0000 7f1491c00640 1 -- 172.21.15.67:0/3480747951 shutdown_connections 2024-03-28T16:09:45.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.138+0000 7f1491c00640 1 -- 172.21.15.67:0/3480747951 wait complete. 2024-03-28T16:09:45.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.139+0000 7f1491c00640 1 Processor -- start 2024-03-28T16:09:45.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.140+0000 7f1491c00640 1 -- start start 2024-03-28T16:09:45.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.140+0000 7f1491c00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f148c101580 0x7f148c075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:45.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.140+0000 7f1491c00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f148c102770 0x7f148c073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:45.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.140+0000 7f1491c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f148c103130 0x7f148c073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:45.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.140+0000 7f1491c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f148c10d100 con 0x7f148c103130 2024-03-28T16:09:45.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.140+0000 7f1491c00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f148c10cf80 con 0x7f148c102770 2024-03-28T16:09:45.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.140+0000 7f1491c00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f148c10d280 con 0x7f148c101580 2024-03-28T16:09:45.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.141+0000 7f1491200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f148c103130 0x7f148c073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:45.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.141+0000 7f148b400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f148c102770 0x7f148c073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:45.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.141+0000 7f1491200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f148c103130 0x7f148c073fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35472/0 (socket says 172.21.15.67:35472) 2024-03-28T16:09:45.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.141+0000 7f1491200640 1 -- 172.21.15.67:0/990902202 learned_addr learned my addr 172.21.15.67:0/990902202 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:09:45.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.141+0000 7f148be00640 1 --2- 172.21.15.67:0/990902202 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f148c101580 0x7f148c075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:45.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.141+0000 7f1491200640 1 -- 172.21.15.67:0/990902202 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f148c101580 msgr2=0x7f148c075450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:45.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.141+0000 7f1491200640 1 --2- 172.21.15.67:0/990902202 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f148c101580 0x7f148c075450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:45.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.141+0000 7f1491200640 1 -- 172.21.15.67:0/990902202 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f148c102770 msgr2=0x7f148c073a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:45.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.141+0000 7f1491200640 1 --2- 172.21.15.67:0/990902202 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f148c102770 0x7f148c073a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:45.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.141+0000 7f1491200640 1 -- 172.21.15.67:0/990902202 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f148c074860 con 0x7f148c103130 2024-03-28T16:09:45.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.141+0000 7f148be00640 1 --2- 172.21.15.67:0/990902202 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f148c101580 0x7f148c075450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T16:09:45.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.142+0000 7f1491200640 1 --2- 172.21.15.67:0/990902202 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f148c103130 0x7f148c073fd0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f148400eaa0 tx=0x7f148400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:45.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.142+0000 7f1488c00640 1 -- 172.21.15.67:0/990902202 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f148400cdb0 con 0x7f148c103130 2024-03-28T16:09:45.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.142+0000 7f1488c00640 1 -- 172.21.15.67:0/990902202 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1484005020 con 0x7f148c103130 2024-03-28T16:09:45.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.142+0000 7f1491c00640 1 -- 172.21.15.67:0/990902202 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f148c1a18c0 con 0x7f148c103130 2024-03-28T16:09:45.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.142+0000 7f1488c00640 1 -- 172.21.15.67:0/990902202 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14840116d0 con 0x7f148c103130 2024-03-28T16:09:45.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.142+0000 7f1491c00640 1 -- 172.21.15.67:0/990902202 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f148c1a1de0 con 0x7f148c103130 2024-03-28T16:09:45.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.143+0000 7f1491c00640 1 -- 172.21.15.67:0/990902202 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f148c068800 con 0x7f148c103130 2024-03-28T16:09:45.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.145+0000 7f1488c00640 1 -- 172.21.15.67:0/990902202 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f14840051d0 con 0x7f148c103130 2024-03-28T16:09:45.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.147+0000 7f1488c00640 1 --2- 172.21.15.67:0/990902202 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f14700a2800 0x7f14700a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:45.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.147+0000 7f1488c00640 1 -- 172.21.15.67:0/990902202 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f14840cf080 con 0x7f148c103130 2024-03-28T16:09:45.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.147+0000 7f148be00640 1 --2- 172.21.15.67:0/990902202 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f14700a2800 0x7f14700a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:45.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.148+0000 7f148be00640 1 --2- 172.21.15.67:0/990902202 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f14700a2800 0x7f14700a4cc0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f148c1025d0 tx=0x7f147c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:45.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.149+0000 7f1488c00640 1 -- 172.21.15.67:0/990902202 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1484016030 con 0x7f148c103130 2024-03-28T16:09:45.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:44 smithi150 ceph-mon[34825]: pgmap v624: 97 pgs: 97 active+clean; 980 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:09:45.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:44 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/786946058' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-28T16:09:45.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:44 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/1829813571' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:09:45.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:44 smithi067 ceph-mon[31707]: pgmap v624: 97 pgs: 97 active+clean; 980 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:09:45.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:44 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/786946058' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-28T16:09:45.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:44 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/1829813571' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:09:45.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:44 smithi183 ceph-mon[35580]: pgmap v624: 97 pgs: 97 active+clean; 980 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:09:45.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:44 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/786946058' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-28T16:09:45.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:44 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/1829813571' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:09:45.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.350+0000 7f1491c00640 1 -- 172.21.15.67:0/990902202 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f148c02d070 con 0x7f148c103130 2024-03-28T16:09:45.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.351+0000 7f1488c00640 1 -- 172.21.15.67:0/990902202 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v92) v1 ==== 104+0+11 (secure 0 0 0) 0x7f148409c0d0 con 0x7f148c103130 2024-03-28T16:09:45.351 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-03-28T16:09:45.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.354+0000 7f1491c00640 1 -- 172.21.15.67:0/990902202 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f14700a2800 msgr2=0x7f14700a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:45.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.354+0000 7f1491c00640 1 --2- 172.21.15.67:0/990902202 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f14700a2800 0x7f14700a4cc0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f148c1025d0 tx=0x7f147c005f90 comp rx=0 tx=0).stop 2024-03-28T16:09:45.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.354+0000 7f1491c00640 1 -- 172.21.15.67:0/990902202 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f148c103130 msgr2=0x7f148c073fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:45.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.354+0000 7f1491c00640 1 --2- 172.21.15.67:0/990902202 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f148c103130 0x7f148c073fd0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f148400eaa0 tx=0x7f148400ef60 comp rx=0 tx=0).stop 2024-03-28T16:09:45.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.354+0000 7f1491c00640 1 -- 172.21.15.67:0/990902202 shutdown_connections 2024-03-28T16:09:45.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.354+0000 7f1491c00640 1 --2- 172.21.15.67:0/990902202 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f14700a2800 0x7f14700a4cc0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:45.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.354+0000 7f1491c00640 1 --2- 172.21.15.67:0/990902202 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f148c103130 0x7f148c073fd0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:45.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.354+0000 7f1491c00640 1 --2- 172.21.15.67:0/990902202 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f148c102770 0x7f148c073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:45.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.354+0000 7f1491c00640 1 --2- 172.21.15.67:0/990902202 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f148c101580 0x7f148c075450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:45.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.354+0000 7f1491c00640 1 -- 172.21.15.67:0/990902202 >> 172.21.15.67:0/990902202 conn(0x7f148c0fcb10 msgr2=0x7f148c105a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:45.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.354+0000 7f1491c00640 1 -- 172.21.15.67:0/990902202 shutdown_connections 2024-03-28T16:09:45.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.354+0000 7f1491c00640 1 -- 172.21.15.67:0/990902202 wait complete. 2024-03-28T16:09:45.371 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T16:09:45.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.596+0000 7f6fb2800640 1 -- 172.21.15.67:0/553635953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6fac105370 msgr2=0x7f6fac10bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:45.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.596+0000 7f6fb2800640 1 --2- 172.21.15.67:0/553635953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6fac105370 0x7f6fac10bc00 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f6fa4009ca0 tx=0x7f6fa402c100 comp rx=0 tx=0).stop 2024-03-28T16:09:45.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.596+0000 7f6fb2800640 1 -- 172.21.15.67:0/553635953 shutdown_connections 2024-03-28T16:09:45.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.596+0000 7f6fb2800640 1 --2- 172.21.15.67:0/553635953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6fac105370 0x7f6fac10bc00 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:45.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.596+0000 7f6fb2800640 1 --2- 172.21.15.67:0/553635953 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6fac1049b0 0x7f6fac104e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:45.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.596+0000 7f6fb2800640 1 --2- 172.21.15.67:0/553635953 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6fac1037c0 0x7f6fac103be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:45.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.596+0000 7f6fb2800640 1 -- 172.21.15.67:0/553635953 >> 172.21.15.67:0/553635953 conn(0x7f6fac0fed50 msgr2=0x7f6fac101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:45.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.596+0000 7f6fb2800640 1 -- 172.21.15.67:0/553635953 shutdown_connections 2024-03-28T16:09:45.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.596+0000 7f6fb2800640 1 -- 172.21.15.67:0/553635953 wait complete. 2024-03-28T16:09:45.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.597+0000 7f6fb2800640 1 Processor -- start 2024-03-28T16:09:45.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.597+0000 7f6fb2800640 1 -- start start 2024-03-28T16:09:45.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.598+0000 7f6fb2800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6fac1037c0 0x7f6fac199940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:45.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.598+0000 7f6fb2800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6fac1049b0 0x7f6fac199e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:45.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.598+0000 7f6fb2800640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6fac105370 0x7f6fac1a0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:45.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.598+0000 7f6fb2800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fac10f2d0 con 0x7f6fac1049b0 2024-03-28T16:09:45.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.598+0000 7f6fb2800640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fac10f150 con 0x7f6fac105370 2024-03-28T16:09:45.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.598+0000 7f6fb2800640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fac10f450 con 0x7f6fac1037c0 2024-03-28T16:09:45.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.598+0000 7f6fb0a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6fac1049b0 0x7f6fac199e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:45.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.598+0000 7f6fb0a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6fac1049b0 0x7f6fac199e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35494/0 (socket says 172.21.15.67:35494) 2024-03-28T16:09:45.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.598+0000 7f6fb0a00640 1 -- 172.21.15.67:0/2864014608 learned_addr learned my addr 172.21.15.67:0/2864014608 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:09:45.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.598+0000 7f6fb1400640 1 --2- 172.21.15.67:0/2864014608 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6fac1037c0 0x7f6fac199940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:45.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.598+0000 7f6fb0a00640 1 -- 172.21.15.67:0/2864014608 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6fac1037c0 msgr2=0x7f6fac199940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:45.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.598+0000 7f6fb0a00640 1 --2- 172.21.15.67:0/2864014608 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6fac1037c0 0x7f6fac199940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:45.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.598+0000 7f6fb1e00640 1 --2- 172.21.15.67:0/2864014608 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6fac105370 0x7f6fac1a0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:45.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.598+0000 7f6fb0a00640 1 -- 172.21.15.67:0/2864014608 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6fac105370 msgr2=0x7f6fac1a0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:45.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.598+0000 7f6fb0a00640 1 --2- 172.21.15.67:0/2864014608 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6fac105370 0x7f6fac1a0ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:45.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.598+0000 7f6fb0a00640 1 -- 172.21.15.67:0/2864014608 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fac1a1560 con 0x7f6fac1049b0 2024-03-28T16:09:45.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.598+0000 7f6fb1e00640 1 --2- 172.21.15.67:0/2864014608 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6fac105370 0x7f6fac1a0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:09:45.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.599+0000 7f6fb0a00640 1 --2- 172.21.15.67:0/2864014608 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6fac1049b0 0x7f6fac199e80 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f6f9800d950 tx=0x7f6f9800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:45.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.599+0000 7f6faa000640 1 -- 172.21.15.67:0/2864014608 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f98017070 con 0x7f6fac1049b0 2024-03-28T16:09:45.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.599+0000 7f6fb2800640 1 -- 172.21.15.67:0/2864014608 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fac1a1850 con 0x7f6fac1049b0 2024-03-28T16:09:45.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.599+0000 7f6faa000640 1 -- 172.21.15.67:0/2864014608 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6f98009c30 con 0x7f6fac1049b0 2024-03-28T16:09:45.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.599+0000 7f6faa000640 1 -- 172.21.15.67:0/2864014608 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f9800f620 con 0x7f6fac1049b0 2024-03-28T16:09:45.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.599+0000 7f6fb2800640 1 -- 172.21.15.67:0/2864014608 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fac1a1da0 con 0x7f6fac1049b0 2024-03-28T16:09:45.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.600+0000 7f6fb2800640 1 -- 172.21.15.67:0/2864014608 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fac10aae0 con 0x7f6fac1049b0 2024-03-28T16:09:45.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.603+0000 7f6faa000640 1 -- 172.21.15.67:0/2864014608 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f6f980109a0 con 0x7f6fac1049b0 2024-03-28T16:09:45.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.604+0000 7f6faa000640 1 --2- 172.21.15.67:0/2864014608 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f6f940a2800 0x7f6f940a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:45.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.604+0000 7f6faa000640 1 -- 172.21.15.67:0/2864014608 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f6f980cdfb0 con 0x7f6fac1049b0 2024-03-28T16:09:45.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.605+0000 7f6fb1400640 1 --2- 172.21.15.67:0/2864014608 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f6f940a2800 0x7f6f940a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:45.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.605+0000 7f6fb1400640 1 --2- 172.21.15.67:0/2864014608 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f6f940a2800 0x7f6f940a4cc0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f6fa0005fd0 tx=0x7f6fa0005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:45.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.609+0000 7f6faa000640 1 -- 172.21.15.67:0/2864014608 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6f98009de0 con 0x7f6fac1049b0 2024-03-28T16:09:45.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.813+0000 7f6fb2800640 1 -- 172.21.15.67:0/2864014608 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6fac02d2d0 con 0x7f6fac1049b0 2024-03-28T16:09:45.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.814+0000 7f6faa000640 1 -- 172.21.15.67:0/2864014608 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6f9809b100 con 0x7f6fac1049b0 2024-03-28T16:09:45.814 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-03-28T16:09:45.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.817+0000 7f6fb2800640 1 -- 172.21.15.67:0/2864014608 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f6f940a2800 msgr2=0x7f6f940a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:45.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.817+0000 7f6fb2800640 1 --2- 172.21.15.67:0/2864014608 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f6f940a2800 0x7f6f940a4cc0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f6fa0005fd0 tx=0x7f6fa0005850 comp rx=0 tx=0).stop 2024-03-28T16:09:45.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.817+0000 7f6fb2800640 1 -- 172.21.15.67:0/2864014608 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6fac1049b0 msgr2=0x7f6fac199e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:45.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.817+0000 7f6fb2800640 1 --2- 172.21.15.67:0/2864014608 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6fac1049b0 0x7f6fac199e80 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f6f9800d950 tx=0x7f6f9800de10 comp rx=0 tx=0).stop 2024-03-28T16:09:45.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.817+0000 7f6fb2800640 1 -- 172.21.15.67:0/2864014608 shutdown_connections 2024-03-28T16:09:45.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.817+0000 7f6fb2800640 1 --2- 172.21.15.67:0/2864014608 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f6f940a2800 0x7f6f940a4cc0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:45.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.817+0000 7f6fb2800640 1 --2- 172.21.15.67:0/2864014608 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6fac105370 0x7f6fac1a0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:45.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.817+0000 7f6fb2800640 1 --2- 172.21.15.67:0/2864014608 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6fac1049b0 0x7f6fac199e80 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:45.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.817+0000 7f6fb2800640 1 --2- 172.21.15.67:0/2864014608 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6fac1037c0 0x7f6fac199940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:45.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.817+0000 7f6fb2800640 1 -- 172.21.15.67:0/2864014608 >> 172.21.15.67:0/2864014608 conn(0x7f6fac0fed50 msgr2=0x7f6fac107cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:45.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.817+0000 7f6fb2800640 1 -- 172.21.15.67:0/2864014608 shutdown_connections 2024-03-28T16:09:45.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:45.817+0000 7f6fb2800640 1 -- 172.21.15.67:0/2864014608 wait complete. 2024-03-28T16:09:45.836 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-28T16:09:46.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:46.058+0000 7fb80ee00640 1 -- 172.21.15.67:0/923755149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb808100480 msgr2=0x7fb8081008a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:46.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:46.058+0000 7fb80ee00640 1 --2- 172.21.15.67:0/923755149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb808100480 0x7fb8081008a0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc009900 tx=0x7fb7fc02bc30 comp rx=0 tx=0).stop 2024-03-28T16:09:46.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:46.059+0000 7fb80ee00640 1 -- 172.21.15.67:0/923755149 shutdown_connections 2024-03-28T16:09:46.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:46.059+0000 7fb80ee00640 1 --2- 172.21.15.67:0/923755149 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb808102030 0x7fb8081088c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:46.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:46.059+0000 7fb80ee00640 1 --2- 172.21.15.67:0/923755149 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb808101670 0x7fb808101af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:46.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:46.059+0000 7fb80ee00640 1 --2- 172.21.15.67:0/923755149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb808100480 0x7fb8081008a0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:46.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:46.059+0000 7fb80ee00640 1 -- 172.21.15.67:0/923755149 >> 172.21.15.67:0/923755149 conn(0x7fb8080fba10 msgr2=0x7fb8080fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:46.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:46.059+0000 7fb80ee00640 1 -- 172.21.15.67:0/923755149 shutdown_connections 2024-03-28T16:09:46.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:46.059+0000 7fb80ee00640 1 -- 172.21.15.67:0/923755149 wait complete. 2024-03-28T16:09:46.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:46.060+0000 7fb80ee00640 1 Processor -- start 2024-03-28T16:09:46.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:46.060+0000 7fb80ee00640 1 -- start start 2024-03-28T16:09:46.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:46.060+0000 7fb80ee00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb808100480 0x7fb808104410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:46.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:46.060+0000 7fb80ee00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb808101670 0x7fb808102a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:46.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:46.060+0000 7fb80ee00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb808102030 0x7fb808102f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:46.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:46.060+0000 7fb80ee00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb80810c010 con 0x7fb808100480 2024-03-28T16:09:46.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:46.060+0000 7fb80ee00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb80810be90 con 0x7fb808101670 2024-03-28T16:09:46.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:46.060+0000 7fb80ee00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb80810c190 con 0x7fb808102030 2024-03-28T16:09:46.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:46.061+0000 7fb80da00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb808100480 0x7fb808104410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:46.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:46.061+0000 7fb80da00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb808100480 0x7fb808104410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35502/0 (socket says 172.21.15.67:35502) 2024-03-28T16:09:46.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:46.061+0000 7fb80da00640 1 -- 172.21.15.67:0/3554914567 learned_addr learned my addr 172.21.15.67:0/3554914567 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:09:46.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:46.061+0000 7fb80e400640 1 --2- 172.21.15.67:0/3554914567 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb808102030 0x7fb808102f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:46.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:46.061+0000 7fb80d000640 1 --2- 172.21.15.67:0/3554914567 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb808101670 0x7fb808102a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:46.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:46.061+0000 7fb80da00640 1 -- 172.21.15.67:0/3554914567 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb808102030 msgr2=0x7fb808102f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:46.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:46.061+0000 7fb80da00640 1 --2- 172.21.15.67:0/3554914567 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb808102030 0x7fb808102f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:46.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:46.061+0000 7fb80da00640 1 -- 172.21.15.67:0/3554914567 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb808101670 msgr2=0x7fb808102a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:46.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:46.061+0000 7fb80da00640 1 --2- 172.21.15.67:0/3554914567 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb808101670 0x7fb808102a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:46.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:46.061+0000 7fb80da00640 1 -- 172.21.15.67:0/3554914567 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb808103820 con 0x7fb808100480 2024-03-28T16:09:46.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:46.062+0000 7fb80da00640 1 --2- 172.21.15.67:0/3554914567 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb808100480 0x7fb808104410 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc000c00 tx=0x7fb7fc0040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:46.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:46.062+0000 7fb806000640 1 -- 172.21.15.67:0/3554914567 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7fc043020 con 0x7fb808100480 2024-03-28T16:09:46.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:46.062+0000 7fb80ee00640 1 -- 172.21.15.67:0/3554914567 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8081a1910 con 0x7fb808100480 2024-03-28T16:09:46.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:46.062+0000 7fb806000640 1 -- 172.21.15.67:0/3554914567 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb7fc0044c0 con 0x7fb808100480 2024-03-28T16:09:46.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:46.062+0000 7fb806000640 1 -- 172.21.15.67:0/3554914567 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7fc003c60 con 0x7fb808100480 2024-03-28T16:09:46.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:46.062+0000 7fb80ee00640 1 -- 172.21.15.67:0/3554914567 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8081a1eb0 con 0x7fb808100480 2024-03-28T16:09:46.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:46.063+0000 7fb80ee00640 1 -- 172.21.15.67:0/3554914567 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8081a2080 con 0x7fb808100480 2024-03-28T16:09:46.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:46.067+0000 7fb806000640 1 -- 172.21.15.67:0/3554914567 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fb7fc046020 con 0x7fb808100480 2024-03-28T16:09:46.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:46.069+0000 7fb806000640 1 --2- 172.21.15.67:0/3554914567 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb7f00a2880 0x7fb7f00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:46.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:46.069+0000 7fb806000640 1 -- 172.21.15.67:0/3554914567 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fb7fc0f0d60 con 0x7fb808100480 2024-03-28T16:09:46.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:46.069+0000 7fb80d000640 1 --2- 172.21.15.67:0/3554914567 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb7f00a2880 0x7fb7f00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:46.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:46.070+0000 7fb80d000640 1 --2- 172.21.15.67:0/3554914567 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb7f00a2880 0x7fb7f00a4d40 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fb808104180 tx=0x7fb7f4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:46.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:46.074+0000 7fb806000640 1 -- 172.21.15.67:0/3554914567 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb7fc04d020 con 0x7fb808100480 2024-03-28T16:09:46.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:46.283+0000 7fb80ee00640 1 -- 172.21.15.67:0/3554914567 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fb8081a2250 con 0x7fb808100480 2024-03-28T16:09:46.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:46.284+0000 7fb806000640 1 -- 172.21.15.67:0/3554914567 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v92) v1 ==== 81+0+592 (secure 0 0 0) 0x7fb7fc0dd030 con 0x7fb808100480 2024-03-28T16:09:46.285 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:6801/1631001968 2024-03-29T15:51:38.875334+0000 2024-03-28T16:09:46.285 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:6800/1631001968 2024-03-29T15:51:38.875334+0000 2024-03-28T16:09:46.285 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:0/1533056310 2024-03-29T15:51:38.875334+0000 2024-03-28T16:09:46.285 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:6801/3580491071 2024-03-29T15:51:03.859740+0000 2024-03-28T16:09:46.285 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:0/2030412458 2024-03-29T15:51:03.859740+0000 2024-03-28T16:09:46.285 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:6800/3580491071 2024-03-29T15:51:03.859740+0000 2024-03-28T16:09:46.285 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:0/2905307533 2024-03-29T15:51:03.859740+0000 2024-03-28T16:09:46.285 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:0/3763759988 2024-03-29T15:51:38.875334+0000 2024-03-28T16:09:46.285 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:0/4089399880 2024-03-29T15:51:03.859740+0000 2024-03-28T16:09:46.285 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:0/4239108661 2024-03-29T15:51:38.875334+0000 2024-03-28T16:09:46.285 INFO:teuthology.orchestra.run.smithi067.stderr:listed 10 entries 2024-03-28T16:09:46.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:46.287+0000 7fb80ee00640 1 -- 172.21.15.67:0/3554914567 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb7f00a2880 msgr2=0x7fb7f00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:46.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:46.287+0000 7fb80ee00640 1 --2- 172.21.15.67:0/3554914567 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb7f00a2880 0x7fb7f00a4d40 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fb808104180 tx=0x7fb7f4005f90 comp rx=0 tx=0).stop 2024-03-28T16:09:46.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:46.287+0000 7fb80ee00640 1 -- 172.21.15.67:0/3554914567 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb808100480 msgr2=0x7fb808104410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:46.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:46.287+0000 7fb80ee00640 1 --2- 172.21.15.67:0/3554914567 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb808100480 0x7fb808104410 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc000c00 tx=0x7fb7fc0040c0 comp rx=0 tx=0).stop 2024-03-28T16:09:46.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:46.287+0000 7fb80ee00640 1 -- 172.21.15.67:0/3554914567 shutdown_connections 2024-03-28T16:09:46.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:46.287+0000 7fb80ee00640 1 --2- 172.21.15.67:0/3554914567 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fb7f00a2880 0x7fb7f00a4d40 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:46.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:46.287+0000 7fb80ee00640 1 --2- 172.21.15.67:0/3554914567 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb808102030 0x7fb808102f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:46.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:46.287+0000 7fb80ee00640 1 --2- 172.21.15.67:0/3554914567 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb808101670 0x7fb808102a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:46.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:46.287+0000 7fb80ee00640 1 --2- 172.21.15.67:0/3554914567 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb808100480 0x7fb808104410 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:46.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:46.287+0000 7fb80ee00640 1 -- 172.21.15.67:0/3554914567 >> 172.21.15.67:0/3554914567 conn(0x7fb8080fba10 msgr2=0x7fb808068830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:46.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:46.287+0000 7fb80ee00640 1 -- 172.21.15.67:0/3554914567 shutdown_connections 2024-03-28T16:09:46.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:46.287+0000 7fb80ee00640 1 -- 172.21.15.67:0/3554914567 wait complete. 2024-03-28T16:09:46.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:45 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/1714244002' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T16:09:46.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:45 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/990902202' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T16:09:46.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:45 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/2864014608' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T16:09:46.303 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:45 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/1714244002' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T16:09:46.303 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:45 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/990902202' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T16:09:46.303 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:45 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/2864014608' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T16:09:46.305 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi067.front.sepia.ceph.com... 2024-03-28T16:09:46.305 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T16:09:46.305 DEBUG:teuthology.orchestra.run.smithi067:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-28T16:09:46.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:45 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/1714244002' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T16:09:46.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:45 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/990902202' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T16:09:46.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:45 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/2864014608' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T16:09:46.381 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T16:09:46.396 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi067.stderr:ceph-fuse[83688]: fuse finished with error 0 and tester_r 0 2024-03-28T16:09:47.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:46 smithi150 ceph-mon[34825]: pgmap v625: 97 pgs: 97 active+clean; 980 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:09:47.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:46 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T16:09:47.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:46 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/3554914567' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-28T16:09:47.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:46 smithi067 ceph-mon[31707]: pgmap v625: 97 pgs: 97 active+clean; 980 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:09:47.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:46 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T16:09:47.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:46 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/3554914567' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-28T16:09:47.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:46 smithi183 ceph-mon[35580]: pgmap v625: 97 pgs: 97 active+clean; 980 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T16:09:47.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:46 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T16:09:47.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:46 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/3554914567' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-28T16:09:48.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:47 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:09:48.304 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:47 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:09:48.304 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:47 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:09:48.304 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:47 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:09:48.304 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:47 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:09:48.304 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:47 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:09:48.304 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:47 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:09:48.304 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:47 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:09:48.304 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:47 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:09:48.304 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:47 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T16:09:48.305 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:47 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:09:48.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:47 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:09:48.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:47 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:09:48.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:47 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:09:48.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:47 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:09:48.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:47 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:09:48.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:47 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:09:48.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:47 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:09:48.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:47 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:09:48.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:47 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:09:48.321 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:47 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T16:09:48.321 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:47 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:09:48.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:47 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:09:48.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:47 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:09:48.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:47 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:09:48.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:47 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:09:48.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:47 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:09:48.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:47 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:09:48.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:47 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:09:48.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:47 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:09:48.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:47 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:09:48.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:47 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T16:09:48.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:47 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' 2024-03-28T16:09:49.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:48 smithi150 ceph-mon[34825]: pgmap v626: 97 pgs: 97 active+clean; 980 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:09:49.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:48 smithi150 ceph-mon[34825]: Adjusting osd_memory_target on smithi067 to 176.8M 2024-03-28T16:09:49.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:48 smithi150 ceph-mon[34825]: Adjusting osd_memory_target on smithi150 to 176.8M 2024-03-28T16:09:49.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:48 smithi150 ceph-mon[34825]: Unable to set osd_memory_target on smithi067 to 185395302: error parsing value: Value '185395302' is below minimum 939524096 2024-03-28T16:09:49.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:48 smithi150 ceph-mon[34825]: Unable to set osd_memory_target on smithi150 to 185389568: error parsing value: Value '185389568' is below minimum 939524096 2024-03-28T16:09:49.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:48 smithi067 ceph-mon[31707]: pgmap v626: 97 pgs: 97 active+clean; 980 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:09:49.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:48 smithi067 ceph-mon[31707]: Adjusting osd_memory_target on smithi067 to 176.8M 2024-03-28T16:09:49.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:48 smithi067 ceph-mon[31707]: Adjusting osd_memory_target on smithi150 to 176.8M 2024-03-28T16:09:49.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:48 smithi067 ceph-mon[31707]: Unable to set osd_memory_target on smithi067 to 185395302: error parsing value: Value '185395302' is below minimum 939524096 2024-03-28T16:09:49.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:48 smithi067 ceph-mon[31707]: Unable to set osd_memory_target on smithi150 to 185389568: error parsing value: Value '185389568' is below minimum 939524096 2024-03-28T16:09:49.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:48 smithi183 ceph-mon[35580]: pgmap v626: 97 pgs: 97 active+clean; 980 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-28T16:09:49.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:48 smithi183 ceph-mon[35580]: Adjusting osd_memory_target on smithi067 to 176.8M 2024-03-28T16:09:49.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:48 smithi183 ceph-mon[35580]: Adjusting osd_memory_target on smithi150 to 176.8M 2024-03-28T16:09:49.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:48 smithi183 ceph-mon[35580]: Unable to set osd_memory_target on smithi067 to 185395302: error parsing value: Value '185395302' is below minimum 939524096 2024-03-28T16:09:49.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:48 smithi183 ceph-mon[35580]: Unable to set osd_memory_target on smithi150 to 185389568: error parsing value: Value '185389568' is below minimum 939524096 2024-03-28T16:09:51.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:50 smithi150 ceph-mon[34825]: pgmap v627: 97 pgs: 97 active+clean; 980 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 0 B/s wr, 1 op/s 2024-03-28T16:09:51.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:50 smithi067 ceph-mon[31707]: pgmap v627: 97 pgs: 97 active+clean; 980 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 0 B/s wr, 1 op/s 2024-03-28T16:09:51.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:50 smithi183 ceph-mon[35580]: pgmap v627: 97 pgs: 97 active+clean; 980 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 0 B/s wr, 1 op/s 2024-03-28T16:09:52.482 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi067.front.sepia.ceph.com 2024-03-28T16:09:52.482 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T16:09:52.482 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-28T16:09:52.557 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T16:09:52.557 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-03-28T16:09:52.612 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-28T16:09:52.613 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T16:09:52.614 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-28T16:09:52.718 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T16:09:52.719 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link delete ceph-brx 2024-03-28T16:09:52.857 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-28T16:09:52.868 INFO:tasks.cephadm:Teardown begin 2024-03-28T16:09:52.869 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-28T16:09:52.869 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T16:09:53.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.065+0000 7f1595600640 1 -- 172.21.15.67:0/2163477165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15901052a0 msgr2=0x7f159010bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:53.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.065+0000 7f1595600640 1 --2- 172.21.15.67:0/2163477165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15901052a0 0x7f159010bb30 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f1588009ca0 tx=0x7f158802c0f0 comp rx=0 tx=0).stop 2024-03-28T16:09:53.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.066+0000 7f1595600640 1 -- 172.21.15.67:0/2163477165 shutdown_connections 2024-03-28T16:09:53.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.066+0000 7f1595600640 1 --2- 172.21.15.67:0/2163477165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15901052a0 0x7f159010bb30 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:53.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.066+0000 7f1595600640 1 --2- 172.21.15.67:0/2163477165 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15901048e0 0x7f1590104d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:53.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.066+0000 7f1595600640 1 --2- 172.21.15.67:0/2163477165 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f15901036f0 0x7f1590103b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:53.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.066+0000 7f1595600640 1 -- 172.21.15.67:0/2163477165 >> 172.21.15.67:0/2163477165 conn(0x7f15900fec80 msgr2=0x7f15901010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:53.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.066+0000 7f1595600640 1 -- 172.21.15.67:0/2163477165 shutdown_connections 2024-03-28T16:09:53.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.066+0000 7f1595600640 1 -- 172.21.15.67:0/2163477165 wait complete. 2024-03-28T16:09:53.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.067+0000 7f1595600640 1 Processor -- start 2024-03-28T16:09:53.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.067+0000 7f1595600640 1 -- start start 2024-03-28T16:09:53.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.067+0000 7f1595600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15901036f0 0x7f15901998a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:53.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.067+0000 7f1595600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f15901048e0 0x7f1590199de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:53.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.067+0000 7f1595600640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15901052a0 0x7f15901a0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:53.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.067+0000 7f1595600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f159010f200 con 0x7f15901036f0 2024-03-28T16:09:53.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.067+0000 7f1595600640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f159010f080 con 0x7f15901052a0 2024-03-28T16:09:53.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.067+0000 7f1595600640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f159010f380 con 0x7f15901048e0 2024-03-28T16:09:53.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.067+0000 7f158fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15901036f0 0x7f15901998a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:53.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.067+0000 7f158fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15901036f0 0x7f15901998a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46482/0 (socket says 172.21.15.67:46482) 2024-03-28T16:09:53.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.068+0000 7f158fe00640 1 -- 172.21.15.67:0/20967011 learned_addr learned my addr 172.21.15.67:0/20967011 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:09:53.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.068+0000 7f1587e00640 1 --2- 172.21.15.67:0/20967011 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f15901048e0 0x7f1590199de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:53.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.068+0000 7f1594c00640 1 --2- 172.21.15.67:0/20967011 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15901052a0 0x7f15901a0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:53.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.068+0000 7f158fe00640 1 -- 172.21.15.67:0/20967011 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f15901048e0 msgr2=0x7f1590199de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:53.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.068+0000 7f158fe00640 1 --2- 172.21.15.67:0/20967011 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f15901048e0 0x7f1590199de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:53.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.068+0000 7f158fe00640 1 -- 172.21.15.67:0/20967011 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15901052a0 msgr2=0x7f15901a0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:53.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.068+0000 7f158fe00640 1 --2- 172.21.15.67:0/20967011 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15901052a0 0x7f15901a0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:53.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.068+0000 7f158fe00640 1 -- 172.21.15.67:0/20967011 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15901a14c0 con 0x7f15901036f0 2024-03-28T16:09:53.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.068+0000 7f158fe00640 1 --2- 172.21.15.67:0/20967011 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15901036f0 0x7f15901998a0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f158000d770 tx=0x7f158000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:53.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.069+0000 7f158d600640 1 -- 172.21.15.67:0/20967011 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1580017070 con 0x7f15901036f0 2024-03-28T16:09:53.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.069+0000 7f158d600640 1 -- 172.21.15.67:0/20967011 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1580009c30 con 0x7f15901036f0 2024-03-28T16:09:53.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.069+0000 7f158d600640 1 -- 172.21.15.67:0/20967011 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1580003d00 con 0x7f15901036f0 2024-03-28T16:09:53.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.069+0000 7f1595600640 1 -- 172.21.15.67:0/20967011 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15901a17b0 con 0x7f15901036f0 2024-03-28T16:09:53.073 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:52 smithi067 ceph-mon[31707]: pgmap v628: 97 pgs: 97 active+clean; 981 KiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-28T16:09:53.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.069+0000 7f1595600640 1 -- 172.21.15.67:0/20967011 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15901a1cb0 con 0x7f15901036f0 2024-03-28T16:09:53.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.070+0000 7f1595600640 1 -- 172.21.15.67:0/20967011 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f159010aa10 con 0x7f15901036f0 2024-03-28T16:09:53.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.072+0000 7f158d600640 1 -- 172.21.15.67:0/20967011 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f1580003ea0 con 0x7f15901036f0 2024-03-28T16:09:53.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.074+0000 7f158d600640 1 --2- 172.21.15.67:0/20967011 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f15740a2800 0x7f15740a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:53.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.074+0000 7f158d600640 1 -- 172.21.15.67:0/20967011 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f15800ce890 con 0x7f15901036f0 2024-03-28T16:09:53.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.074+0000 7f1587e00640 1 --2- 172.21.15.67:0/20967011 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f15740a2800 0x7f15740a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:53.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.075+0000 7f1587e00640 1 --2- 172.21.15.67:0/20967011 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f15740a2800 0x7f15740a4cc0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f1578005fd0 tx=0x7f1578005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:53.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.078+0000 7f158d600640 1 -- 172.21.15.67:0/20967011 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f158009b8e0 con 0x7f15901036f0 2024-03-28T16:09:53.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:52 smithi150 ceph-mon[34825]: pgmap v628: 97 pgs: 97 active+clean; 981 KiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-28T16:09:53.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.325+0000 7f1595600640 1 -- 172.21.15.67:0/20967011 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1590062340 con 0x7f15901036f0 2024-03-28T16:09:53.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.327+0000 7f158d600640 1 -- 172.21.15.67:0/20967011 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11341 (secure 0 0 0) 0x7f1580009de0 con 0x7f15901036f0 2024-03-28T16:09:53.328 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:09:53.328 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2008297659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2008297659},{"type":"v1","addr":"172.21.15.67:6835","nonce":2008297659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14532,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/320295745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":320295745},{"type":"v1","addr":"172.21.15.67:6837","nonce":320295745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6839/691821273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":691821273},{"type":"v1","addr":"172.21.15.67:6839","nonce":691821273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6841/3211254875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6840","nonce":3211254875},{"type":"v1","addr":"172.21.15.67:6841","nonce":3211254875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2129665423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2129665423},{"type":"v1","addr":"172.21.15.183:6833","nonce":2129665423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24496,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/1589784425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":1589784425},{"type":"v1","addr":"172.21.15.183:6835","nonce":1589784425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24508,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/2185863835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":2185863835},{"type":"v1","addr":"172.21.15.183:6837","nonce":2185863835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24520,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6839/1242814398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":1242814398},{"type":"v1","addr":"172.21.15.183:6839","nonce":1242814398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":34409,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1008335521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1008335521},{"type":"v1","addr":"172.21.15.150:6833","nonce":1008335521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":34445,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3269935695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3269935695},{"type":"v1","addr":"172.21.15.150:6835","nonce":3269935695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T16:01:07.696824+0000","modified":"2024-03-28T16:01:11.997773+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":34508},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34475":{"gid":34475,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/1112886277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":1112886277},{"type":"v1","addr":"172.21.15.150:6837","nonce":1112886277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_34508":{"gid":34508,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6839/1211011614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":1211011614},{"type":"v1","addr":"172.21.15.150:6839","nonce":1211011614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34508,"qdb_cluster":[34508]},"id":1}]} 2024-03-28T16:09:53.328 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 19 2024-03-28T16:09:53.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.330+0000 7f1595600640 1 -- 172.21.15.67:0/20967011 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f15740a2800 msgr2=0x7f15740a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:53.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.330+0000 7f1595600640 1 --2- 172.21.15.67:0/20967011 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f15740a2800 0x7f15740a4cc0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f1578005fd0 tx=0x7f1578005850 comp rx=0 tx=0).stop 2024-03-28T16:09:53.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.330+0000 7f1595600640 1 -- 172.21.15.67:0/20967011 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15901036f0 msgr2=0x7f15901998a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:53.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.330+0000 7f1595600640 1 --2- 172.21.15.67:0/20967011 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15901036f0 0x7f15901998a0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f158000d770 tx=0x7f158000dc30 comp rx=0 tx=0).stop 2024-03-28T16:09:53.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.330+0000 7f1595600640 1 -- 172.21.15.67:0/20967011 shutdown_connections 2024-03-28T16:09:53.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.330+0000 7f1595600640 1 --2- 172.21.15.67:0/20967011 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f15740a2800 0x7f15740a4cc0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:53.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.330+0000 7f1595600640 1 --2- 172.21.15.67:0/20967011 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15901052a0 0x7f15901a0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:53.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.330+0000 7f1595600640 1 --2- 172.21.15.67:0/20967011 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f15901048e0 0x7f1590199de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:53.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.330+0000 7f1595600640 1 --2- 172.21.15.67:0/20967011 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15901036f0 0x7f15901998a0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:53.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.330+0000 7f1595600640 1 -- 172.21.15.67:0/20967011 >> 172.21.15.67:0/20967011 conn(0x7f15900fec80 msgr2=0x7f1590107be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:53.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.330+0000 7f1595600640 1 -- 172.21.15.67:0/20967011 shutdown_connections 2024-03-28T16:09:53.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.330+0000 7f1595600640 1 -- 172.21.15.67:0/20967011 wait complete. 2024-03-28T16:09:53.350 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T16:09:53.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:52 smithi183 ceph-mon[35580]: pgmap v628: 97 pgs: 97 active+clean; 981 KiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-28T16:09:53.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.574+0000 7f689a000640 1 -- 172.21.15.67:0/441083500 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6894107b00 msgr2=0x7f6894107f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:53.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.574+0000 7f689a000640 1 --2- 172.21.15.67:0/441083500 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6894107b00 0x7f6894107f20 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f6888009900 tx=0x7f688802bc30 comp rx=0 tx=0).stop 2024-03-28T16:09:53.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.574+0000 7f689a000640 1 -- 172.21.15.67:0/441083500 shutdown_connections 2024-03-28T16:09:53.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.575+0000 7f689a000640 1 --2- 172.21.15.67:0/441083500 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68941096b0 0x7f689410ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:53.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.575+0000 7f689a000640 1 --2- 172.21.15.67:0/441083500 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6894108cf0 0x7f6894109170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:53.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.575+0000 7f689a000640 1 --2- 172.21.15.67:0/441083500 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6894107b00 0x7f6894107f20 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:53.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.575+0000 7f689a000640 1 -- 172.21.15.67:0/441083500 >> 172.21.15.67:0/441083500 conn(0x7f6894076ab0 msgr2=0x7f6894078ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:53.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.575+0000 7f689a000640 1 -- 172.21.15.67:0/441083500 shutdown_connections 2024-03-28T16:09:53.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.575+0000 7f689a000640 1 -- 172.21.15.67:0/441083500 wait complete. 2024-03-28T16:09:53.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.576+0000 7f689a000640 1 Processor -- start 2024-03-28T16:09:53.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.576+0000 7f689a000640 1 -- start start 2024-03-28T16:09:53.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.576+0000 7f689a000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6894107b00 0x7f689419dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:53.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.576+0000 7f689a000640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6894108cf0 0x7f689419e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:53.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.576+0000 7f689a000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f68941096b0 0x7f68941a5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:53.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.576+0000 7f689a000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6894113650 con 0x7f6894107b00 2024-03-28T16:09:53.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.576+0000 7f689a000640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68941134d0 con 0x7f6894108cf0 2024-03-28T16:09:53.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.576+0000 7f689a000640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68941137d0 con 0x7f68941096b0 2024-03-28T16:09:53.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.576+0000 7f6898c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6894107b00 0x7f689419dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:53.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.576+0000 7f6898c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6894107b00 0x7f689419dd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46490/0 (socket says 172.21.15.67:46490) 2024-03-28T16:09:53.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.576+0000 7f6898c00640 1 -- 172.21.15.67:0/1216521833 learned_addr learned my addr 172.21.15.67:0/1216521833 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:09:53.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.577+0000 7f6898c00640 1 -- 172.21.15.67:0/1216521833 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f68941096b0 msgr2=0x7f68941a5300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:53.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.577+0000 7f6899600640 1 --2- 172.21.15.67:0/1216521833 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f68941096b0 0x7f68941a5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:53.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.577+0000 7f6898c00640 1 --2- 172.21.15.67:0/1216521833 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f68941096b0 0x7f68941a5300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:53.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.577+0000 7f6898c00640 1 -- 172.21.15.67:0/1216521833 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6894108cf0 msgr2=0x7f689419e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:53.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.577+0000 7f6893e00640 1 --2- 172.21.15.67:0/1216521833 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6894108cf0 0x7f689419e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:53.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.577+0000 7f6898c00640 1 --2- 172.21.15.67:0/1216521833 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6894108cf0 0x7f689419e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:53.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.577+0000 7f6898c00640 1 -- 172.21.15.67:0/1216521833 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68941a5970 con 0x7f6894107b00 2024-03-28T16:09:53.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.577+0000 7f6893e00640 1 --2- 172.21.15.67:0/1216521833 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6894108cf0 0x7f689419e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:09:53.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.577+0000 7f6898c00640 1 --2- 172.21.15.67:0/1216521833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6894107b00 0x7f689419dd50 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f6888005ab0 tx=0x7f68880042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:53.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.578+0000 7f6891600640 1 -- 172.21.15.67:0/1216521833 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6888043020 con 0x7f6894107b00 2024-03-28T16:09:53.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.578+0000 7f6891600640 1 -- 172.21.15.67:0/1216521833 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f68880054c0 con 0x7f6894107b00 2024-03-28T16:09:53.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.578+0000 7f6891600640 1 -- 172.21.15.67:0/1216521833 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6888003d10 con 0x7f6894107b00 2024-03-28T16:09:53.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.578+0000 7f689a000640 1 -- 172.21.15.67:0/1216521833 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68941a5c00 con 0x7f6894107b00 2024-03-28T16:09:53.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.578+0000 7f689a000640 1 -- 172.21.15.67:0/1216521833 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68941a6100 con 0x7f6894107b00 2024-03-28T16:09:53.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.579+0000 7f689a000640 1 -- 172.21.15.67:0/1216521833 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f689410eef0 con 0x7f6894107b00 2024-03-28T16:09:53.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.581+0000 7f6891600640 1 -- 172.21.15.67:0/1216521833 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f6888046020 con 0x7f6894107b00 2024-03-28T16:09:53.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.582+0000 7f6891600640 1 --2- 172.21.15.67:0/1216521833 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f687c0a2800 0x7f687c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:53.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.582+0000 7f6891600640 1 -- 172.21.15.67:0/1216521833 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f68880f0f00 con 0x7f6894107b00 2024-03-28T16:09:53.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.583+0000 7f6893e00640 1 --2- 172.21.15.67:0/1216521833 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f687c0a2800 0x7f687c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:53.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.583+0000 7f6893e00640 1 --2- 172.21.15.67:0/1216521833 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f687c0a2800 0x7f687c0a4cc0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f689419f260 tx=0x7f6880005e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:53.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.587+0000 7f6891600640 1 -- 172.21.15.67:0/1216521833 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f688804d020 con 0x7f6894107b00 2024-03-28T16:09:53.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.832+0000 7f689a000640 1 -- 172.21.15.67:0/1216521833 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f689402d070 con 0x7f6894107b00 2024-03-28T16:09:53.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.834+0000 7f6891600640 1 -- 172.21.15.67:0/1216521833 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11341 (secure 0 0 0) 0x7f689402d070 con 0x7f6894107b00 2024-03-28T16:09:53.834 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:09:53.835 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2008297659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2008297659},{"type":"v1","addr":"172.21.15.67:6835","nonce":2008297659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14532,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/320295745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":320295745},{"type":"v1","addr":"172.21.15.67:6837","nonce":320295745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6839/691821273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":691821273},{"type":"v1","addr":"172.21.15.67:6839","nonce":691821273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6841/3211254875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6840","nonce":3211254875},{"type":"v1","addr":"172.21.15.67:6841","nonce":3211254875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2129665423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2129665423},{"type":"v1","addr":"172.21.15.183:6833","nonce":2129665423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24496,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/1589784425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":1589784425},{"type":"v1","addr":"172.21.15.183:6835","nonce":1589784425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24508,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/2185863835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":2185863835},{"type":"v1","addr":"172.21.15.183:6837","nonce":2185863835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24520,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6839/1242814398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":1242814398},{"type":"v1","addr":"172.21.15.183:6839","nonce":1242814398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":34409,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1008335521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1008335521},{"type":"v1","addr":"172.21.15.150:6833","nonce":1008335521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":34445,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3269935695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3269935695},{"type":"v1","addr":"172.21.15.150:6835","nonce":3269935695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T16:01:07.696824+0000","modified":"2024-03-28T16:01:11.997773+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":34508},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34475":{"gid":34475,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/1112886277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":1112886277},{"type":"v1","addr":"172.21.15.150:6837","nonce":1112886277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_34508":{"gid":34508,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6839/1211011614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":1211011614},{"type":"v1","addr":"172.21.15.150:6839","nonce":1211011614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34508,"qdb_cluster":[34508]},"id":1}]} 2024-03-28T16:09:53.835 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 19 2024-03-28T16:09:53.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.837+0000 7f689a000640 1 -- 172.21.15.67:0/1216521833 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f687c0a2800 msgr2=0x7f687c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:53.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.837+0000 7f689a000640 1 --2- 172.21.15.67:0/1216521833 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f687c0a2800 0x7f687c0a4cc0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f689419f260 tx=0x7f6880005e50 comp rx=0 tx=0).stop 2024-03-28T16:09:53.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.837+0000 7f689a000640 1 -- 172.21.15.67:0/1216521833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6894107b00 msgr2=0x7f689419dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:53.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.837+0000 7f689a000640 1 --2- 172.21.15.67:0/1216521833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6894107b00 0x7f689419dd50 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f6888005ab0 tx=0x7f68880042f0 comp rx=0 tx=0).stop 2024-03-28T16:09:53.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.837+0000 7f689a000640 1 -- 172.21.15.67:0/1216521833 shutdown_connections 2024-03-28T16:09:53.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.837+0000 7f689a000640 1 --2- 172.21.15.67:0/1216521833 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f687c0a2800 0x7f687c0a4cc0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:53.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.837+0000 7f689a000640 1 --2- 172.21.15.67:0/1216521833 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f68941096b0 0x7f68941a5300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:53.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.837+0000 7f689a000640 1 --2- 172.21.15.67:0/1216521833 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6894108cf0 0x7f689419e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:53.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.837+0000 7f689a000640 1 --2- 172.21.15.67:0/1216521833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6894107b00 0x7f689419dd50 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:53.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.837+0000 7f689a000640 1 -- 172.21.15.67:0/1216521833 >> 172.21.15.67:0/1216521833 conn(0x7f6894076ab0 msgr2=0x7f689410c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:53.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.837+0000 7f689a000640 1 -- 172.21.15.67:0/1216521833 shutdown_connections 2024-03-28T16:09:53.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:53.837+0000 7f689a000640 1 -- 172.21.15.67:0/1216521833 wait complete. 2024-03-28T16:09:53.858 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T16:09:54.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.085+0000 7fee3be00640 1 -- 172.21.15.67:0/206624165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee3c1048e0 msgr2=0x7fee3c104d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:54.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.085+0000 7fee3be00640 1 --2- 172.21.15.67:0/206624165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee3c1048e0 0x7fee3c104d60 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fee24009900 tx=0x7fee2402bc30 comp rx=0 tx=0).stop 2024-03-28T16:09:54.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.085+0000 7fee3be00640 1 -- 172.21.15.67:0/206624165 shutdown_connections 2024-03-28T16:09:54.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.085+0000 7fee3be00640 1 --2- 172.21.15.67:0/206624165 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fee3c1052a0 0x7fee3c10bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:54.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.085+0000 7fee3be00640 1 --2- 172.21.15.67:0/206624165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee3c1048e0 0x7fee3c104d60 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:54.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.085+0000 7fee3be00640 1 --2- 172.21.15.67:0/206624165 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee3c1036f0 0x7fee3c103b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:54.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.085+0000 7fee3be00640 1 -- 172.21.15.67:0/206624165 >> 172.21.15.67:0/206624165 conn(0x7fee3c0fec80 msgr2=0x7fee3c1010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:54.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.085+0000 7fee3be00640 1 -- 172.21.15.67:0/206624165 shutdown_connections 2024-03-28T16:09:54.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.085+0000 7fee3be00640 1 -- 172.21.15.67:0/206624165 wait complete. 2024-03-28T16:09:54.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.086+0000 7fee3be00640 1 Processor -- start 2024-03-28T16:09:54.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.086+0000 7fee3be00640 1 -- start start 2024-03-28T16:09:54.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.087+0000 7fee3be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee3c1036f0 0x7fee3c1998c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:54.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.087+0000 7fee3be00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee3c1048e0 0x7fee3c199e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:54.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.087+0000 7fee3be00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fee3c1052a0 0x7fee3c1a0e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:54.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.087+0000 7fee3be00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee3c10f270 con 0x7fee3c1036f0 2024-03-28T16:09:54.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.087+0000 7fee3be00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee3c10f0f0 con 0x7fee3c1048e0 2024-03-28T16:09:54.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.087+0000 7fee3be00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee3c10f3f0 con 0x7fee3c1052a0 2024-03-28T16:09:54.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.087+0000 7fee3aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee3c1036f0 0x7fee3c1998c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:54.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.087+0000 7fee3aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee3c1036f0 0x7fee3c1998c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46522/0 (socket says 172.21.15.67:46522) 2024-03-28T16:09:54.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.087+0000 7fee3aa00640 1 -- 172.21.15.67:0/2126176533 learned_addr learned my addr 172.21.15.67:0/2126176533 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:09:54.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.087+0000 7fee32000640 1 --2- 172.21.15.67:0/2126176533 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee3c1048e0 0x7fee3c199e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:54.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.088+0000 7fee3b400640 1 --2- 172.21.15.67:0/2126176533 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fee3c1052a0 0x7fee3c1a0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:54.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.088+0000 7fee3aa00640 1 -- 172.21.15.67:0/2126176533 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fee3c1052a0 msgr2=0x7fee3c1a0e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:54.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.088+0000 7fee3aa00640 1 --2- 172.21.15.67:0/2126176533 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fee3c1052a0 0x7fee3c1a0e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:54.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.088+0000 7fee3aa00640 1 -- 172.21.15.67:0/2126176533 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee3c1048e0 msgr2=0x7fee3c199e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:54.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.088+0000 7fee3aa00640 1 --2- 172.21.15.67:0/2126176533 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee3c1048e0 0x7fee3c199e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:54.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.088+0000 7fee3aa00640 1 -- 172.21.15.67:0/2126176533 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee3c1a14e0 con 0x7fee3c1036f0 2024-03-28T16:09:54.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.088+0000 7fee3aa00640 1 --2- 172.21.15.67:0/2126176533 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee3c1036f0 0x7fee3c1998c0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fee2c00d950 tx=0x7fee2c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:54.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.088+0000 7fee33e00640 1 -- 172.21.15.67:0/2126176533 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee2c017020 con 0x7fee3c1036f0 2024-03-28T16:09:54.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.089+0000 7fee33e00640 1 -- 172.21.15.67:0/2126176533 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fee2c009c30 con 0x7fee3c1036f0 2024-03-28T16:09:54.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.089+0000 7fee3be00640 1 -- 172.21.15.67:0/2126176533 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee3c1a17d0 con 0x7fee3c1036f0 2024-03-28T16:09:54.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.089+0000 7fee33e00640 1 -- 172.21.15.67:0/2126176533 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee2c00f6b0 con 0x7fee3c1036f0 2024-03-28T16:09:54.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.089+0000 7fee3be00640 1 -- 172.21.15.67:0/2126176533 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee3c1a1d40 con 0x7fee3c1036f0 2024-03-28T16:09:54.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.090+0000 7fee3be00640 1 -- 172.21.15.67:0/2126176533 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee3c10aa10 con 0x7fee3c1036f0 2024-03-28T16:09:54.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.092+0000 7fee33e00640 1 -- 172.21.15.67:0/2126176533 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fee2c01a020 con 0x7fee3c1036f0 2024-03-28T16:09:54.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.093+0000 7fee33e00640 1 --2- 172.21.15.67:0/2126176533 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fee200a2880 0x7fee200a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:54.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.093+0000 7fee33e00640 1 -- 172.21.15.67:0/2126176533 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fee2c0cedd0 con 0x7fee3c1036f0 2024-03-28T16:09:54.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.094+0000 7fee32000640 1 --2- 172.21.15.67:0/2126176533 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fee200a2880 0x7fee200a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:54.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.094+0000 7fee32000640 1 --2- 172.21.15.67:0/2126176533 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fee200a2880 0x7fee200a4d40 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fee24000c00 tx=0x7fee2402c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:54.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.096+0000 7fee33e00640 1 -- 172.21.15.67:0/2126176533 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fee2c021020 con 0x7fee3c1036f0 2024-03-28T16:09:54.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.301+0000 7fee3be00640 1 -- 172.21.15.67:0/2126176533 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fee3c02d320 con 0x7fee3c1036f0 2024-03-28T16:09:54.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:53 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/20967011' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:09:54.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:53 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/1216521833' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:09:54.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.305+0000 7fee33e00640 1 -- 172.21.15.67:0/2126176533 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) v1 ==== 74+0+22673 (secure 0 0 0) 0x7fee2c009de0 con 0x7fee3c1036f0 2024-03-28T16:09:54.307 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-28T16:09:54.307 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":92,"fsid":"b5f8225a-ed1a-11ee-95d0-87774f69a715","created":"2024-03-28T15:50:21.471924+0000","modified":"2024-03-28T16:01:40.643215+0000","last_up_change":"2024-03-28T16:00:01.369123+0000","last_in_change":"2024-03-28T15:59:37.515483+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T15:54:55.588007+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T16:01:05.719801+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T16:01:07.206303+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7c1791b6-9c0e-4512-a327-cad166bb5581","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6803","nonce":1894681701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6805","nonce":1894681701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6809","nonce":1894681701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1894681701},{"type":"v1","addr":"172.21.15.67:6807","nonce":1894681701}]},"public_addr":"172.21.15.67:6803/1894681701","cluster_addr":"172.21.15.67:6805/1894681701","heartbeat_back_addr":"172.21.15.67:6809/1894681701","heartbeat_front_addr":"172.21.15.67:6807/1894681701","state":["exists","up"]},{"osd":1,"uuid":"0cdda7f3-d00c-4436-bcc1-49fb8072b2e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6801","nonce":1734465762}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6803","nonce":1734465762}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6807","nonce":1734465762}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1734465762},{"type":"v1","addr":"172.21.15.150:6805","nonce":1734465762}]},"public_addr":"172.21.15.150:6801/1734465762","cluster_addr":"172.21.15.150:6803/1734465762","heartbeat_back_addr":"172.21.15.150:6807/1734465762","heartbeat_front_addr":"172.21.15.150:6805/1734465762","state":["exists","up"]},{"osd":2,"uuid":"a55b9a02-c3fd-4061-95c0-53a24a17ade0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6801","nonce":3398095931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6803","nonce":3398095931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6807","nonce":3398095931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":3398095931},{"type":"v1","addr":"172.21.15.183:6805","nonce":3398095931}]},"public_addr":"172.21.15.183:6801/3398095931","cluster_addr":"172.21.15.183:6803/3398095931","heartbeat_back_addr":"172.21.15.183:6807/3398095931","heartbeat_front_addr":"172.21.15.183:6805/3398095931","state":["exists","up"]},{"osd":3,"uuid":"d34940c0-5ce9-4b8b-aa36-4872414a043f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6811","nonce":1933541875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6813","nonce":1933541875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6817","nonce":1933541875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1933541875},{"type":"v1","addr":"172.21.15.67:6815","nonce":1933541875}]},"public_addr":"172.21.15.67:6811/1933541875","cluster_addr":"172.21.15.67:6813/1933541875","heartbeat_back_addr":"172.21.15.67:6817/1933541875","heartbeat_front_addr":"172.21.15.67:6815/1933541875","state":["exists","up"]},{"osd":4,"uuid":"5cc3e260-83a3-4a4d-aa6a-34a3b12b2b5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6809","nonce":3775269993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6811","nonce":3775269993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6815","nonce":3775269993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3775269993},{"type":"v1","addr":"172.21.15.150:6813","nonce":3775269993}]},"public_addr":"172.21.15.150:6809/3775269993","cluster_addr":"172.21.15.150:6811/3775269993","heartbeat_back_addr":"172.21.15.150:6815/3775269993","heartbeat_front_addr":"172.21.15.150:6813/3775269993","state":["exists","up"]},{"osd":5,"uuid":"a2542b21-d6ac-413e-8b9b-4a6548b20840","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6809","nonce":164284137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6811","nonce":164284137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6815","nonce":164284137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":164284137},{"type":"v1","addr":"172.21.15.183:6813","nonce":164284137}]},"public_addr":"172.21.15.183:6809/164284137","cluster_addr":"172.21.15.183:6811/164284137","heartbeat_back_addr":"172.21.15.183:6815/164284137","heartbeat_front_addr":"172.21.15.183:6813/164284137","state":["exists","up"]},{"osd":6,"uuid":"8905a2d3-0f1b-4853-a218-22da2635150a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6819","nonce":3565528013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6821","nonce":3565528013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6825","nonce":3565528013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3565528013},{"type":"v1","addr":"172.21.15.67:6823","nonce":3565528013}]},"public_addr":"172.21.15.67:6819/3565528013","cluster_addr":"172.21.15.67:6821/3565528013","heartbeat_back_addr":"172.21.15.67:6825/3565528013","heartbeat_front_addr":"172.21.15.67:6823/3565528013","state":["exists","up"]},{"osd":7,"uuid":"b275055b-b451-4180-91b1-e6e61a828c48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6817","nonce":3735115340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6819","nonce":3735115340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6823","nonce":3735115340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3735115340},{"type":"v1","addr":"172.21.15.150:6821","nonce":3735115340}]},"public_addr":"172.21.15.150:6817/3735115340","cluster_addr":"172.21.15.150:6819/3735115340","heartbeat_back_addr":"172.21.15.150:6823/3735115340","heartbeat_front_addr":"172.21.15.150:6821/3735115340","state":["exists","up"]},{"osd":8,"uuid":"b6abf7ad-eb34-4f60-a4eb-348a92d7195b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6817","nonce":1025400095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6819","nonce":1025400095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6823","nonce":1025400095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":1025400095},{"type":"v1","addr":"172.21.15.183:6821","nonce":1025400095}]},"public_addr":"172.21.15.183:6817/1025400095","cluster_addr":"172.21.15.183:6819/1025400095","heartbeat_back_addr":"172.21.15.183:6823/1025400095","heartbeat_front_addr":"172.21.15.183:6821/1025400095","state":["exists","up"]},{"osd":9,"uuid":"28a78e3a-0657-40aa-816f-16a6fadadcbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6827","nonce":76885409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6829","nonce":76885409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6833","nonce":76885409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":76885409},{"type":"v1","addr":"172.21.15.67:6831","nonce":76885409}]},"public_addr":"172.21.15.67:6827/76885409","cluster_addr":"172.21.15.67:6829/76885409","heartbeat_back_addr":"172.21.15.67:6833/76885409","heartbeat_front_addr":"172.21.15.67:6831/76885409","state":["exists","up"]},{"osd":10,"uuid":"d287907e-424f-4b49-a06e-4613c6de0474","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6825","nonce":3489982080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6827","nonce":3489982080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6831","nonce":3489982080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3489982080},{"type":"v1","addr":"172.21.15.150:6829","nonce":3489982080}]},"public_addr":"172.21.15.150:6825/3489982080","cluster_addr":"172.21.15.150:6827/3489982080","heartbeat_back_addr":"172.21.15.150:6831/3489982080","heartbeat_front_addr":"172.21.15.150:6829/3489982080","state":["exists","up"]},{"osd":11,"uuid":"a3c29c47-a228-4f42-a23c-fbdb0c093699","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":85,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6825","nonce":1892639849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6826","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6827","nonce":1892639849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6831","nonce":1892639849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":1892639849},{"type":"v1","addr":"172.21.15.183:6829","nonce":1892639849}]},"public_addr":"172.21.15.183:6825/1892639849","cluster_addr":"172.21.15.183:6827/1892639849","heartbeat_back_addr":"172.21.15.183:6831/1892639849","heartbeat_front_addr":"172.21.15.183:6829/1892639849","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:53:45.646211+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:54:18.956590+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:54:52.161646+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:55:27.502822+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:56:01.162932+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:56:34.674452+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:57:09.298679+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:57:43.230406+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:58:17.003821+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:58:51.085863+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:59:25.665065+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T16:00:00.263457+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":1}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6801/1631001968":"2024-03-29T15:51:38.875334+0000","172.21.15.67:6800/1631001968":"2024-03-29T15:51:38.875334+0000","172.21.15.67:0/1533056310":"2024-03-29T15:51:38.875334+0000","172.21.15.67:6801/3580491071":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/2030412458":"2024-03-29T15:51:03.859740+0000","172.21.15.67:6800/3580491071":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/2905307533":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/3763759988":"2024-03-29T15:51:38.875334+0000","172.21.15.67:0/4089399880":"2024-03-29T15:51:03.859740+0000","172.21.15.67:0/4239108661":"2024-03-29T15:51:38.875334+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T16:09:54.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.308+0000 7fee3be00640 1 -- 172.21.15.67:0/2126176533 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fee200a2880 msgr2=0x7fee200a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:54.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.308+0000 7fee3be00640 1 --2- 172.21.15.67:0/2126176533 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fee200a2880 0x7fee200a4d40 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fee24000c00 tx=0x7fee2402c7e0 comp rx=0 tx=0).stop 2024-03-28T16:09:54.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.308+0000 7fee3be00640 1 -- 172.21.15.67:0/2126176533 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee3c1036f0 msgr2=0x7fee3c1998c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:54.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.308+0000 7fee3be00640 1 --2- 172.21.15.67:0/2126176533 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee3c1036f0 0x7fee3c1998c0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fee2c00d950 tx=0x7fee2c00de10 comp rx=0 tx=0).stop 2024-03-28T16:09:54.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.308+0000 7fee3be00640 1 -- 172.21.15.67:0/2126176533 shutdown_connections 2024-03-28T16:09:54.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.308+0000 7fee3be00640 1 --2- 172.21.15.67:0/2126176533 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fee200a2880 0x7fee200a4d40 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:54.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.308+0000 7fee3be00640 1 --2- 172.21.15.67:0/2126176533 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fee3c1052a0 0x7fee3c1a0e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:54.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.308+0000 7fee3be00640 1 --2- 172.21.15.67:0/2126176533 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee3c1048e0 0x7fee3c199e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:54.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.308+0000 7fee3be00640 1 --2- 172.21.15.67:0/2126176533 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee3c1036f0 0x7fee3c1998c0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:54.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.308+0000 7fee3be00640 1 -- 172.21.15.67:0/2126176533 >> 172.21.15.67:0/2126176533 conn(0x7fee3c0fec80 msgr2=0x7fee3c107be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:54.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.308+0000 7fee3be00640 1 -- 172.21.15.67:0/2126176533 shutdown_connections 2024-03-28T16:09:54.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.308+0000 7fee3be00640 1 -- 172.21.15.67:0/2126176533 wait complete. 2024-03-28T16:09:54.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:53 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/20967011' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:09:54.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:53 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/1216521833' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:09:54.329 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-28T16:09:54.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:53 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/20967011' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:09:54.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:53 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/1216521833' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:09:54.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.547+0000 7fae76800640 1 -- 172.21.15.67:0/3677965973 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae70103150 msgr2=0x7fae701099e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:54.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.547+0000 7fae76800640 1 --2- 172.21.15.67:0/3677965973 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae70103150 0x7fae701099e0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fae68009ca0 tx=0x7fae6802c0f0 comp rx=0 tx=0).stop 2024-03-28T16:09:54.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.547+0000 7fae76800640 1 -- 172.21.15.67:0/3677965973 shutdown_connections 2024-03-28T16:09:54.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.547+0000 7fae76800640 1 --2- 172.21.15.67:0/3677965973 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae70103150 0x7fae701099e0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:54.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.547+0000 7fae76800640 1 --2- 172.21.15.67:0/3677965973 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fae70102790 0x7fae70102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:54.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.547+0000 7fae76800640 1 --2- 172.21.15.67:0/3677965973 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae701015a0 0x7fae701019c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:54.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.547+0000 7fae76800640 1 -- 172.21.15.67:0/3677965973 >> 172.21.15.67:0/3677965973 conn(0x7fae700fcb10 msgr2=0x7fae700fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:54.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.547+0000 7fae76800640 1 -- 172.21.15.67:0/3677965973 shutdown_connections 2024-03-28T16:09:54.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.547+0000 7fae76800640 1 -- 172.21.15.67:0/3677965973 wait complete. 2024-03-28T16:09:54.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.548+0000 7fae76800640 1 Processor -- start 2024-03-28T16:09:54.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.548+0000 7fae76800640 1 -- start start 2024-03-28T16:09:54.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.548+0000 7fae76800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae701015a0 0x7fae70075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:54.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.548+0000 7fae76800640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae70102790 0x7fae70073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:54.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.549+0000 7fae76800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fae70103150 0x7fae70073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:54.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.549+0000 7fae76800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae7010d0e0 con 0x7fae701015a0 2024-03-28T16:09:54.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.549+0000 7fae76800640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae7010cf60 con 0x7fae70102790 2024-03-28T16:09:54.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.549+0000 7fae76800640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae7010d260 con 0x7fae70103150 2024-03-28T16:09:54.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.549+0000 7fae75400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae701015a0 0x7fae70075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:54.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.549+0000 7fae75400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae701015a0 0x7fae70075450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46550/0 (socket says 172.21.15.67:46550) 2024-03-28T16:09:54.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.549+0000 7fae75400640 1 -- 172.21.15.67:0/1539647104 learned_addr learned my addr 172.21.15.67:0/1539647104 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:09:54.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.549+0000 7fae74a00640 1 --2- 172.21.15.67:0/1539647104 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae70102790 0x7fae70073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:54.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.549+0000 7fae75e00640 1 --2- 172.21.15.67:0/1539647104 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fae70103150 0x7fae70073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:54.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.549+0000 7fae75400640 1 -- 172.21.15.67:0/1539647104 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fae70103150 msgr2=0x7fae70073fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:54.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.549+0000 7fae75400640 1 --2- 172.21.15.67:0/1539647104 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fae70103150 0x7fae70073fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:54.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.549+0000 7fae75400640 1 -- 172.21.15.67:0/1539647104 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae70102790 msgr2=0x7fae70073a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:54.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.549+0000 7fae75400640 1 --2- 172.21.15.67:0/1539647104 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae70102790 0x7fae70073a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:54.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.549+0000 7fae75400640 1 -- 172.21.15.67:0/1539647104 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae70074860 con 0x7fae701015a0 2024-03-28T16:09:54.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.549+0000 7fae74a00640 1 --2- 172.21.15.67:0/1539647104 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae70102790 0x7fae70073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T16:09:54.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.550+0000 7fae75400640 1 --2- 172.21.15.67:0/1539647104 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae701015a0 0x7fae70075450 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fae6400d770 tx=0x7fae6400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:54.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.550+0000 7fae6e000640 1 -- 172.21.15.67:0/1539647104 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae64017070 con 0x7fae701015a0 2024-03-28T16:09:54.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.550+0000 7fae76800640 1 -- 172.21.15.67:0/1539647104 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae701a1850 con 0x7fae701015a0 2024-03-28T16:09:54.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.550+0000 7fae6e000640 1 -- 172.21.15.67:0/1539647104 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fae64009c30 con 0x7fae701015a0 2024-03-28T16:09:54.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.550+0000 7fae6e000640 1 -- 172.21.15.67:0/1539647104 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae64003d00 con 0x7fae701015a0 2024-03-28T16:09:54.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.550+0000 7fae76800640 1 -- 172.21.15.67:0/1539647104 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae701a1da0 con 0x7fae701015a0 2024-03-28T16:09:54.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.552+0000 7fae76800640 1 -- 172.21.15.67:0/1539647104 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae701088c0 con 0x7fae701015a0 2024-03-28T16:09:54.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.553+0000 7fae6e000640 1 -- 172.21.15.67:0/1539647104 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fae64003ea0 con 0x7fae701015a0 2024-03-28T16:09:54.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.555+0000 7fae6e000640 1 --2- 172.21.15.67:0/1539647104 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fae580a2800 0x7fae580a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:54.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.555+0000 7fae6e000640 1 -- 172.21.15.67:0/1539647104 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fae640ce890 con 0x7fae701015a0 2024-03-28T16:09:54.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.555+0000 7fae74a00640 1 --2- 172.21.15.67:0/1539647104 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fae580a2800 0x7fae580a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:54.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.556+0000 7fae74a00640 1 --2- 172.21.15.67:0/1539647104 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fae580a2800 0x7fae580a4cc0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fae5c0058c0 tx=0x7fae5c005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:54.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.559+0000 7fae6e000640 1 -- 172.21.15.67:0/1539647104 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fae6409b8e0 con 0x7fae701015a0 2024-03-28T16:09:54.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.818+0000 7fae76800640 1 -- 172.21.15.67:0/1539647104 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fae700622a0 con 0x7fae701015a0 2024-03-28T16:09:54.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.823+0000 7fae6e000640 1 -- 172.21.15.67:0/1539647104 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v20) v1 ==== 161+0+0 (secure 0 0 0) 0x7fae64009de0 con 0x7fae701015a0 2024-03-28T16:09:54.824 INFO:teuthology.orchestra.run.smithi067.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-28T16:09:54.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.826+0000 7fae76800640 1 -- 172.21.15.67:0/1539647104 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fae580a2800 msgr2=0x7fae580a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:54.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.826+0000 7fae76800640 1 --2- 172.21.15.67:0/1539647104 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fae580a2800 0x7fae580a4cc0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fae5c0058c0 tx=0x7fae5c005850 comp rx=0 tx=0).stop 2024-03-28T16:09:54.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.826+0000 7fae76800640 1 -- 172.21.15.67:0/1539647104 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae701015a0 msgr2=0x7fae70075450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:54.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.826+0000 7fae76800640 1 --2- 172.21.15.67:0/1539647104 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae701015a0 0x7fae70075450 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fae6400d770 tx=0x7fae6400dc30 comp rx=0 tx=0).stop 2024-03-28T16:09:54.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.827+0000 7fae76800640 1 -- 172.21.15.67:0/1539647104 shutdown_connections 2024-03-28T16:09:54.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.827+0000 7fae76800640 1 --2- 172.21.15.67:0/1539647104 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fae580a2800 0x7fae580a4cc0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:54.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.827+0000 7fae76800640 1 --2- 172.21.15.67:0/1539647104 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fae70103150 0x7fae70073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:54.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.827+0000 7fae76800640 1 --2- 172.21.15.67:0/1539647104 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae70102790 0x7fae70073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:54.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.827+0000 7fae76800640 1 --2- 172.21.15.67:0/1539647104 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae701015a0 0x7fae70075450 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:54.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.827+0000 7fae76800640 1 -- 172.21.15.67:0/1539647104 >> 172.21.15.67:0/1539647104 conn(0x7fae700fcb10 msgr2=0x7fae70105a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:54.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.827+0000 7fae76800640 1 -- 172.21.15.67:0/1539647104 shutdown_connections 2024-03-28T16:09:54.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:54.827+0000 7fae76800640 1 -- 172.21.15.67:0/1539647104 wait complete. 2024-03-28T16:09:54.847 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-28T16:09:55.035 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:54 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mon-a[31703]: 2024-03-28T16:09:54.819+0000 7ff20c600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-28T16:09:55.036 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:54 smithi067 ceph-mon[31707]: pgmap v629: 97 pgs: 97 active+clean; 981 KiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-28T16:09:55.036 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:54 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/2126176533' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:09:55.036 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:54 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/1539647104' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-28T16:09:55.036 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:54 smithi067 ceph-mon[31707]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T16:09:55.036 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:54 smithi067 ceph-mon[31707]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-28T16:09:55.036 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:54 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/1539647104' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-28T16:09:55.036 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:54 smithi067 ceph-mon[31707]: osdmap e93: 12 total, 12 up, 12 in 2024-03-28T16:09:55.036 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:54 smithi067 ceph-mon[31707]: fsmap cephfs:0/1 10 up:standby, 1 failed 2024-03-28T16:09:55.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:55.038+0000 7f74d3200640 1 -- 172.21.15.67:0/1608373930 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f74d41049b0 msgr2=0x7f74d4104e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:55.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:55.038+0000 7f74d3200640 1 --2- 172.21.15.67:0/1608373930 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f74d41049b0 0x7f74d4104e30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f74c0009860 tx=0x7f74c002bb70 comp rx=0 tx=0).stop 2024-03-28T16:09:55.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:55.038+0000 7f74d3200640 1 -- 172.21.15.67:0/1608373930 shutdown_connections 2024-03-28T16:09:55.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:55.038+0000 7f74d3200640 1 --2- 172.21.15.67:0/1608373930 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f74d4105370 0x7f74d410bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:55.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:55.038+0000 7f74d3200640 1 --2- 172.21.15.67:0/1608373930 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f74d41049b0 0x7f74d4104e30 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:55.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:55.038+0000 7f74d3200640 1 --2- 172.21.15.67:0/1608373930 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74d41037c0 0x7f74d4103be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:55.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:55.038+0000 7f74d3200640 1 -- 172.21.15.67:0/1608373930 >> 172.21.15.67:0/1608373930 conn(0x7f74d40fed50 msgr2=0x7f74d4101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:55.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:55.038+0000 7f74d3200640 1 -- 172.21.15.67:0/1608373930 shutdown_connections 2024-03-28T16:09:55.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:55.038+0000 7f74d3200640 1 -- 172.21.15.67:0/1608373930 wait complete. 2024-03-28T16:09:55.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:55.040+0000 7f74d3200640 1 Processor -- start 2024-03-28T16:09:55.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:55.040+0000 7f74d3200640 1 -- start start 2024-03-28T16:09:55.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:55.040+0000 7f74d3200640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f74d41037c0 0x7f74d4199a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:55.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:55.040+0000 7f74d3200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f74d41049b0 0x7f74d4199fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:55.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:55.040+0000 7f74d3200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74d4105370 0x7f74d41a0f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:55.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:55.040+0000 7f74d3200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74d410f2d0 con 0x7f74d4105370 2024-03-28T16:09:55.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:55.040+0000 7f74d3200640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74d410f150 con 0x7f74d41037c0 2024-03-28T16:09:55.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:55.040+0000 7f74d3200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74d410f450 con 0x7f74d41049b0 2024-03-28T16:09:55.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:55.041+0000 7f74d2800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74d4105370 0x7f74d41a0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:55.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:55.041+0000 7f74d2800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74d4105370 0x7f74d41a0f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46572/0 (socket says 172.21.15.67:46572) 2024-03-28T16:09:55.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:55.041+0000 7f74d2800640 1 -- 172.21.15.67:0/3894573121 learned_addr learned my addr 172.21.15.67:0/3894573121 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:09:55.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:55.041+0000 7f74d1e00640 1 --2- 172.21.15.67:0/3894573121 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f74d41037c0 0x7f74d4199a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:55.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:55.041+0000 7f74d1400640 1 --2- 172.21.15.67:0/3894573121 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f74d41049b0 0x7f74d4199fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:55.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:55.041+0000 7f74d2800640 1 -- 172.21.15.67:0/3894573121 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f74d41049b0 msgr2=0x7f74d4199fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:55.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:55.041+0000 7f74d2800640 1 --2- 172.21.15.67:0/3894573121 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f74d41049b0 0x7f74d4199fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:55.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:55.041+0000 7f74d2800640 1 -- 172.21.15.67:0/3894573121 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f74d41037c0 msgr2=0x7f74d4199a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:55.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:55.041+0000 7f74d2800640 1 --2- 172.21.15.67:0/3894573121 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f74d41037c0 0x7f74d4199a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:55.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:55.041+0000 7f74d2800640 1 -- 172.21.15.67:0/3894573121 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74d41a15f0 con 0x7f74d4105370 2024-03-28T16:09:55.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:55.042+0000 7f74d2800640 1 --2- 172.21.15.67:0/3894573121 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74d4105370 0x7f74d41a0f80 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f74cc00eaa0 tx=0x7f74cc00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:55.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:55.042+0000 7f74caa00640 1 -- 172.21.15.67:0/3894573121 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74cc00cdb0 con 0x7f74d4105370 2024-03-28T16:09:55.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:55.042+0000 7f74caa00640 1 -- 172.21.15.67:0/3894573121 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f74cc005020 con 0x7f74d4105370 2024-03-28T16:09:55.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:55.042+0000 7f74d3200640 1 -- 172.21.15.67:0/3894573121 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74d41a18e0 con 0x7f74d4105370 2024-03-28T16:09:55.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:55.042+0000 7f74caa00640 1 -- 172.21.15.67:0/3894573121 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74cc0116d0 con 0x7f74d4105370 2024-03-28T16:09:55.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:55.042+0000 7f74d3200640 1 -- 172.21.15.67:0/3894573121 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74d41a1db0 con 0x7f74d4105370 2024-03-28T16:09:55.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:55.043+0000 7f74d3200640 1 -- 172.21.15.67:0/3894573121 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74d410ab00 con 0x7f74d4105370 2024-03-28T16:09:55.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:55.045+0000 7f74caa00640 1 -- 172.21.15.67:0/3894573121 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f74cc0051d0 con 0x7f74d4105370 2024-03-28T16:09:55.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:55.046+0000 7f74caa00640 1 --2- 172.21.15.67:0/3894573121 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f74b80a2800 0x7f74b80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:55.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:55.046+0000 7f74caa00640 1 -- 172.21.15.67:0/3894573121 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8305+0+0 (secure 0 0 0) 0x7f74cc0cedd0 con 0x7f74d4105370 2024-03-28T16:09:55.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:55.047+0000 7f74d1e00640 1 --2- 172.21.15.67:0/3894573121 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f74b80a2800 0x7f74b80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:55.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:55.047+0000 7f74d1e00640 1 --2- 172.21.15.67:0/3894573121 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f74b80a2800 0x7f74b80a4cc0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f74c40096c0 tx=0x7f74c4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:55.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:55.051+0000 7f74caa00640 1 -- 172.21.15.67:0/3894573121 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f74cc016030 con 0x7f74d4105370 2024-03-28T16:09:55.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:55.288+0000 7f74d3200640 1 -- 172.21.15.67:0/3894573121 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f74d40622a0 con 0x7f74d4105370 2024-03-28T16:09:55.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:54 smithi150 ceph-mon[34825]: pgmap v629: 97 pgs: 97 active+clean; 981 KiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-28T16:09:55.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:54 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/2126176533' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:09:55.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:54 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/1539647104' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-28T16:09:55.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:54 smithi150 ceph-mon[34825]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T16:09:55.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:54 smithi150 ceph-mon[34825]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-28T16:09:55.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:54 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/1539647104' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-28T16:09:55.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:54 smithi150 ceph-mon[34825]: osdmap e93: 12 total, 12 up, 12 in 2024-03-28T16:09:55.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:54 smithi150 ceph-mon[34825]: fsmap cephfs:0/1 10 up:standby, 1 failed 2024-03-28T16:09:55.303 INFO:journalctl@ceph.mds.b.smithi150.stdout:Mar 28 16:09:54 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-b[65534]: ignoring --setuser ceph since I am not root 2024-03-28T16:09:55.303 INFO:journalctl@ceph.mds.b.smithi150.stdout:Mar 28 16:09:54 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-b[65534]: ignoring --setgroup ceph since I am not root 2024-03-28T16:09:55.303 INFO:journalctl@ceph.mds.b.smithi150.stdout:Mar 28 16:09:54 smithi150 ceph-mds[65543]: -- 172.21.15.150:0/3644582318 <== mon.2 v2:172.21.15.150:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f4d5653a40 con 0x55f4d5626800 2024-03-28T16:09:55.303 INFO:journalctl@ceph.mds.b.smithi150.stdout:Mar 28 16:09:54 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-b[65534]: starting mds.b at 2024-03-28T16:09:55.304 INFO:journalctl@ceph.mds.h.smithi150.stdout:Mar 28 16:09:54 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-h[65230]: ignoring --setuser ceph since I am not root 2024-03-28T16:09:55.304 INFO:journalctl@ceph.mds.h.smithi150.stdout:Mar 28 16:09:54 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-h[65230]: ignoring --setgroup ceph since I am not root 2024-03-28T16:09:55.304 INFO:journalctl@ceph.mds.h.smithi150.stdout:Mar 28 16:09:54 smithi150 ceph-mds[65234]: -- 172.21.15.150:0/3244620357 <== mon.2 v2:172.21.15.150:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5651b02bda40 con 0x5651b0290800 2024-03-28T16:09:55.304 INFO:journalctl@ceph.mds.h.smithi150.stdout:Mar 28 16:09:54 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-h[65230]: starting mds.h at 2024-03-28T16:09:55.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:54 smithi183 ceph-mon[35580]: pgmap v629: 97 pgs: 97 active+clean; 981 KiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-28T16:09:55.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:54 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/2126176533' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:09:55.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:54 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/1539647104' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-28T16:09:55.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:54 smithi183 ceph-mon[35580]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T16:09:55.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:54 smithi183 ceph-mon[35580]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-28T16:09:55.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:54 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/1539647104' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-28T16:09:55.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:54 smithi183 ceph-mon[35580]: osdmap e93: 12 total, 12 up, 12 in 2024-03-28T16:09:55.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:54 smithi183 ceph-mon[35580]: fsmap cephfs:0/1 10 up:standby, 1 failed 2024-03-28T16:09:55.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:55.915+0000 7f74caa00640 1 -- 172.21.15.67:0/3894573121 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v21) v1 ==== 104+0+0 (secure 0 0 0) 0x7f74cc09bdc0 con 0x7f74d4105370 2024-03-28T16:09:55.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:55.919+0000 7f74d3200640 1 -- 172.21.15.67:0/3894573121 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f74b80a2800 msgr2=0x7f74b80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:55.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:55.919+0000 7f74d3200640 1 --2- 172.21.15.67:0/3894573121 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f74b80a2800 0x7f74b80a4cc0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f74c40096c0 tx=0x7f74c4009260 comp rx=0 tx=0).stop 2024-03-28T16:09:55.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:55.919+0000 7f74d3200640 1 -- 172.21.15.67:0/3894573121 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74d4105370 msgr2=0x7f74d41a0f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:55.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:55.919+0000 7f74d3200640 1 --2- 172.21.15.67:0/3894573121 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74d4105370 0x7f74d41a0f80 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f74cc00eaa0 tx=0x7f74cc00ef60 comp rx=0 tx=0).stop 2024-03-28T16:09:55.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:55.920+0000 7f74d3200640 1 -- 172.21.15.67:0/3894573121 shutdown_connections 2024-03-28T16:09:55.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:55.920+0000 7f74d3200640 1 --2- 172.21.15.67:0/3894573121 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f74b80a2800 0x7f74b80a4cc0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:55.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:55.920+0000 7f74d3200640 1 --2- 172.21.15.67:0/3894573121 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74d4105370 0x7f74d41a0f80 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:55.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:55.920+0000 7f74d3200640 1 --2- 172.21.15.67:0/3894573121 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f74d41049b0 0x7f74d4199fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:55.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:55.920+0000 7f74d3200640 1 --2- 172.21.15.67:0/3894573121 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f74d41037c0 0x7f74d4199a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:55.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:55.920+0000 7f74d3200640 1 -- 172.21.15.67:0/3894573121 >> 172.21.15.67:0/3894573121 conn(0x7f74d40fed50 msgr2=0x7f74d4107cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:55.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:55.920+0000 7f74d3200640 1 -- 172.21.15.67:0/3894573121 shutdown_connections 2024-03-28T16:09:55.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:55.920+0000 7f74d3200640 1 -- 172.21.15.67:0/3894573121 wait complete. 2024-03-28T16:09:55.944 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-28T16:09:56.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:56.152+0000 7faebda00640 1 -- 172.21.15.67:0/2901971640 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faeb8109870 msgr2=0x7faeb8110100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:56.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:56.152+0000 7faebda00640 1 --2- 172.21.15.67:0/2901971640 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faeb8109870 0x7faeb8110100 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7faeb0009ca0 tx=0x7faeb002c0f0 comp rx=0 tx=0).stop 2024-03-28T16:09:56.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:56.152+0000 7faebda00640 1 -- 172.21.15.67:0/2901971640 shutdown_connections 2024-03-28T16:09:56.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:56.152+0000 7faebda00640 1 --2- 172.21.15.67:0/2901971640 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faeb8109870 0x7faeb8110100 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:56.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:56.152+0000 7faebda00640 1 --2- 172.21.15.67:0/2901971640 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faeb8108eb0 0x7faeb8109330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:56.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:56.152+0000 7faebda00640 1 --2- 172.21.15.67:0/2901971640 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faeb8107cc0 0x7faeb81080e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:56.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:56.152+0000 7faebda00640 1 -- 172.21.15.67:0/2901971640 >> 172.21.15.67:0/2901971640 conn(0x7faeb8076c60 msgr2=0x7faeb80790a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:56.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:56.152+0000 7faebda00640 1 -- 172.21.15.67:0/2901971640 shutdown_connections 2024-03-28T16:09:56.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:56.152+0000 7faebda00640 1 -- 172.21.15.67:0/2901971640 wait complete. 2024-03-28T16:09:56.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:56.153+0000 7faebda00640 1 Processor -- start 2024-03-28T16:09:56.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:56.153+0000 7faebda00640 1 -- start start 2024-03-28T16:09:56.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:56.154+0000 7faebda00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faeb8107cc0 0x7faeb819dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:56.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:56.154+0000 7faebda00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faeb8108eb0 0x7faeb819e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:56.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:56.154+0000 7faebda00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faeb8109870 0x7faeb81a5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:56.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:56.154+0000 7faebda00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeb81137f0 con 0x7faeb8109870 2024-03-28T16:09:56.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:56.154+0000 7faebda00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeb8113670 con 0x7faeb8107cc0 2024-03-28T16:09:56.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:56.154+0000 7faebda00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeb8113970 con 0x7faeb8108eb0 2024-03-28T16:09:56.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:56.154+0000 7faeb7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faeb8109870 0x7faeb81a5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:56.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:56.154+0000 7faeb7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faeb8109870 0x7faeb81a5450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46584/0 (socket says 172.21.15.67:46584) 2024-03-28T16:09:56.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:56.154+0000 7faeb7400640 1 -- 172.21.15.67:0/4242966212 learned_addr learned my addr 172.21.15.67:0/4242966212 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:09:56.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:56.154+0000 7faeb6a00640 1 --2- 172.21.15.67:0/4242966212 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faeb8107cc0 0x7faeb819dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:56.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:56.154+0000 7faeb7400640 1 -- 172.21.15.67:0/4242966212 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faeb8108eb0 msgr2=0x7faeb819e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:56.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:56.154+0000 7faeae000640 1 --2- 172.21.15.67:0/4242966212 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faeb8108eb0 0x7faeb819e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:56.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:56.154+0000 7faeb7400640 1 --2- 172.21.15.67:0/4242966212 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faeb8108eb0 0x7faeb819e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:56.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:56.155+0000 7faeb7400640 1 -- 172.21.15.67:0/4242966212 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faeb8107cc0 msgr2=0x7faeb819dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:56.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:56.155+0000 7faeb7400640 1 --2- 172.21.15.67:0/4242966212 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faeb8107cc0 0x7faeb819dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:56.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:56.155+0000 7faeb7400640 1 -- 172.21.15.67:0/4242966212 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeb81a5ac0 con 0x7faeb8109870 2024-03-28T16:09:56.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:56.155+0000 7faeb7400640 1 --2- 172.21.15.67:0/4242966212 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faeb8109870 0x7faeb81a5450 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7faeb002c700 tx=0x7faeb00328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:56.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:56.155+0000 7faeafe00640 1 -- 172.21.15.67:0/4242966212 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb0041020 con 0x7faeb8109870 2024-03-28T16:09:56.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:56.155+0000 7faebda00640 1 -- 172.21.15.67:0/4242966212 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faeb81a5d50 con 0x7faeb8109870 2024-03-28T16:09:56.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:56.155+0000 7faeafe00640 1 -- 172.21.15.67:0/4242966212 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7faeb0003e20 con 0x7faeb8109870 2024-03-28T16:09:56.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:56.155+0000 7faeafe00640 1 -- 172.21.15.67:0/4242966212 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb0039040 con 0x7faeb8109870 2024-03-28T16:09:56.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:56.155+0000 7faebda00640 1 -- 172.21.15.67:0/4242966212 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faeb81a61e0 con 0x7faeb8109870 2024-03-28T16:09:56.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:56.156+0000 7faebda00640 1 -- 172.21.15.67:0/4242966212 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faeb810efe0 con 0x7faeb8109870 2024-03-28T16:09:56.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:56.158+0000 7faeafe00640 1 -- 172.21.15.67:0/4242966212 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7faeb0044020 con 0x7faeb8109870 2024-03-28T16:09:56.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:56.159+0000 7faeafe00640 1 --2- 172.21.15.67:0/4242966212 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fae9c0a2800 0x7fae9c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:56.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:56.160+0000 7faeafe00640 1 -- 172.21.15.67:0/4242966212 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8305+0+0 (secure 0 0 0) 0x7faeb00f0ef0 con 0x7faeb8109870 2024-03-28T16:09:56.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:56.160+0000 7faeb6a00640 1 --2- 172.21.15.67:0/4242966212 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fae9c0a2800 0x7fae9c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:56.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:56.160+0000 7faeb6a00640 1 --2- 172.21.15.67:0/4242966212 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fae9c0a2800 0x7fae9c0a4cc0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7faea8005fd0 tx=0x7faea8005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:56.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:56.164+0000 7faeafe00640 1 -- 172.21.15.67:0/4242966212 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7faeb004b020 con 0x7faeb8109870 2024-03-28T16:09:56.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:55 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/3894573121' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-28T16:09:56.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:55 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/3894573121' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-28T16:09:56.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:55 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/3894573121' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-28T16:09:56.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:56.374+0000 7faebda00640 1 -- 172.21.15.67:0/4242966212 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7faeb8062340 con 0x7faeb8109870 2024-03-28T16:09:56.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:56.929+0000 7faeafe00640 1 -- 172.21.15.67:0/4242966212 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v94) v1 ==== 181+0+0 (secure 0 0 0) 0x7faeb003b2f0 con 0x7faeb8109870 2024-03-28T16:09:56.930 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' removed 2024-03-28T16:09:56.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:56.933+0000 7faebda00640 1 -- 172.21.15.67:0/4242966212 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fae9c0a2800 msgr2=0x7fae9c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:56.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:56.933+0000 7faebda00640 1 --2- 172.21.15.67:0/4242966212 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fae9c0a2800 0x7fae9c0a4cc0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7faea8005fd0 tx=0x7faea8005850 comp rx=0 tx=0).stop 2024-03-28T16:09:56.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:56.933+0000 7faebda00640 1 -- 172.21.15.67:0/4242966212 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faeb8109870 msgr2=0x7faeb81a5450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:56.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:56.933+0000 7faebda00640 1 --2- 172.21.15.67:0/4242966212 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faeb8109870 0x7faeb81a5450 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7faeb002c700 tx=0x7faeb00328a0 comp rx=0 tx=0).stop 2024-03-28T16:09:56.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:56.934+0000 7faebda00640 1 -- 172.21.15.67:0/4242966212 shutdown_connections 2024-03-28T16:09:56.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:56.934+0000 7faebda00640 1 --2- 172.21.15.67:0/4242966212 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7fae9c0a2800 0x7fae9c0a4cc0 secure :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7faea8005fd0 tx=0x7faea8005850 comp rx=0 tx=0).stop 2024-03-28T16:09:56.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:56.934+0000 7faebda00640 1 --2- 172.21.15.67:0/4242966212 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faeb8109870 0x7faeb81a5450 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:56.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:56.934+0000 7faebda00640 1 --2- 172.21.15.67:0/4242966212 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faeb8108eb0 0x7faeb819e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:56.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:56.934+0000 7faebda00640 1 --2- 172.21.15.67:0/4242966212 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faeb8107cc0 0x7faeb819dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:56.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:56.934+0000 7faebda00640 1 -- 172.21.15.67:0/4242966212 >> 172.21.15.67:0/4242966212 conn(0x7faeb8076c60 msgr2=0x7faeb810c1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:56.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:56.936+0000 7faebda00640 1 -- 172.21.15.67:0/4242966212 shutdown_connections 2024-03-28T16:09:56.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:56.936+0000 7faebda00640 1 -- 172.21.15.67:0/4242966212 wait complete. 2024-03-28T16:09:56.964 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-28T16:09:57.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:57.185+0000 7f8d6fe00640 1 -- 172.21.15.67:0/2697117960 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d701007b0 msgr2=0x7f8d70100c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:57.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:57.185+0000 7f8d6fe00640 1 --2- 172.21.15.67:0/2697117960 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d701007b0 0x7f8d70100c10 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f8d58009900 tx=0x7f8d5802bc50 comp rx=0 tx=0).stop 2024-03-28T16:09:57.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:57.185+0000 7f8d6fe00640 1 -- 172.21.15.67:0/2697117960 shutdown_connections 2024-03-28T16:09:57.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:57.185+0000 7f8d6fe00640 1 --2- 172.21.15.67:0/2697117960 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8d70101150 0x7f8d701015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:57.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:57.185+0000 7f8d6fe00640 1 --2- 172.21.15.67:0/2697117960 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d701007b0 0x7f8d70100c10 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:57.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:57.185+0000 7f8d6fe00640 1 --2- 172.21.15.67:0/2697117960 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d700ffe50 0x7f8d70100270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:57.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:57.185+0000 7f8d6fe00640 1 -- 172.21.15.67:0/2697117960 >> 172.21.15.67:0/2697117960 conn(0x7f8d700fba10 msgr2=0x7f8d700fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:57.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:57.185+0000 7f8d6fe00640 1 -- 172.21.15.67:0/2697117960 shutdown_connections 2024-03-28T16:09:57.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:57.185+0000 7f8d6fe00640 1 -- 172.21.15.67:0/2697117960 wait complete. 2024-03-28T16:09:57.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:57.186+0000 7f8d6fe00640 1 Processor -- start 2024-03-28T16:09:57.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:57.186+0000 7f8d6fe00640 1 -- start start 2024-03-28T16:09:57.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:57.187+0000 7f8d6fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d700ffe50 0x7f8d70199910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:57.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:57.187+0000 7f8d6fe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8d701007b0 0x7f8d70199e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:57.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:57.187+0000 7f8d6fe00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d70101150 0x7f8d701a0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:57.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:57.187+0000 7f8d6fe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d7010f2f0 con 0x7f8d700ffe50 2024-03-28T16:09:57.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:57.187+0000 7f8d6fe00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d7010f170 con 0x7f8d70101150 2024-03-28T16:09:57.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:57.187+0000 7f8d6fe00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d7010f470 con 0x7f8d701007b0 2024-03-28T16:09:57.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:57.187+0000 7f8d6ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d700ffe50 0x7f8d70199910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:57.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:57.187+0000 7f8d6ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d700ffe50 0x7f8d70199910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46616/0 (socket says 172.21.15.67:46616) 2024-03-28T16:09:57.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:57.187+0000 7f8d6ea00640 1 -- 172.21.15.67:0/3290883328 learned_addr learned my addr 172.21.15.67:0/3290883328 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-03-28T16:09:57.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:57.187+0000 7f8d6e000640 1 --2- 172.21.15.67:0/3290883328 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8d701007b0 0x7f8d70199e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:57.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:57.187+0000 7f8d6f400640 1 --2- 172.21.15.67:0/3290883328 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d70101150 0x7f8d701a0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:57.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:57.187+0000 7f8d6ea00640 1 -- 172.21.15.67:0/3290883328 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8d701007b0 msgr2=0x7f8d70199e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:57.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:57.187+0000 7f8d6ea00640 1 --2- 172.21.15.67:0/3290883328 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8d701007b0 0x7f8d70199e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:57.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:57.187+0000 7f8d6ea00640 1 -- 172.21.15.67:0/3290883328 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d70101150 msgr2=0x7f8d701a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:57.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:57.187+0000 7f8d6ea00640 1 --2- 172.21.15.67:0/3290883328 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d70101150 0x7f8d701a0ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:57.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:57.187+0000 7f8d6ea00640 1 -- 172.21.15.67:0/3290883328 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d701a1530 con 0x7f8d700ffe50 2024-03-28T16:09:57.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:57.188+0000 7f8d6ea00640 1 --2- 172.21.15.67:0/3290883328 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d700ffe50 0x7f8d70199910 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f8d6000cce0 tx=0x7f8d6000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:57.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:57.188+0000 7f8d67400640 1 -- 172.21.15.67:0/3290883328 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d6000f650 con 0x7f8d700ffe50 2024-03-28T16:09:57.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:57.188+0000 7f8d67400640 1 -- 172.21.15.67:0/3290883328 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8d6000fca0 con 0x7f8d700ffe50 2024-03-28T16:09:57.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:57.188+0000 7f8d6fe00640 1 -- 172.21.15.67:0/3290883328 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d701a1820 con 0x7f8d700ffe50 2024-03-28T16:09:57.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:57.188+0000 7f8d67400640 1 -- 172.21.15.67:0/3290883328 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d60018b90 con 0x7f8d700ffe50 2024-03-28T16:09:57.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:57.188+0000 7f8d6fe00640 1 -- 172.21.15.67:0/3290883328 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d701a1d70 con 0x7f8d700ffe50 2024-03-28T16:09:57.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:57.190+0000 7f8d6fe00640 1 -- 172.21.15.67:0/3290883328 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d7010acf0 con 0x7f8d700ffe50 2024-03-28T16:09:57.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:57.192+0000 7f8d67400640 1 -- 172.21.15.67:0/3290883328 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f8d6000f7f0 con 0x7f8d700ffe50 2024-03-28T16:09:57.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:57.193+0000 7f8d67400640 1 --2- 172.21.15.67:0/3290883328 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f8d540a2800 0x7f8d540a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:09:57.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:57.193+0000 7f8d67400640 1 -- 172.21.15.67:0/3290883328 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 7772+0+0 (secure 0 0 0) 0x7f8d600cee70 con 0x7f8d700ffe50 2024-03-28T16:09:57.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:57.193+0000 7f8d6e000640 1 --2- 172.21.15.67:0/3290883328 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f8d540a2800 0x7f8d540a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:09:57.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:57.194+0000 7f8d6e000640 1 --2- 172.21.15.67:0/3290883328 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f8d540a2800 0x7f8d540a4cc0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f8d580096c0 tx=0x7f8d58005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:09:57.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:57.197+0000 7f8d67400640 1 -- 172.21.15.67:0/3290883328 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8d60014030 con 0x7f8d700ffe50 2024-03-28T16:09:57.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:56 smithi150 ceph-mon[34825]: pgmap v631: 97 pgs: 97 active+clean; 981 KiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 307 B/s wr, 2 op/s 2024-03-28T16:09:57.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:56 smithi150 ceph-mon[34825]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T16:09:57.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:56 smithi150 ceph-mon[34825]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-28T16:09:57.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:56 smithi150 ceph-mon[34825]: Cluster is now healthy 2024-03-28T16:09:57.304 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:56 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/3894573121' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-28T16:09:57.304 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:56 smithi150 ceph-mon[34825]: mds.? [v2:172.21.15.150:6836/3592899544,v1:172.21.15.150:6837/3592899544] up:boot 2024-03-28T16:09:57.304 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:56 smithi150 ceph-mon[34825]: mds.? [v2:172.21.15.150:6838/762742206,v1:172.21.15.150:6839/762742206] up:boot 2024-03-28T16:09:57.304 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:56 smithi150 ceph-mon[34825]: fsmap 12 up:standby 2024-03-28T16:09:57.304 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:56 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T16:09:57.304 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:56 smithi150 ceph-mon[34825]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T16:09:57.304 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:56 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/4242966212' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T16:09:57.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:56 smithi067 ceph-mon[31707]: pgmap v631: 97 pgs: 97 active+clean; 981 KiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 307 B/s wr, 2 op/s 2024-03-28T16:09:57.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:56 smithi067 ceph-mon[31707]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T16:09:57.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:56 smithi067 ceph-mon[31707]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-28T16:09:57.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:56 smithi067 ceph-mon[31707]: Cluster is now healthy 2024-03-28T16:09:57.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:56 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/3894573121' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-28T16:09:57.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:56 smithi067 ceph-mon[31707]: mds.? [v2:172.21.15.150:6836/3592899544,v1:172.21.15.150:6837/3592899544] up:boot 2024-03-28T16:09:57.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:56 smithi067 ceph-mon[31707]: mds.? [v2:172.21.15.150:6838/762742206,v1:172.21.15.150:6839/762742206] up:boot 2024-03-28T16:09:57.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:56 smithi067 ceph-mon[31707]: fsmap 12 up:standby 2024-03-28T16:09:57.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:56 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T16:09:57.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:56 smithi067 ceph-mon[31707]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T16:09:57.321 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:56 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/4242966212' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T16:09:57.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:56 smithi183 ceph-mon[35580]: pgmap v631: 97 pgs: 97 active+clean; 981 KiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 307 B/s wr, 2 op/s 2024-03-28T16:09:57.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:56 smithi183 ceph-mon[35580]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T16:09:57.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:56 smithi183 ceph-mon[35580]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-28T16:09:57.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:56 smithi183 ceph-mon[35580]: Cluster is now healthy 2024-03-28T16:09:57.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:56 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/3894573121' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-28T16:09:57.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:56 smithi183 ceph-mon[35580]: mds.? [v2:172.21.15.150:6836/3592899544,v1:172.21.15.150:6837/3592899544] up:boot 2024-03-28T16:09:57.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:56 smithi183 ceph-mon[35580]: mds.? [v2:172.21.15.150:6838/762742206,v1:172.21.15.150:6839/762742206] up:boot 2024-03-28T16:09:57.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:56 smithi183 ceph-mon[35580]: fsmap 12 up:standby 2024-03-28T16:09:57.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:56 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T16:09:57.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:56 smithi183 ceph-mon[35580]: from='mgr.14152 172.21.15.67:0/3091822087' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T16:09:57.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:56 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/4242966212' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T16:09:57.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:57.408+0000 7f8d6fe00640 1 -- 172.21.15.67:0/3290883328 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f8d7002d070 con 0x7f8d700ffe50 2024-03-28T16:09:57.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:57.958+0000 7f8d67400640 1 -- 172.21.15.67:0/3290883328 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v95) v1 ==== 169+0+0 (secure 0 0 0) 0x7f8d6009c090 con 0x7f8d700ffe50 2024-03-28T16:09:57.960 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' removed 2024-03-28T16:09:57.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:57.962+0000 7f8d6fe00640 1 -- 172.21.15.67:0/3290883328 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f8d540a2800 msgr2=0x7f8d540a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:57.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:57.962+0000 7f8d6fe00640 1 --2- 172.21.15.67:0/3290883328 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f8d540a2800 0x7f8d540a4cc0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f8d580096c0 tx=0x7f8d58005fd0 comp rx=0 tx=0).stop 2024-03-28T16:09:57.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:57.962+0000 7f8d6fe00640 1 -- 172.21.15.67:0/3290883328 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d700ffe50 msgr2=0x7f8d70199910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:09:57.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:57.962+0000 7f8d6fe00640 1 --2- 172.21.15.67:0/3290883328 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d700ffe50 0x7f8d70199910 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f8d6000cce0 tx=0x7f8d6000a620 comp rx=0 tx=0).stop 2024-03-28T16:09:57.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:57.963+0000 7f8d6fe00640 1 -- 172.21.15.67:0/3290883328 shutdown_connections 2024-03-28T16:09:57.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:57.963+0000 7f8d6fe00640 1 --2- 172.21.15.67:0/3290883328 >> [v2:172.21.15.67:6800/208726654,v1:172.21.15.67:6801/208726654] conn(0x7f8d540a2800 0x7f8d540a4cc0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:57.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:57.963+0000 7f8d6fe00640 1 --2- 172.21.15.67:0/3290883328 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d70101150 0x7f8d701a0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:57.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:57.963+0000 7f8d6fe00640 1 --2- 172.21.15.67:0/3290883328 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8d701007b0 0x7f8d70199e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:57.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:57.963+0000 7f8d6fe00640 1 --2- 172.21.15.67:0/3290883328 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d700ffe50 0x7f8d70199910 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:09:57.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:57.963+0000 7f8d6fe00640 1 -- 172.21.15.67:0/3290883328 >> 172.21.15.67:0/3290883328 conn(0x7f8d700fba10 msgr2=0x7f8d700fd0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:09:57.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:57.965+0000 7f8d6fe00640 1 -- 172.21.15.67:0/3290883328 shutdown_connections 2024-03-28T16:09:57.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-28T16:09:57.966+0000 7f8d6fe00640 1 -- 172.21.15.67:0/3290883328 wait complete. 2024-03-28T16:09:57.999 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-28T16:09:58.075 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-28T16:09:58.178 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-28T16:09:58.251 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:57 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/4242966212' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-28T16:09:58.252 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:57 smithi183 ceph-mon[35580]: osdmap e94: 12 total, 12 up, 12 in 2024-03-28T16:09:58.252 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:57 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/3290883328' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T16:09:58.280 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-28T16:09:58.280 DEBUG:teuthology.orchestra.run.smithi067:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-28T16:09:58.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:57 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/4242966212' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-28T16:09:58.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:57 smithi150 ceph-mon[34825]: osdmap e94: 12 total, 12 up, 12 in 2024-03-28T16:09:58.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:57 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/3290883328' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T16:09:58.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:57 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/4242966212' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-28T16:09:58.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:57 smithi067 ceph-mon[31707]: osdmap e94: 12 total, 12 up, 12 in 2024-03-28T16:09:58.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:57 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/3290883328' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T16:09:58.341 DEBUG:teuthology.orchestra.run.smithi150:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-28T16:09:58.397 DEBUG:teuthology.orchestra.run.smithi183:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-28T16:09:58.458 INFO:tasks.cephadm:Stopping all daemons... 2024-03-28T16:09:58.459 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-28T16:09:58.459 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@mon.a 2024-03-28T16:09:58.820 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:58 smithi067 systemd[1]: Stopping Ceph mon.a for b5f8225a-ed1a-11ee-95d0-87774f69a715... 2024-03-28T16:09:59.302 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:58 smithi150 ceph-mon[34825]: pgmap v633: 65 pgs: 65 active+clean; 577 KiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:09:59.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:58 smithi150 ceph-mon[34825]: from='client.? 172.21.15.67:0/3290883328' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-28T16:09:59.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:09:58 smithi150 ceph-mon[34825]: osdmap e95: 12 total, 12 up, 12 in 2024-03-28T16:09:59.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:58 smithi067 ceph-mon[31707]: pgmap v633: 65 pgs: 65 active+clean; 577 KiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:09:59.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:58 smithi067 ceph-mon[31707]: from='client.? 172.21.15.67:0/3290883328' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-28T16:09:59.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:58 smithi067 ceph-mon[31707]: osdmap e95: 12 total, 12 up, 12 in 2024-03-28T16:09:59.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:59 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mon-a[31703]: 2024-03-28T16:09:59.065+0000 7ff213400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T16:09:59.320 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:59 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mon-a[31703]: 2024-03-28T16:09:59.065+0000 7ff213400700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-28T16:09:59.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:58 smithi183 ceph-mon[35580]: pgmap v633: 65 pgs: 65 active+clean; 577 KiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:09:59.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:58 smithi183 ceph-mon[35580]: from='client.? 172.21.15.67:0/3290883328' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-28T16:09:59.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:09:58 smithi183 ceph-mon[35580]: osdmap e95: 12 total, 12 up, 12 in 2024-03-28T16:09:59.664 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:59 smithi067 podman[90815]: 2024-03-28 16:09:59.347035395 +0000 UTC m=+0.434740057 container died ecc4fb80b575f73bbc7f18649c2d8c13d6756758b2184d98b2c5a8a68d64a63f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mon-a, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-28T16:10:00.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:59 smithi067 podman[90815]: 2024-03-28 16:09:59.867002635 +0000 UTC m=+0.954707271 container cleanup ecc4fb80b575f73bbc7f18649c2d8c13d6756758b2184d98b2c5a8a68d64a63f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mon-a, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T16:10:00.070 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:09:59 smithi067 bash[90815]: ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mon-a 2024-03-28T16:10:00.570 INFO:journalctl@ceph.mon.a.smithi067.stdout:Mar 28 16:10:00 smithi067 podman[90829]: 2024-03-28 16:10:00.134984392 +0000 UTC m=+0.765504973 container remove ecc4fb80b575f73bbc7f18649c2d8c13d6756758b2184d98b2c5a8a68d64a63f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mon-a, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-28T16:10:00.659 DEBUG:teuthology.orchestra.run.smithi067:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@mon.a.service' 2024-03-28T16:10:00.810 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T16:10:00.810 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-28T16:10:00.811 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-28T16:10:00.811 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@mon.b 2024-03-28T16:10:01.303 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:10:00 smithi150 systemd[1]: Stopping Ceph mon.b for b5f8225a-ed1a-11ee-95d0-87774f69a715... 2024-03-28T16:10:01.614 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:10:01 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mon-b[34821]: 2024-03-28T16:10:01.343+0000 7f90a8000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T16:10:01.614 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:10:01 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mon-b[34821]: 2024-03-28T16:10:01.343+0000 7f90a8000700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-28T16:10:01.913 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:10:01 smithi150 podman[68467]: 2024-03-28 16:10:01.612613588 +0000 UTC m=+0.379266863 container died df1c336724bc41d00a40bde07ba59648cd9fcf3f8d1dea8eef68a17a91296af5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mon-b, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-28T16:10:02.249 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:10:02 smithi150 podman[68467]: 2024-03-28 16:10:02.080260242 +0000 UTC m=+0.846913511 container cleanup df1c336724bc41d00a40bde07ba59648cd9fcf3f8d1dea8eef68a17a91296af5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mon-b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-28T16:10:02.250 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:10:02 smithi150 bash[68467]: ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mon-b 2024-03-28T16:10:02.552 INFO:journalctl@ceph.mon.b.smithi150.stdout:Mar 28 16:10:02 smithi150 podman[68481]: 2024-03-28 16:10:02.247974227 +0000 UTC m=+0.620344316 container remove df1c336724bc41d00a40bde07ba59648cd9fcf3f8d1dea8eef68a17a91296af5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mon-b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T16:10:02.791 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@mon.b.service' 2024-03-28T16:10:02.933 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T16:10:02.933 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-28T16:10:02.933 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-28T16:10:02.933 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@mon.c 2024-03-28T16:10:03.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:10:03 smithi183 systemd[1]: Stopping Ceph mon.c for b5f8225a-ed1a-11ee-95d0-87774f69a715... 2024-03-28T16:10:03.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:10:03 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mon-c[35574]: 2024-03-28T16:10:03.538+0000 7f2061400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T16:10:03.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:10:03 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mon-c[35574]: 2024-03-28T16:10:03.538+0000 7f2061400700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-28T16:10:03.851 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:10:03 smithi183 podman[69322]: 2024-03-28 16:10:03.693832914 +0000 UTC m=+0.325903501 container died d8279387b55fcdb3b258590891a48c5a26678be6e42ef866bc9b43ca4057bda3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mon-c, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-28T16:10:04.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:10:04 smithi183 podman[69322]: 2024-03-28 16:10:04.18957179 +0000 UTC m=+0.821642387 container cleanup d8279387b55fcdb3b258590891a48c5a26678be6e42ef866bc9b43ca4057bda3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mon-c, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-28T16:10:04.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:10:04 smithi183 bash[69322]: ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mon-c 2024-03-28T16:10:04.741 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 28 16:10:04 smithi183 podman[69336]: 2024-03-28 16:10:04.50809449 +0000 UTC m=+0.795087648 container remove d8279387b55fcdb3b258590891a48c5a26678be6e42ef866bc9b43ca4057bda3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mon-c, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-28T16:10:04.775 DEBUG:teuthology.orchestra.run.smithi183:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@mon.c.service' 2024-03-28T16:10:04.914 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T16:10:04.914 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-28T16:10:04.914 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-28T16:10:04.914 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@mgr.x 2024-03-28T16:10:05.319 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 16:10:05 smithi067 systemd[1]: Stopping Ceph mgr.x for b5f8225a-ed1a-11ee-95d0-87774f69a715... 2024-03-28T16:10:05.821 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 16:10:05 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T16:10:05.510+0000 7f4feec00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T16:10:05.821 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 16:10:05 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x[31948]: 2024-03-28T16:10:05.510+0000 7f4feec00700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-28T16:10:05.821 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 16:10:05 smithi067 podman[90938]: 2024-03-28 16:10:05.604501152 +0000 UTC m=+0.233718915 container died a4c2b1cc1e783368f00d60cedae51385a8867410c9931a84cb6aba0639ebffe1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-28T16:10:06.320 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 16:10:06 smithi067 podman[90938]: 2024-03-28 16:10:06.087826277 +0000 UTC m=+0.717044017 container cleanup a4c2b1cc1e783368f00d60cedae51385a8867410c9931a84cb6aba0639ebffe1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-28T16:10:06.320 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 16:10:06 smithi067 bash[90938]: ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x 2024-03-28T16:10:06.819 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 28 16:10:06 smithi067 podman[90951]: 2024-03-28 16:10:06.422476106 +0000 UTC m=+0.809832053 container remove a4c2b1cc1e783368f00d60cedae51385a8867410c9931a84cb6aba0639ebffe1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-x, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T16:10:07.019 DEBUG:teuthology.orchestra.run.smithi067:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@mgr.x.service' 2024-03-28T16:10:07.148 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T16:10:07.148 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-28T16:10:07.148 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-28T16:10:07.149 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@mgr.y 2024-03-28T16:10:07.552 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Mar 28 16:10:07 smithi150 systemd[1]: Stopping Ceph mgr.y for b5f8225a-ed1a-11ee-95d0-87774f69a715... 2024-03-28T16:10:08.107 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Mar 28 16:10:07 smithi150 podman[68591]: 2024-03-28 16:10:07.822818853 +0000 UTC m=+0.234312608 container died d5b1a132ba2eba3a4b5e425b8b3f8e5cb60775039278a57d6d87af6f6e22667f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-y, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-28T16:10:08.361 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Mar 28 16:10:08 smithi150 podman[68591]: 2024-03-28 16:10:08.334554632 +0000 UTC m=+0.746048377 container cleanup d5b1a132ba2eba3a4b5e425b8b3f8e5cb60775039278a57d6d87af6f6e22667f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-28T16:10:08.362 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Mar 28 16:10:08 smithi150 bash[68591]: ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-y 2024-03-28T16:10:09.052 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Mar 28 16:10:08 smithi150 podman[68605]: 2024-03-28 16:10:08.594153937 +0000 UTC m=+0.759533550 container remove d5b1a132ba2eba3a4b5e425b8b3f8e5cb60775039278a57d6d87af6f6e22667f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-y, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319) 2024-03-28T16:10:09.052 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Mar 28 16:10:08 smithi150 systemd[1]: ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-03-28T16:10:09.145 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@mgr.y.service' 2024-03-28T16:10:09.264 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T16:10:09.264 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-28T16:10:09.264 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-03-28T16:10:09.265 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@mgr.z 2024-03-28T16:10:09.850 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 28 16:10:09 smithi183 systemd[1]: Stopping Ceph mgr.z for b5f8225a-ed1a-11ee-95d0-87774f69a715... 2024-03-28T16:10:10.275 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 28 16:10:09 smithi183 podman[69444]: 2024-03-28 16:10:09.94200395 +0000 UTC m=+0.273627846 container died 66d2d9e37125865ae3365ce21d6946736aa01c8cc1b7d63eff25beec9d7f5c02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-z, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319) 2024-03-28T16:10:10.600 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 28 16:10:10 smithi183 podman[69444]: 2024-03-28 16:10:10.478747346 +0000 UTC m=+0.810371255 container cleanup 66d2d9e37125865ae3365ce21d6946736aa01c8cc1b7d63eff25beec9d7f5c02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-z, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-28T16:10:10.600 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 28 16:10:10 smithi183 bash[69444]: ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-z 2024-03-28T16:10:11.101 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 28 16:10:10 smithi183 podman[69459]: 2024-03-28 16:10:10.662391013 +0000 UTC m=+0.707186340 container remove 66d2d9e37125865ae3365ce21d6946736aa01c8cc1b7d63eff25beec9d7f5c02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mgr-z, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T16:10:11.101 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Mar 28 16:10:10 smithi183 systemd[1]: ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@mgr.z.service: Main process exited, code=exited, status=143/n/a 2024-03-28T16:10:11.175 DEBUG:teuthology.orchestra.run.smithi183:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@mgr.z.service' 2024-03-28T16:10:11.292 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T16:10:11.293 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-03-28T16:10:11.293 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-28T16:10:11.293 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@osd.0 2024-03-28T16:10:11.819 INFO:journalctl@ceph.osd.0.smithi067.stdout:Mar 28 16:10:11 smithi067 systemd[1]: Stopping Ceph osd.0 for b5f8225a-ed1a-11ee-95d0-87774f69a715... 2024-03-28T16:10:12.201 INFO:journalctl@ceph.osd.0.smithi067.stdout:Mar 28 16:10:11 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-0[46432]: 2024-03-28T16:10:11.880+0000 7fd059200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T16:10:12.201 INFO:journalctl@ceph.osd.0.smithi067.stdout:Mar 28 16:10:11 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-0[46432]: 2024-03-28T16:10:11.880+0000 7fd059200700 -1 osd.0 95 *** Got signal Terminated *** 2024-03-28T16:10:12.201 INFO:journalctl@ceph.osd.0.smithi067.stdout:Mar 28 16:10:11 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-0[46432]: 2024-03-28T16:10:11.880+0000 7fd059200700 -1 osd.0 95 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T16:10:17.699 INFO:journalctl@ceph.osd.0.smithi067.stdout:Mar 28 16:10:17 smithi067 podman[91058]: 2024-03-28 16:10:17.445058996 +0000 UTC m=+5.710048496 container died bbb9444dd28c67b09cddb36bf2607085590356758767e0ec54dc7276e5fd2260 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-0, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-28T16:10:18.036 INFO:journalctl@ceph.osd.0.smithi067.stdout:Mar 28 16:10:17 smithi067 podman[91058]: 2024-03-28 16:10:17.816928579 +0000 UTC m=+6.081918105 container cleanup bbb9444dd28c67b09cddb36bf2607085590356758767e0ec54dc7276e5fd2260 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-28T16:10:18.036 INFO:journalctl@ceph.osd.0.smithi067.stdout:Mar 28 16:10:17 smithi067 bash[91058]: ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-0 2024-03-28T16:10:18.319 INFO:journalctl@ceph.osd.0.smithi067.stdout:Mar 28 16:10:18 smithi067 podman[91091]: 2024-03-28 16:10:18.03456803 +0000 UTC m=+0.576505175 container remove bbb9444dd28c67b09cddb36bf2607085590356758767e0ec54dc7276e5fd2260 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-0, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T16:10:19.251 INFO:journalctl@ceph.osd.0.smithi067.stdout:Mar 28 16:10:18 smithi067 podman[91180]: 2024-03-28 16:10:18.842010461 +0000 UTC m=+0.040096019 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T16:10:19.251 INFO:journalctl@ceph.osd.0.smithi067.stdout:Mar 28 16:10:19 smithi067 podman[91180]: 2024-03-28 16:10:19.093860951 +0000 UTC m=+0.291946546 container create fed7a6e016d7c06e21c462e0b9152a773ba9efbd6de8bbf9e1c3dce98d3d1a37 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS) 2024-03-28T16:10:19.570 INFO:journalctl@ceph.osd.0.smithi067.stdout:Mar 28 16:10:19 smithi067 podman[91180]: 2024-03-28 16:10:19.37289517 +0000 UTC m=+0.570980785 container init fed7a6e016d7c06e21c462e0b9152a773ba9efbd6de8bbf9e1c3dce98d3d1a37 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-0-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, ceph=True, GIT_CLEAN=True) 2024-03-28T16:10:19.570 INFO:journalctl@ceph.osd.0.smithi067.stdout:Mar 28 16:10:19 smithi067 podman[91180]: 2024-03-28 16:10:19.384537344 +0000 UTC m=+0.582622953 container start fed7a6e016d7c06e21c462e0b9152a773ba9efbd6de8bbf9e1c3dce98d3d1a37 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-28T16:10:19.570 INFO:journalctl@ceph.osd.0.smithi067.stdout:Mar 28 16:10:19 smithi067 podman[91180]: 2024-03-28 16:10:19.471198652 +0000 UTC m=+0.669284283 container attach fed7a6e016d7c06e21c462e0b9152a773ba9efbd6de8bbf9e1c3dce98d3d1a37 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-0-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319) 2024-03-28T16:10:20.319 INFO:journalctl@ceph.osd.0.smithi067.stdout:Mar 28 16:10:19 smithi067 podman[91180]: 2024-03-28 16:10:19.844470195 +0000 UTC m=+1.042555783 container died fed7a6e016d7c06e21c462e0b9152a773ba9efbd6de8bbf9e1c3dce98d3d1a37 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True) 2024-03-28T16:10:20.695 DEBUG:teuthology.orchestra.run.smithi067:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@osd.0.service' 2024-03-28T16:10:20.826 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T16:10:20.826 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-28T16:10:20.826 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-28T16:10:20.827 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@osd.1 2024-03-28T16:10:21.302 INFO:journalctl@ceph.osd.1.smithi150.stdout:Mar 28 16:10:20 smithi150 systemd[1]: Stopping Ceph osd.1 for b5f8225a-ed1a-11ee-95d0-87774f69a715... 2024-03-28T16:10:21.803 INFO:journalctl@ceph.osd.1.smithi150.stdout:Mar 28 16:10:21 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-1[39434]: 2024-03-28T16:10:21.411+0000 7febddc00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T16:10:21.803 INFO:journalctl@ceph.osd.1.smithi150.stdout:Mar 28 16:10:21 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-1[39434]: 2024-03-28T16:10:21.411+0000 7febddc00700 -1 osd.1 95 *** Got signal Terminated *** 2024-03-28T16:10:21.803 INFO:journalctl@ceph.osd.1.smithi150.stdout:Mar 28 16:10:21 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-1[39434]: 2024-03-28T16:10:21.411+0000 7febddc00700 -1 osd.1 95 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T16:10:27.053 INFO:journalctl@ceph.osd.1.smithi150.stdout:Mar 28 16:10:26 smithi150 podman[68715]: 2024-03-28 16:10:26.798759312 +0000 UTC m=+5.519395797 container died 4d90ec1f75bde49120ec782cfce4080b11b68fdf1cc2cd4f54515755e47bb024 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-1, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-28T16:10:27.499 INFO:journalctl@ceph.osd.1.smithi150.stdout:Mar 28 16:10:27 smithi150 podman[68715]: 2024-03-28 16:10:27.213744262 +0000 UTC m=+5.934380750 container cleanup 4d90ec1f75bde49120ec782cfce4080b11b68fdf1cc2cd4f54515755e47bb024 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-1, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T16:10:27.499 INFO:journalctl@ceph.osd.1.smithi150.stdout:Mar 28 16:10:27 smithi150 bash[68715]: ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-1 2024-03-28T16:10:27.802 INFO:journalctl@ceph.osd.1.smithi150.stdout:Mar 28 16:10:27 smithi150 podman[68733]: 2024-03-28 16:10:27.498311062 +0000 UTC m=+0.683632107 container remove 4d90ec1f75bde49120ec782cfce4080b11b68fdf1cc2cd4f54515755e47bb024 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-1, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-28T16:10:28.553 INFO:journalctl@ceph.osd.1.smithi150.stdout:Mar 28 16:10:28 smithi150 podman[68823]: 2024-03-28 16:10:28.185899713 +0000 UTC m=+0.038721848 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T16:10:28.553 INFO:journalctl@ceph.osd.1.smithi150.stdout:Mar 28 16:10:28 smithi150 podman[68823]: 2024-03-28 16:10:28.299199824 +0000 UTC m=+0.152021963 container create ba154de2d22c6f74ca0606be1dfcf9563af025bf2673d3fa7a29ace8ead0f987 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-1-deactivate, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-28T16:10:28.996 INFO:journalctl@ceph.osd.1.smithi150.stdout:Mar 28 16:10:28 smithi150 podman[68823]: 2024-03-28 16:10:28.553373229 +0000 UTC m=+0.406195408 container init ba154de2d22c6f74ca0606be1dfcf9563af025bf2673d3fa7a29ace8ead0f987 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T16:10:28.996 INFO:journalctl@ceph.osd.1.smithi150.stdout:Mar 28 16:10:28 smithi150 podman[68823]: 2024-03-28 16:10:28.564251607 +0000 UTC m=+0.417073752 container start ba154de2d22c6f74ca0606be1dfcf9563af025bf2673d3fa7a29ace8ead0f987 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-1-deactivate, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True) 2024-03-28T16:10:28.996 INFO:journalctl@ceph.osd.1.smithi150.stdout:Mar 28 16:10:28 smithi150 podman[68823]: 2024-03-28 16:10:28.676782744 +0000 UTC m=+0.529604889 container attach ba154de2d22c6f74ca0606be1dfcf9563af025bf2673d3fa7a29ace8ead0f987 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-1-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.build-date=20240319, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-28T16:10:29.302 INFO:journalctl@ceph.osd.1.smithi150.stdout:Mar 28 16:10:29 smithi150 podman[68849]: 2024-03-28 16:10:29.056593819 +0000 UTC m=+0.037642580 container died ba154de2d22c6f74ca0606be1dfcf9563af025bf2673d3fa7a29ace8ead0f987 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-1-deactivate, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-28T16:10:29.727 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@osd.1.service' 2024-03-28T16:10:29.852 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T16:10:29.852 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-28T16:10:29.852 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-28T16:10:29.852 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@osd.2 2024-03-28T16:10:30.350 INFO:journalctl@ceph.osd.2.smithi183.stdout:Mar 28 16:10:29 smithi183 systemd[1]: Stopping Ceph osd.2 for b5f8225a-ed1a-11ee-95d0-87774f69a715... 2024-03-28T16:10:30.714 INFO:journalctl@ceph.osd.2.smithi183.stdout:Mar 28 16:10:30 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-2[40716]: 2024-03-28T16:10:30.411+0000 7fa830c00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T16:10:30.715 INFO:journalctl@ceph.osd.2.smithi183.stdout:Mar 28 16:10:30 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-2[40716]: 2024-03-28T16:10:30.411+0000 7fa830c00700 -1 osd.2 95 *** Got signal Terminated *** 2024-03-28T16:10:30.715 INFO:journalctl@ceph.osd.2.smithi183.stdout:Mar 28 16:10:30 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-2[40716]: 2024-03-28T16:10:30.411+0000 7fa830c00700 -1 osd.2 95 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T16:10:36.101 INFO:journalctl@ceph.osd.2.smithi183.stdout:Mar 28 16:10:35 smithi183 podman[69569]: 2024-03-28 16:10:35.848476722 +0000 UTC m=+5.583390928 container died df90457e598419f60aec676a17b2fe8d32f4a26fe1c00668d0e77ee5d539d108 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-2, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-28T16:10:36.601 INFO:journalctl@ceph.osd.2.smithi183.stdout:Mar 28 16:10:36 smithi183 podman[69569]: 2024-03-28 16:10:36.324106491 +0000 UTC m=+6.059020745 container cleanup df90457e598419f60aec676a17b2fe8d32f4a26fe1c00668d0e77ee5d539d108 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-28T16:10:36.601 INFO:journalctl@ceph.osd.2.smithi183.stdout:Mar 28 16:10:36 smithi183 bash[69569]: ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-2 2024-03-28T16:10:37.100 INFO:journalctl@ceph.osd.2.smithi183.stdout:Mar 28 16:10:36 smithi183 podman[69600]: 2024-03-28 16:10:36.616822413 +0000 UTC m=+0.754614016 container remove df90457e598419f60aec676a17b2fe8d32f4a26fe1c00668d0e77ee5d539d108 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T16:10:37.522 INFO:journalctl@ceph.osd.2.smithi183.stdout:Mar 28 16:10:37 smithi183 podman[69688]: 2024-03-28 16:10:37.128276914 +0000 UTC m=+0.036947018 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T16:10:37.522 INFO:journalctl@ceph.osd.2.smithi183.stdout:Mar 28 16:10:37 smithi183 podman[69688]: 2024-03-28 16:10:37.234453689 +0000 UTC m=+0.143123821 container create 78047574c8f594e5352ee301e97e231fbf0909026bcd805bb58ee3e5594c28cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-2-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-28T16:10:37.522 INFO:journalctl@ceph.osd.2.smithi183.stdout:Mar 28 16:10:37 smithi183 podman[69688]: 2024-03-28 16:10:37.521343194 +0000 UTC m=+0.430013333 container init 78047574c8f594e5352ee301e97e231fbf0909026bcd805bb58ee3e5594c28cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-2-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-28T16:10:37.850 INFO:journalctl@ceph.osd.2.smithi183.stdout:Mar 28 16:10:37 smithi183 podman[69688]: 2024-03-28 16:10:37.533236583 +0000 UTC m=+0.441906722 container start 78047574c8f594e5352ee301e97e231fbf0909026bcd805bb58ee3e5594c28cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-2-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True) 2024-03-28T16:10:37.850 INFO:journalctl@ceph.osd.2.smithi183.stdout:Mar 28 16:10:37 smithi183 podman[69688]: 2024-03-28 16:10:37.619949578 +0000 UTC m=+0.528619753 container attach 78047574c8f594e5352ee301e97e231fbf0909026bcd805bb58ee3e5594c28cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-2-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0) 2024-03-28T16:10:38.336 INFO:journalctl@ceph.osd.2.smithi183.stdout:Mar 28 16:10:37 smithi183 podman[69688]: 2024-03-28 16:10:37.964052046 +0000 UTC m=+0.872722182 container died 78047574c8f594e5352ee301e97e231fbf0909026bcd805bb58ee3e5594c28cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-28T16:10:38.635 DEBUG:teuthology.orchestra.run.smithi183:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@osd.2.service' 2024-03-28T16:10:38.756 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T16:10:38.756 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-28T16:10:38.756 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-28T16:10:38.757 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@osd.3 2024-03-28T16:10:39.052 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:10:38 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:10:38.727+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:10:39.100 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 28 16:10:38 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-5[47486]: 2024-03-28T16:10:38.844+0000 7fd7bfc00700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.299545+0000 front 2024-03-28T16:10:16.299299+0000 (oldest deadline 2024-03-28T16:10:37.999325+0000) 2024-03-28T16:10:39.319 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:10:38 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:10:38.992+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.291110+0000 front 2024-03-28T16:10:16.291029+0000 (oldest deadline 2024-03-28T16:10:37.991294+0000) 2024-03-28T16:10:39.320 INFO:journalctl@ceph.osd.3.smithi067.stdout:Mar 28 16:10:38 smithi067 systemd[1]: Stopping Ceph osd.3 for b5f8225a-ed1a-11ee-95d0-87774f69a715... 2024-03-28T16:10:39.320 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:10:38 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:10:38.987+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.533614+0000 front 2024-03-28T16:10:16.534633+0000 (oldest deadline 2024-03-28T16:10:38.233619+0000) 2024-03-28T16:10:39.819 INFO:journalctl@ceph.osd.3.smithi067.stdout:Mar 28 16:10:39 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-3[52980]: 2024-03-28T16:10:39.523+0000 7fc67a600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T16:10:39.819 INFO:journalctl@ceph.osd.3.smithi067.stdout:Mar 28 16:10:39 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-3[52980]: 2024-03-28T16:10:39.523+0000 7fc67a600700 -1 osd.3 95 *** Got signal Terminated *** 2024-03-28T16:10:39.820 INFO:journalctl@ceph.osd.3.smithi067.stdout:Mar 28 16:10:39 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-3[52980]: 2024-03-28T16:10:39.523+0000 7fc67a600700 -1 osd.3 95 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T16:10:40.053 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:10:39 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:10:39.689+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:10:40.100 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 28 16:10:39 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-5[47486]: 2024-03-28T16:10:39.838+0000 7fd7bfc00700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.299545+0000 front 2024-03-28T16:10:16.299299+0000 (oldest deadline 2024-03-28T16:10:37.999325+0000) 2024-03-28T16:10:40.319 INFO:journalctl@ceph.osd.3.smithi067.stdout:Mar 28 16:10:40 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-3[52980]: 2024-03-28T16:10:40.079+0000 7fc670e00700 -1 osd.3 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.317820+0000 front 2024-03-28T16:10:16.318603+0000 (oldest deadline 2024-03-28T16:10:39.817476+0000) 2024-03-28T16:10:40.320 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:10:39 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:10:39.978+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.291110+0000 front 2024-03-28T16:10:16.291029+0000 (oldest deadline 2024-03-28T16:10:37.991294+0000) 2024-03-28T16:10:40.320 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:10:39 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:10:39.995+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.533614+0000 front 2024-03-28T16:10:16.534633+0000 (oldest deadline 2024-03-28T16:10:38.233619+0000) 2024-03-28T16:10:40.850 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:10:40 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:10:40.434+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:10:41.052 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:10:40 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:10:40.680+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:10:41.320 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:10:40 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:10:40.981+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.291110+0000 front 2024-03-28T16:10:16.291029+0000 (oldest deadline 2024-03-28T16:10:37.991294+0000) 2024-03-28T16:10:41.320 INFO:journalctl@ceph.osd.3.smithi067.stdout:Mar 28 16:10:41 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-3[52980]: 2024-03-28T16:10:41.095+0000 7fc670e00700 -1 osd.3 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.317820+0000 front 2024-03-28T16:10:16.318603+0000 (oldest deadline 2024-03-28T16:10:39.817476+0000) 2024-03-28T16:10:41.321 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:10:40 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:10:40.995+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.533614+0000 front 2024-03-28T16:10:16.534633+0000 (oldest deadline 2024-03-28T16:10:38.233619+0000) 2024-03-28T16:10:41.350 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 28 16:10:40 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-5[47486]: 2024-03-28T16:10:40.883+0000 7fd7bfc00700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.299545+0000 front 2024-03-28T16:10:16.299299+0000 (oldest deadline 2024-03-28T16:10:37.999325+0000) 2024-03-28T16:10:41.837 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:10:41 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:10:41.420+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:10:42.052 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:10:41 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:10:41.682+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:10:42.100 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 28 16:10:41 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-5[47486]: 2024-03-28T16:10:41.836+0000 7fd7bfc00700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.299545+0000 front 2024-03-28T16:10:16.299299+0000 (oldest deadline 2024-03-28T16:10:37.999325+0000) 2024-03-28T16:10:42.257 INFO:journalctl@ceph.osd.3.smithi067.stdout:Mar 28 16:10:42 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-3[52980]: 2024-03-28T16:10:42.117+0000 7fc670e00700 -1 osd.3 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.317820+0000 front 2024-03-28T16:10:16.318603+0000 (oldest deadline 2024-03-28T16:10:39.817476+0000) 2024-03-28T16:10:42.257 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:10:41 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:10:41.988+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.291110+0000 front 2024-03-28T16:10:16.291029+0000 (oldest deadline 2024-03-28T16:10:37.991294+0000) 2024-03-28T16:10:42.257 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:10:41 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:10:41.994+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.533614+0000 front 2024-03-28T16:10:16.534633+0000 (oldest deadline 2024-03-28T16:10:38.233619+0000) 2024-03-28T16:10:42.791 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:10:42 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:10:42.470+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:10:43.052 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:10:42 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:10:42.711+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:10:43.100 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 28 16:10:42 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-5[47486]: 2024-03-28T16:10:42.790+0000 7fd7bfc00700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.299545+0000 front 2024-03-28T16:10:16.299299+0000 (oldest deadline 2024-03-28T16:10:37.999325+0000) 2024-03-28T16:10:43.320 INFO:journalctl@ceph.osd.3.smithi067.stdout:Mar 28 16:10:43 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-3[52980]: 2024-03-28T16:10:43.074+0000 7fc670e00700 -1 osd.3 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.317820+0000 front 2024-03-28T16:10:16.318603+0000 (oldest deadline 2024-03-28T16:10:39.817476+0000) 2024-03-28T16:10:43.320 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:10:42 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:10:42.947+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.533614+0000 front 2024-03-28T16:10:16.534633+0000 (oldest deadline 2024-03-28T16:10:38.233619+0000) 2024-03-28T16:10:43.320 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:10:42 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:10:42.968+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.291110+0000 front 2024-03-28T16:10:16.291029+0000 (oldest deadline 2024-03-28T16:10:37.991294+0000) 2024-03-28T16:10:43.827 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:10:43 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:10:43.505+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:10:44.052 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:10:43 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:10:43.742+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:10:44.100 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 28 16:10:43 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-5[47486]: 2024-03-28T16:10:43.826+0000 7fd7bfc00700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.299545+0000 front 2024-03-28T16:10:16.299299+0000 (oldest deadline 2024-03-28T16:10:37.999325+0000) 2024-03-28T16:10:44.319 INFO:journalctl@ceph.osd.3.smithi067.stdout:Mar 28 16:10:44 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-3[52980]: 2024-03-28T16:10:44.040+0000 7fc670e00700 -1 osd.3 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.317820+0000 front 2024-03-28T16:10:16.318603+0000 (oldest deadline 2024-03-28T16:10:39.817476+0000) 2024-03-28T16:10:44.320 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:10:44 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:10:44.011+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.291110+0000 front 2024-03-28T16:10:16.291029+0000 (oldest deadline 2024-03-28T16:10:37.991294+0000) 2024-03-28T16:10:44.320 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:10:43 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:10:43.941+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.533614+0000 front 2024-03-28T16:10:16.534633+0000 (oldest deadline 2024-03-28T16:10:38.233619+0000) 2024-03-28T16:10:44.850 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:10:44 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:10:44.484+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:10:45.052 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:10:44 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:10:44.711+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:10:45.319 INFO:journalctl@ceph.osd.3.smithi067.stdout:Mar 28 16:10:45 smithi067 podman[91283]: 2024-03-28 16:10:45.08103571 +0000 UTC m=+5.819699341 container died b47291b09f6d983d10ca13379fff61f40f2e51ad62bf8549afc2f0aefd2775d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-3, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T16:10:45.320 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:10:44 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:10:44.983+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.291110+0000 front 2024-03-28T16:10:16.291029+0000 (oldest deadline 2024-03-28T16:10:37.991294+0000) 2024-03-28T16:10:45.320 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:10:44 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:10:44.952+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.533614+0000 front 2024-03-28T16:10:16.534633+0000 (oldest deadline 2024-03-28T16:10:38.233619+0000) 2024-03-28T16:10:45.350 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 28 16:10:44 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-5[47486]: 2024-03-28T16:10:44.857+0000 7fd7bfc00700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.299545+0000 front 2024-03-28T16:10:16.299299+0000 (oldest deadline 2024-03-28T16:10:37.999325+0000) 2024-03-28T16:10:45.819 INFO:journalctl@ceph.osd.3.smithi067.stdout:Mar 28 16:10:45 smithi067 podman[91283]: 2024-03-28 16:10:45.617070685 +0000 UTC m=+6.355734359 container cleanup b47291b09f6d983d10ca13379fff61f40f2e51ad62bf8549afc2f0aefd2775d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-3, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-28T16:10:45.820 INFO:journalctl@ceph.osd.3.smithi067.stdout:Mar 28 16:10:45 smithi067 bash[91283]: ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-3 2024-03-28T16:10:45.850 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:10:45 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:10:45.525+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:10:46.052 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:10:45 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:10:45.691+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:10:46.319 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:10:45 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:10:45.971+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.291110+0000 front 2024-03-28T16:10:16.291029+0000 (oldest deadline 2024-03-28T16:10:37.991294+0000) 2024-03-28T16:10:46.320 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:10:45 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:10:45.905+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.533614+0000 front 2024-03-28T16:10:16.534633+0000 (oldest deadline 2024-03-28T16:10:38.233619+0000) 2024-03-28T16:10:46.320 INFO:journalctl@ceph.osd.3.smithi067.stdout:Mar 28 16:10:45 smithi067 podman[91314]: 2024-03-28 16:10:45.893396393 +0000 UTC m=+0.796763208 container remove b47291b09f6d983d10ca13379fff61f40f2e51ad62bf8549afc2f0aefd2775d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-3, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-28T16:10:46.350 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 28 16:10:45 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-5[47486]: 2024-03-28T16:10:45.884+0000 7fd7bfc00700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.299545+0000 front 2024-03-28T16:10:16.299299+0000 (oldest deadline 2024-03-28T16:10:37.999325+0000) 2024-03-28T16:10:46.850 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:10:46 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:10:46.544+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:10:47.031 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:10:46 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:10:46.983+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.291110+0000 front 2024-03-28T16:10:16.291029+0000 (oldest deadline 2024-03-28T16:10:37.991294+0000) 2024-03-28T16:10:47.031 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:10:46 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:10:46.944+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.533614+0000 front 2024-03-28T16:10:16.534633+0000 (oldest deadline 2024-03-28T16:10:38.233619+0000) 2024-03-28T16:10:47.032 INFO:journalctl@ceph.osd.3.smithi067.stdout:Mar 28 16:10:46 smithi067 podman[91405]: 2024-03-28 16:10:46.660936189 +0000 UTC m=+0.033646280 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T16:10:47.032 INFO:journalctl@ceph.osd.3.smithi067.stdout:Mar 28 16:10:46 smithi067 podman[91405]: 2024-03-28 16:10:46.769219841 +0000 UTC m=+0.141929956 container create 2fa2ff36e16388d7da96a1e43ea9f46b9fc1f270f7262ee0901c0e2d3a6d6b23 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-3-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS) 2024-03-28T16:10:47.032 INFO:journalctl@ceph.osd.3.smithi067.stdout:Mar 28 16:10:47 smithi067 podman[91405]: 2024-03-28 16:10:47.031075767 +0000 UTC m=+0.403785889 container init 2fa2ff36e16388d7da96a1e43ea9f46b9fc1f270f7262ee0901c0e2d3a6d6b23 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-3-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T16:10:47.052 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:10:46 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:10:46.654+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:10:47.319 INFO:journalctl@ceph.osd.3.smithi067.stdout:Mar 28 16:10:47 smithi067 podman[91405]: 2024-03-28 16:10:47.042765882 +0000 UTC m=+0.415475987 container start 2fa2ff36e16388d7da96a1e43ea9f46b9fc1f270f7262ee0901c0e2d3a6d6b23 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-3-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-28T16:10:47.320 INFO:journalctl@ceph.osd.3.smithi067.stdout:Mar 28 16:10:47 smithi067 podman[91405]: 2024-03-28 16:10:47.129906603 +0000 UTC m=+0.502616714 container attach 2fa2ff36e16388d7da96a1e43ea9f46b9fc1f270f7262ee0901c0e2d3a6d6b23 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-3-deactivate, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-28T16:10:47.350 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 28 16:10:46 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-5[47486]: 2024-03-28T16:10:46.849+0000 7fd7bfc00700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.299545+0000 front 2024-03-28T16:10:16.299299+0000 (oldest deadline 2024-03-28T16:10:37.999325+0000) 2024-03-28T16:10:47.820 INFO:journalctl@ceph.osd.3.smithi067.stdout:Mar 28 16:10:47 smithi067 podman[91405]: 2024-03-28 16:10:47.495318186 +0000 UTC m=+0.868028316 container died 2fa2ff36e16388d7da96a1e43ea9f46b9fc1f270f7262ee0901c0e2d3a6d6b23 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-28T16:10:47.850 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:10:47 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:10:47.565+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:10:47.983 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:10:47 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:10:47.669+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:10:48.302 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:10:47 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:10:47.982+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.173988+0000 front 2024-03-28T16:10:23.173853+0000 (oldest deadline 2024-03-28T16:10:47.273946+0000) 2024-03-28T16:10:48.317 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:10:47 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:10:47.955+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.291110+0000 front 2024-03-28T16:10:16.291029+0000 (oldest deadline 2024-03-28T16:10:37.991294+0000) 2024-03-28T16:10:48.317 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:10:47 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:10:47.955+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:25.898350+0000 front 2024-03-28T16:10:25.898465+0000 (oldest deadline 2024-03-28T16:10:47.598638+0000) 2024-03-28T16:10:48.318 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:10:47 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:10:47.902+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.533614+0000 front 2024-03-28T16:10:16.534633+0000 (oldest deadline 2024-03-28T16:10:38.233619+0000) 2024-03-28T16:10:48.318 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:10:47 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:10:47.902+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:21.735742+0000 front 2024-03-28T16:10:21.735828+0000 (oldest deadline 2024-03-28T16:10:47.635538+0000) 2024-03-28T16:10:48.350 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 28 16:10:47 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-5[47486]: 2024-03-28T16:10:47.892+0000 7fd7bfc00700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.299545+0000 front 2024-03-28T16:10:16.299299+0000 (oldest deadline 2024-03-28T16:10:37.999325+0000) 2024-03-28T16:10:48.351 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:10:48 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:10:48.103+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.809005+0000 front 2024-03-28T16:10:24.808952+0000 (oldest deadline 2024-03-28T16:10:47.108813+0000) 2024-03-28T16:10:48.365 DEBUG:teuthology.orchestra.run.smithi067:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@osd.3.service' 2024-03-28T16:10:48.491 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T16:10:48.491 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-28T16:10:48.491 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-28T16:10:48.492 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@osd.4 2024-03-28T16:10:48.803 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:10:48 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:10:48.694+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:10:48.803 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:10:48 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:10:48.694+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.601392+0000 front 2024-03-28T16:10:23.601326+0000 (oldest deadline 2024-03-28T16:10:47.701522+0000) 2024-03-28T16:10:48.803 INFO:journalctl@ceph.osd.4.smithi150.stdout:Mar 28 16:10:48 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-4[46365]: 2024-03-28T16:10:48.526+0000 7f5081200700 -1 osd.4 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:25.611238+0000 front 2024-03-28T16:10:25.611041+0000 (oldest deadline 2024-03-28T16:10:47.911231+0000) 2024-03-28T16:10:48.803 INFO:journalctl@ceph.osd.4.smithi150.stdout:Mar 28 16:10:48 smithi150 systemd[1]: Stopping Ceph osd.4 for b5f8225a-ed1a-11ee-95d0-87774f69a715... 2024-03-28T16:10:48.850 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:10:48 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:10:48.556+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:10:49.302 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:10:48 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:10:48.991+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.173988+0000 front 2024-03-28T16:10:23.173853+0000 (oldest deadline 2024-03-28T16:10:47.273946+0000) 2024-03-28T16:10:49.303 INFO:journalctl@ceph.osd.4.smithi150.stdout:Mar 28 16:10:49 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-4[46365]: 2024-03-28T16:10:49.072+0000 7f5085a00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T16:10:49.303 INFO:journalctl@ceph.osd.4.smithi150.stdout:Mar 28 16:10:49 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-4[46365]: 2024-03-28T16:10:49.072+0000 7f5085a00700 -1 osd.4 95 *** Got signal Terminated *** 2024-03-28T16:10:49.303 INFO:journalctl@ceph.osd.4.smithi150.stdout:Mar 28 16:10:49 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-4[46365]: 2024-03-28T16:10:49.072+0000 7f5085a00700 -1 osd.4 95 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T16:10:49.319 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:10:48 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:10:48.883+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.533614+0000 front 2024-03-28T16:10:16.534633+0000 (oldest deadline 2024-03-28T16:10:38.233619+0000) 2024-03-28T16:10:49.320 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:10:48 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:10:48.883+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:21.735742+0000 front 2024-03-28T16:10:21.735828+0000 (oldest deadline 2024-03-28T16:10:47.635538+0000) 2024-03-28T16:10:49.320 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:10:48 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:10:48.938+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.291110+0000 front 2024-03-28T16:10:16.291029+0000 (oldest deadline 2024-03-28T16:10:37.991294+0000) 2024-03-28T16:10:49.320 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:10:48 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:10:48.938+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:25.898350+0000 front 2024-03-28T16:10:25.898465+0000 (oldest deadline 2024-03-28T16:10:47.598638+0000) 2024-03-28T16:10:49.350 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:10:49 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:10:49.129+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.809005+0000 front 2024-03-28T16:10:24.808952+0000 (oldest deadline 2024-03-28T16:10:47.108813+0000) 2024-03-28T16:10:49.350 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 28 16:10:48 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-5[47486]: 2024-03-28T16:10:48.883+0000 7fd7bfc00700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.299545+0000 front 2024-03-28T16:10:16.299299+0000 (oldest deadline 2024-03-28T16:10:37.999325+0000) 2024-03-28T16:10:49.802 INFO:journalctl@ceph.osd.4.smithi150.stdout:Mar 28 16:10:49 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-4[46365]: 2024-03-28T16:10:49.504+0000 7f5081200700 -1 osd.4 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:25.611238+0000 front 2024-03-28T16:10:25.611041+0000 (oldest deadline 2024-03-28T16:10:47.911231+0000) 2024-03-28T16:10:49.802 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:10:49 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:10:49.711+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:10:49.803 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:10:49 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:10:49.711+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.601392+0000 front 2024-03-28T16:10:23.601326+0000 (oldest deadline 2024-03-28T16:10:47.701522+0000) 2024-03-28T16:10:49.850 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:10:49 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:10:49.514+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:10:49.850 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:10:49 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:10:49.514+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.053696+0000 front 2024-03-28T16:10:24.053904+0000 (oldest deadline 2024-03-28T16:10:48.753813+0000) 2024-03-28T16:10:50.127 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 28 16:10:49 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-5[47486]: 2024-03-28T16:10:49.876+0000 7fd7bfc00700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.299545+0000 front 2024-03-28T16:10:16.299299+0000 (oldest deadline 2024-03-28T16:10:37.999325+0000) 2024-03-28T16:10:50.127 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 28 16:10:49 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-5[47486]: 2024-03-28T16:10:49.876+0000 7fd7bfc00700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.402160+0000 front 2024-03-28T16:10:24.402709+0000 (oldest deadline 2024-03-28T16:10:49.702229+0000) 2024-03-28T16:10:50.302 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:10:49 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:10:49.957+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.173988+0000 front 2024-03-28T16:10:23.173853+0000 (oldest deadline 2024-03-28T16:10:47.273946+0000) 2024-03-28T16:10:50.319 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:10:49 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:10:49.958+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.291110+0000 front 2024-03-28T16:10:16.291029+0000 (oldest deadline 2024-03-28T16:10:37.991294+0000) 2024-03-28T16:10:50.319 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:10:49 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:10:49.958+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:25.898350+0000 front 2024-03-28T16:10:25.898465+0000 (oldest deadline 2024-03-28T16:10:47.598638+0000) 2024-03-28T16:10:50.320 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:10:49 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:10:49.891+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.533614+0000 front 2024-03-28T16:10:16.534633+0000 (oldest deadline 2024-03-28T16:10:38.233619+0000) 2024-03-28T16:10:50.320 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:10:49 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:10:49.891+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:21.735742+0000 front 2024-03-28T16:10:21.735828+0000 (oldest deadline 2024-03-28T16:10:47.635538+0000) 2024-03-28T16:10:50.549 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:10:50 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:10:50.126+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.809005+0000 front 2024-03-28T16:10:24.808952+0000 (oldest deadline 2024-03-28T16:10:47.108813+0000) 2024-03-28T16:10:50.802 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:10:50 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:10:50.734+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:10:50.802 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:10:50 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:10:50.734+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.601392+0000 front 2024-03-28T16:10:23.601326+0000 (oldest deadline 2024-03-28T16:10:47.701522+0000) 2024-03-28T16:10:50.802 INFO:journalctl@ceph.osd.4.smithi150.stdout:Mar 28 16:10:50 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-4[46365]: 2024-03-28T16:10:50.523+0000 7f5081200700 -1 osd.4 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:25.611238+0000 front 2024-03-28T16:10:25.611041+0000 (oldest deadline 2024-03-28T16:10:47.911231+0000) 2024-03-28T16:10:50.850 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:10:50 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:10:50.548+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:10:50.850 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:10:50 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:10:50.548+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.053696+0000 front 2024-03-28T16:10:24.053904+0000 (oldest deadline 2024-03-28T16:10:48.753813+0000) 2024-03-28T16:10:51.302 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:10:51 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:10:51.002+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.173988+0000 front 2024-03-28T16:10:23.173853+0000 (oldest deadline 2024-03-28T16:10:47.273946+0000) 2024-03-28T16:10:51.319 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:10:50 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:10:50.954+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.291110+0000 front 2024-03-28T16:10:16.291029+0000 (oldest deadline 2024-03-28T16:10:37.991294+0000) 2024-03-28T16:10:51.319 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:10:50 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:10:50.954+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:25.898350+0000 front 2024-03-28T16:10:25.898465+0000 (oldest deadline 2024-03-28T16:10:47.598638+0000) 2024-03-28T16:10:51.320 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:10:50 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:10:50.894+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.533614+0000 front 2024-03-28T16:10:16.534633+0000 (oldest deadline 2024-03-28T16:10:38.233619+0000) 2024-03-28T16:10:51.320 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:10:50 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:10:50.894+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:21.735742+0000 front 2024-03-28T16:10:21.735828+0000 (oldest deadline 2024-03-28T16:10:47.635538+0000) 2024-03-28T16:10:51.350 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:10:51 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:10:51.079+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.809005+0000 front 2024-03-28T16:10:24.808952+0000 (oldest deadline 2024-03-28T16:10:47.108813+0000) 2024-03-28T16:10:51.350 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 28 16:10:50 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-5[47486]: 2024-03-28T16:10:50.891+0000 7fd7bfc00700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.299545+0000 front 2024-03-28T16:10:16.299299+0000 (oldest deadline 2024-03-28T16:10:37.999325+0000) 2024-03-28T16:10:51.350 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 28 16:10:50 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-5[47486]: 2024-03-28T16:10:50.891+0000 7fd7bfc00700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.402160+0000 front 2024-03-28T16:10:24.402709+0000 (oldest deadline 2024-03-28T16:10:49.702229+0000) 2024-03-28T16:10:51.802 INFO:journalctl@ceph.osd.4.smithi150.stdout:Mar 28 16:10:51 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-4[46365]: 2024-03-28T16:10:51.539+0000 7f5081200700 -1 osd.4 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:25.611238+0000 front 2024-03-28T16:10:25.611041+0000 (oldest deadline 2024-03-28T16:10:47.911231+0000) 2024-03-28T16:10:51.803 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:10:51 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:10:51.690+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:10:51.803 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:10:51 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:10:51.690+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.601392+0000 front 2024-03-28T16:10:23.601326+0000 (oldest deadline 2024-03-28T16:10:47.701522+0000) 2024-03-28T16:10:51.850 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:10:51 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:10:51.535+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:10:51.850 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:10:51 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:10:51.535+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.053696+0000 front 2024-03-28T16:10:24.053904+0000 (oldest deadline 2024-03-28T16:10:48.753813+0000) 2024-03-28T16:10:52.302 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:10:52 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:10:52.045+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.173988+0000 front 2024-03-28T16:10:23.173853+0000 (oldest deadline 2024-03-28T16:10:47.273946+0000) 2024-03-28T16:10:52.319 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:10:51 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:10:51.920+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.533614+0000 front 2024-03-28T16:10:16.534633+0000 (oldest deadline 2024-03-28T16:10:38.233619+0000) 2024-03-28T16:10:52.319 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:10:51 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:10:51.920+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:21.735742+0000 front 2024-03-28T16:10:21.735828+0000 (oldest deadline 2024-03-28T16:10:47.635538+0000) 2024-03-28T16:10:52.320 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:10:52 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:10:52.003+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.291110+0000 front 2024-03-28T16:10:16.291029+0000 (oldest deadline 2024-03-28T16:10:37.991294+0000) 2024-03-28T16:10:52.320 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:10:52 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:10:52.003+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:25.898350+0000 front 2024-03-28T16:10:25.898465+0000 (oldest deadline 2024-03-28T16:10:47.598638+0000) 2024-03-28T16:10:52.350 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:10:52 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:10:52.107+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.809005+0000 front 2024-03-28T16:10:24.808952+0000 (oldest deadline 2024-03-28T16:10:47.108813+0000) 2024-03-28T16:10:52.350 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 28 16:10:51 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-5[47486]: 2024-03-28T16:10:51.884+0000 7fd7bfc00700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.299545+0000 front 2024-03-28T16:10:16.299299+0000 (oldest deadline 2024-03-28T16:10:37.999325+0000) 2024-03-28T16:10:52.350 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 28 16:10:51 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-5[47486]: 2024-03-28T16:10:51.884+0000 7fd7bfc00700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.402160+0000 front 2024-03-28T16:10:24.402709+0000 (oldest deadline 2024-03-28T16:10:49.702229+0000) 2024-03-28T16:10:52.845 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:10:52 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:10:52.527+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:10:52.846 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:10:52 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:10:52.527+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.053696+0000 front 2024-03-28T16:10:24.053904+0000 (oldest deadline 2024-03-28T16:10:48.753813+0000) 2024-03-28T16:10:53.050 INFO:journalctl@ceph.osd.4.smithi150.stdout:Mar 28 16:10:52 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-4[46365]: 2024-03-28T16:10:52.578+0000 7f5081200700 -1 osd.4 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:25.611238+0000 front 2024-03-28T16:10:25.611041+0000 (oldest deadline 2024-03-28T16:10:47.911231+0000) 2024-03-28T16:10:53.051 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:10:52 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:10:52.707+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:10:53.051 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:10:52 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:10:52.707+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.601392+0000 front 2024-03-28T16:10:23.601326+0000 (oldest deadline 2024-03-28T16:10:47.701522+0000) 2024-03-28T16:10:53.100 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 28 16:10:52 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-5[47486]: 2024-03-28T16:10:52.844+0000 7fd7bfc00700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.299545+0000 front 2024-03-28T16:10:16.299299+0000 (oldest deadline 2024-03-28T16:10:37.999325+0000) 2024-03-28T16:10:53.100 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 28 16:10:52 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-5[47486]: 2024-03-28T16:10:52.844+0000 7fd7bfc00700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.402160+0000 front 2024-03-28T16:10:24.402709+0000 (oldest deadline 2024-03-28T16:10:49.702229+0000) 2024-03-28T16:10:53.302 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:10:53 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:10:53.049+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.173988+0000 front 2024-03-28T16:10:23.173853+0000 (oldest deadline 2024-03-28T16:10:47.273946+0000) 2024-03-28T16:10:53.319 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:10:52 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:10:52.959+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.291110+0000 front 2024-03-28T16:10:16.291029+0000 (oldest deadline 2024-03-28T16:10:37.991294+0000) 2024-03-28T16:10:53.319 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:10:52 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:10:52.959+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:25.898350+0000 front 2024-03-28T16:10:25.898465+0000 (oldest deadline 2024-03-28T16:10:47.598638+0000) 2024-03-28T16:10:53.319 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:10:52 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:10:52.934+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.533614+0000 front 2024-03-28T16:10:16.534633+0000 (oldest deadline 2024-03-28T16:10:38.233619+0000) 2024-03-28T16:10:53.320 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:10:52 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:10:52.934+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:21.735742+0000 front 2024-03-28T16:10:21.735828+0000 (oldest deadline 2024-03-28T16:10:47.635538+0000) 2024-03-28T16:10:53.567 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:10:53 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:10:53.119+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.809005+0000 front 2024-03-28T16:10:24.808952+0000 (oldest deadline 2024-03-28T16:10:47.108813+0000) 2024-03-28T16:10:53.818 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:10:53 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:10:53.566+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:10:53.819 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:10:53 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:10:53.566+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.053696+0000 front 2024-03-28T16:10:24.053904+0000 (oldest deadline 2024-03-28T16:10:48.753813+0000) 2024-03-28T16:10:54.052 INFO:journalctl@ceph.osd.4.smithi150.stdout:Mar 28 16:10:53 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-4[46365]: 2024-03-28T16:10:53.584+0000 7f5081200700 -1 osd.4 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:25.611238+0000 front 2024-03-28T16:10:25.611041+0000 (oldest deadline 2024-03-28T16:10:47.911231+0000) 2024-03-28T16:10:54.053 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:10:53 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:10:53.679+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:10:54.053 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:10:53 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:10:53.679+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.601392+0000 front 2024-03-28T16:10:23.601326+0000 (oldest deadline 2024-03-28T16:10:47.701522+0000) 2024-03-28T16:10:54.100 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 28 16:10:53 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-5[47486]: 2024-03-28T16:10:53.817+0000 7fd7bfc00700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.299545+0000 front 2024-03-28T16:10:16.299299+0000 (oldest deadline 2024-03-28T16:10:37.999325+0000) 2024-03-28T16:10:54.100 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 28 16:10:53 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-5[47486]: 2024-03-28T16:10:53.817+0000 7fd7bfc00700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.402160+0000 front 2024-03-28T16:10:24.402709+0000 (oldest deadline 2024-03-28T16:10:49.702229+0000) 2024-03-28T16:10:54.320 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:10:53 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:10:53.963+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.291110+0000 front 2024-03-28T16:10:16.291029+0000 (oldest deadline 2024-03-28T16:10:37.991294+0000) 2024-03-28T16:10:54.320 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:10:53 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:10:53.963+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:25.898350+0000 front 2024-03-28T16:10:25.898465+0000 (oldest deadline 2024-03-28T16:10:47.598638+0000) 2024-03-28T16:10:54.321 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:10:53 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:10:53.929+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.533614+0000 front 2024-03-28T16:10:16.534633+0000 (oldest deadline 2024-03-28T16:10:38.233619+0000) 2024-03-28T16:10:54.321 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:10:53 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:10:53.929+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:21.735742+0000 front 2024-03-28T16:10:21.735828+0000 (oldest deadline 2024-03-28T16:10:47.635538+0000) 2024-03-28T16:10:54.550 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:10:54 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:10:54.127+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.809005+0000 front 2024-03-28T16:10:24.808952+0000 (oldest deadline 2024-03-28T16:10:47.108813+0000) 2024-03-28T16:10:54.552 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:10:54 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:10:54.053+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.173988+0000 front 2024-03-28T16:10:23.173853+0000 (oldest deadline 2024-03-28T16:10:47.273946+0000) 2024-03-28T16:10:54.823 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:10:54 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:10:54.549+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:10:54.823 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:10:54 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:10:54.549+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.053696+0000 front 2024-03-28T16:10:24.053904+0000 (oldest deadline 2024-03-28T16:10:48.753813+0000) 2024-03-28T16:10:54.888 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:10:54 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:10:54.634+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:10:54.888 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:10:54 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:10:54.634+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.601392+0000 front 2024-03-28T16:10:23.601326+0000 (oldest deadline 2024-03-28T16:10:47.701522+0000) 2024-03-28T16:10:54.888 INFO:journalctl@ceph.osd.4.smithi150.stdout:Mar 28 16:10:54 smithi150 podman[68941]: 2024-03-28 16:10:54.562667502 +0000 UTC m=+5.631139814 container died 94d188816840ccf5793d52a6c25bed82cda6624184d73dbf8809e3caa3e1f03c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-4, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T16:10:55.083 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 28 16:10:54 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-5[47486]: 2024-03-28T16:10:54.822+0000 7fd7bfc00700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.299545+0000 front 2024-03-28T16:10:16.299299+0000 (oldest deadline 2024-03-28T16:10:37.999325+0000) 2024-03-28T16:10:55.083 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 28 16:10:54 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-5[47486]: 2024-03-28T16:10:54.822+0000 7fd7bfc00700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.402160+0000 front 2024-03-28T16:10:24.402709+0000 (oldest deadline 2024-03-28T16:10:49.702229+0000) 2024-03-28T16:10:55.302 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:10:55 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:10:55.044+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.173988+0000 front 2024-03-28T16:10:23.173853+0000 (oldest deadline 2024-03-28T16:10:47.273946+0000) 2024-03-28T16:10:55.302 INFO:journalctl@ceph.osd.4.smithi150.stdout:Mar 28 16:10:55 smithi150 podman[68941]: 2024-03-28 16:10:55.03316927 +0000 UTC m=+6.101641645 container cleanup 94d188816840ccf5793d52a6c25bed82cda6624184d73dbf8809e3caa3e1f03c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-4, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-28T16:10:55.303 INFO:journalctl@ceph.osd.4.smithi150.stdout:Mar 28 16:10:55 smithi150 bash[68941]: ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-4 2024-03-28T16:10:55.319 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:10:54 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:10:54.927+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.533614+0000 front 2024-03-28T16:10:16.534633+0000 (oldest deadline 2024-03-28T16:10:38.233619+0000) 2024-03-28T16:10:55.319 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:10:54 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:10:54.927+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:21.735742+0000 front 2024-03-28T16:10:21.735828+0000 (oldest deadline 2024-03-28T16:10:47.635538+0000) 2024-03-28T16:10:55.320 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:10:54 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:10:54.964+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.291110+0000 front 2024-03-28T16:10:16.291029+0000 (oldest deadline 2024-03-28T16:10:37.991294+0000) 2024-03-28T16:10:55.320 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:10:54 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:10:54.964+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:25.898350+0000 front 2024-03-28T16:10:25.898465+0000 (oldest deadline 2024-03-28T16:10:47.598638+0000) 2024-03-28T16:10:55.350 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:10:55 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:10:55.081+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.809005+0000 front 2024-03-28T16:10:24.808952+0000 (oldest deadline 2024-03-28T16:10:47.108813+0000) 2024-03-28T16:10:55.644 INFO:journalctl@ceph.osd.4.smithi150.stdout:Mar 28 16:10:55 smithi150 podman[68963]: 2024-03-28 16:10:55.353108799 +0000 UTC m=+0.777355421 container remove 94d188816840ccf5793d52a6c25bed82cda6624184d73dbf8809e3caa3e1f03c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2) 2024-03-28T16:10:55.850 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:10:55 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:10:55.552+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:10:55.850 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:10:55 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:10:55.552+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.053696+0000 front 2024-03-28T16:10:24.053904+0000 (oldest deadline 2024-03-28T16:10:48.753813+0000) 2024-03-28T16:10:55.960 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:10:55 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:10:55.644+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:10:55.960 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:10:55 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:10:55.644+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.601392+0000 front 2024-03-28T16:10:23.601326+0000 (oldest deadline 2024-03-28T16:10:47.701522+0000) 2024-03-28T16:10:56.302 INFO:journalctl@ceph.osd.4.smithi150.stdout:Mar 28 16:10:55 smithi150 podman[69052]: 2024-03-28 16:10:55.866657944 +0000 UTC m=+0.033118708 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T16:10:56.302 INFO:journalctl@ceph.osd.4.smithi150.stdout:Mar 28 16:10:55 smithi150 podman[69052]: 2024-03-28 16:10:55.981848969 +0000 UTC m=+0.148309770 container create 397a0535877f478bfc2e8886b3d8ff2ff5cc53efa39ed8d9eb3f198a7e6686d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T16:10:56.303 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:10:56 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:10:56.041+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.173988+0000 front 2024-03-28T16:10:23.173853+0000 (oldest deadline 2024-03-28T16:10:47.273946+0000) 2024-03-28T16:10:56.319 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:10:55 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:10:55.948+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.533614+0000 front 2024-03-28T16:10:16.534633+0000 (oldest deadline 2024-03-28T16:10:38.233619+0000) 2024-03-28T16:10:56.319 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:10:55 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:10:55.948+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:21.735742+0000 front 2024-03-28T16:10:21.735828+0000 (oldest deadline 2024-03-28T16:10:47.635538+0000) 2024-03-28T16:10:56.320 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:10:55 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:10:55.936+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.291110+0000 front 2024-03-28T16:10:16.291029+0000 (oldest deadline 2024-03-28T16:10:37.991294+0000) 2024-03-28T16:10:56.320 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:10:55 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:10:55.936+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:25.898350+0000 front 2024-03-28T16:10:25.898465+0000 (oldest deadline 2024-03-28T16:10:47.598638+0000) 2024-03-28T16:10:56.350 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:10:56 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:10:56.062+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.809005+0000 front 2024-03-28T16:10:24.808952+0000 (oldest deadline 2024-03-28T16:10:47.108813+0000) 2024-03-28T16:10:56.350 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 28 16:10:55 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-5[47486]: 2024-03-28T16:10:55.853+0000 7fd7bfc00700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.299545+0000 front 2024-03-28T16:10:16.299299+0000 (oldest deadline 2024-03-28T16:10:37.999325+0000) 2024-03-28T16:10:56.350 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 28 16:10:55 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-5[47486]: 2024-03-28T16:10:55.853+0000 7fd7bfc00700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.402160+0000 front 2024-03-28T16:10:24.402709+0000 (oldest deadline 2024-03-28T16:10:49.702229+0000) 2024-03-28T16:10:56.596 INFO:journalctl@ceph.osd.4.smithi150.stdout:Mar 28 16:10:56 smithi150 podman[69052]: 2024-03-28 16:10:56.313510013 +0000 UTC m=+0.479970820 container init 397a0535877f478bfc2e8886b3d8ff2ff5cc53efa39ed8d9eb3f198a7e6686d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-4-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T16:10:56.597 INFO:journalctl@ceph.osd.4.smithi150.stdout:Mar 28 16:10:56 smithi150 podman[69052]: 2024-03-28 16:10:56.324613911 +0000 UTC m=+0.491074681 container start 397a0535877f478bfc2e8886b3d8ff2ff5cc53efa39ed8d9eb3f198a7e6686d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, ceph=True, GIT_CLEAN=True) 2024-03-28T16:10:56.597 INFO:journalctl@ceph.osd.4.smithi150.stdout:Mar 28 16:10:56 smithi150 podman[69052]: 2024-03-28 16:10:56.395688314 +0000 UTC m=+0.562149161 container attach 397a0535877f478bfc2e8886b3d8ff2ff5cc53efa39ed8d9eb3f198a7e6686d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-28T16:10:56.850 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:10:56 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:10:56.514+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:10:56.850 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:10:56 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:10:56.514+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.053696+0000 front 2024-03-28T16:10:24.053904+0000 (oldest deadline 2024-03-28T16:10:48.753813+0000) 2024-03-28T16:10:56.991 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:10:56 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:10:56.595+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:10:56.991 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:10:56 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:10:56.595+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.601392+0000 front 2024-03-28T16:10:23.601326+0000 (oldest deadline 2024-03-28T16:10:47.701522+0000) 2024-03-28T16:10:56.991 INFO:journalctl@ceph.osd.4.smithi150.stdout:Mar 28 16:10:56 smithi150 podman[69052]: 2024-03-28 16:10:56.748929485 +0000 UTC m=+0.915390289 container died 397a0535877f478bfc2e8886b3d8ff2ff5cc53efa39ed8d9eb3f198a7e6686d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-4-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319) 2024-03-28T16:10:57.302 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:10:57 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:10:57.006+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.173988+0000 front 2024-03-28T16:10:23.173853+0000 (oldest deadline 2024-03-28T16:10:47.273946+0000) 2024-03-28T16:10:57.302 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:10:57 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:10:57.006+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:32.978519+0000 front 2024-03-28T16:10:32.978267+0000 (oldest deadline 2024-03-28T16:10:56.477715+0000) 2024-03-28T16:10:57.319 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:10:56 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:10:56.953+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.533614+0000 front 2024-03-28T16:10:16.534633+0000 (oldest deadline 2024-03-28T16:10:38.233619+0000) 2024-03-28T16:10:57.319 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:10:56 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:10:56.953+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:21.735742+0000 front 2024-03-28T16:10:21.735828+0000 (oldest deadline 2024-03-28T16:10:47.635538+0000) 2024-03-28T16:10:57.320 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:10:56 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:10:56.956+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.291110+0000 front 2024-03-28T16:10:16.291029+0000 (oldest deadline 2024-03-28T16:10:37.991294+0000) 2024-03-28T16:10:57.320 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:10:56 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:10:56.956+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:25.898350+0000 front 2024-03-28T16:10:25.898465+0000 (oldest deadline 2024-03-28T16:10:47.598638+0000) 2024-03-28T16:10:57.350 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:10:57 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:10:57.041+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.809005+0000 front 2024-03-28T16:10:24.808952+0000 (oldest deadline 2024-03-28T16:10:47.108813+0000) 2024-03-28T16:10:57.350 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 28 16:10:56 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-5[47486]: 2024-03-28T16:10:56.872+0000 7fd7bfc00700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.299545+0000 front 2024-03-28T16:10:16.299299+0000 (oldest deadline 2024-03-28T16:10:37.999325+0000) 2024-03-28T16:10:57.350 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 28 16:10:56 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-5[47486]: 2024-03-28T16:10:56.872+0000 7fd7bfc00700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.402160+0000 front 2024-03-28T16:10:24.402709+0000 (oldest deadline 2024-03-28T16:10:49.702229+0000) 2024-03-28T16:10:57.367 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@osd.4.service' 2024-03-28T16:10:57.471 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T16:10:57.471 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-28T16:10:57.471 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-28T16:10:57.471 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@osd.5 2024-03-28T16:10:57.830 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:10:57 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:10:57.535+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:10:57.830 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:10:57 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:10:57.535+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.053696+0000 front 2024-03-28T16:10:24.053904+0000 (oldest deadline 2024-03-28T16:10:48.753813+0000) 2024-03-28T16:10:57.830 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 28 16:10:57 smithi183 systemd[1]: Stopping Ceph osd.5 for b5f8225a-ed1a-11ee-95d0-87774f69a715... 2024-03-28T16:10:57.830 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 28 16:10:57 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-5[47486]: 2024-03-28T16:10:57.829+0000 7fd7bfc00700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.299545+0000 front 2024-03-28T16:10:16.299299+0000 (oldest deadline 2024-03-28T16:10:37.999325+0000) 2024-03-28T16:10:57.830 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 28 16:10:57 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-5[47486]: 2024-03-28T16:10:57.829+0000 7fd7bfc00700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.402160+0000 front 2024-03-28T16:10:24.402709+0000 (oldest deadline 2024-03-28T16:10:49.702229+0000) 2024-03-28T16:10:57.830 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 28 16:10:57 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-5[47486]: 2024-03-28T16:10:57.829+0000 7fd7bfc00700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:34.305365+0000 front 2024-03-28T16:10:34.305031+0000 (oldest deadline 2024-03-28T16:10:57.205045+0000) 2024-03-28T16:10:58.043 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:10:57 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:10:57.596+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:10:58.043 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:10:57 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:10:57.596+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.601392+0000 front 2024-03-28T16:10:23.601326+0000 (oldest deadline 2024-03-28T16:10:47.701522+0000) 2024-03-28T16:10:58.100 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:10:57 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:10:57.996+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.809005+0000 front 2024-03-28T16:10:24.808952+0000 (oldest deadline 2024-03-28T16:10:47.108813+0000) 2024-03-28T16:10:58.100 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 28 16:10:58 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-5[47486]: 2024-03-28T16:10:58.048+0000 7fd7c9400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T16:10:58.100 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 28 16:10:58 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-5[47486]: 2024-03-28T16:10:58.048+0000 7fd7c9400700 -1 osd.5 95 *** Got signal Terminated *** 2024-03-28T16:10:58.100 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 28 16:10:58 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-5[47486]: 2024-03-28T16:10:58.048+0000 7fd7c9400700 -1 osd.5 95 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T16:10:58.303 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:10:58 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:10:58.042+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.173988+0000 front 2024-03-28T16:10:23.173853+0000 (oldest deadline 2024-03-28T16:10:47.273946+0000) 2024-03-28T16:10:58.303 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:10:58 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:10:58.042+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:32.978519+0000 front 2024-03-28T16:10:32.978267+0000 (oldest deadline 2024-03-28T16:10:56.477715+0000) 2024-03-28T16:10:58.319 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:10:57 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:10:57.993+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.533614+0000 front 2024-03-28T16:10:16.534633+0000 (oldest deadline 2024-03-28T16:10:38.233619+0000) 2024-03-28T16:10:58.320 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:10:57 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:10:57.993+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:21.735742+0000 front 2024-03-28T16:10:21.735828+0000 (oldest deadline 2024-03-28T16:10:47.635538+0000) 2024-03-28T16:10:58.320 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:10:57 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:10:57.921+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.291110+0000 front 2024-03-28T16:10:16.291029+0000 (oldest deadline 2024-03-28T16:10:37.991294+0000) 2024-03-28T16:10:58.320 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:10:57 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:10:57.921+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:25.898350+0000 front 2024-03-28T16:10:25.898465+0000 (oldest deadline 2024-03-28T16:10:47.598638+0000) 2024-03-28T16:10:58.799 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:10:58 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:10:58.489+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:10:58.799 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:10:58 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:10:58.489+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.053696+0000 front 2024-03-28T16:10:24.053904+0000 (oldest deadline 2024-03-28T16:10:48.753813+0000) 2024-03-28T16:10:58.800 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:10:58 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:10:58.489+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:33.258214+0000 front 2024-03-28T16:10:33.258863+0000 (oldest deadline 2024-03-28T16:10:57.958232+0000) 2024-03-28T16:10:59.043 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:10:58 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:10:58.640+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:10:59.043 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:10:58 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:10:58.640+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.601392+0000 front 2024-03-28T16:10:23.601326+0000 (oldest deadline 2024-03-28T16:10:47.701522+0000) 2024-03-28T16:10:59.100 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:10:58 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:10:58.962+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.809005+0000 front 2024-03-28T16:10:24.808952+0000 (oldest deadline 2024-03-28T16:10:47.108813+0000) 2024-03-28T16:10:59.100 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 28 16:10:58 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-5[47486]: 2024-03-28T16:10:58.798+0000 7fd7bfc00700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.299545+0000 front 2024-03-28T16:10:16.299299+0000 (oldest deadline 2024-03-28T16:10:37.999325+0000) 2024-03-28T16:10:59.101 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 28 16:10:58 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-5[47486]: 2024-03-28T16:10:58.798+0000 7fd7bfc00700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.402160+0000 front 2024-03-28T16:10:24.402709+0000 (oldest deadline 2024-03-28T16:10:49.702229+0000) 2024-03-28T16:10:59.101 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 28 16:10:58 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-5[47486]: 2024-03-28T16:10:58.798+0000 7fd7bfc00700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:34.305365+0000 front 2024-03-28T16:10:34.305031+0000 (oldest deadline 2024-03-28T16:10:57.205045+0000) 2024-03-28T16:10:59.302 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:10:59 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:10:59.042+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.173988+0000 front 2024-03-28T16:10:23.173853+0000 (oldest deadline 2024-03-28T16:10:47.273946+0000) 2024-03-28T16:10:59.302 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:10:59 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:10:59.042+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:32.978519+0000 front 2024-03-28T16:10:32.978267+0000 (oldest deadline 2024-03-28T16:10:56.477715+0000) 2024-03-28T16:10:59.319 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:10:58 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:10:58.980+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.533614+0000 front 2024-03-28T16:10:16.534633+0000 (oldest deadline 2024-03-28T16:10:38.233619+0000) 2024-03-28T16:10:59.320 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:10:58 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:10:58.980+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:21.735742+0000 front 2024-03-28T16:10:21.735828+0000 (oldest deadline 2024-03-28T16:10:47.635538+0000) 2024-03-28T16:10:59.320 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:10:58 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:10:58.980+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:34.639789+0000 front 2024-03-28T16:10:34.639852+0000 (oldest deadline 2024-03-28T16:10:58.139257+0000) 2024-03-28T16:10:59.320 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:10:58 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:10:58.917+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.291110+0000 front 2024-03-28T16:10:16.291029+0000 (oldest deadline 2024-03-28T16:10:37.991294+0000) 2024-03-28T16:10:59.320 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:10:58 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:10:58.917+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:25.898350+0000 front 2024-03-28T16:10:25.898465+0000 (oldest deadline 2024-03-28T16:10:47.598638+0000) 2024-03-28T16:10:59.320 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:10:58 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:10:58.917+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.202100+0000 front 2024-03-28T16:10:35.202222+0000 (oldest deadline 2024-03-28T16:10:58.101986+0000) 2024-03-28T16:10:59.752 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:10:59 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:10:59.497+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:10:59.752 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:10:59 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:10:59.497+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.053696+0000 front 2024-03-28T16:10:24.053904+0000 (oldest deadline 2024-03-28T16:10:48.753813+0000) 2024-03-28T16:10:59.752 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:10:59 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:10:59.497+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:33.258214+0000 front 2024-03-28T16:10:33.258863+0000 (oldest deadline 2024-03-28T16:10:57.958232+0000) 2024-03-28T16:11:00.011 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 28 16:10:59 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-5[47486]: 2024-03-28T16:10:59.751+0000 7fd7bfc00700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.299545+0000 front 2024-03-28T16:10:16.299299+0000 (oldest deadline 2024-03-28T16:10:37.999325+0000) 2024-03-28T16:11:00.012 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 28 16:10:59 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-5[47486]: 2024-03-28T16:10:59.751+0000 7fd7bfc00700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.402160+0000 front 2024-03-28T16:10:24.402709+0000 (oldest deadline 2024-03-28T16:10:49.702229+0000) 2024-03-28T16:11:00.012 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 28 16:10:59 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-5[47486]: 2024-03-28T16:10:59.751+0000 7fd7bfc00700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:34.305365+0000 front 2024-03-28T16:10:34.305031+0000 (oldest deadline 2024-03-28T16:10:57.205045+0000) 2024-03-28T16:11:00.013 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:10:59 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:10:59.605+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:11:00.013 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:10:59 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:10:59.605+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.601392+0000 front 2024-03-28T16:10:23.601326+0000 (oldest deadline 2024-03-28T16:10:47.701522+0000) 2024-03-28T16:11:00.013 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:10:59 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:10:59.605+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.304820+0000 front 2024-03-28T16:10:35.304495+0000 (oldest deadline 2024-03-28T16:10:59.404389+0000) 2024-03-28T16:11:00.302 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:00 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:11:00.011+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.173988+0000 front 2024-03-28T16:10:23.173853+0000 (oldest deadline 2024-03-28T16:10:47.273946+0000) 2024-03-28T16:11:00.302 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:00 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:11:00.011+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:32.978519+0000 front 2024-03-28T16:10:32.978267+0000 (oldest deadline 2024-03-28T16:10:56.477715+0000) 2024-03-28T16:11:00.319 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:10:59 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:10:59.968+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.533614+0000 front 2024-03-28T16:10:16.534633+0000 (oldest deadline 2024-03-28T16:10:38.233619+0000) 2024-03-28T16:11:00.319 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:10:59 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:10:59.968+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:21.735742+0000 front 2024-03-28T16:10:21.735828+0000 (oldest deadline 2024-03-28T16:10:47.635538+0000) 2024-03-28T16:11:00.319 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:10:59 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:10:59.968+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:34.639789+0000 front 2024-03-28T16:10:34.639852+0000 (oldest deadline 2024-03-28T16:10:58.139257+0000) 2024-03-28T16:11:00.320 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:10:59 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:10:59.913+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.291110+0000 front 2024-03-28T16:10:16.291029+0000 (oldest deadline 2024-03-28T16:10:37.991294+0000) 2024-03-28T16:11:00.320 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:10:59 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:10:59.913+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:25.898350+0000 front 2024-03-28T16:10:25.898465+0000 (oldest deadline 2024-03-28T16:10:47.598638+0000) 2024-03-28T16:11:00.320 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:10:59 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:10:59.913+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.202100+0000 front 2024-03-28T16:10:35.202222+0000 (oldest deadline 2024-03-28T16:10:58.101986+0000) 2024-03-28T16:11:00.350 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:00 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:00.010+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.809005+0000 front 2024-03-28T16:10:24.808952+0000 (oldest deadline 2024-03-28T16:10:47.108813+0000) 2024-03-28T16:11:00.350 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:00 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:00.010+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.115020+0000 front 2024-03-28T16:10:35.114954+0000 (oldest deadline 2024-03-28T16:10:59.815004+0000) 2024-03-28T16:11:00.850 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:00 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:00.538+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:11:00.850 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:00 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:00.538+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.053696+0000 front 2024-03-28T16:10:24.053904+0000 (oldest deadline 2024-03-28T16:10:48.753813+0000) 2024-03-28T16:11:00.851 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:00 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:00.538+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:33.258214+0000 front 2024-03-28T16:10:33.258863+0000 (oldest deadline 2024-03-28T16:10:57.958232+0000) 2024-03-28T16:11:00.851 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 28 16:11:00 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-5[47486]: 2024-03-28T16:11:00.772+0000 7fd7bfc00700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.299545+0000 front 2024-03-28T16:10:16.299299+0000 (oldest deadline 2024-03-28T16:10:37.999325+0000) 2024-03-28T16:11:00.851 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 28 16:11:00 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-5[47486]: 2024-03-28T16:11:00.772+0000 7fd7bfc00700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.402160+0000 front 2024-03-28T16:10:24.402709+0000 (oldest deadline 2024-03-28T16:10:49.702229+0000) 2024-03-28T16:11:00.851 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 28 16:11:00 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-5[47486]: 2024-03-28T16:11:00.772+0000 7fd7bfc00700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:34.305365+0000 front 2024-03-28T16:10:34.305031+0000 (oldest deadline 2024-03-28T16:10:57.205045+0000) 2024-03-28T16:11:00.995 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:00 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:00.642+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:11:00.995 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:00 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:00.642+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.601392+0000 front 2024-03-28T16:10:23.601326+0000 (oldest deadline 2024-03-28T16:10:47.701522+0000) 2024-03-28T16:11:00.995 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:00 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:00.642+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.304820+0000 front 2024-03-28T16:10:35.304495+0000 (oldest deadline 2024-03-28T16:10:59.404389+0000) 2024-03-28T16:11:01.302 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:00 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:11:00.994+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.173988+0000 front 2024-03-28T16:10:23.173853+0000 (oldest deadline 2024-03-28T16:10:47.273946+0000) 2024-03-28T16:11:01.302 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:00 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:11:00.994+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:32.978519+0000 front 2024-03-28T16:10:32.978267+0000 (oldest deadline 2024-03-28T16:10:56.477715+0000) 2024-03-28T16:11:01.319 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:11:00 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:11:00.909+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.291110+0000 front 2024-03-28T16:10:16.291029+0000 (oldest deadline 2024-03-28T16:10:37.991294+0000) 2024-03-28T16:11:01.320 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:11:00 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:11:00.909+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:25.898350+0000 front 2024-03-28T16:10:25.898465+0000 (oldest deadline 2024-03-28T16:10:47.598638+0000) 2024-03-28T16:11:01.320 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:11:00 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:11:00.909+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.202100+0000 front 2024-03-28T16:10:35.202222+0000 (oldest deadline 2024-03-28T16:10:58.101986+0000) 2024-03-28T16:11:01.320 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:00 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:00.922+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.533614+0000 front 2024-03-28T16:10:16.534633+0000 (oldest deadline 2024-03-28T16:10:38.233619+0000) 2024-03-28T16:11:01.320 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:00 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:00.922+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:21.735742+0000 front 2024-03-28T16:10:21.735828+0000 (oldest deadline 2024-03-28T16:10:47.635538+0000) 2024-03-28T16:11:01.320 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:00 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:00.922+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:34.639789+0000 front 2024-03-28T16:10:34.639852+0000 (oldest deadline 2024-03-28T16:10:58.139257+0000) 2024-03-28T16:11:01.350 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:01 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:01.023+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.809005+0000 front 2024-03-28T16:10:24.808952+0000 (oldest deadline 2024-03-28T16:10:47.108813+0000) 2024-03-28T16:11:01.350 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:01 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:01.023+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.115020+0000 front 2024-03-28T16:10:35.114954+0000 (oldest deadline 2024-03-28T16:10:59.815004+0000) 2024-03-28T16:11:01.850 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:01 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:01.557+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:11:01.850 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:01 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:01.557+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.053696+0000 front 2024-03-28T16:10:24.053904+0000 (oldest deadline 2024-03-28T16:10:48.753813+0000) 2024-03-28T16:11:01.850 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:01 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:01.557+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:33.258214+0000 front 2024-03-28T16:10:33.258863+0000 (oldest deadline 2024-03-28T16:10:57.958232+0000) 2024-03-28T16:11:01.851 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 28 16:11:01 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-5[47486]: 2024-03-28T16:11:01.793+0000 7fd7bfc00700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.299545+0000 front 2024-03-28T16:10:16.299299+0000 (oldest deadline 2024-03-28T16:10:37.999325+0000) 2024-03-28T16:11:01.851 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 28 16:11:01 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-5[47486]: 2024-03-28T16:11:01.793+0000 7fd7bfc00700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.402160+0000 front 2024-03-28T16:10:24.402709+0000 (oldest deadline 2024-03-28T16:10:49.702229+0000) 2024-03-28T16:11:01.851 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 28 16:11:01 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-5[47486]: 2024-03-28T16:11:01.793+0000 7fd7bfc00700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:34.305365+0000 front 2024-03-28T16:10:34.305031+0000 (oldest deadline 2024-03-28T16:10:57.205045+0000) 2024-03-28T16:11:01.981 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:01 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:01.615+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:11:01.981 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:01 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:01.615+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.601392+0000 front 2024-03-28T16:10:23.601326+0000 (oldest deadline 2024-03-28T16:10:47.701522+0000) 2024-03-28T16:11:01.981 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:01 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:01.615+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.304820+0000 front 2024-03-28T16:10:35.304495+0000 (oldest deadline 2024-03-28T16:10:59.404389+0000) 2024-03-28T16:11:02.303 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:01 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:11:01.980+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.173988+0000 front 2024-03-28T16:10:23.173853+0000 (oldest deadline 2024-03-28T16:10:47.273946+0000) 2024-03-28T16:11:02.303 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:01 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:11:01.980+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:32.978519+0000 front 2024-03-28T16:10:32.978267+0000 (oldest deadline 2024-03-28T16:10:56.477715+0000) 2024-03-28T16:11:02.320 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:01 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:01.885+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.533614+0000 front 2024-03-28T16:10:16.534633+0000 (oldest deadline 2024-03-28T16:10:38.233619+0000) 2024-03-28T16:11:02.320 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:01 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:01.885+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:21.735742+0000 front 2024-03-28T16:10:21.735828+0000 (oldest deadline 2024-03-28T16:10:47.635538+0000) 2024-03-28T16:11:02.320 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:01 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:01.885+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:34.639789+0000 front 2024-03-28T16:10:34.639852+0000 (oldest deadline 2024-03-28T16:10:58.139257+0000) 2024-03-28T16:11:02.320 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:11:01 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:11:01.885+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.291110+0000 front 2024-03-28T16:10:16.291029+0000 (oldest deadline 2024-03-28T16:10:37.991294+0000) 2024-03-28T16:11:02.320 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:11:01 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:11:01.885+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:25.898350+0000 front 2024-03-28T16:10:25.898465+0000 (oldest deadline 2024-03-28T16:10:47.598638+0000) 2024-03-28T16:11:02.320 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:11:01 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:11:01.885+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.202100+0000 front 2024-03-28T16:10:35.202222+0000 (oldest deadline 2024-03-28T16:10:58.101986+0000) 2024-03-28T16:11:02.350 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:02 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:02.053+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.809005+0000 front 2024-03-28T16:10:24.808952+0000 (oldest deadline 2024-03-28T16:10:47.108813+0000) 2024-03-28T16:11:02.350 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:02 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:02.053+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.115020+0000 front 2024-03-28T16:10:35.114954+0000 (oldest deadline 2024-03-28T16:10:59.815004+0000) 2024-03-28T16:11:02.798 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:02 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:02.522+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:11:02.798 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:02 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:02.522+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.053696+0000 front 2024-03-28T16:10:24.053904+0000 (oldest deadline 2024-03-28T16:10:48.753813+0000) 2024-03-28T16:11:02.798 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:02 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:02.522+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:33.258214+0000 front 2024-03-28T16:10:33.258863+0000 (oldest deadline 2024-03-28T16:10:57.958232+0000) 2024-03-28T16:11:02.982 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:02 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:02.605+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:11:02.982 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:02 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:02.605+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.601392+0000 front 2024-03-28T16:10:23.601326+0000 (oldest deadline 2024-03-28T16:10:47.701522+0000) 2024-03-28T16:11:02.982 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:02 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:02.605+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.304820+0000 front 2024-03-28T16:10:35.304495+0000 (oldest deadline 2024-03-28T16:10:59.404389+0000) 2024-03-28T16:11:03.100 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:03 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:03.045+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.809005+0000 front 2024-03-28T16:10:24.808952+0000 (oldest deadline 2024-03-28T16:10:47.108813+0000) 2024-03-28T16:11:03.100 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:03 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:03.045+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.115020+0000 front 2024-03-28T16:10:35.114954+0000 (oldest deadline 2024-03-28T16:10:59.815004+0000) 2024-03-28T16:11:03.101 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 28 16:11:02 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-5[47486]: 2024-03-28T16:11:02.797+0000 7fd7bfc00700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.299545+0000 front 2024-03-28T16:10:16.299299+0000 (oldest deadline 2024-03-28T16:10:37.999325+0000) 2024-03-28T16:11:03.101 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 28 16:11:02 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-5[47486]: 2024-03-28T16:11:02.797+0000 7fd7bfc00700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.402160+0000 front 2024-03-28T16:10:24.402709+0000 (oldest deadline 2024-03-28T16:10:49.702229+0000) 2024-03-28T16:11:03.101 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 28 16:11:02 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-5[47486]: 2024-03-28T16:11:02.797+0000 7fd7bfc00700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:34.305365+0000 front 2024-03-28T16:10:34.305031+0000 (oldest deadline 2024-03-28T16:10:57.205045+0000) 2024-03-28T16:11:03.302 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:02 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:11:02.981+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.173988+0000 front 2024-03-28T16:10:23.173853+0000 (oldest deadline 2024-03-28T16:10:47.273946+0000) 2024-03-28T16:11:03.302 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:02 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:11:02.981+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:32.978519+0000 front 2024-03-28T16:10:32.978267+0000 (oldest deadline 2024-03-28T16:10:56.477715+0000) 2024-03-28T16:11:03.319 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:02 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:02.838+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.533614+0000 front 2024-03-28T16:10:16.534633+0000 (oldest deadline 2024-03-28T16:10:38.233619+0000) 2024-03-28T16:11:03.319 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:02 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:02.838+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:21.735742+0000 front 2024-03-28T16:10:21.735828+0000 (oldest deadline 2024-03-28T16:10:47.635538+0000) 2024-03-28T16:11:03.319 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:02 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:02.838+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:34.639789+0000 front 2024-03-28T16:10:34.639852+0000 (oldest deadline 2024-03-28T16:10:58.139257+0000) 2024-03-28T16:11:03.320 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:11:02 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:11:02.923+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.291110+0000 front 2024-03-28T16:10:16.291029+0000 (oldest deadline 2024-03-28T16:10:37.991294+0000) 2024-03-28T16:11:03.320 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:11:02 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:11:02.923+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:25.898350+0000 front 2024-03-28T16:10:25.898465+0000 (oldest deadline 2024-03-28T16:10:47.598638+0000) 2024-03-28T16:11:03.320 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:11:02 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:11:02.923+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.202100+0000 front 2024-03-28T16:10:35.202222+0000 (oldest deadline 2024-03-28T16:10:58.101986+0000) 2024-03-28T16:11:03.851 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 28 16:11:03 smithi183 podman[69793]: 2024-03-28 16:11:03.634133091 +0000 UTC m=+5.731273314 container died 8e103876fd53c2ff8018a8ebd3c04f3aae22699323167d8a2e42c39e67de75be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-28T16:11:03.851 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:03 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:03.537+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:11:03.851 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:03 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:03.537+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.053696+0000 front 2024-03-28T16:10:24.053904+0000 (oldest deadline 2024-03-28T16:10:48.753813+0000) 2024-03-28T16:11:03.851 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:03 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:03.537+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:33.258214+0000 front 2024-03-28T16:10:33.258863+0000 (oldest deadline 2024-03-28T16:10:57.958232+0000) 2024-03-28T16:11:03.954 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:03 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:03.593+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:11:03.954 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:03 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:03.593+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.601392+0000 front 2024-03-28T16:10:23.601326+0000 (oldest deadline 2024-03-28T16:10:47.701522+0000) 2024-03-28T16:11:03.954 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:03 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:03.593+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.304820+0000 front 2024-03-28T16:10:35.304495+0000 (oldest deadline 2024-03-28T16:10:59.404389+0000) 2024-03-28T16:11:04.268 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:04 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:04.024+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.809005+0000 front 2024-03-28T16:10:24.808952+0000 (oldest deadline 2024-03-28T16:10:47.108813+0000) 2024-03-28T16:11:04.268 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:04 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:04.024+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.115020+0000 front 2024-03-28T16:10:35.114954+0000 (oldest deadline 2024-03-28T16:10:59.815004+0000) 2024-03-28T16:11:04.268 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 28 16:11:04 smithi183 podman[69793]: 2024-03-28 16:11:04.091080628 +0000 UTC m=+6.188220883 container cleanup 8e103876fd53c2ff8018a8ebd3c04f3aae22699323167d8a2e42c39e67de75be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-28T16:11:04.268 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 28 16:11:04 smithi183 bash[69793]: ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-5 2024-03-28T16:11:04.302 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:03 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:11:03.953+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.173988+0000 front 2024-03-28T16:10:23.173853+0000 (oldest deadline 2024-03-28T16:10:47.273946+0000) 2024-03-28T16:11:04.302 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:03 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:11:03.953+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:32.978519+0000 front 2024-03-28T16:10:32.978267+0000 (oldest deadline 2024-03-28T16:10:56.477715+0000) 2024-03-28T16:11:04.320 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:11:03 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:11:03.905+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.291110+0000 front 2024-03-28T16:10:16.291029+0000 (oldest deadline 2024-03-28T16:10:37.991294+0000) 2024-03-28T16:11:04.320 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:11:03 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:11:03.905+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:25.898350+0000 front 2024-03-28T16:10:25.898465+0000 (oldest deadline 2024-03-28T16:10:47.598638+0000) 2024-03-28T16:11:04.320 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:11:03 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:11:03.905+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.202100+0000 front 2024-03-28T16:10:35.202222+0000 (oldest deadline 2024-03-28T16:10:58.101986+0000) 2024-03-28T16:11:04.320 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:03 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:03.826+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.533614+0000 front 2024-03-28T16:10:16.534633+0000 (oldest deadline 2024-03-28T16:10:38.233619+0000) 2024-03-28T16:11:04.320 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:03 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:03.826+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:21.735742+0000 front 2024-03-28T16:10:21.735828+0000 (oldest deadline 2024-03-28T16:10:47.635538+0000) 2024-03-28T16:11:04.320 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:03 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:03.826+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:34.639789+0000 front 2024-03-28T16:10:34.639852+0000 (oldest deadline 2024-03-28T16:10:58.139257+0000) 2024-03-28T16:11:04.600 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 28 16:11:04 smithi183 podman[69813]: 2024-03-28 16:11:04.267104701 +0000 UTC m=+0.622171210 container remove 8e103876fd53c2ff8018a8ebd3c04f3aae22699323167d8a2e42c39e67de75be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-28T16:11:04.600 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:04 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:04.498+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:11:04.600 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:04 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:04.498+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.053696+0000 front 2024-03-28T16:10:24.053904+0000 (oldest deadline 2024-03-28T16:10:48.753813+0000) 2024-03-28T16:11:04.600 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:04 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:04.498+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:33.258214+0000 front 2024-03-28T16:10:33.258863+0000 (oldest deadline 2024-03-28T16:10:57.958232+0000) 2024-03-28T16:11:04.989 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:04 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:04.635+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:11:04.989 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:04 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:04.635+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.601392+0000 front 2024-03-28T16:10:23.601326+0000 (oldest deadline 2024-03-28T16:10:47.701522+0000) 2024-03-28T16:11:04.989 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:04 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:04.635+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.304820+0000 front 2024-03-28T16:10:35.304495+0000 (oldest deadline 2024-03-28T16:10:59.404389+0000) 2024-03-28T16:11:05.302 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:04 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:11:04.988+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.173988+0000 front 2024-03-28T16:10:23.173853+0000 (oldest deadline 2024-03-28T16:10:47.273946+0000) 2024-03-28T16:11:05.303 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:04 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:11:04.988+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:32.978519+0000 front 2024-03-28T16:10:32.978267+0000 (oldest deadline 2024-03-28T16:10:56.477715+0000) 2024-03-28T16:11:05.320 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:11:04 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:11:04.946+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.291110+0000 front 2024-03-28T16:10:16.291029+0000 (oldest deadline 2024-03-28T16:10:37.991294+0000) 2024-03-28T16:11:05.320 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:11:04 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:11:04.946+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:25.898350+0000 front 2024-03-28T16:10:25.898465+0000 (oldest deadline 2024-03-28T16:10:47.598638+0000) 2024-03-28T16:11:05.320 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:11:04 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:11:04.946+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.202100+0000 front 2024-03-28T16:10:35.202222+0000 (oldest deadline 2024-03-28T16:10:58.101986+0000) 2024-03-28T16:11:05.320 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:04 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:04.841+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.533614+0000 front 2024-03-28T16:10:16.534633+0000 (oldest deadline 2024-03-28T16:10:38.233619+0000) 2024-03-28T16:11:05.320 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:04 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:04.841+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:21.735742+0000 front 2024-03-28T16:10:21.735828+0000 (oldest deadline 2024-03-28T16:10:47.635538+0000) 2024-03-28T16:11:05.321 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:04 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:04.841+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:34.639789+0000 front 2024-03-28T16:10:34.639852+0000 (oldest deadline 2024-03-28T16:10:58.139257+0000) 2024-03-28T16:11:05.332 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:05 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:05.001+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.809005+0000 front 2024-03-28T16:10:24.808952+0000 (oldest deadline 2024-03-28T16:10:47.108813+0000) 2024-03-28T16:11:05.332 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:05 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:05.001+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.115020+0000 front 2024-03-28T16:10:35.114954+0000 (oldest deadline 2024-03-28T16:10:59.815004+0000) 2024-03-28T16:11:05.332 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 28 16:11:05 smithi183 podman[69899]: 2024-03-28 16:11:05.076544914 +0000 UTC m=+0.126885354 container create e95d85a3e81e9621c9edf0e96c31d7f2166abb7c6db15ba6e9a6c341daa9d611 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-5-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-28T16:11:05.332 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 28 16:11:05 smithi183 podman[69899]: 2024-03-28 16:11:04.982131729 +0000 UTC m=+0.032472144 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T16:11:05.332 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 28 16:11:05 smithi183 podman[69899]: 2024-03-28 16:11:05.331707306 +0000 UTC m=+0.382047753 container init e95d85a3e81e9621c9edf0e96c31d7f2166abb7c6db15ba6e9a6c341daa9d611 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-5-deactivate, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-28T16:11:05.601 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 28 16:11:05 smithi183 podman[69899]: 2024-03-28 16:11:05.34267067 +0000 UTC m=+0.393011089 container start e95d85a3e81e9621c9edf0e96c31d7f2166abb7c6db15ba6e9a6c341daa9d611 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-5-deactivate, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-28T16:11:05.601 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 28 16:11:05 smithi183 podman[69899]: 2024-03-28 16:11:05.420172229 +0000 UTC m=+0.470512709 container attach e95d85a3e81e9621c9edf0e96c31d7f2166abb7c6db15ba6e9a6c341daa9d611 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-28T16:11:05.601 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:05 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:05.542+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:11:05.601 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:05 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:05.542+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.053696+0000 front 2024-03-28T16:10:24.053904+0000 (oldest deadline 2024-03-28T16:10:48.753813+0000) 2024-03-28T16:11:05.601 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:05 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:05.542+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:33.258214+0000 front 2024-03-28T16:10:33.258863+0000 (oldest deadline 2024-03-28T16:10:57.958232+0000) 2024-03-28T16:11:05.993 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:05 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:05.636+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:11:05.993 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:05 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:05.636+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.601392+0000 front 2024-03-28T16:10:23.601326+0000 (oldest deadline 2024-03-28T16:10:47.701522+0000) 2024-03-28T16:11:05.993 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:05 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:05.636+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.304820+0000 front 2024-03-28T16:10:35.304495+0000 (oldest deadline 2024-03-28T16:10:59.404389+0000) 2024-03-28T16:11:06.049 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 28 16:11:05 smithi183 podman[69899]: 2024-03-28 16:11:05.777147523 +0000 UTC m=+0.827487963 container died e95d85a3e81e9621c9edf0e96c31d7f2166abb7c6db15ba6e9a6c341daa9d611 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-5-deactivate, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T16:11:06.302 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:05 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:11:05.992+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.173988+0000 front 2024-03-28T16:10:23.173853+0000 (oldest deadline 2024-03-28T16:10:47.273946+0000) 2024-03-28T16:11:06.303 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:05 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:11:05.992+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:32.978519+0000 front 2024-03-28T16:10:32.978267+0000 (oldest deadline 2024-03-28T16:10:56.477715+0000) 2024-03-28T16:11:06.303 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:05 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:11:05.992+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:43.480465+0000 front 2024-03-28T16:10:43.480862+0000 (oldest deadline 2024-03-28T16:11:05.780413+0000) 2024-03-28T16:11:06.320 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:11:05 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:11:05.910+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.291110+0000 front 2024-03-28T16:10:16.291029+0000 (oldest deadline 2024-03-28T16:10:37.991294+0000) 2024-03-28T16:11:06.320 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:11:05 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:11:05.910+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:25.898350+0000 front 2024-03-28T16:10:25.898465+0000 (oldest deadline 2024-03-28T16:10:47.598638+0000) 2024-03-28T16:11:06.320 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:11:05 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:11:05.910+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.202100+0000 front 2024-03-28T16:10:35.202222+0000 (oldest deadline 2024-03-28T16:10:58.101986+0000) 2024-03-28T16:11:06.320 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:05 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:05.873+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.533614+0000 front 2024-03-28T16:10:16.534633+0000 (oldest deadline 2024-03-28T16:10:38.233619+0000) 2024-03-28T16:11:06.320 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:05 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:05.873+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:21.735742+0000 front 2024-03-28T16:10:21.735828+0000 (oldest deadline 2024-03-28T16:10:47.635538+0000) 2024-03-28T16:11:06.320 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:05 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:05.873+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:34.639789+0000 front 2024-03-28T16:10:34.639852+0000 (oldest deadline 2024-03-28T16:10:58.139257+0000) 2024-03-28T16:11:06.350 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:06 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:06.048+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.809005+0000 front 2024-03-28T16:10:24.808952+0000 (oldest deadline 2024-03-28T16:10:47.108813+0000) 2024-03-28T16:11:06.350 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:06 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:06.048+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.115020+0000 front 2024-03-28T16:10:35.114954+0000 (oldest deadline 2024-03-28T16:10:59.815004+0000) 2024-03-28T16:11:06.350 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:06 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:06.048+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:40.317024+0000 front 2024-03-28T16:10:40.317425+0000 (oldest deadline 2024-03-28T16:11:05.617169+0000) 2024-03-28T16:11:06.485 DEBUG:teuthology.orchestra.run.smithi183:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@osd.5.service' 2024-03-28T16:11:06.601 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:06 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:06.550+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:11:06.601 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:06 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:06.550+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.053696+0000 front 2024-03-28T16:10:24.053904+0000 (oldest deadline 2024-03-28T16:10:48.753813+0000) 2024-03-28T16:11:06.601 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:06 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:06.550+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:33.258214+0000 front 2024-03-28T16:10:33.258863+0000 (oldest deadline 2024-03-28T16:10:57.958232+0000) 2024-03-28T16:11:06.606 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T16:11:06.606 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-28T16:11:06.606 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-28T16:11:06.606 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@osd.6 2024-03-28T16:11:07.002 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:06 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:06.671+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:11:07.002 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:06 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:06.671+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.601392+0000 front 2024-03-28T16:10:23.601326+0000 (oldest deadline 2024-03-28T16:10:47.701522+0000) 2024-03-28T16:11:07.002 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:06 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:06.671+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.304820+0000 front 2024-03-28T16:10:35.304495+0000 (oldest deadline 2024-03-28T16:10:59.404389+0000) 2024-03-28T16:11:07.069 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:11:06 smithi067 systemd[1]: Stopping Ceph osd.6 for b5f8225a-ed1a-11ee-95d0-87774f69a715... 2024-03-28T16:11:07.070 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:11:06 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:11:06.927+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.291110+0000 front 2024-03-28T16:10:16.291029+0000 (oldest deadline 2024-03-28T16:10:37.991294+0000) 2024-03-28T16:11:07.070 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:11:06 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:11:06.927+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:25.898350+0000 front 2024-03-28T16:10:25.898465+0000 (oldest deadline 2024-03-28T16:10:47.598638+0000) 2024-03-28T16:11:07.070 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:11:06 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:11:06.927+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.202100+0000 front 2024-03-28T16:10:35.202222+0000 (oldest deadline 2024-03-28T16:10:58.101986+0000) 2024-03-28T16:11:07.070 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:11:06 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:11:06.927+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:41.004400+0000 front 2024-03-28T16:10:41.004243+0000 (oldest deadline 2024-03-28T16:11:06.304407+0000) 2024-03-28T16:11:07.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:06 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:06.840+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.533614+0000 front 2024-03-28T16:10:16.534633+0000 (oldest deadline 2024-03-28T16:10:38.233619+0000) 2024-03-28T16:11:07.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:06 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:06.840+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:21.735742+0000 front 2024-03-28T16:10:21.735828+0000 (oldest deadline 2024-03-28T16:10:47.635538+0000) 2024-03-28T16:11:07.071 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:06 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:06.840+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:34.639789+0000 front 2024-03-28T16:10:34.639852+0000 (oldest deadline 2024-03-28T16:10:58.139257+0000) 2024-03-28T16:11:07.302 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:07 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:11:07.001+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.173988+0000 front 2024-03-28T16:10:23.173853+0000 (oldest deadline 2024-03-28T16:10:47.273946+0000) 2024-03-28T16:11:07.302 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:07 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:11:07.001+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:32.978519+0000 front 2024-03-28T16:10:32.978267+0000 (oldest deadline 2024-03-28T16:10:56.477715+0000) 2024-03-28T16:11:07.303 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:07 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:11:07.001+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:43.480465+0000 front 2024-03-28T16:10:43.480862+0000 (oldest deadline 2024-03-28T16:11:05.780413+0000) 2024-03-28T16:11:07.350 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:07 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:07.031+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.809005+0000 front 2024-03-28T16:10:24.808952+0000 (oldest deadline 2024-03-28T16:10:47.108813+0000) 2024-03-28T16:11:07.350 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:07 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:07.031+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.115020+0000 front 2024-03-28T16:10:35.114954+0000 (oldest deadline 2024-03-28T16:10:59.815004+0000) 2024-03-28T16:11:07.351 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:07 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:07.031+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:40.317024+0000 front 2024-03-28T16:10:40.317425+0000 (oldest deadline 2024-03-28T16:11:05.617169+0000) 2024-03-28T16:11:07.569 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:11:07 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:11:07.175+0000 7ff6e7200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T16:11:07.570 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:11:07 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:11:07.175+0000 7ff6e7200700 -1 osd.6 95 *** Got signal Terminated *** 2024-03-28T16:11:07.570 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:11:07 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:11:07.175+0000 7ff6e7200700 -1 osd.6 95 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T16:11:07.850 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:07 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:07.565+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:11:07.851 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:07 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:07.565+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.053696+0000 front 2024-03-28T16:10:24.053904+0000 (oldest deadline 2024-03-28T16:10:48.753813+0000) 2024-03-28T16:11:07.851 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:07 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:07.565+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:33.258214+0000 front 2024-03-28T16:10:33.258863+0000 (oldest deadline 2024-03-28T16:10:57.958232+0000) 2024-03-28T16:11:08.005 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:07 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:07.668+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:11:08.006 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:07 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:07.668+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.601392+0000 front 2024-03-28T16:10:23.601326+0000 (oldest deadline 2024-03-28T16:10:47.701522+0000) 2024-03-28T16:11:08.006 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:07 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:07.668+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.304820+0000 front 2024-03-28T16:10:35.304495+0000 (oldest deadline 2024-03-28T16:10:59.404389+0000) 2024-03-28T16:11:08.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:07 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:07.799+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.533614+0000 front 2024-03-28T16:10:16.534633+0000 (oldest deadline 2024-03-28T16:10:38.233619+0000) 2024-03-28T16:11:08.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:07 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:07.799+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:21.735742+0000 front 2024-03-28T16:10:21.735828+0000 (oldest deadline 2024-03-28T16:10:47.635538+0000) 2024-03-28T16:11:08.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:07 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:07.799+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:34.639789+0000 front 2024-03-28T16:10:34.639852+0000 (oldest deadline 2024-03-28T16:10:58.139257+0000) 2024-03-28T16:11:08.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:07 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:07.799+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:41.041611+0000 front 2024-03-28T16:10:41.041696+0000 (oldest deadline 2024-03-28T16:11:06.941328+0000) 2024-03-28T16:11:08.070 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:11:07 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:11:07.905+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.291110+0000 front 2024-03-28T16:10:16.291029+0000 (oldest deadline 2024-03-28T16:10:37.991294+0000) 2024-03-28T16:11:08.070 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:11:07 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:11:07.905+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:25.898350+0000 front 2024-03-28T16:10:25.898465+0000 (oldest deadline 2024-03-28T16:10:47.598638+0000) 2024-03-28T16:11:08.070 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:11:07 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:11:07.905+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.202100+0000 front 2024-03-28T16:10:35.202222+0000 (oldest deadline 2024-03-28T16:10:58.101986+0000) 2024-03-28T16:11:08.070 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:11:07 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:11:07.905+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:41.004400+0000 front 2024-03-28T16:10:41.004243+0000 (oldest deadline 2024-03-28T16:11:06.304407+0000) 2024-03-28T16:11:08.286 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:08 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:08.027+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.809005+0000 front 2024-03-28T16:10:24.808952+0000 (oldest deadline 2024-03-28T16:10:47.108813+0000) 2024-03-28T16:11:08.286 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:08 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:08.027+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.115020+0000 front 2024-03-28T16:10:35.114954+0000 (oldest deadline 2024-03-28T16:10:59.815004+0000) 2024-03-28T16:11:08.286 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:08 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:08.027+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:40.317024+0000 front 2024-03-28T16:10:40.317425+0000 (oldest deadline 2024-03-28T16:11:05.617169+0000) 2024-03-28T16:11:08.302 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:08 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:11:08.004+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.173988+0000 front 2024-03-28T16:10:23.173853+0000 (oldest deadline 2024-03-28T16:10:47.273946+0000) 2024-03-28T16:11:08.302 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:08 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:11:08.004+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:32.978519+0000 front 2024-03-28T16:10:32.978267+0000 (oldest deadline 2024-03-28T16:10:56.477715+0000) 2024-03-28T16:11:08.302 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:08 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:11:08.004+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:43.480465+0000 front 2024-03-28T16:10:43.480862+0000 (oldest deadline 2024-03-28T16:11:05.780413+0000) 2024-03-28T16:11:08.850 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:08 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:08.560+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:11:08.850 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:08 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:08.560+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.053696+0000 front 2024-03-28T16:10:24.053904+0000 (oldest deadline 2024-03-28T16:10:48.753813+0000) 2024-03-28T16:11:08.850 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:08 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:08.560+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:33.258214+0000 front 2024-03-28T16:10:33.258863+0000 (oldest deadline 2024-03-28T16:10:57.958232+0000) 2024-03-28T16:11:08.985 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:08 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:08.634+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:11:08.985 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:08 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:08.634+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.601392+0000 front 2024-03-28T16:10:23.601326+0000 (oldest deadline 2024-03-28T16:10:47.701522+0000) 2024-03-28T16:11:08.985 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:08 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:08.635+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.304820+0000 front 2024-03-28T16:10:35.304495+0000 (oldest deadline 2024-03-28T16:10:59.404389+0000) 2024-03-28T16:11:08.985 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:08 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:08.635+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:44.107005+0000 front 2024-03-28T16:10:44.106857+0000 (oldest deadline 2024-03-28T16:11:08.206356+0000) 2024-03-28T16:11:09.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:08 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:08.760+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.533614+0000 front 2024-03-28T16:10:16.534633+0000 (oldest deadline 2024-03-28T16:10:38.233619+0000) 2024-03-28T16:11:09.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:08 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:08.760+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:21.735742+0000 front 2024-03-28T16:10:21.735828+0000 (oldest deadline 2024-03-28T16:10:47.635538+0000) 2024-03-28T16:11:09.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:08 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:08.760+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:34.639789+0000 front 2024-03-28T16:10:34.639852+0000 (oldest deadline 2024-03-28T16:10:58.139257+0000) 2024-03-28T16:11:09.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:08 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:08.760+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:41.041611+0000 front 2024-03-28T16:10:41.041696+0000 (oldest deadline 2024-03-28T16:11:06.941328+0000) 2024-03-28T16:11:09.071 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:11:08 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:11:08.919+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.291110+0000 front 2024-03-28T16:10:16.291029+0000 (oldest deadline 2024-03-28T16:10:37.991294+0000) 2024-03-28T16:11:09.071 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:11:08 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:11:08.919+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:25.898350+0000 front 2024-03-28T16:10:25.898465+0000 (oldest deadline 2024-03-28T16:10:47.598638+0000) 2024-03-28T16:11:09.071 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:11:08 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:11:08.919+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.202100+0000 front 2024-03-28T16:10:35.202222+0000 (oldest deadline 2024-03-28T16:10:58.101986+0000) 2024-03-28T16:11:09.071 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:11:08 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:11:08.919+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:41.004400+0000 front 2024-03-28T16:10:41.004243+0000 (oldest deadline 2024-03-28T16:11:06.304407+0000) 2024-03-28T16:11:09.302 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:08 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:11:08.984+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.173988+0000 front 2024-03-28T16:10:23.173853+0000 (oldest deadline 2024-03-28T16:10:47.273946+0000) 2024-03-28T16:11:09.302 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:08 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:11:08.984+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:32.978519+0000 front 2024-03-28T16:10:32.978267+0000 (oldest deadline 2024-03-28T16:10:56.477715+0000) 2024-03-28T16:11:09.302 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:08 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:11:08.984+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:43.480465+0000 front 2024-03-28T16:10:43.480862+0000 (oldest deadline 2024-03-28T16:11:05.780413+0000) 2024-03-28T16:11:09.350 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:09 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:09.001+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.809005+0000 front 2024-03-28T16:10:24.808952+0000 (oldest deadline 2024-03-28T16:10:47.108813+0000) 2024-03-28T16:11:09.350 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:09 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:09.001+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.115020+0000 front 2024-03-28T16:10:35.114954+0000 (oldest deadline 2024-03-28T16:10:59.815004+0000) 2024-03-28T16:11:09.351 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:09 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:09.001+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:40.317024+0000 front 2024-03-28T16:10:40.317425+0000 (oldest deadline 2024-03-28T16:11:05.617169+0000) 2024-03-28T16:11:09.850 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:09 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:09.520+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:11:09.850 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:09 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:09.520+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.053696+0000 front 2024-03-28T16:10:24.053904+0000 (oldest deadline 2024-03-28T16:10:48.753813+0000) 2024-03-28T16:11:09.851 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:09 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:09.520+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:33.258214+0000 front 2024-03-28T16:10:33.258863+0000 (oldest deadline 2024-03-28T16:10:57.958232+0000) 2024-03-28T16:11:09.937 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:09 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:09.588+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:11:09.937 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:09 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:09.588+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.601392+0000 front 2024-03-28T16:10:23.601326+0000 (oldest deadline 2024-03-28T16:10:47.701522+0000) 2024-03-28T16:11:09.937 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:09 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:09.588+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.304820+0000 front 2024-03-28T16:10:35.304495+0000 (oldest deadline 2024-03-28T16:10:59.404389+0000) 2024-03-28T16:11:09.937 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:09 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:09.588+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:44.107005+0000 front 2024-03-28T16:10:44.106857+0000 (oldest deadline 2024-03-28T16:11:08.206356+0000) 2024-03-28T16:11:10.069 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:09 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:09.715+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.533614+0000 front 2024-03-28T16:10:16.534633+0000 (oldest deadline 2024-03-28T16:10:38.233619+0000) 2024-03-28T16:11:10.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:09 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:09.715+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:21.735742+0000 front 2024-03-28T16:10:21.735828+0000 (oldest deadline 2024-03-28T16:10:47.635538+0000) 2024-03-28T16:11:10.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:09 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:09.715+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:34.639789+0000 front 2024-03-28T16:10:34.639852+0000 (oldest deadline 2024-03-28T16:10:58.139257+0000) 2024-03-28T16:11:10.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:09 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:09.715+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:41.041611+0000 front 2024-03-28T16:10:41.041696+0000 (oldest deadline 2024-03-28T16:11:06.941328+0000) 2024-03-28T16:11:10.070 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:11:09 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:11:09.947+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.291110+0000 front 2024-03-28T16:10:16.291029+0000 (oldest deadline 2024-03-28T16:10:37.991294+0000) 2024-03-28T16:11:10.070 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:11:09 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:11:09.947+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:25.898350+0000 front 2024-03-28T16:10:25.898465+0000 (oldest deadline 2024-03-28T16:10:47.598638+0000) 2024-03-28T16:11:10.070 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:11:09 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:11:09.947+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.202100+0000 front 2024-03-28T16:10:35.202222+0000 (oldest deadline 2024-03-28T16:10:58.101986+0000) 2024-03-28T16:11:10.070 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:11:09 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:11:09.947+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:41.004400+0000 front 2024-03-28T16:10:41.004243+0000 (oldest deadline 2024-03-28T16:11:06.304407+0000) 2024-03-28T16:11:10.302 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:09 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:11:09.936+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.173988+0000 front 2024-03-28T16:10:23.173853+0000 (oldest deadline 2024-03-28T16:10:47.273946+0000) 2024-03-28T16:11:10.302 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:09 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:11:09.936+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:32.978519+0000 front 2024-03-28T16:10:32.978267+0000 (oldest deadline 2024-03-28T16:10:56.477715+0000) 2024-03-28T16:11:10.302 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:09 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:11:09.936+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:43.480465+0000 front 2024-03-28T16:10:43.480862+0000 (oldest deadline 2024-03-28T16:11:05.780413+0000) 2024-03-28T16:11:10.350 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:10 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:10.010+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.809005+0000 front 2024-03-28T16:10:24.808952+0000 (oldest deadline 2024-03-28T16:10:47.108813+0000) 2024-03-28T16:11:10.350 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:10 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:10.010+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.115020+0000 front 2024-03-28T16:10:35.114954+0000 (oldest deadline 2024-03-28T16:10:59.815004+0000) 2024-03-28T16:11:10.350 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:10 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:10.010+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:40.317024+0000 front 2024-03-28T16:10:40.317425+0000 (oldest deadline 2024-03-28T16:11:05.617169+0000) 2024-03-28T16:11:10.850 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:10 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:10.501+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:11:10.850 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:10 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:10.501+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.053696+0000 front 2024-03-28T16:10:24.053904+0000 (oldest deadline 2024-03-28T16:10:48.753813+0000) 2024-03-28T16:11:10.850 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:10 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:10.501+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:33.258214+0000 front 2024-03-28T16:10:33.258863+0000 (oldest deadline 2024-03-28T16:10:57.958232+0000) 2024-03-28T16:11:10.964 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:10 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:10.604+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:11:10.964 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:10 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:10.604+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.601392+0000 front 2024-03-28T16:10:23.601326+0000 (oldest deadline 2024-03-28T16:10:47.701522+0000) 2024-03-28T16:11:10.964 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:10 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:10.604+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.304820+0000 front 2024-03-28T16:10:35.304495+0000 (oldest deadline 2024-03-28T16:10:59.404389+0000) 2024-03-28T16:11:10.964 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:10 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:10.604+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:44.107005+0000 front 2024-03-28T16:10:44.106857+0000 (oldest deadline 2024-03-28T16:11:08.206356+0000) 2024-03-28T16:11:10.996 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:10 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:10.716+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.533614+0000 front 2024-03-28T16:10:16.534633+0000 (oldest deadline 2024-03-28T16:10:38.233619+0000) 2024-03-28T16:11:10.996 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:10 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:10.716+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:21.735742+0000 front 2024-03-28T16:10:21.735828+0000 (oldest deadline 2024-03-28T16:10:47.635538+0000) 2024-03-28T16:11:10.996 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:10 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:10.716+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:34.639789+0000 front 2024-03-28T16:10:34.639852+0000 (oldest deadline 2024-03-28T16:10:58.139257+0000) 2024-03-28T16:11:10.997 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:10 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:10.716+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:41.041611+0000 front 2024-03-28T16:10:41.041696+0000 (oldest deadline 2024-03-28T16:11:06.941328+0000) 2024-03-28T16:11:11.302 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:10 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:11:10.986+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.173988+0000 front 2024-03-28T16:10:23.173853+0000 (oldest deadline 2024-03-28T16:10:47.273946+0000) 2024-03-28T16:11:11.302 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:10 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:11:10.986+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:32.978519+0000 front 2024-03-28T16:10:32.978267+0000 (oldest deadline 2024-03-28T16:10:56.477715+0000) 2024-03-28T16:11:11.302 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:10 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:11:10.986+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:43.480465+0000 front 2024-03-28T16:10:43.480862+0000 (oldest deadline 2024-03-28T16:11:05.780413+0000) 2024-03-28T16:11:11.319 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:11:10 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:11:10.995+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.291110+0000 front 2024-03-28T16:10:16.291029+0000 (oldest deadline 2024-03-28T16:10:37.991294+0000) 2024-03-28T16:11:11.320 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:11:10 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:11:10.995+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:25.898350+0000 front 2024-03-28T16:10:25.898465+0000 (oldest deadline 2024-03-28T16:10:47.598638+0000) 2024-03-28T16:11:11.320 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:11:10 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:11:10.995+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.202100+0000 front 2024-03-28T16:10:35.202222+0000 (oldest deadline 2024-03-28T16:10:58.101986+0000) 2024-03-28T16:11:11.320 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:11:10 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:11:10.995+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:41.004400+0000 front 2024-03-28T16:10:41.004243+0000 (oldest deadline 2024-03-28T16:11:06.304407+0000) 2024-03-28T16:11:11.350 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:10 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:10.997+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.809005+0000 front 2024-03-28T16:10:24.808952+0000 (oldest deadline 2024-03-28T16:10:47.108813+0000) 2024-03-28T16:11:11.350 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:10 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:10.997+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.115020+0000 front 2024-03-28T16:10:35.114954+0000 (oldest deadline 2024-03-28T16:10:59.815004+0000) 2024-03-28T16:11:11.350 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:10 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:10.997+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:40.317024+0000 front 2024-03-28T16:10:40.317425+0000 (oldest deadline 2024-03-28T16:11:05.617169+0000) 2024-03-28T16:11:11.850 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:11 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:11.460+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:11:11.850 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:11 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:11.460+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.053696+0000 front 2024-03-28T16:10:24.053904+0000 (oldest deadline 2024-03-28T16:10:48.753813+0000) 2024-03-28T16:11:11.850 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:11 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:11.460+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:33.258214+0000 front 2024-03-28T16:10:33.258863+0000 (oldest deadline 2024-03-28T16:10:57.958232+0000) 2024-03-28T16:11:11.968 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:11 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:11.721+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.533614+0000 front 2024-03-28T16:10:16.534633+0000 (oldest deadline 2024-03-28T16:10:38.233619+0000) 2024-03-28T16:11:11.968 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:11 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:11.721+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:21.735742+0000 front 2024-03-28T16:10:21.735828+0000 (oldest deadline 2024-03-28T16:10:47.635538+0000) 2024-03-28T16:11:11.968 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:11 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:11.721+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:34.639789+0000 front 2024-03-28T16:10:34.639852+0000 (oldest deadline 2024-03-28T16:10:58.139257+0000) 2024-03-28T16:11:11.968 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:11 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:11.721+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:41.041611+0000 front 2024-03-28T16:10:41.041696+0000 (oldest deadline 2024-03-28T16:11:06.941328+0000) 2024-03-28T16:11:11.989 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:11 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:11.602+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:11:11.989 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:11 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:11.602+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.601392+0000 front 2024-03-28T16:10:23.601326+0000 (oldest deadline 2024-03-28T16:10:47.701522+0000) 2024-03-28T16:11:11.990 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:11 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:11.602+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.304820+0000 front 2024-03-28T16:10:35.304495+0000 (oldest deadline 2024-03-28T16:10:59.404389+0000) 2024-03-28T16:11:11.990 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:11 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:11.602+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:44.107005+0000 front 2024-03-28T16:10:44.106857+0000 (oldest deadline 2024-03-28T16:11:08.206356+0000) 2024-03-28T16:11:12.302 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:11 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:11:11.988+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.173988+0000 front 2024-03-28T16:10:23.173853+0000 (oldest deadline 2024-03-28T16:10:47.273946+0000) 2024-03-28T16:11:12.302 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:11 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:11:11.988+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:32.978519+0000 front 2024-03-28T16:10:32.978267+0000 (oldest deadline 2024-03-28T16:10:56.477715+0000) 2024-03-28T16:11:12.302 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:11 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:11:11.988+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:43.480465+0000 front 2024-03-28T16:10:43.480862+0000 (oldest deadline 2024-03-28T16:11:05.780413+0000) 2024-03-28T16:11:12.319 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:11:11 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:11:11.967+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.291110+0000 front 2024-03-28T16:10:16.291029+0000 (oldest deadline 2024-03-28T16:10:37.991294+0000) 2024-03-28T16:11:12.319 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:11:11 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:11:11.967+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:25.898350+0000 front 2024-03-28T16:10:25.898465+0000 (oldest deadline 2024-03-28T16:10:47.598638+0000) 2024-03-28T16:11:12.319 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:11:11 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:11:11.967+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.202100+0000 front 2024-03-28T16:10:35.202222+0000 (oldest deadline 2024-03-28T16:10:58.101986+0000) 2024-03-28T16:11:12.319 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:11:11 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6[59907]: 2024-03-28T16:11:11.967+0000 7ff6dda00700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:41.004400+0000 front 2024-03-28T16:10:41.004243+0000 (oldest deadline 2024-03-28T16:11:06.304407+0000) 2024-03-28T16:11:12.350 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:11 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:11.970+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.809005+0000 front 2024-03-28T16:10:24.808952+0000 (oldest deadline 2024-03-28T16:10:47.108813+0000) 2024-03-28T16:11:12.350 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:11 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:11.970+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.115020+0000 front 2024-03-28T16:10:35.114954+0000 (oldest deadline 2024-03-28T16:10:59.815004+0000) 2024-03-28T16:11:12.350 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:11 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:11.970+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:40.317024+0000 front 2024-03-28T16:10:40.317425+0000 (oldest deadline 2024-03-28T16:11:05.617169+0000) 2024-03-28T16:11:12.803 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:12 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:12.581+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:11:12.803 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:12 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:12.581+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.601392+0000 front 2024-03-28T16:10:23.601326+0000 (oldest deadline 2024-03-28T16:10:47.701522+0000) 2024-03-28T16:11:12.803 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:12 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:12.581+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.304820+0000 front 2024-03-28T16:10:35.304495+0000 (oldest deadline 2024-03-28T16:10:59.404389+0000) 2024-03-28T16:11:12.803 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:12 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:12.581+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:44.107005+0000 front 2024-03-28T16:10:44.106857+0000 (oldest deadline 2024-03-28T16:11:08.206356+0000) 2024-03-28T16:11:12.850 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:12 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:12.461+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:11:12.850 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:12 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:12.461+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.053696+0000 front 2024-03-28T16:10:24.053904+0000 (oldest deadline 2024-03-28T16:10:48.753813+0000) 2024-03-28T16:11:12.850 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:12 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:12.461+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:33.258214+0000 front 2024-03-28T16:10:33.258863+0000 (oldest deadline 2024-03-28T16:10:57.958232+0000) 2024-03-28T16:11:12.924 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:11:12 smithi067 podman[91512]: 2024-03-28 16:11:12.614838394 +0000 UTC m=+5.585565208 container died 65fb70e8afa491584b7fbb42762972e0a1036beeef907b630e6f9d813b6ebc4e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319) 2024-03-28T16:11:12.924 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:12 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:12.729+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.533614+0000 front 2024-03-28T16:10:16.534633+0000 (oldest deadline 2024-03-28T16:10:38.233619+0000) 2024-03-28T16:11:12.924 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:12 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:12.729+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:21.735742+0000 front 2024-03-28T16:10:21.735828+0000 (oldest deadline 2024-03-28T16:10:47.635538+0000) 2024-03-28T16:11:12.924 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:12 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:12.729+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:34.639789+0000 front 2024-03-28T16:10:34.639852+0000 (oldest deadline 2024-03-28T16:10:58.139257+0000) 2024-03-28T16:11:12.925 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:12 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:12.729+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:41.041611+0000 front 2024-03-28T16:10:41.041696+0000 (oldest deadline 2024-03-28T16:11:06.941328+0000) 2024-03-28T16:11:13.271 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:11:13 smithi067 podman[91512]: 2024-03-28 16:11:13.060839453 +0000 UTC m=+6.031566305 container cleanup 65fb70e8afa491584b7fbb42762972e0a1036beeef907b630e6f9d813b6ebc4e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240319) 2024-03-28T16:11:13.271 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:11:13 smithi067 bash[91512]: ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6 2024-03-28T16:11:13.302 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:13 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:11:13.038+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.173988+0000 front 2024-03-28T16:10:23.173853+0000 (oldest deadline 2024-03-28T16:10:47.273946+0000) 2024-03-28T16:11:13.302 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:13 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:11:13.038+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:32.978519+0000 front 2024-03-28T16:10:32.978267+0000 (oldest deadline 2024-03-28T16:10:56.477715+0000) 2024-03-28T16:11:13.302 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:13 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:11:13.038+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:43.480465+0000 front 2024-03-28T16:10:43.480862+0000 (oldest deadline 2024-03-28T16:11:05.780413+0000) 2024-03-28T16:11:13.350 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:12 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:12.923+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.809005+0000 front 2024-03-28T16:10:24.808952+0000 (oldest deadline 2024-03-28T16:10:47.108813+0000) 2024-03-28T16:11:13.350 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:12 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:12.923+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.115020+0000 front 2024-03-28T16:10:35.114954+0000 (oldest deadline 2024-03-28T16:10:59.815004+0000) 2024-03-28T16:11:13.350 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:12 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:12.923+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:40.317024+0000 front 2024-03-28T16:10:40.317425+0000 (oldest deadline 2024-03-28T16:11:05.617169+0000) 2024-03-28T16:11:13.569 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:11:13 smithi067 podman[91542]: 2024-03-28 16:11:13.270045045 +0000 UTC m=+0.641098239 container remove 65fb70e8afa491584b7fbb42762972e0a1036beeef907b630e6f9d813b6ebc4e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319) 2024-03-28T16:11:13.850 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:13 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:13.438+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:11:13.850 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:13 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:13.438+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.053696+0000 front 2024-03-28T16:10:24.053904+0000 (oldest deadline 2024-03-28T16:10:48.753813+0000) 2024-03-28T16:11:13.850 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:13 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:13.438+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:33.258214+0000 front 2024-03-28T16:10:33.258863+0000 (oldest deadline 2024-03-28T16:10:57.958232+0000) 2024-03-28T16:11:14.052 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:13 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:13.574+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:11:14.053 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:13 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:13.574+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.601392+0000 front 2024-03-28T16:10:23.601326+0000 (oldest deadline 2024-03-28T16:10:47.701522+0000) 2024-03-28T16:11:14.053 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:13 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:13.574+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.304820+0000 front 2024-03-28T16:10:35.304495+0000 (oldest deadline 2024-03-28T16:10:59.404389+0000) 2024-03-28T16:11:14.053 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:13 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:13.574+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:44.107005+0000 front 2024-03-28T16:10:44.106857+0000 (oldest deadline 2024-03-28T16:11:08.206356+0000) 2024-03-28T16:11:14.069 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:13 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:13.702+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.533614+0000 front 2024-03-28T16:10:16.534633+0000 (oldest deadline 2024-03-28T16:10:38.233619+0000) 2024-03-28T16:11:14.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:13 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:13.702+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:21.735742+0000 front 2024-03-28T16:10:21.735828+0000 (oldest deadline 2024-03-28T16:10:47.635538+0000) 2024-03-28T16:11:14.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:13 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:13.702+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:34.639789+0000 front 2024-03-28T16:10:34.639852+0000 (oldest deadline 2024-03-28T16:10:58.139257+0000) 2024-03-28T16:11:14.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:13 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:13.702+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:41.041611+0000 front 2024-03-28T16:10:41.041696+0000 (oldest deadline 2024-03-28T16:11:06.941328+0000) 2024-03-28T16:11:14.350 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:13 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:13.875+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.809005+0000 front 2024-03-28T16:10:24.808952+0000 (oldest deadline 2024-03-28T16:10:47.108813+0000) 2024-03-28T16:11:14.350 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:13 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:13.875+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.115020+0000 front 2024-03-28T16:10:35.114954+0000 (oldest deadline 2024-03-28T16:10:59.815004+0000) 2024-03-28T16:11:14.351 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:13 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:13.875+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:40.317024+0000 front 2024-03-28T16:10:40.317425+0000 (oldest deadline 2024-03-28T16:11:05.617169+0000) 2024-03-28T16:11:14.427 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:11:14 smithi067 podman[91633]: 2024-03-28 16:11:14.056254864 +0000 UTC m=+0.039107185 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T16:11:14.427 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:11:14 smithi067 podman[91633]: 2024-03-28 16:11:14.162560596 +0000 UTC m=+0.145412954 container create b3a9f126b7a1415e17e7ed9b6fe43a60ef6106437b31714db0600cfaaa29659c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6-deactivate, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-28T16:11:14.427 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:11:14 smithi067 podman[91633]: 2024-03-28 16:11:14.425525474 +0000 UTC m=+0.408377862 container init b3a9f126b7a1415e17e7ed9b6fe43a60ef6106437b31714db0600cfaaa29659c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6-deactivate, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-28T16:11:14.552 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:14 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:11:14.077+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.173988+0000 front 2024-03-28T16:10:23.173853+0000 (oldest deadline 2024-03-28T16:10:47.273946+0000) 2024-03-28T16:11:14.552 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:14 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:11:14.077+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:32.978519+0000 front 2024-03-28T16:10:32.978267+0000 (oldest deadline 2024-03-28T16:10:56.477715+0000) 2024-03-28T16:11:14.552 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:14 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:11:14.077+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:43.480465+0000 front 2024-03-28T16:10:43.480862+0000 (oldest deadline 2024-03-28T16:11:05.780413+0000) 2024-03-28T16:11:14.720 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:11:14 smithi067 podman[91633]: 2024-03-28 16:11:14.439536742 +0000 UTC m=+0.422389101 container start b3a9f126b7a1415e17e7ed9b6fe43a60ef6106437b31714db0600cfaaa29659c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-28T16:11:14.720 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:11:14 smithi067 podman[91633]: 2024-03-28 16:11:14.555470875 +0000 UTC m=+0.538323244 container attach b3a9f126b7a1415e17e7ed9b6fe43a60ef6106437b31714db0600cfaaa29659c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True) 2024-03-28T16:11:14.850 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:14 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:14.424+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:11:14.850 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:14 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:14.424+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.053696+0000 front 2024-03-28T16:10:24.053904+0000 (oldest deadline 2024-03-28T16:10:48.753813+0000) 2024-03-28T16:11:14.850 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:14 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:14.424+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:33.258214+0000 front 2024-03-28T16:10:33.258863+0000 (oldest deadline 2024-03-28T16:10:57.958232+0000) 2024-03-28T16:11:15.052 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:14 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:14.594+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:11:15.052 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:14 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:14.594+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.601392+0000 front 2024-03-28T16:10:23.601326+0000 (oldest deadline 2024-03-28T16:10:47.701522+0000) 2024-03-28T16:11:15.053 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:14 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:14.594+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.304820+0000 front 2024-03-28T16:10:35.304495+0000 (oldest deadline 2024-03-28T16:10:59.404389+0000) 2024-03-28T16:11:15.053 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:14 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:14.594+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:44.107005+0000 front 2024-03-28T16:10:44.106857+0000 (oldest deadline 2024-03-28T16:11:08.206356+0000) 2024-03-28T16:11:15.069 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 28 16:11:14 smithi067 podman[91633]: 2024-03-28 16:11:14.886253076 +0000 UTC m=+0.869105435 container died b3a9f126b7a1415e17e7ed9b6fe43a60ef6106437b31714db0600cfaaa29659c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-6-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T16:11:15.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:14 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:14.719+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.533614+0000 front 2024-03-28T16:10:16.534633+0000 (oldest deadline 2024-03-28T16:10:38.233619+0000) 2024-03-28T16:11:15.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:14 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:14.719+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:21.735742+0000 front 2024-03-28T16:10:21.735828+0000 (oldest deadline 2024-03-28T16:10:47.635538+0000) 2024-03-28T16:11:15.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:14 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:14.719+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:34.639789+0000 front 2024-03-28T16:10:34.639852+0000 (oldest deadline 2024-03-28T16:10:58.139257+0000) 2024-03-28T16:11:15.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:14 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:14.719+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:41.041611+0000 front 2024-03-28T16:10:41.041696+0000 (oldest deadline 2024-03-28T16:11:06.941328+0000) 2024-03-28T16:11:15.350 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:14 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:14.900+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.809005+0000 front 2024-03-28T16:10:24.808952+0000 (oldest deadline 2024-03-28T16:10:47.108813+0000) 2024-03-28T16:11:15.350 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:14 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:14.900+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.115020+0000 front 2024-03-28T16:10:35.114954+0000 (oldest deadline 2024-03-28T16:10:59.815004+0000) 2024-03-28T16:11:15.350 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:14 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:14.900+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:40.317024+0000 front 2024-03-28T16:10:40.317425+0000 (oldest deadline 2024-03-28T16:11:05.617169+0000) 2024-03-28T16:11:15.514 DEBUG:teuthology.orchestra.run.smithi067:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@osd.6.service' 2024-03-28T16:11:15.552 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:15 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:11:15.067+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.173988+0000 front 2024-03-28T16:10:23.173853+0000 (oldest deadline 2024-03-28T16:10:47.273946+0000) 2024-03-28T16:11:15.552 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:15 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:11:15.067+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:32.978519+0000 front 2024-03-28T16:10:32.978267+0000 (oldest deadline 2024-03-28T16:10:56.477715+0000) 2024-03-28T16:11:15.553 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:15 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:11:15.067+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:43.480465+0000 front 2024-03-28T16:10:43.480862+0000 (oldest deadline 2024-03-28T16:11:05.780413+0000) 2024-03-28T16:11:15.642 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T16:11:15.642 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-28T16:11:15.642 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-28T16:11:15.642 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@osd.7 2024-03-28T16:11:15.850 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:15 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:15.409+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:11:15.850 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:15 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:15.409+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.053696+0000 front 2024-03-28T16:10:24.053904+0000 (oldest deadline 2024-03-28T16:10:48.753813+0000) 2024-03-28T16:11:15.850 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:15 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:15.409+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:33.258214+0000 front 2024-03-28T16:10:33.258863+0000 (oldest deadline 2024-03-28T16:10:57.958232+0000) 2024-03-28T16:11:16.052 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:15 smithi150 systemd[1]: Stopping Ceph osd.7 for b5f8225a-ed1a-11ee-95d0-87774f69a715... 2024-03-28T16:11:16.053 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:15 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:15.644+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:11:16.053 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:15 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:15.644+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.601392+0000 front 2024-03-28T16:10:23.601326+0000 (oldest deadline 2024-03-28T16:10:47.701522+0000) 2024-03-28T16:11:16.053 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:15 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:15.644+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.304820+0000 front 2024-03-28T16:10:35.304495+0000 (oldest deadline 2024-03-28T16:10:59.404389+0000) 2024-03-28T16:11:16.053 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:15 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:15.644+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:44.107005+0000 front 2024-03-28T16:10:44.106857+0000 (oldest deadline 2024-03-28T16:11:08.206356+0000) 2024-03-28T16:11:16.069 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:15 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:15.729+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.533614+0000 front 2024-03-28T16:10:16.534633+0000 (oldest deadline 2024-03-28T16:10:38.233619+0000) 2024-03-28T16:11:16.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:15 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:15.729+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:21.735742+0000 front 2024-03-28T16:10:21.735828+0000 (oldest deadline 2024-03-28T16:10:47.635538+0000) 2024-03-28T16:11:16.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:15 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:15.729+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:34.639789+0000 front 2024-03-28T16:10:34.639852+0000 (oldest deadline 2024-03-28T16:10:58.139257+0000) 2024-03-28T16:11:16.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:15 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:15.729+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:41.041611+0000 front 2024-03-28T16:10:41.041696+0000 (oldest deadline 2024-03-28T16:11:06.941328+0000) 2024-03-28T16:11:16.350 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:15 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:15.933+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.809005+0000 front 2024-03-28T16:10:24.808952+0000 (oldest deadline 2024-03-28T16:10:47.108813+0000) 2024-03-28T16:11:16.351 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:15 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:15.933+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.115020+0000 front 2024-03-28T16:10:35.114954+0000 (oldest deadline 2024-03-28T16:10:59.815004+0000) 2024-03-28T16:11:16.351 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:15 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:15.933+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:40.317024+0000 front 2024-03-28T16:10:40.317425+0000 (oldest deadline 2024-03-28T16:11:05.617169+0000) 2024-03-28T16:11:16.351 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:15 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:15.933+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:49.619830+0000 front 2024-03-28T16:10:49.620058+0000 (oldest deadline 2024-03-28T16:11:14.919627+0000) 2024-03-28T16:11:16.553 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:16 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:11:16.102+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.173988+0000 front 2024-03-28T16:10:23.173853+0000 (oldest deadline 2024-03-28T16:10:47.273946+0000) 2024-03-28T16:11:16.553 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:16 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:11:16.102+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:32.978519+0000 front 2024-03-28T16:10:32.978267+0000 (oldest deadline 2024-03-28T16:10:56.477715+0000) 2024-03-28T16:11:16.553 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:16 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:11:16.102+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:43.480465+0000 front 2024-03-28T16:10:43.480862+0000 (oldest deadline 2024-03-28T16:11:05.780413+0000) 2024-03-28T16:11:16.553 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:16 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:11:16.191+0000 7f425ee00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T16:11:16.554 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:16 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:11:16.191+0000 7f425ee00700 -1 osd.7 95 *** Got signal Terminated *** 2024-03-28T16:11:16.554 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:16 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:11:16.191+0000 7f425ee00700 -1 osd.7 95 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T16:11:16.850 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:16 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:16.406+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:11:16.850 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:16 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:16.406+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.053696+0000 front 2024-03-28T16:10:24.053904+0000 (oldest deadline 2024-03-28T16:10:48.753813+0000) 2024-03-28T16:11:16.850 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:16 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:16.406+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:33.258214+0000 front 2024-03-28T16:10:33.258863+0000 (oldest deadline 2024-03-28T16:10:57.958232+0000) 2024-03-28T16:11:17.053 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:16 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:16.611+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:11:17.053 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:16 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:16.611+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.601392+0000 front 2024-03-28T16:10:23.601326+0000 (oldest deadline 2024-03-28T16:10:47.701522+0000) 2024-03-28T16:11:17.053 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:16 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:16.611+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.304820+0000 front 2024-03-28T16:10:35.304495+0000 (oldest deadline 2024-03-28T16:10:59.404389+0000) 2024-03-28T16:11:17.053 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:16 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:16.611+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:44.107005+0000 front 2024-03-28T16:10:44.106857+0000 (oldest deadline 2024-03-28T16:11:08.206356+0000) 2024-03-28T16:11:17.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:16 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:16.778+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.533614+0000 front 2024-03-28T16:10:16.534633+0000 (oldest deadline 2024-03-28T16:10:38.233619+0000) 2024-03-28T16:11:17.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:16 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:16.778+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:21.735742+0000 front 2024-03-28T16:10:21.735828+0000 (oldest deadline 2024-03-28T16:10:47.635538+0000) 2024-03-28T16:11:17.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:16 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:16.778+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:34.639789+0000 front 2024-03-28T16:10:34.639852+0000 (oldest deadline 2024-03-28T16:10:58.139257+0000) 2024-03-28T16:11:17.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:16 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:16.778+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:41.041611+0000 front 2024-03-28T16:10:41.041696+0000 (oldest deadline 2024-03-28T16:11:06.941328+0000) 2024-03-28T16:11:17.350 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:16 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:16.890+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.809005+0000 front 2024-03-28T16:10:24.808952+0000 (oldest deadline 2024-03-28T16:10:47.108813+0000) 2024-03-28T16:11:17.351 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:16 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:16.890+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.115020+0000 front 2024-03-28T16:10:35.114954+0000 (oldest deadline 2024-03-28T16:10:59.815004+0000) 2024-03-28T16:11:17.351 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:16 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:16.890+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:40.317024+0000 front 2024-03-28T16:10:40.317425+0000 (oldest deadline 2024-03-28T16:11:05.617169+0000) 2024-03-28T16:11:17.351 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:16 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:16.890+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:49.619830+0000 front 2024-03-28T16:10:49.620058+0000 (oldest deadline 2024-03-28T16:11:14.919627+0000) 2024-03-28T16:11:17.552 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:17 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:11:17.091+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.173988+0000 front 2024-03-28T16:10:23.173853+0000 (oldest deadline 2024-03-28T16:10:47.273946+0000) 2024-03-28T16:11:17.552 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:17 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:11:17.091+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:32.978519+0000 front 2024-03-28T16:10:32.978267+0000 (oldest deadline 2024-03-28T16:10:56.477715+0000) 2024-03-28T16:11:17.552 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:17 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:11:17.091+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:43.480465+0000 front 2024-03-28T16:10:43.480862+0000 (oldest deadline 2024-03-28T16:11:05.780413+0000) 2024-03-28T16:11:17.553 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:17 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:11:17.091+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:51.082602+0000 front 2024-03-28T16:10:51.082344+0000 (oldest deadline 2024-03-28T16:11:16.982675+0000) 2024-03-28T16:11:17.850 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:17 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:17.447+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:11:17.850 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:17 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:17.447+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.053696+0000 front 2024-03-28T16:10:24.053904+0000 (oldest deadline 2024-03-28T16:10:48.753813+0000) 2024-03-28T16:11:17.850 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:17 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:17.447+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:33.258214+0000 front 2024-03-28T16:10:33.258863+0000 (oldest deadline 2024-03-28T16:10:57.958232+0000) 2024-03-28T16:11:18.052 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:17 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:17.610+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:11:18.053 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:17 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:17.610+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.601392+0000 front 2024-03-28T16:10:23.601326+0000 (oldest deadline 2024-03-28T16:10:47.701522+0000) 2024-03-28T16:11:18.053 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:17 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:17.610+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.304820+0000 front 2024-03-28T16:10:35.304495+0000 (oldest deadline 2024-03-28T16:10:59.404389+0000) 2024-03-28T16:11:18.053 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:17 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:17.610+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:44.107005+0000 front 2024-03-28T16:10:44.106857+0000 (oldest deadline 2024-03-28T16:11:08.206356+0000) 2024-03-28T16:11:18.069 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:17 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:17.760+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.533614+0000 front 2024-03-28T16:10:16.534633+0000 (oldest deadline 2024-03-28T16:10:38.233619+0000) 2024-03-28T16:11:18.069 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:17 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:17.760+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:21.735742+0000 front 2024-03-28T16:10:21.735828+0000 (oldest deadline 2024-03-28T16:10:47.635538+0000) 2024-03-28T16:11:18.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:17 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:17.760+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:34.639789+0000 front 2024-03-28T16:10:34.639852+0000 (oldest deadline 2024-03-28T16:10:58.139257+0000) 2024-03-28T16:11:18.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:17 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:17.760+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:41.041611+0000 front 2024-03-28T16:10:41.041696+0000 (oldest deadline 2024-03-28T16:11:06.941328+0000) 2024-03-28T16:11:18.350 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:17 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:17.876+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.809005+0000 front 2024-03-28T16:10:24.808952+0000 (oldest deadline 2024-03-28T16:10:47.108813+0000) 2024-03-28T16:11:18.351 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:17 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:17.876+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.115020+0000 front 2024-03-28T16:10:35.114954+0000 (oldest deadline 2024-03-28T16:10:59.815004+0000) 2024-03-28T16:11:18.351 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:17 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:17.876+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:40.317024+0000 front 2024-03-28T16:10:40.317425+0000 (oldest deadline 2024-03-28T16:11:05.617169+0000) 2024-03-28T16:11:18.351 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:17 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:17.876+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:49.619830+0000 front 2024-03-28T16:10:49.620058+0000 (oldest deadline 2024-03-28T16:11:14.919627+0000) 2024-03-28T16:11:18.552 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:18 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:11:18.108+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.173988+0000 front 2024-03-28T16:10:23.173853+0000 (oldest deadline 2024-03-28T16:10:47.273946+0000) 2024-03-28T16:11:18.553 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:18 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:11:18.108+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:32.978519+0000 front 2024-03-28T16:10:32.978267+0000 (oldest deadline 2024-03-28T16:10:56.477715+0000) 2024-03-28T16:11:18.553 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:18 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:11:18.108+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:43.480465+0000 front 2024-03-28T16:10:43.480862+0000 (oldest deadline 2024-03-28T16:11:05.780413+0000) 2024-03-28T16:11:18.553 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:18 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:11:18.108+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:51.082602+0000 front 2024-03-28T16:10:51.082344+0000 (oldest deadline 2024-03-28T16:11:16.982675+0000) 2024-03-28T16:11:18.841 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:18 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:18.422+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:11:18.842 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:18 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:18.422+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.053696+0000 front 2024-03-28T16:10:24.053904+0000 (oldest deadline 2024-03-28T16:10:48.753813+0000) 2024-03-28T16:11:18.842 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:18 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:18.422+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:33.258214+0000 front 2024-03-28T16:10:33.258863+0000 (oldest deadline 2024-03-28T16:10:57.958232+0000) 2024-03-28T16:11:19.053 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:18 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:18.646+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:11:19.053 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:18 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:18.646+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.601392+0000 front 2024-03-28T16:10:23.601326+0000 (oldest deadline 2024-03-28T16:10:47.701522+0000) 2024-03-28T16:11:19.054 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:18 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:18.646+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.304820+0000 front 2024-03-28T16:10:35.304495+0000 (oldest deadline 2024-03-28T16:10:59.404389+0000) 2024-03-28T16:11:19.054 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:18 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:18.646+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:44.107005+0000 front 2024-03-28T16:10:44.106857+0000 (oldest deadline 2024-03-28T16:11:08.206356+0000) 2024-03-28T16:11:19.069 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:18 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:18.725+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.533614+0000 front 2024-03-28T16:10:16.534633+0000 (oldest deadline 2024-03-28T16:10:38.233619+0000) 2024-03-28T16:11:19.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:18 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:18.725+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:21.735742+0000 front 2024-03-28T16:10:21.735828+0000 (oldest deadline 2024-03-28T16:10:47.635538+0000) 2024-03-28T16:11:19.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:18 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:18.725+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:34.639789+0000 front 2024-03-28T16:10:34.639852+0000 (oldest deadline 2024-03-28T16:10:58.139257+0000) 2024-03-28T16:11:19.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:18 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:18.725+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:41.041611+0000 front 2024-03-28T16:10:41.041696+0000 (oldest deadline 2024-03-28T16:11:06.941328+0000) 2024-03-28T16:11:19.100 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:18 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:18.840+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.809005+0000 front 2024-03-28T16:10:24.808952+0000 (oldest deadline 2024-03-28T16:10:47.108813+0000) 2024-03-28T16:11:19.100 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:18 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:18.840+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.115020+0000 front 2024-03-28T16:10:35.114954+0000 (oldest deadline 2024-03-28T16:10:59.815004+0000) 2024-03-28T16:11:19.101 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:18 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:18.840+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:40.317024+0000 front 2024-03-28T16:10:40.317425+0000 (oldest deadline 2024-03-28T16:11:05.617169+0000) 2024-03-28T16:11:19.101 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:18 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:18.840+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:49.619830+0000 front 2024-03-28T16:10:49.620058+0000 (oldest deadline 2024-03-28T16:11:14.919627+0000) 2024-03-28T16:11:19.552 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:19 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:11:19.126+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.173988+0000 front 2024-03-28T16:10:23.173853+0000 (oldest deadline 2024-03-28T16:10:47.273946+0000) 2024-03-28T16:11:19.552 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:19 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:11:19.126+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:32.978519+0000 front 2024-03-28T16:10:32.978267+0000 (oldest deadline 2024-03-28T16:10:56.477715+0000) 2024-03-28T16:11:19.552 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:19 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:11:19.126+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:43.480465+0000 front 2024-03-28T16:10:43.480862+0000 (oldest deadline 2024-03-28T16:11:05.780413+0000) 2024-03-28T16:11:19.552 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:19 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:11:19.126+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:51.082602+0000 front 2024-03-28T16:10:51.082344+0000 (oldest deadline 2024-03-28T16:11:16.982675+0000) 2024-03-28T16:11:19.600 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:19 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:19.393+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:11:19.601 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:19 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:19.393+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.053696+0000 front 2024-03-28T16:10:24.053904+0000 (oldest deadline 2024-03-28T16:10:48.753813+0000) 2024-03-28T16:11:19.601 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:19 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:19.393+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:33.258214+0000 front 2024-03-28T16:10:33.258863+0000 (oldest deadline 2024-03-28T16:10:57.958232+0000) 2024-03-28T16:11:19.601 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:19 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:19.393+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:53.262996+0000 front 2024-03-28T16:10:53.262892+0000 (oldest deadline 2024-03-28T16:11:18.463901+0000) 2024-03-28T16:11:20.053 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:19 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:19.669+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:11:20.053 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:19 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:19.669+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.601392+0000 front 2024-03-28T16:10:23.601326+0000 (oldest deadline 2024-03-28T16:10:47.701522+0000) 2024-03-28T16:11:20.053 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:19 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:19.669+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.304820+0000 front 2024-03-28T16:10:35.304495+0000 (oldest deadline 2024-03-28T16:10:59.404389+0000) 2024-03-28T16:11:20.053 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:19 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:19.669+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:44.107005+0000 front 2024-03-28T16:10:44.106857+0000 (oldest deadline 2024-03-28T16:11:08.206356+0000) 2024-03-28T16:11:20.053 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:19 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:19.669+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:52.908191+0000 front 2024-03-28T16:10:52.908270+0000 (oldest deadline 2024-03-28T16:11:18.808335+0000) 2024-03-28T16:11:20.069 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:19 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:19.742+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.533614+0000 front 2024-03-28T16:10:16.534633+0000 (oldest deadline 2024-03-28T16:10:38.233619+0000) 2024-03-28T16:11:20.069 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:19 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:19.742+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:21.735742+0000 front 2024-03-28T16:10:21.735828+0000 (oldest deadline 2024-03-28T16:10:47.635538+0000) 2024-03-28T16:11:20.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:19 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:19.742+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:34.639789+0000 front 2024-03-28T16:10:34.639852+0000 (oldest deadline 2024-03-28T16:10:58.139257+0000) 2024-03-28T16:11:20.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:19 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:19.742+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:41.041611+0000 front 2024-03-28T16:10:41.041696+0000 (oldest deadline 2024-03-28T16:11:06.941328+0000) 2024-03-28T16:11:20.350 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:19 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:19.864+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.809005+0000 front 2024-03-28T16:10:24.808952+0000 (oldest deadline 2024-03-28T16:10:47.108813+0000) 2024-03-28T16:11:20.351 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:19 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:19.864+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.115020+0000 front 2024-03-28T16:10:35.114954+0000 (oldest deadline 2024-03-28T16:10:59.815004+0000) 2024-03-28T16:11:20.351 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:19 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:19.864+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:40.317024+0000 front 2024-03-28T16:10:40.317425+0000 (oldest deadline 2024-03-28T16:11:05.617169+0000) 2024-03-28T16:11:20.351 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:19 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:19.864+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:49.619830+0000 front 2024-03-28T16:10:49.620058+0000 (oldest deadline 2024-03-28T16:11:14.919627+0000) 2024-03-28T16:11:20.552 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:20 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:11:20.115+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.173988+0000 front 2024-03-28T16:10:23.173853+0000 (oldest deadline 2024-03-28T16:10:47.273946+0000) 2024-03-28T16:11:20.552 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:20 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:11:20.115+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:32.978519+0000 front 2024-03-28T16:10:32.978267+0000 (oldest deadline 2024-03-28T16:10:56.477715+0000) 2024-03-28T16:11:20.552 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:20 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:11:20.115+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:43.480465+0000 front 2024-03-28T16:10:43.480862+0000 (oldest deadline 2024-03-28T16:11:05.780413+0000) 2024-03-28T16:11:20.552 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:20 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:11:20.115+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:51.082602+0000 front 2024-03-28T16:10:51.082344+0000 (oldest deadline 2024-03-28T16:11:16.982675+0000) 2024-03-28T16:11:20.833 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:20 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:20.379+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:11:20.848 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:20 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:20.379+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.053696+0000 front 2024-03-28T16:10:24.053904+0000 (oldest deadline 2024-03-28T16:10:48.753813+0000) 2024-03-28T16:11:20.849 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:20 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:20.379+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:33.258214+0000 front 2024-03-28T16:10:33.258863+0000 (oldest deadline 2024-03-28T16:10:57.958232+0000) 2024-03-28T16:11:20.849 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:20 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:20.379+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:53.262996+0000 front 2024-03-28T16:10:53.262892+0000 (oldest deadline 2024-03-28T16:11:18.463901+0000) 2024-03-28T16:11:21.053 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:20 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:20.719+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:11:21.053 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:20 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:20.719+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.601392+0000 front 2024-03-28T16:10:23.601326+0000 (oldest deadline 2024-03-28T16:10:47.701522+0000) 2024-03-28T16:11:21.053 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:20 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:20.719+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.304820+0000 front 2024-03-28T16:10:35.304495+0000 (oldest deadline 2024-03-28T16:10:59.404389+0000) 2024-03-28T16:11:21.053 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:20 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:20.719+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:44.107005+0000 front 2024-03-28T16:10:44.106857+0000 (oldest deadline 2024-03-28T16:11:08.206356+0000) 2024-03-28T16:11:21.053 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:20 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:20.719+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:52.908191+0000 front 2024-03-28T16:10:52.908270+0000 (oldest deadline 2024-03-28T16:11:18.808335+0000) 2024-03-28T16:11:21.069 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:20 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:20.718+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.533614+0000 front 2024-03-28T16:10:16.534633+0000 (oldest deadline 2024-03-28T16:10:38.233619+0000) 2024-03-28T16:11:21.069 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:20 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:20.718+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:21.735742+0000 front 2024-03-28T16:10:21.735828+0000 (oldest deadline 2024-03-28T16:10:47.635538+0000) 2024-03-28T16:11:21.069 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:20 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:20.718+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:34.639789+0000 front 2024-03-28T16:10:34.639852+0000 (oldest deadline 2024-03-28T16:10:58.139257+0000) 2024-03-28T16:11:21.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:20 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:20.718+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:41.041611+0000 front 2024-03-28T16:10:41.041696+0000 (oldest deadline 2024-03-28T16:11:06.941328+0000) 2024-03-28T16:11:21.100 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:20 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:20.831+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.809005+0000 front 2024-03-28T16:10:24.808952+0000 (oldest deadline 2024-03-28T16:10:47.108813+0000) 2024-03-28T16:11:21.101 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:20 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:20.831+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.115020+0000 front 2024-03-28T16:10:35.114954+0000 (oldest deadline 2024-03-28T16:10:59.815004+0000) 2024-03-28T16:11:21.101 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:20 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:20.831+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:40.317024+0000 front 2024-03-28T16:10:40.317425+0000 (oldest deadline 2024-03-28T16:11:05.617169+0000) 2024-03-28T16:11:21.101 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:20 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:20.831+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:49.619830+0000 front 2024-03-28T16:10:49.620058+0000 (oldest deadline 2024-03-28T16:11:14.919627+0000) 2024-03-28T16:11:21.552 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:21 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:11:21.155+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.173988+0000 front 2024-03-28T16:10:23.173853+0000 (oldest deadline 2024-03-28T16:10:47.273946+0000) 2024-03-28T16:11:21.552 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:21 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:11:21.155+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:32.978519+0000 front 2024-03-28T16:10:32.978267+0000 (oldest deadline 2024-03-28T16:10:56.477715+0000) 2024-03-28T16:11:21.552 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:21 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:11:21.155+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:43.480465+0000 front 2024-03-28T16:10:43.480862+0000 (oldest deadline 2024-03-28T16:11:05.780413+0000) 2024-03-28T16:11:21.553 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:21 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7[53449]: 2024-03-28T16:11:21.155+0000 7f4255600700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:51.082602+0000 front 2024-03-28T16:10:51.082344+0000 (oldest deadline 2024-03-28T16:11:16.982675+0000) 2024-03-28T16:11:21.796 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:21 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:21.423+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:11:21.796 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:21 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:21.423+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.053696+0000 front 2024-03-28T16:10:24.053904+0000 (oldest deadline 2024-03-28T16:10:48.753813+0000) 2024-03-28T16:11:21.796 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:21 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:21.423+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:33.258214+0000 front 2024-03-28T16:10:33.258863+0000 (oldest deadline 2024-03-28T16:10:57.958232+0000) 2024-03-28T16:11:21.797 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:21 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:21.423+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:53.262996+0000 front 2024-03-28T16:10:53.262892+0000 (oldest deadline 2024-03-28T16:11:18.463901+0000) 2024-03-28T16:11:22.052 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:21 smithi150 podman[69182]: 2024-03-28 16:11:21.75425027 +0000 UTC m=+5.706650761 container died 53c7cd410f369bb8df927376c1ce9245937bc146516d6bbc04e8f747199b229d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T16:11:22.053 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:21 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:21.736+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:11:22.053 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:21 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:21.736+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.601392+0000 front 2024-03-28T16:10:23.601326+0000 (oldest deadline 2024-03-28T16:10:47.701522+0000) 2024-03-28T16:11:22.053 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:21 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:21.736+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.304820+0000 front 2024-03-28T16:10:35.304495+0000 (oldest deadline 2024-03-28T16:10:59.404389+0000) 2024-03-28T16:11:22.053 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:21 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:21.736+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:44.107005+0000 front 2024-03-28T16:10:44.106857+0000 (oldest deadline 2024-03-28T16:11:08.206356+0000) 2024-03-28T16:11:22.053 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:21 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:21.736+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:52.908191+0000 front 2024-03-28T16:10:52.908270+0000 (oldest deadline 2024-03-28T16:11:18.808335+0000) 2024-03-28T16:11:22.069 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:21 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:21.717+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.533614+0000 front 2024-03-28T16:10:16.534633+0000 (oldest deadline 2024-03-28T16:10:38.233619+0000) 2024-03-28T16:11:22.069 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:21 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:21.717+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:21.735742+0000 front 2024-03-28T16:10:21.735828+0000 (oldest deadline 2024-03-28T16:10:47.635538+0000) 2024-03-28T16:11:22.069 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:21 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:21.717+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:34.639789+0000 front 2024-03-28T16:10:34.639852+0000 (oldest deadline 2024-03-28T16:10:58.139257+0000) 2024-03-28T16:11:22.069 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:21 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:21.717+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:41.041611+0000 front 2024-03-28T16:10:41.041696+0000 (oldest deadline 2024-03-28T16:11:06.941328+0000) 2024-03-28T16:11:22.100 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:21 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:21.795+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.809005+0000 front 2024-03-28T16:10:24.808952+0000 (oldest deadline 2024-03-28T16:10:47.108813+0000) 2024-03-28T16:11:22.100 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:21 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:21.795+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.115020+0000 front 2024-03-28T16:10:35.114954+0000 (oldest deadline 2024-03-28T16:10:59.815004+0000) 2024-03-28T16:11:22.100 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:21 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:21.795+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:40.317024+0000 front 2024-03-28T16:10:40.317425+0000 (oldest deadline 2024-03-28T16:11:05.617169+0000) 2024-03-28T16:11:22.101 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:21 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:21.795+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:49.619830+0000 front 2024-03-28T16:10:49.620058+0000 (oldest deadline 2024-03-28T16:11:14.919627+0000) 2024-03-28T16:11:22.336 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:22 smithi150 podman[69182]: 2024-03-28 16:11:22.168190394 +0000 UTC m=+6.120590914 container cleanup 53c7cd410f369bb8df927376c1ce9245937bc146516d6bbc04e8f747199b229d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-28T16:11:22.336 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:22 smithi150 bash[69182]: ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7 2024-03-28T16:11:22.629 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:22 smithi150 podman[69201]: 2024-03-28 16:11:22.335878219 +0000 UTC m=+0.566683498 container remove 53c7cd410f369bb8df927376c1ce9245937bc146516d6bbc04e8f747199b229d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True) 2024-03-28T16:11:22.754 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:22 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:22.420+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:11:22.754 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:22 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:22.420+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.053696+0000 front 2024-03-28T16:10:24.053904+0000 (oldest deadline 2024-03-28T16:10:48.753813+0000) 2024-03-28T16:11:22.754 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:22 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:22.420+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:33.258214+0000 front 2024-03-28T16:10:33.258863+0000 (oldest deadline 2024-03-28T16:10:57.958232+0000) 2024-03-28T16:11:22.754 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:22 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:22.420+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:53.262996+0000 front 2024-03-28T16:10:53.262892+0000 (oldest deadline 2024-03-28T16:11:18.463901+0000) 2024-03-28T16:11:23.052 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:22 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:22.785+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:11:23.052 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:22 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:22.785+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.601392+0000 front 2024-03-28T16:10:23.601326+0000 (oldest deadline 2024-03-28T16:10:47.701522+0000) 2024-03-28T16:11:23.053 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:22 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:22.785+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.304820+0000 front 2024-03-28T16:10:35.304495+0000 (oldest deadline 2024-03-28T16:10:59.404389+0000) 2024-03-28T16:11:23.053 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:22 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:22.785+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:44.107005+0000 front 2024-03-28T16:10:44.106857+0000 (oldest deadline 2024-03-28T16:11:08.206356+0000) 2024-03-28T16:11:23.053 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:22 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:22.785+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:52.908191+0000 front 2024-03-28T16:10:52.908270+0000 (oldest deadline 2024-03-28T16:11:18.808335+0000) 2024-03-28T16:11:23.069 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:22 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:22.684+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.533614+0000 front 2024-03-28T16:10:16.534633+0000 (oldest deadline 2024-03-28T16:10:38.233619+0000) 2024-03-28T16:11:23.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:22 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:22.684+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:21.735742+0000 front 2024-03-28T16:10:21.735828+0000 (oldest deadline 2024-03-28T16:10:47.635538+0000) 2024-03-28T16:11:23.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:22 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:22.684+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:34.639789+0000 front 2024-03-28T16:10:34.639852+0000 (oldest deadline 2024-03-28T16:10:58.139257+0000) 2024-03-28T16:11:23.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:22 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:22.684+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:41.041611+0000 front 2024-03-28T16:10:41.041696+0000 (oldest deadline 2024-03-28T16:11:06.941328+0000) 2024-03-28T16:11:23.100 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:22 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:22.753+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.809005+0000 front 2024-03-28T16:10:24.808952+0000 (oldest deadline 2024-03-28T16:10:47.108813+0000) 2024-03-28T16:11:23.100 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:22 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:22.753+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.115020+0000 front 2024-03-28T16:10:35.114954+0000 (oldest deadline 2024-03-28T16:10:59.815004+0000) 2024-03-28T16:11:23.100 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:22 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:22.753+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:40.317024+0000 front 2024-03-28T16:10:40.317425+0000 (oldest deadline 2024-03-28T16:11:05.617169+0000) 2024-03-28T16:11:23.100 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:22 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:22.753+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:49.619830+0000 front 2024-03-28T16:10:49.620058+0000 (oldest deadline 2024-03-28T16:11:14.919627+0000) 2024-03-28T16:11:23.417 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:23 smithi150 podman[69288]: 2024-03-28 16:11:23.031709571 +0000 UTC m=+0.031905565 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T16:11:23.417 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:23 smithi150 podman[69288]: 2024-03-28 16:11:23.187165579 +0000 UTC m=+0.187361594 container create d27d2dd61a19c505c610b25aee2646ddb280e2f3f196f07dd660224029d2f9e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-28T16:11:23.417 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:23 smithi150 podman[69288]: 2024-03-28 16:11:23.41590325 +0000 UTC m=+0.416099272 container init d27d2dd61a19c505c610b25aee2646ddb280e2f3f196f07dd660224029d2f9e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7-deactivate, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-28T16:11:23.764 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:23 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:23.432+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:11:23.764 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:23 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:23.432+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.053696+0000 front 2024-03-28T16:10:24.053904+0000 (oldest deadline 2024-03-28T16:10:48.753813+0000) 2024-03-28T16:11:23.764 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:23 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:23.432+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:33.258214+0000 front 2024-03-28T16:10:33.258863+0000 (oldest deadline 2024-03-28T16:10:57.958232+0000) 2024-03-28T16:11:23.764 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:23 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:23.432+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:53.262996+0000 front 2024-03-28T16:10:53.262892+0000 (oldest deadline 2024-03-28T16:11:18.463901+0000) 2024-03-28T16:11:23.802 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:23 smithi150 podman[69288]: 2024-03-28 16:11:23.428851197 +0000 UTC m=+0.429047188 container start d27d2dd61a19c505c610b25aee2646ddb280e2f3f196f07dd660224029d2f9e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T16:11:23.802 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:23 smithi150 podman[69288]: 2024-03-28 16:11:23.522730115 +0000 UTC m=+0.522926196 container attach d27d2dd61a19c505c610b25aee2646ddb280e2f3f196f07dd660224029d2f9e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-28T16:11:24.069 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:23 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:23.665+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.533614+0000 front 2024-03-28T16:10:16.534633+0000 (oldest deadline 2024-03-28T16:10:38.233619+0000) 2024-03-28T16:11:24.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:23 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:23.665+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:21.735742+0000 front 2024-03-28T16:10:21.735828+0000 (oldest deadline 2024-03-28T16:10:47.635538+0000) 2024-03-28T16:11:24.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:23 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:23.665+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:34.639789+0000 front 2024-03-28T16:10:34.639852+0000 (oldest deadline 2024-03-28T16:10:58.139257+0000) 2024-03-28T16:11:24.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:23 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:23.665+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:41.041611+0000 front 2024-03-28T16:10:41.041696+0000 (oldest deadline 2024-03-28T16:11:06.941328+0000) 2024-03-28T16:11:24.100 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:23 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:23.762+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.809005+0000 front 2024-03-28T16:10:24.808952+0000 (oldest deadline 2024-03-28T16:10:47.108813+0000) 2024-03-28T16:11:24.100 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:23 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:23.762+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.115020+0000 front 2024-03-28T16:10:35.114954+0000 (oldest deadline 2024-03-28T16:10:59.815004+0000) 2024-03-28T16:11:24.100 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:23 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:23.762+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:40.317024+0000 front 2024-03-28T16:10:40.317425+0000 (oldest deadline 2024-03-28T16:11:05.617169+0000) 2024-03-28T16:11:24.100 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:23 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:23.762+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:49.619830+0000 front 2024-03-28T16:10:49.620058+0000 (oldest deadline 2024-03-28T16:11:14.919627+0000) 2024-03-28T16:11:24.216 INFO:journalctl@ceph.osd.7.smithi150.stdout:Mar 28 16:11:23 smithi150 podman[69288]: 2024-03-28 16:11:23.864404156 +0000 UTC m=+0.864600229 container died d27d2dd61a19c505c610b25aee2646ddb280e2f3f196f07dd660224029d2f9e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-7-deactivate, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5) 2024-03-28T16:11:24.216 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:23 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:23.832+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:11:24.216 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:23 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:23.832+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.601392+0000 front 2024-03-28T16:10:23.601326+0000 (oldest deadline 2024-03-28T16:10:47.701522+0000) 2024-03-28T16:11:24.216 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:23 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:23.832+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.304820+0000 front 2024-03-28T16:10:35.304495+0000 (oldest deadline 2024-03-28T16:10:59.404389+0000) 2024-03-28T16:11:24.216 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:23 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:23.832+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:44.107005+0000 front 2024-03-28T16:10:44.106857+0000 (oldest deadline 2024-03-28T16:11:08.206356+0000) 2024-03-28T16:11:24.216 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:23 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:23.832+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:52.908191+0000 front 2024-03-28T16:10:52.908270+0000 (oldest deadline 2024-03-28T16:11:18.808335+0000) 2024-03-28T16:11:24.592 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@osd.7.service' 2024-03-28T16:11:24.711 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T16:11:24.711 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-28T16:11:24.712 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-03-28T16:11:24.712 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@osd.8 2024-03-28T16:11:24.768 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:24 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:24.447+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:11:24.768 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:24 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:24.447+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.053696+0000 front 2024-03-28T16:10:24.053904+0000 (oldest deadline 2024-03-28T16:10:48.753813+0000) 2024-03-28T16:11:24.769 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:24 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:24.447+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:33.258214+0000 front 2024-03-28T16:10:33.258863+0000 (oldest deadline 2024-03-28T16:10:57.958232+0000) 2024-03-28T16:11:24.769 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:24 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:24.447+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:53.262996+0000 front 2024-03-28T16:10:53.262892+0000 (oldest deadline 2024-03-28T16:11:18.463901+0000) 2024-03-28T16:11:24.769 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:24 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:24.447+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:00.165831+0000 front 2024-03-28T16:11:00.165617+0000 (oldest deadline 2024-03-28T16:11:24.265608+0000) 2024-03-28T16:11:24.802 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:24 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:24.787+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:11:24.802 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:24 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:24.787+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.601392+0000 front 2024-03-28T16:10:23.601326+0000 (oldest deadline 2024-03-28T16:10:47.701522+0000) 2024-03-28T16:11:24.803 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:24 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:24.787+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.304820+0000 front 2024-03-28T16:10:35.304495+0000 (oldest deadline 2024-03-28T16:10:59.404389+0000) 2024-03-28T16:11:24.803 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:24 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:24.787+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:44.107005+0000 front 2024-03-28T16:10:44.106857+0000 (oldest deadline 2024-03-28T16:11:08.206356+0000) 2024-03-28T16:11:24.803 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:24 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:24.787+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:52.908191+0000 front 2024-03-28T16:10:52.908270+0000 (oldest deadline 2024-03-28T16:11:18.808335+0000) 2024-03-28T16:11:25.069 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:24 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:24.636+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.533614+0000 front 2024-03-28T16:10:16.534633+0000 (oldest deadline 2024-03-28T16:10:38.233619+0000) 2024-03-28T16:11:25.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:24 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:24.636+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:21.735742+0000 front 2024-03-28T16:10:21.735828+0000 (oldest deadline 2024-03-28T16:10:47.635538+0000) 2024-03-28T16:11:25.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:24 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:24.636+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:34.639789+0000 front 2024-03-28T16:10:34.639852+0000 (oldest deadline 2024-03-28T16:10:58.139257+0000) 2024-03-28T16:11:25.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:24 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:24.636+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:41.041611+0000 front 2024-03-28T16:10:41.041696+0000 (oldest deadline 2024-03-28T16:11:06.941328+0000) 2024-03-28T16:11:25.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:24 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:24.636+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:00.847365+0000 front 2024-03-28T16:11:00.847285+0000 (oldest deadline 2024-03-28T16:11:24.347544+0000) 2024-03-28T16:11:25.100 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:24 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:24.767+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.809005+0000 front 2024-03-28T16:10:24.808952+0000 (oldest deadline 2024-03-28T16:10:47.108813+0000) 2024-03-28T16:11:25.100 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:24 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:24.767+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.115020+0000 front 2024-03-28T16:10:35.114954+0000 (oldest deadline 2024-03-28T16:10:59.815004+0000) 2024-03-28T16:11:25.100 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:24 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:24.767+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:40.317024+0000 front 2024-03-28T16:10:40.317425+0000 (oldest deadline 2024-03-28T16:11:05.617169+0000) 2024-03-28T16:11:25.100 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:24 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:24.767+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:49.619830+0000 front 2024-03-28T16:10:49.620058+0000 (oldest deadline 2024-03-28T16:11:14.919627+0000) 2024-03-28T16:11:25.100 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:24 smithi183 systemd[1]: Stopping Ceph osd.8 for b5f8225a-ed1a-11ee-95d0-87774f69a715... 2024-03-28T16:11:25.600 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:25 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:25.264+0000 7f76b5000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T16:11:25.601 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:25 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:25.264+0000 7f76b5000700 -1 osd.8 95 *** Got signal Terminated *** 2024-03-28T16:11:25.601 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:25 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:25.264+0000 7f76b5000700 -1 osd.8 95 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T16:11:25.601 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:25 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:25.489+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:11:25.601 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:25 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:25.489+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.053696+0000 front 2024-03-28T16:10:24.053904+0000 (oldest deadline 2024-03-28T16:10:48.753813+0000) 2024-03-28T16:11:25.601 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:25 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:25.489+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:33.258214+0000 front 2024-03-28T16:10:33.258863+0000 (oldest deadline 2024-03-28T16:10:57.958232+0000) 2024-03-28T16:11:25.601 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:25 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:25.489+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:53.262996+0000 front 2024-03-28T16:10:53.262892+0000 (oldest deadline 2024-03-28T16:11:18.463901+0000) 2024-03-28T16:11:25.601 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:25 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:25.489+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:00.165831+0000 front 2024-03-28T16:11:00.165617+0000 (oldest deadline 2024-03-28T16:11:24.265608+0000) 2024-03-28T16:11:26.052 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:25 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:25.783+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:11:26.052 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:25 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:25.783+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.601392+0000 front 2024-03-28T16:10:23.601326+0000 (oldest deadline 2024-03-28T16:10:47.701522+0000) 2024-03-28T16:11:26.053 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:25 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:25.783+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.304820+0000 front 2024-03-28T16:10:35.304495+0000 (oldest deadline 2024-03-28T16:10:59.404389+0000) 2024-03-28T16:11:26.053 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:25 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:25.783+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:44.107005+0000 front 2024-03-28T16:10:44.106857+0000 (oldest deadline 2024-03-28T16:11:08.206356+0000) 2024-03-28T16:11:26.053 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:25 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:25.783+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:52.908191+0000 front 2024-03-28T16:10:52.908270+0000 (oldest deadline 2024-03-28T16:11:18.808335+0000) 2024-03-28T16:11:26.069 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:25 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:25.636+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.533614+0000 front 2024-03-28T16:10:16.534633+0000 (oldest deadline 2024-03-28T16:10:38.233619+0000) 2024-03-28T16:11:26.069 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:25 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:25.636+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:21.735742+0000 front 2024-03-28T16:10:21.735828+0000 (oldest deadline 2024-03-28T16:10:47.635538+0000) 2024-03-28T16:11:26.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:25 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:25.636+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:34.639789+0000 front 2024-03-28T16:10:34.639852+0000 (oldest deadline 2024-03-28T16:10:58.139257+0000) 2024-03-28T16:11:26.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:25 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:25.636+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:41.041611+0000 front 2024-03-28T16:10:41.041696+0000 (oldest deadline 2024-03-28T16:11:06.941328+0000) 2024-03-28T16:11:26.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:25 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:25.636+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:00.847365+0000 front 2024-03-28T16:11:00.847285+0000 (oldest deadline 2024-03-28T16:11:24.347544+0000) 2024-03-28T16:11:26.101 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:25 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:25.723+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.809005+0000 front 2024-03-28T16:10:24.808952+0000 (oldest deadline 2024-03-28T16:10:47.108813+0000) 2024-03-28T16:11:26.101 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:25 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:25.723+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.115020+0000 front 2024-03-28T16:10:35.114954+0000 (oldest deadline 2024-03-28T16:10:59.815004+0000) 2024-03-28T16:11:26.101 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:25 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:25.723+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:40.317024+0000 front 2024-03-28T16:10:40.317425+0000 (oldest deadline 2024-03-28T16:11:05.617169+0000) 2024-03-28T16:11:26.101 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:25 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:25.723+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:49.619830+0000 front 2024-03-28T16:10:49.620058+0000 (oldest deadline 2024-03-28T16:11:14.919627+0000) 2024-03-28T16:11:26.850 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:26 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:26.686+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.809005+0000 front 2024-03-28T16:10:24.808952+0000 (oldest deadline 2024-03-28T16:10:47.108813+0000) 2024-03-28T16:11:26.850 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:26 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:26.686+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.115020+0000 front 2024-03-28T16:10:35.114954+0000 (oldest deadline 2024-03-28T16:10:59.815004+0000) 2024-03-28T16:11:26.850 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:26 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:26.686+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:40.317024+0000 front 2024-03-28T16:10:40.317425+0000 (oldest deadline 2024-03-28T16:11:05.617169+0000) 2024-03-28T16:11:26.851 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:26 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:26.686+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:49.619830+0000 front 2024-03-28T16:10:49.620058+0000 (oldest deadline 2024-03-28T16:11:14.919627+0000) 2024-03-28T16:11:26.851 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:26 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:26.509+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:11:26.851 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:26 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:26.509+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.053696+0000 front 2024-03-28T16:10:24.053904+0000 (oldest deadline 2024-03-28T16:10:48.753813+0000) 2024-03-28T16:11:26.851 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:26 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:26.509+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:33.258214+0000 front 2024-03-28T16:10:33.258863+0000 (oldest deadline 2024-03-28T16:10:57.958232+0000) 2024-03-28T16:11:26.851 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:26 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:26.509+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:53.262996+0000 front 2024-03-28T16:10:53.262892+0000 (oldest deadline 2024-03-28T16:11:18.463901+0000) 2024-03-28T16:11:26.851 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:26 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:26.509+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:00.165831+0000 front 2024-03-28T16:11:00.165617+0000 (oldest deadline 2024-03-28T16:11:24.265608+0000) 2024-03-28T16:11:27.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:26 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:26.633+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.533614+0000 front 2024-03-28T16:10:16.534633+0000 (oldest deadline 2024-03-28T16:10:38.233619+0000) 2024-03-28T16:11:27.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:26 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:26.633+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:21.735742+0000 front 2024-03-28T16:10:21.735828+0000 (oldest deadline 2024-03-28T16:10:47.635538+0000) 2024-03-28T16:11:27.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:26 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:26.633+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:34.639789+0000 front 2024-03-28T16:10:34.639852+0000 (oldest deadline 2024-03-28T16:10:58.139257+0000) 2024-03-28T16:11:27.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:26 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:26.633+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:41.041611+0000 front 2024-03-28T16:10:41.041696+0000 (oldest deadline 2024-03-28T16:11:06.941328+0000) 2024-03-28T16:11:27.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:26 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:26.633+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:00.847365+0000 front 2024-03-28T16:11:00.847285+0000 (oldest deadline 2024-03-28T16:11:24.347544+0000) 2024-03-28T16:11:27.302 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:26 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:26.803+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:11:27.302 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:26 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:26.803+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.601392+0000 front 2024-03-28T16:10:23.601326+0000 (oldest deadline 2024-03-28T16:10:47.701522+0000) 2024-03-28T16:11:27.302 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:26 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:26.803+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.304820+0000 front 2024-03-28T16:10:35.304495+0000 (oldest deadline 2024-03-28T16:10:59.404389+0000) 2024-03-28T16:11:27.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:26 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:26.803+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:44.107005+0000 front 2024-03-28T16:10:44.106857+0000 (oldest deadline 2024-03-28T16:11:08.206356+0000) 2024-03-28T16:11:27.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:26 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:26.803+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:52.908191+0000 front 2024-03-28T16:10:52.908270+0000 (oldest deadline 2024-03-28T16:11:18.808335+0000) 2024-03-28T16:11:27.850 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:27 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:27.691+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.809005+0000 front 2024-03-28T16:10:24.808952+0000 (oldest deadline 2024-03-28T16:10:47.108813+0000) 2024-03-28T16:11:27.850 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:27 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:27.691+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.115020+0000 front 2024-03-28T16:10:35.114954+0000 (oldest deadline 2024-03-28T16:10:59.815004+0000) 2024-03-28T16:11:27.851 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:27 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:27.691+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:40.317024+0000 front 2024-03-28T16:10:40.317425+0000 (oldest deadline 2024-03-28T16:11:05.617169+0000) 2024-03-28T16:11:27.851 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:27 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:27.691+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:49.619830+0000 front 2024-03-28T16:10:49.620058+0000 (oldest deadline 2024-03-28T16:11:14.919627+0000) 2024-03-28T16:11:27.851 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:27 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:27.691+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:02.423734+0000 front 2024-03-28T16:11:02.423590+0000 (oldest deadline 2024-03-28T16:11:27.123627+0000) 2024-03-28T16:11:27.851 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:27 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:27.517+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:11:27.851 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:27 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:27.517+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.053696+0000 front 2024-03-28T16:10:24.053904+0000 (oldest deadline 2024-03-28T16:10:48.753813+0000) 2024-03-28T16:11:27.851 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:27 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:27.517+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:33.258214+0000 front 2024-03-28T16:10:33.258863+0000 (oldest deadline 2024-03-28T16:10:57.958232+0000) 2024-03-28T16:11:27.851 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:27 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:27.517+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:53.262996+0000 front 2024-03-28T16:10:53.262892+0000 (oldest deadline 2024-03-28T16:11:18.463901+0000) 2024-03-28T16:11:27.851 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:27 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:27.517+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:00.165831+0000 front 2024-03-28T16:11:00.165617+0000 (oldest deadline 2024-03-28T16:11:24.265608+0000) 2024-03-28T16:11:28.069 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:27 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:27.632+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.533614+0000 front 2024-03-28T16:10:16.534633+0000 (oldest deadline 2024-03-28T16:10:38.233619+0000) 2024-03-28T16:11:28.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:27 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:27.632+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:21.735742+0000 front 2024-03-28T16:10:21.735828+0000 (oldest deadline 2024-03-28T16:10:47.635538+0000) 2024-03-28T16:11:28.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:27 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:27.632+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:34.639789+0000 front 2024-03-28T16:10:34.639852+0000 (oldest deadline 2024-03-28T16:10:58.139257+0000) 2024-03-28T16:11:28.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:27 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:27.632+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:41.041611+0000 front 2024-03-28T16:10:41.041696+0000 (oldest deadline 2024-03-28T16:11:06.941328+0000) 2024-03-28T16:11:28.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:27 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:27.632+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:00.847365+0000 front 2024-03-28T16:11:00.847285+0000 (oldest deadline 2024-03-28T16:11:24.347544+0000) 2024-03-28T16:11:28.302 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:27 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:27.805+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:11:28.302 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:27 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:27.805+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.601392+0000 front 2024-03-28T16:10:23.601326+0000 (oldest deadline 2024-03-28T16:10:47.701522+0000) 2024-03-28T16:11:28.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:27 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:27.805+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.304820+0000 front 2024-03-28T16:10:35.304495+0000 (oldest deadline 2024-03-28T16:10:59.404389+0000) 2024-03-28T16:11:28.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:27 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:27.805+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:44.107005+0000 front 2024-03-28T16:10:44.106857+0000 (oldest deadline 2024-03-28T16:11:08.206356+0000) 2024-03-28T16:11:28.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:27 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:27.805+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:52.908191+0000 front 2024-03-28T16:10:52.908270+0000 (oldest deadline 2024-03-28T16:11:18.808335+0000) 2024-03-28T16:11:28.850 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:28 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:28.696+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.809005+0000 front 2024-03-28T16:10:24.808952+0000 (oldest deadline 2024-03-28T16:10:47.108813+0000) 2024-03-28T16:11:28.851 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:28 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:28.696+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.115020+0000 front 2024-03-28T16:10:35.114954+0000 (oldest deadline 2024-03-28T16:10:59.815004+0000) 2024-03-28T16:11:28.851 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:28 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:28.696+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:40.317024+0000 front 2024-03-28T16:10:40.317425+0000 (oldest deadline 2024-03-28T16:11:05.617169+0000) 2024-03-28T16:11:28.851 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:28 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:28.696+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:49.619830+0000 front 2024-03-28T16:10:49.620058+0000 (oldest deadline 2024-03-28T16:11:14.919627+0000) 2024-03-28T16:11:28.851 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:28 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:28.696+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:02.423734+0000 front 2024-03-28T16:11:02.423590+0000 (oldest deadline 2024-03-28T16:11:27.123627+0000) 2024-03-28T16:11:28.851 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:28 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:28.525+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:11:28.851 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:28 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:28.525+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.053696+0000 front 2024-03-28T16:10:24.053904+0000 (oldest deadline 2024-03-28T16:10:48.753813+0000) 2024-03-28T16:11:28.851 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:28 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:28.525+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:33.258214+0000 front 2024-03-28T16:10:33.258863+0000 (oldest deadline 2024-03-28T16:10:57.958232+0000) 2024-03-28T16:11:28.851 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:28 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:28.525+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:53.262996+0000 front 2024-03-28T16:10:53.262892+0000 (oldest deadline 2024-03-28T16:11:18.463901+0000) 2024-03-28T16:11:28.851 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:28 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:28.525+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:00.165831+0000 front 2024-03-28T16:11:00.165617+0000 (oldest deadline 2024-03-28T16:11:24.265608+0000) 2024-03-28T16:11:29.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:28 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:28.661+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.533614+0000 front 2024-03-28T16:10:16.534633+0000 (oldest deadline 2024-03-28T16:10:38.233619+0000) 2024-03-28T16:11:29.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:28 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:28.661+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:21.735742+0000 front 2024-03-28T16:10:21.735828+0000 (oldest deadline 2024-03-28T16:10:47.635538+0000) 2024-03-28T16:11:29.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:28 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:28.661+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:34.639789+0000 front 2024-03-28T16:10:34.639852+0000 (oldest deadline 2024-03-28T16:10:58.139257+0000) 2024-03-28T16:11:29.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:28 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:28.661+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:41.041611+0000 front 2024-03-28T16:10:41.041696+0000 (oldest deadline 2024-03-28T16:11:06.941328+0000) 2024-03-28T16:11:29.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:28 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:28.661+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:00.847365+0000 front 2024-03-28T16:11:00.847285+0000 (oldest deadline 2024-03-28T16:11:24.347544+0000) 2024-03-28T16:11:29.302 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:28 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:28.816+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:11:29.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:28 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:28.816+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.601392+0000 front 2024-03-28T16:10:23.601326+0000 (oldest deadline 2024-03-28T16:10:47.701522+0000) 2024-03-28T16:11:29.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:28 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:28.816+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.304820+0000 front 2024-03-28T16:10:35.304495+0000 (oldest deadline 2024-03-28T16:10:59.404389+0000) 2024-03-28T16:11:29.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:28 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:28.816+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:44.107005+0000 front 2024-03-28T16:10:44.106857+0000 (oldest deadline 2024-03-28T16:11:08.206356+0000) 2024-03-28T16:11:29.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:28 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:28.816+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:52.908191+0000 front 2024-03-28T16:10:52.908270+0000 (oldest deadline 2024-03-28T16:11:18.808335+0000) 2024-03-28T16:11:29.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:28 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:28.816+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:02.112246+0000 front 2024-03-28T16:11:02.111944+0000 (oldest deadline 2024-03-28T16:11:28.011785+0000) 2024-03-28T16:11:29.850 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:29 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:29.701+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.809005+0000 front 2024-03-28T16:10:24.808952+0000 (oldest deadline 2024-03-28T16:10:47.108813+0000) 2024-03-28T16:11:29.850 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:29 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:29.701+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.115020+0000 front 2024-03-28T16:10:35.114954+0000 (oldest deadline 2024-03-28T16:10:59.815004+0000) 2024-03-28T16:11:29.851 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:29 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:29.701+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:40.317024+0000 front 2024-03-28T16:10:40.317425+0000 (oldest deadline 2024-03-28T16:11:05.617169+0000) 2024-03-28T16:11:29.851 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:29 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:29.701+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:49.619830+0000 front 2024-03-28T16:10:49.620058+0000 (oldest deadline 2024-03-28T16:11:14.919627+0000) 2024-03-28T16:11:29.851 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:29 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8[54208]: 2024-03-28T16:11:29.701+0000 7f76b0800700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:02.423734+0000 front 2024-03-28T16:11:02.423590+0000 (oldest deadline 2024-03-28T16:11:27.123627+0000) 2024-03-28T16:11:29.851 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:29 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:29.570+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:11:29.851 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:29 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:29.570+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.053696+0000 front 2024-03-28T16:10:24.053904+0000 (oldest deadline 2024-03-28T16:10:48.753813+0000) 2024-03-28T16:11:29.851 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:29 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:29.570+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:33.258214+0000 front 2024-03-28T16:10:33.258863+0000 (oldest deadline 2024-03-28T16:10:57.958232+0000) 2024-03-28T16:11:29.851 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:29 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:29.570+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:53.262996+0000 front 2024-03-28T16:10:53.262892+0000 (oldest deadline 2024-03-28T16:11:18.463901+0000) 2024-03-28T16:11:29.851 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:29 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:29.570+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:00.165831+0000 front 2024-03-28T16:11:00.165617+0000 (oldest deadline 2024-03-28T16:11:24.265608+0000) 2024-03-28T16:11:30.069 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:29 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:29.691+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.533614+0000 front 2024-03-28T16:10:16.534633+0000 (oldest deadline 2024-03-28T16:10:38.233619+0000) 2024-03-28T16:11:30.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:29 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:29.691+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:21.735742+0000 front 2024-03-28T16:10:21.735828+0000 (oldest deadline 2024-03-28T16:10:47.635538+0000) 2024-03-28T16:11:30.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:29 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:29.691+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:34.639789+0000 front 2024-03-28T16:10:34.639852+0000 (oldest deadline 2024-03-28T16:10:58.139257+0000) 2024-03-28T16:11:30.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:29 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:29.691+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:41.041611+0000 front 2024-03-28T16:10:41.041696+0000 (oldest deadline 2024-03-28T16:11:06.941328+0000) 2024-03-28T16:11:30.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:29 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:29.691+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:00.847365+0000 front 2024-03-28T16:11:00.847285+0000 (oldest deadline 2024-03-28T16:11:24.347544+0000) 2024-03-28T16:11:30.302 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:29 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:29.864+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:11:30.302 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:29 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:29.864+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.601392+0000 front 2024-03-28T16:10:23.601326+0000 (oldest deadline 2024-03-28T16:10:47.701522+0000) 2024-03-28T16:11:30.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:29 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:29.864+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.304820+0000 front 2024-03-28T16:10:35.304495+0000 (oldest deadline 2024-03-28T16:10:59.404389+0000) 2024-03-28T16:11:30.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:29 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:29.864+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:44.107005+0000 front 2024-03-28T16:10:44.106857+0000 (oldest deadline 2024-03-28T16:11:08.206356+0000) 2024-03-28T16:11:30.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:29 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:29.864+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:52.908191+0000 front 2024-03-28T16:10:52.908270+0000 (oldest deadline 2024-03-28T16:11:18.808335+0000) 2024-03-28T16:11:30.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:29 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:29.864+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:02.112246+0000 front 2024-03-28T16:11:02.111944+0000 (oldest deadline 2024-03-28T16:11:28.011785+0000) 2024-03-28T16:11:30.818 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:30 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:30.546+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:11:30.819 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:30 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:30.546+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.053696+0000 front 2024-03-28T16:10:24.053904+0000 (oldest deadline 2024-03-28T16:10:48.753813+0000) 2024-03-28T16:11:30.819 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:30 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:30.546+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:33.258214+0000 front 2024-03-28T16:10:33.258863+0000 (oldest deadline 2024-03-28T16:10:57.958232+0000) 2024-03-28T16:11:30.819 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:30 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:30.546+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:53.262996+0000 front 2024-03-28T16:10:53.262892+0000 (oldest deadline 2024-03-28T16:11:18.463901+0000) 2024-03-28T16:11:30.819 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:30 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:30.546+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:00.165831+0000 front 2024-03-28T16:11:00.165617+0000 (oldest deadline 2024-03-28T16:11:24.265608+0000) 2024-03-28T16:11:31.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:30 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:30.685+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.533614+0000 front 2024-03-28T16:10:16.534633+0000 (oldest deadline 2024-03-28T16:10:38.233619+0000) 2024-03-28T16:11:31.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:30 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:30.685+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:21.735742+0000 front 2024-03-28T16:10:21.735828+0000 (oldest deadline 2024-03-28T16:10:47.635538+0000) 2024-03-28T16:11:31.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:30 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:30.685+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:34.639789+0000 front 2024-03-28T16:10:34.639852+0000 (oldest deadline 2024-03-28T16:10:58.139257+0000) 2024-03-28T16:11:31.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:30 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:30.685+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:41.041611+0000 front 2024-03-28T16:10:41.041696+0000 (oldest deadline 2024-03-28T16:11:06.941328+0000) 2024-03-28T16:11:31.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:30 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:30.685+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:00.847365+0000 front 2024-03-28T16:11:00.847285+0000 (oldest deadline 2024-03-28T16:11:24.347544+0000) 2024-03-28T16:11:31.100 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:30 smithi183 podman[70017]: 2024-03-28 16:11:30.818128281 +0000 UTC m=+5.669981001 container died c61fd9797c7cb9e56029b5f126c856484e80f32253cb697b12cd987bdab4c5dd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8, ceph=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T16:11:31.302 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:30 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:30.875+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:11:31.302 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:30 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:30.875+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.601392+0000 front 2024-03-28T16:10:23.601326+0000 (oldest deadline 2024-03-28T16:10:47.701522+0000) 2024-03-28T16:11:31.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:30 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:30.875+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.304820+0000 front 2024-03-28T16:10:35.304495+0000 (oldest deadline 2024-03-28T16:10:59.404389+0000) 2024-03-28T16:11:31.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:30 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:30.875+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:44.107005+0000 front 2024-03-28T16:10:44.106857+0000 (oldest deadline 2024-03-28T16:11:08.206356+0000) 2024-03-28T16:11:31.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:30 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:30.875+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:52.908191+0000 front 2024-03-28T16:10:52.908270+0000 (oldest deadline 2024-03-28T16:11:18.808335+0000) 2024-03-28T16:11:31.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:30 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:30.875+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:02.112246+0000 front 2024-03-28T16:11:02.111944+0000 (oldest deadline 2024-03-28T16:11:28.011785+0000) 2024-03-28T16:11:31.383 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:31 smithi183 podman[70017]: 2024-03-28 16:11:31.232823973 +0000 UTC m=+6.084676717 container cleanup c61fd9797c7cb9e56029b5f126c856484e80f32253cb697b12cd987bdab4c5dd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-28T16:11:31.384 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:31 smithi183 bash[70017]: ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8 2024-03-28T16:11:31.678 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:31 smithi183 podman[70037]: 2024-03-28 16:11:31.417163349 +0000 UTC m=+0.583623370 container remove c61fd9797c7cb9e56029b5f126c856484e80f32253cb697b12cd987bdab4c5dd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-28T16:11:31.678 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:31 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:31.592+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:11:31.678 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:31 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:31.592+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.053696+0000 front 2024-03-28T16:10:24.053904+0000 (oldest deadline 2024-03-28T16:10:48.753813+0000) 2024-03-28T16:11:31.678 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:31 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:31.592+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:33.258214+0000 front 2024-03-28T16:10:33.258863+0000 (oldest deadline 2024-03-28T16:10:57.958232+0000) 2024-03-28T16:11:31.679 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:31 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:31.592+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:53.262996+0000 front 2024-03-28T16:10:53.262892+0000 (oldest deadline 2024-03-28T16:11:18.463901+0000) 2024-03-28T16:11:31.679 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:31 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:31.592+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:00.165831+0000 front 2024-03-28T16:11:00.165617+0000 (oldest deadline 2024-03-28T16:11:24.265608+0000) 2024-03-28T16:11:32.069 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:31 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:31.704+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.533614+0000 front 2024-03-28T16:10:16.534633+0000 (oldest deadline 2024-03-28T16:10:38.233619+0000) 2024-03-28T16:11:32.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:31 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:31.704+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:21.735742+0000 front 2024-03-28T16:10:21.735828+0000 (oldest deadline 2024-03-28T16:10:47.635538+0000) 2024-03-28T16:11:32.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:31 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:31.704+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:34.639789+0000 front 2024-03-28T16:10:34.639852+0000 (oldest deadline 2024-03-28T16:10:58.139257+0000) 2024-03-28T16:11:32.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:31 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:31.704+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:41.041611+0000 front 2024-03-28T16:10:41.041696+0000 (oldest deadline 2024-03-28T16:11:06.941328+0000) 2024-03-28T16:11:32.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:31 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:31.704+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:00.847365+0000 front 2024-03-28T16:11:00.847285+0000 (oldest deadline 2024-03-28T16:11:24.347544+0000) 2024-03-28T16:11:32.302 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:31 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:31.835+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:11:32.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:31 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:31.835+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.601392+0000 front 2024-03-28T16:10:23.601326+0000 (oldest deadline 2024-03-28T16:10:47.701522+0000) 2024-03-28T16:11:32.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:31 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:31.835+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.304820+0000 front 2024-03-28T16:10:35.304495+0000 (oldest deadline 2024-03-28T16:10:59.404389+0000) 2024-03-28T16:11:32.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:31 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:31.835+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:44.107005+0000 front 2024-03-28T16:10:44.106857+0000 (oldest deadline 2024-03-28T16:11:08.206356+0000) 2024-03-28T16:11:32.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:31 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:31.835+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:52.908191+0000 front 2024-03-28T16:10:52.908270+0000 (oldest deadline 2024-03-28T16:11:18.808335+0000) 2024-03-28T16:11:32.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:31 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:31.835+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:02.112246+0000 front 2024-03-28T16:11:02.111944+0000 (oldest deadline 2024-03-28T16:11:28.011785+0000) 2024-03-28T16:11:32.428 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:32 smithi183 podman[70131]: 2024-03-28 16:11:32.011712655 +0000 UTC m=+0.032953344 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T16:11:32.429 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:32 smithi183 podman[70131]: 2024-03-28 16:11:32.109602518 +0000 UTC m=+0.130843230 container create b191a7b8e8353c96899778932e765520b73ec13c6717e83cbf383c81ecef7668 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8-deactivate, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2) 2024-03-28T16:11:32.429 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:32 smithi183 podman[70131]: 2024-03-28 16:11:32.337145339 +0000 UTC m=+0.358386022 container init b191a7b8e8353c96899778932e765520b73ec13c6717e83cbf383c81ecef7668 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T16:11:32.429 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:32 smithi183 podman[70131]: 2024-03-28 16:11:32.347131893 +0000 UTC m=+0.368372591 container start b191a7b8e8353c96899778932e765520b73ec13c6717e83cbf383c81ecef7668 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T16:11:32.429 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:32 smithi183 podman[70131]: 2024-03-28 16:11:32.428384909 +0000 UTC m=+0.449625644 container attach b191a7b8e8353c96899778932e765520b73ec13c6717e83cbf383c81ecef7668 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8-deactivate, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-28T16:11:32.790 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:32 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:32.581+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:11:32.790 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:32 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:32.581+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.053696+0000 front 2024-03-28T16:10:24.053904+0000 (oldest deadline 2024-03-28T16:10:48.753813+0000) 2024-03-28T16:11:32.790 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:32 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:32.581+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:33.258214+0000 front 2024-03-28T16:10:33.258863+0000 (oldest deadline 2024-03-28T16:10:57.958232+0000) 2024-03-28T16:11:32.790 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:32 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:32.581+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:53.262996+0000 front 2024-03-28T16:10:53.262892+0000 (oldest deadline 2024-03-28T16:11:18.463901+0000) 2024-03-28T16:11:32.790 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:32 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:32.581+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:00.165831+0000 front 2024-03-28T16:11:00.165617+0000 (oldest deadline 2024-03-28T16:11:24.265608+0000) 2024-03-28T16:11:33.069 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:32 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:32.674+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.533614+0000 front 2024-03-28T16:10:16.534633+0000 (oldest deadline 2024-03-28T16:10:38.233619+0000) 2024-03-28T16:11:33.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:32 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:32.674+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:21.735742+0000 front 2024-03-28T16:10:21.735828+0000 (oldest deadline 2024-03-28T16:10:47.635538+0000) 2024-03-28T16:11:33.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:32 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:32.674+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:34.639789+0000 front 2024-03-28T16:10:34.639852+0000 (oldest deadline 2024-03-28T16:10:58.139257+0000) 2024-03-28T16:11:33.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:32 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:32.674+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:41.041611+0000 front 2024-03-28T16:10:41.041696+0000 (oldest deadline 2024-03-28T16:11:06.941328+0000) 2024-03-28T16:11:33.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:32 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:32.674+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:00.847365+0000 front 2024-03-28T16:11:00.847285+0000 (oldest deadline 2024-03-28T16:11:24.347544+0000) 2024-03-28T16:11:33.099 INFO:journalctl@ceph.osd.8.smithi183.stdout:Mar 28 16:11:32 smithi183 podman[70131]: 2024-03-28 16:11:32.790181502 +0000 UTC m=+0.811422209 container died b191a7b8e8353c96899778932e765520b73ec13c6717e83cbf383c81ecef7668 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-8-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T16:11:33.302 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:32 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:32.801+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:11:33.302 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:32 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:32.801+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.601392+0000 front 2024-03-28T16:10:23.601326+0000 (oldest deadline 2024-03-28T16:10:47.701522+0000) 2024-03-28T16:11:33.302 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:32 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:32.801+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.304820+0000 front 2024-03-28T16:10:35.304495+0000 (oldest deadline 2024-03-28T16:10:59.404389+0000) 2024-03-28T16:11:33.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:32 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:32.801+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:44.107005+0000 front 2024-03-28T16:10:44.106857+0000 (oldest deadline 2024-03-28T16:11:08.206356+0000) 2024-03-28T16:11:33.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:32 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:32.801+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:52.908191+0000 front 2024-03-28T16:10:52.908270+0000 (oldest deadline 2024-03-28T16:11:18.808335+0000) 2024-03-28T16:11:33.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:32 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:32.801+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:02.112246+0000 front 2024-03-28T16:11:02.111944+0000 (oldest deadline 2024-03-28T16:11:28.011785+0000) 2024-03-28T16:11:33.442 DEBUG:teuthology.orchestra.run.smithi183:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@osd.8.service' 2024-03-28T16:11:33.566 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T16:11:33.567 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-03-28T16:11:33.567 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-03-28T16:11:33.567 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@osd.9 2024-03-28T16:11:33.850 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:33 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:33.570+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:11:33.851 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:33 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:33.570+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.053696+0000 front 2024-03-28T16:10:24.053904+0000 (oldest deadline 2024-03-28T16:10:48.753813+0000) 2024-03-28T16:11:33.851 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:33 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:33.570+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:33.258214+0000 front 2024-03-28T16:10:33.258863+0000 (oldest deadline 2024-03-28T16:10:57.958232+0000) 2024-03-28T16:11:33.851 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:33 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:33.570+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:53.262996+0000 front 2024-03-28T16:10:53.262892+0000 (oldest deadline 2024-03-28T16:11:18.463901+0000) 2024-03-28T16:11:33.851 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:33 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:33.570+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:00.165831+0000 front 2024-03-28T16:11:00.165617+0000 (oldest deadline 2024-03-28T16:11:24.265608+0000) 2024-03-28T16:11:33.851 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:33 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:33.570+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-03-28T16:11:07.068385+0000 front 2024-03-28T16:11:07.068776+0000 (oldest deadline 2024-03-28T16:11:32.968340+0000) 2024-03-28T16:11:34.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:33 smithi067 systemd[1]: Stopping Ceph osd.9 for b5f8225a-ed1a-11ee-95d0-87774f69a715... 2024-03-28T16:11:34.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:33 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:33.712+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.533614+0000 front 2024-03-28T16:10:16.534633+0000 (oldest deadline 2024-03-28T16:10:38.233619+0000) 2024-03-28T16:11:34.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:33 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:33.712+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:21.735742+0000 front 2024-03-28T16:10:21.735828+0000 (oldest deadline 2024-03-28T16:10:47.635538+0000) 2024-03-28T16:11:34.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:33 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:33.712+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:34.639789+0000 front 2024-03-28T16:10:34.639852+0000 (oldest deadline 2024-03-28T16:10:58.139257+0000) 2024-03-28T16:11:34.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:33 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:33.712+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:41.041611+0000 front 2024-03-28T16:10:41.041696+0000 (oldest deadline 2024-03-28T16:11:06.941328+0000) 2024-03-28T16:11:34.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:33 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:33.712+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:00.847365+0000 front 2024-03-28T16:11:00.847285+0000 (oldest deadline 2024-03-28T16:11:24.347544+0000) 2024-03-28T16:11:34.071 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:33 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:33.712+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-03-28T16:11:09.451890+0000 front 2024-03-28T16:11:09.451813+0000 (oldest deadline 2024-03-28T16:11:32.951463+0000) 2024-03-28T16:11:34.302 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:33 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:33.805+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:11:34.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:33 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:33.805+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.601392+0000 front 2024-03-28T16:10:23.601326+0000 (oldest deadline 2024-03-28T16:10:47.701522+0000) 2024-03-28T16:11:34.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:33 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:33.805+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.304820+0000 front 2024-03-28T16:10:35.304495+0000 (oldest deadline 2024-03-28T16:10:59.404389+0000) 2024-03-28T16:11:34.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:33 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:33.805+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:44.107005+0000 front 2024-03-28T16:10:44.106857+0000 (oldest deadline 2024-03-28T16:11:08.206356+0000) 2024-03-28T16:11:34.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:33 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:33.805+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:52.908191+0000 front 2024-03-28T16:10:52.908270+0000 (oldest deadline 2024-03-28T16:11:18.808335+0000) 2024-03-28T16:11:34.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:33 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:33.805+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:02.112246+0000 front 2024-03-28T16:11:02.111944+0000 (oldest deadline 2024-03-28T16:11:28.011785+0000) 2024-03-28T16:11:34.570 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:34 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:34.160+0000 7f7fd1a00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T16:11:34.570 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:34 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:34.160+0000 7f7fd1a00700 -1 osd.9 95 *** Got signal Terminated *** 2024-03-28T16:11:34.570 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:34 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:34.160+0000 7f7fd1a00700 -1 osd.9 95 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T16:11:35.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:34 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:34.685+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.533614+0000 front 2024-03-28T16:10:16.534633+0000 (oldest deadline 2024-03-28T16:10:38.233619+0000) 2024-03-28T16:11:35.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:34 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:34.685+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:21.735742+0000 front 2024-03-28T16:10:21.735828+0000 (oldest deadline 2024-03-28T16:10:47.635538+0000) 2024-03-28T16:11:35.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:34 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:34.685+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:34.639789+0000 front 2024-03-28T16:10:34.639852+0000 (oldest deadline 2024-03-28T16:10:58.139257+0000) 2024-03-28T16:11:35.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:34 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:34.685+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:41.041611+0000 front 2024-03-28T16:10:41.041696+0000 (oldest deadline 2024-03-28T16:11:06.941328+0000) 2024-03-28T16:11:35.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:34 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:34.685+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:00.847365+0000 front 2024-03-28T16:11:00.847285+0000 (oldest deadline 2024-03-28T16:11:24.347544+0000) 2024-03-28T16:11:35.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:34 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:34.685+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-03-28T16:11:09.451890+0000 front 2024-03-28T16:11:09.451813+0000 (oldest deadline 2024-03-28T16:11:32.951463+0000) 2024-03-28T16:11:35.100 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:34 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:34.603+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:11:35.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:34 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:34.603+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.053696+0000 front 2024-03-28T16:10:24.053904+0000 (oldest deadline 2024-03-28T16:10:48.753813+0000) 2024-03-28T16:11:35.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:34 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:34.603+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:33.258214+0000 front 2024-03-28T16:10:33.258863+0000 (oldest deadline 2024-03-28T16:10:57.958232+0000) 2024-03-28T16:11:35.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:34 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:34.603+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:53.262996+0000 front 2024-03-28T16:10:53.262892+0000 (oldest deadline 2024-03-28T16:11:18.463901+0000) 2024-03-28T16:11:35.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:34 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:34.603+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:00.165831+0000 front 2024-03-28T16:11:00.165617+0000 (oldest deadline 2024-03-28T16:11:24.265608+0000) 2024-03-28T16:11:35.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:34 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:34.603+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-03-28T16:11:07.068385+0000 front 2024-03-28T16:11:07.068776+0000 (oldest deadline 2024-03-28T16:11:32.968340+0000) 2024-03-28T16:11:35.302 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:34 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:34.831+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:11:35.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:34 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:34.831+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.601392+0000 front 2024-03-28T16:10:23.601326+0000 (oldest deadline 2024-03-28T16:10:47.701522+0000) 2024-03-28T16:11:35.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:34 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:34.831+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.304820+0000 front 2024-03-28T16:10:35.304495+0000 (oldest deadline 2024-03-28T16:10:59.404389+0000) 2024-03-28T16:11:35.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:34 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:34.831+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:44.107005+0000 front 2024-03-28T16:10:44.106857+0000 (oldest deadline 2024-03-28T16:11:08.206356+0000) 2024-03-28T16:11:35.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:34 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:34.831+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:52.908191+0000 front 2024-03-28T16:10:52.908270+0000 (oldest deadline 2024-03-28T16:11:18.808335+0000) 2024-03-28T16:11:35.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:34 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:34.831+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:02.112246+0000 front 2024-03-28T16:11:02.111944+0000 (oldest deadline 2024-03-28T16:11:28.011785+0000) 2024-03-28T16:11:35.850 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:35 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:35.573+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:11:35.851 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:35 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:35.573+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.053696+0000 front 2024-03-28T16:10:24.053904+0000 (oldest deadline 2024-03-28T16:10:48.753813+0000) 2024-03-28T16:11:35.851 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:35 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:35.573+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:33.258214+0000 front 2024-03-28T16:10:33.258863+0000 (oldest deadline 2024-03-28T16:10:57.958232+0000) 2024-03-28T16:11:35.851 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:35 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:35.573+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:53.262996+0000 front 2024-03-28T16:10:53.262892+0000 (oldest deadline 2024-03-28T16:11:18.463901+0000) 2024-03-28T16:11:35.851 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:35 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:35.573+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:00.165831+0000 front 2024-03-28T16:11:00.165617+0000 (oldest deadline 2024-03-28T16:11:24.265608+0000) 2024-03-28T16:11:35.851 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:35 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:35.573+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-03-28T16:11:07.068385+0000 front 2024-03-28T16:11:07.068776+0000 (oldest deadline 2024-03-28T16:11:32.968340+0000) 2024-03-28T16:11:36.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:35 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:35.728+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.533614+0000 front 2024-03-28T16:10:16.534633+0000 (oldest deadline 2024-03-28T16:10:38.233619+0000) 2024-03-28T16:11:36.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:35 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:35.728+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:21.735742+0000 front 2024-03-28T16:10:21.735828+0000 (oldest deadline 2024-03-28T16:10:47.635538+0000) 2024-03-28T16:11:36.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:35 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:35.728+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:34.639789+0000 front 2024-03-28T16:10:34.639852+0000 (oldest deadline 2024-03-28T16:10:58.139257+0000) 2024-03-28T16:11:36.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:35 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:35.728+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:41.041611+0000 front 2024-03-28T16:10:41.041696+0000 (oldest deadline 2024-03-28T16:11:06.941328+0000) 2024-03-28T16:11:36.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:35 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:35.728+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:00.847365+0000 front 2024-03-28T16:11:00.847285+0000 (oldest deadline 2024-03-28T16:11:24.347544+0000) 2024-03-28T16:11:36.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:35 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:35.728+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-03-28T16:11:09.451890+0000 front 2024-03-28T16:11:09.451813+0000 (oldest deadline 2024-03-28T16:11:32.951463+0000) 2024-03-28T16:11:36.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:35 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:35.808+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:11:36.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:35 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:35.808+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.601392+0000 front 2024-03-28T16:10:23.601326+0000 (oldest deadline 2024-03-28T16:10:47.701522+0000) 2024-03-28T16:11:36.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:35 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:35.808+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.304820+0000 front 2024-03-28T16:10:35.304495+0000 (oldest deadline 2024-03-28T16:10:59.404389+0000) 2024-03-28T16:11:36.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:35 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:35.808+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:44.107005+0000 front 2024-03-28T16:10:44.106857+0000 (oldest deadline 2024-03-28T16:11:08.206356+0000) 2024-03-28T16:11:36.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:35 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:35.808+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:52.908191+0000 front 2024-03-28T16:10:52.908270+0000 (oldest deadline 2024-03-28T16:11:18.808335+0000) 2024-03-28T16:11:36.304 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:35 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:35.808+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:02.112246+0000 front 2024-03-28T16:11:02.111944+0000 (oldest deadline 2024-03-28T16:11:28.011785+0000) 2024-03-28T16:11:37.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:36 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:36.708+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.533614+0000 front 2024-03-28T16:10:16.534633+0000 (oldest deadline 2024-03-28T16:10:38.233619+0000) 2024-03-28T16:11:37.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:36 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:36.708+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:21.735742+0000 front 2024-03-28T16:10:21.735828+0000 (oldest deadline 2024-03-28T16:10:47.635538+0000) 2024-03-28T16:11:37.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:36 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:36.708+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:34.639789+0000 front 2024-03-28T16:10:34.639852+0000 (oldest deadline 2024-03-28T16:10:58.139257+0000) 2024-03-28T16:11:37.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:36 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:36.708+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:41.041611+0000 front 2024-03-28T16:10:41.041696+0000 (oldest deadline 2024-03-28T16:11:06.941328+0000) 2024-03-28T16:11:37.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:36 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:36.708+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:00.847365+0000 front 2024-03-28T16:11:00.847285+0000 (oldest deadline 2024-03-28T16:11:24.347544+0000) 2024-03-28T16:11:37.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:36 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:36.708+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-03-28T16:11:09.451890+0000 front 2024-03-28T16:11:09.451813+0000 (oldest deadline 2024-03-28T16:11:32.951463+0000) 2024-03-28T16:11:37.100 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:36 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:36.604+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:11:37.100 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:36 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:36.604+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.053696+0000 front 2024-03-28T16:10:24.053904+0000 (oldest deadline 2024-03-28T16:10:48.753813+0000) 2024-03-28T16:11:37.100 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:36 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:36.604+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:33.258214+0000 front 2024-03-28T16:10:33.258863+0000 (oldest deadline 2024-03-28T16:10:57.958232+0000) 2024-03-28T16:11:37.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:36 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:36.604+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:53.262996+0000 front 2024-03-28T16:10:53.262892+0000 (oldest deadline 2024-03-28T16:11:18.463901+0000) 2024-03-28T16:11:37.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:36 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:36.604+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:00.165831+0000 front 2024-03-28T16:11:00.165617+0000 (oldest deadline 2024-03-28T16:11:24.265608+0000) 2024-03-28T16:11:37.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:36 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:36.604+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-03-28T16:11:07.068385+0000 front 2024-03-28T16:11:07.068776+0000 (oldest deadline 2024-03-28T16:11:32.968340+0000) 2024-03-28T16:11:37.302 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:36 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:36.830+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:11:37.302 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:36 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:36.830+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.601392+0000 front 2024-03-28T16:10:23.601326+0000 (oldest deadline 2024-03-28T16:10:47.701522+0000) 2024-03-28T16:11:37.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:36 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:36.830+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.304820+0000 front 2024-03-28T16:10:35.304495+0000 (oldest deadline 2024-03-28T16:10:59.404389+0000) 2024-03-28T16:11:37.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:36 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:36.830+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:44.107005+0000 front 2024-03-28T16:10:44.106857+0000 (oldest deadline 2024-03-28T16:11:08.206356+0000) 2024-03-28T16:11:37.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:36 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:36.830+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:52.908191+0000 front 2024-03-28T16:10:52.908270+0000 (oldest deadline 2024-03-28T16:11:18.808335+0000) 2024-03-28T16:11:37.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:36 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:36.830+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:02.112246+0000 front 2024-03-28T16:11:02.111944+0000 (oldest deadline 2024-03-28T16:11:28.011785+0000) 2024-03-28T16:11:38.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:37 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:37.708+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.533614+0000 front 2024-03-28T16:10:16.534633+0000 (oldest deadline 2024-03-28T16:10:38.233619+0000) 2024-03-28T16:11:38.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:37 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:37.708+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:21.735742+0000 front 2024-03-28T16:10:21.735828+0000 (oldest deadline 2024-03-28T16:10:47.635538+0000) 2024-03-28T16:11:38.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:37 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:37.708+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:34.639789+0000 front 2024-03-28T16:10:34.639852+0000 (oldest deadline 2024-03-28T16:10:58.139257+0000) 2024-03-28T16:11:38.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:37 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:37.708+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:41.041611+0000 front 2024-03-28T16:10:41.041696+0000 (oldest deadline 2024-03-28T16:11:06.941328+0000) 2024-03-28T16:11:38.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:37 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:37.708+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:00.847365+0000 front 2024-03-28T16:11:00.847285+0000 (oldest deadline 2024-03-28T16:11:24.347544+0000) 2024-03-28T16:11:38.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:37 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:37.708+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-03-28T16:11:09.451890+0000 front 2024-03-28T16:11:09.451813+0000 (oldest deadline 2024-03-28T16:11:32.951463+0000) 2024-03-28T16:11:38.100 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:37 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:37.607+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:11:38.100 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:37 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:37.607+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.053696+0000 front 2024-03-28T16:10:24.053904+0000 (oldest deadline 2024-03-28T16:10:48.753813+0000) 2024-03-28T16:11:38.100 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:37 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:37.607+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:33.258214+0000 front 2024-03-28T16:10:33.258863+0000 (oldest deadline 2024-03-28T16:10:57.958232+0000) 2024-03-28T16:11:38.100 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:37 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:37.607+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:53.262996+0000 front 2024-03-28T16:10:53.262892+0000 (oldest deadline 2024-03-28T16:11:18.463901+0000) 2024-03-28T16:11:38.100 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:37 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:37.608+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:00.165831+0000 front 2024-03-28T16:11:00.165617+0000 (oldest deadline 2024-03-28T16:11:24.265608+0000) 2024-03-28T16:11:38.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:37 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:37.608+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-03-28T16:11:07.068385+0000 front 2024-03-28T16:11:07.068776+0000 (oldest deadline 2024-03-28T16:11:32.968340+0000) 2024-03-28T16:11:38.302 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:37 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:37.852+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:11:38.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:37 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:37.852+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.601392+0000 front 2024-03-28T16:10:23.601326+0000 (oldest deadline 2024-03-28T16:10:47.701522+0000) 2024-03-28T16:11:38.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:37 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:37.852+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.304820+0000 front 2024-03-28T16:10:35.304495+0000 (oldest deadline 2024-03-28T16:10:59.404389+0000) 2024-03-28T16:11:38.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:37 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:37.852+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:44.107005+0000 front 2024-03-28T16:10:44.106857+0000 (oldest deadline 2024-03-28T16:11:08.206356+0000) 2024-03-28T16:11:38.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:37 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:37.852+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:52.908191+0000 front 2024-03-28T16:10:52.908270+0000 (oldest deadline 2024-03-28T16:11:18.808335+0000) 2024-03-28T16:11:38.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:37 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:37.852+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:02.112246+0000 front 2024-03-28T16:11:02.111944+0000 (oldest deadline 2024-03-28T16:11:28.011785+0000) 2024-03-28T16:11:38.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:37 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:37.852+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-03-28T16:11:12.113739+0000 front 2024-03-28T16:11:12.114043+0000 (oldest deadline 2024-03-28T16:11:37.413699+0000) 2024-03-28T16:11:39.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:38 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:38.659+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.533614+0000 front 2024-03-28T16:10:16.534633+0000 (oldest deadline 2024-03-28T16:10:38.233619+0000) 2024-03-28T16:11:39.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:38 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:38.659+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:21.735742+0000 front 2024-03-28T16:10:21.735828+0000 (oldest deadline 2024-03-28T16:10:47.635538+0000) 2024-03-28T16:11:39.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:38 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:38.659+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:34.639789+0000 front 2024-03-28T16:10:34.639852+0000 (oldest deadline 2024-03-28T16:10:58.139257+0000) 2024-03-28T16:11:39.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:38 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:38.659+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:41.041611+0000 front 2024-03-28T16:10:41.041696+0000 (oldest deadline 2024-03-28T16:11:06.941328+0000) 2024-03-28T16:11:39.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:38 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:38.659+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:00.847365+0000 front 2024-03-28T16:11:00.847285+0000 (oldest deadline 2024-03-28T16:11:24.347544+0000) 2024-03-28T16:11:39.070 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:38 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9[66948]: 2024-03-28T16:11:38.659+0000 7f7fcd200700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-03-28T16:11:09.451890+0000 front 2024-03-28T16:11:09.451813+0000 (oldest deadline 2024-03-28T16:11:32.951463+0000) 2024-03-28T16:11:39.100 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:38 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:38.650+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:11:39.100 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:38 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:38.650+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.053696+0000 front 2024-03-28T16:10:24.053904+0000 (oldest deadline 2024-03-28T16:10:48.753813+0000) 2024-03-28T16:11:39.100 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:38 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:38.650+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:33.258214+0000 front 2024-03-28T16:10:33.258863+0000 (oldest deadline 2024-03-28T16:10:57.958232+0000) 2024-03-28T16:11:39.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:38 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:38.650+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:53.262996+0000 front 2024-03-28T16:10:53.262892+0000 (oldest deadline 2024-03-28T16:11:18.463901+0000) 2024-03-28T16:11:39.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:38 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:38.650+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:00.165831+0000 front 2024-03-28T16:11:00.165617+0000 (oldest deadline 2024-03-28T16:11:24.265608+0000) 2024-03-28T16:11:39.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:38 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:38.650+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-03-28T16:11:07.068385+0000 front 2024-03-28T16:11:07.068776+0000 (oldest deadline 2024-03-28T16:11:32.968340+0000) 2024-03-28T16:11:39.205 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:38 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:38.808+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:11:39.205 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:38 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:38.808+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.601392+0000 front 2024-03-28T16:10:23.601326+0000 (oldest deadline 2024-03-28T16:10:47.701522+0000) 2024-03-28T16:11:39.205 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:38 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:38.808+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.304820+0000 front 2024-03-28T16:10:35.304495+0000 (oldest deadline 2024-03-28T16:10:59.404389+0000) 2024-03-28T16:11:39.205 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:38 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:38.808+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:44.107005+0000 front 2024-03-28T16:10:44.106857+0000 (oldest deadline 2024-03-28T16:11:08.206356+0000) 2024-03-28T16:11:39.205 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:38 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:38.808+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:52.908191+0000 front 2024-03-28T16:10:52.908270+0000 (oldest deadline 2024-03-28T16:11:18.808335+0000) 2024-03-28T16:11:39.205 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:38 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:38.808+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:02.112246+0000 front 2024-03-28T16:11:02.111944+0000 (oldest deadline 2024-03-28T16:11:28.011785+0000) 2024-03-28T16:11:39.205 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:38 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:38.808+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-03-28T16:11:12.113739+0000 front 2024-03-28T16:11:12.114043+0000 (oldest deadline 2024-03-28T16:11:37.413699+0000) 2024-03-28T16:11:39.888 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:39 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:39.777+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:11:39.889 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:39 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:39.777+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.601392+0000 front 2024-03-28T16:10:23.601326+0000 (oldest deadline 2024-03-28T16:10:47.701522+0000) 2024-03-28T16:11:39.889 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:39 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:39.777+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.304820+0000 front 2024-03-28T16:10:35.304495+0000 (oldest deadline 2024-03-28T16:10:59.404389+0000) 2024-03-28T16:11:39.889 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:39 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:39.777+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:44.107005+0000 front 2024-03-28T16:10:44.106857+0000 (oldest deadline 2024-03-28T16:11:08.206356+0000) 2024-03-28T16:11:39.889 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:39 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:39.777+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:52.908191+0000 front 2024-03-28T16:10:52.908270+0000 (oldest deadline 2024-03-28T16:11:18.808335+0000) 2024-03-28T16:11:39.889 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:39 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:39.777+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:02.112246+0000 front 2024-03-28T16:11:02.111944+0000 (oldest deadline 2024-03-28T16:11:28.011785+0000) 2024-03-28T16:11:39.889 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:39 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:39.777+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-03-28T16:11:12.113739+0000 front 2024-03-28T16:11:12.114043+0000 (oldest deadline 2024-03-28T16:11:37.413699+0000) 2024-03-28T16:11:39.933 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:39 smithi067 podman[91737]: 2024-03-28 16:11:39.633296053 +0000 UTC m=+5.625200851 container died 8fd3cd101854274d8668beef0bd4a7f825ee698c71a6dd4a6d02ac032828f1a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-28T16:11:40.100 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:39 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:39.673+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:11:40.100 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:39 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:39.673+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.053696+0000 front 2024-03-28T16:10:24.053904+0000 (oldest deadline 2024-03-28T16:10:48.753813+0000) 2024-03-28T16:11:40.100 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:39 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:39.673+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:33.258214+0000 front 2024-03-28T16:10:33.258863+0000 (oldest deadline 2024-03-28T16:10:57.958232+0000) 2024-03-28T16:11:40.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:39 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:39.673+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:53.262996+0000 front 2024-03-28T16:10:53.262892+0000 (oldest deadline 2024-03-28T16:11:18.463901+0000) 2024-03-28T16:11:40.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:39 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:39.673+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:00.165831+0000 front 2024-03-28T16:11:00.165617+0000 (oldest deadline 2024-03-28T16:11:24.265608+0000) 2024-03-28T16:11:40.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:39 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:39.673+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-03-28T16:11:07.068385+0000 front 2024-03-28T16:11:07.068776+0000 (oldest deadline 2024-03-28T16:11:32.968340+0000) 2024-03-28T16:11:40.312 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:40 smithi067 podman[91737]: 2024-03-28 16:11:40.052400623 +0000 UTC m=+6.044305432 container cleanup 8fd3cd101854274d8668beef0bd4a7f825ee698c71a6dd4a6d02ac032828f1a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-28T16:11:40.312 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:40 smithi067 bash[91737]: ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9 2024-03-28T16:11:40.569 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:40 smithi067 podman[91755]: 2024-03-28 16:11:40.311724396 +0000 UTC m=+0.664159815 container remove 8fd3cd101854274d8668beef0bd4a7f825ee698c71a6dd4a6d02ac032828f1a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-28T16:11:41.053 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:40 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:40.819+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:11:41.053 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:40 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:40.819+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.601392+0000 front 2024-03-28T16:10:23.601326+0000 (oldest deadline 2024-03-28T16:10:47.701522+0000) 2024-03-28T16:11:41.053 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:40 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:40.819+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.304820+0000 front 2024-03-28T16:10:35.304495+0000 (oldest deadline 2024-03-28T16:10:59.404389+0000) 2024-03-28T16:11:41.053 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:40 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:40.819+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:44.107005+0000 front 2024-03-28T16:10:44.106857+0000 (oldest deadline 2024-03-28T16:11:08.206356+0000) 2024-03-28T16:11:41.054 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:40 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:40.819+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:52.908191+0000 front 2024-03-28T16:10:52.908270+0000 (oldest deadline 2024-03-28T16:11:18.808335+0000) 2024-03-28T16:11:41.054 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:40 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:40.819+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:02.112246+0000 front 2024-03-28T16:11:02.111944+0000 (oldest deadline 2024-03-28T16:11:28.011785+0000) 2024-03-28T16:11:41.054 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:40 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:40.819+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-03-28T16:11:12.113739+0000 front 2024-03-28T16:11:12.114043+0000 (oldest deadline 2024-03-28T16:11:37.413699+0000) 2024-03-28T16:11:41.100 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:40 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:40.657+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:11:41.100 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:40 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:40.657+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.053696+0000 front 2024-03-28T16:10:24.053904+0000 (oldest deadline 2024-03-28T16:10:48.753813+0000) 2024-03-28T16:11:41.100 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:40 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:40.657+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:33.258214+0000 front 2024-03-28T16:10:33.258863+0000 (oldest deadline 2024-03-28T16:10:57.958232+0000) 2024-03-28T16:11:41.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:40 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:40.657+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:53.262996+0000 front 2024-03-28T16:10:53.262892+0000 (oldest deadline 2024-03-28T16:11:18.463901+0000) 2024-03-28T16:11:41.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:40 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:40.657+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:00.165831+0000 front 2024-03-28T16:11:00.165617+0000 (oldest deadline 2024-03-28T16:11:24.265608+0000) 2024-03-28T16:11:41.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:40 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:40.657+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-03-28T16:11:07.068385+0000 front 2024-03-28T16:11:07.068776+0000 (oldest deadline 2024-03-28T16:11:32.968340+0000) 2024-03-28T16:11:41.459 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:41 smithi067 podman[91841]: 2024-03-28 16:11:41.096221747 +0000 UTC m=+0.034586114 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T16:11:41.459 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:41 smithi067 podman[91841]: 2024-03-28 16:11:41.195834985 +0000 UTC m=+0.134199372 container create 79fd027da25b307e0402248710d4fa4b6ba1ac7eb109067830844ebe187da7e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-28T16:11:41.459 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:41 smithi067 podman[91841]: 2024-03-28 16:11:41.458240108 +0000 UTC m=+0.396604502 container init 79fd027da25b307e0402248710d4fa4b6ba1ac7eb109067830844ebe187da7e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-28T16:11:41.820 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:41 smithi067 podman[91841]: 2024-03-28 16:11:41.469072812 +0000 UTC m=+0.407437171 container start 79fd027da25b307e0402248710d4fa4b6ba1ac7eb109067830844ebe187da7e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9-deactivate, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319) 2024-03-28T16:11:41.820 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:41 smithi067 podman[91841]: 2024-03-28 16:11:41.539827155 +0000 UTC m=+0.478191592 container attach 79fd027da25b307e0402248710d4fa4b6ba1ac7eb109067830844ebe187da7e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319) 2024-03-28T16:11:42.052 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:41 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:41.676+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:11:42.053 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:41 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:41.676+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.053696+0000 front 2024-03-28T16:10:24.053904+0000 (oldest deadline 2024-03-28T16:10:48.753813+0000) 2024-03-28T16:11:42.053 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:41 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:41.676+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:33.258214+0000 front 2024-03-28T16:10:33.258863+0000 (oldest deadline 2024-03-28T16:10:57.958232+0000) 2024-03-28T16:11:42.053 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:41 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:41.676+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:53.262996+0000 front 2024-03-28T16:10:53.262892+0000 (oldest deadline 2024-03-28T16:11:18.463901+0000) 2024-03-28T16:11:42.053 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:41 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:41.676+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:00.165831+0000 front 2024-03-28T16:11:00.165617+0000 (oldest deadline 2024-03-28T16:11:24.265608+0000) 2024-03-28T16:11:42.053 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:41 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:41.676+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-03-28T16:11:07.068385+0000 front 2024-03-28T16:11:07.068776+0000 (oldest deadline 2024-03-28T16:11:32.968340+0000) 2024-03-28T16:11:42.053 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:41 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:41.799+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:11:42.053 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:41 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:41.799+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.601392+0000 front 2024-03-28T16:10:23.601326+0000 (oldest deadline 2024-03-28T16:10:47.701522+0000) 2024-03-28T16:11:42.054 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:41 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:41.799+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.304820+0000 front 2024-03-28T16:10:35.304495+0000 (oldest deadline 2024-03-28T16:10:59.404389+0000) 2024-03-28T16:11:42.054 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:41 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:41.799+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:44.107005+0000 front 2024-03-28T16:10:44.106857+0000 (oldest deadline 2024-03-28T16:11:08.206356+0000) 2024-03-28T16:11:42.054 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:41 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:41.799+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:52.908191+0000 front 2024-03-28T16:10:52.908270+0000 (oldest deadline 2024-03-28T16:11:18.808335+0000) 2024-03-28T16:11:42.054 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:41 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:41.799+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:02.112246+0000 front 2024-03-28T16:11:02.111944+0000 (oldest deadline 2024-03-28T16:11:28.011785+0000) 2024-03-28T16:11:42.054 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:41 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:41.799+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-03-28T16:11:12.113739+0000 front 2024-03-28T16:11:12.114043+0000 (oldest deadline 2024-03-28T16:11:37.413699+0000) 2024-03-28T16:11:42.277 INFO:journalctl@ceph.osd.9.smithi067.stdout:Mar 28 16:11:41 smithi067 podman[91841]: 2024-03-28 16:11:41.918899463 +0000 UTC m=+0.857263857 container died 79fd027da25b307e0402248710d4fa4b6ba1ac7eb109067830844ebe187da7e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-9-deactivate, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T16:11:42.625 DEBUG:teuthology.orchestra.run.smithi067:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@osd.9.service' 2024-03-28T16:11:42.748 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T16:11:42.748 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-03-28T16:11:42.749 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-03-28T16:11:42.749 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@osd.10 2024-03-28T16:11:43.100 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:42 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:42.667+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:11:43.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:42 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:42.667+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.053696+0000 front 2024-03-28T16:10:24.053904+0000 (oldest deadline 2024-03-28T16:10:48.753813+0000) 2024-03-28T16:11:43.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:42 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:42.667+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:33.258214+0000 front 2024-03-28T16:10:33.258863+0000 (oldest deadline 2024-03-28T16:10:57.958232+0000) 2024-03-28T16:11:43.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:42 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:42.667+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:53.262996+0000 front 2024-03-28T16:10:53.262892+0000 (oldest deadline 2024-03-28T16:11:18.463901+0000) 2024-03-28T16:11:43.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:42 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:42.667+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:00.165831+0000 front 2024-03-28T16:11:00.165617+0000 (oldest deadline 2024-03-28T16:11:24.265608+0000) 2024-03-28T16:11:43.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:42 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:42.667+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-03-28T16:11:07.068385+0000 front 2024-03-28T16:11:07.068776+0000 (oldest deadline 2024-03-28T16:11:32.968340+0000) 2024-03-28T16:11:43.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:42 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:42.843+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:11:43.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:42 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:42.843+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.601392+0000 front 2024-03-28T16:10:23.601326+0000 (oldest deadline 2024-03-28T16:10:47.701522+0000) 2024-03-28T16:11:43.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:42 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:42.843+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.304820+0000 front 2024-03-28T16:10:35.304495+0000 (oldest deadline 2024-03-28T16:10:59.404389+0000) 2024-03-28T16:11:43.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:42 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:42.843+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:44.107005+0000 front 2024-03-28T16:10:44.106857+0000 (oldest deadline 2024-03-28T16:11:08.206356+0000) 2024-03-28T16:11:43.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:42 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:42.843+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:52.908191+0000 front 2024-03-28T16:10:52.908270+0000 (oldest deadline 2024-03-28T16:11:18.808335+0000) 2024-03-28T16:11:43.304 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:42 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:42.843+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:02.112246+0000 front 2024-03-28T16:11:02.111944+0000 (oldest deadline 2024-03-28T16:11:28.011785+0000) 2024-03-28T16:11:43.304 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:42 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:42.843+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-03-28T16:11:12.113739+0000 front 2024-03-28T16:11:12.114043+0000 (oldest deadline 2024-03-28T16:11:37.413699+0000) 2024-03-28T16:11:43.304 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:42 smithi150 systemd[1]: Stopping Ceph osd.10 for b5f8225a-ed1a-11ee-95d0-87774f69a715... 2024-03-28T16:11:43.802 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:43 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:43.336+0000 7f6b25200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T16:11:43.802 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:43 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:43.336+0000 7f6b25200700 -1 osd.10 95 *** Got signal Terminated *** 2024-03-28T16:11:43.802 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:43 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:43.336+0000 7f6b25200700 -1 osd.10 95 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T16:11:44.100 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:43 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:43.655+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:11:44.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:43 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:43.655+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.053696+0000 front 2024-03-28T16:10:24.053904+0000 (oldest deadline 2024-03-28T16:10:48.753813+0000) 2024-03-28T16:11:44.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:43 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:43.655+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:33.258214+0000 front 2024-03-28T16:10:33.258863+0000 (oldest deadline 2024-03-28T16:10:57.958232+0000) 2024-03-28T16:11:44.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:43 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:43.655+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:53.262996+0000 front 2024-03-28T16:10:53.262892+0000 (oldest deadline 2024-03-28T16:11:18.463901+0000) 2024-03-28T16:11:44.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:43 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:43.655+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:00.165831+0000 front 2024-03-28T16:11:00.165617+0000 (oldest deadline 2024-03-28T16:11:24.265608+0000) 2024-03-28T16:11:44.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:43 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:43.655+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-03-28T16:11:07.068385+0000 front 2024-03-28T16:11:07.068776+0000 (oldest deadline 2024-03-28T16:11:32.968340+0000) 2024-03-28T16:11:44.302 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:43 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:43.887+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:11:44.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:43 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:43.887+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.601392+0000 front 2024-03-28T16:10:23.601326+0000 (oldest deadline 2024-03-28T16:10:47.701522+0000) 2024-03-28T16:11:44.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:43 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:43.887+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.304820+0000 front 2024-03-28T16:10:35.304495+0000 (oldest deadline 2024-03-28T16:10:59.404389+0000) 2024-03-28T16:11:44.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:43 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:43.887+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:44.107005+0000 front 2024-03-28T16:10:44.106857+0000 (oldest deadline 2024-03-28T16:11:08.206356+0000) 2024-03-28T16:11:44.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:43 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:43.887+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:52.908191+0000 front 2024-03-28T16:10:52.908270+0000 (oldest deadline 2024-03-28T16:11:18.808335+0000) 2024-03-28T16:11:44.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:43 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:43.887+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:02.112246+0000 front 2024-03-28T16:11:02.111944+0000 (oldest deadline 2024-03-28T16:11:28.011785+0000) 2024-03-28T16:11:44.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:43 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:43.887+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-03-28T16:11:12.113739+0000 front 2024-03-28T16:11:12.114043+0000 (oldest deadline 2024-03-28T16:11:37.413699+0000) 2024-03-28T16:11:45.100 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:44 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:44.684+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:11:45.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:44 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:44.684+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.053696+0000 front 2024-03-28T16:10:24.053904+0000 (oldest deadline 2024-03-28T16:10:48.753813+0000) 2024-03-28T16:11:45.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:44 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:44.684+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:33.258214+0000 front 2024-03-28T16:10:33.258863+0000 (oldest deadline 2024-03-28T16:10:57.958232+0000) 2024-03-28T16:11:45.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:44 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:44.684+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:53.262996+0000 front 2024-03-28T16:10:53.262892+0000 (oldest deadline 2024-03-28T16:11:18.463901+0000) 2024-03-28T16:11:45.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:44 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:44.684+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:00.165831+0000 front 2024-03-28T16:11:00.165617+0000 (oldest deadline 2024-03-28T16:11:24.265608+0000) 2024-03-28T16:11:45.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:44 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:44.684+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-03-28T16:11:07.068385+0000 front 2024-03-28T16:11:07.068776+0000 (oldest deadline 2024-03-28T16:11:32.968340+0000) 2024-03-28T16:11:45.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:44 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:44.684+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6820 osd.7 since back 2024-03-28T16:11:18.870289+0000 front 2024-03-28T16:11:18.870354+0000 (oldest deadline 2024-03-28T16:11:44.170006+0000) 2024-03-28T16:11:45.302 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:44 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:44.935+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:11:45.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:44 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:44.935+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.601392+0000 front 2024-03-28T16:10:23.601326+0000 (oldest deadline 2024-03-28T16:10:47.701522+0000) 2024-03-28T16:11:45.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:44 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:44.935+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.304820+0000 front 2024-03-28T16:10:35.304495+0000 (oldest deadline 2024-03-28T16:10:59.404389+0000) 2024-03-28T16:11:45.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:44 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:44.935+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:44.107005+0000 front 2024-03-28T16:10:44.106857+0000 (oldest deadline 2024-03-28T16:11:08.206356+0000) 2024-03-28T16:11:45.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:44 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:44.935+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:52.908191+0000 front 2024-03-28T16:10:52.908270+0000 (oldest deadline 2024-03-28T16:11:18.808335+0000) 2024-03-28T16:11:45.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:44 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:44.935+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:02.112246+0000 front 2024-03-28T16:11:02.111944+0000 (oldest deadline 2024-03-28T16:11:28.011785+0000) 2024-03-28T16:11:45.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:44 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:44.935+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-03-28T16:11:12.113739+0000 front 2024-03-28T16:11:12.114043+0000 (oldest deadline 2024-03-28T16:11:37.413699+0000) 2024-03-28T16:11:46.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:45 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:45.731+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:11:46.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:45 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:45.731+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.053696+0000 front 2024-03-28T16:10:24.053904+0000 (oldest deadline 2024-03-28T16:10:48.753813+0000) 2024-03-28T16:11:46.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:45 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:45.731+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:33.258214+0000 front 2024-03-28T16:10:33.258863+0000 (oldest deadline 2024-03-28T16:10:57.958232+0000) 2024-03-28T16:11:46.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:45 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:45.731+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:53.262996+0000 front 2024-03-28T16:10:53.262892+0000 (oldest deadline 2024-03-28T16:11:18.463901+0000) 2024-03-28T16:11:46.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:45 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:45.731+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:00.165831+0000 front 2024-03-28T16:11:00.165617+0000 (oldest deadline 2024-03-28T16:11:24.265608+0000) 2024-03-28T16:11:46.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:45 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:45.731+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-03-28T16:11:07.068385+0000 front 2024-03-28T16:11:07.068776+0000 (oldest deadline 2024-03-28T16:11:32.968340+0000) 2024-03-28T16:11:46.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:45 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:45.731+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6820 osd.7 since back 2024-03-28T16:11:18.870289+0000 front 2024-03-28T16:11:18.870354+0000 (oldest deadline 2024-03-28T16:11:44.170006+0000) 2024-03-28T16:11:46.302 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:45 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:45.961+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:11:46.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:45 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:45.961+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.601392+0000 front 2024-03-28T16:10:23.601326+0000 (oldest deadline 2024-03-28T16:10:47.701522+0000) 2024-03-28T16:11:46.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:45 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:45.961+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.304820+0000 front 2024-03-28T16:10:35.304495+0000 (oldest deadline 2024-03-28T16:10:59.404389+0000) 2024-03-28T16:11:46.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:45 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:45.961+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:44.107005+0000 front 2024-03-28T16:10:44.106857+0000 (oldest deadline 2024-03-28T16:11:08.206356+0000) 2024-03-28T16:11:46.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:45 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:45.961+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:52.908191+0000 front 2024-03-28T16:10:52.908270+0000 (oldest deadline 2024-03-28T16:11:18.808335+0000) 2024-03-28T16:11:46.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:45 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:45.961+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:02.112246+0000 front 2024-03-28T16:11:02.111944+0000 (oldest deadline 2024-03-28T16:11:28.011785+0000) 2024-03-28T16:11:46.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:45 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:45.961+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-03-28T16:11:12.113739+0000 front 2024-03-28T16:11:12.114043+0000 (oldest deadline 2024-03-28T16:11:37.413699+0000) 2024-03-28T16:11:47.100 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:46 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:46.719+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:11:47.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:46 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:46.719+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.053696+0000 front 2024-03-28T16:10:24.053904+0000 (oldest deadline 2024-03-28T16:10:48.753813+0000) 2024-03-28T16:11:47.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:46 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:46.719+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:33.258214+0000 front 2024-03-28T16:10:33.258863+0000 (oldest deadline 2024-03-28T16:10:57.958232+0000) 2024-03-28T16:11:47.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:46 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:46.719+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:53.262996+0000 front 2024-03-28T16:10:53.262892+0000 (oldest deadline 2024-03-28T16:11:18.463901+0000) 2024-03-28T16:11:47.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:46 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:46.719+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:00.165831+0000 front 2024-03-28T16:11:00.165617+0000 (oldest deadline 2024-03-28T16:11:24.265608+0000) 2024-03-28T16:11:47.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:46 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:46.719+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-03-28T16:11:07.068385+0000 front 2024-03-28T16:11:07.068776+0000 (oldest deadline 2024-03-28T16:11:32.968340+0000) 2024-03-28T16:11:47.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:46 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:46.719+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6820 osd.7 since back 2024-03-28T16:11:18.870289+0000 front 2024-03-28T16:11:18.870354+0000 (oldest deadline 2024-03-28T16:11:44.170006+0000) 2024-03-28T16:11:47.302 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:46 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:46.942+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:11:47.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:46 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:46.942+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.601392+0000 front 2024-03-28T16:10:23.601326+0000 (oldest deadline 2024-03-28T16:10:47.701522+0000) 2024-03-28T16:11:47.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:46 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:46.942+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.304820+0000 front 2024-03-28T16:10:35.304495+0000 (oldest deadline 2024-03-28T16:10:59.404389+0000) 2024-03-28T16:11:47.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:46 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:46.942+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:44.107005+0000 front 2024-03-28T16:10:44.106857+0000 (oldest deadline 2024-03-28T16:11:08.206356+0000) 2024-03-28T16:11:47.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:46 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:46.942+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:52.908191+0000 front 2024-03-28T16:10:52.908270+0000 (oldest deadline 2024-03-28T16:11:18.808335+0000) 2024-03-28T16:11:47.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:46 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:46.942+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:02.112246+0000 front 2024-03-28T16:11:02.111944+0000 (oldest deadline 2024-03-28T16:11:28.011785+0000) 2024-03-28T16:11:47.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:46 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:46.942+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-03-28T16:11:12.113739+0000 front 2024-03-28T16:11:12.114043+0000 (oldest deadline 2024-03-28T16:11:37.413699+0000) 2024-03-28T16:11:48.100 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:47 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:47.695+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:11:48.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:47 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:47.695+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.053696+0000 front 2024-03-28T16:10:24.053904+0000 (oldest deadline 2024-03-28T16:10:48.753813+0000) 2024-03-28T16:11:48.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:47 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:47.695+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:33.258214+0000 front 2024-03-28T16:10:33.258863+0000 (oldest deadline 2024-03-28T16:10:57.958232+0000) 2024-03-28T16:11:48.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:47 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:47.695+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:53.262996+0000 front 2024-03-28T16:10:53.262892+0000 (oldest deadline 2024-03-28T16:11:18.463901+0000) 2024-03-28T16:11:48.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:47 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:47.695+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:00.165831+0000 front 2024-03-28T16:11:00.165617+0000 (oldest deadline 2024-03-28T16:11:24.265608+0000) 2024-03-28T16:11:48.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:47 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:47.695+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-03-28T16:11:07.068385+0000 front 2024-03-28T16:11:07.068776+0000 (oldest deadline 2024-03-28T16:11:32.968340+0000) 2024-03-28T16:11:48.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:47 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:47.695+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6820 osd.7 since back 2024-03-28T16:11:18.870289+0000 front 2024-03-28T16:11:18.870354+0000 (oldest deadline 2024-03-28T16:11:44.170006+0000) 2024-03-28T16:11:48.302 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:47 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:47.954+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:16.796944+0000 front 2024-03-28T16:10:16.797572+0000 (oldest deadline 2024-03-28T16:10:38.497160+0000) 2024-03-28T16:11:48.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:47 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:47.954+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:23.601392+0000 front 2024-03-28T16:10:23.601326+0000 (oldest deadline 2024-03-28T16:10:47.701522+0000) 2024-03-28T16:11:48.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:47 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:47.954+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:35.304820+0000 front 2024-03-28T16:10:35.304495+0000 (oldest deadline 2024-03-28T16:10:59.404389+0000) 2024-03-28T16:11:48.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:47 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:47.954+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-03-28T16:10:44.107005+0000 front 2024-03-28T16:10:44.106857+0000 (oldest deadline 2024-03-28T16:11:08.206356+0000) 2024-03-28T16:11:48.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:47 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:47.954+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:52.908191+0000 front 2024-03-28T16:10:52.908270+0000 (oldest deadline 2024-03-28T16:11:18.808335+0000) 2024-03-28T16:11:48.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:47 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:47.954+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:02.112246+0000 front 2024-03-28T16:11:02.111944+0000 (oldest deadline 2024-03-28T16:11:28.011785+0000) 2024-03-28T16:11:48.303 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:47 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10[60027]: 2024-03-28T16:11:47.954+0000 7f6b1ba00700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-03-28T16:11:12.113739+0000 front 2024-03-28T16:11:12.114043+0000 (oldest deadline 2024-03-28T16:11:37.413699+0000) 2024-03-28T16:11:49.100 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:48 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:48.729+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:11:49.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:48 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:48.729+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.053696+0000 front 2024-03-28T16:10:24.053904+0000 (oldest deadline 2024-03-28T16:10:48.753813+0000) 2024-03-28T16:11:49.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:48 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:48.729+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:33.258214+0000 front 2024-03-28T16:10:33.258863+0000 (oldest deadline 2024-03-28T16:10:57.958232+0000) 2024-03-28T16:11:49.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:48 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:48.729+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:53.262996+0000 front 2024-03-28T16:10:53.262892+0000 (oldest deadline 2024-03-28T16:11:18.463901+0000) 2024-03-28T16:11:49.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:48 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:48.729+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:00.165831+0000 front 2024-03-28T16:11:00.165617+0000 (oldest deadline 2024-03-28T16:11:24.265608+0000) 2024-03-28T16:11:49.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:48 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:48.729+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-03-28T16:11:07.068385+0000 front 2024-03-28T16:11:07.068776+0000 (oldest deadline 2024-03-28T16:11:32.968340+0000) 2024-03-28T16:11:49.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:48 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:48.729+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6820 osd.7 since back 2024-03-28T16:11:18.870289+0000 front 2024-03-28T16:11:18.870354+0000 (oldest deadline 2024-03-28T16:11:44.170006+0000) 2024-03-28T16:11:49.184 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:48 smithi150 podman[69406]: 2024-03-28 16:11:48.826900518 +0000 UTC m=+5.653230149 container died 7fb4df68f28e12c7636570c5fbdb59bcec1fcebe9f8e88cfec57c5331c593ece (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-28T16:11:49.480 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:49 smithi150 podman[69406]: 2024-03-28 16:11:49.303075208 +0000 UTC m=+6.129404859 container cleanup 7fb4df68f28e12c7636570c5fbdb59bcec1fcebe9f8e88cfec57c5331c593ece (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True) 2024-03-28T16:11:49.480 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:49 smithi150 bash[69406]: ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10 2024-03-28T16:11:49.802 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:49 smithi150 podman[69425]: 2024-03-28 16:11:49.479139337 +0000 UTC m=+0.638734519 container remove 7fb4df68f28e12c7636570c5fbdb59bcec1fcebe9f8e88cfec57c5331c593ece (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-28T16:11:50.100 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:49 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:49.696+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:11:50.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:49 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:49.696+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.053696+0000 front 2024-03-28T16:10:24.053904+0000 (oldest deadline 2024-03-28T16:10:48.753813+0000) 2024-03-28T16:11:50.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:49 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:49.696+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:33.258214+0000 front 2024-03-28T16:10:33.258863+0000 (oldest deadline 2024-03-28T16:10:57.958232+0000) 2024-03-28T16:11:50.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:49 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:49.696+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:53.262996+0000 front 2024-03-28T16:10:53.262892+0000 (oldest deadline 2024-03-28T16:11:18.463901+0000) 2024-03-28T16:11:50.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:49 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:49.696+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:00.165831+0000 front 2024-03-28T16:11:00.165617+0000 (oldest deadline 2024-03-28T16:11:24.265608+0000) 2024-03-28T16:11:50.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:49 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:49.696+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-03-28T16:11:07.068385+0000 front 2024-03-28T16:11:07.068776+0000 (oldest deadline 2024-03-28T16:11:32.968340+0000) 2024-03-28T16:11:50.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:49 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:49.696+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6820 osd.7 since back 2024-03-28T16:11:18.870289+0000 front 2024-03-28T16:11:18.870354+0000 (oldest deadline 2024-03-28T16:11:44.170006+0000) 2024-03-28T16:11:50.492 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:50 smithi150 podman[69513]: 2024-03-28 16:11:50.179124898 +0000 UTC m=+0.037550790 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T16:11:50.492 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:50 smithi150 podman[69513]: 2024-03-28 16:11:50.346512106 +0000 UTC m=+0.204937983 container create 7db363fef5ae5e04436659b5ffdd44ac0dbd1fa053146ae704233275a43200db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD) 2024-03-28T16:11:51.036 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:50 smithi150 podman[69513]: 2024-03-28 16:11:50.599703442 +0000 UTC m=+0.458129336 container init 7db363fef5ae5e04436659b5ffdd44ac0dbd1fa053146ae704233275a43200db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319) 2024-03-28T16:11:51.037 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:50 smithi150 podman[69513]: 2024-03-28 16:11:50.606992148 +0000 UTC m=+0.465418028 container start 7db363fef5ae5e04436659b5ffdd44ac0dbd1fa053146ae704233275a43200db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319) 2024-03-28T16:11:51.037 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:50 smithi150 podman[69513]: 2024-03-28 16:11:50.690404077 +0000 UTC m=+0.548830040 container attach 7db363fef5ae5e04436659b5ffdd44ac0dbd1fa053146ae704233275a43200db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319) 2024-03-28T16:11:51.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:50 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:50.692+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:11:51.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:50 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:50.692+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.053696+0000 front 2024-03-28T16:10:24.053904+0000 (oldest deadline 2024-03-28T16:10:48.753813+0000) 2024-03-28T16:11:51.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:50 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:50.692+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:33.258214+0000 front 2024-03-28T16:10:33.258863+0000 (oldest deadline 2024-03-28T16:10:57.958232+0000) 2024-03-28T16:11:51.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:50 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:50.692+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:53.262996+0000 front 2024-03-28T16:10:53.262892+0000 (oldest deadline 2024-03-28T16:11:18.463901+0000) 2024-03-28T16:11:51.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:50 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:50.692+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:00.165831+0000 front 2024-03-28T16:11:00.165617+0000 (oldest deadline 2024-03-28T16:11:24.265608+0000) 2024-03-28T16:11:51.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:50 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:50.692+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-03-28T16:11:07.068385+0000 front 2024-03-28T16:11:07.068776+0000 (oldest deadline 2024-03-28T16:11:32.968340+0000) 2024-03-28T16:11:51.102 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:50 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:50.692+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6820 osd.7 since back 2024-03-28T16:11:18.870289+0000 front 2024-03-28T16:11:18.870354+0000 (oldest deadline 2024-03-28T16:11:44.170006+0000) 2024-03-28T16:11:51.302 INFO:journalctl@ceph.osd.10.smithi150.stdout:Mar 28 16:11:51 smithi150 podman[69513]: 2024-03-28 16:11:51.036704658 +0000 UTC m=+0.895130608 container died 7db363fef5ae5e04436659b5ffdd44ac0dbd1fa053146ae704233275a43200db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-10-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-28T16:11:51.718 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@osd.10.service' 2024-03-28T16:11:51.882 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T16:11:51.882 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-03-28T16:11:51.882 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-03-28T16:11:51.882 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@osd.11 2024-03-28T16:11:51.968 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:51 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:51.675+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:11:51.969 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:51 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:51.675+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.053696+0000 front 2024-03-28T16:10:24.053904+0000 (oldest deadline 2024-03-28T16:10:48.753813+0000) 2024-03-28T16:11:51.969 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:51 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:51.675+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:33.258214+0000 front 2024-03-28T16:10:33.258863+0000 (oldest deadline 2024-03-28T16:10:57.958232+0000) 2024-03-28T16:11:51.969 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:51 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:51.675+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:53.262996+0000 front 2024-03-28T16:10:53.262892+0000 (oldest deadline 2024-03-28T16:11:18.463901+0000) 2024-03-28T16:11:51.969 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:51 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:51.675+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:00.165831+0000 front 2024-03-28T16:11:00.165617+0000 (oldest deadline 2024-03-28T16:11:24.265608+0000) 2024-03-28T16:11:51.969 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:51 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:51.675+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-03-28T16:11:07.068385+0000 front 2024-03-28T16:11:07.068776+0000 (oldest deadline 2024-03-28T16:11:32.968340+0000) 2024-03-28T16:11:51.969 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:51 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:51.675+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6820 osd.7 since back 2024-03-28T16:11:18.870289+0000 front 2024-03-28T16:11:18.870354+0000 (oldest deadline 2024-03-28T16:11:44.170006+0000) 2024-03-28T16:11:51.969 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:51 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:51.675+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6820 osd.8 since back 2024-03-28T16:11:28.172423+0000 front 2024-03-28T16:11:28.172295+0000 (oldest deadline 2024-03-28T16:11:51.072232+0000) 2024-03-28T16:11:52.350 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:52 smithi183 systemd[1]: Stopping Ceph osd.11 for b5f8225a-ed1a-11ee-95d0-87774f69a715... 2024-03-28T16:11:52.725 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:52 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:52.454+0000 7ffb57e00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T16:11:52.726 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:52 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:52.454+0000 7ffb57e00700 -1 osd.11 95 *** Got signal Terminated *** 2024-03-28T16:11:52.726 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:52 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:52.454+0000 7ffb57e00700 -1 osd.11 95 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T16:11:52.726 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:52 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:52.723+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:11:52.726 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:52 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:52.723+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.053696+0000 front 2024-03-28T16:10:24.053904+0000 (oldest deadline 2024-03-28T16:10:48.753813+0000) 2024-03-28T16:11:52.726 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:52 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:52.723+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:33.258214+0000 front 2024-03-28T16:10:33.258863+0000 (oldest deadline 2024-03-28T16:10:57.958232+0000) 2024-03-28T16:11:53.100 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:52 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:52.723+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:53.262996+0000 front 2024-03-28T16:10:53.262892+0000 (oldest deadline 2024-03-28T16:11:18.463901+0000) 2024-03-28T16:11:53.100 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:52 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:52.723+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:00.165831+0000 front 2024-03-28T16:11:00.165617+0000 (oldest deadline 2024-03-28T16:11:24.265608+0000) 2024-03-28T16:11:53.100 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:52 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:52.723+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-03-28T16:11:07.068385+0000 front 2024-03-28T16:11:07.068776+0000 (oldest deadline 2024-03-28T16:11:32.968340+0000) 2024-03-28T16:11:53.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:52 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:52.723+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6820 osd.7 since back 2024-03-28T16:11:18.870289+0000 front 2024-03-28T16:11:18.870354+0000 (oldest deadline 2024-03-28T16:11:44.170006+0000) 2024-03-28T16:11:53.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:52 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:52.723+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6820 osd.8 since back 2024-03-28T16:11:28.172423+0000 front 2024-03-28T16:11:28.172295+0000 (oldest deadline 2024-03-28T16:11:51.072232+0000) 2024-03-28T16:11:54.100 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:53 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:53.730+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:11:54.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:53 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:53.730+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.053696+0000 front 2024-03-28T16:10:24.053904+0000 (oldest deadline 2024-03-28T16:10:48.753813+0000) 2024-03-28T16:11:54.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:53 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:53.730+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:33.258214+0000 front 2024-03-28T16:10:33.258863+0000 (oldest deadline 2024-03-28T16:10:57.958232+0000) 2024-03-28T16:11:54.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:53 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:53.730+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:53.262996+0000 front 2024-03-28T16:10:53.262892+0000 (oldest deadline 2024-03-28T16:11:18.463901+0000) 2024-03-28T16:11:54.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:53 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:53.730+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:00.165831+0000 front 2024-03-28T16:11:00.165617+0000 (oldest deadline 2024-03-28T16:11:24.265608+0000) 2024-03-28T16:11:54.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:53 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:53.730+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-03-28T16:11:07.068385+0000 front 2024-03-28T16:11:07.068776+0000 (oldest deadline 2024-03-28T16:11:32.968340+0000) 2024-03-28T16:11:54.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:53 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:53.730+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6820 osd.7 since back 2024-03-28T16:11:18.870289+0000 front 2024-03-28T16:11:18.870354+0000 (oldest deadline 2024-03-28T16:11:44.170006+0000) 2024-03-28T16:11:54.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:53 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:53.730+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6820 osd.8 since back 2024-03-28T16:11:28.172423+0000 front 2024-03-28T16:11:28.172295+0000 (oldest deadline 2024-03-28T16:11:51.072232+0000) 2024-03-28T16:11:55.100 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:54 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:54.766+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:11:55.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:54 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:54.766+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.053696+0000 front 2024-03-28T16:10:24.053904+0000 (oldest deadline 2024-03-28T16:10:48.753813+0000) 2024-03-28T16:11:55.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:54 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:54.766+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:33.258214+0000 front 2024-03-28T16:10:33.258863+0000 (oldest deadline 2024-03-28T16:10:57.958232+0000) 2024-03-28T16:11:55.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:54 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:54.766+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:53.262996+0000 front 2024-03-28T16:10:53.262892+0000 (oldest deadline 2024-03-28T16:11:18.463901+0000) 2024-03-28T16:11:55.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:54 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:54.767+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:00.165831+0000 front 2024-03-28T16:11:00.165617+0000 (oldest deadline 2024-03-28T16:11:24.265608+0000) 2024-03-28T16:11:55.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:54 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:54.767+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-03-28T16:11:07.068385+0000 front 2024-03-28T16:11:07.068776+0000 (oldest deadline 2024-03-28T16:11:32.968340+0000) 2024-03-28T16:11:55.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:54 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:54.767+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6820 osd.7 since back 2024-03-28T16:11:18.870289+0000 front 2024-03-28T16:11:18.870354+0000 (oldest deadline 2024-03-28T16:11:44.170006+0000) 2024-03-28T16:11:55.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:54 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:54.767+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6820 osd.8 since back 2024-03-28T16:11:28.172423+0000 front 2024-03-28T16:11:28.172295+0000 (oldest deadline 2024-03-28T16:11:51.072232+0000) 2024-03-28T16:11:56.100 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:55 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:55.789+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:11:56.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:55 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:55.789+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.053696+0000 front 2024-03-28T16:10:24.053904+0000 (oldest deadline 2024-03-28T16:10:48.753813+0000) 2024-03-28T16:11:56.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:55 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:55.789+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:33.258214+0000 front 2024-03-28T16:10:33.258863+0000 (oldest deadline 2024-03-28T16:10:57.958232+0000) 2024-03-28T16:11:56.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:55 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:55.789+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:53.262996+0000 front 2024-03-28T16:10:53.262892+0000 (oldest deadline 2024-03-28T16:11:18.463901+0000) 2024-03-28T16:11:56.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:55 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:55.789+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:00.165831+0000 front 2024-03-28T16:11:00.165617+0000 (oldest deadline 2024-03-28T16:11:24.265608+0000) 2024-03-28T16:11:56.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:55 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:55.789+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-03-28T16:11:07.068385+0000 front 2024-03-28T16:11:07.068776+0000 (oldest deadline 2024-03-28T16:11:32.968340+0000) 2024-03-28T16:11:56.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:55 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:55.789+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6820 osd.7 since back 2024-03-28T16:11:18.870289+0000 front 2024-03-28T16:11:18.870354+0000 (oldest deadline 2024-03-28T16:11:44.170006+0000) 2024-03-28T16:11:56.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:55 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:55.789+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6820 osd.8 since back 2024-03-28T16:11:28.172423+0000 front 2024-03-28T16:11:28.172295+0000 (oldest deadline 2024-03-28T16:11:51.072232+0000) 2024-03-28T16:11:57.100 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:56 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:56.758+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-03-28T16:10:15.950026+0000 front 2024-03-28T16:10:15.950307+0000 (oldest deadline 2024-03-28T16:10:40.050245+0000) 2024-03-28T16:11:57.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:56 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:56.758+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2024-03-28T16:10:24.053696+0000 front 2024-03-28T16:10:24.053904+0000 (oldest deadline 2024-03-28T16:10:48.753813+0000) 2024-03-28T16:11:57.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:56 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:56.758+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2024-03-28T16:10:33.258214+0000 front 2024-03-28T16:10:33.258863+0000 (oldest deadline 2024-03-28T16:10:57.958232+0000) 2024-03-28T16:11:57.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:56 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:56.758+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-03-28T16:10:53.262996+0000 front 2024-03-28T16:10:53.262892+0000 (oldest deadline 2024-03-28T16:11:18.463901+0000) 2024-03-28T16:11:57.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:56 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:56.758+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6812 osd.5 since back 2024-03-28T16:11:00.165831+0000 front 2024-03-28T16:11:00.165617+0000 (oldest deadline 2024-03-28T16:11:24.265608+0000) 2024-03-28T16:11:57.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:56 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:56.758+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-03-28T16:11:07.068385+0000 front 2024-03-28T16:11:07.068776+0000 (oldest deadline 2024-03-28T16:11:32.968340+0000) 2024-03-28T16:11:57.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:56 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:56.758+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.150:6820 osd.7 since back 2024-03-28T16:11:18.870289+0000 front 2024-03-28T16:11:18.870354+0000 (oldest deadline 2024-03-28T16:11:44.170006+0000) 2024-03-28T16:11:57.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:56 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11[60868]: 2024-03-28T16:11:56.758+0000 7ffb53600700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.183:6820 osd.8 since back 2024-03-28T16:11:28.172423+0000 front 2024-03-28T16:11:28.172295+0000 (oldest deadline 2024-03-28T16:11:51.072232+0000) 2024-03-28T16:11:58.214 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:57 smithi183 podman[70240]: 2024-03-28 16:11:57.873041675 +0000 UTC m=+5.549003362 container died 91ed0d98f521c07abfd91fcfaa7dc4ec7719a5cc9445043d8a2cdb5d528029ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T16:11:58.513 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:58 smithi183 podman[70240]: 2024-03-28 16:11:58.32374663 +0000 UTC m=+5.999708352 container cleanup 91ed0d98f521c07abfd91fcfaa7dc4ec7719a5cc9445043d8a2cdb5d528029ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, ceph=True) 2024-03-28T16:11:58.513 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:58 smithi183 bash[70240]: ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11 2024-03-28T16:11:58.850 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:58 smithi183 podman[70260]: 2024-03-28 16:11:58.512045063 +0000 UTC m=+0.629272159 container remove 91ed0d98f521c07abfd91fcfaa7dc4ec7719a5cc9445043d8a2cdb5d528029ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319) 2024-03-28T16:11:59.600 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:59 smithi183 podman[70346]: 2024-03-28 16:11:59.252864589 +0000 UTC m=+0.034695275 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T16:11:59.600 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:59 smithi183 podman[70346]: 2024-03-28 16:11:59.392370071 +0000 UTC m=+0.174200788 container create 51c55dbbe7131b9bfbea62c3143d83857fe3f90c3561971d961d85555a1661ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-28T16:12:00.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:59 smithi183 podman[70346]: 2024-03-28 16:11:59.663557512 +0000 UTC m=+0.445388236 container init 51c55dbbe7131b9bfbea62c3143d83857fe3f90c3561971d961d85555a1661ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-28T16:12:00.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:59 smithi183 podman[70346]: 2024-03-28 16:11:59.676304609 +0000 UTC m=+0.458135363 container start 51c55dbbe7131b9bfbea62c3143d83857fe3f90c3561971d961d85555a1661ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11-deactivate, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-28T16:12:00.101 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:11:59 smithi183 podman[70346]: 2024-03-28 16:11:59.777946615 +0000 UTC m=+0.559777327 container attach 51c55dbbe7131b9bfbea62c3143d83857fe3f90c3561971d961d85555a1661ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-28T16:12:00.469 INFO:journalctl@ceph.osd.11.smithi183.stdout:Mar 28 16:12:00 smithi183 podman[70346]: 2024-03-28 16:12:00.106634578 +0000 UTC m=+0.888465301 container died 51c55dbbe7131b9bfbea62c3143d83857fe3f90c3561971d961d85555a1661ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-osd-11-deactivate, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-28T16:12:00.900 DEBUG:teuthology.orchestra.run.smithi183:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@osd.11.service' 2024-03-28T16:12:01.001 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T16:12:01.001 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-03-28T16:12:01.001 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-03-28T16:12:01.001 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@mds.a 2024-03-28T16:12:01.570 INFO:journalctl@ceph.mds.a.smithi067.stdout:Mar 28 16:12:01 smithi067 systemd[1]: Stopping Ceph mds.a for b5f8225a-ed1a-11ee-95d0-87774f69a715... 2024-03-28T16:12:02.070 INFO:journalctl@ceph.mds.a.smithi067.stdout:Mar 28 16:12:01 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-a[73695]: 2024-03-28T16:12:01.601+0000 7ff517800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T16:12:02.070 INFO:journalctl@ceph.mds.a.smithi067.stdout:Mar 28 16:12:01 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-a[73695]: 2024-03-28T16:12:01.601+0000 7ff517800700 -1 mds.a *** got signal Terminated *** 2024-03-28T16:12:05.907 INFO:journalctl@ceph.mds.a.smithi067.stdout:Mar 28 16:12:05 smithi067 podman[91948]: 2024-03-28 16:12:05.6233057 +0000 UTC m=+4.159483281 container died 0d13172c8488f7ad865be21abfebfc4e6959d0b76871909e042f0d526032d664 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-a, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.schema-version=1.0) 2024-03-28T16:12:06.286 INFO:journalctl@ceph.mds.a.smithi067.stdout:Mar 28 16:12:06 smithi067 podman[91948]: 2024-03-28 16:12:06.018488964 +0000 UTC m=+4.554666538 container cleanup 0d13172c8488f7ad865be21abfebfc4e6959d0b76871909e042f0d526032d664 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-a, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T16:12:06.287 INFO:journalctl@ceph.mds.a.smithi067.stdout:Mar 28 16:12:06 smithi067 bash[91948]: ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-a 2024-03-28T16:12:06.563 INFO:journalctl@ceph.mds.a.smithi067.stdout:Mar 28 16:12:06 smithi067 podman[91962]: 2024-03-28 16:12:06.286070908 +0000 UTC m=+0.650023762 container remove 0d13172c8488f7ad865be21abfebfc4e6959d0b76871909e042f0d526032d664 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-28T16:12:06.597 DEBUG:teuthology.orchestra.run.smithi067:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@mds.a.service' 2024-03-28T16:12:06.721 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T16:12:06.722 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-03-28T16:12:06.722 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-03-28T16:12:06.722 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@mds.d 2024-03-28T16:12:07.319 INFO:journalctl@ceph.mds.d.smithi067.stdout:Mar 28 16:12:06 smithi067 systemd[1]: Stopping Ceph mds.d for b5f8225a-ed1a-11ee-95d0-87774f69a715... 2024-03-28T16:12:07.819 INFO:journalctl@ceph.mds.d.smithi067.stdout:Mar 28 16:12:07 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-d[72659]: 2024-03-28T16:12:07.329+0000 7fd874e00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T16:12:07.820 INFO:journalctl@ceph.mds.d.smithi067.stdout:Mar 28 16:12:07 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-d[72659]: 2024-03-28T16:12:07.329+0000 7fd874e00700 -1 mds.d *** got signal Terminated *** 2024-03-28T16:12:11.069 INFO:journalctl@ceph.mds.d.smithi067.stdout:Mar 28 16:12:10 smithi067 podman[92067]: 2024-03-28 16:12:10.653831654 +0000 UTC m=+3.452623925 container died 92b9cad2c191477db5fbece171102b6ea0d7bef324555141fb330efa45681363 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-d, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T16:12:11.415 INFO:journalctl@ceph.mds.d.smithi067.stdout:Mar 28 16:12:11 smithi067 podman[92067]: 2024-03-28 16:12:11.213507972 +0000 UTC m=+4.012300252 container cleanup 92b9cad2c191477db5fbece171102b6ea0d7bef324555141fb330efa45681363 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-d, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-28T16:12:11.415 INFO:journalctl@ceph.mds.d.smithi067.stdout:Mar 28 16:12:11 smithi067 bash[92067]: ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-d 2024-03-28T16:12:11.819 INFO:journalctl@ceph.mds.d.smithi067.stdout:Mar 28 16:12:11 smithi067 podman[92081]: 2024-03-28 16:12:11.414464442 +0000 UTC m=+0.738284279 container remove 92b9cad2c191477db5fbece171102b6ea0d7bef324555141fb330efa45681363 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-d, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-28T16:12:11.958 DEBUG:teuthology.orchestra.run.smithi067:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@mds.d.service' 2024-03-28T16:12:12.089 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T16:12:12.089 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-03-28T16:12:12.089 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-03-28T16:12:12.089 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@mds.g 2024-03-28T16:12:12.569 INFO:journalctl@ceph.mds.g.smithi067.stdout:Mar 28 16:12:12 smithi067 systemd[1]: Stopping Ceph mds.g for b5f8225a-ed1a-11ee-95d0-87774f69a715... 2024-03-28T16:12:13.069 INFO:journalctl@ceph.mds.g.smithi067.stdout:Mar 28 16:12:12 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-g[73348]: 2024-03-28T16:12:12.671+0000 7f1cb4200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T16:12:13.069 INFO:journalctl@ceph.mds.g.smithi067.stdout:Mar 28 16:12:12 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-g[73348]: 2024-03-28T16:12:12.671+0000 7f1cb4200700 -1 mds.g *** got signal Terminated *** 2024-03-28T16:12:14.300 INFO:journalctl@ceph.mds.g.smithi067.stdout:Mar 28 16:12:14 smithi067 podman[92189]: 2024-03-28 16:12:14.010419429 +0000 UTC m=+1.465213532 container died a38fb47a48acf7de183e2f918df2ffb5714a08dbca0f7e3e6ff41eef037f214c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-g, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T16:12:15.069 INFO:journalctl@ceph.mds.g.smithi067.stdout:Mar 28 16:12:14 smithi067 podman[92189]: 2024-03-28 16:12:14.591831835 +0000 UTC m=+2.046625918 container cleanup a38fb47a48acf7de183e2f918df2ffb5714a08dbca0f7e3e6ff41eef037f214c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-g, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2) 2024-03-28T16:12:15.070 INFO:journalctl@ceph.mds.g.smithi067.stdout:Mar 28 16:12:14 smithi067 bash[92189]: ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-g 2024-03-28T16:12:15.070 INFO:journalctl@ceph.mds.g.smithi067.stdout:Mar 28 16:12:14 smithi067 podman[92214]: 2024-03-28 16:12:14.77630595 +0000 UTC m=+0.746193905 container remove a38fb47a48acf7de183e2f918df2ffb5714a08dbca0f7e3e6ff41eef037f214c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-g, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-28T16:12:15.316 DEBUG:teuthology.orchestra.run.smithi067:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@mds.g.service' 2024-03-28T16:12:15.439 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T16:12:15.439 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-03-28T16:12:15.439 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-03-28T16:12:15.439 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@mds.j 2024-03-28T16:12:15.819 INFO:journalctl@ceph.mds.j.smithi067.stdout:Mar 28 16:12:15 smithi067 systemd[1]: Stopping Ceph mds.j for b5f8225a-ed1a-11ee-95d0-87774f69a715... 2024-03-28T16:12:16.569 INFO:journalctl@ceph.mds.j.smithi067.stdout:Mar 28 16:12:16 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-j[72279]: 2024-03-28T16:12:16.109+0000 7fdf4ba00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T16:12:16.570 INFO:journalctl@ceph.mds.j.smithi067.stdout:Mar 28 16:12:16 smithi067 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-j[72279]: 2024-03-28T16:12:16.109+0000 7fdf4ba00700 -1 mds.j *** got signal Terminated *** 2024-03-28T16:12:19.027 INFO:journalctl@ceph.mds.j.smithi067.stdout:Mar 28 16:12:18 smithi067 podman[92319]: 2024-03-28 16:12:18.711742889 +0000 UTC m=+2.835228998 container died dd754e78743777780c556beef45db1f136a9c16c7d60e16424c5c73cfbf5799c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-j, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-28T16:12:19.319 INFO:journalctl@ceph.mds.j.smithi067.stdout:Mar 28 16:12:19 smithi067 podman[92319]: 2024-03-28 16:12:19.139133214 +0000 UTC m=+3.262619336 container cleanup dd754e78743777780c556beef45db1f136a9c16c7d60e16424c5c73cfbf5799c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-j, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-28T16:12:19.319 INFO:journalctl@ceph.mds.j.smithi067.stdout:Mar 28 16:12:19 smithi067 bash[92319]: ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-j 2024-03-28T16:12:19.819 INFO:journalctl@ceph.mds.j.smithi067.stdout:Mar 28 16:12:19 smithi067 podman[92333]: 2024-03-28 16:12:19.448269372 +0000 UTC m=+0.712645727 container remove dd754e78743777780c556beef45db1f136a9c16c7d60e16424c5c73cfbf5799c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-j, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux ) 2024-03-28T16:12:19.978 DEBUG:teuthology.orchestra.run.smithi067:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@mds.j.service' 2024-03-28T16:12:20.102 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T16:12:20.103 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-03-28T16:12:20.103 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-03-28T16:12:20.103 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@mds.b 2024-03-28T16:12:20.552 INFO:journalctl@ceph.mds.b.smithi150.stdout:Mar 28 16:12:20 smithi150 systemd[1]: Stopping Ceph mds.b for b5f8225a-ed1a-11ee-95d0-87774f69a715... 2024-03-28T16:12:21.052 INFO:journalctl@ceph.mds.b.smithi150.stdout:Mar 28 16:12:20 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-b[65534]: 2024-03-28T16:12:20.667+0000 7fdf3c000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T16:12:21.052 INFO:journalctl@ceph.mds.b.smithi150.stdout:Mar 28 16:12:20 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-b[65534]: 2024-03-28T16:12:20.667+0000 7fdf3c000700 -1 mds.b *** got signal Terminated *** 2024-03-28T16:12:23.302 INFO:journalctl@ceph.mds.b.smithi150.stdout:Mar 28 16:12:22 smithi150 podman[69623]: 2024-03-28 16:12:22.998089113 +0000 UTC m=+2.461413946 container died b6cc8cd55de5de20837d9aab3cb242ab0d26a09a30e15aa846e4ab95c4357468 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-b, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T16:12:23.662 INFO:journalctl@ceph.mds.b.smithi150.stdout:Mar 28 16:12:23 smithi150 podman[69623]: 2024-03-28 16:12:23.402401921 +0000 UTC m=+2.865726774 container cleanup b6cc8cd55de5de20837d9aab3cb242ab0d26a09a30e15aa846e4ab95c4357468 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-b, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, ceph=True) 2024-03-28T16:12:23.662 INFO:journalctl@ceph.mds.b.smithi150.stdout:Mar 28 16:12:23 smithi150 bash[69623]: ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-b 2024-03-28T16:12:23.923 INFO:journalctl@ceph.mds.b.smithi150.stdout:Mar 28 16:12:23 smithi150 podman[69636]: 2024-03-28 16:12:23.661699088 +0000 UTC m=+0.644946204 container remove b6cc8cd55de5de20837d9aab3cb242ab0d26a09a30e15aa846e4ab95c4357468 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-b, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-28T16:12:23.942 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@mds.b.service' 2024-03-28T16:12:24.061 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T16:12:24.062 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-03-28T16:12:24.062 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-03-28T16:12:24.062 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@mds.e 2024-03-28T16:12:24.302 INFO:journalctl@ceph.mds.e.smithi150.stdout:Mar 28 16:12:24 smithi150 systemd[1]: Stopping Ceph mds.e for b5f8225a-ed1a-11ee-95d0-87774f69a715... 2024-03-28T16:12:25.052 INFO:journalctl@ceph.mds.e.smithi150.stdout:Mar 28 16:12:24 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-e[64924]: 2024-03-28T16:12:24.628+0000 7fe83ac00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T16:12:25.053 INFO:journalctl@ceph.mds.e.smithi150.stdout:Mar 28 16:12:24 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-e[64924]: 2024-03-28T16:12:24.628+0000 7fe83ac00700 -1 mds.e *** got signal Terminated *** 2024-03-28T16:12:27.003 INFO:journalctl@ceph.mds.e.smithi150.stdout:Mar 28 16:12:26 smithi150 podman[69743]: 2024-03-28 16:12:26.724579143 +0000 UTC m=+2.238466453 container died d7898e35df8d35841269c2dae298d516b6034cedba45e277904e7d4ef5985454 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-e, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-28T16:12:27.303 INFO:journalctl@ceph.mds.e.smithi150.stdout:Mar 28 16:12:27 smithi150 podman[69743]: 2024-03-28 16:12:27.123205888 +0000 UTC m=+2.637093238 container cleanup d7898e35df8d35841269c2dae298d516b6034cedba45e277904e7d4ef5985454 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-e, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-28T16:12:27.303 INFO:journalctl@ceph.mds.e.smithi150.stdout:Mar 28 16:12:27 smithi150 bash[69743]: ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-e 2024-03-28T16:12:27.802 INFO:journalctl@ceph.mds.e.smithi150.stdout:Mar 28 16:12:27 smithi150 podman[69769]: 2024-03-28 16:12:27.41454068 +0000 UTC m=+0.683622094 container remove d7898e35df8d35841269c2dae298d516b6034cedba45e277904e7d4ef5985454 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-e, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS) 2024-03-28T16:12:27.914 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@mds.e.service' 2024-03-28T16:12:28.032 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T16:12:28.032 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-03-28T16:12:28.033 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-03-28T16:12:28.033 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@mds.h 2024-03-28T16:12:28.552 INFO:journalctl@ceph.mds.h.smithi150.stdout:Mar 28 16:12:28 smithi150 systemd[1]: Stopping Ceph mds.h for b5f8225a-ed1a-11ee-95d0-87774f69a715... 2024-03-28T16:12:29.052 INFO:journalctl@ceph.mds.h.smithi150.stdout:Mar 28 16:12:28 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-h[65230]: 2024-03-28T16:12:28.656+0000 7f675ca00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T16:12:29.052 INFO:journalctl@ceph.mds.h.smithi150.stdout:Mar 28 16:12:28 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-h[65230]: 2024-03-28T16:12:28.656+0000 7f675ca00700 -1 mds.h *** got signal Terminated *** 2024-03-28T16:12:31.263 INFO:journalctl@ceph.mds.h.smithi150.stdout:Mar 28 16:12:30 smithi150 podman[69874]: 2024-03-28 16:12:30.987362117 +0000 UTC m=+2.466940154 container died b4dfaf989e0853cb88f1b39854334810b4a6e95e1976b7495a50ca4a87f5c040 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-h, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-28T16:12:31.552 INFO:journalctl@ceph.mds.h.smithi150.stdout:Mar 28 16:12:31 smithi150 podman[69874]: 2024-03-28 16:12:31.383498609 +0000 UTC m=+2.863076636 container cleanup b4dfaf989e0853cb88f1b39854334810b4a6e95e1976b7495a50ca4a87f5c040 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-h, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-28T16:12:31.552 INFO:journalctl@ceph.mds.h.smithi150.stdout:Mar 28 16:12:31 smithi150 bash[69874]: ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-h 2024-03-28T16:12:31.913 INFO:journalctl@ceph.mds.h.smithi150.stdout:Mar 28 16:12:31 smithi150 podman[69889]: 2024-03-28 16:12:31.651256466 +0000 UTC m=+0.655951127 container remove b4dfaf989e0853cb88f1b39854334810b4a6e95e1976b7495a50ca4a87f5c040 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, ceph=True) 2024-03-28T16:12:31.930 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@mds.h.service' 2024-03-28T16:12:32.049 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T16:12:32.050 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-03-28T16:12:32.050 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-03-28T16:12:32.050 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@mds.k 2024-03-28T16:12:32.552 INFO:journalctl@ceph.mds.k.smithi150.stdout:Mar 28 16:12:32 smithi150 systemd[1]: Stopping Ceph mds.k for b5f8225a-ed1a-11ee-95d0-87774f69a715... 2024-03-28T16:12:33.052 INFO:journalctl@ceph.mds.k.smithi150.stdout:Mar 28 16:12:32 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-k[64609]: 2024-03-28T16:12:32.660+0000 7fd145e00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T16:12:33.052 INFO:journalctl@ceph.mds.k.smithi150.stdout:Mar 28 16:12:32 smithi150 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-k[64609]: 2024-03-28T16:12:32.660+0000 7fd145e00700 -1 mds.k *** got signal Terminated *** 2024-03-28T16:12:35.052 INFO:journalctl@ceph.mds.k.smithi150.stdout:Mar 28 16:12:34 smithi150 podman[69995]: 2024-03-28 16:12:34.795758657 +0000 UTC m=+2.275440137 container died 8b52dd296b4cdc7ec44a3b6050ca01fec8636133bc39934a4e0c9523dff8ffac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-k, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T16:12:35.380 INFO:journalctl@ceph.mds.k.smithi150.stdout:Mar 28 16:12:35 smithi150 podman[69995]: 2024-03-28 16:12:35.170343812 +0000 UTC m=+2.650025277 container cleanup 8b52dd296b4cdc7ec44a3b6050ca01fec8636133bc39934a4e0c9523dff8ffac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-k, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T16:12:35.380 INFO:journalctl@ceph.mds.k.smithi150.stdout:Mar 28 16:12:35 smithi150 bash[69995]: ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-k 2024-03-28T16:12:35.802 INFO:journalctl@ceph.mds.k.smithi150.stdout:Mar 28 16:12:35 smithi150 podman[70009]: 2024-03-28 16:12:35.37985471 +0000 UTC m=+0.570913276 container remove 8b52dd296b4cdc7ec44a3b6050ca01fec8636133bc39934a4e0c9523dff8ffac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-k, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T16:12:35.829 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@mds.k.service' 2024-03-28T16:12:35.936 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T16:12:35.936 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-03-28T16:12:35.937 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-03-28T16:12:35.937 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@mds.c 2024-03-28T16:12:36.350 INFO:journalctl@ceph.mds.c.smithi183.stdout:Mar 28 16:12:36 smithi183 systemd[1]: Stopping Ceph mds.c for b5f8225a-ed1a-11ee-95d0-87774f69a715... 2024-03-28T16:12:36.716 INFO:journalctl@ceph.mds.c.smithi183.stdout:Mar 28 16:12:36 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-c[66213]: 2024-03-28T16:12:36.493+0000 7f0fd1a00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T16:12:36.716 INFO:journalctl@ceph.mds.c.smithi183.stdout:Mar 28 16:12:36 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-c[66213]: 2024-03-28T16:12:36.493+0000 7f0fd1a00700 -1 mds.c *** got signal Terminated *** 2024-03-28T16:12:38.534 INFO:journalctl@ceph.mds.c.smithi183.stdout:Mar 28 16:12:38 smithi183 podman[70462]: 2024-03-28 16:12:38.219057819 +0000 UTC m=+1.857103559 container died 2b20d7441f6a459f45a53251e72fb1a1158c8e056016982dddf223f28a09162c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-c, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-28T16:12:38.850 INFO:journalctl@ceph.mds.c.smithi183.stdout:Mar 28 16:12:38 smithi183 podman[70462]: 2024-03-28 16:12:38.64597961 +0000 UTC m=+2.284025345 container cleanup 2b20d7441f6a459f45a53251e72fb1a1158c8e056016982dddf223f28a09162c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-c, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T16:12:38.850 INFO:journalctl@ceph.mds.c.smithi183.stdout:Mar 28 16:12:38 smithi183 bash[70462]: ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-c 2024-03-28T16:12:39.350 INFO:journalctl@ceph.mds.c.smithi183.stdout:Mar 28 16:12:38 smithi183 podman[70488]: 2024-03-28 16:12:38.897051028 +0000 UTC m=+0.658713461 container remove 2b20d7441f6a459f45a53251e72fb1a1158c8e056016982dddf223f28a09162c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-c, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-28T16:12:39.410 DEBUG:teuthology.orchestra.run.smithi183:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@mds.c.service' 2024-03-28T16:12:39.526 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T16:12:39.527 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-03-28T16:12:39.527 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-03-28T16:12:39.527 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@mds.f 2024-03-28T16:12:40.100 INFO:journalctl@ceph.mds.f.smithi183.stdout:Mar 28 16:12:39 smithi183 systemd[1]: Stopping Ceph mds.f for b5f8225a-ed1a-11ee-95d0-87774f69a715... 2024-03-28T16:12:40.600 INFO:journalctl@ceph.mds.f.smithi183.stdout:Mar 28 16:12:40 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-f[65896]: 2024-03-28T16:12:40.105+0000 7fc562000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T16:12:40.600 INFO:journalctl@ceph.mds.f.smithi183.stdout:Mar 28 16:12:40 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-f[65896]: 2024-03-28T16:12:40.106+0000 7fc562000700 -1 mds.f *** got signal Terminated *** 2024-03-28T16:12:42.771 INFO:journalctl@ceph.mds.f.smithi183.stdout:Mar 28 16:12:42 smithi183 podman[70593]: 2024-03-28 16:12:42.456370683 +0000 UTC m=+2.473657155 container died fc616700c00247e3cb65f7b0b5c6bb9b9bae245cf7eeae84bd0accf3158a14b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-f, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-28T16:12:43.100 INFO:journalctl@ceph.mds.f.smithi183.stdout:Mar 28 16:12:42 smithi183 podman[70593]: 2024-03-28 16:12:42.900185735 +0000 UTC m=+2.917472162 container cleanup fc616700c00247e3cb65f7b0b5c6bb9b9bae245cf7eeae84bd0accf3158a14b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-f, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T16:12:43.100 INFO:journalctl@ceph.mds.f.smithi183.stdout:Mar 28 16:12:42 smithi183 bash[70593]: ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-f 2024-03-28T16:12:43.600 INFO:journalctl@ceph.mds.f.smithi183.stdout:Mar 28 16:12:43 smithi183 podman[70607]: 2024-03-28 16:12:43.149856642 +0000 UTC m=+0.674882421 container remove fc616700c00247e3cb65f7b0b5c6bb9b9bae245cf7eeae84bd0accf3158a14b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-f, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T16:12:43.729 DEBUG:teuthology.orchestra.run.smithi183:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@mds.f.service' 2024-03-28T16:12:43.843 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T16:12:43.843 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-03-28T16:12:43.843 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-03-28T16:12:43.843 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@mds.i 2024-03-28T16:12:44.100 INFO:journalctl@ceph.mds.i.smithi183.stdout:Mar 28 16:12:43 smithi183 systemd[1]: Stopping Ceph mds.i for b5f8225a-ed1a-11ee-95d0-87774f69a715... 2024-03-28T16:12:44.850 INFO:journalctl@ceph.mds.i.smithi183.stdout:Mar 28 16:12:44 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-i[66519]: 2024-03-28T16:12:44.382+0000 7f3c73e00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T16:12:44.850 INFO:journalctl@ceph.mds.i.smithi183.stdout:Mar 28 16:12:44 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-i[66519]: 2024-03-28T16:12:44.382+0000 7f3c73e00700 -1 mds.i *** got signal Terminated *** 2024-03-28T16:12:46.265 INFO:journalctl@ceph.mds.i.smithi183.stdout:Mar 28 16:12:45 smithi183 podman[70712]: 2024-03-28 16:12:45.962917524 +0000 UTC m=+1.695640605 container died 999fdaab14075feb5960df431524dd27359a5e8a477a34b884dd583e3698ce82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-i, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-28T16:12:46.600 INFO:journalctl@ceph.mds.i.smithi183.stdout:Mar 28 16:12:46 smithi183 podman[70712]: 2024-03-28 16:12:46.368887903 +0000 UTC m=+2.101611021 container cleanup 999fdaab14075feb5960df431524dd27359a5e8a477a34b884dd583e3698ce82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-i, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , ceph=True) 2024-03-28T16:12:46.600 INFO:journalctl@ceph.mds.i.smithi183.stdout:Mar 28 16:12:46 smithi183 bash[70712]: ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-i 2024-03-28T16:12:46.904 INFO:journalctl@ceph.mds.i.smithi183.stdout:Mar 28 16:12:46 smithi183 podman[70726]: 2024-03-28 16:12:46.644683992 +0000 UTC m=+0.663148569 container remove 999fdaab14075feb5960df431524dd27359a5e8a477a34b884dd583e3698ce82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-i, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.build-date=20240319) 2024-03-28T16:12:46.942 DEBUG:teuthology.orchestra.run.smithi183:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@mds.i.service' 2024-03-28T16:12:47.056 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T16:12:47.056 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-03-28T16:12:47.056 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-03-28T16:12:47.057 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@mds.l 2024-03-28T16:12:47.350 INFO:journalctl@ceph.mds.l.smithi183.stdout:Mar 28 16:12:47 smithi183 systemd[1]: Stopping Ceph mds.l for b5f8225a-ed1a-11ee-95d0-87774f69a715... 2024-03-28T16:12:47.993 INFO:journalctl@ceph.mds.l.smithi183.stdout:Mar 28 16:12:47 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-l[65574]: 2024-03-28T16:12:47.620+0000 7ffb10000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T16:12:47.993 INFO:journalctl@ceph.mds.l.smithi183.stdout:Mar 28 16:12:47 smithi183 ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-l[65574]: 2024-03-28T16:12:47.620+0000 7ffb10000700 -1 mds.l *** got signal Terminated *** 2024-03-28T16:12:51.293 INFO:journalctl@ceph.mds.l.smithi183.stdout:Mar 28 16:12:50 smithi183 podman[70834]: 2024-03-28 16:12:50.960682495 +0000 UTC m=+3.495758028 container died 4489b0bde9085bba82870b23a47a0018f73b0e52b526dc8d98a72242ea6be629 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-l, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T16:12:51.600 INFO:journalctl@ceph.mds.l.smithi183.stdout:Mar 28 16:12:51 smithi183 podman[70834]: 2024-03-28 16:12:51.404803442 +0000 UTC m=+3.939879040 container cleanup 4489b0bde9085bba82870b23a47a0018f73b0e52b526dc8d98a72242ea6be629 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-l, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True) 2024-03-28T16:12:51.600 INFO:journalctl@ceph.mds.l.smithi183.stdout:Mar 28 16:12:51 smithi183 bash[70834]: ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-l 2024-03-28T16:12:52.100 INFO:journalctl@ceph.mds.l.smithi183.stdout:Mar 28 16:12:51 smithi183 podman[70848]: 2024-03-28 16:12:51.697245823 +0000 UTC m=+0.721737439 container remove 4489b0bde9085bba82870b23a47a0018f73b0e52b526dc8d98a72242ea6be629 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715-mds-l, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T16:12:52.154 DEBUG:teuthology.orchestra.run.smithi183:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b5f8225a-ed1a-11ee-95d0-87774f69a715@mds.l.service' 2024-03-28T16:12:52.263 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T16:12:52.264 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-03-28T16:12:52.264 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 --force --keep-logs 2024-03-28T16:12:52.604 INFO:teuthology.orchestra.run.smithi067.stdout:Deleting cluster with fsid: b5f8225a-ed1a-11ee-95d0-87774f69a715 2024-03-28T16:12:57.548 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 --force --keep-logs 2024-03-28T16:12:57.869 INFO:teuthology.orchestra.run.smithi150.stdout:Deleting cluster with fsid: b5f8225a-ed1a-11ee-95d0-87774f69a715 2024-03-28T16:13:02.734 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 --force --keep-logs 2024-03-28T16:13:03.057 INFO:teuthology.orchestra.run.smithi183.stdout:Deleting cluster with fsid: b5f8225a-ed1a-11ee-95d0-87774f69a715 2024-03-28T16:13:07.999 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-28T16:13:08.106 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-28T16:13:08.208 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-28T16:13:08.293 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-28T16:13:08.295 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625322/remote/smithi067/crash 2024-03-28T16:13:08.296 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/crash -- . 2024-03-28T16:13:08.413 INFO:teuthology.orchestra.run.smithi067.stderr:tar: /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/crash: Cannot open: No such file or directory 2024-03-28T16:13:08.413 INFO:teuthology.orchestra.run.smithi067.stderr:tar: Error is not recoverable: exiting now 2024-03-28T16:13:08.420 DEBUG:teuthology.misc:Transferring archived files from smithi150:/var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625322/remote/smithi150/crash 2024-03-28T16:13:08.420 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar c -f - -C /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/crash -- . 2024-03-28T16:13:08.567 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/crash: Cannot open: No such file or directory 2024-03-28T16:13:08.567 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2024-03-28T16:13:08.573 DEBUG:teuthology.misc:Transferring archived files from smithi183:/var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625322/remote/smithi183/crash 2024-03-28T16:13:08.574 DEBUG:teuthology.orchestra.run.smithi183:> sudo tar c -f - -C /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/crash -- . 2024-03-28T16:13:08.661 INFO:teuthology.orchestra.run.smithi183.stderr:tar: /var/lib/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/crash: Cannot open: No such file or directory 2024-03-28T16:13:08.662 INFO:teuthology.orchestra.run.smithi183.stderr:tar: Error is not recoverable: exiting now 2024-03-28T16:13:08.665 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-28T16:13:08.666 DEBUG:teuthology.orchestra.run.smithi067:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-28T16:13:08.782 INFO:teuthology.orchestra.run.smithi067.stdout:2024-03-28T15:58:21.700776+0000 mon.a (mon.0) 683 : cluster [WRN] Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-03-28T16:13:08.783 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-28T16:13:08.783 DEBUG:teuthology.orchestra.run.smithi067:> sudo egrep '\[SEC\]' /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-28T16:13:08.871 DEBUG:teuthology.orchestra.run.smithi067:> sudo egrep '\[ERR\]' /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-28T16:13:08.960 DEBUG:teuthology.orchestra.run.smithi067:> sudo egrep '\[WRN\]' /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-28T16:13:09.053 INFO:teuthology.orchestra.run.smithi067.stdout:2024-03-28T15:58:21.700776+0000 mon.a (mon.0) 683 : cluster [WRN] Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-03-28T16:13:09.054 INFO:tasks.cephadm:Compressing logs... 2024-03-28T16:13:09.054 DEBUG:teuthology.orchestra.run.smithi067:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-28T16:13:09.059 DEBUG:teuthology.orchestra.run.smithi150:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-28T16:13:09.068 DEBUG:teuthology.orchestra.run.smithi183:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-28T16:13:09.128 INFO:teuthology.orchestra.run.smithi067.stderr:find: ‘/var/log/rbd-target-api’gzip: No such file or directory -5 2024-03-28T16:13:09.128 INFO:teuthology.orchestra.run.smithi067.stderr: --verbose -- /var/log/ceph/cephadm.log 2024-03-28T16:13:09.128 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.log 2024-03-28T16:13:09.128 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-osd.6.log 2024-03-28T16:13:09.129 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.0.log: gzip -5 --verbose -- /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-mon.a.log 2024-03-28T16:13:09.129 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph.audit.log 2024-03-28T16:13:09.130 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-mds.g.log 2024-03-28T16:13:09.131 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-mds.j.log 2024-03-28T16:13:09.133 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-osd.3.log 2024-03-28T16:13:09.137 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-mds.j.log: /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-osd.3.log: 90.1% -- replaced with /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph.audit.log.gz 2024-03-28T16:13:09.137 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-mgr.x.log 2024-03-28T16:13:09.137 INFO:teuthology.orchestra.run.smithi067.stderr: 93.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-28T16:13:09.138 INFO:teuthology.orchestra.run.smithi067.stderr: 94.6% -- replaced with /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-mds.g.log.gz 2024-03-28T16:13:09.138 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-osd.0.log 2024-03-28T16:13:09.139 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-mgr.x.log: 94.6% -- replaced with /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-mds.j.log.gz 2024-03-28T16:13:09.139 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph.cephadm.log 2024-03-28T16:13:09.142 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph.log 2024-03-28T16:13:09.143 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph.cephadm.log: 82.6% -- replaced with /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph.cephadm.log.gz 2024-03-28T16:13:09.145 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-volume.log 2024-03-28T16:13:09.146 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-osd.9.log 2024-03-28T16:13:09.149 INFO:teuthology.orchestra.run.smithi150.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-28T16:13:09.149 INFO:teuthology.orchestra.run.smithi150.stderr:: No such file or directory 2024-03-28T16:13:09.150 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-mds.d.log 2024-03-28T16:13:09.150 INFO:teuthology.orchestra.run.smithi067.stderr: 87.0% -- replaced with /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph.log.gz 2024-03-28T16:13:09.150 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-mds.a.log 2024-03-28T16:13:09.151 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-osd.10.log 2024-03-28T16:13:09.152 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph.audit.log 2024-03-28T16:13:09.152 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-osd.10.log: 91.7% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-28T16:13:09.152 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-mds.b.log 2024-03-28T16:13:09.154 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-osd.4.log 2024-03-28T16:13:09.155 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph.audit.log: /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-osd.1.log 2024-03-28T16:13:09.156 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-mds.k.log 2024-03-28T16:13:09.158 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-mds.h.log 2024-03-28T16:13:09.158 INFO:teuthology.orchestra.run.smithi150.stderr: 90.3% -- replaced with /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph.audit.log.gz 2024-03-28T16:13:09.160 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-osd.9.log: /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-mds.a.log: /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-mds.d.log: 94.6% -- replaced with /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-mds.d.log.gz 2024-03-28T16:13:09.162 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-mds.k.log: /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-osd.7.log 2024-03-28T16:13:09.166 INFO:teuthology.orchestra.run.smithi183.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-28T16:13:09.167 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-28T16:13:09.167 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-mds.l.log 2024-03-28T16:13:09.167 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph.audit.log 2024-03-28T16:13:09.174 INFO:teuthology.orchestra.run.smithi067.stderr: 94.6% 90.8% -- replaced with /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-volume.log.gz 2024-03-28T16:13:09.174 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-mds.a.log.gz 2024-03-28T16:13:09.174 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-mds.e.log 2024-03-28T16:13:09.175 INFO:teuthology.orchestra.run.smithi150.stderr: 94.5% -- replaced with /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-mds.k.log.gz 2024-03-28T16:13:09.175 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-mgr.y.log 2024-03-28T16:13:09.175 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-mgr.y.log: /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph.cephadm.log 2024-03-28T16:13:09.181 INFO:teuthology.orchestra.run.smithi150.stderr: 94.5% -- replaced with /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-mds.e.log.gz 2024-03-28T16:13:09.181 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph.log 2024-03-28T16:13:09.182 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph.cephadm.log: 82.2% -- replaced with /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph.cephadm.log.gz 2024-03-28T16:13:09.182 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-volume.log 2024-03-28T16:13:09.182 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-mon.b.log 2024-03-28T16:13:09.184 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-mds.l.log: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-28T16:13:09.184 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-mds.i.log 2024-03-28T16:13:09.184 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-osd.11.log 2024-03-28T16:13:09.184 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph.audit.log: /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-mgr.z.log 2024-03-28T16:13:09.185 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-osd.11.log: 94.6% -- replaced with /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-mds.l.log.gz 2024-03-28T16:13:09.185 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-osd.8.log 2024-03-28T16:13:09.185 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-mgr.z.log: gzip -5 94.6% --verbose -- replaced with /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-mds.i.log.gz -- 2024-03-28T16:13:09.185 INFO:teuthology.orchestra.run.smithi183.stderr: /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-osd.5.log 2024-03-28T16:13:09.185 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-osd.8.log: 90.3% -- replaced with /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph.audit.log.gz 2024-03-28T16:13:09.185 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-mds.f.log 2024-03-28T16:13:09.185 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph.cephadm.log 2024-03-28T16:13:09.186 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph.log 2024-03-28T16:13:09.186 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-osd.2.log 2024-03-28T16:13:09.186 INFO:teuthology.orchestra.run.smithi183.stderr: 82.1% -- replaced with /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph.cephadm.log.gz 2024-03-28T16:13:09.186 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph.log: 92.6% -- replaced with /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-mgr.z.log.gz 2024-03-28T16:13:09.186 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-mds.c.log 2024-03-28T16:13:09.186 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-volume.log 2024-03-28T16:13:09.186 INFO:teuthology.orchestra.run.smithi183.stderr: 94.7% -- replaced with /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-mds.f.log.gz 2024-03-28T16:13:09.186 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-mds.c.log: 87.0% -- replaced with /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph.log.gz 2024-03-28T16:13:09.187 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-mon.c.log 2024-03-28T16:13:09.188 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-volume.log: /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-mon.c.log: 94.5% -- replaced with /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-mds.c.log.gz 2024-03-28T16:13:09.188 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-volume.log: /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-mon.b.log: 92.6% -- replaced with /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-mgr.y.log.gz 2024-03-28T16:13:09.188 INFO:teuthology.orchestra.run.smithi150.stderr: 87.0% -- replaced with /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph.log.gz 2024-03-28T16:13:09.195 INFO:teuthology.orchestra.run.smithi183.stderr: 90.8% -- replaced with /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-volume.log.gz 2024-03-28T16:13:09.218 INFO:teuthology.orchestra.run.smithi150.stderr: 90.8% -- replaced with /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-volume.log.gz 2024-03-28T16:13:09.247 INFO:teuthology.orchestra.run.smithi150.stderr: 93.3% -- replaced with /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-mds.h.log.gz 2024-03-28T16:13:09.412 INFO:teuthology.orchestra.run.smithi067.stderr: 89.7% -- replaced with /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-mgr.x.log.gz 2024-03-28T16:13:09.426 INFO:teuthology.orchestra.run.smithi150.stderr: 92.6% -- replaced with /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-mds.b.log.gz 2024-03-28T16:13:09.519 INFO:teuthology.orchestra.run.smithi183.stderr: 92.0% -- replaced with /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-mon.c.log.gz 2024-03-28T16:13:09.534 INFO:teuthology.orchestra.run.smithi150.stderr: 91.8% -- replaced with /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-mon.b.log.gz 2024-03-28T16:13:09.749 INFO:teuthology.orchestra.run.smithi183.stderr: 92.7% -- replaced with /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-osd.8.log.gz 2024-03-28T16:13:09.833 INFO:teuthology.orchestra.run.smithi150.stderr: 92.9% -- replaced with /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-osd.4.log.gz 2024-03-28T16:13:09.837 INFO:teuthology.orchestra.run.smithi183.stderr: 92.8% -- replaced with /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-osd.2.log.gz 2024-03-28T16:13:09.906 INFO:teuthology.orchestra.run.smithi183.stderr: 92.7% -- replaced with /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-osd.5.log.gz 2024-03-28T16:13:09.914 INFO:teuthology.orchestra.run.smithi183.stderr: 93.1% -- replaced with /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-osd.11.log.gz 2024-03-28T16:13:09.919 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-28T16:13:09.919 INFO:teuthology.orchestra.run.smithi183.stderr:real 0m0.794s 2024-03-28T16:13:09.919 INFO:teuthology.orchestra.run.smithi183.stderr:user 0m2.879s 2024-03-28T16:13:09.919 INFO:teuthology.orchestra.run.smithi183.stderr:sys 0m0.336s 2024-03-28T16:13:09.924 INFO:teuthology.orchestra.run.smithi150.stderr: 93.0% -- replaced with /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-osd.7.log.gz 2024-03-28T16:13:09.929 INFO:teuthology.orchestra.run.smithi067.stderr: 92.6% -- replaced with /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-osd.9.log.gz 2024-03-28T16:13:09.971 INFO:teuthology.orchestra.run.smithi067.stderr: 93.0% -- replaced with /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-osd.0.log.gz 2024-03-28T16:13:09.984 INFO:teuthology.orchestra.run.smithi150.stderr: 92.8% -- replaced with /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-osd.10.log.gz 2024-03-28T16:13:10.097 INFO:teuthology.orchestra.run.smithi067.stderr: 92.9% -- replaced with /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-osd.6.log.gz 2024-03-28T16:13:10.100 INFO:teuthology.orchestra.run.smithi150.stderr: 93.0% -- replaced with /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-osd.1.log.gz 2024-03-28T16:13:10.105 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-03-28T16:13:10.105 INFO:teuthology.orchestra.run.smithi150.stderr:real 0m0.992s 2024-03-28T16:13:10.105 INFO:teuthology.orchestra.run.smithi150.stderr:user 0m3.657s 2024-03-28T16:13:10.105 INFO:teuthology.orchestra.run.smithi150.stderr:sys 0m0.391s 2024-03-28T16:13:10.158 INFO:teuthology.orchestra.run.smithi067.stderr: 92.9% -- replaced with /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-osd.3.log.gz 2024-03-28T16:13:10.169 INFO:teuthology.orchestra.run.smithi067.stderr: 90.5% -- replaced with /var/log/ceph/b5f8225a-ed1a-11ee-95d0-87774f69a715/ceph-mon.a.log.gz 2024-03-28T16:13:17.950 INFO:teuthology.orchestra.run.smithi067.stderr: 96.3% -- replaced with /var/log/ceph/ceph-client.0.log.gz 2024-03-28T16:13:17.954 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-28T16:13:17.954 INFO:teuthology.orchestra.run.smithi067.stderr:real 0m8.862s 2024-03-28T16:13:17.954 INFO:teuthology.orchestra.run.smithi067.stderr:user 0m12.735s 2024-03-28T16:13:17.954 INFO:teuthology.orchestra.run.smithi067.stderr:sys 0m1.153s 2024-03-28T16:13:17.955 INFO:tasks.cephadm:Archiving logs... 2024-03-28T16:13:17.955 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/log/ceph to /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625322/remote/smithi067/log 2024-03-28T16:13:17.956 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-28T16:13:19.569 DEBUG:teuthology.misc:Transferring archived files from smithi150:/var/log/ceph to /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625322/remote/smithi150/log 2024-03-28T16:13:19.574 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-28T16:13:20.190 DEBUG:teuthology.misc:Transferring archived files from smithi183:/var/log/ceph to /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625322/remote/smithi183/log 2024-03-28T16:13:20.191 DEBUG:teuthology.orchestra.run.smithi183:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-28T16:13:20.759 INFO:tasks.cephadm:Removing cluster... 2024-03-28T16:13:20.759 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 --force 2024-03-28T16:13:21.103 INFO:teuthology.orchestra.run.smithi067.stdout:Deleting cluster with fsid: b5f8225a-ed1a-11ee-95d0-87774f69a715 2024-03-28T16:13:21.743 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 --force 2024-03-28T16:13:22.071 INFO:teuthology.orchestra.run.smithi150.stdout:Deleting cluster with fsid: b5f8225a-ed1a-11ee-95d0-87774f69a715 2024-03-28T16:13:22.649 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid b5f8225a-ed1a-11ee-95d0-87774f69a715 --force 2024-03-28T16:13:22.977 INFO:teuthology.orchestra.run.smithi183.stdout:Deleting cluster with fsid: b5f8225a-ed1a-11ee-95d0-87774f69a715 2024-03-28T16:13:23.562 INFO:tasks.cephadm:Removing cephadm ... 2024-03-28T16:13:23.562 DEBUG:teuthology.orchestra.run.smithi067:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-28T16:13:23.624 DEBUG:teuthology.orchestra.run.smithi150:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-28T16:13:23.679 DEBUG:teuthology.orchestra.run.smithi183:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-28T16:13:23.734 INFO:tasks.cephadm:Teardown complete 2024-03-28T16:13:23.735 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-28T16:13:23.746 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-28T16:13:23.746 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-28T16:13:23.754 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-28T16:13:23.764 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-28T16:13:23.930 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-28T16:13:23.941 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-28T16:13:23.941 DEBUG:teuthology.orchestra.run.smithi067:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T16:13:23.954 DEBUG:teuthology.orchestra.run.smithi150:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T16:13:23.962 DEBUG:teuthology.orchestra.run.smithi183:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T16:13:24.000 INFO:teuthology.orchestra.run.smithi067.stderr:bash: line 1: ntpq: command not found 2024-03-28T16:13:24.005 INFO:teuthology.orchestra.run.smithi150.stderr:bash: line 1: ntpq: command not found 2024-03-28T16:13:24.009 INFO:teuthology.orchestra.run.smithi067.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T16:13:24.009 INFO:teuthology.orchestra.run.smithi067.stdout:=============================================================================== 2024-03-28T16:13:24.010 INFO:teuthology.orchestra.run.smithi067.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 112 +326us[ +319us] +/- 77ms 2024-03-28T16:13:24.010 INFO:teuthology.orchestra.run.smithi067.stdout:^* hv02.front.sepia.ceph.com 3 6 377 43 -87us[ -89us] +/- 61ms 2024-03-28T16:13:24.010 INFO:teuthology.orchestra.run.smithi067.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 43 -247us[ -247us] +/- 70ms 2024-03-28T16:13:24.010 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T16:13:24.015 INFO:teuthology.orchestra.run.smithi183.stderr:bash: line 1: ntpq: command not found 2024-03-28T16:13:24.015 INFO:teuthology.orchestra.run.smithi150.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T16:13:24.015 INFO:teuthology.orchestra.run.smithi150.stdout:=============================================================================== 2024-03-28T16:13:24.015 INFO:teuthology.orchestra.run.smithi150.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 43 +297us[ +297us] +/- 78ms 2024-03-28T16:13:24.015 INFO:teuthology.orchestra.run.smithi150.stdout:^* hv02.front.sepia.ceph.com 3 6 377 44 -33us[ -26us] +/- 61ms 2024-03-28T16:13:24.015 INFO:teuthology.orchestra.run.smithi150.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 45 -223us[ -216us] +/- 70ms 2024-03-28T16:13:24.015 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T16:13:24.023 INFO:teuthology.orchestra.run.smithi183.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T16:13:24.024 INFO:teuthology.orchestra.run.smithi183.stdout:=============================================================================== 2024-03-28T16:13:24.024 INFO:teuthology.orchestra.run.smithi183.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 45 +289us[ +296us] +/- 78ms 2024-03-28T16:13:24.024 INFO:teuthology.orchestra.run.smithi183.stdout:^* hv02.front.sepia.ceph.com 3 6 377 44 -34us[ -27us] +/- 61ms 2024-03-28T16:13:24.024 INFO:teuthology.orchestra.run.smithi183.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 46 -208us[ -201us] +/- 70ms 2024-03-28T16:13:24.024 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T16:13:24.025 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-28T16:13:24.037 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-28T16:13:24.037 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-28T16:13:24.048 DEBUG:teuthology.orchestra.run.smithi067:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-28T16:13:24.248 DEBUG:teuthology.orchestra.run.smithi150:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-28T16:13:24.463 DEBUG:teuthology.orchestra.run.smithi183:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-28T16:13:24.683 DEBUG:teuthology.orchestra.run.smithi067:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T16:13:24.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T16:13:24.777 DEBUG:teuthology.orchestra.run.smithi150:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T16:13:24.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T16:13:24.878 DEBUG:teuthology.orchestra.run.smithi183:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T16:13:24.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T16:13:24.982 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-28T16:13:24.993 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-28T16:13:25.003 INFO:teuthology.task.internal:Duration was 2007.346562 seconds 2024-03-28T16:13:25.004 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-28T16:13:25.023 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-28T16:13:25.024 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-28T16:13:25.032 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-28T16:13:25.039 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-28T16:13:25.178 INFO:teuthology.orchestra.run.smithi067.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T16:13:25.183 INFO:teuthology.orchestra.run.smithi150.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T16:13:25.194 INFO:teuthology.orchestra.run.smithi183.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T16:13:25.651 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-28T16:13:25.651 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi067.front.sepia.ceph.com 2024-03-28T16:13:25.652 DEBUG:teuthology.orchestra.run.smithi067:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-28T16:13:25.713 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi150.front.sepia.ceph.com 2024-03-28T16:13:25.713 DEBUG:teuthology.orchestra.run.smithi150:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-28T16:13:25.785 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi183.front.sepia.ceph.com 2024-03-28T16:13:25.785 DEBUG:teuthology.orchestra.run.smithi183:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-28T16:13:25.860 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-28T16:13:25.860 DEBUG:teuthology.orchestra.run.smithi067:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-28T16:13:25.868 DEBUG:teuthology.orchestra.run.smithi150:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-28T16:13:25.875 DEBUG:teuthology.orchestra.run.smithi183:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-28T16:13:25.969 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-28T16:13:25.970 DEBUG:teuthology.orchestra.run.smithi067:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-28T16:13:25.975 DEBUG:teuthology.orchestra.run.smithi150:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-28T16:13:25.981 DEBUG:teuthology.orchestra.run.smithi183:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-28T16:13:26.459 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-28T16:13:26.471 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-28T16:13:26.471 DEBUG:teuthology.orchestra.run.smithi067:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-28T16:13:26.570 DEBUG:teuthology.orchestra.run.smithi150:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-28T16:13:26.671 DEBUG:teuthology.orchestra.run.smithi183:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-28T16:13:26.770 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-28T16:13:26.780 DEBUG:teuthology.orchestra.run.smithi067:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-28T16:13:26.789 DEBUG:teuthology.orchestra.run.smithi150:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-28T16:13:26.799 DEBUG:teuthology.orchestra.run.smithi183:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-28T16:13:26.898 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern = core 2024-03-28T16:13:26.899 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern = core 2024-03-28T16:13:26.906 INFO:teuthology.orchestra.run.smithi183.stdout:kernel.core_pattern = core 2024-03-28T16:13:26.962 DEBUG:teuthology.orchestra.run.smithi067:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-28T16:13:27.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T16:13:27.015 DEBUG:teuthology.orchestra.run.smithi150:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-28T16:13:27.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T16:13:27.073 DEBUG:teuthology.orchestra.run.smithi183:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-28T16:13:27.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T16:13:27.128 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-28T16:13:27.141 INFO:teuthology.task.internal:Transferring archived files... 2024-03-28T16:13:27.142 DEBUG:teuthology.misc:Transferring archived files from smithi067:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625322/remote/smithi067 2024-03-28T16:13:27.142 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-28T16:13:27.270 DEBUG:teuthology.misc:Transferring archived files from smithi150:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625322/remote/smithi150 2024-03-28T16:13:27.270 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-28T16:13:27.381 DEBUG:teuthology.misc:Transferring archived files from smithi183:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625322/remote/smithi183 2024-03-28T16:13:27.382 DEBUG:teuthology.orchestra.run.smithi183:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-28T16:13:27.492 INFO:teuthology.task.internal:Removing archive directory... 2024-03-28T16:13:27.492 DEBUG:teuthology.orchestra.run.smithi067:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-28T16:13:27.500 DEBUG:teuthology.orchestra.run.smithi150:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-28T16:13:27.508 DEBUG:teuthology.orchestra.run.smithi183:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-28T16:13:27.565 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-28T16:13:27.576 INFO:teuthology.task.internal:Not uploading archives. 2024-03-28T16:13:27.576 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-28T16:13:27.588 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-28T16:13:27.588 DEBUG:teuthology.orchestra.run.smithi067:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-28T16:13:27.596 DEBUG:teuthology.orchestra.run.smithi150:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-28T16:13:27.604 DEBUG:teuthology.orchestra.run.smithi183:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-28T16:13:27.647 INFO:teuthology.orchestra.run.smithi067.stdout: 393229 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 28 16:13 /home/ubuntu/cephtest 2024-03-28T16:13:27.652 INFO:teuthology.orchestra.run.smithi150.stdout: 393229 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 28 16:13 /home/ubuntu/cephtest 2024-03-28T16:13:27.659 INFO:teuthology.orchestra.run.smithi183.stdout: 393229 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 28 16:13 /home/ubuntu/cephtest 2024-03-28T16:13:27.663 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-28T16:13:27.679 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-28T16:13:27.711 INFO:teuthology.nuke:Checking targets against current locks 2024-03-28T16:13:27.735 DEBUG:teuthology.nuke:shortname: smithi067 2024-03-28T16:13:27.735 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-28T16:13:27.756 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi067.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625322', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 15:30:11.382265', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJhDUpZ2IauSqSlufwftGvEaQ5Vkiy4L5JFlRhYpBj2aTCwofUuXYlSN6ib1OrRxcYTlXmvuJV4XyHSzUqmpg/w='} 2024-03-28T16:13:27.761 DEBUG:teuthology.nuke:shortname: smithi150 2024-03-28T16:13:27.761 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-28T16:13:27.781 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi150.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625322', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 15:30:11.383491', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKoK70VPdmC0/eaUeMW21Z+c7KnYoeDaBs5A+2KtrsJOjJKEZ7ly0mD2SHeQpR/0WAIJQPxBMGQVnQWzZw0qjng='} 2024-03-28T16:13:27.786 DEBUG:teuthology.nuke:shortname: smithi183 2024-03-28T16:13:27.787 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-28T16:13:27.811 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi183.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625322', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 15:30:11.380785', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEeOeHmr9R9+ygjpgtqLUQT+UiCsT5fjGaFjqEiHSMmwGHnuZq/Ym1IW+aDBvoirIO+eDo7o3EdxLx0EEdGZCP8='} 2024-03-28T16:13:27.818 INFO:teuthology.orchestra.console.smithi067:Power off 2024-03-28T16:13:27.819 DEBUG:teuthology.orchestra.console.smithi067:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-28T16:13:27.845 DEBUG:teuthology.orchestra.console.smithi067:power off output: Chassis Power Control: Down/Off 2024-03-28T16:13:27.845 DEBUG:teuthology.orchestra.console.smithi067:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T16:13:27.863 INFO:teuthology.orchestra.console.smithi150:Power off 2024-03-28T16:13:27.863 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-28T16:13:27.877 DEBUG:teuthology.orchestra.console.smithi067:check power output: Chassis Power is on 2024-03-28T16:13:27.885 DEBUG:teuthology.orchestra.console.smithi150:power off output: Chassis Power Control: Down/Off 2024-03-28T16:13:27.886 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T16:13:27.909 INFO:teuthology.orchestra.console.smithi183:Power off 2024-03-28T16:13:27.909 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-28T16:13:27.919 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is on 2024-03-28T16:13:27.933 DEBUG:teuthology.orchestra.console.smithi183:power off output: Chassis Power Control: Down/Off 2024-03-28T16:13:27.933 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T16:13:27.957 DEBUG:teuthology.orchestra.console.smithi183:check power output: Chassis Power is on 2024-03-28T16:13:31.878 DEBUG:teuthology.orchestra.console.smithi067:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T16:13:31.921 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T16:13:31.958 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T16:13:31.989 DEBUG:teuthology.orchestra.console.smithi067:check power output: Chassis Power is on 2024-03-28T16:13:32.032 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is on 2024-03-28T16:13:32.068 DEBUG:teuthology.orchestra.console.smithi183:check power output: Chassis Power is on 2024-03-28T16:13:35.990 DEBUG:teuthology.orchestra.console.smithi067:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T16:13:36.032 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T16:13:36.069 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T16:13:36.103 DEBUG:teuthology.orchestra.console.smithi067:check power output: Chassis Power is on 2024-03-28T16:13:36.143 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is off 2024-03-28T16:13:36.181 DEBUG:teuthology.orchestra.console.smithi183:check power output: Chassis Power is on 2024-03-28T16:13:36.244 INFO:teuthology.orchestra.console.smithi150:Power off completed 2024-03-28T16:13:40.106 DEBUG:teuthology.orchestra.console.smithi067:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T16:13:40.182 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T16:13:40.220 DEBUG:teuthology.orchestra.console.smithi067:check power output: Chassis Power is off 2024-03-28T16:13:40.293 DEBUG:teuthology.orchestra.console.smithi183:check power output: Chassis Power is off 2024-03-28T16:13:40.321 INFO:teuthology.orchestra.console.smithi067:Power off completed 2024-03-28T16:13:40.394 INFO:teuthology.orchestra.console.smithi183:Power off completed 2024-03-28T16:13:40.500 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/iozone}} duration: 2007.346562385559 failure_reason: '"2024-03-28T15:58:21.700776+0000 mon.a (mon.0) 683 : cluster [WRN] Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED)" in cluster log' flavor: default mon.a-kernel-sha1: 1df0c7e8a2354c42d738e5454ba83ea7332deac2 mon.b-kernel-sha1: 1df0c7e8a2354c42d738e5454ba83ea7332deac2 mon.c-kernel-sha1: 1df0c7e8a2354c42d738e5454ba83ea7332deac2 owner: mchangir success: false 2024-03-28T16:13:40.501 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-28T16:13:40.597 INFO:teuthology.run:FAIL